system

Abstract

We provide the system. [Solution] A camera system that captures images in real time, An AI analysis method that analyzes captured images and counts the number of people within the area, A recording means for saving the analysis results to a database, A management system that retrieves congestion status from stored data and allocates resources, A system that includes a notification mechanism to inform users of congestion status and resource allocation.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a modern office environment, there is a problem that congestion occurs on specific weekdays or time zones, making it difficult for employees to work efficiently. In particular, the shortage of meeting rooms and workstations causes a decline in work efficiency. Also, since the congestion situation of rooms or seats cannot be grasped in advance, wasted movement and waiting time are likely to occur. Therefore, there is a need for a system that visualizes the congestion situation of an office in real time and performs appropriate resource allocation.

Means for Solving the Problems

[0005] This invention provides a camera means for capturing images in real time, and an AI analysis means for analyzing the captured images and counting the number of people in an area. Furthermore, it includes a recording means for saving the analysis results to a database, and a management means for obtaining congestion status from the saved data and allocating resources. By notifying users of congestion status and resource allocation using a notification means, efficient office use can be promoted, and a more comfortable working environment can be created. In summary, this invention provides a system that visualizes office congestion and realizes efficient resource management.

[0006] A "camera device" is a device that captures images of a specified area in real time.

[0007] "AI analysis means" refers to a device or program that uses an artificial intelligence model capable of analyzing captured images and counting the number of people within a specified area.

[0008] "Recording means" refers to a device or program for storing data obtained as analysis results in a database.

[0009] "Management means" refers to a device or program that has the function of obtaining congestion status from a stored database and allocating resources.

[0010] "Notification means" refers to a device or program for notifying users of congestion status and resource allocation information.

[0011] "Image preprocessing" refers to the process of performing operations such as resizing and normalizing images before AI analysis.

[0012] "Resource allocation" is the process of dynamically arranging office resources such as meeting rooms and workspaces to ensure efficient use. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, when an emotion engine is combined. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a tagged processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a tagged RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a tagged storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention relates to a system that visualizes congestion levels in an office environment in real time and enables efficient resource management. Specific embodiments of this system are described below.

[0035] This system includes a camera for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, and a notification means for notifying users of congestion status and resource allocation.

[0036] Program Processing Description

[0037] Image capture

[0038] The terminal (camera) captures images in real time based on the specified camera ID. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[0039] Image analysis

[0040] Based on the image data received by the server, the number of people in the room is counted using AI analysis methods. Specifically, the images are first preprocessed (e.g., resizing, normalization), and then the preprocessed images are input into an AI model for analysis. The AI ​​model uses object detection technology and deep learning algorithms to detect people in the images and count their numbers.

[0041] Database update

[0042] The server uses AI analysis to obtain the number of people and the current timestamp, and stores this information in a database using recording devices. This allows the congestion status of each room and area to be stored as time-series data.

[0043] Retrieving congestion status

[0044] The server retrieves the latest congestion information from a stored database. This involves extracting the most recent number of people in each room or area and calculating the degree of congestion. The congestion level is expressed as the utilization rate for each room or area.

[0045] Resource allocation

[0046] Based on the acquired congestion information, the server dynamically allocates resources such as meeting rooms and office spaces using a management system. Resource allocation is performed by prioritizing the use of areas and rooms with low congestion levels. For example, if a meeting room is full, the server will suggest other available meeting rooms to the user.

[0047] notification

[0048] Using notification mechanisms, the server informs users of real-time congestion status and resource allocation information. This allows users to know in advance which rooms or seats are suitable and to use the facilities efficiently.

[0049] Specific example

[0050] Example 1: Analysis and notification of meeting room congestion status

[0051] A terminal (a camera installed in the meeting room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis and counts that there are currently 10 people in the meeting room. After saving the analysis results and timestamps to a database, the server retrieves the latest congestion status and notifies the user that "Meeting Room 1 is currently available for 10 people."

[0052] Example 2: Managing the work area and allocating resources

[0053] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and count the number of people currently in the area, which is 20. The analysis results are stored in a database to obtain congestion status, and if a certain level of congestion is exceeded, the server notifies users of other available seats. For example, it might notify users that "Seat A is currently available," providing them with an efficient work environment.

[0054] As described above, the system according to the present invention can provide users with a user-friendly work environment by enabling real-time visualization of congestion status and efficient resource management.

[0055] The following describes the processing flow.

[0056] Step 1:

[0057] The terminal (camera) captures images of a designated area. Based on a specific camera ID (e.g., Camera 1), the terminal acquires real-time images of the conference room or office area. The acquired image data is then sent to the server.

[0058] Step 2:

[0059] The server receives image data sent from the terminal. The server preprocesses the received image, performing necessary operations such as resizing and normalization.

[0060] Step 3:

[0061] The server analyzes the pre-processed images using AI analysis tools. Specifically, the images are input into an AI model (e.g., an object detection model), and the analysis begins. The AI ​​model detects people in the images and counts the number of each person.

[0062] Step 4:

[0063] The server retrieves the number of people information obtained as a result of the AI ​​model's analysis. Next, it retrieves the current timestamp and saves it to the database along with the analysis results.

[0064] Step 5:

[0065] The server retrieves the latest occupancy data from the database in real time to obtain congestion status. It then aggregates the latest occupancy information for each room and area and calculates the level of congestion.

[0066] Step 6:

[0067] The server manages resource allocation based on congestion levels. For example, if a meeting room is fully booked, it suggests an alternative, available meeting room to the user. It dynamically optimizes resource allocation by adjusting allocations according to congestion levels.

[0068] Step 7:

[0069] The server notifies users of congestion status and resource allocation information. Using notification methods, users are informed of the availability of meeting rooms and workspaces in real time. This allows users to efficiently utilize the most suitable resources.

[0070] By following these steps, this system can visualize office congestion in real time and enable efficient resource management.

[0071] (Example 1)

[0072] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0073] To efficiently manage congestion in office environments and improve user convenience, real-time situation monitoring and dynamic resource allocation are necessary. However, conventional methods make it difficult to intuitively grasp congestion levels and optimize resource use. Therefore, there is a need for an efficient resource management system based on real-time data.

[0074] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0075] In this invention, the server includes a shooting means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, and a storage means for saving the analysis results to a database. This makes it possible to grasp the congestion status of each area in real time and to allocate resources efficiently.

[0076] "Shooting means" refers to a device that uses a camera or video capture device to acquire images in real time.

[0077] "Artificial intelligence analysis means" refers to a system that includes artificial intelligence and algorithms for processing and analyzing received image data and for identifying and counting objects and people within the image.

[0078] "Storage means" refers to a device or system for recording and storing analysis results and related data in a storage system such as a database.

[0079] A "management system" is a control system that obtains congestion status based on stored data and allocates resources accordingly.

[0080] A "notification system" is a system for notifying users in real time about congestion levels and resource allocation information.

[0081] An "artificial intelligence model" is an algorithm trained to analyze image data based on deep learning or machine learning, and to detect objects and people.

[0082] "Congestion level" is an indicator that shows the density of people in a particular area, and is calculated based on real-time data.

[0083] This invention relates to a system that visualizes congestion levels in an office environment in real time and enables efficient resource management. This system includes a means for capturing images, an artificial intelligence analysis means, a storage means, a management means, and a notification means. Embodiments of this invention are described in detail below.

[0084] System Configuration

[0085] Photography methods

[0086] The terminal (camera) captures images in real time based on the specified camera ID. For example, cameras installed in an office conference room or work area take pictures every second and send the image data to the server. The cameras used consist of general IP cameras and high-resolution cameras.

[0087] Artificial intelligence analysis methods

[0088] The server counts the number of people using AI analysis based on image data received from the terminal. The images are first preprocessed (resizing and normalization), and then input into the AI ​​model for analysis. The AI ​​model is built using TENSORFLOW® and PyTorch, and utilizes YOLO and SSD models as representative object detection techniques. Through analysis, people in the image are detected and counted.

[0089] Preservation means

[0090] The server stores the number of people and timestamps obtained by the AI ​​analysis method in a database. This database uses relational databases such as MySQL® or PostgreSQL. The stored data allows for the management of congestion levels in each room and area as time-series data.

[0091] management measures

[0092] The server retrieves the latest congestion information from the database. Specifically, it extracts the latest number of people in each room or area from the database and calculates the degree of congestion. For example, it can calculate the congestion level of a particular room as "50% utilization." Based on the congestion level, the server dynamically allocates resources. For example, if meeting room A is full, the server will suggest the available meeting room B to the user.

[0093] Notification means

[0094] The server uses notification methods to inform users in real time about congestion levels and resource allocation information. These notifications are sent as push notifications to smartphones and PCs. Notification services such as Firebase Cloud Messaging (FCM) are used for these notifications.

[0095] Specific example

[0096] Analysis and notification of meeting room congestion status

[0097] A terminal (a camera installed in the meeting room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis and counts that there are currently 10 people in the meeting room. After saving the analysis results and timestamps to a database, the server retrieves the latest congestion status and notifies the user that "Meeting Room 1 is currently available for 10 people."

[0098] Management of the work area and resource allocation

[0099] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and count the number of people currently in the area, which is 20. The analysis results are stored in a database to obtain congestion status, and if a certain level of congestion is exceeded, the server notifies users of other available seats. For example, it might notify users that "Seat A is currently available," providing them with an efficient work environment.

[0100] Thus, the system according to the present invention can provide users with a user-friendly work environment by enabling real-time visualization of congestion and efficient resource management.

[0101] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0102] Step 1: Image Capture

[0103] The device (camera) captures images in real time.

[0104] Input: Camera ID, current timestamp

[0105] Operation: The camera takes a picture in JPEG format every second and sends the image data to the server as an HTTP request.

[0106] Output: Sending JPEG image data to the server

[0107] Step 2: Receiving Image Data

[0108] The server receives image data sent from the terminal.

[0109] Input: JPEG image data sent from the terminal, HTTP request

[0110] Operation: The server parses the payload of the received HTTP request and temporarily saves the JPEG image to a directory.

[0111] Output: Saved JPEG image file

[0112] Step 3: Image preprocessing

[0113] The server preprocesses the received image data.

[0114] Input: Saved JPEG image file

[0115] Operation: The server uses the OpenCV library in Python to resize and normalize images. Specifically, it resizes images to 300x300 pixels and normalizes each pixel value to a range of 0 to 1.

[0116] Output: Resized and normalized image data

[0117] Step 4: Image Analysis

[0118] The server analyzes the pre-processed image data using artificial intelligence analysis tools.

[0119] Input: Resized and normalized image data

[0120] Operation: The server inputs image data into an AI model (e.g., YOLOv3) using TensorFlow or PyTorch and performs object detection. As an analysis result, it obtains the coordinates and number of people in the image.

[0121] Output: Person detection results (e.g., person coordinates, number of people data)

[0122] Step 5: Database Update

[0123] The server saves the analysis results (number of people) and the current timestamp to the database.

[0124] Input: Person detection result, timestamp, camera ID

[0125] Operation: The server connects to a MySQL or PostgreSQL database and executes INSERT queries to save data. Specifically, it records the number of people, camera ID, and timestamp.

[0126] Output: Number of people and timestamp stored in the database

[0127] Step 6: Obtain congestion status

[0128] The server retrieves the latest congestion information from the database.

[0129] Input: Number of people in the database, timestamp

[0130] Operation: The server executes a SELECT query to extract the latest number of people data. It calculates the level of congestion and determines the room occupancy rate.

[0131] Output: Congestion level information (occupancy rate) for each room and area.

[0132] Step 7: Resource Allocation

[0133] The server dynamically allocates resources based on congestion levels.

[0134] Input: Congestion level information

[0135] Operation: The server prioritizes selecting less congested areas or rooms based on the level of congestion and generates suggestion information. For example, it creates dynamic allocation information such as, "If meeting room A is full, use meeting room B."

[0136] Output: Resource allocation information

[0137] Step 8: Notification

[0138] The server notifies the user of resource allocation information.

[0139] Input: Resource allocation information, user's device ID

[0140] Operation: Uses Firebase Cloud Messaging (FCM) and other tools to send notifications to smartphones and computers. Specifically, it sends messages such as "Meeting Room 1 is currently available for 10 people."

[0141] Output: Notification message sent to the user's device

[0142] (Application Example 1)

[0143] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0144] Traditional food delivery systems struggled to accurately track restaurant congestion and cooking times, making it difficult to provide customers with accurate waiting times. Furthermore, delivery personnel were unable to be instructed on optimal routes or priorities, requiring efficient operations.

[0145] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0146] In this invention, the server includes an image acquisition means for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a resource management means for obtaining congestion status from the saved data and allocating resources, a notification means for notifying users of congestion status and resource allocation, a restaurant management means for monitoring the cooking status and waiting time of restaurants and informing customers of the optimal ordering timing and waiting time, and a delivery management means for optimizing order priorities and delivery routes based on congestion status. This enables accurate waiting time provision and efficient resource management.

[0147] "Image acquisition means" refers to devices or systems for capturing images in real time.

[0148] "AI analysis means" refers to devices or systems that use artificial intelligence technology to analyze captured images and count the number of people within a given area.

[0149] "Recording means" refers to devices or mechanisms for saving analysis results to a database.

[0150] A "resource management means" is a device or mechanism for obtaining congestion status from stored data and allocating resources.

[0151] "Notification means" refers to devices or mechanisms for notifying users of congestion status and resource allocation.

[0152] "Restaurant management systems" refer to devices and mechanisms that monitor the cooking status and waiting times of restaurants and inform customers of the optimal timing for ordering and waiting times.

[0153] A "delivery management system" refers to a device or mechanism for optimizing order priorities and delivery routes based on congestion levels.

[0154] The system for carrying out this invention is configured as follows.

[0155] Hardware and software configuration

[0156] Image acquisition method:

[0157] Use cameras (e.g., IP cameras, webcams, etc.) to capture images in real time. These cameras will be installed in the cooking area and waiting area of ​​the restaurant to capture current video.

[0158] AI analysis means:

[0159] A server will be set up to analyze the received image data. This server will be equipped with an AI model that performs object detection technology (e.g., YOLO, SSD). Python libraries (e.g., OpenCV, TensorFlow, PyTorch) will be used for image preprocessing and analysis software.

[0160] Recording means:

[0161] To store the analysis results in a database, a relational database such as MySQL or PostgreSQL is used. This database is used to manage time-series data for each area.

[0162] Resource management methods:

[0163] This includes server processing logic for retrieving congestion status from stored data and dynamically allocating resources. Specifically, it uses programming languages ​​such as Python and Java (registered trademark) to calculate congestion information and perform optimal resource allocation.

[0164] Notification method:

[0165] Smartphone apps and web applications are being developed to provide users with real-time notifications. These applications will deliver information on congestion levels and resource allocation via push notifications and email notifications.

[0166] Restaurant management means:

[0167] This system includes features to monitor the cooking status and waiting times in restaurants, and to inform customers of the optimal timing for ordering and waiting times. This primarily works in conjunction with the AI ​​analysis methods described above.

[0168] Delivery management methods:

[0169] This includes a system for optimizing order priorities and delivery routes based on congestion levels, resulting in more efficient operations.

[0170] Processing flow and data processing

[0171] The server receives the image and uses AI analysis to analyze the number of people and cooking status within the image. This data is stored in a recording system and processed by a resource management system. Congestion status and resource allocation information are notified to users and delivery personnel in real time via a notification system. Each process involves specific hardware and software, ensuring efficient data calculation and processing.

[0172] Specific example

[0173] Example 1: Notification of waiting time during cooking

[0174] A customer places an order at a restaurant. A camera captures the kitchen and sends the image to a server. The server uses the YOLO model to count the number of orders being prepared and predict the waiting time. This information is then sent to the customer's smartphone.

[0175] Prompt message: "Please check the cooking status and waiting times of currently busy restaurants."

[0176] Example 2: Optimizing delivery routes

[0177] A delivery driver receives multiple orders. The server analyzes the received images and calculates the optimal delivery route based on congestion levels. This information is then sent to the delivery driver's smartphone, enabling efficient delivery.

[0178] Prompt: "Calculate the optimal delivery route based on traffic conditions."

[0179] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0180] Step 1:

[0181] Image capture

[0182] The terminal (camera) captures images of the cooking area and waiting area inside the restaurant in real time. The captured image data is transmitted to a server via the network.

[0183] Input: Camera video

[0184] Output: Captured image data

[0185] Specific operation: The camera captures video frame by frame and sends the image data to the server.

[0186] Step 2:

[0187] Image preprocessing

[0188] The server preprocesses the received image data before AI analysis. Specifically, it resizes the images, normalizes them, and converts them into a format suitable for the AI ​​model.

[0189] Input: Captured image data

[0190] Output: Preprocessed image data

[0191] Specific operation: Use OpenCV to perform image preprocessing such as resizing and normalization.

[0192] Step 3:

[0193] AI analysis

[0194] The server inputs pre-processed image data into an AI model to detect people and cooking conditions within the images. Specifically, it uses object detection models such as YOLO and SSD.

[0195] Input: Preprocessed image data

[0196] Output: Analysis results (location and number of people, cooking status)

[0197] Specific operation: The AI ​​model detects people and objects in the image and outputs that information.

[0198] Step 4:

[0199] Save to database

[0200] The server saves the analysis results and timestamps to a database. This records the congestion and cooking status of each area as time-series data.

[0201] Input: Analysis results, timestamp

[0202] Output: Analysis results saved in the database

[0203] Specific operation: Execute INSERT statements into a database using MySQL or PostgreSQL.

[0204] Step 5:

[0205] Retrieving congestion status

[0206] The server retrieves the latest congestion information from the database and calculates the latest personnel information and congestion level for each area.

[0207] Input: Database analysis results

[0208] Output: Latest congestion data

[0209] Specific operation: Retrieve the latest records using a query and run a script to calculate the level of congestion.

[0210] Step 6:

[0211] Resource allocation

[0212] The server dynamically assigns order priorities and delivery routes based on congestion levels. If congestion is high, it takes measures such as suggesting an alternative delivery route.

[0213] Input: Latest congestion data

[0214] Output: Resource allocation instructions

[0215] Specific operation: Execute the algorithm using a Python script to determine efficient resource allocation.

[0216] Step 7:

[0217] notification

[0218] The server notifies users and delivery drivers of the latest congestion status, waiting times, and delivery route information. Notifications are sent via smartphone apps and web applications.

[0219] Input: Resource allocation instruction

[0220] Output: Notification Information

[0221] Specific operation: Use APIs for push notifications and email notifications to send information in real time.

[0222] Step 8:

[0223] Display of the user interface

[0224] Users can check current congestion levels, waiting times, and recommended delivery routes through smartphone apps and web applications.

[0225] Input: Notification information

[0226] Output: Information displayed on the screen

[0227] Specific operation: The application's frontend retrieves the latest information and displays it in the user interface.

[0228] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0229] This invention combines a system that visualizes congestion levels in an office environment in real time and enables efficient resource management with an emotion engine that recognizes user emotions. Specific embodiments of this system are described below.

[0230] This system includes a camera for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, a notification means for notifying users of congestion status and resource allocation, and an emotion engine for analyzing user emotions.

[0231] Program Processing Description

[0232] Image capture

[0233] The terminal (camera) captures images of a designated area in real time. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[0234] Image analysis

[0235] Based on the image data received by the server, the number of people in the room is counted using AI analysis methods. Specifically, the images are first preprocessed (e.g., resizing, normalization), and then the preprocessed images are input into an AI model for analysis. The AI ​​model uses object detection technology and deep learning algorithms to detect people in the images and count their numbers.

[0236] Database update

[0237] The server uses AI analysis to obtain the number of people and the current timestamp, and stores this information in a database using recording devices. This allows the congestion status of each room and area to be stored as time-series data.

[0238] Emotion analysis

[0239] The server uses an emotion engine to analyze the user's emotional state. Specifically, it inputs image data acquired from the camera and audio data acquired from the microphone into the emotion engine, and identifies the user's emotional state from their facial expressions and tone of voice.

[0240] Retrieving congestion status

[0241] The server retrieves the latest congestion information from a stored database. This involves extracting the most recent number of people in each room or area and calculating the degree of congestion. The congestion level is expressed as the utilization rate for each room or area.

[0242] Resource allocation

[0243] Based on acquired congestion levels and emotional states, the server dynamically allocates resources in meeting rooms and workspaces using a management system. Resource allocation not only prioritizes the use of less congested areas and rooms, but also provides optimal resources according to the user's emotional state. For example, it assigns quieter areas to users experiencing stress.

[0244] notification

[0245] Using notification mechanisms, the server informs users of real-time congestion and resource allocation information. This allows users to know in advance which rooms or seats are suitable and to use them efficiently. Furthermore, it can provide users with resource recommendations and customized notifications based on their emotional state.

[0246] Specific example

[0247] Example 1: Analysis of conference room congestion and sentiment analysis

[0248] A terminal (a camera installed in the conference room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis tools and counts that there are currently 10 people in the conference room. After saving the analysis results and timestamps to a database, the server uses an emotion engine to analyze the users' emotions from the video and detects that many users are in a stressed state. Based on the congestion level and emotional state, the server notifies users that "Conference Room 1 is currently available with 10 people, and the stress level is high."

[0249] Example 2: Managing work areas and assigning resources based on emotional state

[0250] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and counts that there are currently 20 people in the area. After saving the analysis results to a database to obtain the congestion status, the server uses an emotion engine to analyze the users' stress levels. Based on the congestion status and emotional state, the server assigns a quiet area to users who are stressed and notifies them with a message such as, "Seat A is currently vacant. It is a quiet area that can help reduce stress."

[0251] As described above, the system according to the present invention can achieve real-time visualization of congestion status and efficient resource management, as well as optimal resource allocation and notification that takes into account the user's emotional state.

[0252] The following describes the processing flow.

[0253] Step 1:

[0254] The terminal (camera) captures images of a designated area in real time. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[0255] Step 2:

[0256] The server receives image data sent from the terminal. The server preprocesses the received image, performing necessary operations such as resizing and normalization.

[0257] Step 3:

[0258] The server analyzes the pre-processed images using AI analysis tools. Specifically, the images are input into an AI model (e.g., an object detection model), and the analysis begins. The AI ​​model detects people in the images and counts the number of each person.

[0259] Step 4:

[0260] The server retrieves the number of people information obtained as a result of the AI ​​model's analysis. Next, it retrieves the current timestamp and saves it to the database along with the analysis results.

[0261] Step 5:

[0262] The server uses an emotion engine to analyze the user's emotional state. Image data acquired from the camera and audio data acquired from the microphone are input into the emotion engine, which then identifies the user's emotional state from their facial expressions and tone of voice.

[0263] Step 6:

[0264] The server retrieves the latest occupancy data from the database in real time to obtain congestion status. It then aggregates the latest occupancy information for each room and area and calculates the level of congestion.

[0265] Step 7:

[0266] The server manages resource allocation based on congestion levels and user emotions. For example, if a meeting room is full, it suggests an alternative, available meeting room to the user. It dynamically optimizes resources by adjusting allocations according to congestion levels. It also prioritizes assigning quieter areas to users who are stressed.

[0267] Step 8:

[0268] The server notifies users of congestion status and resource allocation information. It uses notification methods to inform users in real time about the availability of meeting rooms and workspaces. It also provides resource suggestions and customized notifications based on the user's emotional state. For example, it might provide users with a specific notification such as, "Meeting Room A is currently available. It's a quiet area that can help reduce stress."

[0269] Through the steps described above, this system can visualize office congestion and user emotional states in real time, enabling efficient and optimal resource management and notifications.

[0270] (Example 2)

[0271] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0272] Traditional office resource management systems fail to adequately visualize congestion levels and efficiently allocate resources. Furthermore, they cannot provide resources while considering users' emotional states, making it difficult to properly manage user satisfaction and stress levels. Therefore, it is necessary to efficiently utilize office space resources while optimally allocating resources in accordance with users' emotional states.

[0273] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0274] In this invention, the server includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, an emotion recognition means for analyzing the emotional state of users, and a notification means for notifying users of congestion status and resource allocation. This enables real-time visualization of congestion status and efficient resource management, as well as optimal resource allocation that takes into account the emotional state of users.

[0275] "Real-time" refers to a process or response that occurs almost instantly, with virtually no time delay.

[0276] "Imaging means" refers to a device that captures images or videos of a designated area. A camera is a specific example of this.

[0277] "Artificial intelligence analysis means" refers to artificial intelligence technology used to analyze captured images and obtain specific information (for example, counting the number of people).

[0278] An "artificial intelligence model" refers to a model used for analysis that possesses algorithms and structures employed by artificial intelligence analysis tools. Deep learning models are an example of this type of model.

[0279] "Recording means" refers to a device or system for saving analysis results to a database or similar.

[0280] The "database" refers to a digital system for organizing the storage and management of analysis results and other information.

[0281] The "emotion recognition means" refers to a system or technology for analyzing the emotional state of a user from their facial expressions, voice, etc.

[0282] The "management means" refers to a system or device for overall management of resource allocation and the operation of the entire system.

[0283] The "notification means" refers to a method or device for transmitting the analysis results of the system and resource allocation information to the user. Specifically, this includes emails and push notifications.

[0284] The "congestion situation" refers to a state indicating the density of the number of people and the utilization rate within a specific area.

[0285] "Resource allocation" refers to the act of efficiently and appropriately distributing the available resources within the system (e.g., meeting rooms and workstations).

[0286] The "user" refers to an individual or group that uses this system to manage and utilize the resources within the office.

[0287] The present invention is a system that visualizes the congestion situation in the office environment in real time and realizes efficient resource management, and further has an emotion recognition function for recognizing the emotions of users. A specific example of this system will be described below.

[0288] This system includes imaging means for capturing images in real time, artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, recording means for storing the analysis results in a database, management means for obtaining the congestion situation from the stored data and performing resource allocation, emotion recognition means for analyzing the emotions of the user, and notification means for notifying the user of the congestion situation and resource allocation.

[0289] The terminal (camera) captures images of a designated area, such as an office conference room or work area, in real time. The camera is high-resolution and periodically captures images, sending the data to a server. A network (e.g., Wi-Fi) is used for data transmission.

[0290] The server preprocesses the received image data (e.g., resizing, normalization) and counts the number of people in the image using AI analysis tools. Specifically, it uses artificial intelligence models (e.g., YOLO, OpenCV) to perform object detection and person counting. The analysis results are stored in a database (e.g., MySQL, PostgreSQL).

[0291] Furthermore, the server uses emotion recognition means (e.g., emotion recognition API) to analyze captured image and audio data and identify the user's emotional state. This allows for the determination of states such as stress or relaxation.

[0292] The server retrieves the latest congestion information from a stored database and calculates the degree of congestion. Based on the latest number of people in each area, the congestion status is visualized as a utilization rate. Resource allocation is dynamically performed using management tools based on congestion status and sentiment.

[0293] For example, resources can be allocated to less crowded areas or according to the emotional state of users. For instance, users experiencing stress could be assigned to quieter areas.

[0294] Using notification methods, the server informs users of real-time congestion status and resource allocation information. Notifications are sent via email, smartphone app push notifications, etc. This allows users to not only use rooms and seats efficiently, but also receive resource recommendations based on their emotional state.

[0295] Specific example

[0296] Example 1: Analysis of conference room congestion and sentiment analysis

[0297] A terminal (a camera installed in the conference room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis tools (e.g., YOLO) and counts that there are currently 10 people in the conference room. After saving the analysis results and timestamps to a database, the server uses emotion recognition tools to analyze the emotions of the users from the video and detects that many users are in a stressed state. Based on the congestion level and emotional state, the server notifies users that "Conference Room 1 is currently available with 10 people, and the stress level is high."

[0298] Example 2: Managing work areas and assigning resources based on emotional state

[0299] Cameras installed in the work area capture images and send them to a server. The server analyzes the images using AI analysis tools (e.g., OpenCV) and counts that there are currently 20 people in the area. After saving the analysis results to a database to obtain the congestion status, the server analyzes the stress levels of users using emotion recognition tools. Based on the congestion status and emotional state, the server assigns a quiet area to users who are stressed and notifies them with a message such as, "Seat A is currently vacant. It is a quiet area that can help reduce stress."

[0300] Examples of prompts for generative AI models

[0301] Prompt message 1:

[0302] "Please provide a detailed explanation of the system's process flow, which analyzes the current congestion level and user emotional state of office meeting rooms in real time to determine the optimal resource allocation."

[0303] Prompt message 2:

[0304] "Please detail the specific process for allocating resources to quieter areas based on the level of congestion in the work area and the stress levels of users."

[0305] As described above, in addition to visualizing the congestion situation in real time and efficient resource management, the present invention can realize optimal resource allocation and notification considering the congestion situation and the emotional state of users.

[0306] The flow of specific processing in Example 2 will be described using FIG. 13.

[0307] Step 1:

[0308] The terminal (camera) captures an image of the designated area. The camera is set to capture an image at a high resolution of 1 image per minute. The input is the current camera video, and the output is the captured image data. As a specific operation, the camera acquires the video in the conference room of the office and generates the image file.

[0309] Step 2:

[0310] The terminal transmits the captured image data to the server. The input is the image data captured by the camera, and the output is the image data uploaded to the server. As a specific operation, the camera uploads the image file to the server using the HTTP protocol. Wi-Fi connection is used for data transmission.

[0311] Step 3:

[0312] The server preprocesses the received image data. Specifically, image resizing and normalization are performed. The input is the image data uploaded to the server, and the output is the preprocessed image data. As a specific operation, the resolution of the image is standardized and color normalization is performed.

[0313] Step 4:

[0314] The server analyzes pre-processed image data using artificial intelligence analysis tools. The input is pre-processed image data, and the output is the count of people within a given area. Specifically, the server uses the YOLO model to detect people in the image and counts their numbers.

[0315] Step 5:

[0316] The server saves the analysis results to a database. The input is the AI ​​analysis results (number of people count) and a timestamp, and the output is time-series data stored in the database. Specifically, the server saves the data to a database such as MySQL using the INSERT command.

[0317] Step 6:

[0318] The server analyzes the user's emotional state using emotion recognition technology. The input is captured image and audio data, and the output is the emotion analysis result. Specifically, the server sends data to an emotion recognition API, which identifies emotions from the user's facial expressions and tone of voice.

[0319] Step 7:

[0320] The server retrieves the latest congestion status from a stored database. The input is the latest user count data in the database, and the output is the congestion level. Specifically, the server executes an SQL query to extract the latest data from the database and calculates the utilization rate.

[0321] Step 8:

[0322] The server allocates resources based on congestion levels and sentiment levels. Inputs are congestion data and sentiment analysis results, while output is the resource allocation result. Specifically, the server dynamically selects the most suitable meeting room or area and allocates resources accordingly.

[0323] Step 9:

[0324] The server notifies users of information using a notification system. The input is resource allocation results and congestion status data, and the output is a notification message. Specifically, the server sends notification emails to users using a mail server and, if necessary, sends push notifications to smartphone applications.

[0325] (Application Example 2)

[0326] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0327] In modern brick-and-mortar stores, understanding customer congestion in real time and using that information to manage resources efficiently is a crucial challenge. Furthermore, providing services that consider the emotional state of visitors is expected to improve customer satisfaction. However, conventional systems have struggled to simultaneously achieve both real-time analysis and resource allocation based on emotional state. To solve this problem, a system is needed that performs both congestion and emotional analysis in real time and allocates resources appropriately for each visitor.

[0328] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0329] In this invention, the server includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a management means for obtaining congestion status from stored data and allocating resources, an emotion analysis means for analyzing the emotions of users, a notification means for notifying users of congestion status and resource allocation, and a means for recommending resources according to the emotional state of users. This makes it possible to visualize the congestion status in a physical store in real time and to dynamically allocate the optimal resources based on the emotional state of visitors.

[0330] An "imaging means" is a device for capturing images in real time.

[0331] "Artificial intelligence analysis means" refers to AI technology that analyzes captured images and counts the number of people in those images.

[0332] "Recording means" refers to a function for saving analysis results and acquired data to a database.

[0333] "Management means" refers to a function that retrieves congestion status from stored data and dynamically allocates resources.

[0334] "Emotion analysis means" refers to technology that analyzes a user's image and audio data to identify their emotional state.

[0335] "Notification means" refers to a function that notifies users in real time about congestion status and resource allocation information.

[0336] A "resource recommendation mechanism" is a function that dynamically suggests the most suitable resources based on the user's emotional state.

[0337] This invention relates to a system that visualizes congestion levels in physical stores in real time, enabling efficient resource management and service provision based on customer sentiment. The system includes imaging means, artificial intelligence analysis means, recording means, management means, sentiment analysis means, notification means, and resource recommendation means.

[0338] This system uses cameras as an imaging means to capture images in real time. For example, surveillance cameras installed in each area of ​​a physical store acquire current video footage and transmit that video data to a server.

[0339] The server uses artificial intelligence analysis to count the number of people in the area based on the received video data. This process utilizes pre-trained deep learning models and object detection algorithms. Specifically, the images are first pre-processed, such as resizing and normalization, and then the pre-processed images are input into the artificial intelligence model for analysis.

[0340] The analysis results are stored in a database using recording devices. For example, the number of people and timestamps for each area are recorded as time-series data and used for subsequent congestion analysis.

[0341] Next, the server analyzes the customer's emotional state using emotion analysis tools. This involves identifying the customer's emotional state from their facial expressions and tone of voice using video data acquired by the camera and audio data acquired by the microphone. A separate deep learning model is used for emotion analysis to estimate various emotions (e.g., stress, joy, etc.).

[0342] The management system retrieves the latest congestion status from stored data and dynamically allocates resources along with the user's emotional state. For example, if a particular area is congested or a customer's stress level is high, it will prioritize recommending less crowded or quieter areas.

[0343] Using notification mechanisms, the server informs users in real time about congestion levels and resource allocation. This allows users to select appropriate areas and take actions to avoid congestion. Furthermore, it provides customized services based on the customer's emotional state through resource recommendation mechanisms.

[0344] Specific example

[0345] Example 1: Analysis of congestion levels and sentiment analysis in physical stores

[0346] A terminal (a camera installed in the physical store) captures images in real time and sends them to a server. The server analyzes the images using artificial intelligence analysis and counts that there are currently 50 people in the store. After saving the analysis results and timestamps to a database, the server uses emotion analysis to analyze the emotions of the customers from the video and detects that many customers are in a stressed state. Based on the congestion level and emotional state, the server notifies store staff and customers that "Store area A is currently available with 50 people, and the stress level is high."

[0347] Example 2: Managing dedicated areas and assigning resources based on emotional state

[0348] Cameras installed in a designated area capture images and send them to a server. The server uses artificial intelligence analysis to analyze the images and counts that there are currently 30 people in the area. After saving the analysis results to a database to obtain the congestion status, the server uses emotion analysis to analyze the stress levels of the customers. Based on the congestion status and emotional state, the server assigns a quiet area to customers who are stressed and notifies them with a message such as, "Area B is currently empty. It is a quiet area that can help reduce stress."

[0349] Example of a prompt:

[0350] "Please explain how to apply a system that visualizes office congestion levels in real time."

[0351] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0352] Step 1:

[0353] The device captures images in real time.

[0354] Input: Video data captured by the camera

[0355] Operation: Cameras installed in each area of ​​the store capture images at specified intervals and send the data to the server in streaming format.

[0356] Output: Captured image data

[0357] Step 2:

[0358] The server uses artificial intelligence analysis to count the number of people based on the image data it receives.

[0359] Input: Captured image data

[0360] Operation: The received image data is preprocessed, such as by resizing and normalizing, and the preprocessed image is input into a deep learning model. The AI ​​model uses an object detection algorithm to detect people in the image and counts the number of people.

[0361] Output: Data on the number of people counted

[0362] Step 3:

[0363] The server saves the analysis results to a database.

[0364] Input: Counted number of people data, timestamp

[0365] Operation: Records the analysis results (number of people data) and the current timestamp in the database.

[0366] Output: Updated database

[0367] Step 4:

[0368] The server uses emotion analysis tools to analyze the customer's emotional state.

[0369] Input: Captured image data, audio data (if necessary)

[0370] Operation: Image and audio data are input into another deep learning model to estimate the customer's emotional state from their facial expressions and tone of voice. Various emotions are identified (e.g., stress, joy), and analysis results are obtained.

[0371] Output: Customer sentiment data

[0372] Step 5:

[0373] The server retrieves the latest congestion information from stored data and dynamically allocates resources.

[0374] Input: Number of people data, emotion data

[0375] Operation: Retrieves the latest headcount and sentiment data from the database and uses management tools to allocate resources based on congestion levels and sentiment states. Selects areas with low congestion or areas suitable for the customer's sentiment state.

[0376] Output: Resource allocation data

[0377] Step 6:

[0378] The server notifies the user of the information it has obtained.

[0379] Input: Resource allocation data, congestion status data

[0380] Operation: Uses notification methods to provide users (customers and staff) with real-time information on congestion levels and resource allocation. For example, it may notify users that a specific area is available or that a quiet area is suitable for stress relief.

[0381] Output: Notified information

[0382] Example of a prompt:

[0383] "Please explain how to apply a system that visualizes office congestion levels in real time."

[0384] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0385] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0386] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0387] [Second Embodiment]

[0388] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0389] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0390] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0391] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0392] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0393] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0394] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0395] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0396] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0397] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0398] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0399] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0400] This invention relates to a system that visualizes congestion levels in an office environment in real time and enables efficient resource management. Specific embodiments of this system are described below.

[0401] This system includes a camera for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, and a notification means for notifying users of congestion status and resource allocation.

[0402] Program Processing Description

[0403] Image capture

[0404] The terminal (camera) captures images in real time based on the specified camera ID. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[0405] Image analysis

[0406] Based on the image data received by the server, the number of people in the room is counted using AI analysis methods. Specifically, the images are first preprocessed (e.g., resizing, normalization), and then the preprocessed images are input into an AI model for analysis. The AI ​​model uses object detection technology and deep learning algorithms to detect people in the images and count their numbers.

[0407] Database update

[0408] The server uses AI analysis to obtain the number of people and the current timestamp, and stores this information in a database using recording devices. This allows the congestion status of each room and area to be stored as time-series data.

[0409] Retrieving congestion status

[0410] The server retrieves the latest congestion information from a stored database. This involves extracting the most recent number of people in each room or area and calculating the degree of congestion. The congestion level is expressed as the utilization rate for each room or area.

[0411] Resource allocation

[0412] Based on the acquired congestion information, the server dynamically allocates resources such as meeting rooms and office spaces using a management system. Resource allocation is performed by prioritizing the use of areas and rooms with low congestion levels. For example, if a meeting room is full, the server will suggest other available meeting rooms to the user.

[0413] notification

[0414] Using notification mechanisms, the server informs users of real-time congestion status and resource allocation information. This allows users to know in advance which rooms or seats are suitable and to use the facilities efficiently.

[0415] Specific example

[0416] Example 1: Analysis and notification of meeting room congestion status

[0417] A terminal (a camera installed in the meeting room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis and counts that there are currently 10 people in the meeting room. After saving the analysis results and timestamps to a database, the server retrieves the latest congestion status and notifies the user that "Meeting Room 1 is currently available for 10 people."

[0418] Example 2: Managing the work area and allocating resources

[0419] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and count the number of people currently in the area, which is 20. The analysis results are stored in a database to obtain congestion status, and if a certain level of congestion is exceeded, the server notifies users of other available seats. For example, it might notify users that "Seat A is currently available," providing them with an efficient work environment.

[0420] As described above, the system according to the present invention can provide users with a user-friendly work environment by enabling real-time visualization of congestion status and efficient resource management.

[0421] The following describes the processing flow.

[0422] Step 1:

[0423] The terminal (camera) captures images of a designated area. Based on a specific camera ID (e.g., Camera 1), the terminal acquires real-time images of the conference room or office area. The acquired image data is then sent to the server.

[0424] Step 2:

[0425] The server receives image data sent from the terminal. The server preprocesses the received image, performing necessary operations such as resizing and normalization.

[0426] Step 3:

[0427] The server analyzes the pre-processed images using AI analysis tools. Specifically, the images are input into an AI model (e.g., an object detection model), and the analysis begins. The AI ​​model detects people in the images and counts the number of each person.

[0428] Step 4:

[0429] The server retrieves the number of people information obtained as a result of the AI ​​model's analysis. Next, it retrieves the current timestamp and saves it to the database along with the analysis results.

[0430] Step 5:

[0431] The server retrieves the latest occupancy data from the database in real time to obtain congestion status. It then aggregates the latest occupancy information for each room and area and calculates the level of congestion.

[0432] Step 6:

[0433] The server manages resource allocation based on congestion levels. For example, if a meeting room is fully booked, it suggests an alternative, available meeting room to the user. It dynamically optimizes resource allocation by adjusting allocations according to congestion levels.

[0434] Step 7:

[0435] The server notifies users of congestion status and resource allocation information. Using notification methods, users are informed of the availability of meeting rooms and workspaces in real time. This allows users to efficiently utilize the most suitable resources.

[0436] By following these steps, this system can visualize office congestion in real time and enable efficient resource management.

[0437] (Example 1)

[0438] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0439] To efficiently manage congestion in office environments and improve user convenience, real-time situation monitoring and dynamic resource allocation are necessary. However, conventional methods make it difficult to intuitively grasp congestion levels and optimize resource use. Therefore, there is a need for an efficient resource management system based on real-time data.

[0440] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0441] In this invention, the server includes a shooting means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, and a storage means for saving the analysis results to a database. This makes it possible to grasp the congestion status of each area in real time and to allocate resources efficiently.

[0442] "Shooting means" refers to a device that uses a camera or video capture device to acquire images in real time.

[0443] "Artificial intelligence analysis means" refers to a system that includes artificial intelligence and algorithms for processing and analyzing received image data and for identifying and counting objects and people within the image.

[0444] "Storage means" refers to a device or system for recording and storing analysis results and related data in a storage system such as a database.

[0445] A "management system" is a control system that obtains congestion status based on stored data and allocates resources accordingly.

[0446] A "notification system" is a system for notifying users in real time about congestion levels and resource allocation information.

[0447] An "artificial intelligence model" is an algorithm trained to analyze image data based on deep learning or machine learning, and to detect objects and people.

[0448] "Congestion level" is an indicator that shows the density of people in a particular area, and is calculated based on real-time data.

[0449] This invention relates to a system that visualizes congestion levels in an office environment in real time and enables efficient resource management. This system includes a means for capturing images, an artificial intelligence analysis means, a storage means, a management means, and a notification means. Embodiments of this invention are described in detail below.

[0450] System Configuration

[0451] Photography methods

[0452] The terminal (camera) captures images in real time based on the specified camera ID. For example, cameras installed in an office conference room or work area take pictures every second and send the image data to the server. The cameras used consist of general IP cameras and high-resolution cameras.

[0453] Artificial intelligence analysis methods

[0454] The server counts the number of people using AI analysis based on image data received from the terminal. The images are first preprocessed (resizing and normalization), and then input into the AI ​​model for analysis. The AI ​​model is built using TensorFlow or PyTorch, and utilizes YOLO and SSD models as representative object detection techniques. Through analysis, people in the image are detected and counted.

[0455] Preservation means

[0456] The server stores the number of people and timestamps obtained through AI analysis into a database. This database uses a relational database such as MySQL or PostgreSQL. The stored data allows for the management of congestion levels in each room and area as time-series data.

[0457] management measures

[0458] The server retrieves the latest congestion information from the database. Specifically, it extracts the latest number of people in each room or area from the database and calculates the degree of congestion. For example, it can calculate the congestion level of a particular room as "50% utilization." Based on the congestion level, the server dynamically allocates resources. For example, if meeting room A is full, the server will suggest the available meeting room B to the user.

[0459] Notification means

[0460] The server uses notification methods to inform users in real time about congestion levels and resource allocation information. These notifications are sent as push notifications to smartphones and PCs. Notification services such as Firebase Cloud Messaging (FCM) are used for these notifications.

[0461] Specific example

[0462] Analysis and notification of meeting room congestion status

[0463] A terminal (a camera installed in the meeting room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis and counts that there are currently 10 people in the meeting room. After saving the analysis results and timestamps to a database, the server retrieves the latest congestion status and notifies the user that "Meeting Room 1 is currently available for 10 people."

[0464] Management of the work area and resource allocation

[0465] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and count the number of people currently in the area, which is 20. The analysis results are stored in a database to obtain congestion status, and if a certain level of congestion is exceeded, the server notifies users of other available seats. For example, it might notify users that "Seat A is currently available," providing them with an efficient work environment.

[0466] Thus, the system according to the present invention can provide users with a user-friendly work environment by enabling real-time visualization of congestion and efficient resource management.

[0467] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0468] Step 1: Image Capture

[0469] The device (camera) captures images in real time.

[0470] Input: Camera ID, current timestamp

[0471] Operation: The camera takes a picture in JPEG format every second and sends the image data to the server as an HTTP request.

[0472] Output: Sending JPEG image data to the server

[0473] Step 2: Receiving Image Data

[0474] The server receives image data sent from the terminal.

[0475] Input: JPEG image data sent from the terminal, HTTP request

[0476] Operation: The server parses the payload of the received HTTP request and temporarily saves the JPEG image to a directory.

[0477] Output: Saved JPEG image file

[0478] Step 3: Image preprocessing

[0479] The server preprocesses the received image data.

[0480] Input: Saved JPEG image file

[0481] Operation: The server uses the OpenCV library in Python to resize and normalize images. Specifically, it resizes images to 300x300 pixels and normalizes each pixel value to a range of 0 to 1.

[0482] Output: Resized and normalized image data

[0483] Step 4: Image Analysis

[0484] The server analyzes the pre-processed image data using artificial intelligence analysis tools.

[0485] Input: Resized and normalized image data

[0486] Operation: The server inputs image data into an AI model (e.g., YOLOv3) using TensorFlow or PyTorch and performs object detection. As an analysis result, it obtains the coordinates and number of people in the image.

[0487] Output: Person detection results (e.g., person coordinates, number of people data)

[0488] Step 5: Database Update

[0489] The server saves the analysis results (number of people) and the current timestamp to the database.

[0490] Input: Person detection result, timestamp, camera ID

[0491] Operation: The server connects to a MySQL or PostgreSQL database and executes INSERT queries to save data. Specifically, it records the number of people, camera ID, and timestamp.

[0492] Output: Number of people and timestamp stored in the database

[0493] Step 6: Obtain congestion status

[0494] The server retrieves the latest congestion information from the database.

[0495] Input: Number of people in the database, timestamp

[0496] Operation: The server executes a SELECT query to extract the latest number of people data. It calculates the level of congestion and determines the room occupancy rate.

[0497] Output: Congestion level information (occupancy rate) for each room and area.

[0498] Step 7: Resource Allocation

[0499] The server dynamically allocates resources based on congestion levels.

[0500] Input: Congestion level information

[0501] Operation: The server prioritizes selecting less congested areas or rooms based on the level of congestion and generates suggestion information. For example, it creates dynamic allocation information such as, "If meeting room A is full, use meeting room B."

[0502] Output: Resource allocation information

[0503] Step 8: Notification

[0504] The server notifies the user of resource allocation information.

[0505] Input: Resource allocation information, user's device ID

[0506] Operation: Uses Firebase Cloud Messaging (FCM) and other tools to send notifications to smartphones and computers. Specifically, it sends messages such as "Meeting Room 1 is currently available for 10 people."

[0507] Output: Notification message sent to the user's device

[0508] (Application Example 1)

[0509] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0510] Traditional food delivery systems struggled to accurately track restaurant congestion and cooking times, making it difficult to provide customers with accurate waiting times. Furthermore, delivery personnel were unable to be instructed on optimal routes or priorities, requiring efficient operations.

[0511] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0512] In this invention, the server includes an image acquisition means for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a resource management means for obtaining congestion status from the saved data and allocating resources, a notification means for notifying users of congestion status and resource allocation, a restaurant management means for monitoring the cooking status and waiting time of restaurants and informing customers of the optimal ordering timing and waiting time, and a delivery management means for optimizing order priorities and delivery routes based on congestion status. This enables accurate waiting time provision and efficient resource management.

[0513] "Image acquisition means" refers to devices or systems for capturing images in real time.

[0514] "AI analysis means" refers to devices or systems that use artificial intelligence technology to analyze captured images and count the number of people within a given area.

[0515] "Recording means" refers to devices or mechanisms for saving analysis results to a database.

[0516] A "resource management means" is a device or mechanism for obtaining congestion status from stored data and allocating resources.

[0517] "Notification means" refers to devices or mechanisms for notifying users of congestion status and resource allocation.

[0518] "Restaurant management systems" refer to devices and mechanisms that monitor the cooking status and waiting times of restaurants and inform customers of the optimal timing for ordering and waiting times.

[0519] A "delivery management system" refers to a device or mechanism for optimizing order priorities and delivery routes based on congestion levels.

[0520] The system for carrying out this invention is configured as follows.

[0521] Hardware and software configuration

[0522] Image acquisition method:

[0523] Use cameras (e.g., IP cameras, webcams, etc.) to capture images in real time. These cameras will be installed in the cooking area and waiting area of ​​the restaurant to capture current video.

[0524] AI analysis means:

[0525] A server will be set up to analyze the received image data. This server will be equipped with an AI model that performs object detection technology (e.g., YOLO, SSD). Python libraries (e.g., OpenCV, TensorFlow, PyTorch) will be used for image preprocessing and analysis software.

[0526] Recording means:

[0527] To store the analysis results in a database, a relational database such as MySQL or PostgreSQL is used. This database is used to manage time-series data for each area.

[0528] Resource management methods:

[0529] This includes server processing logic for retrieving congestion information from stored data and dynamically allocating resources. Specifically, it uses programming languages ​​such as Python or Java to calculate congestion information and perform optimal resource allocation.

[0530] Notification method:

[0531] Smartphone apps and web applications are being developed to provide users with real-time notifications. These applications will deliver information on congestion levels and resource allocation via push notifications and email notifications.

[0532] Restaurant management means:

[0533] This system includes features to monitor the cooking status and waiting times in restaurants, and to inform customers of the optimal timing for ordering and waiting times. This primarily works in conjunction with the AI ​​analysis methods described above.

[0534] Delivery management methods:

[0535] This includes a system for optimizing order priorities and delivery routes based on congestion levels, resulting in more efficient operations.

[0536] Processing flow and data processing

[0537] The server receives the image and uses AI analysis to analyze the number of people and cooking status within the image. This data is stored in a recording system and processed by a resource management system. Congestion status and resource allocation information are notified to users and delivery personnel in real time via a notification system. Each process involves specific hardware and software, ensuring efficient data calculation and processing.

[0538] Specific example

[0539] Example 1: Notification of waiting time during cooking

[0540] A customer places an order at a restaurant. A camera captures the kitchen and sends the image to a server. The server uses the YOLO model to count the number of orders being prepared and predict the waiting time. This information is then sent to the customer's smartphone.

[0541] Prompt message: "Please check the cooking status and waiting times of currently busy restaurants."

[0542] Example 2: Optimizing delivery routes

[0543] A delivery driver receives multiple orders. The server analyzes the received images and calculates the optimal delivery route based on congestion levels. This information is then sent to the delivery driver's smartphone, enabling efficient delivery.

[0544] Prompt: "Calculate the optimal delivery route based on traffic conditions."

[0545] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0546] Step 1:

[0547] Image capture

[0548] The terminal (camera) captures images of the cooking area and waiting area inside the restaurant in real time. The captured image data is transmitted to a server via the network.

[0549] Input: Camera video

[0550] Output: Captured image data

[0551] Specific operation: The camera captures video frame by frame and sends the image data to the server.

[0552] Step 2:

[0553] Image preprocessing

[0554] The server preprocesses the received image data before AI analysis. Specifically, it resizes the images, normalizes them, and converts them into a format suitable for the AI ​​model.

[0555] Input: Captured image data

[0556] Output: Preprocessed image data

[0557] Specific operation: Use OpenCV to perform image preprocessing such as resizing and normalization.

[0558] Step 3:

[0559] AI analysis

[0560] The server inputs pre-processed image data into an AI model to detect people and cooking conditions within the images. Specifically, it uses object detection models such as YOLO and SSD.

[0561] Input: Preprocessed image data

[0562] Output: Analysis results (location and number of people, cooking status)

[0563] Specific operation: The AI ​​model detects people and objects in the image and outputs that information.

[0564] Step 4:

[0565] Save to database

[0566] The server saves the analysis results and timestamps to a database. This records the congestion and cooking status of each area as time-series data.

[0567] Input: Analysis results, timestamp

[0568] Output: Analysis results saved in the database

[0569] Specific operation: Execute INSERT statements into a database using MySQL or PostgreSQL.

[0570] Step 5:

[0571] Retrieving congestion status

[0572] The server retrieves the latest congestion information from the database and calculates the latest personnel information and congestion level for each area.

[0573] Input: Database analysis results

[0574] Output: Latest congestion data

[0575] Specific operation: Retrieve the latest records using a query and run a script to calculate the level of congestion.

[0576] Step 6:

[0577] Resource allocation

[0578] The server dynamically assigns order priorities and delivery routes based on congestion levels. If congestion is high, it takes measures such as suggesting an alternative delivery route.

[0579] Input: Latest congestion data

[0580] Output: Resource allocation instructions

[0581] Specific operation: Execute the algorithm using a Python script to determine efficient resource allocation.

[0582] Step 7:

[0583] notification

[0584] The server notifies users and delivery drivers of the latest congestion status, waiting times, and delivery route information. Notifications are sent via smartphone apps and web applications.

[0585] Input: Resource allocation instruction

[0586] Output: Notification Information

[0587] Specific operation: Use APIs for push notifications and email notifications to send information in real time.

[0588] Step 8:

[0589] Display of the user interface

[0590] Users can check current congestion levels, waiting times, and recommended delivery routes through smartphone apps and web applications.

[0591] Input: Notification information

[0592] Output: Information displayed on the screen

[0593] Specific operation: The application's frontend retrieves the latest information and displays it in the user interface.

[0594] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0595] This invention combines a system that visualizes congestion levels in an office environment in real time and enables efficient resource management with an emotion engine that recognizes user emotions. Specific embodiments of this system are described below.

[0596] This system includes a camera for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, a notification means for notifying users of congestion status and resource allocation, and an emotion engine for analyzing user emotions.

[0597] Program Processing Description

[0598] Image capture

[0599] The terminal (camera) captures images of a designated area in real time. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[0600] Image analysis

[0601] Based on the image data received by the server, the number of people in the room is counted using AI analysis methods. Specifically, the images are first preprocessed (e.g., resizing, normalization), and then the preprocessed images are input into an AI model for analysis. The AI ​​model uses object detection technology and deep learning algorithms to detect people in the images and count their numbers.

[0602] Database update

[0603] The server uses AI analysis to obtain the number of people and the current timestamp, and stores this information in a database using recording devices. This allows the congestion status of each room and area to be stored as time-series data.

[0604] Emotion analysis

[0605] The server uses an emotion engine to analyze the user's emotional state. Specifically, it inputs image data acquired from the camera and audio data acquired from the microphone into the emotion engine, and identifies the user's emotional state from their facial expressions and tone of voice.

[0606] Retrieving congestion status

[0607] The server retrieves the latest congestion information from a stored database. This involves extracting the most recent number of people in each room or area and calculating the degree of congestion. The congestion level is expressed as the utilization rate for each room or area.

[0608] Resource allocation

[0609] Based on acquired congestion levels and emotional states, the server dynamically allocates resources in meeting rooms and workspaces using a management system. Resource allocation not only prioritizes the use of less congested areas and rooms, but also provides optimal resources according to the user's emotional state. For example, it assigns quieter areas to users experiencing stress.

[0610] notification

[0611] Using notification mechanisms, the server informs users of real-time congestion and resource allocation information. This allows users to know in advance which rooms or seats are suitable and to use them efficiently. Furthermore, it can provide users with resource recommendations and customized notifications based on their emotional state.

[0612] Specific example

[0613] Example 1: Analysis of conference room congestion and sentiment analysis

[0614] A terminal (a camera installed in the conference room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis tools and counts that there are currently 10 people in the conference room. After saving the analysis results and timestamps to a database, the server uses an emotion engine to analyze the users' emotions from the video and detects that many users are in a stressed state. Based on the congestion level and emotional state, the server notifies users that "Conference Room 1 is currently available with 10 people, and the stress level is high."

[0615] Example 2: Managing work areas and assigning resources based on emotional state

[0616] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and counts that there are currently 20 people in the area. After saving the analysis results to a database to obtain the congestion status, the server uses an emotion engine to analyze the users' stress levels. Based on the congestion status and emotional state, the server assigns a quiet area to users who are stressed and notifies them with a message such as, "Seat A is currently vacant. It is a quiet area that can help reduce stress."

[0617] As described above, the system according to the present invention can visualize congestion status in real time and manage resources efficiently, as well as achieve optimal resource allocation and notification that takes into account the user's emotional state.

[0618] The following describes the processing flow.

[0619] Step 1:

[0620] The terminal (camera) captures images of a designated area in real time. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[0621] Step 2:

[0622] The server receives image data sent from the terminal. The server preprocesses the received image, performing necessary operations such as resizing and normalization.

[0623] Step 3:

[0624] The server analyzes the pre-processed images using AI analysis tools. Specifically, the images are input into an AI model (e.g., an object detection model), and the analysis begins. The AI ​​model detects people in the images and counts the number of each person.

[0625] Step 4:

[0626] The server retrieves the number of people information obtained as a result of the AI ​​model's analysis. Next, it retrieves the current timestamp and saves it to the database along with the analysis results.

[0627] Step 5:

[0628] The server uses an emotion engine to analyze the user's emotional state. Image data acquired from the camera and audio data acquired from the microphone are input into the emotion engine, which then identifies the user's emotional state from their facial expressions and tone of voice.

[0629] Step 6:

[0630] The server retrieves the latest occupancy data from the database in real time to obtain congestion status. It then aggregates the latest occupancy information for each room and area and calculates the level of congestion.

[0631] Step 7:

[0632] The server manages resource allocation based on congestion levels and user emotions. For example, if a meeting room is full, it suggests an alternative, available meeting room to the user. It dynamically optimizes resources by adjusting allocations according to congestion levels. It also prioritizes assigning quieter areas to users who are stressed.

[0633] Step 8:

[0634] The server notifies users of congestion status and resource allocation information. It uses notification methods to inform users in real time about the availability of meeting rooms and workspaces. It also provides resource suggestions and customized notifications based on the user's emotional state. For example, it might provide users with a specific notification such as, "Meeting Room A is currently available. It's a quiet area that can help reduce stress."

[0635] Through the steps described above, this system can visualize office congestion and user emotional states in real time, enabling efficient and optimal resource management and notifications.

[0636] (Example 2)

[0637] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0638] Traditional office resource management systems fail to adequately visualize congestion levels and efficiently allocate resources. Furthermore, they cannot provide resources while considering users' emotional states, making it difficult to properly manage user satisfaction and stress levels. Therefore, it is necessary to efficiently utilize office space resources while optimally allocating resources in accordance with users' emotional states.

[0639] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0640] In this invention, the server includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, an emotion recognition means for analyzing the emotional state of users, and a notification means for notifying users of congestion status and resource allocation. This enables real-time visualization of congestion status and efficient resource management, as well as optimal resource allocation that takes into account the emotional state of users.

[0641] "Real-time" refers to a process or response that occurs almost instantly, with virtually no time delay.

[0642] "Imaging means" refers to a device that captures images or videos of a designated area. A camera is a specific example of this.

[0643] "Artificial intelligence analysis means" refers to artificial intelligence technology used to analyze captured images and obtain specific information (for example, counting the number of people).

[0644] An "artificial intelligence model" refers to a model used for analysis that possesses algorithms and structures employed by artificial intelligence analysis tools. Deep learning models are an example of this type of model.

[0645] "Recording means" refers to a device or system for saving analysis results to a database or similar.

[0646] A "database" refers to a digital system used to systematically store and manage analysis results and other information.

[0647] "Emotion recognition means" refers to a system or technology that analyzes a user's emotional state from their facial expressions, voice, etc.

[0648] "Management means" refers to a system or device that oversees and manages resource allocation and the operation of the entire system.

[0649] "Notification means" refers to methods or devices for communicating system analysis results and resource allocation information to users. Specifically, this includes email and push notifications.

[0650] "Congestion status" refers to the state of a particular area, indicating the density of people or the utilization rate.

[0651] "Resource allocation" refers to the act of efficiently and appropriately distributing available resources within a system (for example, meeting rooms or office desks).

[0652] "User" refers to an individual or organization that uses this system to manage and utilize resources within the office.

[0653] This invention provides a system that visualizes congestion levels in an office environment in real time, enabling efficient resource management, and further incorporates an emotion recognition function that recognizes the emotions of users. A specific example of this system is described below.

[0654] This system includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, an emotion recognition means for analyzing the emotions of users, and a notification means for notifying users of congestion status and resource allocation.

[0655] The terminal (camera) captures images of a designated area, such as an office conference room or work area, in real time. The camera is high-resolution and periodically captures images, sending the data to a server. A network (e.g., Wi-Fi) is used for data transmission.

[0656] The server preprocesses the received image data (e.g., resizing, normalization) and counts the number of people in the image using AI analysis tools. Specifically, it uses artificial intelligence models (e.g., YOLO, OpenCV) to perform object detection and person counting. The analysis results are stored in a database (e.g., MySQL, PostgreSQL).

[0657] Furthermore, the server uses emotion recognition means (e.g., emotion recognition API) to analyze captured image and audio data and identify the user's emotional state. This allows for the determination of states such as stress or relaxation.

[0658] The server retrieves the latest congestion information from a stored database and calculates the degree of congestion. Based on the latest number of people in each area, the congestion status is visualized as a utilization rate. Resource allocation is dynamically performed using management tools based on congestion status and sentiment.

[0659] For example, resources can be allocated to less crowded areas or according to the emotional state of users. For instance, users experiencing stress could be assigned to quieter areas.

[0660] Using notification methods, the server informs users of real-time congestion status and resource allocation information. Notifications are sent via email, smartphone app push notifications, etc. This allows users to not only use rooms and seats efficiently, but also receive resource recommendations based on their emotional state.

[0661] Specific example

[0662] Example 1: Analysis of conference room congestion and sentiment analysis

[0663] A terminal (a camera installed in the conference room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis tools (e.g., YOLO) and counts that there are currently 10 people in the conference room. After saving the analysis results and timestamps to a database, the server uses emotion recognition tools to analyze the emotions of the users from the video and detects that many users are in a stressed state. Based on the congestion level and emotional state, the server notifies users that "Conference Room 1 is currently available with 10 people, and the stress level is high."

[0664] Example 2: Managing work areas and assigning resources based on emotional state

[0665] Cameras installed in the work area capture images and send them to a server. The server analyzes the images using AI analysis tools (e.g., OpenCV) and counts that there are currently 20 people in the area. After saving the analysis results to a database to obtain the congestion status, the server analyzes the stress levels of users using emotion recognition tools. Based on the congestion status and emotional state, the server assigns a quiet area to users who are stressed and notifies them with a message such as, "Seat A is currently vacant. It is a quiet area that can help reduce stress."

[0666] Examples of prompts for generative AI models

[0667] Prompt message 1:

[0668] "Please provide a detailed explanation of the system's process flow, which analyzes the current congestion level and user emotional state of office meeting rooms in real time to determine the optimal resource allocation."

[0669] Prompt message 2:

[0670] "Please detail the specific process for allocating resources to quieter areas based on the level of congestion in the work area and the stress levels of users."

[0671] As described above, the present invention enables real-time visualization of congestion status and efficient resource management, as well as optimal resource allocation and notification that takes into account congestion status and the emotional state of users.

[0672] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0673] Step 1:

[0674] The device (camera) captures images of a specified area. The camera is high-resolution and set to capture one image per minute. The input is the current camera feed, and the output is the captured image data. Specifically, the camera acquires video from an office conference room and generates an image file.

[0675] Step 2:

[0676] The device sends the captured image data to the server. The input is the image data captured by the camera, and the output is the image data uploaded to the server. Specifically, the camera uploads the image file to the server using the HTTP protocol. A Wi-Fi connection is used for data transmission.

[0677] Step 3:

[0678] The server preprocesses the received image data. Specifically, it resizes and normalizes the images. The input is the image data uploaded to the server, and the output is the preprocessed image data. Specifically, it standardizes the image resolution and normalizes the colors.

[0679] Step 4:

[0680] The server analyzes pre-processed image data using artificial intelligence analysis tools. The input is pre-processed image data, and the output is the count of people within a given area. Specifically, the server uses the YOLO model to detect people in the image and counts their numbers.

[0681] Step 5:

[0682] The server saves the analysis results to a database. The input is the AI ​​analysis results (number of people count) and a timestamp, and the output is time-series data stored in the database. Specifically, the server saves the data to a database such as MySQL using the INSERT command.

[0683] Step 6:

[0684] The server analyzes the user's emotional state using emotion recognition technology. The input is captured image and audio data, and the output is the emotion analysis result. Specifically, the server sends data to an emotion recognition API, which identifies emotions from the user's facial expressions and tone of voice.

[0685] Step 7:

[0686] The server retrieves the latest congestion status from a stored database. The input is the latest user count data in the database, and the output is the congestion level. Specifically, the server executes an SQL query to extract the latest data from the database and calculates the utilization rate.

[0687] Step 8:

[0688] The server allocates resources based on congestion levels and sentiment levels. Inputs are congestion data and sentiment analysis results, while output is the resource allocation result. Specifically, the server dynamically selects the most suitable meeting room or area and allocates resources accordingly.

[0689] Step 9:

[0690] The server notifies users of information using a notification system. The input is resource allocation results and congestion status data, and the output is a notification message. Specifically, the server sends notification emails to users using a mail server and, if necessary, sends push notifications to smartphone applications.

[0691] (Application Example 2)

[0692] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0693] In modern brick-and-mortar stores, understanding customer congestion in real time and using that information to manage resources efficiently is a crucial challenge. Furthermore, providing services that consider the emotional state of visitors is expected to improve customer satisfaction. However, conventional systems have struggled to simultaneously achieve both real-time analysis and resource allocation based on emotional state. To solve this problem, a system is needed that performs both congestion and emotional analysis in real time and allocates resources appropriately for each visitor.

[0694] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0695] In this invention, the server includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a management means for obtaining congestion status from stored data and allocating resources, an emotion analysis means for analyzing the emotions of users, a notification means for notifying users of congestion status and resource allocation, and a means for recommending resources according to the emotional state of users. This makes it possible to visualize the congestion status in a physical store in real time and to dynamically allocate the optimal resources based on the emotional state of visitors.

[0696] An "imaging means" is a device for capturing images in real time.

[0697] "Artificial intelligence analysis means" refers to AI technology that analyzes captured images and counts the number of people in those images.

[0698] "Recording means" refers to a function for saving analysis results and acquired data to a database.

[0699] "Management means" refers to a function that retrieves congestion status from stored data and dynamically allocates resources.

[0700] "Emotion analysis means" refers to technology that analyzes a user's image and audio data to identify their emotional state.

[0701] "Notification means" refers to a function that notifies users in real time about congestion status and resource allocation information.

[0702] A "resource recommendation mechanism" is a function that dynamically suggests the most suitable resources based on the user's emotional state.

[0703] This invention relates to a system that visualizes congestion levels in physical stores in real time, enabling efficient resource management and service provision based on customer sentiment. The system includes imaging means, artificial intelligence analysis means, recording means, management means, sentiment analysis means, notification means, and resource recommendation means.

[0704] This system uses cameras as an imaging means to capture images in real time. For example, surveillance cameras installed in each area of ​​a physical store acquire current video footage and transmit that video data to a server.

[0705] The server uses artificial intelligence analysis to count the number of people in the area based on the received video data. This process utilizes pre-trained deep learning models and object detection algorithms. Specifically, the images are first pre-processed, such as resizing and normalization, and then the pre-processed images are input into the artificial intelligence model for analysis.

[0706] The analysis results are stored in a database using recording devices. For example, the number of people and timestamps for each area are recorded as time-series data and used for subsequent congestion analysis.

[0707] Next, the server analyzes the customer's emotional state using emotion analysis tools. This involves identifying the customer's emotional state from their facial expressions and tone of voice using video data acquired by the camera and audio data acquired by the microphone. A separate deep learning model is used for emotion analysis to estimate various emotions (e.g., stress, joy, etc.).

[0708] The management system retrieves the latest congestion status from stored data and dynamically allocates resources along with the user's emotional state. For example, if a particular area is congested or a customer's stress level is high, it will prioritize recommending less crowded or quieter areas.

[0709] Using notification mechanisms, the server informs users in real time about congestion levels and resource allocation. This allows users to select appropriate areas and take actions to avoid congestion. Furthermore, it provides customized services based on the customer's emotional state through resource recommendation mechanisms.

[0710] Specific example

[0711] Example 1: Analysis of congestion levels and sentiment analysis in physical stores

[0712] A terminal (a camera installed in the physical store) captures images in real time and sends them to a server. The server analyzes the images using artificial intelligence analysis and counts that there are currently 50 people in the store. After saving the analysis results and timestamps to a database, the server uses emotion analysis to analyze the emotions of the customers from the video and detects that many customers are in a stressed state. Based on the congestion level and emotional state, the server notifies store staff and customers that "Store area A is currently available with 50 people, and the stress level is high."

[0713] Example 2: Managing dedicated areas and assigning resources based on emotional state

[0714] Cameras installed in a designated area capture images and send them to a server. The server uses artificial intelligence analysis to analyze the images and counts that there are currently 30 people in the area. After saving the analysis results to a database to obtain the congestion status, the server uses emotion analysis to analyze the stress levels of the customers. Based on the congestion status and emotional state, the server assigns a quiet area to customers who are stressed and notifies them with a message such as, "Area B is currently empty. It is a quiet area that can help reduce stress."

[0715] Example of a prompt:

[0716] "Please explain how to apply a system that visualizes office congestion levels in real time."

[0717] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0718] Step 1:

[0719] The device captures images in real time.

[0720] Input: Video data captured by the camera

[0721] Operation: Cameras installed in each area of ​​the store capture images at specified intervals and send the data to the server in streaming format.

[0722] Output: Captured image data

[0723] Step 2:

[0724] The server uses artificial intelligence analysis to count the number of people based on the image data it receives.

[0725] Input: Captured image data

[0726] Operation: The received image data is preprocessed, such as by resizing and normalizing, and the preprocessed image is input into a deep learning model. The AI ​​model uses an object detection algorithm to detect people in the image and counts the number of people.

[0727] Output: Data on the number of people counted

[0728] Step 3:

[0729] The server saves the analysis results to a database.

[0730] Input: Counted number of people data, timestamp

[0731] Operation: Records the analysis results (number of people data) and the current timestamp in the database.

[0732] Output: Updated database

[0733] Step 4:

[0734] The server uses emotion analysis tools to analyze the customer's emotional state.

[0735] Input: Captured image data, audio data (if necessary)

[0736] Operation: Image and audio data are input into another deep learning model to estimate the customer's emotional state from their facial expressions and tone of voice. Various emotions are identified (e.g., stress, joy), and analysis results are obtained.

[0737] Output: Customer sentiment data

[0738] Step 5:

[0739] The server retrieves the latest congestion information from stored data and dynamically allocates resources.

[0740] Input: Number of people data, emotion data

[0741] Operation: Retrieves the latest headcount and sentiment data from the database and uses management tools to allocate resources based on congestion levels and sentiment states. Selects areas with low congestion or areas suitable for the customer's sentiment state.

[0742] Output: Resource allocation data

[0743] Step 6:

[0744] The server notifies the user of the information it has obtained.

[0745] Input: Resource allocation data, congestion status data

[0746] Operation: Uses notification methods to provide users (customers and staff) with real-time information on congestion levels and resource allocation. For example, it may notify users that a specific area is available or that a quiet area is suitable for stress relief.

[0747] Output: Notified information

[0748] Example of a prompt:

[0749] "Please explain how to apply a system that visualizes office congestion levels in real time."

[0750] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0751] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0752] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0753] [Third Embodiment]

[0754] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0755] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0756] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0757] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0758] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0759] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0760] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0761] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0762] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0763] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0764] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0765] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0766] This invention relates to a system that visualizes congestion levels in an office environment in real time and enables efficient resource management. Specific embodiments of this system are described below.

[0767] This system includes a camera for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, and a notification means for notifying users of congestion status and resource allocation.

[0768] Program Processing Description

[0769] Image capture

[0770] The terminal (camera) captures images in real time based on the specified camera ID. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[0771] Image analysis

[0772] Based on the image data received by the server, the number of people in the room is counted using AI analysis methods. Specifically, the images are first preprocessed (e.g., resizing, normalization), and then the preprocessed images are input into an AI model for analysis. The AI ​​model uses object detection technology and deep learning algorithms to detect people in the images and count their numbers.

[0773] Database update

[0774] The server uses AI analysis to obtain the number of people and the current timestamp, and stores this information in a database using recording devices. This allows the congestion status of each room and area to be stored as time-series data.

[0775] Retrieving congestion status

[0776] The server retrieves the latest congestion information from a stored database. This involves extracting the most recent number of people in each room or area and calculating the degree of congestion. The congestion level is expressed as the utilization rate for each room or area.

[0777] Resource allocation

[0778] Based on the acquired congestion information, the server dynamically allocates resources such as meeting rooms and office spaces using a management system. Resource allocation is performed by prioritizing the use of areas and rooms with low congestion levels. For example, if a meeting room is full, the server will suggest other available meeting rooms to the user.

[0779] notification

[0780] Using notification mechanisms, the server informs users of real-time congestion status and resource allocation information. This allows users to know in advance which rooms or seats are suitable and to use the facilities efficiently.

[0781] Specific example

[0782] Example 1: Analysis and notification of meeting room congestion status

[0783] A terminal (a camera installed in the meeting room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis and counts that there are currently 10 people in the meeting room. After saving the analysis results and timestamps to a database, the server retrieves the latest congestion status and notifies the user that "Meeting Room 1 is currently available for 10 people."

[0784] Example 2: Managing the work area and allocating resources

[0785] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and count the number of people currently in the area, which is 20. The analysis results are stored in a database to obtain congestion status, and if a certain level of congestion is exceeded, the server notifies users of other available seats. For example, it might notify users that "Seat A is currently available," providing them with an efficient work environment.

[0786] As described above, the system according to the present invention can provide users with a user-friendly work environment by enabling real-time visualization of congestion status and efficient resource management.

[0787] The following describes the processing flow.

[0788] Step 1:

[0789] The terminal (camera) captures images of a designated area. Based on a specific camera ID (e.g., Camera 1), the terminal acquires real-time images of the conference room or office area. The acquired image data is then sent to the server.

[0790] Step 2:

[0791] The server receives image data sent from the terminal. The server preprocesses the received image, performing necessary operations such as resizing and normalization.

[0792] Step 3:

[0793] The server analyzes the pre-processed images using AI analysis tools. Specifically, the images are input into an AI model (e.g., an object detection model), and the analysis begins. The AI ​​model detects people in the images and counts the number of each person.

[0794] Step 4:

[0795] The server retrieves the number of people information obtained as a result of the AI ​​model's analysis. Next, it retrieves the current timestamp and saves it to the database along with the analysis results.

[0796] Step 5:

[0797] The server retrieves the latest occupancy data from the database in real time to obtain congestion status. It then aggregates the latest occupancy information for each room and area and calculates the level of congestion.

[0798] Step 6:

[0799] The server manages resource allocation based on congestion levels. For example, if a meeting room is fully booked, it suggests an alternative, available meeting room to the user. It dynamically optimizes resource allocation by adjusting allocations according to congestion levels.

[0800] Step 7:

[0801] The server notifies users of congestion status and resource allocation information. Using notification methods, users are informed of the availability of meeting rooms and workspaces in real time. This allows users to efficiently utilize the most suitable resources.

[0802] By following these steps, this system can visualize office congestion in real time and enable efficient resource management.

[0803] (Example 1)

[0804] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0805] To efficiently manage congestion in office environments and improve user convenience, real-time situation monitoring and dynamic resource allocation are necessary. However, conventional methods make it difficult to intuitively grasp congestion levels and optimize resource use. Therefore, there is a need for an efficient resource management system based on real-time data.

[0806] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0807] In this invention, the server includes a shooting means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, and a storage means for saving the analysis results to a database. This makes it possible to grasp the congestion status of each area in real time and to allocate resources efficiently.

[0808] "Shooting means" refers to a device that uses a camera or video capture device to acquire images in real time.

[0809] "Artificial intelligence analysis means" refers to a system that includes artificial intelligence and algorithms for processing and analyzing received image data and for identifying and counting objects and people within the image.

[0810] "Storage means" refers to a device or system for recording and storing analysis results and related data in a storage system such as a database.

[0811] A "management system" is a control system that obtains congestion status based on stored data and allocates resources accordingly.

[0812] A "notification system" is a system for notifying users in real time about congestion levels and resource allocation information.

[0813] An "artificial intelligence model" is an algorithm trained to analyze image data based on deep learning or machine learning, and to detect objects and people.

[0814] "Congestion level" is an indicator that shows the density of people in a particular area, and is calculated based on real-time data.

[0815] This invention relates to a system that visualizes congestion levels in an office environment in real time and enables efficient resource management. This system includes a means for capturing images, an artificial intelligence analysis means, a storage means, a management means, and a notification means. Embodiments of this invention are described in detail below.

[0816] System Configuration

[0817] Photography methods

[0818] The terminal (camera) captures images in real time based on the specified camera ID. For example, cameras installed in an office conference room or work area take pictures every second and send the image data to the server. The cameras used consist of general IP cameras and high-resolution cameras.

[0819] Artificial intelligence analysis methods

[0820] The server counts the number of people using AI analysis based on image data received from the terminal. The images are first preprocessed (resizing and normalization), and then input into the AI ​​model for analysis. The AI ​​model is built using TensorFlow or PyTorch, and utilizes YOLO and SSD models as representative object detection techniques. Through analysis, people in the image are detected and counted.

[0821] Preservation means

[0822] The server stores the number of people and timestamps obtained through AI analysis into a database. This database uses a relational database such as MySQL or PostgreSQL. The stored data allows for the management of congestion levels in each room and area as time-series data.

[0823] management measures

[0824] The server retrieves the latest congestion information from the database. Specifically, it extracts the latest number of people in each room or area from the database and calculates the degree of congestion. For example, it can calculate the congestion level of a particular room as "50% utilization." Based on the congestion level, the server dynamically allocates resources. For example, if meeting room A is full, the server will suggest the available meeting room B to the user.

[0825] Notification means

[0826] The server uses notification methods to inform users in real time about congestion levels and resource allocation information. These notifications are sent as push notifications to smartphones and PCs. Notification services such as Firebase Cloud Messaging (FCM) are used for these notifications.

[0827] Specific example

[0828] Analysis and notification of meeting room congestion status

[0829] A terminal (a camera installed in the meeting room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis and counts that there are currently 10 people in the meeting room. After saving the analysis results and timestamps to a database, the server retrieves the latest congestion status and notifies the user that "Meeting Room 1 is currently available for 10 people."

[0830] Management of the work area and resource allocation

[0831] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and count the number of people currently in the area, which is 20. The analysis results are stored in a database to obtain congestion status, and if a certain level of congestion is exceeded, the server notifies users of other available seats. For example, it might notify users that "Seat A is currently available," providing them with an efficient work environment.

[0832] Thus, the system according to the present invention can provide users with a user-friendly work environment by enabling real-time visualization of congestion and efficient resource management.

[0833] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0834] Step 1: Image Capture

[0835] The device (camera) captures images in real time.

[0836] Input: Camera ID, current timestamp

[0837] Operation: The camera takes a picture in JPEG format every second and sends the image data to the server as an HTTP request.

[0838] Output: Sending JPEG image data to the server

[0839] Step 2: Receiving Image Data

[0840] The server receives image data sent from the terminal.

[0841] Input: JPEG image data sent from the terminal, HTTP request

[0842] Operation: The server parses the payload of the received HTTP request and temporarily saves the JPEG image to a directory.

[0843] Output: Saved JPEG image file

[0844] Step 3: Image preprocessing

[0845] The server preprocesses the received image data.

[0846] Input: Saved JPEG image file

[0847] Operation: The server uses the OpenCV library in Python to resize and normalize images. Specifically, it resizes images to 300x300 pixels and normalizes each pixel value to a range of 0 to 1.

[0848] Output: Resized and normalized image data

[0849] Step 4: Image Analysis

[0850] The server analyzes the pre-processed image data using artificial intelligence analysis tools.

[0851] Input: Resized and normalized image data

[0852] Operation: The server inputs image data into an AI model (e.g., YOLOv3) using TensorFlow or PyTorch and performs object detection. As an analysis result, it obtains the coordinates and number of people in the image.

[0853] Output: Person detection results (e.g., person coordinates, number of people data)

[0854] Step 5: Database Update

[0855] The server saves the analysis results (number of people) and the current timestamp to the database.

[0856] Input: Person detection result, timestamp, camera ID

[0857] Operation: The server connects to a MySQL or PostgreSQL database and executes INSERT queries to save data. Specifically, it records the number of people, camera ID, and timestamp.

[0858] Output: Number of people and timestamp stored in the database

[0859] Step 6: Obtain congestion status

[0860] The server retrieves the latest congestion information from the database.

[0861] Input: Number of people in the database, timestamp

[0862] Operation: The server executes a SELECT query to extract the latest number of people data. It calculates the level of congestion and determines the room occupancy rate.

[0863] Output: Congestion level information (occupancy rate) for each room and area.

[0864] Step 7: Resource Allocation

[0865] The server dynamically allocates resources based on congestion levels.

[0866] Input: Congestion level information

[0867] Operation: The server prioritizes selecting less congested areas or rooms based on the level of congestion and generates suggestion information. For example, it creates dynamic allocation information such as, "If meeting room A is full, use meeting room B."

[0868] Output: Resource allocation information

[0869] Step 8: Notification

[0870] The server notifies the user of resource allocation information.

[0871] Input: Resource allocation information, user's device ID

[0872] Operation: Uses Firebase Cloud Messaging (FCM) and other tools to send notifications to smartphones and computers. Specifically, it sends messages such as "Meeting Room 1 is currently available for 10 people."

[0873] Output: Notification message sent to the user's device

[0874] (Application Example 1)

[0875] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0876] Traditional food delivery systems struggled to accurately track restaurant congestion and cooking times, making it difficult to provide customers with accurate waiting times. Furthermore, delivery personnel were unable to be instructed on optimal routes or priorities, requiring efficient operations.

[0877] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0878] In this invention, the server includes an image acquisition means for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a resource management means for obtaining congestion status from the saved data and allocating resources, a notification means for notifying users of congestion status and resource allocation, a restaurant management means for monitoring the cooking status and waiting time of restaurants and informing customers of the optimal ordering timing and waiting time, and a delivery management means for optimizing order priorities and delivery routes based on congestion status. This enables accurate waiting time provision and efficient resource management.

[0879] "Image acquisition means" refers to devices or systems for capturing images in real time.

[0880] "AI analysis means" refers to devices or systems that use artificial intelligence technology to analyze captured images and count the number of people within a given area.

[0881] "Recording means" refers to devices or mechanisms for saving analysis results to a database.

[0882] A "resource management means" is a device or mechanism for obtaining congestion status from stored data and allocating resources.

[0883] "Notification means" refers to devices or mechanisms for notifying users of congestion status and resource allocation.

[0884] "Restaurant management systems" refer to devices and mechanisms that monitor the cooking status and waiting times of restaurants and inform customers of the optimal timing for ordering and waiting times.

[0885] A "delivery management system" refers to a device or mechanism for optimizing order priorities and delivery routes based on congestion levels.

[0886] The system for carrying out this invention is configured as follows.

[0887] Hardware and software configuration

[0888] Image acquisition method:

[0889] Use cameras (e.g., IP cameras, webcams, etc.) to capture images in real time. These cameras will be installed in the cooking area and waiting area of ​​the restaurant to capture current video.

[0890] AI analysis means:

[0891] A server will be set up to analyze the received image data. This server will be equipped with an AI model that performs object detection technology (e.g., YOLO, SSD). Python libraries (e.g., OpenCV, TensorFlow, PyTorch) will be used for image preprocessing and analysis software.

[0892] Recording means:

[0893] To store the analysis results in a database, a relational database such as MySQL or PostgreSQL is used. This database is used to manage time-series data for each area.

[0894] Resource management methods:

[0895] This includes server processing logic for retrieving congestion information from stored data and dynamically allocating resources. Specifically, it uses programming languages ​​such as Python or Java to calculate congestion information and perform optimal resource allocation.

[0896] Notification method:

[0897] Smartphone apps and web applications are being developed to provide users with real-time notifications. These applications will deliver information on congestion levels and resource allocation via push notifications and email notifications.

[0898] Restaurant management means:

[0899] This system includes features to monitor the cooking status and waiting times in restaurants, and to inform customers of the optimal timing for ordering and waiting times. This primarily works in conjunction with the AI ​​analysis methods described above.

[0900] Delivery management methods:

[0901] This includes a system for optimizing order priorities and delivery routes based on congestion levels, resulting in more efficient operations.

[0902] Processing flow and data processing

[0903] The server receives the image and uses AI analysis to analyze the number of people and cooking status within the image. This data is stored in a recording system and processed by a resource management system. Congestion status and resource allocation information are notified to users and delivery personnel in real time via a notification system. Each process involves specific hardware and software, ensuring efficient data calculation and processing.

[0904] Specific example

[0905] Example 1: Notification of waiting time during cooking

[0906] A customer places an order at a restaurant. A camera captures the kitchen and sends the image to a server. The server uses the YOLO model to count the number of orders being prepared and predict the waiting time. This information is then sent to the customer's smartphone.

[0907] Prompt message: "Please check the cooking status and waiting times of currently busy restaurants."

[0908] Example 2: Optimizing delivery routes

[0909] A delivery driver receives multiple orders. The server analyzes the received images and calculates the optimal delivery route based on congestion levels. This information is then sent to the delivery driver's smartphone, enabling efficient delivery.

[0910] Prompt: "Calculate the optimal delivery route based on traffic conditions."

[0911] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0912] Step 1:

[0913] Image capture

[0914] The terminal (camera) captures images of the cooking area and waiting area inside the restaurant in real time. The captured image data is transmitted to a server via the network.

[0915] Input: Camera video

[0916] Output: Captured image data

[0917] Specific operation: The camera captures video frame by frame and sends the image data to the server.

[0918] Step 2:

[0919] Image preprocessing

[0920] The server preprocesses the received image data before AI analysis. Specifically, it resizes the images, normalizes them, and converts them into a format suitable for the AI ​​model.

[0921] Input: Captured image data

[0922] Output: Preprocessed image data

[0923] Specific operation: Use OpenCV to perform image preprocessing such as resizing and normalization.

[0924] Step 3:

[0925] AI analysis

[0926] The server inputs pre-processed image data into an AI model to detect people and cooking conditions within the images. Specifically, it uses object detection models such as YOLO and SSD.

[0927] Input: Preprocessed image data

[0928] Output: Analysis results (location and number of people, cooking status)

[0929] Specific operation: The AI ​​model detects people and objects in the image and outputs that information.

[0930] Step 4:

[0931] Save to database

[0932] The server saves the analysis results and timestamps to a database. This records the congestion and cooking status of each area as time-series data.

[0933] Input: Analysis results, timestamp

[0934] Output: Analysis results saved in the database

[0935] Specific operation: Execute INSERT statements into a database using MySQL or PostgreSQL.

[0936] Step 5:

[0937] Retrieving congestion status

[0938] The server retrieves the latest congestion information from the database and calculates the latest personnel information and congestion level for each area.

[0939] Input: Database analysis results

[0940] Output: Latest congestion data

[0941] Specific operation: Retrieve the latest records using a query and run a script to calculate the level of congestion.

[0942] Step 6:

[0943] Resource allocation

[0944] The server dynamically assigns order priorities and delivery routes based on congestion levels. If congestion is high, it takes measures such as suggesting an alternative delivery route.

[0945] Input: Latest congestion data

[0946] Output: Resource allocation instructions

[0947] Specific operation: Execute the algorithm using a Python script to determine efficient resource allocation.

[0948] Step 7:

[0949] notification

[0950] The server notifies users and delivery drivers of the latest congestion status, waiting times, and delivery route information. Notifications are sent via smartphone apps and web applications.

[0951] Input: Resource allocation instruction

[0952] Output: Notification Information

[0953] Specific operation: Use APIs for push notifications and email notifications to send information in real time.

[0954] Step 8:

[0955] Display of the user interface

[0956] Users can check current congestion levels, waiting times, and recommended delivery routes through smartphone apps and web applications.

[0957] Input: Notification information

[0958] Output: Information displayed on the screen

[0959] Specific operation: The application's frontend retrieves the latest information and displays it in the user interface.

[0960] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0961] This invention combines a system that visualizes congestion levels in an office environment in real time and enables efficient resource management with an emotion engine that recognizes user emotions. Specific embodiments of this system are described below.

[0962] This system includes a camera for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, a notification means for notifying users of congestion status and resource allocation, and an emotion engine for analyzing user emotions.

[0963] Program Processing Description

[0964] Image capture

[0965] The terminal (camera) captures images of a designated area in real time. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[0966] Image analysis

[0967] Based on the image data received by the server, the number of people in the room is counted using AI analysis methods. Specifically, the images are first preprocessed (e.g., resizing, normalization), and then the preprocessed images are input into an AI model for analysis. The AI ​​model uses object detection technology and deep learning algorithms to detect people in the images and count their numbers.

[0968] Database update

[0969] The server uses AI analysis to obtain the number of people and the current timestamp, and stores this information in a database using recording devices. This allows the congestion status of each room and area to be stored as time-series data.

[0970] Emotion analysis

[0971] The server uses an emotion engine to analyze the user's emotional state. Specifically, it inputs image data acquired from the camera and audio data acquired from the microphone into the emotion engine, and identifies the user's emotional state from their facial expressions and tone of voice.

[0972] Retrieving congestion status

[0973] The server retrieves the latest congestion information from a stored database. This involves extracting the most recent number of people in each room or area and calculating the degree of congestion. The congestion level is expressed as the utilization rate for each room or area.

[0974] Resource allocation

[0975] Based on acquired congestion levels and emotional states, the server dynamically allocates resources in meeting rooms and workspaces using a management system. Resource allocation not only prioritizes the use of less congested areas and rooms, but also provides optimal resources according to the user's emotional state. For example, it assigns quieter areas to users experiencing stress.

[0976] notification

[0977] Using notification mechanisms, the server informs users of real-time congestion and resource allocation information. This allows users to know in advance which rooms or seats are suitable and to use them efficiently. Furthermore, it can provide users with resource recommendations and customized notifications based on their emotional state.

[0978] Specific example

[0979] Example 1: Analysis of conference room congestion and sentiment analysis

[0980] A terminal (a camera installed in the conference room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis tools and counts that there are currently 10 people in the conference room. After saving the analysis results and timestamps to a database, the server uses an emotion engine to analyze the users' emotions from the video and detects that many users are in a stressed state. Based on the congestion level and emotional state, the server notifies users that "Conference Room 1 is currently available with 10 people, and the stress level is high."

[0981] Example 2: Managing work areas and assigning resources based on emotional state

[0982] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and counts that there are currently 20 people in the area. After saving the analysis results to a database to obtain the congestion status, the server uses an emotion engine to analyze the users' stress levels. Based on the congestion status and emotional state, the server assigns a quiet area to users who are stressed and notifies them with a message such as, "Seat A is currently vacant. It is a quiet area that can help reduce stress."

[0983] As described above, the system according to the present invention can achieve real-time visualization of congestion status and efficient resource management, as well as optimal resource allocation and notification that takes into account the user's emotional state.

[0984] The following describes the processing flow.

[0985] Step 1:

[0986] The terminal (camera) captures images of a designated area in real time. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[0987] Step 2:

[0988] The server receives image data sent from the terminal. The server preprocesses the received image, performing necessary operations such as resizing and normalization.

[0989] Step 3:

[0990] The server analyzes the pre-processed images using AI analysis tools. Specifically, the images are input into an AI model (e.g., an object detection model), and the analysis begins. The AI ​​model detects people in the images and counts the number of each person.

[0991] Step 4:

[0992] The server retrieves the number of people information obtained as a result of the AI ​​model's analysis. Next, it retrieves the current timestamp and saves it to the database along with the analysis results.

[0993] Step 5:

[0994] The server uses an emotion engine to analyze the user's emotional state. Image data acquired from the camera and audio data acquired from the microphone are input into the emotion engine, which then identifies the user's emotional state from their facial expressions and tone of voice.

[0995] Step 6:

[0996] The server retrieves the latest occupancy data from the database in real time to obtain congestion status. It then aggregates the latest occupancy information for each room and area and calculates the level of congestion.

[0997] Step 7:

[0998] The server manages resource allocation based on congestion levels and user emotions. For example, if a meeting room is full, it suggests an alternative, available meeting room to the user. It dynamically optimizes resources by adjusting allocations according to congestion levels. It also prioritizes assigning quieter areas to users who are stressed.

[0999] Step 8:

[1000] The server notifies users of congestion status and resource allocation information. It uses notification methods to inform users in real time about the availability of meeting rooms and workspaces. It also provides resource suggestions and customized notifications based on the user's emotional state. For example, it might provide users with a specific notification such as, "Meeting Room A is currently available. It's a quiet area that can help reduce stress."

[1001] Through the steps described above, this system can visualize office congestion and user emotional states in real time, enabling efficient and optimal resource management and notifications.

[1002] (Example 2)

[1003] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1004] Traditional office resource management systems fail to adequately visualize congestion levels and efficiently allocate resources. Furthermore, they cannot provide resources while considering users' emotional states, making it difficult to properly manage user satisfaction and stress levels. Therefore, it is necessary to efficiently utilize office space resources while optimally allocating resources in accordance with users' emotional states.

[1005] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1006] In this invention, the server includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, an emotion recognition means for analyzing the emotional state of users, and a notification means for notifying users of congestion status and resource allocation. This enables real-time visualization of congestion status and efficient resource management, as well as optimal resource allocation that takes into account the emotional state of users.

[1007] "Real-time" refers to a process or response that occurs almost instantly, with virtually no time delay.

[1008] "Imaging means" refers to a device that captures images or videos of a designated area. A camera is a specific example of this.

[1009] "Artificial intelligence analysis means" refers to artificial intelligence technology used to analyze captured images and obtain specific information (for example, counting the number of people).

[1010] An "artificial intelligence model" refers to a model used for analysis that possesses algorithms and structures employed by artificial intelligence analysis tools. Deep learning models are an example of this type of model.

[1011] "Recording means" refers to a device or system for saving analysis results to a database or similar.

[1012] A "database" refers to a digital system used to systematically store and manage analysis results and other information.

[1013] "Emotion recognition means" refers to a system or technology that analyzes a user's emotional state from their facial expressions, voice, etc.

[1014] "Management means" refers to a system or device that oversees and manages resource allocation and the operation of the entire system.

[1015] "Notification means" refers to methods or devices for communicating system analysis results and resource allocation information to users. Specifically, this includes email and push notifications.

[1016] "Congestion status" refers to the state of a particular area, indicating the density of people or the utilization rate.

[1017] "Resource allocation" refers to the act of efficiently and appropriately distributing available resources within a system (for example, meeting rooms or office desks).

[1018] "User" refers to an individual or organization that uses this system to manage and utilize resources within the office.

[1019] This invention provides a system that visualizes congestion levels in an office environment in real time, enabling efficient resource management, and further incorporates an emotion recognition function that recognizes the emotions of users. A specific example of this system is described below.

[1020] This system includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, an emotion recognition means for analyzing the emotions of users, and a notification means for notifying users of congestion status and resource allocation.

[1021] The terminal (camera) captures images of a designated area, such as an office conference room or work area, in real time. The camera is high-resolution and periodically captures images, sending the data to a server. A network (e.g., Wi-Fi) is used for data transmission.

[1022] The server preprocesses the received image data (e.g., resizing, normalization) and counts the number of people in the image using AI analysis tools. Specifically, it uses artificial intelligence models (e.g., YOLO, OpenCV) to perform object detection and person counting. The analysis results are stored in a database (e.g., MySQL, PostgreSQL).

[1023] Furthermore, the server uses emotion recognition means (e.g., emotion recognition API) to analyze captured image and audio data and identify the user's emotional state. This allows for the determination of states such as stress or relaxation.

[1024] The server retrieves the latest congestion information from a stored database and calculates the degree of congestion. Based on the latest number of people in each area, the congestion status is visualized as a utilization rate. Resource allocation is dynamically performed using management tools based on congestion status and sentiment.

[1025] For example, resources can be allocated to less crowded areas or according to the emotional state of users. For instance, users experiencing stress could be assigned to quieter areas.

[1026] Using notification methods, the server informs users of real-time congestion status and resource allocation information. Notifications are sent via email, smartphone app push notifications, etc. This allows users to not only use rooms and seats efficiently, but also receive resource recommendations based on their emotional state.

[1027] Specific example

[1028] Example 1: Analysis of conference room congestion and sentiment analysis

[1029] A terminal (a camera installed in the conference room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis tools (e.g., YOLO) and counts that there are currently 10 people in the conference room. After saving the analysis results and timestamps to a database, the server uses emotion recognition tools to analyze the emotions of the users from the video and detects that many users are in a stressed state. Based on the congestion level and emotional state, the server notifies users that "Conference Room 1 is currently available with 10 people, and the stress level is high."

[1030] Example 2: Managing work areas and assigning resources based on emotional state

[1031] Cameras installed in the work area capture images and send them to a server. The server analyzes the images using AI analysis tools (e.g., OpenCV) and counts that there are currently 20 people in the area. After saving the analysis results to a database to obtain the congestion status, the server analyzes the stress levels of users using emotion recognition tools. Based on the congestion status and emotional state, the server assigns a quiet area to users who are stressed and notifies them with a message such as, "Seat A is currently vacant. It is a quiet area that can help reduce stress."

[1032] Examples of prompts for generative AI models

[1033] Prompt message 1:

[1034] "Please provide a detailed explanation of the system's process flow, which analyzes the current congestion level and user emotional state of office meeting rooms in real time to determine the optimal resource allocation."

[1035] Prompt message 2:

[1036] "Please detail the specific process for allocating resources to quieter areas based on the level of congestion in the work area and the stress levels of users."

[1037] As described above, the present invention enables real-time visualization of congestion status and efficient resource management, as well as optimal resource allocation and notification that takes into account congestion status and the emotional state of users.

[1038] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1039] Step 1:

[1040] The device (camera) captures images of a specified area. The camera is high-resolution and set to capture one image per minute. The input is the current camera feed, and the output is the captured image data. Specifically, the camera acquires video from an office conference room and generates an image file.

[1041] Step 2:

[1042] The device sends the captured image data to the server. The input is the image data captured by the camera, and the output is the image data uploaded to the server. Specifically, the camera uploads the image file to the server using the HTTP protocol. A Wi-Fi connection is used for data transmission.

[1043] Step 3:

[1044] The server preprocesses the received image data. Specifically, it resizes and normalizes the images. The input is the image data uploaded to the server, and the output is the preprocessed image data. Specifically, it standardizes the image resolution and normalizes the colors.

[1045] Step 4:

[1046] The server analyzes pre-processed image data using artificial intelligence analysis tools. The input is pre-processed image data, and the output is the count of people within a given area. Specifically, the server uses the YOLO model to detect people in the image and counts their numbers.

[1047] Step 5:

[1048] The server saves the analysis results to a database. The input is the AI ​​analysis results (number of people count) and a timestamp, and the output is time-series data stored in the database. Specifically, the server saves the data to a database such as MySQL using the INSERT command.

[1049] Step 6:

[1050] The server analyzes the user's emotional state using emotion recognition technology. The input is captured image and audio data, and the output is the emotion analysis result. Specifically, the server sends data to an emotion recognition API, which identifies emotions from the user's facial expressions and tone of voice.

[1051] Step 7:

[1052] The server retrieves the latest congestion status from a stored database. The input is the latest user count data in the database, and the output is the congestion level. Specifically, the server executes an SQL query to extract the latest data from the database and calculates the utilization rate.

[1053] Step 8:

[1054] The server allocates resources based on congestion levels and sentiment levels. Inputs are congestion data and sentiment analysis results, while output is the resource allocation result. Specifically, the server dynamically selects the most suitable meeting room or area and allocates resources accordingly.

[1055] Step 9:

[1056] The server notifies users of information using a notification system. The input is resource allocation results and congestion status data, and the output is a notification message. Specifically, the server sends notification emails to users using a mail server and, if necessary, sends push notifications to smartphone applications.

[1057] (Application Example 2)

[1058] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1059] In modern brick-and-mortar stores, understanding customer congestion in real time and using that information to manage resources efficiently is a crucial challenge. Furthermore, providing services that consider the emotional state of visitors is expected to improve customer satisfaction. However, conventional systems have struggled to simultaneously achieve both real-time analysis and resource allocation based on emotional state. To solve this problem, a system is needed that performs both congestion and emotional analysis in real time and allocates resources appropriately for each visitor.

[1060] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1061] In this invention, the server includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a management means for obtaining congestion status from stored data and allocating resources, an emotion analysis means for analyzing the emotions of users, a notification means for notifying users of congestion status and resource allocation, and a means for recommending resources according to the emotional state of users. This makes it possible to visualize the congestion status in a physical store in real time and to dynamically allocate the optimal resources based on the emotional state of visitors.

[1062] An "imaging means" is a device for capturing images in real time.

[1063] "Artificial intelligence analysis means" refers to AI technology that analyzes captured images and counts the number of people in those images.

[1064] "Recording means" refers to a function for saving analysis results and acquired data to a database.

[1065] "Management means" refers to a function that retrieves congestion status from stored data and dynamically allocates resources.

[1066] "Emotion analysis means" refers to technology that analyzes a user's image and audio data to identify their emotional state.

[1067] "Notification means" refers to a function that notifies users in real time about congestion status and resource allocation information.

[1068] A "resource recommendation mechanism" is a function that dynamically suggests the most suitable resources based on the user's emotional state.

[1069] This invention relates to a system that visualizes congestion levels in physical stores in real time, enabling efficient resource management and service provision based on customer sentiment. The system includes imaging means, artificial intelligence analysis means, recording means, management means, sentiment analysis means, notification means, and resource recommendation means.

[1070] This system uses cameras as an imaging means to capture images in real time. For example, surveillance cameras installed in each area of ​​a physical store acquire current video footage and transmit that video data to a server.

[1071] The server uses artificial intelligence analysis to count the number of people in the area based on the received video data. This process utilizes pre-trained deep learning models and object detection algorithms. Specifically, the images are first pre-processed, such as resizing and normalization, and then the pre-processed images are input into the artificial intelligence model for analysis.

[1072] The analysis results are stored in a database using recording devices. For example, the number of people and timestamps for each area are recorded as time-series data and used for subsequent congestion analysis.

[1073] Next, the server analyzes the customer's emotional state using emotion analysis tools. This involves identifying the customer's emotional state from their facial expressions and tone of voice using video data acquired by the camera and audio data acquired by the microphone. A separate deep learning model is used for emotion analysis to estimate various emotions (e.g., stress, joy, etc.).

[1074] The management system retrieves the latest congestion status from stored data and dynamically allocates resources along with the user's emotional state. For example, if a particular area is congested or a customer's stress level is high, it will prioritize recommending less crowded or quieter areas.

[1075] Using notification mechanisms, the server informs users in real time about congestion levels and resource allocation. This allows users to select appropriate areas and take actions to avoid congestion. Furthermore, it provides customized services based on the customer's emotional state through resource recommendation mechanisms.

[1076] Specific example

[1077] Example 1: Analysis of congestion levels and sentiment analysis in physical stores

[1078] A terminal (a camera installed in the physical store) captures images in real time and sends them to a server. The server analyzes the images using artificial intelligence analysis and counts that there are currently 50 people in the store. After saving the analysis results and timestamps to a database, the server uses emotion analysis to analyze the emotions of the customers from the video and detects that many customers are in a stressed state. Based on the congestion level and emotional state, the server notifies store staff and customers that "Store area A is currently available with 50 people, and the stress level is high."

[1079] Example 2: Managing dedicated areas and assigning resources based on emotional state

[1080] Cameras installed in a designated area capture images and send them to a server. The server uses artificial intelligence analysis to analyze the images and counts that there are currently 30 people in the area. After saving the analysis results to a database to obtain the congestion status, the server uses emotion analysis to analyze the stress levels of the customers. Based on the congestion status and emotional state, the server assigns a quiet area to customers who are stressed and notifies them with a message such as, "Area B is currently empty. It is a quiet area that can help reduce stress."

[1081] Example of a prompt:

[1082] "Please explain how to apply a system that visualizes office congestion levels in real time."

[1083] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1084] Step 1:

[1085] The device captures images in real time.

[1086] Input: Video data captured by the camera

[1087] Operation: Cameras installed in each area of ​​the store capture images at specified intervals and send the data to the server in streaming format.

[1088] Output: Captured image data

[1089] Step 2:

[1090] The server uses artificial intelligence analysis to count the number of people based on the image data it receives.

[1091] Input: Captured image data

[1092] Operation: The received image data is preprocessed, such as by resizing and normalizing, and the preprocessed image is input into a deep learning model. The AI ​​model uses an object detection algorithm to detect people in the image and counts the number of people.

[1093] Output: Data on the number of people counted

[1094] Step 3:

[1095] The server saves the analysis results to a database.

[1096] Input: Counted number of people data, timestamp

[1097] Operation: Records the analysis results (number of people data) and the current timestamp in the database.

[1098] Output: Updated database

[1099] Step 4:

[1100] The server uses emotion analysis tools to analyze the customer's emotional state.

[1101] Input: Captured image data, audio data (if necessary)

[1102] Operation: Image and audio data are input into another deep learning model to estimate the customer's emotional state from their facial expressions and tone of voice. Various emotions are identified (e.g., stress, joy), and analysis results are obtained.

[1103] Output: Customer sentiment data

[1104] Step 5:

[1105] The server retrieves the latest congestion information from stored data and dynamically allocates resources.

[1106] Input: Number of people data, emotion data

[1107] Operation: Retrieves the latest headcount and sentiment data from the database and uses management tools to allocate resources based on congestion levels and sentiment states. Selects areas with low congestion or areas suitable for the customer's sentiment state.

[1108] Output: Resource allocation data

[1109] Step 6:

[1110] The server notifies the user of the information it has obtained.

[1111] Input: Resource allocation data, congestion status data

[1112] Operation: Uses notification methods to provide users (customers and staff) with real-time information on congestion levels and resource allocation. For example, it may notify users that a specific area is available or that a quiet area is suitable for stress relief.

[1113] Output: Notified information

[1114] Example of a prompt:

[1115] "Please explain how to apply a system that visualizes office congestion levels in real time."

[1116] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1117] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1118] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1119] [Fourth Embodiment]

[1120] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1121] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1122] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1123] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1124] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1125] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1126] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1127] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1128] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1129] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1130] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1131] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1132] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1133] This invention relates to a system that visualizes congestion levels in an office environment in real time and enables efficient resource management. Specific embodiments of this system are described below.

[1134] This system includes a camera for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, and a notification means for notifying users of congestion status and resource allocation.

[1135] Program Processing Description

[1136] Image capture

[1137] The terminal (camera) captures images in real time based on the specified camera ID. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[1138] Image analysis

[1139] Based on the image data received by the server, the number of people in the room is counted using AI analysis methods. Specifically, the images are first preprocessed (e.g., resizing, normalization), and then the preprocessed images are input into an AI model for analysis. The AI ​​model uses object detection technology and deep learning algorithms to detect people in the images and count their numbers.

[1140] Database update

[1141] The server uses AI analysis to obtain the number of people and the current timestamp, and stores this information in a database using recording devices. This allows the congestion status of each room and area to be stored as time-series data.

[1142] Retrieving congestion status

[1143] The server retrieves the latest congestion information from a stored database. This involves extracting the most recent number of people in each room or area and calculating the degree of congestion. The congestion level is expressed as the utilization rate for each room or area.

[1144] Resource allocation

[1145] Based on the acquired congestion information, the server dynamically allocates resources such as meeting rooms and office spaces using a management system. Resource allocation is performed by prioritizing the use of areas and rooms with low congestion levels. For example, if a meeting room is full, the server will suggest other available meeting rooms to the user.

[1146] notification

[1147] Using notification mechanisms, the server informs users of real-time congestion status and resource allocation information. This allows users to know in advance which rooms or seats are suitable and to use the facilities efficiently.

[1148] Specific example

[1149] Example 1: Analysis and notification of meeting room congestion status

[1150] A terminal (a camera installed in the meeting room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis and counts that there are currently 10 people in the meeting room. After saving the analysis results and timestamps to a database, the server retrieves the latest congestion status and notifies the user that "Meeting Room 1 is currently available for 10 people."

[1151] Example 2: Managing the work area and allocating resources

[1152] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and count the number of people currently in the area, which is 20. The analysis results are stored in a database to obtain congestion status, and if a certain level of congestion is exceeded, the server notifies users of other available seats. For example, it might notify users that "Seat A is currently available," providing them with an efficient work environment.

[1153] As described above, the system according to the present invention can provide users with a user-friendly work environment by enabling real-time visualization of congestion status and efficient resource management.

[1154] The following describes the processing flow.

[1155] Step 1:

[1156] The terminal (camera) captures images of a designated area. Based on a specific camera ID (e.g., Camera 1), the terminal acquires real-time images of the conference room or office area. The acquired image data is then sent to the server.

[1157] Step 2:

[1158] The server receives image data sent from the terminal. The server preprocesses the received image, performing necessary operations such as resizing and normalization.

[1159] Step 3:

[1160] The server analyzes the pre-processed images using AI analysis tools. Specifically, the images are input into an AI model (e.g., an object detection model), and the analysis begins. The AI ​​model detects people in the images and counts the number of each person.

[1161] Step 4:

[1162] The server retrieves the number of people information obtained as a result of the AI ​​model's analysis. Next, it retrieves the current timestamp and saves it to the database along with the analysis results.

[1163] Step 5:

[1164] The server retrieves the latest occupancy data from the database in real time to obtain congestion status. It then aggregates the latest occupancy information for each room and area and calculates the level of congestion.

[1165] Step 6:

[1166] The server manages resource allocation based on congestion levels. For example, if a meeting room is fully booked, it suggests an alternative, available meeting room to the user. It dynamically optimizes resource allocation by adjusting allocations according to congestion levels.

[1167] Step 7:

[1168] The server notifies users of congestion status and resource allocation information. Using notification methods, users are informed of the availability of meeting rooms and workspaces in real time. This allows users to efficiently utilize the most suitable resources.

[1169] By following these steps, this system can visualize office congestion in real time and enable efficient resource management.

[1170] (Example 1)

[1171] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1172] To efficiently manage congestion in office environments and improve user convenience, real-time situation monitoring and dynamic resource allocation are necessary. However, conventional methods make it difficult to intuitively grasp congestion levels and optimize resource use. Therefore, there is a need for an efficient resource management system based on real-time data.

[1173] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1174] In this invention, the server includes a shooting means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, and a storage means for saving the analysis results to a database. This makes it possible to grasp the congestion status of each area in real time and to allocate resources efficiently.

[1175] "Shooting means" refers to a device that uses a camera or video capture device to acquire images in real time.

[1176] "Artificial intelligence analysis means" refers to a system that includes artificial intelligence and algorithms for processing and analyzing received image data and for identifying and counting objects and people within the image.

[1177] "Storage means" refers to a device or system for recording and storing analysis results and related data in a storage system such as a database.

[1178] A "management system" is a control system that obtains congestion status based on stored data and allocates resources accordingly.

[1179] A "notification system" is a system for notifying users in real time about congestion levels and resource allocation information.

[1180] An "artificial intelligence model" is an algorithm trained to analyze image data based on deep learning or machine learning, and to detect objects and people.

[1181] "Congestion level" is an indicator that shows the density of people in a particular area, and is calculated based on real-time data.

[1182] This invention relates to a system that visualizes congestion levels in an office environment in real time and enables efficient resource management. This system includes a means for capturing images, an artificial intelligence analysis means, a storage means, a management means, and a notification means. Embodiments of this invention are described in detail below.

[1183] System Configuration

[1184] Photography methods

[1185] The terminal (camera) captures images in real time based on the specified camera ID. For example, cameras installed in an office conference room or work area take pictures every second and send the image data to the server. The cameras used consist of general IP cameras and high-resolution cameras.

[1186] Artificial intelligence analysis methods

[1187] The server counts the number of people using AI analysis based on image data received from the terminal. The images are first preprocessed (resizing and normalization), and then input into the AI ​​model for analysis. The AI ​​model is built using TensorFlow or PyTorch, and utilizes YOLO and SSD models as representative object detection techniques. Through analysis, people in the image are detected and counted.

[1188] Preservation means

[1189] The server stores the number of people and timestamps obtained through AI analysis into a database. This database uses a relational database such as MySQL or PostgreSQL. The stored data allows for the management of congestion levels in each room and area as time-series data.

[1190] management measures

[1191] The server retrieves the latest congestion information from the database. Specifically, it extracts the latest number of people in each room or area from the database and calculates the degree of congestion. For example, it can calculate the congestion level of a particular room as "50% utilization." Based on the congestion level, the server dynamically allocates resources. For example, if meeting room A is full, the server will suggest the available meeting room B to the user.

[1192] Notification means

[1193] The server uses notification methods to inform users in real time about congestion levels and resource allocation information. These notifications are sent as push notifications to smartphones and PCs. Notification services such as Firebase Cloud Messaging (FCM) are used for these notifications.

[1194] Specific example

[1195] Analysis and notification of meeting room congestion status

[1196] A terminal (a camera installed in the meeting room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis and counts that there are currently 10 people in the meeting room. After saving the analysis results and timestamps to a database, the server retrieves the latest congestion status and notifies the user that "Meeting Room 1 is currently available for 10 people."

[1197] Management of the work area and resource allocation

[1198] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and count the number of people currently in the area, which is 20. The analysis results are stored in a database to obtain congestion status, and if a certain level of congestion is exceeded, the server notifies users of other available seats. For example, it might notify users that "Seat A is currently available," providing them with an efficient work environment.

[1199] Thus, the system according to the present invention can provide users with a user-friendly work environment by enabling real-time visualization of congestion and efficient resource management.

[1200] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1201] Step 1: Image Capture

[1202] The device (camera) captures images in real time.

[1203] Input: Camera ID, current timestamp

[1204] Operation: The camera takes a picture in JPEG format every second and sends the image data to the server as an HTTP request.

[1205] Output: Sending JPEG image data to the server

[1206] Step 2: Receiving Image Data

[1207] The server receives image data sent from the terminal.

[1208] Input: JPEG image data sent from the terminal, HTTP request

[1209] Operation: The server parses the payload of the received HTTP request and temporarily saves the JPEG image to a directory.

[1210] Output: Saved JPEG image file

[1211] Step 3: Image preprocessing

[1212] The server preprocesses the received image data.

[1213] Input: Saved JPEG image file

[1214] Operation: The server uses the OpenCV library in Python to resize and normalize images. Specifically, it resizes images to 300x300 pixels and normalizes each pixel value to a range of 0 to 1.

[1215] Output: Resized and normalized image data

[1216] Step 4: Image Analysis

[1217] The server analyzes the pre-processed image data using artificial intelligence analysis tools.

[1218] Input: Resized and normalized image data

[1219] Operation: The server inputs image data into an AI model (e.g., YOLOv3) using TensorFlow or PyTorch and performs object detection. As an analysis result, it obtains the coordinates and number of people in the image.

[1220] Output: Person detection results (e.g., person coordinates, number of people data)

[1221] Step 5: Database Update

[1222] The server saves the analysis results (number of people) and the current timestamp to the database.

[1223] Input: Person detection result, timestamp, camera ID

[1224] Operation: The server connects to a MySQL or PostgreSQL database and executes INSERT queries to save data. Specifically, it records the number of people, camera ID, and timestamp.

[1225] Output: Number of people and timestamp stored in the database

[1226] Step 6: Obtain congestion status

[1227] The server retrieves the latest congestion information from the database.

[1228] Input: Number of people in the database, timestamp

[1229] Operation: The server executes a SELECT query to extract the latest number of people data. It calculates the level of congestion and determines the room occupancy rate.

[1230] Output: Congestion level information (occupancy rate) for each room and area.

[1231] Step 7: Resource Allocation

[1232] The server dynamically allocates resources based on congestion levels.

[1233] Input: Congestion level information

[1234] Operation: The server prioritizes selecting less congested areas or rooms based on the level of congestion and generates suggestion information. For example, it creates dynamic allocation information such as, "If meeting room A is full, use meeting room B."

[1235] Output: Resource allocation information

[1236] Step 8: Notification

[1237] The server notifies the user of resource allocation information.

[1238] Input: Resource allocation information, user's device ID

[1239] Operation: Uses Firebase Cloud Messaging (FCM) and other tools to send notifications to smartphones and computers. Specifically, it sends messages such as "Meeting Room 1 is currently available for 10 people."

[1240] Output: Notification message sent to the user's device

[1241] (Application Example 1)

[1242] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1243] Traditional food delivery systems struggled to accurately track restaurant congestion and cooking times, making it difficult to provide customers with accurate waiting times. Furthermore, delivery personnel were unable to be instructed on optimal routes or priorities, requiring efficient operations.

[1244] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1245] In this invention, the server includes an image acquisition means for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a resource management means for obtaining congestion status from the saved data and allocating resources, a notification means for notifying users of congestion status and resource allocation, a restaurant management means for monitoring the cooking status and waiting time of restaurants and informing customers of the optimal ordering timing and waiting time, and a delivery management means for optimizing order priorities and delivery routes based on congestion status. This enables accurate waiting time provision and efficient resource management.

[1246] "Image acquisition means" refers to devices or systems for capturing images in real time.

[1247] "AI analysis means" refers to devices or systems that use artificial intelligence technology to analyze captured images and count the number of people within a given area.

[1248] "Recording means" refers to devices or mechanisms for saving analysis results to a database.

[1249] A "resource management means" is a device or mechanism for obtaining congestion status from stored data and allocating resources.

[1250] "Notification means" refers to devices or mechanisms for notifying users of congestion status and resource allocation.

[1251] "Restaurant management systems" refer to devices and mechanisms that monitor the cooking status and waiting times of restaurants and inform customers of the optimal timing for ordering and waiting times.

[1252] A "delivery management system" refers to a device or mechanism for optimizing order priorities and delivery routes based on congestion levels.

[1253] The system for carrying out this invention is configured as follows.

[1254] Hardware and software configuration

[1255] Image acquisition method:

[1256] Use cameras (e.g., IP cameras, webcams, etc.) to capture images in real time. These cameras will be installed in the cooking area and waiting area of ​​the restaurant to capture current video.

[1257] AI analysis means:

[1258] A server will be set up to analyze the received image data. This server will be equipped with an AI model that performs object detection technology (e.g., YOLO, SSD). Python libraries (e.g., OpenCV, TensorFlow, PyTorch) will be used for image preprocessing and analysis software.

[1259] Recording means:

[1260] To store the analysis results in a database, a relational database such as MySQL or PostgreSQL is used. This database is used to manage time-series data for each area.

[1261] Resource management methods:

[1262] This includes server processing logic for retrieving congestion information from stored data and dynamically allocating resources. Specifically, it uses programming languages ​​such as Python or Java to calculate congestion information and perform optimal resource allocation.

[1263] Notification method:

[1264] Smartphone apps and web applications are being developed to provide users with real-time notifications. These applications will deliver information on congestion levels and resource allocation via push notifications and email notifications.

[1265] Restaurant management means:

[1266] This system includes features to monitor the cooking status and waiting times in restaurants, and to inform customers of the optimal timing for ordering and waiting times. This primarily works in conjunction with the AI ​​analysis methods described above.

[1267] Delivery management methods:

[1268] This includes a system for optimizing order priorities and delivery routes based on congestion levels, resulting in more efficient operations.

[1269] Processing flow and data processing

[1270] The server receives the image and uses AI analysis to analyze the number of people and cooking status within the image. This data is stored in a recording system and processed by a resource management system. Congestion status and resource allocation information are notified to users and delivery personnel in real time via a notification system. Each process involves specific hardware and software, ensuring efficient data calculation and processing.

[1271] Specific example

[1272] Example 1: Notification of waiting time during cooking

[1273] A customer places an order at a restaurant. A camera captures the kitchen and sends the image to a server. The server uses the YOLO model to count the number of orders being prepared and predict the waiting time. This information is then sent to the customer's smartphone.

[1274] Prompt message: "Please check the cooking status and waiting times of currently busy restaurants."

[1275] Example 2: Optimizing delivery routes

[1276] A delivery driver receives multiple orders. The server analyzes the received images and calculates the optimal delivery route based on congestion levels. This information is then sent to the delivery driver's smartphone, enabling efficient delivery.

[1277] Prompt: "Calculate the optimal delivery route based on traffic conditions."

[1278] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1279] Step 1:

[1280] Image capture

[1281] The terminal (camera) captures images of the cooking area and waiting area inside the restaurant in real time. The captured image data is transmitted to a server via the network.

[1282] Input: Camera video

[1283] Output: Captured image data

[1284] Specific operation: The camera captures video frame by frame and sends the image data to the server.

[1285] Step 2:

[1286] Image preprocessing

[1287] The server preprocesses the received image data before AI analysis. Specifically, it resizes the images, normalizes them, and converts them into a format suitable for the AI ​​model.

[1288] Input: Captured image data

[1289] Output: Preprocessed image data

[1290] Specific operation: Use OpenCV to perform image preprocessing such as resizing and normalization.

[1291] Step 3:

[1292] AI analysis

[1293] The server inputs pre-processed image data into an AI model to detect people and cooking conditions within the images. Specifically, it uses object detection models such as YOLO and SSD.

[1294] Input: Preprocessed image data

[1295] Output: Analysis results (location and number of people, cooking status)

[1296] Specific operation: The AI ​​model detects people and objects in the image and outputs that information.

[1297] Step 4:

[1298] Save to database

[1299] The server saves the analysis results and timestamps to a database. This records the congestion and cooking status of each area as time-series data.

[1300] Input: Analysis results, timestamp

[1301] Output: Analysis results saved in the database

[1302] Specific operation: Execute INSERT statements into a database using MySQL or PostgreSQL.

[1303] Step 5:

[1304] Retrieving congestion status

[1305] The server retrieves the latest congestion information from the database and calculates the latest personnel information and congestion level for each area.

[1306] Input: Database analysis results

[1307] Output: Latest congestion data

[1308] Specific operation: Retrieve the latest records using a query and run a script to calculate the level of congestion.

[1309] Step 6:

[1310] Resource allocation

[1311] The server dynamically assigns order priorities and delivery routes based on congestion levels. If congestion is high, it takes measures such as suggesting an alternative delivery route.

[1312] Input: Latest congestion data

[1313] Output: Resource allocation instructions

[1314] Specific operation: Execute the algorithm using a Python script to determine efficient resource allocation.

[1315] Step 7:

[1316] notification

[1317] The server notifies users and delivery drivers of the latest congestion status, waiting times, and delivery route information. Notifications are sent via smartphone apps and web applications.

[1318] Input: Resource allocation instruction

[1319] Output: Notification Information

[1320] Specific operation: Use APIs for push notifications and email notifications to send information in real time.

[1321] Step 8:

[1322] Display of the user interface

[1323] Users can check current congestion levels, waiting times, and recommended delivery routes through smartphone apps and web applications.

[1324] Input: Notification information

[1325] Output: Information displayed on the screen

[1326] Specific operation: The application's frontend retrieves the latest information and displays it in the user interface.

[1327] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1328] This invention combines a system that visualizes congestion levels in an office environment in real time and enables efficient resource management with an emotion engine that recognizes user emotions. Specific embodiments of this system are described below.

[1329] This system includes a camera for capturing images in real time, an AI analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, a notification means for notifying users of congestion status and resource allocation, and an emotion engine for analyzing user emotions.

[1330] Program Processing Description

[1331] Image capture

[1332] The terminal (camera) captures images of a designated area in real time. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[1333] Image analysis

[1334] Based on the image data received by the server, the number of people in the room is counted using AI analysis methods. Specifically, the images are first preprocessed (e.g., resizing, normalization), and then the preprocessed images are input into an AI model for analysis. The AI ​​model uses object detection technology and deep learning algorithms to detect people in the images and count their numbers.

[1335] Database update

[1336] The server uses AI analysis to obtain the number of people and the current timestamp, and stores this information in a database using recording devices. This allows the congestion status of each room and area to be stored as time-series data.

[1337] Emotion analysis

[1338] The server uses an emotion engine to analyze the user's emotional state. Specifically, it inputs image data acquired from the camera and audio data acquired from the microphone into the emotion engine, and identifies the user's emotional state from their facial expressions and tone of voice.

[1339] Retrieving congestion status

[1340] The server retrieves the latest congestion information from a stored database. This involves extracting the most recent number of people in each room or area and calculating the degree of congestion. The congestion level is expressed as the utilization rate for each room or area.

[1341] Resource allocation

[1342] Based on acquired congestion levels and emotional states, the server dynamically allocates resources in meeting rooms and workspaces using a management system. Resource allocation not only prioritizes the use of less congested areas and rooms, but also provides optimal resources according to the user's emotional state. For example, it assigns quieter areas to users experiencing stress.

[1343] notification

[1344] Using notification mechanisms, the server informs users of real-time congestion and resource allocation information. This allows users to know in advance which rooms or seats are suitable and to use them efficiently. Furthermore, it can provide users with resource recommendations and customized notifications based on their emotional state.

[1345] Specific example

[1346] Example 1: Analysis of conference room congestion and sentiment analysis

[1347] A terminal (a camera installed in the conference room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis tools and counts that there are currently 10 people in the conference room. After saving the analysis results and timestamps to a database, the server uses an emotion engine to analyze the users' emotions from the video and detects that many users are in a stressed state. Based on the congestion level and emotional state, the server notifies users that "Conference Room 1 is currently available with 10 people, and the stress level is high."

[1348] Example 2: Managing work areas and assigning resources based on emotional state

[1349] Cameras installed in the work area capture images and send them to a server. The server uses AI analysis to analyze the images and counts that there are currently 20 people in the area. After saving the analysis results to a database to obtain the congestion status, the server uses an emotion engine to analyze the users' stress levels. Based on the congestion status and emotional state, the server assigns a quiet area to users who are stressed and notifies them with a message such as, "Seat A is currently vacant. It is a quiet area that can help reduce stress."

[1350] As described above, the system according to the present invention can achieve real-time visualization of congestion status and efficient resource management, as well as optimal resource allocation and notification that takes into account the user's emotional state.

[1351] The following describes the processing flow.

[1352] Step 1:

[1353] The terminal (camera) captures images of a designated area in real time. For example, a camera installed in an office conference room or work area acquires current video footage and sends that video data to a server.

[1354] Step 2:

[1355] The server receives image data sent from the terminal. The server preprocesses the received image, performing necessary operations such as resizing and normalization.

[1356] Step 3:

[1357] The server analyzes the pre-processed images using AI analysis tools. Specifically, the images are input into an AI model (e.g., an object detection model), and the analysis begins. The AI ​​model detects people in the images and counts the number of each person.

[1358] Step 4:

[1359] The server retrieves the number of people information obtained as a result of the AI ​​model's analysis. Next, it retrieves the current timestamp and saves it to the database along with the analysis results.

[1360] Step 5:

[1361] The server uses an emotion engine to analyze the user's emotional state. Image data acquired from the camera and audio data acquired from the microphone are input into the emotion engine, which then identifies the user's emotional state from their facial expressions and tone of voice.

[1362] Step 6:

[1363] The server retrieves the latest occupancy data from the database in real time to obtain congestion status. It then aggregates the latest occupancy information for each room and area and calculates the level of congestion.

[1364] Step 7:

[1365] The server manages resource allocation based on congestion levels and user emotions. For example, if a meeting room is full, it suggests an alternative, available meeting room to the user. It dynamically optimizes resources by adjusting allocations according to congestion levels. It also prioritizes assigning quieter areas to users who are stressed.

[1366] Step 8:

[1367] The server notifies users of congestion status and resource allocation information. It uses notification methods to inform users in real time about the availability of meeting rooms and workspaces. It also provides resource suggestions and customized notifications based on the user's emotional state. For example, it might provide users with a specific notification such as, "Meeting Room A is currently available. It's a quiet area that can help reduce stress."

[1368] Through the steps described above, this system can visualize office congestion and user emotional states in real time, enabling efficient and optimal resource management and notifications.

[1369] (Example 2)

[1370] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1371] Traditional office resource management systems fail to adequately visualize congestion levels and efficiently allocate resources. Furthermore, they cannot provide resources while considering users' emotional states, making it difficult to properly manage user satisfaction and stress levels. Therefore, it is necessary to efficiently utilize office space resources while optimally allocating resources in accordance with users' emotional states.

[1372] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1373] In this invention, the server includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, an emotion recognition means for analyzing the emotional state of users, and a notification means for notifying users of congestion status and resource allocation. This enables real-time visualization of congestion status and efficient resource management, as well as optimal resource allocation that takes into account the emotional state of users.

[1374] "Real-time" refers to a process or response that occurs almost instantly, with virtually no time delay.

[1375] "Imaging means" refers to a device that captures images or videos of a designated area. A camera is a specific example of this.

[1376] "Artificial intelligence analysis means" refers to artificial intelligence technology used to analyze captured images and obtain specific information (for example, counting the number of people).

[1377] An "artificial intelligence model" refers to a model used for analysis that possesses algorithms and structures employed by artificial intelligence analysis tools. Deep learning models are an example of this type of model.

[1378] "Recording means" refers to a device or system for saving analysis results to a database or similar.

[1379] A "database" refers to a digital system used to systematically store and manage analysis results and other information.

[1380] "Emotion recognition means" refers to a system or technology that analyzes a user's emotional state from their facial expressions, voice, etc.

[1381] "Management means" refers to a system or device that oversees and manages resource allocation and the operation of the entire system.

[1382] "Notification means" refers to methods or devices for communicating system analysis results and resource allocation information to users. Specifically, this includes email and push notifications.

[1383] "Congestion status" refers to the state of a particular area, indicating the density of people or the utilization rate.

[1384] "Resource allocation" refers to the act of efficiently and appropriately distributing available resources within a system (for example, meeting rooms or office desks).

[1385] "User" refers to an individual or organization that uses this system to manage and utilize resources within the office.

[1386] This invention provides a system that visualizes congestion levels in an office environment in real time, enabling efficient resource management, and further incorporates an emotion recognition function that recognizes the emotions of users. A specific example of this system is described below.

[1387] This system includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a recording means for saving the analysis results to a database, a management means for obtaining congestion status from the saved data and allocating resources, an emotion recognition means for analyzing the emotions of users, and a notification means for notifying users of congestion status and resource allocation.

[1388] The terminal (camera) captures images of a designated area, such as an office conference room or work area, in real time. The camera is high-resolution and periodically captures images, sending the data to a server. A network (e.g., Wi-Fi) is used for data transmission.

[1389] The server preprocesses the received image data (e.g., resizing, normalization) and counts the number of people in the image using AI analysis tools. Specifically, it uses artificial intelligence models (e.g., YOLO, OpenCV) to perform object detection and person counting. The analysis results are stored in a database (e.g., MySQL, PostgreSQL).

[1390] Furthermore, the server uses emotion recognition means (e.g., emotion recognition API) to analyze captured image and audio data and identify the user's emotional state. This allows for the determination of states such as stress or relaxation.

[1391] The server retrieves the latest congestion information from a stored database and calculates the degree of congestion. Based on the latest number of people in each area, the congestion status is visualized as a utilization rate. Resource allocation is dynamically performed using management tools based on congestion status and sentiment.

[1392] For example, resources can be allocated to less crowded areas or according to the emotional state of users. For instance, users experiencing stress could be assigned to quieter areas.

[1393] Using notification methods, the server informs users of real-time congestion status and resource allocation information. Notifications are sent via email, smartphone app push notifications, etc. This allows users to not only use rooms and seats efficiently, but also receive resource recommendations based on their emotional state.

[1394] Specific example

[1395] Example 1: Analysis of conference room congestion and sentiment analysis

[1396] A terminal (a camera installed in the conference room) captures images in real time and sends them to the server. The server analyzes the images using AI analysis tools (e.g., YOLO) and counts that there are currently 10 people in the conference room. After saving the analysis results and timestamps to a database, the server uses emotion recognition tools to analyze the emotions of the users from the video and detects that many users are in a stressed state. Based on the congestion level and emotional state, the server notifies users that "Conference Room 1 is currently available with 10 people, and the stress level is high."

[1397] Example 2: Managing work areas and assigning resources based on emotional state

[1398] Cameras installed in the work area capture images and send them to a server. The server analyzes the images using AI analysis tools (e.g., OpenCV) and counts that there are currently 20 people in the area. After saving the analysis results to a database to obtain the congestion status, the server analyzes the stress levels of users using emotion recognition tools. Based on the congestion status and emotional state, the server assigns a quiet area to users who are stressed and notifies them with a message such as, "Seat A is currently vacant. It is a quiet area that can help reduce stress."

[1399] Examples of prompts for generative AI models

[1400] Prompt message 1:

[1401] "Please provide a detailed explanation of the system's process flow, which analyzes the current congestion level and user emotional state of office meeting rooms in real time to determine the optimal resource allocation."

[1402] Prompt message 2:

[1403] "Please detail the specific process for allocating resources to quieter areas based on the level of congestion in the work area and the stress levels of users."

[1404] As described above, the present invention enables real-time visualization of congestion status and efficient resource management, as well as optimal resource allocation and notification that takes into account congestion status and the emotional state of users.

[1405] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1406] Step 1:

[1407] The device (camera) captures images of a specified area. The camera is high-resolution and set to capture one image per minute. The input is the current camera feed, and the output is the captured image data. Specifically, the camera acquires video from an office conference room and generates an image file.

[1408] Step 2:

[1409] The device sends the captured image data to the server. The input is the image data captured by the camera, and the output is the image data uploaded to the server. Specifically, the camera uploads the image file to the server using the HTTP protocol. A Wi-Fi connection is used for data transmission.

[1410] Step 3:

[1411] The server preprocesses the received image data. Specifically, it resizes and normalizes the images. The input is the image data uploaded to the server, and the output is the preprocessed image data. Specifically, it standardizes the image resolution and normalizes the colors.

[1412] Step 4:

[1413] The server analyzes pre-processed image data using artificial intelligence analysis tools. The input is pre-processed image data, and the output is the count of people within a given area. Specifically, the server uses the YOLO model to detect people in the image and counts their numbers.

[1414] Step 5:

[1415] The server saves the analysis results to a database. The input is the AI ​​analysis results (number of people count) and a timestamp, and the output is time-series data stored in the database. Specifically, the server saves the data to a database such as MySQL using the INSERT command.

[1416] Step 6:

[1417] The server analyzes the user's emotional state using emotion recognition technology. The input is captured image and audio data, and the output is the emotion analysis result. Specifically, the server sends data to an emotion recognition API, which identifies emotions from the user's facial expressions and tone of voice.

[1418] Step 7:

[1419] The server retrieves the latest congestion status from a stored database. The input is the latest user count data in the database, and the output is the congestion level. Specifically, the server executes an SQL query to extract the latest data from the database and calculates the utilization rate.

[1420] Step 8:

[1421] The server allocates resources based on congestion levels and sentiment levels. Inputs are congestion data and sentiment analysis results, while output is the resource allocation result. Specifically, the server dynamically selects the most suitable meeting room or area and allocates resources accordingly.

[1422] Step 9:

[1423] The server notifies users of information using a notification system. The input is resource allocation results and congestion status data, and the output is a notification message. Specifically, the server sends notification emails to users using a mail server and, if necessary, sends push notifications to smartphone applications.

[1424] (Application Example 2)

[1425] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1426] In modern brick-and-mortar stores, understanding customer congestion in real time and using that information to manage resources efficiently is a crucial challenge. Furthermore, providing services that consider the emotional state of visitors is expected to improve customer satisfaction. However, conventional systems have struggled to simultaneously achieve both real-time analysis and resource allocation based on emotional state. To solve this problem, a system is needed that performs both congestion and emotional analysis in real time and allocates resources appropriately for each visitor.

[1427] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1428] In this invention, the server includes an imaging means for capturing images in real time, an artificial intelligence analysis means for analyzing the captured images and counting the number of people in the area, a management means for obtaining congestion status from stored data and allocating resources, an emotion analysis means for analyzing the emotions of users, a notification means for notifying users of congestion status and resource allocation, and a means for recommending resources according to the emotional state of users. This makes it possible to visualize the congestion status in a physical store in real time and to dynamically allocate the optimal resources based on the emotional state of visitors.

[1429] An "imaging means" is a device for capturing images in real time.

[1430] "Artificial intelligence analysis means" refers to AI technology that analyzes captured images and counts the number of people in those images.

[1431] "Recording means" refers to a function for saving analysis results and acquired data to a database.

[1432] "Management means" refers to a function that retrieves congestion status from stored data and dynamically allocates resources.

[1433] "Emotion analysis means" refers to technology that analyzes a user's image and audio data to identify their emotional state.

[1434] "Notification means" refers to a function that notifies users in real time about congestion status and resource allocation information.

[1435] A "resource recommendation mechanism" is a function that dynamically suggests the most suitable resources based on the user's emotional state.

[1436] This invention relates to a system that visualizes congestion levels in physical stores in real time, enabling efficient resource management and service provision based on customer sentiment. The system includes imaging means, artificial intelligence analysis means, recording means, management means, sentiment analysis means, notification means, and resource recommendation means.

[1437] This system uses cameras as an imaging means to capture images in real time. For example, surveillance cameras installed in each area of ​​a physical store acquire current video footage and transmit that video data to a server.

[1438] The server uses artificial intelligence analysis to count the number of people in the area based on the received video data. This process utilizes pre-trained deep learning models and object detection algorithms. Specifically, the images are first pre-processed, such as resizing and normalization, and then the pre-processed images are input into the artificial intelligence model for analysis.

[1439] The analysis results are stored in a database using recording devices. For example, the number of people and timestamps for each area are recorded as time-series data and used for subsequent congestion analysis.

[1440] Next, the server analyzes the customer's emotional state using emotion analysis tools. This involves identifying the customer's emotional state from their facial expressions and tone of voice using video data acquired by the camera and audio data acquired by the microphone. A separate deep learning model is used for emotion analysis to estimate various emotions (e.g., stress, joy, etc.).

[1441] The management system retrieves the latest congestion status from stored data and dynamically allocates resources along with the user's emotional state. For example, if a particular area is congested or a customer's stress level is high, it will prioritize recommending less crowded or quieter areas.

[1442] Using notification mechanisms, the server informs users in real time about congestion levels and resource allocation. This allows users to select appropriate areas and take actions to avoid congestion. Furthermore, it provides customized services based on the customer's emotional state through resource recommendation mechanisms.

[1443] Specific example

[1444] Example 1: Analysis of congestion levels and sentiment analysis in physical stores

[1445] A terminal (a camera installed in the physical store) captures images in real time and sends them to a server. The server analyzes the images using artificial intelligence analysis and counts that there are currently 50 people in the store. After saving the analysis results and timestamps to a database, the server uses emotion analysis to analyze the emotions of the customers from the video and detects that many customers are in a stressed state. Based on the congestion level and emotional state, the server notifies store staff and customers that "Store area A is currently available with 50 people, and the stress level is high."

[1446] Example 2: Managing dedicated areas and assigning resources based on emotional state

[1447] Cameras installed in a designated area capture images and send them to a server. The server uses artificial intelligence analysis to analyze the images and counts that there are currently 30 people in the area. After saving the analysis results to a database to obtain the congestion status, the server uses emotion analysis to analyze the stress levels of the customers. Based on the congestion status and emotional state, the server assigns a quiet area to customers who are stressed and notifies them with a message such as, "Area B is currently empty. It is a quiet area that can help reduce stress."

[1448] Example of a prompt:

[1449] "Please explain how to apply a system that visualizes office congestion levels in real time."

[1450] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1451] Step 1:

[1452] The device captures images in real time.

[1453] Input: Video data captured by the camera

[1454] Operation: Cameras installed in each area of ​​the store capture images at specified intervals and send the data to the server in streaming format.

[1455] Output: Captured image data

[1456] Step 2:

[1457] The server uses artificial intelligence analysis to count the number of people based on the image data it receives.

[1458] Input: Captured image data

[1459] Operation: The received image data is preprocessed, such as by resizing and normalizing, and the preprocessed image is input into a deep learning model. The AI ​​model uses an object detection algorithm to detect people in the image and counts the number of people.

[1460] Output: Data on the number of people counted

[1461] Step 3:

[1462] The server saves the analysis results to a database.

[1463] Input: Counted number of people data, timestamp

[1464] Operation: Records the analysis results (number of people data) and the current timestamp in the database.

[1465] Output: Updated database

[1466] Step 4:

[1467] The server uses emotion analysis tools to analyze the customer's emotional state.

[1468] Input: Captured image data, audio data (if necessary)

[1469] Operation: Image and audio data are input into another deep learning model to estimate the customer's emotional state from their facial expressions and tone of voice. Various emotions are identified (e.g., stress, joy), and analysis results are obtained.

[1470] Output: Customer sentiment data

[1471] Step 5:

[1472] The server retrieves the latest congestion information from stored data and dynamically allocates resources.

[1473] Input: Number of people data, emotion data

[1474] Operation: Retrieves the latest headcount and sentiment data from the database and uses management tools to allocate resources based on congestion levels and sentiment states. Selects areas with low congestion or areas suitable for the customer's sentiment state.

[1475] Output: Resource allocation data

[1476] Step 6:

[1477] The server notifies the user of the information it has obtained.

[1478] Input: Resource allocation data, congestion status data

[1479] Operation: Uses notification methods to provide users (customers and staff) with real-time information on congestion levels and resource allocation. For example, it may notify users that a specific area is available or that a quiet area is suitable for stress relief.

[1480] Output: Notified information

[1481] Example of a prompt:

[1482] "Please explain how to apply a system that visualizes office congestion levels in real time."

[1483] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1484] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1485] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1486] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1487] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1488] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1489] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1490] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1491] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1492] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1493] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1494] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1495] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1496] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1497] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1498] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1499] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1500] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1501] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1502] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1503] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1504] The following is further disclosed regarding the embodiments described above.

[1505] (Claim 1)

[1506] A camera system that captures images in real time,

[1507] An AI analysis method that analyzes captured images and counts the number of people within the area,

[1508] A recording means for saving the analysis results to a database,

[1509] A management system that retrieves congestion status from stored data and allocates resources,

[1510] A system that includes a notification mechanism to inform users of congestion status and resource allocation.

[1511] (Claim 2)

[1512] The system according to claim 1, wherein an AI analysis means preprocesses an image and inputs the preprocessed image into an AI model.

[1513] (Claim 3)

[1514] The system according to claim 1, wherein the management means dynamically allocates resources based on congestion status.

[1515] "Example 1"

[1516] (Claim 1)

[1517] A means of capturing images in real time,

[1518] An artificial intelligence analysis method that analyzes captured images and counts the number of people in the area,

[1519] A means of saving the analysis results to a database,

[1520] A management system that retrieves congestion status from stored data and allocates resources,

[1521] A system that includes a notification mechanism to inform users of congestion status and resource allocation.

[1522] (Claim 2)

[1523] The system according to claim 1, wherein an artificial intelligence analysis means preprocesses an image and inputs the preprocessed image into an artificial intelligence model.

[1524] (Claim 3)

[1525] The system according to claim 1, wherein the management means includes the step of extracting the latest number of people data from a database and calculating the degree of congestion.

[1526] (Claim 4)

[1527] The system according to claim 1, wherein the management means dynamically allocates resources based on congestion status.

[1528] "Application Example 1"

[1529] (Claim 1)

[1530] An image acquisition method that captures images in real time,

[1531] An AI analysis method that analyzes captured images and counts the number of people within the area,

[1532] A recording means for saving the analysis results to a database,

[1533] A resource management system that retrieves congestion status from stored data and allocates resources,

[1534] A notification mechanism for informing users of congestion status and resource allocation,

[1535] A restaurant management system that monitors the cooking status and waiting times of restaurants and informs customers of the optimal ordering time and waiting time,

[1536] A system that includes delivery management features to optimize order priorities and delivery routes based on congestion levels.

[1537] (Claim 2)

[1538] The system according to claim 1, wherein an AI analysis means preprocesses an image and inputs the preprocessed image into an AI model.

[1539] (Claim 3)

[1540] The system according to claim 1, wherein the resource management means dynamically allocates resources based on congestion status.

[1541] "Example 2 of combining an emotion engine"

[1542] (Claim 1)

[1543] An imaging means for capturing images in real time,

[1544] An artificial intelligence analysis method that analyzes captured images and counts the number of people in the area,

[1545] A recording means for saving the analysis results to a database,

[1546] A management system that retrieves congestion status from stored data and allocates resources,

[1547] An emotion recognition method for analyzing the emotional state of the user,

[1548] A system that includes a notification mechanism to inform users of congestion status and resource allocation.

[1549] (Claim 2)

[1550] The system according to claim 1, wherein an artificial intelligence analysis means preprocesses an image and inputs the preprocessed image into an artificial intelligence model.

[1551] (Claim 3)

[1552] The system according to claim 1, wherein the management means dynamically allocates resources based on congestion status and emotional state.

[1553] "Application example 2 of combining emotional engines"

[1554] (Claim 1)

[1555] An imaging means for capturing images in real time,

[1556] An artificial intelligence analysis tool that analyzes captured images and counts the number of people in the area,

[1557] A recording means for saving the analysis results to a database,

[1558] A management system that retrieves congestion status from stored data and allocates resources,

[1559] A means of analyzing the emotions of users,

[1560] A notification mechanism for informing users of congestion status and resource allocation,

[1561] A system that includes means for recommending resources according to the user's emotional state.

[1562] (Claim 2)

[1563] The system according to claim 1, wherein an artificial intelligence analysis means preprocesses an image and inputs the preprocessed image into an artificial intelligence model.

[1564] (Claim 3)

[1565] The system according to claim 1, wherein the management means dynamically allocates resources based on congestion levels and the emotional state of users.

[1566] keyword:

[1567] Generative AI model, prompt sentence [Explanation of Symbols]

[1568] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A camera system that captures images in real time, An AI analysis method that analyzes captured images and counts the number of people within the area, A recording means for saving the analysis results to a database, A management system that retrieves congestion status from stored data and allocates resources, A system that includes a notification mechanism to inform users of congestion status and resource allocation.

2. The system according to claim 1, wherein an AI analysis means preprocesses an image and inputs the preprocessed image into an AI model.

3. The system according to claim 1, wherein the management means dynamically allocates resources based on the congestion status.

Images