Device and method for displaying occupancy status

Abstract

To enable information having higher reliability about a congestion state to be presented to a user by predicting a person leaving a facility in near feature on the basis of movement of each person.SOLUTION: Persons existing in a facility are detected, persons whose existing time becomes a prescribed threshold or greater are extracted from among existing persons as prediction objects, leaving of persons of the predicted objects is predicted on the basis of a prescribed leaving sign movement, and at least the number of existing persons (the number of attending persons) and the number persons predicted to leave (the number of leaving-predicted persons) are displayed on a guide screen of a user terminal. Also, time of updating the number of existing persons and the number of persons predicted to leave is displayed on the guide screen of the user terminal on the basis of the latest processing result of processing of detecting the person of existing persons and processing of predicting leaving. Also, the number of users currently browsing the guide screen displayed on the user terminal is displayed on the user terminal in response to access from the user terminal.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a device and method for displaying the status of a person's stay in a facility, which generates and presents information regarding the status of a person's stay in the facility based on video footage of the facility. [Background technology]

[0002] By distributing real-time congestion information for restaurants and other facilities to user terminals, and showing the current congestion status to users who are planning to use the facility, convenience for users can be improved. On the other hand, since the congestion level of a facility changes as people leave, it would be beneficial to predict future congestion levels and present those predictions to users as well.

[0003] Conventionally, a technology is known for predicting and presenting the congestion status of such facilities to users. This technology measures the number of people present based on detection results from sensors (such as motion sensors and cameras), and uses a predictive model generated by machine learning based on information about past congestion status at the facility to predict future congestion status and present the prediction results to users (see Patent Document 1). [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-189708 [Overview of the project] [Problems that the invention aims to solve]

[0005] Conventional technologies can predict future congestion levels using predictive models. However, these conventional technologies rely on machine learning based on historical congestion information for the entire facility, and therefore only predict trends in congestion levels—that is, how congestion will progress—from a statistical perspective. As a result, congestion often does not resolve as predicted, and there is a need to provide users with more reliable information regarding congestion levels.

[0006] Therefore, the main objective of the present invention is to provide a occupancy status display device and a occupancy status display method that can present users with more reliable information regarding congestion status by predicting which people will leave the facility in the near future based on the actions of each person. [Means for solving the problem]

[0007] The present invention provides a occupancy status display device that uses a processor to generate display information regarding the degree of congestion in a facility based on video footage of the area where people are staying in the facility. The processor detects people staying in the area, extracts people whose stay time exceeds a predetermined threshold from among the people staying, predicts the departure of the people selected as prediction targets based on predetermined departure prediction actions, and generates display information that includes at least the number of people staying and the number of people whose departure is predicted.

[0008] Furthermore, the present invention provides a method for displaying the status of occupancy, which involves a processor performing a process to generate display information regarding the degree of congestion in a facility based on video footage of the area where people are staying in the facility. The method includes detecting people staying in the area, extracting from among the people staying whose stay time exceeds a predetermined threshold as prediction targets, predicting the departure of the prediction targets based on predetermined departure prediction actions, and generating the display information which includes at least the number of people staying and the number of people whose departure is predicted. [Effects of the Invention]

[0009] According to the present invention, the number of people currently present and the number of people expected to leave are presented to the user. By considering the number of people expected to leave, the user can adjust and judge the level of congestion in the facility based on the number of people currently present. This allows for the presentation of more reliable information regarding the congestion status of the facility to the user. Furthermore, the process of predicting people's departure is performed only on people whose occupancy time exceeds a predetermined threshold, and the processing target is narrowed down to people who are likely to leave, thus reducing the processing load. [Brief explanation of the drawing]

[0010] [Figure 1] Overall configuration diagram of the occupancy status display system according to this embodiment. [Figure 2] Diagram illustrating the camera setup within the facility. [Figure 3] Block diagram showing the general configuration of the server. [Figure 4] Diagram illustrating the contents of the database of occupants managed on the server. [Figure 5] This diagram illustrates the types of behaviors indicating departure from one's seat and provides an example of the predictive score obtained for each type of behavior. [Figure 6] This diagram illustrates the overview of the server-side vacancy prediction process. [Figure 7] A flowchart illustrating an example of the procedures for presence detection, prediction target extraction, and absence prediction processing performed on the server. [Figure 8] An explanatory diagram showing the guidance screen displayed on the user's terminal. [Figure 9] Diagram illustrating the settings screen displayed on the administrator's terminal. [Modes for carrying out the invention]

[0011] A first invention made to solve the above problems is a stay situation presentation device that executes, by a processor, generation processing of display information regarding the congestion level of a facility based on an image obtained by photographing a stay area of a person in the facility. The processor detects a person staying in the stay area, extracts, as prediction targets, persons whose stay time is equal to or longer than a predetermined threshold from among the persons staying, predicts the departure of the persons targeted for prediction based on a predetermined departure omen action, and generates the display information including at least the number of persons staying and the number of persons whose departure is predicted.

[0012] According to this, the number of persons staying and the number of persons whose departure is predicted are presented to the user. By considering the number of persons whose departure is predicted, the user can correct and determine the congestion level of the facility based on the number of persons staying. Thereby, more reliable information regarding the congestion situation of the facility can be presented to the user. Further, the process of predicting the departure of a person is executed for persons whose attendance time is equal to or longer than a predetermined threshold, and the target of the process is narrowed down to persons with a high possibility of leaving, so that the processing load can be reduced.

[0013] Further, a second invention is configured such that the stay area is a presence detection area for detecting a person present, and the departure omen action is a departure omen action performed by a person present immediately before leaving.

[0014] According to this, appropriate information regarding the congestion situation of a facility provided with seats can be presented to the user.

[0015] Further, a third invention is configured such that the processor transmits the display information to the terminal device in response to an access from the terminal device operated by the user.

[0016] According to this, the user can easily view information regarding the congestion situation of the facility.

[0017] Furthermore, the fourth invention is configured such that the processor sets the threshold in response to an operation performed by the facility manager to specify the threshold.

[0018] According to this, facility managers can arbitrarily specify thresholds, taking into account the characteristics of the facility.

[0019] Furthermore, the fifth invention is configured such that the processor generates the display information, including the time at which the number of people currently staying and the number of people expected to leave were updated, based on the latest processing results of the process for detecting people currently staying and the process for predicting their departure.

[0020] According to this, users can easily understand how up-to-date the displayed information is.

[0021] Furthermore, the sixth invention is configured such that the processor generates the display information, including the number of users viewing the display information, in response to access from a terminal device operated by a user.

[0022] According to this, users can adjust their assessment of facility congestion based on the number of people currently present by considering the number of users viewing the displayed information, in addition to the number of people expected to leave.

[0023] Furthermore, the seventh invention is configured such that the departure warning action is at least one of the following: putting on clothing, picking up luggage, finishing handling electronic devices, putting away items, or putting items into a bag.

[0024] According to this, it is possible to accurately predict who will leave the facility in the near future.

[0025] Furthermore, the eighth invention is a method for displaying the status of occupancy in a facility, in which a processor performs a process to generate display information regarding the degree of congestion in a facility based on video footage of the area where people are staying in the facility, the method detects people staying in the area, extracts people from among the people staying whose stay time exceeds a predetermined threshold as prediction targets, predicts the departure of the prediction targets based on predetermined departure prediction actions, and generates the display information which includes at least the number of people staying and the number of people whose departure is predicted.

[0026] According to this, similar to the first invention, by predicting which people will leave the facility in the near future based on their actions, more reliable information about congestion can be presented to users.

[0027] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0028] (First Embodiment) Figure 1 is an overall configuration diagram of the occupancy status display system according to this embodiment.

[0029] This system displays the congestion status (occupancy rate) of a target facility (such as a restaurant) to users. The system consists of Camera 1, Server 2 (occupancy status display device), User Terminal 3 (terminal device), and Administrator Terminal 4 (terminal device). Camera 1, Server 2, User Terminal 3, and Administrator Terminal 4 are connected via a network.

[0030] Camera 1 is installed in the target facility and takes pictures of the inside of the facility.

[0031] Server 2 acquires video footage from Camera 1 of the facility and generates information about the facility's congestion status based on that footage. Server 2 has the functionality of a web server and distributes the facility congestion status information to user terminal 3. Server 2 may be installed in the target facility and perform processing for a single facility, or it may be configured as a cloud computer and perform processing for multiple facilities.

[0032] User terminal 3 is operated by users who are about to use the facility (prospective users). User terminal 3 displays a guidance screen (see Figure 8) regarding the facility's congestion status, which is distributed from server 2. Users can understand the facility's congestion status by viewing the guidance screen.

[0033] Administrator terminal 4 is operated by the facility administrator. Administrator terminal 4 displays a settings screen (see Figure 9) for setting conditions for processing performed by server 2. The administrator can specify processing conditions on the settings screen. Server 2 sets the processing conditions according to the administrator's actions.

[0034] In this embodiment, the facilities targeted are not particularly limited, but may include restaurants or employee cafeterias primarily used by individuals or small groups. Furthermore, the target facilities may be offices employing a free-address system where users can freely choose their seats. They may also include sports gyms or entertainment facilities where people gather at specific times.

[0035] Next, we will explain the overview of the processes performed by this system. Figure 2 is an explanatory diagram showing the shooting status of camera 1 in the facility.

[0036] The target facilities (such as restaurants) are equipped with seats (chairs) and tables. Camera 1 is installed on the ceiling of the facility. Camera 1 films people who are present in the facility.

[0037] In this embodiment, a seat occupancy detection area is pre-set on the image from camera 1. Based on the image of this seat occupancy detection area, the presence of a person, i.e., whether or not a person is sitting in a seat, is detected, and the current number of occupants is displayed to the user. This allows the user to understand the congestion level of the facility.

[0038] On the other hand, even if the current number of people present is high, if some people will finish using the facilities and leave in the near future, the actual level of congestion can be considered low. Therefore, relying solely on the current number of people present does not accurately reflect the congestion situation.

[0039] Therefore, in this embodiment, motion prediction technology is used to predict which people currently seated will leave their seats in the near future (departure prediction processing), distinguish between people who will remain seated and those who will leave in the near future, and present the number of people who will leave in the near future (predicted number of departures) in addition to the current number of people seated. This allows users to more accurately understand the congestion status of the facility.

[0040] Furthermore, performing absence prediction processing on all individuals currently present would place a heavy processing load on the system. Therefore, in this embodiment, since individuals who have just arrived are unlikely to leave their seats in the near future, the presence time (elapsed time from the start of presence) is calculated for each individual currently present. Individuals whose presence time exceeds a predetermined threshold are then extracted as prediction targets (targets for absence prediction processing) and the absence prediction processing is performed. This narrows the target of the absence prediction processing to individuals who are highly likely to leave their seats, thereby reducing the processing load on the absence prediction processing.

[0041] Here, the length of time customers spend in a facility varies depending on the establishment. For example, in establishments that primarily serve beverages, such as cafes, the length of stay is relatively short, while in establishments that primarily serve meals, such as family restaurants, the length of stay is relatively long. Therefore, it is desirable to set thresholds for each facility based on the trends in the length of stay at each facility. For example, in facilities where the length of stay tends to be short, the threshold could be set to 10 minutes, and in facilities where the length of stay tends to be long, the threshold could be set to 30 minutes.

[0042] In this embodiment, the administrator of the target facility can specify a threshold for the amount of time spent at the facility. While the threshold is set individually for each facility, it may also be set according to the type of facility. That is, statistical processing regarding the amount of time spent at the facility for each individual in the past may be performed for each type of facility, and the threshold for the amount of time spent at the facility may be set for each type of facility.

[0043] Furthermore, in this embodiment, the facility is targeted at restaurants and offices, and the congestion status of the facility is presented to the user as the occupancy status of the seats provided within the facility. However, it may also be possible to present the congestion status of facilities that do not have seats, such as clothing stores, to the user. In this case, in order to avoid confusing people moving within the facility, it is preferable to perform person tracking on the video feed from camera 1.

[0044] Furthermore, it is preferable to use a box camera with a predetermined field of view for camera 1. This makes it possible to acquire video footage that enables highly accurate seat absence prediction processing.

[0045] On the other hand, an omnidirectional camera that captures images in all directions may be used as part of camera 1. While an omnidirectional camera can acquire images suitable for presence detection processing, the images from an omnidirectional camera capture people from directly above, making it difficult to properly recognize people's movements, and therefore unsuitable for absence prediction processing. For this reason, images for absence prediction processing may be acquired by a box camera, while images for presence detection processing may be acquired by an omnidirectional camera. In this case, it is preferable to perform a process (calibration) that associates the position on the box camera's image with the position on the omnidirectional camera's image.

[0046] Next, we will describe the general configuration of Server 2. Figure 3 is a block diagram showing the general configuration of Server 2.

[0047] Server 2 comprises a communication unit 21, a storage unit 22, and a processor 23.

[0048] The communication unit 21 communicates with the camera 1, the user terminal 3, and the administrator terminal 4 via the network.

[0049] The memory unit 22 stores programs executed by the processor 23, etc. The memory unit 22 also stores video footage received from the camera 1. The memory unit 22 also stores registration information for the occupant person database (such as the start time of occupancy and the duration of occupancy).

[0050] The processor 23 performs various processes by executing programs stored in the memory unit 22. In this embodiment, the processor 23 performs tasks such as presence detection, prediction target extraction, absence prediction, statistical processing, and display processing.

[0051] In the occupancy detection process (stay detection process), the processor 23 detects people who are present in a seat based on the video feed from camera 1. Specifically, an occupancy detection area is pre-set on the video feed from camera 1, and an image of the occupancy detection area is extracted from the video feed from camera 1. Based on this image, it is determined whether or not a person is sitting in the seat. When an occupant is detected, that person is assigned a person ID.

[0052] In the prediction target extraction process, the processor 23 extracts individuals who are to be predicted (targets for the absence prediction process) from among the individuals whose presence has been detected in the presence detection process. Specifically, the presence time of each individual is compared with a predetermined threshold, and individuals whose presence time is equal to or greater than the threshold are extracted as prediction targets.

[0053] In the departure prediction process, the processor 23 predicts, based on the video feed from camera 1, whether the person being predicted will leave their seat in the near future (within a predetermined time from the current time). The departure prediction process detects when the person being predicted has performed a predetermined departure warning action. A departure warning action is an action performed by a person who is about to leave their seat (an action performed immediately before leaving), and when this departure warning action is detected, it can be predicted that the person who performed that action will leave their seat in the near future.

[0054] Here, "leaving one's seat" refers to the act of leaving one's seat in order to leave the establishment. Examples of actions indicating an intention to leave one's seat include putting on clothing (such as a jacket or hat), picking up belongings (such as a bag), finishing the use of electronic devices (such as a laptop, smartphone, or tablet), putting away items that were being used (such as books or dishes), and standing up from one's seat. It should be noted that actions indicating an intention to leave one's seat are actions performed by a person intending to leave their seat, and can be distinguished from actions performed when a person temporarily leaves their seat to go to the restroom or for other errands, leaving their belongings unattended.

[0055] Furthermore, the absence prediction process utilizes action prediction technology. This technology predicts actions to be performed at a future time (a predetermined time after the present) based on video footage from the past (a predetermined period of time prior to the present) to the present.

[0056] The seat absence prediction process includes video extraction, feature extraction, prediction score acquisition, and seat absence detection.

[0057] In the video extraction process, processor 23 extracts clip footage from the stored video, based on the current time. Clip footage is video footage included in the target period from the past (a predetermined period prior to the present) to the present. In addition, in the video extraction process, processor 23 extracts unit footage for each time point from the clip footage. Unit footage is a collection (video) of a predetermined number (e.g., 6) of past captured images (frames).

[0058] In the feature extraction process, the processor 23 extracts feature information for each time point from the unit video for each time point. In this embodiment, spatial feature information, motion feature information, and object feature information are extracted. Spatial feature information is information about the positional relationships of objects that appear in the video. Motion feature information is information about the movements that appear in the video. Object feature information is information about the types of objects that appear in the video. Note that one or two of the spatial feature information, motion feature information, and object feature information may be extracted.

[0059] Furthermore, the feature extraction process can utilize a feature extraction model (machine learning model) generated by machine learning, such as deep learning. In this case, the unit video is input to the feature extraction model, and the feature information output from the feature extraction model is obtained.

[0060] In the prediction score acquisition process, processor 23 acquires a prediction score for each type of pre-leaving action (such as putting on a jacket or picking up a bag) based on the feature information for each time point acquired in the feature extraction process. The prediction score is an evaluation value that represents the probability that a person performed a pre-leaving action.

[0061] Furthermore, the prediction score acquisition process can utilize a prediction score acquisition model (machine learning model) generated by machine learning, such as deep learning. In this case, feature information is input to the prediction score acquisition model, and a prediction score for each type of absenteeism behavior (behavior class) output from the prediction score acquisition model is obtained. For example, a neural network may be used for the prediction score acquisition model. Alternatively, a prediction score acquisition model may be provided for each type of feature information (spatial feature information, motion feature information, object feature information), and a prediction score may be acquired for each type of feature information.

[0062] In the absence determination process, the processor 23 determines whether the person to be predicted will leave their seat in the near future (within a predetermined time from the current time) based on the predicted score for each type of absence-indicating behavior obtained in the prediction score acquisition process. Specifically, if the highest predicted score among the predicted scores for each type of absence-indicating behavior is above a predetermined threshold, it is determined that the person to be predicted will leave their seat in the near future.

[0063] Furthermore, the absence detection process can utilize an absence detection model (machine learning model) generated by machine learning, such as deep learning. In this case, the predicted score for each type of absence-indicating behavior is input to the absence detection model, and the absence detection result output from the absence detection model is obtained. For example, a support vector machine (SVM) may be used as the absence detection model.

[0064] In the statistical processing, processor 23 generates statistical information to be presented to the administrator. This statistical information is presented to the administrator as a reference when specifying a threshold for presence time when extracting individuals to be targeted for prediction (target of absence prediction processing) from among those currently present.

[0065] During the display process, processor 23 generates display information for the guidance screen (see Figure 8) to be displayed on the user terminal 3. Also during the display process, processor 23 generates display information for the settings screen (see Figure 9) to be displayed on the administrator terminal 4.

[0066] Next, we will explain the database of occupants managed by Server 2. Figure 4 is an explanatory diagram showing the contents of the occupant database managed by Server 2.

[0067] Server 2 performs an occupancy detection process based on the video feed from Camera 1 to detect people who are present. When an occupant is detected, a person ID is assigned to that person, and this person ID and the time they started being present are registered in the occupant database. Server 2 also calculates the occupancy time (the elapsed time from the start of their presence to the current time) for each occupant, and this occupancy time is registered in the occupant database. In this example, the current time is 13:50.

[0068] Furthermore, Server 2 performs a process (prediction target extraction process) to extract individuals who will be targeted for prediction (targets for absence prediction processing) from among the individuals whose presence has been detected by the presence detection process. In the prediction target extraction process, the presence time of each individual is compared with a predetermined threshold by referring to the presence database, and individuals whose presence time is equal to or greater than the threshold are extracted as prediction targets. In this example, individuals with person IDs 11 and 12 have presence times equal to or greater than the threshold (30 minutes), so these individuals are extracted as prediction targets.

[0069] Next, we will explain the signs of leaving one's seat. Figure 5 is an explanatory diagram showing the types of signs of leaving one's seat and an example of the prediction score obtained for each type of sign of leaving one's seat.

[0070] On Server 2, for each type of behavior indicating absence (behavior class), a process is performed to obtain a prediction score that represents the probability that a person performed that behavior (prediction score acquisition process).

[0071] Actions that indicate leaving one's seat include putting on a jacket. Other actions that indicate leaving one's seat include putting on clothing, such as putting on a hat. Also, actions that indicate leaving one's seat include picking up a bag. Other actions that indicate leaving one's seat include picking up luggage such as a cart. Furthermore, actions that indicate leaving one's seat include closing a laptop computer, putting a laptop computer into a bag, and putting a smartphone into a bag. Other actions that indicate leaving one's seat include ending the use of various electronic devices. Also, actions that indicate leaving one's seat include closing a book. Other actions that indicate leaving one's seat include putting away items that were being used, such as stacking dishes. Furthermore, actions that indicate leaving one's seat include putting a book into a bag. Other actions that indicate leaving one's seat include putting various items placed on the table into a bag. Furthermore, if the type of premonitory behavior for leaving one's seat has not been identified when an employee leaves their seat, a new type may be established and added as a premonitory behavior for leaving one's seat by accumulating the actions taken immediately before the employee leaves and performing additional learning.

[0072] Next, the departure prediction process performed by server 2 will be described. FIG. 6 is an explanatory diagram showing an overview of the departure prediction process.

[0073] In server 2, a process is performed to predict whether a person to be predicted will depart in the near future (within a predetermined time from the current time) (departure prediction process). In the example shown in FIG. 6, time t 14 is the current time, and time t 15 is the prediction time. α is a parameter that determines how far in the future to predict from the current time, and the prediction time is set according to α. For example, if α = 10s, then 10 seconds ahead is predicted.

[0074] In server 2, first, from the accumulated video, a clip video based on the current time is cut out, and at the same time, unit videos V1 to V 14 at each time t1 to t 14 are extracted (video extraction process). The clip video is a video included in the target period from the past up to the present that goes back a predetermined period from the current time. The unit video V t at time t is a set (video) of a predetermined number (for example, 6) of past captured images (frames) based on time t.

[0075] [[ID=2--1]] Next, in server 2, a process is performed to extract feature information at each time t1 to t 14 from the unit videos V1 to V 14 at each time t1 to t 14 (feature extraction process).

[0076] Next, in server 2, based on the feature information at each time t1 to t 14 , a process is performed to obtain a prediction score for each type of departure omen action as a prediction result (prediction score acquisition process).

[0077] In the example shown in Figure 6, from time t1 to time t5, only the process of extracting feature information for each time from the unit images V1 to V5 for each time is performed. At time t6, feature information is extracted from unit image V6, and the prediction result for time t6 is obtained based on the feature information for each time extracted from unit images V1 to V6 from time t1 to t6. The same process is repeated for times t7 to t 14 The prediction result is obtained. At each time point, the previous prediction result is successively updated, and the prediction result for that time point is obtained.

[0078] Next, on server 2, the current time t 14 Based on the prediction score for each type of behavior indicating departure, which is the prediction result, a process is performed to determine whether the person being predicted will leave their seat in the near future (within a predetermined time from the current time) (departure determination process). In the example shown in Figure 6, at time t 14 Based on the prediction results, time t 15 In this step, it is determined whether or not the person being predicted will leave their seat.

[0079] Next, we will describe the procedures for presence detection, prediction target extraction, and absence prediction performed on Server 2. Figure 7 is a flowchart showing an example of the procedures for presence detection, prediction target extraction, and absence prediction. These flows are executed periodically.

[0080] As shown in Figure 7(A), in the occupancy detection process, Server 2 first acquires the image captured by Camera 1 (ST101). Next, Server 2 extracts an image of the occupancy detection area for each seat from the image captured by Camera 1, and detects a person present in the occupancy detection area based on that image (person detection process) (ST102).

[0081] Next, Server 2 determines whether a person in the presence detection area is present or absent (Presence Determination Process) (ST103). At this time, if a person in the presence detection area is detected consecutively a predetermined number of times or more, it is determined that the person in the presence detection area is present. For example, if images captured by Camera 1 are acquired at 10-second intervals, and a person in the presence detection area is detected 6 times consecutively, that is, if a person is present in the presence detection area for 1 minute consecutively, it is determined that the person is present. In the presence determination process, a classifier that has been pre-trained using deep learning to distinguish between present and absent states can be used. During the presence determination process, the classifier calculates the likelihood (probability) of the present state and the likelihood of the absent state for the image in the presence detection area, and by comparing the two likelihoods, it is possible to determine whether the person is present or absent.

[0082] If a person in the presence detection area is determined to be present (Yes in ST103), Server 2 registers the person ID assigned to the person determined to be present, as well as the start time of presence, in the present person database (ST104).

[0083] As shown in Figure 7(B), in the prediction target extraction process, Server 2 first selects the person to be determined from among the people whose presence has been detected in the presence detection process (ST201).

[0084] Next, Server 2 determines whether the presence time of the person being evaluated is equal to or greater than the threshold (ST202).

[0085] If the presence time of the person being evaluated exceeds a threshold (Yes in ST202), Server 2 sets that person as a prediction target (target for absence prediction processing) (ST203).

[0086] As shown in Figure 7(C), in the absence prediction process, Server 2 first extracts clip video based on the current time from the stored video, and then extracts unit video for each time point from that clip video (video extraction process) (ST301).

[0087] Next, Server 2 uses a feature extraction model (machine learning model) to extract feature information from the unit video (feature extraction process) (ST302).

[0088] Next, Server 2 uses a prediction score acquisition model (machine learning model) to acquire a prediction score for each type of behavior indicating absence from the desk, based on the characteristic information of each time point included in the target period (prediction score acquisition process) (ST303).

[0089] Next, Server 2 determines whether the person being predicted will leave their desk in the near future (within a predetermined time from the current time) based on the prediction score for each type of behavior indicating absence (absence determination process) (ST304).

[0090] If it is determined that the person being predicted will leave their seat in the near future (Yes in ST304), that person is added to the count of people predicted to leave their seats (ST305).

[0091] Next, we will explain the guidance screen 31 displayed on the user terminal 3. Figure 8 is an explanatory diagram showing the guidance screen 31 displayed on the user terminal 3.

[0092] On user terminal 3, a guidance screen 31 is displayed, informing the user of the store's congestion status. By viewing the guidance screen 31, the user can understand the current congestion status of the store they wish to visit.

[0093] The guidance screen 31 is equipped with a connection destination display unit 32. When a user terminal 3 accesses server 2 by specifying the address (URL) for viewing a specific store, the guidance screen 31 (Web page) is displayed on the user terminal 3, and the address for viewing the store is displayed in the connection destination display unit 32.

[0094] Furthermore, the guidance screen 31 is equipped with a facility information display unit 33. The facility information display unit 33 displays information about the target facility (for example, name, location, etc.). Facility information is registered in advance by the administrator. In addition, the facility information may include privacy-protected images (images with masked people) generated from live images of the store.

[0095] Furthermore, the guidance screen 31 is equipped with an occupancy display unit 34 and an estimated number of people leaving their seats display unit 35. The occupancy display unit 34 displays the occupancy number, that is, the number of people currently present in the facility. The estimated number of people leaving their seats display unit 35 displays the estimated number of people leaving their seats, that is, the number of people currently present in the facility who are predicted to leave their seats in the near future. This allows users to adjust their assessment of the facility's congestion level based on the occupancy number by considering the estimated number of people leaving their seats. Specifically, for example, even if the facility is crowded due to a high number of occupants, if the estimated number of people leaving their seats is high, users can determine that the congestion will soon ease. The occupancy number is obtained through occupancy detection processing. The estimated number of people leaving their seats is obtained through departure prediction processing.

[0096] Furthermore, the guidance screen 31 is equipped with an update time display unit 36. The update time display unit 36 ​​displays the update time of the information displayed on the guidance screen 31, that is, the time when the number of people present and the number of people predicted to be absent were updated based on the latest processing results of the presence detection process and the absence prediction process. This allows users to understand how up-to-date the displayed information (number of people present and number of people predicted to be absent) is.

[0097] The displayed information will be updated periodically at predetermined intervals (e.g., every 15 minutes). Furthermore, the update interval may be longer than the interval at which the occupancy detection and vacancy prediction processes are executed (e.g., every 1 minute). The frequency of the displayed information updates may also be varied depending on the characteristics of the facility and the level of congestion.

[0098] Furthermore, the guidance screen 31 is equipped with a viewer count display unit 37. The viewer count display unit 37 displays the number of people currently viewing the information screen 31, that is, the number of users currently viewing the information screen 31. This allows users to adjust their assessment of the facility's congestion level based on the number of people present by considering the number of people currently viewing the information in addition to the number of people predicted to be away from their seats. Specifically, for example, even if the number of people predicted to be away from their seats is high, if the number of people currently viewing the information screen is also high, users can determine that congestion will not be significantly alleviated as more and more users will continue to visit. The number of people currently viewing the information screen is obtained based on the number of user terminals 3 that have transmitted the information displayed on the information screen 31 in response to access from user terminals 3.

[0099] Next, we will explain the settings screen 41 displayed on the administrator terminal 4. Figure 9 is an explanatory diagram showing the settings screen 41 displayed on the administrator terminal 4.

[0100] On the administrator terminal 4, a settings screen 41 is displayed for setting the conditions for the processing performed on server 2 in response to the administrator's operations. On the settings screen 41, the administrator can specify a threshold value for presence time when extracting individuals to be used as prediction targets (targets for absence prediction processing) from among those currently present in the prediction target extraction process, based on the length of their presence. In addition, the settings screen 41 presents the administrator with statistical information to help them determine the presence time threshold value.

[0101] The settings screen 41 is provided with a connection destination display section 42. When a user terminal 3 accesses server 2 by specifying the settings address (URL) of a specific store, the settings screen 41 (Web page) is displayed on the user terminal 3, and the store's settings address is displayed in the connection destination display section 42.

[0102] Furthermore, the settings screen 41 is provided with a statistical graph display unit 43. The statistical graph display unit 43 displays a histogram as a statistical graph, representing the distribution of attendance time. In the histogram, the vertical axis represents the number of people (frequency), and the horizontal axis represents the range of attendance time (less than 15 minutes, 15 minutes or more but less than 30 minutes, 30 minutes or more but less than 45 minutes, 60 minutes or more). The statistical graph is generated based on statistical information obtained through statistical processing.

[0103] Furthermore, the settings screen 41 is provided with a display period specification section 44. In the display period specification section 44, the administrator can input the display period, that is, the period (start date and time and end date and time) for the statistical graph to be displayed in the statistical graph display section 43. Alternatively, the administrator may be allowed to select from pre-set periods (for example, the entire past period, the past year, the past three months, etc.).

[0104] Furthermore, the settings screen 41 is provided with a recommended value display section 45. The recommended value display section 45 displays recommended values ​​for the attendance time threshold. Server 2 calculates recommended values ​​for the threshold by performing statistical processing on past attendance times for each person stored in the memory unit 22. For example, the mode of attendance time (the attendance time with the highest number of people (frequency)) is set as the recommended value. In addition, the average attendance time is set as the recommended value.

[0105] The settings screen 41 also includes a threshold input section 46 and an OK button 47. In the threshold input section 46, the administrator can input a threshold value for occupancy time. The administrator can input a threshold value they have independently determined, referring to the statistical graph displayed in the statistical graph display section 43. The administrator can also adopt the recommended threshold value displayed in the recommended value display section 45. For example, if the threshold input section 46 is left blank and the administrator operates the OK button 47, the server 2 will set the threshold value to the value displayed in the recommended value display section 45. This allows setting a threshold value that corresponds to the actual usage of the target store (facility). Furthermore, regarding statistical information, multiple statistical graphs may be displayed for various periods, such as morning, afternoon, weekdays, and holidays, and recommended threshold values ​​or threshold inputs may be accepted for each period, setting threshold values ​​that reflect the situation for each period.

[0106] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. [Industrial applicability]

[0107] The occupancy status display device and method according to the present invention have the effect of being able to present more reliable information regarding congestion to users by predicting which people will leave the facility in the near future based on the actions of each person, and are useful as an occupancy status display device and method that generate and present information regarding the occupancy status of people in a facility based on video footage of the facility. [Explanation of Symbols]

[0108] 1 Camera 2. Server (Status display device) 3. User terminal (terminal device) 4. Administrator terminal (terminal device) 22 Memory section 23 processors 31. Guidance screen 34 Number of people present display section 35. Display section for the predicted number of people away from their seats. 36 Update time display section 37. Display section for the number of people currently viewing the page 41 Settings screen 46. ​​Threshold Input Section

Claims

1. A dwelling status display device that uses a processor to generate display information regarding the degree of congestion of a facility based on video footage of the areas where people are staying within the facility, The aforementioned processor, The system detects a person staying in the aforementioned area, From among the persons staying, those whose length of stay exceeds a predetermined threshold are selected as the subjects for prediction. Based on predetermined departure indicators, the departure of the person targeted for prediction is predicted. A occupancy status display device characterized by generating the display information which includes at least the number of people currently staying and the number of people expected to leave.

2. The aforementioned occupancy area is an occupancy detection area that detects people who are present. The occupancy status display device according to claim 1, characterized in that the aforementioned departure warning action is a departure warning action performed by a person who is currently present immediately before leaving their seat.

3. The aforementioned processor, The occupancy status display device according to claim 1, characterized in that it transmits the display information to the terminal device in response to access from the terminal device operated by the user.

4. The aforementioned processor, The occupancy status display device according to claim 1, characterized in that the threshold is set in response to an operation performed by the facility manager to specify the threshold.

5. The aforementioned processor, The occupancy status display device according to claim 1, characterized in that it generates the display information including the time when the number of people currently occupying the premises and the number of people expected to leave are updated, based on the latest processing results of the process for detecting people currently occupying the premises and the process for predicting departure.

6. The aforementioned processor, The occupancy status display device according to claim 1, characterized in that it generates the display information, including the number of users viewing the display information transmitted to the terminal device in response to access from a terminal device operated by a user.

7. The device for indicating the status of stay according to claim 1, characterized in that the aforementioned departure indicator action is at least one of the following: putting on clothing, picking up luggage, ending the use of electronic devices, putting away items, or putting items into a bag.

8. A method for displaying the occupancy status of a facility, which uses a processor to generate display information regarding the degree of congestion of a facility based on video footage of the area where a person is staying within the facility, The system detects a person staying in the aforementioned area, From among the persons staying, those whose length of stay exceeds a predetermined threshold are selected as the subjects for prediction. Based on predetermined departure indicators, the departure of the person targeted for prediction is predicted. A method for displaying the status of a person's stay, characterized by generating the display information which includes at least the number of people currently staying and the number of people who are expected to leave.

Images