State estimation system and state estimation program

Abstract

To provide a state estimating system and a state estimating program for precisely estimating the state of an attention region within each frame image input from a camera to reduce the process load of an estimating process.SOLUTION: A state estimating process estimates, on the basis of a label given to the image of a table (an attention region) in each frame image, the provisional state of the table, and estimates, on the basis of the provisional state (the provisional state between times T to (T+2)) of the table in the frame image after the time T and of the final state (the final state between times (T-2) to (T-1)) of the table in the frame image before the time T, the final state of the table in the frame image at the time T. Accordingly, using a simple MLC model, even if a label is given to the image of the attention region, the adverse effect of noises in the label on the estimation result of the final state can be reduced, it is unnecessary to use a complicated MLC model, and the process load of the estimating process can be reduced.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to a state estimation system and a state estimation program. [Background technology]

[0002] Conventionally, in a deep neural network (DNN) for object recognition, there is a deep neural network model (neural network model for multi-label classification) that returns (outputs) multiple correct labels when an image is input. By using this neural network model for multi-label classification (MLC (Multi Label Classification)), it is possible to determine the type of each object for one or more objects reflected in an image of a region of interest in each frame image input from a camera, and it is possible to estimate the current state of the region of interest. If it is possible to estimate the state of a region of interest in each frame image input from a camera as described above, it is possible to improve the work efficiency of staff at restaurants and other stores, post offices, banks, hospitals, etc. by notifying the staff of these facilities of the state of the region of interest (for example, the state of a table in a restaurant, or the state of a counter at a post office, bank, or hospital). An example of a multi-label classifier is disclosed in, for example, Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-8992 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in general, output data (e.g., one or more labels and the likelihood of these labels) from a simple multi-label classification neural network model (hereinafter referred to as an "MLC model") of a type that assigns labels based on only one still image (an image of a region of interest in each frame image input from a camera) contains noise, so it is not possible to accurately estimate the state of the region of interest. This is a common problem with simple multi-label classification algorithms of a type that assign labels based on only one still image other than the above simple MLC model.

[0005] Furthermore, in fields such as behavior recognition, complex multi-label classification algorithms (e.g., complex MLC models) are also used that can return accurate inference results based on a time series of frame images (e.g., six frame images at times t, t-1, t-2, t-3, t-4, t-5, and t-6). However, such multi-label classification algorithms process multiple frame images at once, which increases the processing load of the inference process (on the processor).

[0006] The present invention is devised to solve the above-mentioned problems, and aims to provide a state estimation system and a state estimation program that are capable of accurately estimating the state of a region of interest in each frame image input from a camera, while having a small processing load for the inference process. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, a state estimation system according to a first aspect of the present invention includes a label assignment means for assigning one or more labels to an image of a region of interest in each frame image input from a camera, a provisional state estimation means for estimating a provisional state of the region of interest based on the labels assigned to the image of the region of interest by the label assignment means, and a final state estimation means for estimating a final state of the region of interest in a frame image at a predetermined time among a plurality of past frame images input from the camera, wherein the final state estimation means Past ContinuousAmong the plurality of frame images, Time a provisional state of the region of interest in a frame image; A provisional state of the attention area in a frame image after the predetermined time; A final state of the region of interest in the frame image at the given time is estimated based on a final state of the region of interest in a frame image prior to the given time. In this state estimation system, it is desirable for the final state estimation means to estimate the most frequent state among the interim state of the area of ​​interest in the frame image at the specified time, the interim state of the area of ​​interest in one or more frame images after the specified time, and the final state of the area of ​​interest in one or more frame images before the specified time to be the final state of the area of ​​interest in the frame image at the specified time.

[0008] In this state estimation system, the label assignment means may assign a likelihood of each label to the image of the area of ​​interest, and the provisional state estimation means may compare the likelihood of each label with a threshold value of the likelihood of each label to estimate a provisional state of the area of ​​interest.

[0009] In this state estimation system, if it is not permissible from the state machine representing the state transition rules that the final state of the area of ​​interest in the frame image at the specified time be the final state estimated by the final state estimation means, it is desirable to further provide a state transition verification means for maintaining the final state of the area of ​​interest in the frame image at the specified time to be the final state of the area of ​​interest in the frame image immediately preceding this frame image.

[0010] In this state estimation system, it is desirable that the state transition verification means allows the final state of the area of ​​interest to transition to the final state estimated by the final state estimation means when the provisional state of the area of ​​interest estimated by the provisional state estimation means remains the same for a predetermined period of time or more, even if it is not permissible from the perspective of a state machine representing the state transition rules that the final state of the area of ​​interest should become the final state estimated by the final state estimation means.

[0011] This state estimation system may further include an alarm means for notifying a user of the state estimation system, and the alarm means may be configured to notify the user of the state estimation system when the final state of the area of ​​interest estimated by the final state estimation means satisfies a predetermined condition.

[0012] It is preferable that this state estimation system further includes a notification means for notifying a user of the state estimation system, and that the notification means notifies the user of the state estimation system when the final state of the area of ​​interest after verification by the state transition verification means satisfies a predetermined condition.

[0013] In this condition estimation system, it is desirable that the notification means outputs a notification screen to a display device to notify the user, and that the notification screen includes notification information for the user and an image based on the frame image most recently input from the camera.

[0014] A state estimation program according to a second aspect of the present invention causes a computer to function as a label assignment means for assigning one or more labels to an image of a region of interest in each frame image input from a camera, a provisional state estimation means for estimating a provisional state of the region of interest based on the labels assigned to the image of the region of interest by the label assignment means, and a final state estimation means for estimating a final state of the region of interest in a frame image at a predetermined time among a plurality of past frame images input from the camera, the final state estimation means comprising: Past Continuous Among the plurality of frame images, Time a provisional state of the region of interest in a frame image; A provisional state of the attention area in a frame image after the predetermined time; A final state of the region of interest in the frame image at the given time is estimated based on a final state of the region of interest in a frame image prior to the given time. In this state estimation program, it is desirable for the final state estimation means to estimate the most frequent state among the provisional state of the area of ​​interest in the frame image at the specified time, the provisional state of the area of ​​interest in one or more frame images after the specified time, and the final state of the area of ​​interest in one or more frame images before the specified time to be the final state of the area of ​​interest in the frame image at the specified time.

[0015] In this state estimation program, it is desirable to have the computer further function as a state transition verification means that maintains the final state of the area of ​​interest in the frame image at the specified time as the final state of the area of ​​interest in the frame image one frame before this frame image if it is not permissible from the state machine representing the rules of state transition that the final state of the area of ​​interest in the frame image at the specified time be the final state estimated by the final state estimation means. Effect of the Invention

[0016] According to the state estimation system according to the first aspect of the present invention and the state estimation program according to the second aspect of the present invention, Consecutive Among the multiple frame images, a predetermined Time A provisional state of the region of interest in the frame image and the above a provisional state of the region of interest in a frame image after a predetermined time; The final state of the region of interest in the frame image at the specified time is estimated based on the final state of the region of interest in the frame image prior to the specified time. This makes it possible to reduce the effect of noise contained in the label on the estimation result of the final state even if a label is assigned to the image of the region of interest using a simple multi-label classification algorithm (e.g., a simple MLC model) that assigns a label based only on the one still image (the image of the region of interest in each frame image input from the camera). Therefore, it is possible to accurately estimate the (final) state of the region of interest in each frame image.

[0017] Also, according to the first aspect of the present invention State Estimation SystemAccording to the state estimation program of the second aspect, as described above, even if a simple multi-label classification algorithm (e.g., a simple MLC model) that assigns a label based on only one still image (an image of the region of interest in each frame image) is used, the (final) state of the region of interest in each frame image can be accurately estimated, so there is no need to use a complex multi-label classification algorithm (e.g., a complex MLC model) that assigns a label based on a plurality of (temporally) consecutive still images (frame images in a time series). Therefore, it is possible to reduce the processing load of the inference process (in the processor). [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram showing a schematic configuration of a state estimation system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a block diagram showing a schematic hardware configuration of an analysis box in FIG. 1. [Diagram 3] A functional block diagram of the CPU in the analysis box. [Figure 4] 4 is a flowchart showing an overview of a state estimation process in the state estimation system. [Diagram 5] FIG. 5 is an explanatory diagram of the process (pre-processing 1) of S2 in FIG. 4. [Figure 6] FIG. 5 is an explanatory diagram of the process of S5 in FIG. 4. [Figure 7] FIG. 2 is a diagram showing a state machine that represents the rules for state transition of each table in the state estimation system. [Figure 8] 4 is a flowchart of a provisional state estimation process in the state estimation system. [Figure 9] FIG. 7 is a diagram showing an inference result by the inference engine in FIG. 6. [Figure 10] 4 is a flowchart of a final state estimation process in the state estimation system. [Figure 11] FIG. 4 is an explanatory diagram of a state buffer used in the final state estimation process. [Figure 12]FIG. 11 is an explanatory diagram of the processes of S21 to S27 in FIG. [Figure 13] FIG. 11 is an explanatory diagram of a case where a state transition that is not permitted from the perspective of a state machine is permitted in the state estimation system. [Figure 14] FIG. 11 is an explanatory diagram of a case where a notification (alert) is given to staff in the state estimation system. [Figure 15] FIG. 13 is an explanatory diagram of a notification screen used for notifying the above staff. [Figure 16] FIG. 13 is a diagram showing a state machine in a state estimation system of modification 1-1. [Figure 17] FIG. 13 is an explanatory diagram of a region of interest in the state estimation system of modification 1-1. [Figure 18] FIG. 13 is a diagram showing a state machine in a state estimation system of modification 1-2. [Figure 19] FIG. 13 is an explanatory diagram of a region of interest in the state estimation system of modification 1-2. [Figure 20] FIG. 13 is a diagram showing a state machine in a state estimation system of modification 1-3. [Figure 21] FIG. 13 is a diagram showing a notification screen in the state estimation system of modification 1-3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, a state estimation system and a state estimation program according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a state estimation system 10 according to this embodiment. In this embodiment, an analysis box 1 ("computer" in the claims), a 360-degree camera 2 which is a network camera (IP (Internet Protocol) camera), and a signage 3 ("display device" in the claims) are arranged in a restaurant Tune Shop S. As shown in FIG. 1, the state estimation system 10 includes an analysis box 1, one or more 360-degree cameras 2 (hereinafter, abbreviated as "camera 2"), a signage 3, a hub 4, and a router 6 arranged in the store S, and an AI analysis server 7 and a management server 8 on a cloud C. However, among the above components, the router 6, the AI ​​analysis server 7, and the management server 8 are merely auxiliary components in the state estimation system 10.

[0020] The camera 2 is a network camera that has an IP address and can be directly connected to a network, and is also a 360-degree camera that can be attached to a ceiling or the like and capture a wide area in the store S. As shown in FIG. 1, the analysis box 1 is connected to one or more cameras 2 via a LAN (Local Area Network) 5 and a hub 4, and performs inference processing such as estimation processing of the state of each area of ​​interest in each frame image (specifically, the state of each table) based on each frame image input from each of these cameras 2. The signage 3 is also used to display a notification screen 71 (see FIG. 15) described later, and the like.

[0021] 1, the analysis box 1 is connected to an AI analysis server 7 and a management server 8 on the cloud C via a hub 4 and a router 6. The AI ​​analysis server 7 analyzes, for example, the behavior of people in each store based on the inference processing results of the analysis box 1, and converts the information on the analysis results into data that is easy to use for various applications such as marketing and crime prevention, and outputs it.

[0022] The management server 8 manages the numerous analysis boxes 1 arranged in each store and the cameras 2 connected to these analysis boxes 1. Specifically, the management server 8 installs application packages in the analysis boxes 1 in each store and controls the starting and stopping of the cameras 2 connected to these analysis boxes 1.

[0023] Next, the hardware configuration of the analysis box 1 will be described with reference to FIG. 2. The analysis box 1 includes a CPU 11 that controls the entire device and performs various calculations, a hard disk 12 that stores various data and programs, a RAM (Random Access Memory) 13, inference chips (hereinafter abbreviated as "chips") 14a to 14h that are DNN inference processors, and a communication control IC 15. The CPU 11 is a general-purpose CPU, or a CPU designed to enhance parallel processing performance for simultaneously processing a large number of video streams. The data stored in the hard disk 12 includes video data (frame image data) 19 obtained by decoding (the data of) the video streams input from each of the cameras 2, and the programs stored in the hard disk 12 include a program for the analysis box OS 20 in addition to the application package 17. The application package 17 also includes a state estimation program 18.

[0024] The (inference) chips 14a to 14h are preferably processors (chips dedicated to inference) optimized for DNN (Deep Neural Networks) inference, but may be general-purpose GPUs (Graphics Processing Units) used for general purposes or other processors. Also, each of the chips 14a to 14h may be a device in which multiple chips (inference processors) are integrated (mounted) on a single board computer.

[0025] 2, the (inference) chips 14a to 14h are connected to the CPU 11 via PCI Express or USB. The communication control IC 15 has a LAN port 16 which is a port for connecting to a LAN of the Ethernet standard.

[0026] 3 shows functional blocks of the CPU 11 in the analysis box 1. The analysis box 1 includes, as functional blocks, a label assignment unit 21, a provisional state estimation unit 22, a final state estimation unit 23, a state transition verification unit 24, a notification unit 25, and a processor allocation unit 26. The label assignment unit 21, the provisional state estimation unit 22, the final state estimation unit 23, the state transition verification unit 24, and the notification unit 25 correspond to the label assignment means, the provisional state estimation means, the final state estimation means, the state transition verification means, and the notification means in the claims, respectively. The functions of the label assignment unit 21, the provisional state estimation unit 22, the final state estimation unit 23, the state transition verification unit 24, and the notification unit 25 are realized by the CPU 11 and the state estimation program 18.

[0027] The labeling unit 21 has a simple deep neural network model for multi-label classification (hereinafter referred to as "MLC model") of a type that assigns a label based on only one still image, and assigns one or more labels to an image of each attention area (an image of each table in the store (hereinafter referred to as "table image")) in each frame image input from the camera 2 using this MLC model. More precisely, the labeling unit 21 assigns likelihood information of each of a plurality of labels to each of the table images using the MLC model. Note that the likelihood information is the likelihood itself in the example shown in FIG. 6, but may be an index of the reliability of the label other than the likelihood. More precisely, the table image is an image of the surroundings of the table including not only the table body 47 but also the chair 48 as shown in FIG. 6. In this embodiment, the "provisional state of each table", the "final state of each table", and the "state of each table" respectively mean, precisely, the "provisional state of the surroundings of each table", the "final state of the surroundings of each table", and the "state of the surroundings of each table".

[0028] The provisional state estimation unit 22 estimates the provisional state of each table in the store based on the labels assigned to each table image by the label assignment unit 21. More precisely, the provisional state estimation unit 22 compares the likelihood of each label assigned to each table image by the label assignment unit 21 with a threshold value for the likelihood of each label, as will be described in detail later.

[0029] The final state estimation unit 23 estimates the final state of each table in a frame image at a predetermined time out of a plurality of past frame images input from camera 2. More precisely, the final state estimation unit 23 estimates the final state of each table in a frame image at a predetermined time out of a plurality of past frame images input from camera 2, based on the provisional state of each table in a frame image after the predetermined time and the final state of each table in a frame image before the predetermined time.

[0030] When the state machine (see FIG. 7) expressing the state transition rules does not allow the final state of each table in the frame image at the predetermined time to become the final state estimated by the final state estimation unit 23, the state transition verification unit 24 maintains the final state of each table in the frame image at the predetermined time as the final state of each table in the frame image immediately preceding this frame image. However, even when the state machine expressing the state transition rules does not allow the final state of each table in the frame image at the predetermined time to become the final state estimated by the final state estimation unit 23, when the provisional state of each table estimated by the provisional state estimation unit 22 remains the same for a predetermined time or more, the state transition verification unit 24 allows the final state of each table to become the final state estimated by the final state estimation unit 23.

[0031] When the final state of each table after verification by state transition verification unit 24 satisfies a predetermined condition, notification unit 25 outputs an alert on a notification screen (see FIG. 15) displayed on signage 3 (display) to notify a user (staff member of store S) of state estimation system 10. The notification screen is an image including notification information for the user (staff member of store S) and an image based on the frame image most recently input from camera 2 (an image in a circular coordinate system captured by camera 2, which is a 360-degree camera, converted into an image in a two-dimensional Cartesian coordinate system (planar development image)).

[0032] The processor allocation unit 26 allocates a chip (inference processor) to be used for inference processing of each neural network model (hereinafter referred to as "NN model"), such as an MLC model, included in each instance of the application package 17, from among the multiple chips 14, based on the inference time required for inference processing of each NN model and its frequency of use.

[0033] FIG. 4 shows an overview of the state estimation process in the state estimation system 10. This state estimation process is performed by the CPU 11 in the analysis box 1 based on the state estimation program 18. When one frame image is input from the camera 2 at a frame rate of 1 fps (S1), the CPU 11 in the analysis box 1 performs a process (pre-processing 1) of cutting out an image of each table (table image) from the frame image based on the state estimation program 18 (S2). Specifically, the images of each table (table images 31a to 31t of No. 1 to 20 in the figure) that are the attention area are cut out from the frame image 30 shown in FIG. 5. Note that in FIG. 5, the frame image in the circular coordinate system captured by the camera 2, which is a 360-degree camera, is converted into an image in the two-dimensional orthogonal coordinate system (planar development image) in order to make it easier to see, and the converted frame image 30 is shown. In the actual image cutting process, the frame image 30 converted into the two-dimensional orthogonal coordinate system as shown in FIG. 5 may be used, or the frame image in the circular coordinate system captured by the camera 2 may be used as it is.

[0034] When the processing in S2 above is completed, the CPU 11 of the analysis box 1 performs processing (pre-processing 2) such as adjusting the size of each of the extracted table images 31a to 31t to a predetermined size (size adjustment) and normalizing the features of each table image (data) after size adjustment (S3).

[0035] When the process of S3 above is completed, the CPU 11 (label assignment unit 21) of the analysis box 1 performs inference processing (S5) using the inference engine 46 (the above-mentioned MLC model) for each table image 41 after the completion of the above-mentioned pre-processing 2, as shown in Fig. 6, and assigns labels with likelihoods (probabilities), which are the results of this inference processing, to each table image 41. In the example shown in Fig. 6, in addition to a table body 47 and a chair 48, a person 42, a glass 43, a tray 44, and a bowl 45 are shown in the table (surrounding) image 41, so the estimation results by the inference engine 46 show that the likelihoods of the labels for person, glass, tray, and bowl are high, and the likelihood of an empty seat is low.

[0036] When the processing of S5 is completed, the CPU 11 of the analysis box 1 performs state estimation processing using a state estimation engine (corresponding to the provisional state estimation unit 22, the final state estimation unit 23, and the state transition verification unit 24 in FIG. 3) described below (S6), and outputs the final state of each table, which is the estimation result. The CPU 11 of the analysis box 1 repeats the processing of S5 and S6 until processing is completed for all table images 41 (images of the attention area) that underwent pre-processing 2 in S3 above (NO in S4). Then, based on the state estimation program 18, the CPU 11 (notification unit 25) of the analysis box 1 outputs an alert on the notification screen 71 (see FIG. 15) displayed on the display of the signage 3 when the final state of each table output in the processing of S6 above satisfies a predetermined condition.

[0037] Next, the state estimation process by the state estimation engine in S6 above will be described in detail. First, the state of each table to be estimated by the state estimation engine will be described. FIG. 7 is a state machine diagram (state machine diagram) showing the rules of transition of the state of each table in the state estimation system 10 of this embodiment. As shown in this diagram, the states of each table in the store S include an empty seat state 51, an order waiting state 52, an order completed state 53, a eating state 54, and a waiting for clearing state 55. The empty seat state 51 is a state in which the table and seat are empty, and the waiting for order state 52 is a state in which a customer takes a seat (at the table) and waits for a staff member to come and take the order. The ordered state 53 is a state in which a staff member takes an order, serves a cup filled with tea or water to the customer, and the customer is waiting for the food he or she ordered. The eating state 54 is a state in which the customer is eating, and the waiting for clearing state 55 is a state in which the customer leaves the seat (leaves the store) and the tray remains on the table.

[0038] As shown in FIG. 7, the state of each table in store S generally transitions in the following order: vacant seat state 51, waiting for order state 52, ordered state 53, eating state 54, waiting for cleanup state 55, before returning to vacant seat state 51.

[0039] The state estimation process by the state estimation engine mainly consists of a provisional state estimation process and a final state estimation. First, the provisional state estimation process will be described. This provisional state estimation process is performed by comparing the result of the inference process by the inference engine 46 (the above-mentioned MLC model) (the likelihood of each label for each table image) with a threshold value of the likelihood of each label.

[0040] More specifically, assume that the table image 41 after the above preprocessing 2 is input to the inference engine 46 (the above MLC model) and the inference results (likelihood of each label) obtained are as shown in Figures 6 and 9. Here, if p(x) is a function that gives the likelihood of label x and h(x) is a function that gives a threshold value of the likelihood of label x, then from the inference results shown in Figures 6 and 9, p(person) = 0.9, p(tray) = 0.99, p(cup) = 0.65, p(bowl) = 0.99, and p(vacant seat) = 0.04. Since the table image 41 shown in Figure 6 shows a person 42, a cup 43, a tray 44, and a bowl 45, the values ​​of the likelihood p of the labels of person, tray, cup, and bowl are high, as described above. In this case, if the likelihood thresholds for each label are h(person) = 0.6, h(tray) = 0.6, h(cup) = 0.63, h(bowl) = 0.6, and h(empty seat) = 0.6, then the labels whose likelihood p is greater than or equal to the likelihood threshold h are person, tray, cup, and bowl.

[0041] In the provisional state estimation process, the CPU 11 (provisional state estimation unit 22) of the analysis box 1 determines that, among the labels of person, tray, cup, bowl, and empty seat, the label whose likelihood p is equal to or greater than the likelihood threshold h is the label assigned to the table image 41 input to the inference engine 46 (MLC model). Therefore, in the case of the example shown in Fig. 6 and Fig. 9 above, the CPU 11 (provisional state estimation unit 22) determines that person, tray, cup, and bowl are the labels assigned to the table image 41.

[0042] Next, an example of a specific flow of the provisional state estimation process will be described with reference to the flowchart shown in Fig. 8. When the likelihood p(vacant seat) of (the label of) an empty seat in the inference result of the inference engine 46 is equal to or greater than the threshold h(vacant seat) (YES in S11), the CPU 11 (provisional state estimation unit 22) estimates that the provisional state is an empty seat state (S12). Also, when the likelihood p(vacant seat) is less than the threshold h(vacant seat) in the judgment process of S11 (NO in S11), the CPU 11 (provisional state estimation unit 22) judges whether the likelihood p(person) of (the label of) a person is equal to or greater than the threshold h(person) (S13). As a result of this determination, when the likelihood p(person) is equal to or greater than the threshold h(person) (YES in S13), the CPU 11 (provisional state estimation unit 22) determines whether or not either of the conditions is satisfied, that the likelihood p(tray) of the tray (label) is equal to or greater than the threshold h(tray), or the likelihood p(bowl) of the bowl (label) is equal to or greater than the threshold h(bowl) (S14). As a result of this determination, when either of the conditions is satisfied, that the likelihood p(tray) of the tray (label) is equal to or greater than the threshold h(tray), or the likelihood p(bowl) of the bowl (label) is equal to or greater than the threshold h(bowl) (YES in S14), the CPU 11 (provisional state estimation unit 22) estimates that the provisional state is a eating state (S15).

[0043] As a result of the judgment in S14 above, if the likelihood p(tray) of the tray (label) is less than the threshold h(tray) and the likelihood p(bowl) of the bowl (label) is less than the threshold h(bowl) (NO in S14), CPU 11 (provisional state estimation unit 22) judges whether the likelihood p(cup) of the cup (label) is equal to or greater than the threshold h(cup) (S16). If the result of this judgment is that the likelihood p(cup) of the cup (label) is equal to or greater than the threshold h(cup) (YES in S16), CPU 11 (provisional state estimation unit 22) estimates that the provisional state is an ordered state (S17). As a result of the judgment in S16 above, when the likelihood p(cup) of the cup (label) is less than the threshold h(cup) (NO in S16), the CPU 11 (its provisional state estimation unit 22) estimates that the provisional state is a waiting for order state (S18).

[0044] Also, when the result of the judgment in S13 above is that the likelihood p(person) is less than the threshold h(person) (NO in S13), the CPU 11 (provisional state estimation unit 22) performs a judgment process similar to S14 above (judgment process of whether or not either of the conditions that the likelihood p(tray) of the tray (label) is equal to or greater than the threshold h(tray) or the likelihood p(bowl) of the bowl (label) is equal to or greater than the threshold h(bowl) is satisfied) (S19). When the result of this judgment is that either of the conditions that the likelihood p(tray) of the tray (label) is equal to or greater than the threshold h(tray) or the likelihood p(bowl) of the bowl (label) is equal to or greater than the threshold h(bowl) is satisfied (YES in S19), the CPU 11 (provisional state estimation unit 22) estimates that the provisional state is a state of waiting for clearing (S20).

[0045] Returning to FIG. 7 above, the relationship between the label given to the table image 41 and the provisional state of the table shown in the relevant table image 41 can be summarized as follows. That is, when the label given to the table image 41 is (only) an unoccupied seat label, the CPU 11 (provisional state estimation unit 22) estimates that the provisional state of the table is an unoccupied seat state 51. Also, when the label given to the table image 41 is only a person label, the CPU 11 (provisional state estimation unit 22) estimates that the provisional state of the table shown in the relevant table image 41 is an order waiting state 52. Furthermore, when the label given to the table image 41 is a person label and a cup label, the CPU 11 (provisional state estimation unit 22) estimates that the provisional state of the table shown in the relevant table image 41 is an ordered state 53. Furthermore, when the labels applied to the table image 41 include at least one of a tray label and a bowl label, and a person label, the CPU 11 (the provisional state estimation unit 22) estimates that the provisional state of the table reflected in the corresponding table image 41 is an eating state 54. When the labels applied to the table image 41 include at least one of a tray label and a bowl label, and do not include a person label, the CPU 11 (the provisional state estimation unit 22) estimates that the provisional state of the table reflected in the corresponding table image 41 is a waiting for clearing state 55.

[0046] Next, the final state estimation process by the above-mentioned state estimation engine will be described with reference to Fig. 10 to Fig. 12. This final state estimation process is performed by (the final state estimation unit 23 of) the CPU 11 of the analysis box 1 after the above-mentioned provisional state estimation process.

[0047] As shown in Fig. 11, the state buffer 60 used in the above-mentioned final state estimation process is a buffer for storing states for (2n+1) frames. In the example of the state buffer 60 shown in Fig. 11, the window size w (the maximum number of states that can be stored in the state buffer 60 (2n+1)) is 11. In this way, when the window size w is 11, if the frame image showing the table whose final state is to be estimated this time is a frame image at time T (the "predetermined time" in the claims), first, the CPU 11 (final state estimation unit 23) of the analysis box 1 stores the final states of the table shown in the frame images from time (T-5) to (T-1) and the tentative states of the table shown in the frame images from time T to (T+5) in the above-mentioned state buffer 60, as shown in Fig. 11.

[0048] That is, as shown in the flowchart of FIG. 10, when the CPU 11 (provisional state estimation unit 22) of the analysis box 1 completes the estimation process of the provisional state of the table reflected in the new frame image (table image 41) (YES in S21), the CPU 11 (final state estimation unit 23) determines whether the length of the current state buffer 60 (the number of states stored in the state buffer 60) is smaller than w (the maximum number of states that can be stored in the state buffer 60 (2n+1)) (S22). Then, if the length of the current state buffer 60 is smaller than w (YES in S22), the provisional state of the table reflected in the new frame image (the provisional state of the table most recently estimated) is added and stored in the state buffer 60 (S23).

[0049] The additional storage process of the provisional state will be described with reference to Fig. 12. In the example shown in Fig. 12, for simplicity, an example will be described in which the window size w (the maximum number of states that can be stored in the state buffer 60 (2n+1)) is 5. As shown in Fig. 12, when t = 1 to 4, the length of the state buffer 60 at that time (the number of states stored in the state buffer 60) is smaller than w (= 5) (YES in S22 of Fig. 10), so the CPU 11 (final state estimation unit 23) adds and stores the provisional state of the table most recently estimated in the state buffer 60 (S23).

[0050] On the other hand, when making the judgment in S22 above, as when t=5 in FIG. 12, the length of the state buffer 60 at that time (the number of states stored in the state buffer 60) is already w (=5) (NO in S22), the state stored first (the state (T-2) at that time) is extracted (removed) from the state buffer 60 using a first-in-first-out method (S24), and the most recently estimated provisional state of the table is stored in the state buffer 60 (as the state (T+2)) (S25).

[0051] Next, the CPU 11 (final state estimation unit 23) judges whether the length of the state buffer 60 (the number of states stored in the state buffer 60) after the processing of S23 or S25 is w (= 5) or not (S26). If the result of this judgment is that the length of the state buffer 60 is w (= 5) (YES in S26), the CPU 11 (final state estimation unit 23) estimates that the most frequent (most frequent) state among the states stored in the state buffer 60 (the final states at times (T-2) and (T-1), and the provisional states from time T to (T+2)) is the final state of (the table shown in the frame image at) time T, and sets this most frequent state as the final state at time T in the state buffer 60 (S27). Specifically, in the example of the state buffer 60 at time (t=5) on the left side of FIG. 12, four of the five stored states are the ordered state, so the "ordered state" is set as the final state 61a at time T in the state buffer 60. Also, in the example of the state buffer 60 at time (t=5) on the right side of FIG. 12, three of the five stored states are the ordered state, so the "ordered state" is set as the final state 61a at time T in the state buffer 60. Note that the function v in FIG. 12 indicates the frequency (number) of each state in the state buffer 60. In detail, v (vacant), v (waiting to order), v (ordered), and v (eating) indicate the number (number) of "vacant state", "waiting to order state", "ordered state", and "eating state" stored in the state buffer 60, respectively.

[0052] When the process of S27 by the final state estimation unit 23 (estimation process of the final state at time T) is completed, the CPU 11 (state transition verification unit 24) judges whether or not it is permissible from the state machine shown in Fig. 7 that the final state of the table (in the frame image) at time T be the final state set in S27 (S28). If the result of this judgment is that it is not permissible (NO in S28), the CPU 11 (state transition verification unit 24) maintains the final state of the table at time T as the final state of the table in the frame image one image before the frame image at time T (the frame image at time (T-1)). In other words, the CPU 11 (state transition verification unit 24) corrects the final state of the table at time T to the final state of the table in the frame image at time (T-1) (S29).

[0053] To be more precise, when viewed from the state machine shown in Figure 7, the CPU 11 (state transition verification unit 24) does not allow the final state of the table (in the frame image) at time T to transition to "ordered state" when the final state of the table (in the frame image) at time (T-1) is "vacant state", so in this case, as a rule, the final state of the table at time T is maintained as the final state of the table at time (T-1) ("vacant state"). However, it is possible that when a customer arrives at the restaurant and sits down at a table, at the same time a staff member places a cup on the table and takes the order (the table state changes (transitions) from "vacant state" to "ordered state"). For this reason, even if it is not permissible from the state machine shown in FIG. 7 that the final state of the table (in the frame image) at time T should become the final state set in S27 above, the CPU 11 (state transition verification unit 24) may allow the final state of the table to become the final state set in S27 above when the provisional state estimated by the provisional state estimation unit 22 remains the same for a predetermined period of time or longer.

[0054] That is, as shown in FIG. 13, when the provisional state of the table estimated by the provisional state estimation unit 22 continues to be the "ordered state" for 10 seconds or more after it has been the "vacant state", the CPU 11 (state transition verification unit 24) allows the final state of the table to transition from the "vacant state" to the "ordered state". Also, as shown in FIG. 13, when the provisional state of the table estimated by the provisional state estimation unit 22 continues to be the "eating state" for 10 seconds or more after it has been the "waiting for order state", the CPU 11 (state transition verification unit 24) allows the final state of the table to transition from the "waiting for order state" to the "eating state". Furthermore, when the provisional state of the table estimated by the provisional state estimation unit 22 continues to be the "vacant state" for 60 seconds or more after it has been the "eating state", the CPU 11 (state transition verification unit 24) allows the final state of the table to transition from the "eating state" to the "vacant state". However, the state transition verification unit 24 of the CPU 11 only allows the above-mentioned transitions from the "vacant state" to the "ordered state", the transition from the "waiting for order state" to the "eating state", and the transition from the "eating state" to the "vacant state" as state transitions that are not permitted from the perspective of the state machine. For example, transitions from the "vacant state" to the "waiting for cleanup state" and from the "waiting for order state" to the "waiting for cleanup state" are not allowed.

[0055] 10, the state transition verification unit 24 of the CPU 11 sends (outputs) the final state of the table at time T after correction to the notification unit 25 (see FIG. 3). Also, if the result of the determination in S28 in FIG. 10 indicates that the final state of the table at time T is allowed to become the final state set in S27 (YES in S28), the state transition verification unit 24 of the CPU 11 sends (outputs) the final state of the table at time T set in S27 to the notification unit 25 (see FIG. 3).

[0056] When the final state of the table at time T sent from the state transition verification unit 24 satisfies the following conditions, the notification unit 25 of the CPU 11 notifies (issues an alert) the staff who is the user of the state estimation system 10. Specifically, as shown in FIG. 14, when the final state of the table at time T changes from vacant state 51 to waiting for order state 52, the notification unit 25 of the CPU 11 assumes that a customer has arrived and sat at the table, issues an alert 70a, and notifies the staff. In addition, when the final state of the table at time T sent from the state transition verification unit 24 is ordered state 53 for 10 minutes or more, the notification unit 25 of the CPU 11 assumes that the customer has been waiting for an order for a long time, issues an alert 70b, and notifies the staff. Furthermore, if the final state of the table at time T sent from the state transition verification unit 24 is in the waiting for clearing state 55 for a predetermined time (e.g., 5 minutes) or more, the notification unit 25 of the CPU 11 determines that the table has not been cleared for a long time (a tray has been left on the table), raises an alert 70c, and notifies the staff.

[0057] The signage 3 shown in Fig. 1 is used to notify the staff. Fig. 15 shows a notification screen 71 displayed on the display of the signage 3. This notification screen 71 is an image including information for notifying the staff and an image based on the frame image most recently input from the camera 2 (an image in a circular coordinate system captured by the camera 2, which is a 360-degree camera, converted into an image in a two-dimensional orthogonal coordinate system (planar development image)). In the following description, the image based on the frame image most recently input from the camera 2 is abbreviated to "frame image most recently input from the camera 2".

[0058] The notification unit 25 of the CPU 11 displays frames 72 to 74 on the notification screen 71 based on the final state of the table at time T sent from the state transition verification unit 24. Of these frames, the frame 72 shown in dashed lines indicates that the table surrounded by this frame 72 is in an "ordered state", the frame 73 shown in dashed lines indicates that the table surrounded by this frame 73 is in an "order waiting state", and the frame 74 shown in solid lines indicates that the table surrounded by this frame 74 is in an "eating state". Furthermore, tables not surrounded by frames on the notification screen 71 are vacant. Although not shown in FIG. 15, a frame indicating that a table is in a "cleaning waiting state" may also be displayed on the notification screen 71.

[0059] Furthermore, the notification unit 25 of the CPU 11 displays the above-mentioned alerts 70a to 70c on the notification screen 71 based on the final state of the table at time T sent from the state transition verification unit 24. That is, when the final state of the table at time T sent from the state transition verification unit 24 changes from vacant state 51 to waiting for order state 52, the notification unit 25 of the CPU 11 performs the above-mentioned alert 70a to the staff by blinking a dashed frame 73 surrounding the corresponding table and outputting a warning sound from the speaker of the signage 3. Furthermore, when the final state of the table at time T sent from the state transition verification unit 24 is the ordered state 53 for 10 minutes or more, the notification unit 25 of the CPU 11 performs the above-mentioned alert 70b to the staff by blinking a dashed-dotted frame 72 surrounding the corresponding table. Furthermore, when the final state of the table at time T sent from the state transition verification unit 24 is the waiting for cleanup state 55 for a predetermined time (e.g., 5 minutes) or more, the notification unit 25 of the CPU 11 issues an alert 70c to the staff member by blinking a frame surrounding the table.

[0060] As described above, the notification unit 25 of the CPU 11 displays not only the above frames 72 to 74 on the notification screen 71, but also the frame image most recently input from the camera 2 (to the analysis box 1). The reason for this is as follows. That is, the notification unit 25 of the CPU 11 displays the above frames 72 to 74 and issues the above alerts 70a to 70c based on the final state (of the table) at time T. However, as mentioned in the explanation of Figs. 11 and 12, in order to estimate the final state at time T, a future (after time T) provisional state (provisional state at times (T+1) to (T+5) in Fig. 11) is required. As described in Fig. 4, the frame rate of the image from the camera 2 is 1 fps, so the provisional state at time (T+5) can be obtained 5 seconds after the provisional state (of the table) at time T is estimated. Therefore, a delay of 5 seconds or more occurs in order to perform the final state estimation process at time T in S27 (see Fig. 10). Moreover, the notification unit 25 receives the final state of the table at time T from the state transition verification unit 24 only after all the processes from S2 to S6 in FIG. 4 are completed after receiving a frame image from the camera 2 in S1.

[0061] Therefore, frames 72-74 of each table and alerts 70a-70c displayed on notification screen 71 are information based on frame images input from camera 2 (to analysis box 1) more than 5 seconds ago (which could be several tens of seconds ago, depending on the performance of CPU 11 and (inference) chip 14). To solve the problem of information delay, notification unit 25 of CPU 11 displays not only frames 72-74 on notification screen 71, but also the frame image most recently input from camera 2 (to analysis box 1), as shown in FIG. In this way, by displaying the most recently input frame image from camera 2 on the notification screen 71, even if there is a delay in updating frames 72-74 for each table displayed on the notification screen 71 and in executing alerts 70a-70c, staff who see the most recent frame image displayed on the notification screen 71 can notice changes in the status of each table and the occurrence of states corresponding to the above-mentioned alerts 70a-70c (a customer arriving, a customer waiting to order for a long time, or a table not being cleared for a long time).

[0062] In addition, in the case where an alert must be issued immediately when the state of the table reaches a predetermined condition, it is possible to reduce or set to zero the number of provisional states (future states) after time T stored in the state buffer 60 shown in FIG. 11 and FIG. 12 to improve the real-time nature of the alert, but the more the number of provisional states after time T stored in the state buffer 60 is reduced, the lower the accuracy of the estimation of the final state at time T becomes. Therefore, it is desirable that the number of provisional states after time T stored in the state buffer 60 is at least 3 or more, and preferably 5 or more. In short, it is desirable that the number of provisional states after time T stored in the state buffer 60 is large from the viewpoint of the accuracy of the estimation of the final state, but it is desirable that the number is small from the viewpoint of the real-time nature of the update of frames 72 to 74 and the alert. In addition, in cases where an alert must be issued immediately when the table state meets a specified condition, if the number of provisional states after time T stored in state buffer 60 is reduced or set to zero, it is desirable from the viewpoint of the accuracy of estimating the final state to increase the number of final states before time T stored in state buffer 60 accordingly.

[0063] As described above, according to the state estimation system 10 and state estimation program 18 of the present embodiment, the final state of the table in the frame image at a predetermined time (time T) is estimated based on the provisional state of the table in the frame images after the predetermined time (time T) (in the example of FIG. 11, the frame images from time T to (T+5)) among the multiple past frame images, and the final state of the table in the frame images before the predetermined time (in the example of FIG. 11, the frame images from time (T-5) to (T-1)). This makes it possible to reduce the influence of noise contained in the label assigned using a simple MLC model of the type that assigns the label based on only one still image on the estimation result of the final state. Therefore, even though the provisional state of the table in each frame image is estimated based on the label including noise assigned using the above-mentioned simple MLC model, the (final) state of the table in each frame image can be accurately estimated based on these provisional states.

[0064] Furthermore, according to the state estimation system 10 and state estimation program 18 of this embodiment, as described above, a simple MLC model that assigns a label based on only one still image (table image in each frame image) is used, so the processing load of the inference process in a processor such as the (inference) chip 14 can be reduced compared to a case where a complex MLC model that assigns a label based on a plurality of (temporally) consecutive still images (time-series frame images) is used. Therefore, the state estimation system 10 can be easily realized using a computer with low processing power arranged on the edge side.

[0065] Moreover, according to the state estimation system 10 of this embodiment, the inference engine 46 (MLC model) compares the likelihood of each label assigned to the table image 41 with a threshold value of the likelihood of each label to estimate the provisional state of the table. This makes it possible to easily estimate the provisional state of the table.

[0066] Furthermore, according to the state estimation system 10 of this embodiment, if the state machine (see FIG. 7) expressing the state transition rules does not permit the final state of the table in the frame image at time T to become the final state estimated by the final state estimation unit 23 (the final state set in the process of S27 in FIG. 10), the final state of the table in the frame image at time T is maintained as the final state of the table in the frame image immediately preceding this frame image (the frame image at time (T-1)). This makes it possible to remove the effects of noise contained in the estimation result of the provisional state based on the labels assigned using the simple MLC model.

[0067] The reason for removing the influence of noise included in the estimation result of the provisional state using a state machine as described above is as follows. That is, as described above, in the state estimation system 10, the final state of the table in the frame image at a predetermined time T is estimated based on the provisional state of the table in the frame image after the predetermined time T and the final state of the table in the frame image before the predetermined time T, thereby making it possible to reduce the influence of noise included in the label given using the simple MLC model on the estimation result of the final state. However, this process alone cannot completely remove the influence of noise included in the label given to each table image on the estimation result of the final state. For this reason, in the state estimation system 10, the state machine is used to prevent transition to a final state that is not permitted according to the state transition rules, thereby further reducing the influence of noise included in the label given using the simple MLC model on the estimation result of the final state.

[0068] Furthermore, according to the state estimation system 10 of this embodiment, even if the state machine expressing the state transition rules does not permit the final state of the table in the frame image at time T to become the final state estimated by the final state estimation unit 23, when the provisional state estimated by the provisional state estimation unit 22 remains the same for a predetermined time or longer, the final state of the table may be permitted to become the final state estimated by the final state estimation unit 23. This is to accommodate exceptional state transition patterns that deviate from the rules indicated by the state machine.

[0069] Furthermore, according to the state estimation system 10 of this embodiment, when the final state of the table (after verification by the state transition verification unit 24) output from the state estimation engine (see FIG. 4) satisfies a predetermined condition, a notification is given to the user (staff) of the state estimation system 10. This allows the staff to be notified based on the accurate final state verified by the state transition verification unit 24, thereby making it possible to improve the efficiency of store operations.

[0070] According to the state estimation system 10 of the present embodiment, the notification unit 25 (of the CPU 11) of the analysis box 1 displays a notification screen 71 for notifying staff on the signage 3 (display), and in addition to information (frames 72-74 in FIG. 15) for notifying staff of the state of each table, the notification screen 71 displays the most recently input frame image from the camera 2. As a result, despite delays in updating the frames 72-74 of each table displayed on the notification screen 71 and in executing the alerts 70a-70c due to the time it takes from estimating the provisional state at time T to completing the estimation of the final state at time T, the staff can notice changes in the state of each table and the occurrence of states corresponding to the above-mentioned alerts 70a-70c (such as a customer's arrival, a state in which a customer is waiting for an order for a long time, or a state in which a table has not been cleared for a long time) by looking at the most recent frame image displayed on the notification screen 71.

[0071] Variations: The present invention is not limited to the configurations of the above-described embodiments, and various modifications are possible without departing from the spirit of the invention.

[0072] Variation 1: In the above embodiment, an example has been described in which the state estimation system 10 is a state estimation system for a restaurant. However, the state estimation system of the present invention is not limited to this, and can be used for various purposes by switching (replacing) the state machine shown in FIG.

[0073] Variation 1-1: For example, by changing the state machine to that shown in Fig. 16, the state estimation system of the present invention can be used as a state estimation system for cash registers in various stores, and counters (for customer service) in banks, post offices, hospitals, rental car stores, etc. In the following description, the cash registers in various stores, and counters (for customer service) in banks, post offices, hospitals, rental car stores, etc., are collectively referred to as "counters." In the examples shown in Figs. 16 and 17, the area of ​​interest includes not only the above-mentioned counter 86 (area) (see Fig. 17), but also a chair 87 in the customer waiting area (area).

[0074] In the state machine shown in Fig. 16, there are three states (of counter 86 and chair 87): vacant state 81, waiting state 82, service providing state 83, and processing state 84. The vacant state 81 is a state in which counter 86 and chair 87 shown in Fig. 17 are vacant, and the waiting state 82 is a state in which a customer is waiting for service from a staff member at counter 86 or chair 87. The service providing state 83 is a state in which a staff member is providing a service to a customer at counter 86, and the processing state 84 is a state in which a staff member is processing a customer request in the so-called back office.

[0075] 16, in this state estimation system, when the label assignment unit assigns a label (assigned label) to an image of the attention area using the MLC model, the provisional state estimation unit estimates that the provisional state of the attention area is an empty state 81, and when the assigned label is only a customer label, the provisional state estimation unit estimates that the provisional state of the attention area is a waiting state 82. When the assigned label is a customer label and a staff label, the provisional state estimation unit estimates that the provisional state of the attention area is a service providing state 83, and when the assigned label is a staff label, the provisional state estimation unit estimates that the provisional state of the attention area is a processing state 84.

[0076] As shown in FIG. 16, the state of the area of ​​interest (the counter 86 and chair 87 shown in FIG. 17) generally transitions through the following states: vacant state 81, waiting state 82, service providing state 83, and processing state 84, and then enters the service providing state 83 before returning to the vacant state 81.

[0077] The state estimation system of this modified example 1-1 notifies the clerk of an alert 80a shown in FIG. 16 based on a frame image showing the counter 86 and the chair 87 when a customer approaches the counter 86 and when the customer sits on a chair 87 in the waiting area. When a customer approaches the counter 86, the above-mentioned alert 80a is notified to all clerks or to a specific clerk with special expertise together with the number of the corresponding counter 86. Here, clerks (staff) at banks, post offices, etc. perform not only service provision work at the counter but also clerical work in the back office. Therefore, by notifying the clerk of the above-mentioned alert 80a, the clerk can concentrate on clerical work in the back office and only check the counter (provide service at the counter) when a customer appears, so that the store operation by the clerk can be optimized.

[0078] The state estimation system of this modified example 1-1 also notifies an alert 80b shown in FIG. 16 based on a frame image showing the counter 86. This alert 80b is issued (1) to notify other store clerks that a service provision operation by a certain store clerk to a customer at the counter 86 has been completed, and (2) to notify a customer that the counter is now available for use. The above-mentioned alerts 80a and 80b are notified by displaying a notification screen on a signage or outputting an alarm sound from a speaker. As can be seen from FIG. 17, the camera used in the state estimation system of this modified example 1-1 is not a 360-degree camera as in the above embodiment, but a normal network camera.

[0079] Variation 1-2: Moreover, by changing the state machine to the state machine shown in Fig. 18, the state estimation system of the present invention can be used as a state estimation system for a bank ATM (automated teller machine). In this example, the area of ​​interest is (the area of) an ATM 94 shown in Fig. 19.

[0080] As shown in Fig. 18, the states of the ATM 94 in the above state machine include an idle state 91, a processing state 92, and a supporting state 93. The idle state 91 is a state in which the ATM 94 shown in Fig. 19 is idle, and the processing state 92 is a state in which a customer is using the services of the ATM 94. Also, the supporting state 93 is a state in which a bank staff member is supporting the customer in operating the ATM because the customer has been in front of the ATM 94 for five minutes or more (the processing state 92 has continued for five minutes or more).

[0081] 18, in this state estimation system, when the label (assigned label) assigned by the label assignment unit using the MLC model to the image of the ATM 94 is an empty label, the provisional state estimation unit estimates that the provisional state of the ATM 94 is an empty state 91, and when the assigned label is only a customer label, the provisional state estimation unit estimates that the provisional state of the ATM 94 is an in-process state 92. When the assigned labels are a customer label and a staff label, the provisional state estimation unit estimates that the provisional state of the ATM 94 is an assisting state 93.

[0082] 18, the state of the ATM 94 basically transitions from an idle state 91 to a processing state 92 and then returns to the idle state 91. However, if five minutes or more have passed since the transition from the idle state 91 to the processing state 92, the bank staff will assist the customer in operating the ATM, so the state transitions from the processing state 92 to a supporting state 93 and then returns to the idle state 91.

[0083] The alert 90a shown in Fig. 18 is an alert for notifying the bank staff of the presence or absence of a customer in front of each ATM 94 (whether each ATM 94 is in an available state 91 or not). The state estimation system (the analysis box in the state estimation system) of the modified example 1-2 issues the above-mentioned alert 90a by displaying a notification screen indicating which ATMs 94 are available and which are in use on a signage installed in an area of ​​the bank counter where the staff are located, based on a frame image showing each ATM 94. When the notification screen indicates that all ATMs 94 are unavailable (in use), the bank staff checks the situation in the area where the ATMs 94 are installed and supports the customer to speed up ATM operations.

[0084] 18 is an alert for notifying bank staff when a customer has been in front of the ATM 94 for 5 minutes or more (when the processing status 92 has continued for 5 minutes or more), as described above. When this alert 90b is notified, the bank staff assists the customer in operating the ATM 94.

[0085] Variation 1-3: Moreover, the state estimation system of the present invention can be made into a state estimation system for a highway bus by changing the state machine to the state machine shown in Fig. 20. In this example, the attention area is each seat of the bus (the area of ​​the seat).

[0086] As shown in FIG. 20, the state of each seat (on a bus) in the above state machine includes an empty seat state 95 and an occupied seat state 96. The empty seat state 95 is a state in which the corresponding seat is empty, and the occupied seat state 96 is a state in which a passenger is sitting in the corresponding seat. In this state estimation system, as shown in FIG. 20, when a label (assigned label) that the label assignment unit assigns to an image of the corresponding seat using the MLC model is an empty seat label, the provisional state estimation unit estimates that the provisional state of the corresponding seat is an empty seat state 95, and when the above-mentioned assigned label is a person label, the provisional state estimation unit estimates that the provisional state of the corresponding seat is an occupied state 96. As shown in FIG. 20, the state of each seat transitions from the empty seat state 95 to the occupied state 96, and then returns to the empty seat state 95.

[0087] The state estimation system (analysis box in the system) of Modification 1-3 displays a notification screen 99 indicating the state of each seat on the bus on a signage 98 installed at the door of the bus, based on a frame image showing each seat on the bus, as shown in Fig. 21. This notification screen 99 is provided with a seat state display field 99a and a number of occupied seats display field 99b. The seat state display field 99a indicates the state of each seat on the bus (which seats are vacant and which seats are occupied by passengers), and the number of occupied seats display field 99b indicates the number of seats on the bus that are estimated to be in an occupied state 96.

[0088] The above-mentioned notification screen 99 is used for (1) passengers selecting seats and (2) the driver checking the number of seated passengers. The above-mentioned (1) passengers selecting seats is performed as follows. That is, when the bus arrives at the bus stop, (the analysis box in) the state estimation system displays the notification screen 99 on the signage 98 installed on the door of the bus. Passengers getting on the bus look at the seat status display field 99a on this notification screen 99 and decide (select) their own seat from among the vacant seats.

[0089] Furthermore, the above-mentioned (2) confirmation of the number of seated passengers by the driver is performed in the following cases: For example, when the bus arrives at a rest area such as a service area and the designated rest time has elapsed, the bus driver checks the number of seats displayed in the seat count display field 99b on the notification screen 99 displayed on the signage 98 to confirm the number of occupied seats (i.e., the number of seated passengers).

[0090] Variation 2: In the above embodiment, an example was shown in which the state estimation system 10 includes the camera 2, the signage 3, etc. in addition to the analysis box 1 ("computer" in the claims), but among these components, the only component essential to the state estimation system of the present invention is the analysis box. In other words, the camera for photographing the area of ​​interest (e.g., a table) may be an existing (external) surveillance camera, and the signage for displaying the notification screen may be an existing (external) display device.

[0091] Variation 3: In the above embodiment, the CPU 11 of the analysis box 1 is configured to include the label assignment unit 21, the provisional state estimation unit 22, the final state estimation unit 23, the state transition verification unit 24, and the notification unit 25, but the state estimation system of the present invention is not limited to this configuration. For example, the cameras installed in each store may be AI (Artificial Intelligence) cameras with so-called edge computing functions, and an application package with an MLC model may be installed in the AI ​​cameras so that the AI ​​cameras have the functions of the above label assignment unit.

[0092] Variation 4: In the above embodiment, an example has been given of a case in which a simple MLC model that assigns labels based on only one still image is used to assign one or more labels to an image of each area of ​​interest (table image) in each frame image input from camera 2. However, the "label assignment means" in the present invention is not limited to one that uses the above-mentioned simple MLC model, and may be one that uses a multi-label classification algorithm that is capable of assigning multiple labels to an image of each area of ​​interest. [Explanation of symbols]

[0093] 1 Analysis Box (Computer) 2 Camera 3. Signage (display device) 10 State Estimation System 18 State Estimation Program 21 Label assignment unit (label assignment means) 22 Provisional state estimation unit (provisional state estimation means) 23 Final state estimation unit (final state estimation means) 24 State transition verification unit (state transition verification means) 25 Notification section (notification means) 31a-31t, 41 Table images (images of the area of ​​interest) 71 Notification screen 72-74 Frame ("Information for notifying the user" in claims) T Time (predetermined time)

Claims

1. a labeling means for assigning one or more labels to an image of a region of interest in each frame image input from a camera; a provisional state estimation means for estimating a provisional state of the region of interest based on the label assigned to the image of the region of interest by the label assignment means; a final state estimation means for estimating a final state of the region of interest in a frame image at a predetermined time among a plurality of past frame images input from the camera, The final state estimation means is a state estimation system that estimates the final state of the area of ​​interest in a frame image at the specified time based on a tentative state of the area of ​​interest in a frame image at the specified time among a plurality of consecutive past frame images, a tentative state of the area of ​​interest in a frame image after the specified time, and the final state of the area of ​​interest in a frame image before the specified time.

2. The state estimation system described in Claim 1, characterized in that the final state estimation means estimates the most frequent state among the interim state of the area of ​​interest in the frame image at the specified time, the interim state of the area of ​​interest in one or more frame images after the specified time, and the final state of the area of ​​interest in one or more frame images before the specified time to be the final state of the area of ​​interest in the frame image at the specified time.

3. The label assignment means assigns a likelihood of each label to the image of the region of interest, 2. The state estimation system according to claim 1, wherein the provisional state estimation means estimates the provisional state of the region of interest by comparing the likelihood of each of the labels with a threshold value of the likelihood of each of the labels.

4. The state estimation system according to any one of claims 1 to 3, further comprising a state transition verification means for maintaining the final state of the area of ​​interest in the frame image at the specified time to be the final state estimated by the final state estimation means when a state machine representing a rule of state transition does not permit the final state of the area of ​​interest in the frame image at the specified time to be the final state estimated by the final state estimation means.

5. The state estimation system according to claim 4, characterized in that the state transition verification means allows the final state of the region of interest to transition to the final state estimated by the final state estimation means when the provisional state of the region of interest estimated by the provisional state estimation means remains the same for a predetermined time or more, even if it is not permissible from the perspective of a state machine representing a rule of state transition that the final state of the region of interest should become the final state estimated by the final state estimation means.

6. a notification means for notifying a user of the state estimation system, 4. The state estimation system according to claim 1, wherein the notification means notifies a user of the state estimation system when the final state of the region of interest estimated by the final state estimation means satisfies a predetermined condition.

7. a notification means for notifying a user of the state estimation system, 6. The state estimation system according to claim 4, wherein the notification means notifies a user of the state estimation system when a final state of the attention area after verification by the state transition verification means satisfies a predetermined condition.

8. The notification means outputs a notification screen for notifying the user to a display device, 8. The state estimation system according to claim 6, wherein the notification screen includes information to be notified to the user and an image based on a frame image most recently input from the camera.

9. Computer, A labeling means for assigning one or more labels to an image of a region of interest in each frame image input from a camera; a provisional state estimation means for estimating a provisional state of the region of interest based on the label assigned to the image of the region of interest by the label assignment means; a state estimation program for causing the program to function as a final state estimation means for estimating a final state of the attention area in a frame image at a predetermined time among a plurality of past frame images input from the camera, the program comprising: The final state estimation means is a state estimation program that estimates the final state of the area of ​​interest in a frame image at the specified time based on a provisional state of the area of ​​interest in a frame image at the specified time among a plurality of consecutive past frame images, a provisional state of the area of ​​interest in a frame image after the specified time, and a final state of the area of ​​interest in a frame image before the specified time.

10. The state estimation program described in Claim 9, characterized in that the final state estimation means estimates the most frequent state among the interim state of the area of ​​interest in the frame image at the specified time, the interim state of the area of ​​interest in one or more frame images after the specified time, and the final state of the area of ​​interest in one or more frame images before the specified time to be the final state of the area of ​​interest in the frame image at the specified time.

11. The computer further comprises: The state estimation program according to claim 9 or 10, characterized in that if it is not permissible from a state machine representing the rules of state transition that the final state of the area of ​​interest in the frame image at the specified time be the final state estimated by the final state estimation means, the state estimation program functions as a state transition verification means that maintains the final state of the area of ​​interest in the frame image at the specified time as the final state of the area of ​​interest in the frame image one frame before this frame image.

Images