Queue detection system, method and program

The matrix detection system accurately tracks queue participants and estimates queue information by using user-input positions and movement models, addressing misidentification and occlusions in existing technologies.

JP7593964B2Active Publication Date: 2024-12-03NEC CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022064304
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2015-09-14
Filing Date
2022-04-08
Publication Date
2024-12-03
Estimated Expiration
2036-09-12

AI Technical Summary

Technical Problem

Existing queue detection technologies mistakenly treat individuals who are merely staying in a queue as those intending to wait, and fail to accurately account for occlusions, leading to incorrect estimates of the number of people and waiting times.

Method used

A matrix detection system that uses user-input matrix positions and different movement models to track objects based on their alignment in a queue, employing tracking units to distinguish between stationary and moving individuals, and generating complete queue information by estimating occluded areas.

Benefits of technology

Accurately distinguishes between individuals intending to wait and those merely present, and estimates queue information including the number of people and waiting times, even in environments with occlusions, using existing surveillance cameras without specialized installations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593964000001
    Figure 0007593964000001
  • Figure 0007593964000002
    Figure 0007593964000002
  • Figure 0007593964000003
    Figure 0007593964000003
Patent Text Reader

Abstract

To provide a queue detection system method and a recording medium that prevent a person who is simply staying in a queue without intending to queue from being mistakenly treated as a person waiting in a queue. [Solution] The queue detection system includes a queue information input unit 2 that accepts queue position input by a user, a determination unit 33 that determines whether an object is in a queue based on the distance between the object detected from the input image and the queue position, and a tracking unit 32 that tracks the position of the object using a movement model that differs depending on whether the object is in a queue or not.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a system, method and recording medium for detecting matrices from an image. [Background technology]

[0002] Various methods have been proposed for detecting queues from camera images and for analyzing the number of people in a queue, the length of the queue, the waiting time, etc.

[0003] The technology described in Patent Document 1 estimates the number of people by calculating the area of ​​the person region for the input image using background subtraction and dividing the area by a predetermined unit area per person. The technology described in Patent Document 1 estimates the waiting time by dividing the estimated number of people by the number of visitors per unit time tallied by a separately specified method.

[0004] The technology described in Patent Document 2 captures the queue from directly above with a camera installed on the ceiling, and extracts the areas of individual people who are staying or moving using background subtraction and optical flow. The technology described in Patent Document 2 generates queue lines by connecting the extracted individual person areas in order, and calculates the waiting time based on the total extension and length of each queue. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2002-329196 A [Patent Document 2] JP 2007-317052 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the techniques described in Patent Documents 1 and 2 above, even if a person is simply hanging around without any intention of joining a queue, that person may be treated as someone waiting in line.

[0007] An object of the present invention is to provide a technique capable of solving the above-mentioned problems. [Means for solving the problem]

[0008] A queue detection system according to one aspect of the present disclosure includes a queue information input means for accepting queue position input by a user, a determination means for determining whether or not an object is in a queue based on the distance between the object detected from an input image and the queue position, and a tracking means for tracking the position of the object using a movement model that differs depending on whether the object is in a queue.

[0009] A matrix detection method according to one aspect of the present disclosure accepts input of a matrix position from a user, determines whether or not an object is in a matrix based on the distance between the object detected from an input image and the matrix position, and tracks the position of the object using a different movement model depending on whether the object is in a matrix.

[0010] A program according to one aspect of the present disclosure causes a computer to perform the following processes: accepting input of a matrix position from a user; determining whether or not an object is in a matrix based on the distance between the object detected from an input image and the matrix position; and tracking the position of the object using a different movement model depending on whether or not the object is in a matrix. Effect of the Invention

[0011] According to the present invention, the problem that a person who is simply hanging around without intending to join a queue may be mistakenly treated as a person waiting in line is solved. [Brief description of the drawings]

[0012] [Figure 1] 1 is a functional block diagram showing a configuration example of a first embodiment of the present invention. [Diagram 2]FIG. 11 is an explanatory diagram showing an example of a method for inputting matrix information. [Diagram 3] FIG. 11 is an explanatory diagram showing an example of a method for inputting matrix information. [Figure 4] FIG. 11 is an explanatory diagram showing an example of a process for generating queue information. [Diagram 5] FIG. 11 is an explanatory diagram showing an example of a process for generating queue information. [Figure 6] 1 is a flowchart showing an example of a processing progress of the first embodiment of the present invention. [Figure 7] 1 is a flowchart showing an example of a processing progress of the first embodiment of the present invention. [Figure 8] FIG. 11 is a functional block diagram showing a configuration example of a second embodiment of the present invention. [Figure 9] 10 is a flowchart showing an example of a process flow according to the second embodiment of the present invention. [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer capable of implementing each embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] First, in order to facilitate understanding of embodiments of the present invention, a background of the present invention will be described.

[0014] In the techniques described in the above-mentioned Patent Documents 1 and 2, even if a person is simply hanging around without any intention of joining a queue, that person may be treated as a person waiting in line.

[0015] For example, the technology described in Patent Document 1 calculates the area of ​​a person region using background subtraction for an input image, so if a person who is not intending to join the queue stops near the queue, the area of ​​the person region will be calculated including the person who is simply standing there. In addition, if there are multiple people standing at a position away from the original queue, the technology described in Patent Document 1 can recognize the multiple people as people who make up the queue. Other issues with the technology described in Patent Document 1 include non-detection due to the occurrence of occlusion. In a method based on background subtraction such as Patent Document 1, differences in people's physiques, the presence or absence of accessories such as luggage, and people overlapping with each other or being hidden by an obstruction easily cause errors in the relationship between the actual person and the person region obtained on a difference basis. In addition, this method does not recognize individual people or their order in the queue, so it is not possible to calculate the waiting time for each person.

[0016] In addition, in the technology described in Patent Document 2, as in Patent Document 1, erroneous processing may occur in which a mere stagnant person is assumed to be a person waiting in line. In addition, as another problem, the method described in Patent Document 2 is premised on good person detection and tracking, but in a real environment, it is expected that people will overlap each other or occlusion will occur due to obstructions due to the installation position of the camera and the structure of the building. This causes people to be missed in the occlusion area, resulting in failure to correctly connect people waiting in line, and erroneous estimation of the number of people and waiting time. In Patent Document 2, in order to solve such occlusion problems, it is proposed to install a camera at a high position and shoot people from directly above. However, in general indoor facilities or outdoors, the environment in which a camera can be installed at a high position is not necessarily in place. In addition, since the installation position and angle of view are different from those of general surveillance cameras, it is necessary to install a new camera dedicated to queue detection, which leads to increased implementation costs. Therefore, it is desirable to solve the occlusion problem by using existing surveillance camera images that shoot people from diagonally above, rather than using a camera with special installation conditions.

[0017] According to the embodiment of the present invention described below, the above-mentioned problems are solved, erroneous detection of people who do not intend to stand in line is prevented, and the line can be detected robustly against occlusion without the need for a special camera such as a ceiling-mounted camera.

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0019] <Embodiment 1> FIG. 1 is a block diagram showing an example of the configuration of the first embodiment of the present invention.

[0020] As shown in FIG. 1, the queue detection system according to this embodiment includes an image input unit 1, a queue information input unit 2, a queue detection unit 3, and an output unit 4.

[0021] The image input unit 1 sequentially inputs pairs of time-series images of a predetermined surveillance area and the shooting times to the queue detection unit 3. The images input by the image input unit 1 to the queue detection unit 3 may be acquired using a shooting device such as a surveillance camera. Alternatively, the image input unit 1 may input to the queue detection unit 3 time-series images obtained by reading video data stored in a storage device (not shown).

[0022] In the present embodiment, the type of object to be detected is not particularly limited, and may be a person, an animal, a car, a robot, or an object. In the following, detection of a person will be described as an example.

[0023] The queue information input unit 2 inputs the occurrence position of the queue to be analyzed (hereinafter referred to as the "queue position"). The queue information input unit 2 accepts the input of the occurrence position of the queue by the user via, for example, a mouse or a touch panel of a tablet terminal. The queue information input unit 2 may input the occurrence position of the queue on a screen displaying a surveillance camera image based on the user's mouse operation or touch panel operation with a touch pen or a finger. Specifically, the queue information input unit 2 may input the route and direction of the queue by drawing the start point, intermediate points, and end points of the feet of people when the queue is generated on the screen. At this time, the user needs to specify at least one start point and one end point, but the intermediate points may be zero. In addition, in order to express a queue including a branch or a merger, the user may input two or more for at least either the start point or the end point. In addition, the queue information input unit 2 may accept the specification of the number of people to line up horizontally for each queue line connecting the start point, intermediate point, and end point from the user.

[0024] In addition, the input start point, intermediate point, end point, and the number of people lining up side by side may be superimposed and drawn on a screen displaying the surveillance camera image, so that the user can easily check the input contents. The queue information may be input in advance before starting the queue detection process, or the changed queue information may be reflected at any timing during the execution of the queue detection process.

[0025] The method of inputting the queue information is not limited to the above. For example, instead of inputting the start point, intermediate points, and end points of the feet of the people when the queue is generated, the user may input the queue positions by directly tracing straight lines or curves on a touch panel. Alternatively, the queue information input unit 2 may automatically input the queue positions by, for example, reading a file in which the queue positions are described in a predetermined format stored inside or outside the device, instead of based on the user's input.

[0026] Here, input of queue information will be specifically described. Fig. 2 is a diagram showing an image of people standing in a queue captured by a surveillance camera. Fig. 2 shows people standing in a queue from the back to the front. Fig. 2 also shows that the queue splits into two at the end, but merges along the way, forming a two-column queue that continues to the front. Fig. 3 shows that two end points (end point 1 and end point 2) at the end of the queue, two relay points (relay point 1 and relay point 2), and one start point (start point 1) at the head of the queue are input as queue information. Fig. 3 also shows that end points 1 and 2 to relay point 1 are input as a one-column queue, and relay point 1 to start point 1 is input as a two-column queue.

[0027] The queue information input unit 2 reflects the information on the start point, intermediate point, end point, and between each point input as described above as a queue position at any timing. For example, a "reflect" button for reflecting the input information may be displayed on the display screen, and when pressing of the button by the user is detected, the queue information input unit 2 may reflect the input information. The queue information reflected in this way is used for the judgment of the judgment unit 33, etc.

[0028] The matrix detection unit 3 analyzes images sequentially input from the image input unit 1, and detects matrices in the images. The matrix detection unit 3 includes a detection unit 31, a tracking unit 32, a determination unit 33, and a generation unit .

[0029] The detection unit 31 detects all or some of the people present in the image acquired from the image input unit 1. The detection unit 31 may perform processing to detect people from the input image, for example, by using a classifier that has been constructed in advance to recognize patterns of specific body regions such as the head and upper body of a person. The detection unit 31 is not limited to a method that uses a classifier, and may use a known method in pattern recognition as a method for detecting people.

[0030] Furthermore, in order to detect only people present in the vicinity of the queue position, the detection unit 31 may perform the detection process by limiting the detection target area based on the queue position input from the queue information input unit 2. That is, the detection unit 31 may perform the detection process only on the area near the input queue position.

[0031] The detection unit 31 outputs, as a detection result, the time of the image acquired from the image input unit 1 and the position coordinates of the detected person. When the calibration information of the camera that captured the input image is known by being calculated in advance, the detection unit 31 may convert the position coordinates on the screen, which are the detection result of the person, into world coordinates using the calibration information and output the converted coordinates.

[0032] The tracking unit 32 tracks people across multiple frames, with the people being detected from each frame image by the detection unit 31. The tracking unit 32 may perform person tracking processing using a known algorithm, such as a particle filter.

[0033] The tracking unit 32 may use different movement models for updating the position of the person to be tracked depending on whether the target is a person waiting in line. The tracking unit 32 may refer to, for example, a queue person ID (IDentifier) ​​list managed by the determination unit 33 to determine whether the target is a person waiting in line. In this case, the tracking unit 32 may determine that the target is a person waiting in line if the person ID of the target is registered in the queue person ID list, and may determine that the target is a person not waiting in line if the person ID is not registered.

[0034] A person waiting in a queue is stationary for most of the time. Therefore, it is appropriate to apply a movement model that represents a stationary state (i.e., hardly moves) for most of the time. In addition, the moving direction for each location is known from the queue position information obtained from the queue information input unit 2. Therefore, a movement model with a limited moving direction can be applied. In other words, the tracking unit 32 can track a person waiting in a queue more robustly by performing a tracking process using a two-state model that represents a moving state and a stationary state. The tracking unit 32 may change the weight of each state model depending on whether the person waiting in a queue is stationary or moving.

[0035] On the other hand, there is no restriction on the direction of movement of people who are not in a queue, so the tracking unit 32 may apply a movement model to people who are not in a queue that assumes that they move in all directions at a predetermined range of movement speeds.

[0036] When the input position information of a person is in planar coordinates on an image, and the calibration information of the camera that captured the input image is known because it has been calculated in advance, the tracking unit 32 may perform tracking processing in the world coordinate system. That is, the tracking unit 32 may use the calibration information to convert the coordinate system on the screen into the world coordinate system before performing tracking processing. When the position information of a person is input in the world coordinate system, the tracking unit 32 may perform tracking processing in the world coordinate system as it is.

[0037] The tracking unit 32 outputs the tracking result. For example, the tracking unit 32 outputs, as the tracking result, information in which the person ID, time, and position coordinates of each person to be tracked are linked together.

[0038] The determination unit 33 calculates the moving speed of each person based on the position coordinates of the person for each time output from the tracking unit 32. The determination unit 33 compares the position coordinates of the person with the coordinates of the matrix position input from the matrix information input unit 2, and calculates the distance between each person and the matrix position.

[0039] The determination unit 33 determines that a person whose moving speed and distance from the queue position are smaller than a predetermined threshold is a person who is waiting in a queue (hereinafter referred to as a "queue person"). The determination unit 33 determines that a person whose moving speed or distance from the queue position is equal to or greater than a predetermined threshold is a person who is not waiting in a queue (hereinafter referred to as a "moving person").

[0040] When the detection unit 31 detects the head as a specific part, the determination unit 33 may estimate a bottom end position corresponding to the person's feet from the position of the head using the camera parameters, and determine whether or not the person is in a queue based on the distance between the estimated bottom end position and the queue position. In this way, even when it is difficult to directly detect the feet, the determination unit 33 can accurately determine whether or not the person is a queue person by detecting the head, which is relatively easy to detect, and estimating the feet from the head. The determination unit 33 may take into consideration the possibility that the person is an adult or a child. Specifically, for example, the determination unit 33 assumes a plurality of variations in the height of the head in the world coordinate system, and estimates a plurality of bottom end positions from the assumed plurality of heights using the camera parameters. The determination unit 33 may determine whether or not the object is in a queue based on the shortest distance between the estimated plurality of bottom end positions and the queue position.

[0041] The determination unit 33 holds a queueing person ID list indicating whether each person ID being tracked is a queueing person or a moving person. The queueing person ID list is a list that manages a pair of the person ID of the person being tracked and information indicating whether the person is a queueing person or a moving person.

[0042] The determination unit 33 updates the queue person ID list using a determination result based on the moving speed and the distance between the queue position and each person ID being tracked.

[0043] The determination unit 33 outputs information on the person determined to be a queueing person. For example, the determination unit 33 outputs information linking the person ID, time, and position coordinates of the person determined to be a queueing person.

[0044] Generation unit 34 generates entire queue information without any missing information, based on the information of the people in queue input from determination unit 33 and the information of the queue position input from queue information input unit 2. That is, generation unit 34 estimates an undetected area, and generates queue information by complementing information of people lining up in the undetected area from the information of the people in queue.

[0045] Specifically, the generating unit 34 estimates the occurrence of occlusion (a region where a person is not detected due to the influence of occlusion) based on the position coordinates of the person in the queue and the queue position. The generating unit 34 estimates the presence of a person who has not been detected due to the influence of occlusion from the estimated occurrence of occlusion and the detection status of the person in the queue before and after the undetected region. The generating unit 34 also estimates a region where a person cannot be detected because the visibility is too small (a region where a person is not detected due to the influence of visibility). The generating unit 34 estimates the presence of a person who has not been detected from the possibility of undetection due to the influence of visibility and the detection status of the person in the queue. The generating unit 34 generates the entire queue information without any missing information by complementing the information of the person who has not been detected but is estimated to exist. The generating unit 34 may generate information such as the number of people waiting in the queue, the length of the queue, the cumulative waiting time of each person waiting in the queue, the progress speed of the queue, and the estimated waiting time from starting to wait in the queue until leaving the queue, as the entire queue information without any missing information.

[0046] A method for estimating the occurrence of occlusion will be described with reference to FIGS.

[0047] Fig. 4 is a diagram showing an image of a queue with a turnaround. In Fig. 4, the upper left corner of the image is the start point (head) of the queue, which turns around twice along the way, and the lower right corner is the end point (tail) of the queue.

[0048] In video shot with a field of view looking diagonally downward, the further away a location is from the camera, the shallower the depression angle. Therefore, even in a procession of people lined up at equal intervals, people further away from the camera are more likely to be hidden by people in front. Also, the further away a person is from the camera, the smaller the person appears. Therefore, a person far from the camera may not be detected if the number of pixels required to detect the person falls below the minimum number. As a concrete example, Figure 4 shows the shaded area where a person went undetected due to occlusion or because the person was too small to be detected.

[0049] In such a situation, when calculating the number of people waiting in the entire queue, for example, the generation unit 34 performs the calculation in the following manner. The generation unit 34 calculates the interrupted queue length based on the position coordinates of the person determined to be a queue person by the determination unit 33 and the queue position input from the queue information input unit 2. The generation unit 34 also estimates the population density (the number of people lining up within a predetermined distance) from the area where the person positions are obtained. The generation unit 34 estimates the number of people present in the interrupted queue length based on the estimated population density, and calculates the number of people waiting by adding this to the number of people that were detected. The generation unit 34 calculates the number of people waiting in the above-mentioned manner, for example.

[0050] Furthermore, when calculating the cumulative waiting time for each person, the generation unit 34 performs the calculation using, for example, the following method. The generation unit 34 estimates the number of people present in the occlusion area using the above-mentioned method. The generation unit 34 calculates the cumulative waiting time from when the people detected immediately before and immediately after the occlusion area started to stand in line. The generation unit 34 divides the time difference between the two calculated cumulative waiting times by the estimated number of people present in the occlusion area. The generation unit 34 performs an interpolation process for the cumulative waiting time by adding the time difference calculated by the division to the cumulative waiting time of the people who are in front of the person in the occlusion area, starting from the first person in the occlusion area.

[0051] The output unit 4 outputs the matrix information input from the matrix detection unit 3. The output mode of the output unit 4 is, for example, display output. In this case, the output unit 4 may display on an external display device. However, the output mode of the output unit 4 is not limited to display and may be other modes.

[0052] Next, the operation of the matrix detection system according to this embodiment will be described with reference to the flowcharts of Fig. 6 and Fig. 7. Fig. 6 shows an example of a process flow when a user inputs a matrix position. Fig. 7 shows an example of a process flow when detecting a matrix after the matrix position is input, and generating the entire matrix information without any missing parts.

[0053] The processing steps described below may be executed in any order or in parallel as long as no inconsistency occurs in the processing content, and other steps may be added between the processing steps. Furthermore, a step described as one step for convenience may be executed as multiple steps, and a step described as multiple steps for convenience may be executed as one step.

[0054] First, the process when the user inputs row and column positions will be described with reference to the flowchart in FIG.

[0055] The queue information input unit 2 acquires an image of a predetermined monitoring area, and displays it on a screen for setting queue positions (step SA1).

[0056] Next, the user checks whether there are any unset matrices on the displayed screen. If the setting of the matrices is not complete and there are any unset matrices, the matrix information input unit 2 accepts the start of input of the matrix positions by the user via a predetermined input device (no in step SA2). Examples of the input device include a mouse and a touch panel, and in the case of a touch panel, the user can input using a touch pen or a finger.

[0057] The matrix information input unit 2 accepts input of a starting point of a matrix by the user (step SA3). If the matrix that the user is trying to set has multiple starting points, the matrix information input unit 2 accepts input of all starting points in step SA3.

[0058] Next, the matrix information input unit 2 accepts input of relay points representing branching points and intermediate points of the matrix by the user (step SA4). If the matrix can be expressed by a straight line connecting the start point and the end point, input of relay points is not necessary, and the process of step SA4 is omitted. Also, if there are multiple relay points in the matrix that the user is trying to set, the matrix information input unit 2 accepts input of all relay points in step SA4.

[0059] Next, the matrix information input unit 2 accepts input of the end points of the matrix (step SA5). If the matrix that the user is trying to set has multiple end points, the matrix information input unit 2 accepts input of all end points in step SA5.

[0060] The start point, intermediate point and end point input in steps SA3, SA4 and SA5 may be displayed on the image acquired in step SA1 each time they are input, allowing the user to confirm each of the input points.

[0061] Next, the user checks whether there are any line segments for which the number of columns (number of people lining up horizontally) has not been set for each line segment connecting the input start point, intermediate point, and end point. If the setting of the number of columns has not been completed and there are any line segments for which the number of columns has not been set, the queue information input unit 2 accepts the start of input of the number of columns by the user via the above-mentioned input device for the unset line segments (no in step SA6). The queue information input unit 2 accepts the input of the number of columns by the user (step SA7).

[0062] In addition, the number of lines and columns connecting the points entered in step SA7 may be displayed on the image acquired in step SA1 each time they are entered, allowing the user to check the number of people on each line entered.

[0063] When the user completes input of the numbers of columns for all line segments (yes in step SA6), the matrix information input unit 2 returns to step SA2 and starts accepting input for the next matrix. The matrix information input unit 2 may determine that the user has completed input of the numbers of columns for all line segments, for example, by accepting a user input to that effect, or by a predetermined time passing without the user starting input.

[0064] When matrix information input unit 2 determines that the user has completed input of all matrices (yes in step SA2), input of matrix information is completed in matrix information input unit 2. Matrix information input unit 2 may determine that the user has completed input of all matrices, for example, by accepting a user input to that effect, or by a predetermined time passing without the user starting input.

[0065] Next, the process of detecting a queue and generating queuing information will be described with reference to the flowchart of FIG.

[0066] The detection unit 31 acquires a photographed image and its photographing time from the input unit 1 (step SB1). The detection unit 31 detects all people present in the image, and outputs the time and the position coordinates of the detected people (step SB2).

[0067] Next, upon receiving the tracking result from the tracking unit 32, the determination unit 33 refers to the queueing person ID list for the tracked person obtained as the processing result for the previous frame image (step SB3) and determines whether the person is a queueing person (step SB4). Note that the queueing person ID list is a list that manages a pair of the person ID of the person being tracked and information indicating whether the person is a queueing person or a moving person.

[0068] If the person is a person in a queue (yes in step SB4), the determination unit 33 predicts the position coordinates in the current frame image using a movement model for people in a queue. The determination unit 33 calculates the position coordinates of the person to be tracked in the current frame image using the predicted position coordinates and the person position coordinates obtained in step SB2 (step SB5a). The determination unit 33 may use, as the movement model for people in a queue, a two-state model of a movement model representing a state in which the queue is stationary and a movement model representing a state in which the queue is moving. Specifically, in the movement model representing a stationary state, the determination unit 33 makes a prediction that there is almost no change in position between the previous frame image and the current frame image. On the other hand, in the movement model representing a moving state, the determination unit 33 makes a prediction in which the movement direction is limited based on the information on the queue position obtained from the queue information input unit 2 between the previous frame image and the current frame image.

[0069] On the other hand, if the person is a moving person (no in step SB4), the determination unit 33 predicts the position coordinates in the current frame image using a movement model for moving people. The determination unit 33 calculates the position coordinates of the tracked person in the current frame image using the predicted position coordinates and the person position coordinates obtained in step SB2 (step SB5b). As the movement model for a moving person, for example, the determination unit 33 may use a model that assumes that the person moves in all directions at a predetermined range of movement speeds since there is no restriction on the movement direction.

[0070] Next, the decision unit 33 calculates the moving speed of the person using the person position coordinates and each piece of time information for the past few frames calculated in steps SB5a and SB5b (step SB6).

[0071] The determination unit 33 calculates an appropriate distance, such as the shortest distance, between the person position coordinates in the current frame and the matrix position obtained from the matrix information input unit 2 (step SB7).

[0072] Using the moving speed calculated in step SB6 and the distance calculated in step SB7, the judgment unit 33 judges whether the moving speed is smaller than a predetermined threshold and the distance is smaller than a predetermined threshold (step SB8). If the above conditions are met (yes in step SB8), the person is judged to be a queueing person (step SB9a). On the other hand, if the above conditions are not met (no in step SB8), the person is judged to be a moving person (step SB9b).

[0073] The determination unit 33 updates the queuing person ID list managed by the determination unit 33 using the results of the determination in steps SB9a and SB9b as to whether or not the person is queuing (step SB10).

[0074] Next, the generation unit 34 estimates the occurrence state of occlusion based on the person position of the person determined to be a queue person in step SB9a and the queue position acquired from the queue information input unit 2. The generation unit 34 performs an interpolation process for queue information present in the occlusion area (step SB11). Specifically, when calculating the number of people waiting as queue information, the generation unit 34 calculates the queue length of the undetected area interrupted by occlusion using the position information of the person determined to be a queue person in step SB9a and the queue position information acquired from the queue information input unit 2. The generation unit 34 also calculates the population density of the queue people (the number of people lining up within a predetermined distance). The generation unit 34 estimates the number of people present in the queue length of the interrupted undetected area based on the calculated population density, and calculates the number of people waiting by adding it to the detected queue people.

[0075] The output unit 4 outputs the matrix information output from the generation unit 34 to an external module such as an application or a storage medium (step SB12).

[0076] Furthermore, the input process of the queue / matrix information shown in Fig. 6 may be performed in advance before starting the queue / matrix detection process shown in Fig. 7. Alternatively, at any timing during execution of the queue / matrix detection process shown in Fig. 7, the process shown in Fig. 6 may be performed to update the queue / matrix information, and the queue / matrix detection process shown in Fig. 7 may be resumed using the updated queue / matrix information.

[0077] The program according to the first embodiment may be any program that causes a computer to execute the above series of processes.

[0078] As described above, the queue detection system of embodiment 1 solves the problem that even if a person is simply hanging around without any intention of queuing, that person may be processed as someone waiting in line.

[0079] Even in an environment where occlusion occurs, the number of people can be interpolated by estimating the length of the queue in the hidden area. Even if the queue shape is complex, the moving direction for each location is known, so the robustness of tracking is improved by utilizing the moving direction in a movement model when tracking people in the queue. Therefore, the queue detection system according to the first embodiment can detect people waiting in a queue and appropriately estimate queue information such as the number of people and waiting time.

[0080] In addition, even if a person who has no intention of joining the queue stops near the queue, it is possible to determine that the person is not a person who will join the queue by appropriately setting a threshold value for the distance between the person and the queue position. Furthermore, by considering not only the distance between the person and the queue position but also the person's movement speed when determining whether or not a person will join the queue, it is possible to accurately determine that the person who passes through the queue near the queue position is not a person who will join the queue.

[0081] <Embodiment 2> Next, with reference to FIG. 8, a functional configuration of a queue detection system according to a second embodiment of the present invention will be described.

[0082] FIG. 8 is a block diagram showing an example of the configuration of the second embodiment of the present invention.

[0083] As shown in FIG. 8, the queue detection system according to this embodiment includes a queue information input unit 51, a detection unit 52, and a determination unit 53.

[0084] The queue / line information input unit 51 accepts input of queue / line positions.

[0085] The detection unit 52 detects an object from an input image.

[0086] The determination unit 53 determines whether or not the object is in a queue, based on the distance between the object and the queue position.

[0087] FIG. 9 is a flow chart showing the operation of the queue detection system according to the second embodiment.

[0088] The queue / matrix information input unit 51 receives input of queue / matrix positions (step SC1).

[0089] The detection unit 52 detects an object from the input image (step SC2).

[0090] The determination unit 53 determines whether or not the object is in a queue, based on the distance between the object and the queue position (step SC3).

[0091] As described above, the queue detection system of embodiment 2 solves the problem that even if a person is simply hanging around and has no intention of queuing, that person may be processed as someone waiting in line.

[0092] <Hardware configuration example> Each component of the queue detection system in each of the above-described embodiments comprises a control unit, a memory, a program loaded into the memory, a storage unit such as a hard disk for storing the program, an interface for network connection, etc., and is realized by any combination of hardware and software.

[0093] FIG. 10 is a block diagram illustrating an example of a hardware configuration of the queue detection system according to the first or second embodiment.

[0094] As shown in Fig. 10, the queue detection system is realized by a computer device including a CPU 11 (Central Processing Unit), a communication IF 12 (communication interface 12) for network connection, a memory 13, and a storage device 14 such as a hard disk for storing a program. However, the configuration of the queue detection system is not limited to the configuration shown in Fig. 10.

[0095] The CPU 11 operates an operating system to control the entire queue detection system. The CPU 11 also reads out programs and data from a recording medium attached to, for example, a drive device into the memory 13. The CPU 11 reads out programs and data into the memory 13, thereby realizing the respective functional units of the image input unit 1, the queue information input unit 2, the queue detection unit 3 (the detection unit 31, the tracking unit 32, the determination unit 33, and the generation unit 34), and the output unit 4.

[0096] The queue detection system may be connected to an image input device, such as another computer, via the communication IF 12, for example.

[0097] The recording device 14 is, for example, an optical disk, a flexible disk, a magneto-optical disk, an external hard disk, a semiconductor memory, etc., and records the computer program in a computer-readable manner. The storage device 14 may be, for example, a data storage unit that stores integrated data. The computer program may also be downloaded from an external computer (not shown) connected to a communication network.

[0098] The queue detection system may be connected to an input device 15 or an output device 16. The input device 15 is, for example, a keyboard, a mouse, etc. The output device 16 may be, for example, a display, etc., and the queue information output by the output unit 4 may be displayed on the display.

[0099] 1 and 8, the components of each embodiment are described as being realized by one physically combined device, but the means for realizing the components is not particularly limited. In other words, the system of each embodiment may be realized by two or more physically separate devices connected by wire or wirelessly.

[0100] Although the present invention has been described above with reference to the embodiments and examples, the present invention is not limited to the above embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0101] <Other expressions of the embodiment> A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0102] (Appendix 1) A matrix information input means for receiving an input of a matrix position; A detection means for detecting an object from an input image; a determining means for determining whether the object is in a queue based on a distance between the object and the queue position; A procession detection system comprising:

[0103] (Appendix 2) a generating means for estimating an area where the object is not detected based on the matrix position and the position of the object determined by the determining means to be in the matrix, complementing the object in the undetected area from the object determined to be in the matrix, and generating matrix information; 2. The procession detection system of claim 1, further comprising:

[0104] (Appendix 3) The generating means complements the objects in the undetected region based on the density of the objects determined by the determining means to be arranged in the matrix. 3. The queue detection system of claim 2.

[0105] (Appendix 4) The generating means includes: Calculating a waiting time for the object determined by the determination means to be in a queue; Calculating a waiting time for an object in the undetected area based on waiting times for objects detected in areas before and after the undetected area 4. The queue detection system of claim 3.

[0106] (Appendix 5) The determining means determines whether the object is in a queue based on a moving speed of the object and a distance between the object and the queue position. 5. The queue detection system according to any one of claims 1 to 4.

[0107] (Appendix 6) a tracking means for tracking the location of the detected object using different movement models depending on whether the object is queued or not; Further comprising: The determining means calculates the moving speed from a tracking result by the tracking means. 6. The queue detection system of claim 5.

[0108] (Appendix 7) The tracking means comprises: For an object determined by the determination means to be waiting in a queue, a movement state from a previous frame to a current frame is predicted based on two movement models representing a waiting state and a movement state of the queue and the queue position and the moving direction acquired from the queue information input means, and For objects that are not determined to be waiting in a queue by the determination means, a movement state from the previous frame to the current frame is predicted based on a movement model that assumes that the objects move in all directions at a speed within a predetermined range. 7. The queue detection system of claim 6.

[0109] (Appendix 8) The matrix information input means receives input of at least one or more starting points of a matrix, at least one or more ending points of a matrix, and zero or more relay points for indicating intermediate points or branch points of the matrix, and receives input of the number of columns in which objects are arranged horizontally with respect to line segments connecting each of the adjacent starting points, relay points, and ending points. A queue detection system according to any one of appendixes 1 to 7.

[0110] (Appendix 9) The matrix information input means includes: The start point, relay point, end point, and the number of columns between each point are drawn on the input image and used as input information, and are reflected as the row and column positions at any timing. 9. The queue detection system of claim 8.

[0111] (Appendix 10) The detection means detects a specific portion of the object, The determination means estimates a bottom end position of the object from the position of the specific part using camera parameters, and determines whether the object is arranged in a queue based on a distance between the estimated bottom end position and the queue position. 10. A queue detection system according to any one of claims 1 to 9.

[0112] (Appendix 11) The determination means assumes a plurality of variations in height of the specific part in the world coordinate system, estimates a plurality of bottom end positions from the plurality of assumed heights using the camera parameters, and determines whether or not the objects are lined up in a queue based on the shortest distance between the plurality of estimated bottom end positions and the queue position. 11. The queue detection system of claim 10.

[0113] (Appendix 12) Accepts input of row and column positions, Detects objects from the input image, determining whether the object is in a queue based on a distance between the object and the queue position; Matrix detection methods.

[0114] (Appendix 13) A process of inputting queue information for receiving an input of a queue position; A detection process for detecting an object from an input image; a determination process for determining whether the object is arranged in a queue based on a distance between the object and the queue position; A program that causes a computer to execute the following.

[0115] This application claims priority based on Japanese Patent Application No. 2015-180260, filed September 14, 2015, the disclosure of which is incorporated herein in its entirety. [Industrial Applicability]

[0116] According to the present invention, a highly accurate queue monitoring system can be provided, and the system can be suitably applied to a system that detects queues occurring within a monitoring area and indicates the number of people waiting and the waiting time.

[0117] In addition, by utilizing previously inputted queue location information, the present invention can estimate the number of people waiting and the waiting time without any loss of data, even in environments where occlusion is likely to occur due to overlapping shelves, pillars, or people, without the need for special camera installation conditions, making it easy to introduce a queue detection system. [Explanation of symbols]

[0118] 1 Image input section 2, 51 Queue information input section 3. Matrix detection section 4 Output section 31, 52 Detector 32 Tracking Department 33, 53 Judgment section 34 Generation part 11 CPU 12 Communication Interface 13. Memory 14 Storage device 15 Input Devices 16 Output Devices

Claims

1. A setting means for setting an area for detecting people standing in a queue on the image based on an operation of a user inputting a queue position; A calculation means for calculating a moving speed of a person appearing in the video; A determination means for determining a person in the area whose calculated moving speed and distance from the matrix position are smaller than a predetermined threshold value; A queue detection system comprising: a detection means for detecting people waiting in a queue from the image based on the determination.

2. The queue detection system according to claim 1 , wherein the detection means detects that the determined person is a person waiting in a queue.

3. 3. The queue detection system according to claim 1, wherein the calculation means calculates a moving speed of the person by using position coordinates of the person in a plurality of frames of the video.

4. Based on the user's input of the queue position, a region for detecting people waiting in the queue is set on the image; Calculate the moving speed of the person in the video, determining a person in the area whose calculated moving speed and distance from the matrix position are less than a predetermined threshold; A queue detection method, comprising: detecting people waiting in a queue from the video based on the determination.

5. The queue detection method according to claim 4 , further comprising detecting that the determined person is a person waiting in a queue.

6. The method according to claim 4 , further comprising the step of calculating a moving speed of the person using position coordinates of the person in a plurality of frames of the video.

7. A setting process for setting an area for detecting people standing in a queue on the video based on an operation of a user inputting a queue position; A calculation process for calculating the moving speed of a person in the video; a determination process for determining a person in the area whose calculated moving speed and distance from the matrix position are smaller than a predetermined threshold; A program that causes a computer to execute a detection process for detecting people lining up in a queue from the image based on the determination.

8. The program according to claim 7 , wherein the detection process detects that the determined person is a person waiting in a queue.

9. The program according to claim 7 , wherein the calculation process calculates a moving speed of the person using position coordinates of the person in a plurality of frames of the video.

Citation Information

Patent Citations

  • Waiting time detection system

    JP2002329196A

  • Elevator controller by use of image monitoring device

    JP2003276963A

  • System for measuring waiting time for lines

    JP2007317052A

  • Systems and methods for determining if objects are in a queue

    US20050117778A1