Information processing system, information processing system control method, and program

Abstract

To estimate a queue from an image and present useful information to users of a system that organizes the queue and to users waiting in line.SOLUTION: An information processing system includes: detection means for acquiring face position information and posture information of multiple persons detected from an image; first estimation means for estimating a direction in which the multiple persons face based on the posture information; calculation means for calculating distances between the multiple persons based on the face position information; second estimation means for estimating a queue area formed by the multiple persons by determining whether a person is present within a predetermined distance along the direction in which the multiple persons face based on the face position information, the posture information, and the distances between the multiple persons; and display control means for causing display means to display an estimation result estimated by the second estimation means.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an information processing system, a control method for an information processing system, and a program.

Background Art

[0002] In recent years, various video analysis techniques have been known, such as a technique for detecting a person or the like from an image (moving image) of a network camera (imaging device) and measuring the number of people, and a technique for estimating a queue.

[0003] Here, in Patent Document 1, using a plurality of images (moving images) in a video, the difference between image frames is obtained to determine a queue area, and the number of people in the queue is calculated from the area of the queue area. Further, in Patent Document 2, a plurality of areas where a queue can be formed are specified in advance, and it is determined whether a queue is formed based on the congestion degree of the corresponding area and the congestion degree of the adjacent area.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when estimating a queue using the technique of Patent Document 1, a plurality of consecutive images are required, so it cannot be applied to a situation where a moving image cannot be acquired. Further, when using the technique of Patent Document 2, it is necessary to specify in advance an area where a queue can be formed, so when the columns are folded back or the columns are meandered according to the congestion situation, the queue cannot be correctly estimated.

[0006] Therefore, an object of the present invention is to provide an information processing system capable of presenting useful information to a user of a system that estimates a matrix from an image and arranges columns, or a user standing in a column.

Means for Solving the Problems

[0007] In order to achieve the above object, an information processing system according to an aspect of the present invention includes: a detection unit that acquires face position information and posture information of a plurality of persons detected from an image; a first estimation unit that estimates a direction in which the plurality of persons are facing based on the posture information; a calculation unit that calculates a distance between the plurality of persons based on the face position information; a second estimation unit that estimates a matrix area formed by the plurality of persons by determining whether there are persons at distances within a predetermined range along the direction in which the plurality of persons are facing based on the face position information, the posture information, and the distance between the plurality of persons; and a display control unit that causes a display unit to display the estimation result estimated by the second estimation unit.

Effects of the Invention

[0008] According to the present invention, it is possible to present useful information to a user of a system that estimates a matrix from an image and arranges columns, or a user standing in a column.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0010] Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings. Note that the following embodiments do not limit the invention related to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by reference numerals, and duplicate explanations are omitted.

[0011] <Embodiment 1> Hereinafter, the video processing system in the present embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the overall configuration of the video processing system 101 in Embodiment 1.

[0012] The video processing system (information processing system) 101 is constructed for outdoor areas such as theaters, stadiums, shopping malls, and parks, as well as meeting rooms and waiting rooms. The video processing system 101 is configured to include a client device 102, a video analysis server 103, a recording server 104, and a network camera 105.

[0013] The client device 102 receives an input from a user, gives instructions to the video analysis server 103, the recording server 104, and the network camera 105, and displays the results. The client device is, for example, an information processing device such as a PC (Personal Computer).

[0014] The video analysis server 103 receives an instruction from the client device 102 and performs video analysis processing (image analysis processing) based on the instruction. The video analysis processing refers to performing analysis processing on videos or images acquired from the recording server 104 or the network camera 105, and discriminating objects such as people shown in the video or image, or analyzing the actions of the objects.

[0015] The recording server 104 receives an instruction from the client device 102 or the video analysis server 103 and performs video recording processing based on the instruction. The video recording processing refers to giving a video acquisition instruction to a plurality of network cameras 105 connected to the same network as the recording server 104, and storing videos and images. Further, the video recording processing also includes the processing of storing meta information obtained from the network camera 105. The meta information includes, for example, time information, position information, camera setting information, information of sensors attached to the network camera, and the like.

[0016] The network camera (imaging device) 105 is installed on the ceiling, wall, utility pole, etc. of a building, and captures and acquires the video around the network camera 105. The network camera 105 receives instructions from the client device 102, the video analysis server 103, and the recording server 104, and performs video acquisition, rotation control of the camera (pan-tilt-zoom control), audio recording, etc. The network camera 105 may be connected with various sensors such as a human sensor, a temperature sensor, an audio sensor, and an infrared sensor for acquiring the above-mentioned meta information. Incidentally, it is preferable that a plurality of network cameras 105 are connected to the network as shown in FIG. 1. When a plurality of network cameras 105 are connected to the network, the video processing system 101 may include the plurality of network cameras 105.

[0017] In this embodiment, the client device 102, the video analysis server 103, the recording server 104, and the network camera 105 are each connected to the network and can communicate with each other. Note that the functions of these client device 102, video analysis server 103, recording server 104, and network camera 105 may be integrated into an information processing device (image processing device) such as a single PC or server, and may be configured as a single unit.

[0018] FIG. 2 is a block diagram showing a hardware configuration example of an information processing device for realizing the client device 102, the video analysis server 103, and the recording server 104. Note that the information processing device is realized by a PC, but may also be realized by an embedded system, a tablet terminal, or the like.

[0019] In FIG. 2, the CPU 201 is a central processing unit, which cooperates with other components based on a computer program and controls the operation of the entire information processing device. The ROM 202 is a read-only memory, which stores basic programs and data used for basic processing. The RAM 203 is a writable memory, which functions as a work area of the CPU 201. The external storage device 204 realizes access to a recording medium, and can load a computer program and data stored in a medium (recording medium) such as a USB memory into this system.

[0020] The storage 205 is a device that functions as a large-capacity memory such as an SSD (Solid State Drive). Various computer programs and data are stored in the storage 205. The operation unit 206 is a device that receives instructions and command inputs from a user, and a keyboard, a pointing device, a touch panel, etc. correspond to this. The display 207 is a display device (display means) that displays commands input from the operation unit 206 and response outputs of the information processing device thereto. The interface (I / F) 208 is a device that relays data exchange with an external device. The system bus 209 is a data bus that controls the flow of data within the information processing device.

[0021] Furthermore, it can also be configured as a replacement for the hardware device by software that realizes functions equivalent to those of the above devices.

[0022] In the video processing system 101 of the present embodiment, software that implements the processing corresponding to each step of the flowchart (FIG. 4) described later is read from the ROM 202, RAM 203, storage 205, etc., and executed using the CPU 201.

[0023] FIG. 3 is a block diagram showing the program configuration 301 of the entire video processing system 101. The video acquisition unit 302 instructs the network camera 105 in the recording server 104 to acquire video and images. The video and images acquired by the video acquisition unit 302 may be stored in the recording server 104, or may be sent to the video analysis server 103 that performs the processing described later without being stored. Also, the video acquisition unit 302 may acquire an image specified by the user.

[0024] The detection unit 303 receives the video and images from the recording server 104 in the video analysis server 103, performs video analysis (image analysis), detects a person, and obtains the detection result. In the detection unit, a regressor is learned in advance using a known machine learning method such as deep learning, and the person position is estimated by applying this regressor. Although machine learning has been described here, if the person position can be estimated, a method such as pattern matching may be used to detect the person. As a result of the detection, information such as the number of people included in the image, the position of the person in the image, the head, face, and body posture of the person is obtained.

[0025] The orientation estimation unit 304 performs video analysis by receiving videos and images from the recording server 104 in the video analysis server 103, and estimates the orientation of a person. The orientation estimation unit receives the result of the detection unit 303 and estimates the direction in which a plurality of persons (each person) are facing. The orientation can be estimated by any method as long as the direction in which the person is facing can be determined. For example, the orientation of the face may be calculated using face information and used as the facing direction, or the orientation of the shoulders may be used as the facing direction using the body posture information.

[0026] The distance calculation unit 305 calculates the distance between persons by receiving the result of the detection unit 303 in the video analysis server 103. The distance calculation unit calculates the distance between a plurality of persons (between each person) from the positions in the image included in the result of the detection unit 303. At this time, the distance in the image can be converted into the actual distance by using information such as the installation height of the camera and the angle of view.

[0027] The area estimation unit 306 performs estimation of the matrix area in the video analysis server 103 by using the results of the detection unit 303, the orientation estimation unit 304, and the distance calculation unit 305. As a result of the estimation of the matrix area, the area estimation unit 306 obtains information such as the matrix area in the image and the number of persons included in the matrix.

[0028] The display control unit (display control means) 307 receives the results of the detection unit 303 and the area estimation unit 306 in the client device 102, and causes a display device such as the display 207 to display the number of persons detected (the detected number of persons) in the image and the estimated matrix area. Note that the display control unit 307 may not only display one result, but also save the analysis result, generate and display the content to be displayed from a plurality of results.

[0029] Hereinafter, the processing flow in the video processing system 101 will be described using the flowchart of FIG. 4. FIG. 4(A) is a flowchart showing the processing of the video processing system 101 in the present embodiment.

[0030] Here, the process of acquiring an image, detecting a person from the image, estimating a matrix area, and displaying the estimation result will be described. This process is periodically executed by the video processing system 101 at intervals of one minute, etc., and the results of each process are saved. Also, refer to FIGS. 5 and 6 as appropriate for explanation. FIG. 5 is a diagram showing an overview of the process of the video processing system 101 in the present embodiment. FIG. 6 is a diagram of an example of a screen for displaying the processing result of the video processing system 101 in Embodiment 1.

[0031] First, in S401, the video acquisition unit 302 acquires an image (or video) that is the target for estimating the matrix area. The image (or video) acquired by the video acquisition unit 302 may be an image (or video) stored in the recording server 104, or may be an image (or video) specified by the user.

[0032] Next, in S402, the detection unit 303 performs analysis (image analysis) on the image acquired in S401 to detect a person and obtain the detection result. As the detection result (person detection result), information such as the number of people in the image, the face area of the person, the position of the person (including the position of the face), the head, face, and body posture of the person is obtained. Also, the detection unit extracts feature amounts of the face from the face area of the person. Here, the information on the position, head, and face of the person means that the area where each part exists is indicated by a rectangle in terms of the X coordinate, Y coordinate, width, and height. The body posture information refers to the coordinate data of feature points such as the joints of the human body. Using these feature points, for example, the posture (the direction being faced) can be obtained from the coordinate positions of the shoulders and waist. In this process, the detection unit 303 also functions as a detection means for acquiring the face position information and posture information of a plurality of people detected from the image acquired in S401 for each detected person.

[0033] Next, in S403, the orientation estimation unit 304 estimates the direction in which each of the multiple persons in the image is facing, using the detection result detected in S402. In this process, the orientation estimation unit 304 uses the body posture information and estimates the direction in which each of the multiple persons in the image is facing, using the detection result detected in S402. Note that regarding the facing direction, it may be roughly obtained in four directions of east, west, south, and north, or it may be obtained in detail according to the estimation accuracy of the orientation. In this process, the orientation estimation unit 304 functions as a first estimation means for estimating the direction in which the multiple persons are facing, based on the posture information acquired in S402.

[0034] Next, in S404, the distance calculation unit 305 calculates the distance between each of the multiple persons in the image, using the detection result detected in S402. The distance between persons is used for determining whether they form a line, so it suffices to be able to determine whether they are in the vicinity of each other. Therefore, the distance between each pair of persons in the image may be calculated, or it may be determined whether there is a person within the vicinity of a certain person, for example, within a predetermined range of 1 m. In the present embodiment, distance calculation is performed based on whether there is another person in the vicinity of the person in the image. For the distance calculation in this process, the calculation is performed using the position information of the person (including the position of the face). In this process, the distance calculation unit 305 also functions as a calculation means for calculating the distance between the multiple persons, based on the position information of the person (including the position of the face) acquired by the detection unit 303. Hereinafter, examples of the processes of S402 to S404 will be described with reference to FIGS. 5(A) and 5(B).

[0035] FIG. 5(A) is a diagram schematically showing a waiting space (for example, a hospital reception, etc.). 501 to 506 represent the persons present in the waiting space. Specifically, the person 501 represents the reception staff. Also, the persons 502, 503, and 504 represent the persons standing in a line. Also, the persons 505 and 506 represent the persons not standing in a line. The starting point 507 indicates the place where the line starts. The place where the line starts means the place where the lining up begins, and in this case, for example, it refers to the reception window.

[0036] FIG. 5(B) is a diagram schematically showing the result of performing the processes of S402 to S405 with respect to FIG. 5(A). The arrows (512, 514, 516, 518) in the figure indicate the directions in which the persons are facing. Also, the circles (513, 515, 517, 519) in the figure indicate the vicinity ranges of the persons and function as determination regions serving as indices when determining whether the persons are in the vicinity of each other. It is preferable that the center of the circle be the center position of the person or the arrow. Note that the determination region is circular in shape, but does not have to be a perfect circle and may be at least partially circular. Alternatively, it may be polygonal.

[0037] As will be described later, if part or all of the ranges of the circles for the persons overlap (when they overlap), the overlapping persons are determined by the region estimation unit 306 to be in the vicinity of each other. In other words, when part or all of the ranges of the circles for the persons overlap, the overlapping persons are determined to be at a distance within a predetermined range.

[0038] Arrow 512 indicates the position and the direction in which person 501 is facing. Circle 513 is a circle indicating the vicinity range of person 501. Arrow 514 indicates the position and the direction in which person 502 is facing. Circle 515 is a circle indicating the vicinity range of person 502. Arrow 516 indicates the position and the direction in which person 503 is facing. Circle 515 is a circle indicating the vicinity range of person 503. Arrow 518 indicates the position and the direction in which person 504 is facing. Circle 519 is a circle indicating the vicinity range of person 504.

[0039] In Fig. 5(B), since circle 513 and circle 515 partially overlap, it can be seen that the persons (person 501 and person 502) indicated by arrow 512 and arrow 514 are in the vicinity. Further, since circle 515 and circle 517 partially overlap, it can be seen that the persons (person 502 and person 503) indicated by arrow 514 and arrow 516 are in the vicinity. Further, since circle 517 and circle 519 partially overlapped, it can be seen that the persons (person 503 and person 504) indicated by arrow 516 and arrow 518 are in the vicinity. Note that 511 shown in Fig. 5(B) corresponds to the starting point 507 shown in Fig. 5(A), and similarly shows the starting point of the column.

[0040] Next, in S405, the region estimation unit 306 estimates the matrix region using the results of each of S402 to S404. As the estimation result of the matrix region, information such as the matrix region in the image and the number of people included in the matrix is obtained. Note that the details of the process of S405 will be described in S411 to S416 of Fig. 4(B) described later.

[0041] Hereinafter, the details of the process of S405 will be described with reference to Fig. 4(B). Fig. 4(B) is a flowchart showing the details of the process of S405. Note that hereinafter, the details of the process of S405 will also be described using the example of Fig. 5(B).

[0042] First, in S411, the region estimation unit 306 sets, as a focus point, something that is considered to be the starting point when a column is estimated to be formed. In the example of Fig. 5(B), the starting point 511 is set as the focus point. Note that the region estimation unit 306 functions as a second estimation means when performing the processes of S405 and S411 to S416 which are the details of S405.

[0043] Next, in S412, the area estimation unit 306 determines whether there is a person near the point of interest set in S411 (whether there is a person within a predetermined range from the point of interest) using the calculation result (distance between multiple persons) calculated in the process of S404. As a result of the determination, if there is a person near the point of interest (if there is a person within a predetermined range from the point of interest), the process proceeds to S413. On the other hand, if there is no person near the point of interest (if there is no person within a predetermined range from the point of interest), the process proceeds to S415. Specifically, as the determination process in S412, the area estimation unit 306 determines whether at least a part of the point of interest falls within the range of the circle of any person.

[0044] In addition, when the process in S412 is the first time (the first time), since the starting point 511 of the column is the point of interest, the area estimation unit 306 determines whether there is a person near the starting point 511. And when there is a person near the starting point 511, the area estimation unit 306 estimates that person as the person at the head of the column. In addition, when the point of interest falls within the range of the circles of multiple persons, it is desirable to estimate the person at the head by adjusting the range of the circle described later.

[0045] Here, in the example of FIG. 5(B), since the starting point 511 which is the point of interest is included in the range of the circle 515 of the person 502 indicated by the arrow 514, the area estimation unit 306 determines that the person 502 is near the point of interest.

[0046] Next, in S413, the area estimation unit 306 determines whether the person determined to be near the point of interest (at a distance within a predetermined range from the point of interest) in S412 is facing the direction of the point of interest using the estimation result (the face directions of multiple persons) obtained in the process of S403. As a result of the determination, if the person determined to be near the point of interest in S412 is facing the direction of the point of interest, the process proceeds to S414. On the other hand, if the person determined to be near the point of interest in S412 is not facing the direction of the point of interest, the process proceeds to S415. In addition, when the process in S413 is the first time (the first time), the area estimation unit 306 determines whether the person determined to be near the point of interest is facing the direction of the starting point 511.

[0047] Here, in the example of FIG. 5(B), since the person 502 indicated by the arrow 514 is facing the direction of the starting point 511 which is the point of interest, the region estimation unit 306 determines that the person 502 is facing the direction of the point of interest.

[0048] Next, in S414, the region estimation unit 306 changes the person who was determined to be near the point of interest (at a distance within a predetermined range from the point of interest) in S412 to the next point of interest. That is, the point of interest is updated. In the example of FIG. 5(B), the person 502 indicated by the arrow 514 is set as the next point of interest by the region estimation unit 306.

[0049] Here, after the process of S414 is completed, the process returns to S412 and the same process is repeated. That is, it is determined whether there is another person near the person set as the next point of interest in the process of S414. If there is another person near the person set as the next point of interest in the process of S414, it is determined whether the person is facing the direction of the point of interest. At this time, since the point of interest has been changed from the starting point to a person, when a part or all of the circle of another person overlaps with the circle of the person set as the point of interest (within the determination region), the overlapping person is determined to be a person near the person set as the point of interest. And if the person determined to be near the point of interest is facing the direction of the point of interest, the person is set as the next point of interest (the point of interest is updated). This series of processes is repeated until there are no more people near the point of interest.

[0050] Here, in the example of FIG. 5(B), after the processes of S412 to S415 for the person 502 indicated by the arrow 514 are completed, next, the person 503 indicated by the arrow 516 becomes the point of interest, and then the person 504 indicated by the arrow 518 becomes the point of interest. Thus, the setting of the point of interest can be performed along the direction of the person. For example, it is preferable to exclude persons not related to the queue such as the receptionist 501 from the determination target before the start of the process.

[0051] Next, in S415, the area estimation unit 306 determines the area that has become the focus in the processes of S411 to S415 and sets it as the matrix area (matrix area estimation process). That is, based on the face position information and pose information of multiple persons and the distances between the multiple persons, the area estimation unit 306 determines whether there are persons at distances within a predetermined range along the direction in which the multiple persons are facing, thereby estimating the matrix area constituted by the multiple persons.

[0052] Note that the area estimation method may be any method as long as the area range including each focus point can be selected. For example, the area estimation unit 306 may generate (calculate) a rectangle (enclosing rectangle) including all the focus points (multiple focus points) or the determination areas (circular areas) of the multiple persons that have become the focus points as the matrix area.

[0053] Here, in the example of FIG. 5(B), the area 520 indicated by the dotted line including a part of the starting point 511, the circles 515, 517, and 519 becomes the matrix area. The above is the detail of the matrix area estimation process in S405.

[0054] Also, when it is determined that there is no person near the focus point of the starting point 511 initially set in the process of S412, or when it is determined that there is a person near the focus point but the person is not facing the direction of the focus point, it is determined that the person has been detected but the matrix has not been formed. In this case, the area estimation unit 306 does not perform the matrix area estimation. As described above, in the present embodiment, by determining whether there are persons at distances within a predetermined range along the direction of the persons and also determining the direction of the persons, the area of the matrix constituted by multiple persons can be estimated from one image.

[0055] Here, although the matrix area estimation process has been described with reference to S411 to S416, in the present embodiment, even when one of the persons lined up in a column is lined up facing a different direction, the matrix area can be estimated. This will be described below with reference to FIG. 5(C).

[0056] Figure 5(C) is a diagram showing a situation where three people (the person of arrow 522, the person of arrow 524, and the person of arrow 526) are lined up starting from the column starting point 521. The circles 523, 525, and 527 shown in Figure 5(C) respectively indicate the determination regions of the person of arrow 522, the person of arrow 524, and the person of arrow 526. In the example shown in Figure 5(C), the range (size of the circle) of the determination region is set to be wider than that in the example of Figure 5(B). Also, in Figure 5(C), the starting point of the column to be the first focus point is the starting point 521.

[0057] When the processes of S411 to S416 are executed for the example shown in Figure 5(C), since the person of arrow 524 is facing a direction different from the focus point, the result is that the person is not included in the column, but the person of arrow 522 and the person of arrow 526 are included in the matrix.

[0058] That is, since a part of the circle 527 of the person of arrow 526 overlaps with the circle 523 of the person of arrow 522, it can be determined that the person of arrow 526 is near the person of arrow 522. Furthermore, since the person of arrow 526 is facing the person of arrow 522 who is the focus point, the person of arrow 526 is set as the next focus point. Therefore, assuming that the people of arrows 522, 524, and 526 are lined up in a column, the determination region (circular region) including the focus points between these people can be set as the matrix region (the matrix region can be estimated). Also, in this case too, for example, a rectangle (enclosing rectangle) including all the focus points (multiple focus points) or the determination regions (circular regions) of the multiple people who have become focus points may be calculated.

[0059] In this way, the region estimation unit 306 can estimate the matrix region by skipping one person between columns by changing the size (for example, the diameter of the circle) of the determination region and executing the estimation process of the matrix region. Therefore, it is possible to improve the processing accuracy of estimating the matrix region by executing the processes of S411 to S416 multiple times while changing the range (size) of the determination region.

[0060] Next, in S406, based on the processing results of S401 to S405, the display control unit 307 generates a display screen (or display screen information) from the analysis results such as the number of people detected from the image and the estimation results of the matrix area. Then, the generated screen is displayed on a display device such as the display 207. After that, the process ends.

[0061] Incidentally, as display examples including the information of the analysis result and the estimation result, there are the number of people in the image, the matrix area, the length of the matrix and the waiting time, and the comparison with the past matrix. Incidentally, the display including the information of the analysis result and the estimation result is not limited to this, and any display that can present the analysis result and the estimation result obtained by the processing of the video processing system 101 of the present embodiment to the user may be used.

[0062] Hereinafter, with reference to FIG. 6, an example of a display screen of the estimation result of the matrix area displayed on the screen will be shown. FIG. 6 is a diagram of an example of a screen for displaying the processing result of the video processing system 101 in the first embodiment. Each display example in FIG. 6 shows a state in which additional information is added to the whole or a part of the image to be analyzed for display.

[0063] FIG. 6(A) is an example of a screen (result screen) to be displayed when the columns are not correctly formed. The area surrounded by the line indicated by 601 shows the matrix area. The matrix area shown in FIG. 6(A) is in an L shape where it should be arranged in a straight line, indicating that the columns are in an undesirable shape. This can be realized by the display control unit 307 determining the shape and size of the matrix area, which is the processing result obtained by the processing of S401 to S405, and comparing the determined shape and size of the matrix area with the desired shape and size. Then, the display control unit 307 displays on the screen of a display device such as the display 207 a screen (display screen as shown in FIG. 6(A)) generated based on the result obtained by the comparison and the respective results obtained by the processing of S401 to S404. Incidentally, the desired shape and size to be compared with the estimated matrix area are set in advance before the start of the process.

[0064] Then, the display control unit 307 can notify the user that the columns are in an undesirable form by displaying the generated screen (a display screen as shown in FIG. 6(A)) on a display device such as the display 207. Further, as shown in FIG. 6(A), it is preferable to also display, as additional information, a predetermined message indicating that the columns are in an undesirable form. Any message may be used as the predetermined message as long as it can notify the user that the columns are in an undesirable form, such as, for example, "There may be a problem with the formation of the columns."

[0065] FIG. 6(B) is an example of a screen (result screen) to be displayed when the columns are not correctly formed. The example of FIG. 6(B) shows that a person has been detected but the matrix has not been detected. This can be achieved by the display control unit 307 determining whether there is a matrix area estimated by the area estimation unit 306. Note that the screen of FIG. 6(B) is an example of a screen to be displayed when the display control unit 307 determines that there is no matrix area. The display control unit 307 displays the generated screen (a screen as shown in FIG. 6(B)) on the screen of a display device such as the display 207 based on the result in this case (the determination result that there is no matrix area) and each result obtained in the processes of S401 to S404. At this time, it is preferable to also display the number of people detected from the image.

[0066] Then, the display control unit 307 can notify the user that a person has been detected but the matrix has not been detected by displaying the generated screen (a screen as shown in FIG. 6(B)) on a display device such as the display 207. Further, as shown in FIG. 6(B), it is preferable to also display, as additional information, a predetermined message notifying that a person has been detected but the matrix has not been detected. Any message may be used as the predetermined message as long as it can notify the user that a person has been detected but the matrix has not been detected, such as, for example, "No matrix has been formed."

[0067] FIG. 6(C) is an example of a screen (display screen) to be displayed when columns are not formed correctly. In the example of FIG. 6(C), as multiple matrix areas, matrix area 602 and matrix area 603 are detected, indicating that the columns are in an undesirable form (split into two branches). This can be achieved by the display control unit 307 determining whether there are multiple matrix areas estimated by the area estimation unit 306. Note that the screen in FIG. 6(C) is an example of a screen to be displayed when the display control unit 307 determines that there are multiple matrix areas. Based on the result in this case (the determination result that there are multiple matrix areas) and each result obtained in the processes of S401 to S404, the display control unit 307 displays the generated screen (a screen as shown in FIG. 6(C)) on the screen of a display device such as the display 207.

[0068] Furthermore, the time 604 when the columns in the matrix area 602 are detected and the time 605 when the columns in the matrix area 603 are detected are displayed as additional information. The matrix area 602 and the matrix area 603 are areas each surrounded by a dotted line.

[0069] Then, by displaying the generated screen (a display screen as shown in FIG. 6(C)) on a display device such as the display 207, the display control unit 307 can notify the user that the columns are in an undesirable form (e.g., split into two branches) due to multiple matrix areas. Furthermore, as shown in FIG. 6(C), it is preferable to also display, as additional information, a predetermined message notifying that the columns are in an undesirable form due to multiple matrix areas on the result screen. Any message can be used as the predetermined message as long as it can notify the user that the columns are in an undesirable form due to multiple matrix areas, such as, for example, "There may be a problem with the correct formation of the columns".

[0070] For example, when the matrix estimation process of S401 to S406 is performed at 1-minute intervals, the time 604 when a column is formed in the area surrounded by the matrix area 602 and the time (605) when a column is formed in the area surrounded by the matrix area 603 are shown. The formation time may be, for example, the time when the area in the corresponding image is estimated (judged) as the matrix area. In this case, since it can be seen that the matrix area 602 was formed at "10:19" and the matrix area 603 was generated at "10:21", displaying the time when the matrix area was generated for each matrix area is useful information for the user who arranges the columns.

[0071] FIG. 6(D) is an example of a screen (result screen) displayed for a person who is about to line up in a column. In the example of FIG. 6(D), when the matrix area 606 exceeds a certain threshold length, the position to line up next is presented in the area of area 607. This can be realized by the display control unit 307 determining whether the length of the matrix area estimated by the area estimation unit 306 exceeds a predetermined threshold. Note that the screen of FIG. 6(D) is an example of a screen displayed when the length of the matrix area exceeds the threshold by the display control unit 307 as described above. The display control unit 307 displays the generated screen (a screen such as that shown in FIG. 6(D)) on the screen of a display device such as the display 207 based on the result in this case (the determination result that the length of the column area exceeds the threshold) and each result obtained by the processes of S401 to S404. Note that the area 607 may be an area adjacent to the last person in the matrix area 606 and where there is no person, or the direction in which the column is to be extended may be set in advance.

[0072] Then, the display control unit 307 can notify the user about the position and area to line up next by displaying the generated screen (the display screen shown in FIG. 6(D)) on a display device such as the display 207. Further, as shown in FIG. 6(D), it is preferable to also display a predetermined message for notifying the user about the position and area to line up next as additional information. As the predetermined message, any message may be used as long as, for example, a message such as "The position or column area to line up next is here" can be notified to the user.

[0073] Next, the presentation of the next queuing position may be in any method as long as the position can be presented to the user. For example, it may be a method of displaying the result screen shown in FIG. 6(D) on a display device such as the display 207, or a method of illuminating the floor surface using a projector or the like capable of projection mapping.

[0074] FIG. 6(E) is an example of a screen presenting the prediction of the queue. In the example of FIG. 6(E), the area 608 is an area surrounded by a solid line and indicates the current queue area. The area 609 is an area indicated by a dotted line. The person icon of the person 610 indicates the predicted queue. The prediction of the queue may be predicted from the degree of elongation of the column, or may be estimated using the history of the queue results (history information of the queue area) such as the previous day or the previous week. That is, the display control unit 307 estimates the transition of the queue area from, for example, the history information of the queue area, and displays the prediction information of the queue area on the screen of a display device such as the display 207 based on the transition of the queue area and the distances between a plurality of persons.

[0075] Then, the display control unit 307 can notify the user about the prediction of the queue by displaying the generated screen (a screen such as that shown in FIG. 6(E)) on a display device such as the display 207. Further, as shown in FIG. 6(E), it is preferable to also display a predetermined message notifying about the prediction of the queue as additional information. As the predetermined message, any message may be used as long as, for example, a message such as "The prediction of the current queue after 10 minutes is as follows" can be notified to the user.

[0076] As described above, according to the video processing system 101 in the present embodiment, it becomes possible to measure the presence or absence and the length of the queue without previously specifying the area of the queue formed by the persons to be measured. Thereby, useful information can be appropriately presented to the user of the system for arranging the columns and the users queuing.

[0077] As described above, the preferred embodiments of the present invention have been described in detail. However, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0078] The present invention can also be realized by executing the following processes. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network for data communication or various storage media. Then, it is a process in which a computer (or CPU, MPU, etc.) of the system or device reads and executes the program. Further, the program may be recorded on a computer-readable recording medium and provided. The present invention is not limited to the above embodiments, and various modifications (including organic combinations of the respective embodiments) are possible based on the gist of the present invention, and they are not excluded from the scope of the present invention.

[0079] The disclosure of the present embodiment includes the following configurations, methods, and programs.

[0080] (Configuration 1) Detection means for acquiring face position information and posture information of a plurality of persons detected from an image, First estimation means for estimating the directions in which the plurality of persons are facing based on the posture information, Calculation means for calculating the distances between the plurality of persons based on the face position information, Second estimation means for estimating a matrix area composed of a plurality of persons by determining whether there are persons at distances within a predetermined range along the directions in which the plurality of persons are facing based on the face position information, the posture information, and the distances between the plurality of persons, Display control means for causing a display means to display the estimation result estimated by the second estimation means, An information processing system characterized by the above.

[0081] (Configuration 2) The second estimation means determines that when part or all of the determination regions, at least a part of which indicates the vicinity range of the person, overlap and at least a part of the overlapping determination regions are circular, the overlapping persons are at a distance within a predetermined range. The information processing system according to Configuration 1.

[0082] (Configuration 3) The second estimation means sets, as a point of interest, something that is the starting point of a column, and when at least a part of the point of interest is within a determination region, at least a part of which indicates the vicinity range of the person and is circular, determines that the person in the determination region is at a distance within a predetermined range from the point of interest. The information processing system according to Configuration 1 or 2.

[0083] (Configuration 4) The second estimation means determines, based on the posture information, whether a person determined to be at a distance within a predetermined range from the point of interest is facing the direction of the point of interest. When a person determined to be at a distance within a predetermined range from the point of interest is facing the direction of the point of interest, the information processing system according to Configuration 3 changes the person determined to be at a distance within a predetermined range from the point of interest to the next point of interest.

[0084] (Configuration 5) When a person becomes the point of interest, the second estimation means determines that a person in a determination region overlapping with the determination region of the person who has become the point of interest is at a distance within a predetermined range from the person who has become the point of interest. The information processing system according to Configuration 3 or 4.

[0085] (Configuration 6) When the second estimation means determines that there is no person at a distance within a predetermined range from the initially set point of interest, or when it determines that there is a person at a distance within a predetermined range from the initially set point of interest but the person is not facing the direction of the point of interest, it determines that no matrix is formed and does not perform the estimation of the matrix region. The information processing system according to any one of Configurations 3 to 5.

[0086] (Configuration 7) The second estimation means estimates the matrix area based on a plurality of points of interest or determination areas at the plurality of points of interest, and is the information processing system according to any one of Configurations 3 to 6, characterized in that.

[0087] (Configuration 8) The second estimation means calculates the matrix area as a rectangular area including the plurality of determination areas that have become the points of interest, and is the information processing system according to any one of Configurations 3 to 7, characterized in that.

[0088] (Configuration 9) The second estimation means changes the range of the determination area, and even if there is a person not facing the direction of the point of interest, calculates the matrix area as a rectangular area including the plurality of determination areas that have become the points of interest, and is the information processing system according to any one of Configurations 3 to 8, characterized in that.

[0089] (Configuration 10) The display control means estimates the transition of the matrix area from the history information of the matrix area, and displays the prediction of the matrix area on the display means based on the transition of the matrix area and the distance between the plurality of persons, and is the information processing system according to any one of Configurations 1 to 9, characterized in that.

[0090] (Configuration 11) When there are a plurality of matrix areas estimated by the second estimation means, the display control means displays on the display means that there are a plurality of matrix areas, and is the information processing system according to any one of Configurations 1 to 10, characterized in that.

[0091] (Configuration 12) When there are a plurality of matrix areas estimated by the second estimation means, the display control means displays on the display means that there are a plurality of matrix areas and the time when the matrix area was generated for each matrix area, and is the information processing system according to any one of Configurations 1 to 11, characterized in that.

[0092] (Configuration 13) The display control means determines whether the matrix area exists. When the matrix area does not exist, the display means displays the number of persons detected by the detection means and a message indicating that the matrix area does not exist. The information processing system according to any one of Configurations 1 to 12, characterized in that.

[0093] (Configuration 14) The display control means compares the shape of the matrix area estimated by the second estimation means with a preset shape, and the display means displays the result obtained by the comparison. The information processing system according to any one of Configurations 1 to 13, characterized in that.

[0094] (Configuration 15) The display control means determines whether the length of the matrix area estimated by the second estimation means exceeds a threshold value. When the length of the estimated matrix area exceeds the threshold value, the display means displays an area at a position different from the estimated matrix area as the area to be lined up next. The information processing system according to any one of Configurations 1 to 14, characterized in that.

[0095] (Configuration 16) A control method for an information processing system, comprising: A detection step of acquiring face position information and posture information of a plurality of persons detected from an image; A first estimation step of estimating the directions in which the plurality of persons are facing based on the posture information; A calculation step of calculating the distances between the plurality of persons based on the face position information; A second estimation step of estimating a matrix area composed of a plurality of persons by determining whether there are persons at distances within a predetermined range along the directions in which the plurality of persons are facing based on the face position information, the posture information, and the distances between the plurality of persons; A display control step of causing a display means to display the estimation result estimated in the second estimation step. A control method for an information processing system, characterized in that.

[0096] (Configuration 17) A program for causing a computer to execute a control method of an information processing system, the program causing the computer to, A detection step of acquiring face position information and posture information of a plurality of persons detected from an image; A first estimation step of estimating the directions in which the plurality of persons are facing based on the posture information; A calculation step of calculating the distances between the plurality of persons based on the face position information; A second estimation step of estimating a matrix area constituted by a plurality of persons by determining whether there are persons at distances within a predetermined range along the directions in which the plurality of persons are facing based on the face position information, the posture information, and the distances between the plurality of persons; A display control step of causing a display means to display the estimation result estimated in the second estimation step. A program characterized by the above.

Explanation of reference numerals

[0097] 302 Video acquisition unit 303 Detection unit 304 Direction estimation unit 305 Distance calculation unit 306 Area estimation unit 307 Display control unit

Claims

1. Detection means for acquiring face position information and pose information of a plurality of persons detected from an image; First estimation means for estimating the direction in which the plurality of persons are facing based on the pose information; Calculation means for calculating the distance between the plurality of persons based on the face position information; Second estimation means for estimating a matrix area composed of a plurality of persons by determining whether there are persons at a distance within a predetermined range along the direction in which the plurality of persons are facing, based on the face position information, the pose information, and the distance between the plurality of persons; Display control means for causing a display means to display the estimation result estimated by the second estimation means. An information processing system characterized by the above.

2. The information processing system according to claim 1, wherein the second estimation means determines that persons who overlap are at a distance within a predetermined range when at least a part of the determination areas indicating the vicinity ranges of the persons are circular and at least a part of them overlaps.

3. The information processing system according to claim 1, wherein the second estimation means sets as a point of interest something that is the starting point of a column, and when at least a part of the point of interest enters a determination area indicating the vicinity range of the person and at least a part of the determination area is circular, determines the person in the determination area to be a person at a distance within a predetermined range from the point of interest.

4. The information processing system according to claim 3, wherein the second estimation means determines whether a person determined to be at a distance within a predetermined range from the point of interest is facing the direction of the point of interest based on the pose information, and when a person determined to be at a distance within a predetermined range from the point of interest is facing the direction of the point of interest, changes the person determined to be at a distance within a predetermined range from the point of interest to the next point of interest.

5. The information processing system according to claim 3, wherein when a person becomes a point of interest, the second estimation means determines a person in a determination area that overlaps with the determination area of the person who has become the point of interest to be a person at a distance within a predetermined range from the person who has become the point of interest.

6. The information processing system according to claim 3, wherein when the second estimation means determines that there is no person at a distance within a predetermined range from the initially set point of interest, or when it is determined that there is a person at a distance within a predetermined range from the initially set point of interest but the person is not facing the direction of the point of interest, the second estimation means determines that no matrix is formed and does not perform the estimation of the matrix area.

7. The information processing system according to claim 3, wherein the second estimation means estimates the matrix area based on a plurality of points of interest or determination areas at the plurality of points of interest.

8. The information processing system according to claim 3, wherein the second estimation means calculates the matrix area as a rectangular area including a plurality of determination areas that have become the points of interest.

9. The information processing system according to claim 3, wherein the second estimation means changes the range of the determination area so that even if there is a person who is not facing the direction of the point of interest, the matrix area is calculated as a rectangular area including a plurality of determination areas that have become the points of interest.

10. The information processing system according to claim 1, wherein the display control means estimates the transition of the matrix area from the history information of the matrix area, and displays the prediction of the matrix area on the display means based on the transition of the matrix area and the distances between the plurality of persons.

11. The information processing system according to claim 1, wherein when there are a plurality of matrix areas estimated by the second estimation means, the display control means displays on the display means that there are a plurality of matrix areas.

12. The information processing system according to claim 1, wherein when there are a plurality of matrix areas estimated by the second estimation means, the display control means displays on the display means for each matrix area that there are a plurality of matrix areas and the time when the matrix area was generated.

13. The information processing system according to claim 1, wherein the display control means determines whether the matrix area exists, and when the matrix area does not exist, the display control means displays on the display means the number of persons detected by the detection means and that the matrix area does not exist.

14. The information processing system according to claim 1, wherein the display control means compares the shape of the matrix area estimated by the second estimation means with a preset shape, and displays the result obtained by the comparison on the display means.

15. The display control means determines whether the length of the matrix area estimated by the second estimation means exceeds a threshold value. When the length of the estimated matrix area exceeds the threshold value, the display control means displays, as the area to be arranged next, an area at a position different from the estimated matrix area. The information processing system according to claim 1, characterized in that.

16. A control method for an information processing system, comprising: a detection step of acquiring face position information and posture information of a plurality of persons detected from an image; a first estimation step of estimating the directions in which the plurality of persons are facing based on the posture information; a calculation step of calculating the distances between the plurality of persons based on the face position information; a second estimation step of estimating a matrix area composed of a plurality of persons by determining whether there are persons at distances within a predetermined range along the directions in which the plurality of persons are facing, based on the face position information, the posture information, and the distances between the plurality of persons; a display control step of causing a display means to display the estimation result estimated in the second estimation step. A control method for an information processing system, characterized by the above.

17. A program for causing a computer to execute a control method for an information processing system, the program causing the computer to execute a detection step of acquiring face position information and posture information of a plurality of persons detected from an image; execute a first estimation step of estimating the directions in which the plurality of persons are facing based on the posture information; execute a calculation step of calculating the distances between the plurality of persons based on the face position information; execute a second estimation step of estimating a matrix area composed of a plurality of persons by determining whether there are persons at distances within a predetermined range along the directions in which the plurality of persons are facing, based on the face position information, the posture information, and the distances between the plurality of persons; execute a display control step of causing a display means to display the estimation result estimated in the second estimation step. A program, characterized by the above.

Images