Person monitoring system, person monitoring method, and person monitoring processing program.

Abstract

To provide a person monitoring system, a person monitoring method, and a person monitoring processing program capable of quickly and accurately monitoring the number of persons and a person-to-person distance with a simple configuration regardless of the states of the persons existing within a predetermined monitoring region.SOLUTION: In a person monitoring system 1, a person measurement device 10 includes: a photographing unit 110 that photographs persons 500a to 500c; and a depth measurement unit 120 that receives reflected waves IL12 to IL32 obtained when indoor illumination light or the like is reflected by the persons 500a to 500c to measure a distance. Further, an information processing device 20 (personal computer 410) includes: a person region identification unit that identifies a person region from an image; a person number calculation unit that calculates the number of persons; a person region distance calculation unit that calculates a first distance that is the distance from the depth measurement unit 120 to the person region for each person; a person-to-person distance calculation unit that calculates a second distance that is the distance between respective persons; and a monitoring unit that determines whether information on the number of persons and information on the second distance satisfy predetermined conditions.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a person monitoring system that monitors the number of people present in a predetermined monitoring area and the distance between the people. [Background technology]

[0002] 2. Description of the Related Art Conventionally, the presence of people and the degree of crowding within a given area have been analyzed using surveillance cameras and the like, and the analysis results have been utilized for security purposes, marketing purposes, and the like.

[0003] For example, Patent Document 1 describes the following human detection system. That is, from a change in luminance value of a pixel in a time-series continuous video sequence captured by a plurality of imaging means, which occurs when an object moves through the time-series continuous video sequence, the velocity vector of the pixel is calculated for each time-series continuous video sequence by using an optical flow estimation calculation. This human detection system is equipped with such a vector calculation means.

[0004] Patent Document 2 also describes a distance estimation device as follows, exemplified by the distance between runners in a relay race. That is, the directions in the video of the recognized people who are estimated to be the first specified person and the second specified person are estimated, and the positional relationship between the first specified person and the second specified person is estimated by referring to the estimated direction of the specified person. Then, the distance between the identified first specified person and the second specified person is estimated by using the estimated positional relationship between the first specified person and the second specified person and a specified object in the video.

[0005] On the other hand, Patent Document 3 describes a human count detection device equipped with a laser radar that acquires distance information at each position in a specified monitoring area by irradiating a laser beam in a scanning manner within the area and detecting the reflected light of the laser beam. That is, the height of a person is determined based on the distance information at each position obtained by the laser radar, and the number of people present in the monitoring area is determined by comparing the height with a threshold value corresponding to the height range from the neck to the head of a person with a standard physique. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2003-109001 A [Patent Document 2] JP 2020-109556 A [Patent Document 3] JP 2006-209318 A Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1 and Patent Document 2, a target person is assumed to be moving, and the person is photographed by multiple cameras, and the person is detected and the distance between the people is specified. In Patent Document 3, a laser light is scanned over the entire target area to determine the characteristics of the target person, and the number of people is specified based on the characteristics. On the other hand, there is a demand for monitoring the number of people and the distance between people in a specified monitoring area quickly and accurately with as simple a configuration as possible, regardless of whether the people in the specified monitoring area are stationary or moving.

[0008] The present disclosure has been made in consideration of such circumstances, and has as its objective the provision of a person monitoring system, a person monitoring method, and a person monitoring processing program that are capable of quickly and accurately monitoring the number of people and the distance between people with as simple a configuration as possible, regardless of the status of people present in a specified monitoring area. [Means for solving the problem]

[0009] The person monitoring system according to this embodiment, which monitors the number of people and the distance between people present within a specified monitoring area, comprises a person measurement device including an imaging unit which captures an image of the person and a depth measurement unit which measures the distance to an object, and an information processing device including a person area identification unit which identifies a person area corresponding to the person from the image, a person number calculation unit which calculates the number of people from information about the person area, a person area distance calculation unit which calculates a first distance, which is the distance from the depth measurement unit to the person area for each person, from the information about the person area and information about the distance measured by the depth measurement unit, a person distance calculation unit which calculates a second distance, which is the distance between each person, if the number of people is two or more, and a monitoring unit which determines whether the information on the number of people and the information on the second distance satisfy specified conditions.

[0010] In addition, in a person monitoring system according to another embodiment of the present invention, the information processing device may further include a person area spatial coordinate calculation unit that calculates spatial coordinates of the person in the monitoring area, and the person distance calculation unit may calculate the second distance based on information of the spatial coordinates.

[0011] Furthermore, the human monitoring system according to another embodiment of the present invention may further include a display device that displays a message indicating that the monitoring unit does not satisfy the predetermined condition, or an alarm device that issues an audio warning to that effect, when the monitoring unit determines that the predetermined condition is not satisfied.

[0012] In addition, in a person monitoring system according to another embodiment of the present invention, the person area identification unit may include a learning model that has undergone machine learning to identify the person from the image and identify the person area of ​​the person.

[0013] In addition, in a person monitoring system according to another embodiment of the present invention, the person measuring device further includes a carbon dioxide concentration measuring unit that measures the carbon dioxide concentration in the air within the monitoring area, and the monitoring unit may determine whether or not the carbon dioxide concentration information measured by the carbon dioxide concentration measuring unit satisfies the specified condition, in addition to the information on the number of people and the information on the second distance.

[0014] In addition, in a person monitoring system according to another embodiment of the present invention, when the number of people is two or more, and the person area identification unit has identified a first person area corresponding to a first person and a second person area corresponding to a second person as the person areas corresponding to the people, and it is determined from the distance information measured by the depth measurement unit that there is an obstruction other than a person between the first person area and the second person area, the monitoring unit may exclude information on the distance between the first person area and the second person area from the second distance information and determine whether the specified condition is satisfied.

[0015] In addition, a person monitoring system according to another embodiment of the present invention may further include an identification means for identifying each of the people corresponding to the person areas identified by the person area identification unit, and the information processing device may further include a memory unit, and the information processing device may store information of the second distance corresponding to each of the people in the memory unit in association with information of the time when the depth measurement unit measured the distance corresponding to the first distance.

[0016] The person monitoring method according to this embodiment for monitoring the number of people present within a specified monitoring area and the distance between the people includes the steps of: capturing an image of the person; measuring a first distance, which is the distance to the person, using a depth measurement unit; identifying a person area corresponding to the person from the image; calculating the number of people from information about the person areas; calculating a first distance, which is the distance from the depth measurement unit to the person area for each person, from information about the person areas and information about the distance measured by the depth measurement unit; if the number of people is two or more, calculating a second distance, which is the distance between each person; and determining whether the information about the number of people and the second distance satisfy a specified condition.

[0017] In this embodiment, the program causing a computer to execute a process of monitoring the number of people present within a specified monitoring area and the distance between the people causes the computer to execute a process of identifying a person area corresponding to the person from an image of the person, a process of calculating the number of people from information about the person area, a process of calculating a first distance, which is the distance from the depth measurement unit to the person area for each person, from information about the person area and information about the distance measured by the depth measurement unit, a process of calculating a second distance, which is the distance between each person, if the number of people is two or more, and a process of determining whether the information on the number of people and the information on the second distance satisfy a specified condition. Effect of the Invention

[0018] According to this embodiment, it is possible to provide a person monitoring system, a person monitoring method, and a person monitoring processing program that are capable of monitoring the number of people and the distance between people quickly and accurately with as simple a configuration as possible, regardless of the status of people present in a specified monitoring area. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic perspective view illustrating a person monitoring system according to a first embodiment. [Diagram 2] 1 is a configuration diagram showing the configuration of a person monitoring system according to a first embodiment. [Diagram 3] 3A and 3B are explanatory diagrams of an image captured by a capturing unit and a plan view of a monitoring area. [Figure 4] This is a cross-sectional view of the monitored area cut along a plane along the Z-axis direction and the direction of each person. [Diagram 5] FIG. 4 is a schematic perspective view illustrating a first distance and a second distance. [Figure 6] FIG. 11 is a schematic perspective view illustrating the first distance from another perspective. [Figure 7] FIG. 2 is a flow diagram illustrating steps of the person monitoring method according to the first embodiment. [Figure 8] FIG. 11 is a schematic perspective view illustrating a person monitoring system according to a second embodiment. [Figure 9] FIG. 11 is a schematic perspective view for explaining a person monitoring system according to a third embodiment. [Figure 10] FIG. 13 is a diagram showing the configuration of a person monitoring system according to a third embodiment. [Figure 11] 13A and 13B are schematic perspective views and image explanatory diagrams for explaining a person monitoring system according to a fourth embodiment. [Figure 12] 13 is an example of an information table in the person monitoring system according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] An example of a person monitoring system, a person monitoring method, and a person monitoring processing program according to the present disclosure will be described below with reference to the drawings, etc. However, the person monitoring system and the like according to the present disclosure are not limited to the embodiments and examples described below.

[0021] The figures shown below are schematic. Therefore, the size and shape of each part are appropriately exaggerated to facilitate understanding. In addition, hatching showing the cross section of a member is appropriately omitted in each figure. The numerical values ​​such as dimensions of each member and the material names described in this specification are examples of embodiments, and are not limited thereto, and can be appropriately selected and used. In this specification, terms that specify shapes and geometric conditions, such as parallel, orthogonal, and vertical, are intended to include substantially the same state in addition to their strict meanings.

[0022] 1. First embodiment A first embodiment of a person monitoring system according to the present disclosure will be described. As shown in Fig. 1, in a predetermined monitoring area S1, people 500a, 500b, and 500c exist as people to be monitored. In addition, in S1, a rack 400 unrelated to the monitoring targets and a personal computer 410 placed on the rack 400 are disposed. In addition, above the wall that divides S1, a person measuring device 10 for acquiring information on people existing in S1 is disposed. The person measuring device 10 includes at least an image capturing unit 110 and a depth measuring unit 120.

[0023] Here, for convenience of explanation of the monitoring area S1, XYZ coordinates are provided for the space of S1. The people 500a, 500b, and 500c and the rack 400, etc. are placed on a floor surface along the XY plane. The vertical direction is the direction of the Z axis, the upper side is the +Z direction, and the lower side is the -Z direction. When the people are viewed along the Y axis direction, the people 500a, 500b, and 500c are stationary and arranged so as to be lined up in order from the +Y direction side. Similarly, when viewed along the X axis direction, the people 500a, 500b, and 500c are arranged so as to be lined up in order from the -X direction side. In addition, the person measurement device 10 is fixed above the wall surface along the YZ plane.

[0024] The photographing unit 110 of the person measuring device 10 is, for example, a CCD camera, and outputs a signal according to the intensity of the light incident on the photographing unit 110, including the reflected light VL1, VL2, and VL3 reflected from indoor illumination light, etc., on the people 500a, 500b, and 500c. These signals can be replaced with pixel information arranged on a plane to form a visualized image. Fig. 3(a) shows an image 420 photographed by the photographing unit 110 of the monitoring area S1.

[0025] The depth measurement unit 120 of the person measurement device 10 emits infrared rays of a predetermined frequency, for example, and receives reflected waves IL12, IL22, and IL32 reflected by the people 500a, 500b, and 500c. It can also measure the time from the emission to the incidence of the infrared rays and calculate the distance to the target object.

[0026] 3(a), a personal computer 410, which is an information processing device 20 described later, identifies person regions 520a, 520b, and 520c of the people 500a, 500b, and 500c from an image 420 captured by the image capture unit 110. Furthermore, the personal computer 410 calculates representative points 510a, 510b, and 510c corresponding to the respective person regions. The representative points form part of the person regions.

[0027] Here, the photographing unit 110 and the depth measuring unit 120 of the person measuring device 10 are adjacent to each other and arranged at almost the same place. Therefore, it is possible to identify that three people 500a, 500b, and 500c are present in the monitoring area S1 from the information obtained from the photographing unit 110 and the depth measuring unit 120. In addition, it is possible to identify the direction of each of the people 500a, 500b, and 500c as viewed from the person measuring device 10, and the distance to each of the people 500a, 500b, and 500c.

[0028] As a result, the personal computer 410, which is the information processing device 20, can calculate the spatial coordinates of each of the persons 500a, 500b, and 500c in the monitoring area S1, as described later. Furthermore, the distance between each person can be calculated based on the spatial coordinates.

[0029] As a result, by setting some conditions for the number of people in the monitoring area S1 and the distance between each person, the people monitoring system 1 can determine whether the state in the monitoring area S1 satisfies the predetermined condition and issue a notification or warning as necessary. Therefore, the people monitoring system 1 according to this embodiment can monitor people with a relatively simple configuration, regardless of whether people in the predetermined monitoring area S1 are stationary or moving.

[0030] That is, the person monitoring system 1 does not require multiple image capturing units, and can be configured with at least a person measurement device 10 consisting of a set of an image capturing unit 110 and a depth measurement unit 120, and an information processing device 20 which can be exemplified as a personal computer 410. In addition, the number of people and the distances between people can be accurately monitored. The configuration of the person monitoring system 1 and the person monitoring method according to this embodiment will be described in detail below.

[0031] [a] Configuration of people monitoring system As shown in FIG. 2, the person monitoring system 1 includes, as a minimum configuration, a person measurement device 10 and an information processing device 20, and further includes an operation / display device 310 and an alarm device 340 in this embodiment.

[0032] [i] Person measuring device As described above, the human measurement device 10 includes, as a minimum configuration, an image capturing unit 110 and a depth measuring unit 120. The image capturing unit 110 is, for example, a color or monochrome two-dimensional CCD camera. The CCD camera outputs to the outside the state of the intensity of the incident light as 256-level luminance values ​​of CCD elements corresponding to each pixel arranged in a planar matrix. In the case of a color CCD, the output is made to the outside as 256-level luminance values ​​of three types of CCD elements, RGB, corresponding to each pixel. The output values ​​of the CCD elements of each pixel are transmitted to the information processing device 20, which will be described later.

[0033] That is, light reflected from objects, including people, within the monitoring area S1 and incident on the image capture unit 110 is output as a luminance gradation signal corresponding to the light intensity of each CCD element, and is taken into the image information acquisition unit 220 via the image capture unit interface 210. Then, a visualized image 420 as shown in FIG. 3(a) is formed.

[0034] The photographing unit 110 may be configured to form an image for invisible light, not limited to an image for visible light. For example, the photographing unit 110 may be configured to have a high sensitivity type monochrome CCD element with a filter that transmits only infrared light of a predetermined frequency band. In this case, even if the amount of light in the monitored area S1 is insufficient, such as in an unlit room at night, an image of the monitored area S1 including a person, such as the image 420, can be obtained satisfactorily.

[0035] On the other hand, the depth measurement unit 120 includes an emission unit that emits infrared rays of a predetermined wavelength to an object, and a sensor unit that receives reflected waves from the object. The time required for the infrared rays emitted from the emission unit to the object to be received by the sensor unit is measured, and the round trip distance to the object can be calculated by multiplying this by the speed of light. As a result, the distance from the depth measurement unit 120 to each person can be calculated. As a method for measuring the required time, various known measurement methods can be used. Note that the distance from the depth measurement unit 120 to each person represents the straight-line distance from the reference point of the depth measurement unit 120 to the reference point of each person, and also represents the shortest distance from a certain plane, for example, the image plane of the shooting unit 110 or a plane corresponding thereto, to the reference point of each person.

[0036] For example, there are those that transmit infrared rays as a predetermined pulse wave and measure the difference between the time the pulse wave is transmitted and the time it is received, and those that detect the phase shift between the time the infrared rays are transmitted and the time they are received and calculate the elapsed time from the amount of shift. For example, a TOF (Time Of Flight) camera uses these methods.

[0037] The required time can also be calculated by continuously observing the change in intensity of the reflected light beam using a technique such as scattered light pulse imaging. Another method is to irradiate an object with infrared light, which is a pattern light of a lattice or stripe pattern, capture the distorted pattern of the reflected light reflected by the object with an infrared CCD camera, and calculate the distance to the object from the degree of deformation. Note that the depth measurement unit 120 does not need to be configured completely independent of the imaging unit 110, and the sensor unit for detecting light reception of the depth measurement unit 120 may be configured to share the CCD element of the imaging unit 110.

[0038] It is advantageous to place the above-mentioned photographing unit 110 and depth measuring unit 120 constituting the person measuring device 10 at a relatively high position in the vertical direction in order to suppress problems in identifying a person area from an image and measuring the distance to the person area due to overlapping of people, etc. The placement height of the person measuring device 10 is preferably placed so that the measurement units of the photographing unit 110 and depth measuring unit 120 are located higher than the average person's height, for example, in a range of 2 m to 4 m from the floor surface, and more preferably in a range of 2.5 m to 3 m.

[0039] [ii] Information processing device 2, the information processing device 20 includes an imaging unit interface 210, an image information acquisition unit 220, a person area identification unit 230, a person direction identification unit 245, and a person number calculation unit 240 as units that mainly process information from the imaging unit 110. The information processing device 20 also includes a depth measurement unit interface 250, a distance information acquisition unit 260, a person area distance calculation unit 270, a person area spatial coordinate calculation unit 280, and a person-to-person distance calculation unit 290 as units that mainly process information from the depth measurement unit 120.

[0040] Furthermore, the information processing device 20 includes a monitoring unit 200 that determines whether or not a predetermined condition is satisfied based on information from the person number calculation unit 240 and the person distance calculation unit 290, and includes a storage unit 295 for storing data on the predetermined condition and other necessary data. In particular, the person area identification unit 230, the person number calculation unit 240, the person area distance calculation unit 270, the person distance calculation unit 290, and the monitoring unit 200 are essential components of the information processing device 20.

[0041] The information processing device 20 is configured with a CPU or MPU, a memory, etc., and performs input / output control with the person measurement device 10 and the operation / display device 310, power control, and various calculations and judgments. The information processing device 20 also includes a storage unit 295 as necessary, and the storage unit 295 is configured with a non-volatile rewritable memory such as a ROM, a RAM, and a flash memory.

[0042] The ROM stores the BIOS, which is a startup program for the CPU, etc., and the OS, etc., and the RAM is used as a temporary working memory for the CPU, etc. The non-volatile rewritable memory is used to store image information and distance information from the person measurement device 10, various application programs and parameters, etc., and to store variable information for holding data in various data tables and databases.

[0043] In this embodiment, the information processing device 20 is realized by a personal computer 410. However, the information processing device 20 is not limited to this, and may be integrated with the person measurement device 10, or may be a mobile terminal such as a smartphone. In addition, the functions of each part of the information processing device 20 described above may be realized as hardware, or may be realized by executing a program such as an application program. Next, each part of the information processing device 20 will be described.

[0044] [1] Camera interface and depth measurement interface The photographing unit interface 210 is a unit that inputs and outputs information to and from the photographing unit 110 of the person measurement device 10. In addition, the depth measurement unit interface 250 is a unit that inputs and outputs information to and from the depth measurement unit 120. These may be integrated as an interface with the person measurement device 10.

[0045] In the case where the person measurement device 10 and the information processing device 20 perform wireless communication, an antenna for transmitting and receiving radio waves for wireless communication with the person measurement device 10 and a rectifier circuit for converting the radio waves into a DC signal may be provided. In this case, the person measurement device 10 also needs to be provided with a similar antenna for communication.

[0046] [2] Image information acquisition section The image information acquisition unit 220 is a part that acquires image information of the monitoring area S1 captured by the imaging unit 110 via the imaging unit interface 210. Specifically, it takes in luminance information expressed in one of 256 steps corresponding to each pixel of the planar image from the imaging unit 110, which is a CCD camera or the like, and converts this into visualized image information arranged in a matrix of the planar image.

[0047] The image information is, for example, that shown in image 420 in Fig. 3(a) . These pieces of image information are used to specify person regions 520a, 520b, and 520c of people 500a, 500b, and 500c, and to calculate the number of people and the distance between people, by various calculation processes and the like described later.

[0048] [3] Human area identification section The person region identification unit 230 identifies person regions 520a, 520b, and 520c in which specific persons 500a, 500b, and 500c exist in the image 420 acquired by the image information acquisition unit 220. In addition, it can calculate representative points 510a, 510b, and 510c corresponding to the person regions 520a, 520b, and 520c as necessary.

[0049] As a method for identifying the presence or absence of a person and the area of ​​the person in the image 420 captured by the image capturing unit 110, for example, there is a method for extracting the contour of the person by an edge detection technique. This can be done by extracting pixels where the brightness difference between adjacent pixels is equal to or greater than a certain level. Alternatively, an unsharp mask image may be created in which the brightness values ​​of a certain number of adjacent pixels within a certain radius are replaced with an average brightness value, and a difference image may be formed by extracting the difference between this image and the original image on a pixel-by-pixel basis. In this way, an area with a contour that has a strong contrast with the background can be extracted.

[0050] Next, such an area is divided into a human area and a non-human area. For an area having a contour extracted as described above, a contour shape feature may be extracted, and whether the area is a human area or a non-human area may be determined based on whether the area has all or part of the features of a head, torso, arms, legs, etc. Alternatively, a reference pattern that is a candidate for a shape showing a characteristic contour of a human area may be registered in advance in the storage unit 295, and the degree of match with the reference pattern may be compared to determine whether the area is a human area or a non-human area.

[0051] Furthermore, the determination of whether an area is a human area or a non-human area may be made not only based on the contour shape of the image but also by registering in advance in the storage unit 295 a reference pattern as a human image including specific image information of a person's head, torso, arms, legs, etc. Then, a successive comparison type pattern matching method may be used to make the determination by comparing the degree of match with the reference pattern.

[0052] Furthermore, the person region identification unit 230 may be provided with a learning model that has undergone machine learning to identify a person from a captured image and identify the person's person region. For example, the person region identification unit 230 is provided with a learning model that has been generated as a result of machine learning using a pattern matching or deep learning technique with a specific person image having a person region as training data. The learning model is configured to input specific image information and output information on the person region identified in the image.

[0053] The image information is, for example, information in which address information of each pixel of a matrix-arranged image is associated with information on the luminance gradation value of the pixel. Also, the information on the person area is, for example, address information of a specific pixel corresponding to the person area, or address information of a specific pixel corresponding to the outline of the person area. However, the image information and the information on the person area may be information other than the above as long as the person area can be appropriately output from the input image by the learning model.

[0054] It is preferable that the teacher data covers all the attributes and behaviors of people who may be present in the monitoring area S1, and it is preferable to prepare image data that takes into consideration gender, age difference, orientation and posture with respect to the imaging unit 110, etc. However, when the attributes and behaviors of people who may be present in the monitoring area S1 are limited, the accuracy of human area detection can be improved by using teacher data specialized for those features.

[0055] Furthermore, the person region identification unit 230 may further calculate representative points 510a, 510b, and 510c of each person based on the person regions 520a, 520b, and 520c identified from the image 420 by the above-mentioned method. The representative points are part of the person regions, but are intended to facilitate the calculation of the distance between people, which will be described later. If the representative points are calculated, the distance between two specific people can be calculated by replacing it with the distance between the representative points of the people, thereby making the calculation fast and easy. The representative points can be determined by any method as positions that are considered to represent the positions of the people.

[0056] For example, the center of gravity, i.e., the centroid, of the person region of the target person may be used as the representative point. Alternatively, the smallest rectangle that encompasses the person region may be created, and the intersection of the diagonals, which is the centroid of the rectangle, may be used as the representative point. Furthermore, if the purpose of monitoring the monitoring region S1 is to prevent droplet infection between people, attention may be focused only on the head regions of the person region, and the centroid or the point that forms the shortest distance between the head regions of two specific people may be used as the representative point.

[0057] The above-described algorithm used for identifying the person region by the person region identifying unit 230, the reference pattern used for pattern matching, the learning model generated as a result of machine learning, and the like may be stored in an internal memory of the person region identifying unit 230 itself or in a storage unit 295 provided in the information processing device 20. In addition, all or a part of these may be stored in a removable memory element such as a USB memory, an SD (registered trademark) card, a CD-ROM, a CD-R, or an IC card.

[0058] Furthermore, all or part of these may be stored in a storage unit fixed to the person region identification unit 230 or the like, and the information may be rewritten at any time from an external device via wired or wireless communication. In this way, the algorithm to be used, the reference pattern used for pattern matching, the learning model generated as a result of machine learning, and the like can be quickly and easily updated, enabling more accurate human region identification.

[0059] [4] Person direction identification part The person direction identifying unit 245 identifies the directions of person areas 520a, 520b, and 520c in the image 420. That is, the person direction identifying unit 245 identifies an angle from the person measurement device 10 toward the person area of ​​the target person, based on a specific plane defined in the monitoring area S1. What the person direction identifying unit 245 identifies may be a range of angles from the person measurement device 10 toward the person area of ​​the target person, or may be an angle toward the representative point of the target person described above.

[0060] In calculating such an angle, for example, a portion that serves as a base plane and a portion that serves as a reference point whose angle from the base plane is known are determined in the monitoring area S1. These portions are then photographed as the image 420, and the angle of the person area is calculated from the difference in the relative position between the portion in the image 420 and the person area.

[0061] In addition, information associating each pixel position of image 420 with its angle from a reference plane may be stored as a data table in storage unit 295 or the like, and angle data corresponding to each pixel position in the person area of ​​an actually captured image may be read out.

[0062] A method for calculating the angle from the person measurement device 10 toward the representative point of the target person will be described. Fig. 3(b) is a plan view of the monitoring area S1 corresponding to the image 420 as seen from the +Z direction. Here, the angles formed clockwise from the photographing unit 110 to the representative points 510a, 510b, and 510c of each person are respectively set to α1, α2, and α3, based on a straight line parallel to the Y axis passing through the photographing unit 110.

[0063] In other words, a plane parallel to the YZ plane passing through the imaging unit 110 is used as the reference plane, and the angles that the line segments connecting the imaging unit 110 and each person's representative points 510a, 510b, and 510c make clockwise around the Z axis from the reference plane as viewed from the +Z direction are α1, α2, and α3, respectively.

[0064] Also, when the straight lines connecting the image capturing unit 110 and the representative points 510a, 510b, and 510c of each person are respectively designated as lines P1, P2, and P3, Fig. 4(a) is a projection diagram in which the image capturing unit 110 and person 500a are projected onto a plane along both the Z axis and line P1, and this is viewed from the side closer to line P2. Similarly, Fig. 4(b) and Fig. 4(c) are projection diagrams in which line P1 in Fig. 4(a) is replaced with lines P2 and P3, respectively, and person 500a is replaced with person 500b and 500c, respectively.

[0065] Here, the angles from the image capturing unit 110 to the representative points 510a, 510b, and 510c of each person in a counterclockwise direction are defined as β1, β2, and β3, respectively, with a straight line parallel to the Z axis passing through the image capturing unit 110 as a reference. In other words, a plane with P1 as a normal is defined as a reference plane, and the angle from the reference plane to the representative point 510a of the person 500a in a clockwise direction around the axis of the direction perpendicular to the P1 axis and the Z axis, as seen from a direction in which the +Z direction appears to be perpendicular to the +P1 direction in a counterclockwise direction, is β1. β2 and β3 can also be defined in the same way by replacing P1 with P2 and P3, and the representative point 510a with the representative points 510b and 510c.

[0066] In this way, the directions, i.e., angles, from the photographing unit 110, i.e., the person measuring device 10, to the representative points 510a, 510b, and 510c of the people 500a, 500b, and 500c in the image 420 can be determined by obtaining the angles α1, α2, α3 and angles β1, β2, β3 described above. Also, without particularly defining the representative points 510a, 510b, and 510c, the angles may be obtained for each pixel corresponding to the person areas 520a, 520b, and 520c, or a plurality of representative points may be defined for one person and their angles may be obtained.

[0067] However, as described below, if the depth measurement unit 120 can measure the shortest distance from the image plane of the imaging unit 110 or a plane corresponding thereto to each person, the information processing device 20 does not need to include such a person direction identification unit 245. In this case, even if the directions of the person areas 520a, 520b, and 520c cannot be identified for the image 420, the first distance and the second distance described below can be easily calculated.

[0068] [5] Number of people calculation section The person number calculation unit 240 calculates the number of people present in the image 420 based on information on person areas identified from the image 420. For example, in this embodiment, the person areas identified in the image 420 are 520a, 520b, and 520c, which indicates that there are three people, 500a, 500b, and 500c. Therefore, the number of people in the monitoring area S1 in this case is determined to be three.

[0069] [6] Distance information acquisition unit The distance information acquisition unit 260 acquires an output value according to the time from emission to incidence of infrared rays on the object measured by the depth measurement unit 120 of the person measurement device 10 via the depth measurement unit interface 250, and converts this into a specific distance value from the depth measurement unit 120 to the object. The distance information acquisition unit 260 may be configured to constantly repeat irradiation of infrared rays while changing the radiation angle, and output of time information from emission to incidence of infrared rays.

[0070] Alternatively, distance information acquisition section 260 may be configured to irradiate a specific infrared ray only in the direction of a specific person area or representative point, and output time information from emission to incidence of the infrared ray, in accordance with an instruction from person area distance calculation section 270 (described later). By configuring distance information acquisition section 260 in the latter manner, it is only necessary to irradiate a specific infrared ray only at necessary locations and output time information from emission to incidence, enabling high-speed and efficient distance measurement.

[0071] [7] Person area distance calculation unit Person area distance calculation unit 270 calculates the distance from depth measurement unit 120 to person areas 520a, 520b, and 520c of each of persons 500a, 500b, and 500c identified by person area identification unit 230 by acquiring the distance from distance information acquisition unit 260. As described above, the calculation method may be performed by extracting only information corresponding to the distance to person areas 520a, 520b, and 520c based on the results of infrared irradiation and output of required time information already performed by depth measurement unit 120 while changing the emission angle.

[0072] Furthermore, person area distance calculation section 270 may instruct distance information acquisition section 260 or depth measurement section 120 on information relating to the positions of person areas 520a, 520b, and 520c identified by person area identification section 230. Then, depth measurement section 120 may irradiate infrared rays and output required time information only in the corresponding direction, and person area distance calculation section 270 may acquire the results.

[0073] In the latter case, the flow is as follows: Fig. 5(a) is a schematic perspective view of a monitoring area S1 corresponding to Fig. 1. The depth measurement unit 120 emits predetermined infrared irradiation lights IL11, IL21, and IL31 from the emission unit toward representative points 510a, 510b, and 510c of person areas 520a, 520b, and 520c in a spot manner.

[0074] In response to this, reflected light IL12, IL22, and IL32 from each of the people 500a, 500b, and 500c are received by the sensor unit of the depth measurement unit 120. This makes it possible to calculate the distance from the depth measurement unit 120 to the representative points 510a, 510b, and 510c of each person by the method described above.

[0075] [8] Person area spatial coordinate calculation unit The person area spatial coordinate calculation unit 280 calculates the spatial coordinates of each person based on information on the direction from the imaging unit 110 to the person area of ​​each person, acquired from the person direction identification unit 245, and information on the distance to the person area of ​​each person, acquired from the above-mentioned person area distance calculation unit 270. In this embodiment, the person area is treated as a representative point, so a method of calculating the spatial coordinates of each person will be described based on this. Note that the description will be given assuming that the imaging unit 110 and the depth measurement unit 120 are arranged at approximately the same position, but if the distance between them cannot be ignored, this distance may be corrected before calculation.

[0076] The distances from the depth measurement unit 120 to the representative points 510a, 510b, and 510c of the people 500a, 500b, and 500c, respectively, calculated by the person area distance calculation unit 270, are assumed to be Da, Db, and Dc. These distances are also referred to as first distances. First, the spatial coordinates of the representative point 510a of the person 500a are calculated.

[0077] As shown in Fig. 4(a), distance Da is the distance from the imaging unit 110 to the representative point 510a on the P1Z plane. Here, the projection distance of this distance onto the P1 axis is taken as Da1. This is equal to the distance from the imaging unit 110 to the representative point 510a in the plan view of the monitoring area S1 seen from the +Z direction, as shown in Fig. 3(b).

[0078] The coordinates of the photographing unit 110 and the depth measuring unit 120 in the XYZ coordinate system are (x0, y0, z0), and the coordinates of the representative point 510a are (xa, ya, za). In this case, xa=x0+Da1×sinα1, and ya=y0+Da1×cosα1. Here, since Da1=Da×cosβ1, xa=x0+Da×sinα1×cosβ1, and ya=y0+Da×cosα1×sinβ1. Also, za=z0-Da×cosβ1.

[0079] Based on a similar concept, when the coordinates of representative points 510b and 510c are (xb, yb, zb) and (xc, yc, zc), xb=x0+Db×sinα2×cosβ2 and yb=y0+Db×cosα2×sinβ2. Furthermore, xc=x0+Dc×sinα3×cosβ3 and yc=y0+Dc×cosα3×sinβ3. Furthermore, zb=z0-Db×cosβ2 and zc=z0-Dc×cosβ3. When multiple representative points are provided in the person area or when spatial coordinates are obtained for each pixel in the person area, the above-described method of calculating the representative points can be repeated to calculate the desired coordinates.

[0080] [9] Person distance calculation section The person distance calculation unit 290 calculates the distance between each person based on the information of the spatial coordinates of the person area of ​​each person calculated by the person area spatial coordinate calculation unit 280. The distance between people is calculated based on the number of people calculated by the person number calculation unit 240, and is calculated for each combination of two people selected from all people.

[0081] For example, since the number of people is three in this embodiment, three distances are calculated as combinations of selecting two people from three people: between the people 500a and 500b, between the people 500a and 500c, and between the people 500b and 500c.

[0082] FIG. 5(b) is a schematic perspective view of the monitoring area S1 corresponding to FIG. 1 and FIG. 5(a), where the distance between person 500a and person 500b is Dab, the distance between person 500a and person 500c is Dac, and the distance between person 500b and person 500c is Dbc. These distances are also referred to as second distances. In this embodiment, the inter-person distance is the distance between the representative points of each person. Here, the spatial coordinates of each person are calculated by the person area spatial coordinate calculation unit 280 as described above.

[0083] From this, the distance between each person can be easily calculated. That is, Dab = ((xa-xb) 2 +(ya-yb) 2 +(za-zb) 2 )1 / 2 Similarly, Dac=((xa-xc) 2 +(ya-yc) 2 +(za-zc) 2 ) 1 / 2 and Dbc=((xb-xc) 2 +(yb-yc) 2 +(zb-zc) 2 ) 1 / 2 It is.

[0084] If the depth measurement unit 120 can measure the shortest distance from the image plane of the image capture unit 110 or a plane corresponding to the image plane to each person, the first distance and the second distance are calculated as follows. Kinect (registered trademark) is an example of a depth measurement unit 120 that employs such a method, and the principle of the depth measurement is described in detail in "Information Processing Society of Japan Research Report Vol. 2014-CVIM-192 No. 22" and the like.

[0085] Here, FIG. 6(a) is a diagram showing the same image 420 as FIG. 3(a), with the origin at the upper left corner of the image 420, the U axis in the horizontal direction, and the V axis in the vertical direction. The UV plane is a plane that includes the image surface of the image 420. The right direction of the U axis is the +U direction, the left direction is the -U direction, the downward direction of the V axis is the +V direction, and the upward direction is the -V direction. The coordinate system that displays the image surface of the image 420 composed of the U axis and the V axis is also called a screen coordinate system. Furthermore, the Zuv axis is taken in a direction perpendicular to the UV plane, the direction from the image surface of the image 420 toward the object is the +Zuv direction, and the opposite direction is the +Zuv direction. The Zuv axis is approximately the same as the optical axis direction of the infrared radiation of the depth measurement unit 120. The UVZuv coordinate system can be said to be a coordinate system that is set relatively depending on the mounting position of the human measurement device 10.

[0086] 6(b) is a plan view of the monitoring area S1 corresponding to the image 420 viewed from the -V direction. Here, the distances from the plane T1 including the image plane of the image 420 of the photographing unit 110 and the depth measuring unit 120 to the representative points 510a, 510b, and 510c of the people 500a, 500b, and 500c are Da2, Db2, and Dc2, respectively. These distances can also be called the first distance. First, the spatial coordinates of the representative point 510a of the person 500a are calculated.

[0087] The coordinates of the reference point of the photographing unit 110 and the depth measuring unit 120 in the UVZuv coordinate system are (u0, v0, zuv0), and the coordinates of the representative point 510a are (ua, va, zuva). At this time, the above-mentioned reference point is the intersection of the U axis and the V axis, i.e., the origin, and the distance from the origin along the U axis to the representative point 510a of the person 500a in the image 420 is Dua1, and the distance from the origin along the V axis is Dva1. At this time, ua=u0+Dua1, and va=v0+Dva1. Also, zuva=zuv0+Da2.

[0088] Furthermore, if the coordinates of representative point 510b of person 500b are (ub, vb, zuvb), then the distance from the origin along the U axis to representative point 510b is Dub1, the distance from the origin along the V axis is Dvb1, and so ub=u0+Dub1, vb=v0+Dvb1, and zuvb=zuv0+Db2. Similarly, if the coordinates of representative point 510c of person 500c are (uc, vc, zuvc), then the distance from the origin along the U axis to representative point 510c is Duc1, the distance from the origin along the V axis is Dvc1, and so uc=u0+Duc1, vc=v0+Dvc1, and zuvc=zuv0+Dc2.

[0089] In addition, since the UVZuv coordinate system is merely the aforementioned XYZ coordinate system rotated by a predetermined angle and translated in parallel, it goes without saying that the above coordinate values ​​can be converted into coordinate values ​​of the XYZ coordinate system as appropriate by performing a predetermined transformation.

[0090]

[10] Monitoring department The monitoring unit 200 acquires information on the number of people calculated by the person number calculation unit 240 and information on the distance between people calculated by the person distance calculation unit 290, judges whether or not these satisfy the set predetermined conditions, and instructs the operation / display device 310 and the alarm device 340 described later to display or warn as necessary.

[0091] The predetermined condition may be some kind of judgment algorithm, such as a formula, stored as an application program or parameter data in the monitoring unit 200 itself or in the storage unit 295, etc. Also, judgment contents corresponding to the number of people and the distance between people may be stored as a data table in the storage unit 295, etc., and judgment may be made by comparing this with the actual state.

[0092] The predetermined condition may be set arbitrarily, for example, the allowable range of the number of people and the allowable range of the distance between people may be set as independent allowable ranges, and if either one of the conditions exceeds the allowable range, it may be determined that the predetermined condition is not met. In one example, if the allowable range of the number of people is set to 5 or less and the allowable range of the distance between people is set to 1 meter or more, it is determined that the predetermined condition is not met if the number of people in the monitoring area S1 is 6 or more, or if any of the distances between people is less than 1 meter.

[0093] Also, an allowable range can be set that correlates the number of people with the distance between people. For example, the allowable range of the number of people can be set to 8 or less, and the allowable range of the distance between people can be changed depending on the number of people, so that when the number of people is 3 or less, the allowable range is 1 m or more, when the number is 4 to 6 or less, the allowable range is 1.5 m or more, and when the number is 7 to 8 or less, the allowable range is 2 m or more. Alternatively, the allowable range can be set to the number of combinations of people corresponding to the distance between people, such as the number of combinations where the distance between people is 1 m or more and less than 1.5 m is 2 or less, the number of combinations where the distance between people is 1.5 m or more and less than 2 m is 3 or less, and the number of combinations where the distance between people is 2 m or more and less than 2.5 m is 5 or less.

[0094] In addition, application programs and data such as algorithms and reference data tables for determining whether or not the specified conditions are satisfied by the monitoring unit 200 may be stored in the built-in memory of the monitoring unit 200 itself or in a storage unit 295 provided in the information processing device 20.

[0095] Also, similarly to the person region identification unit 230, all or part of these may be stored in a removable memory element such as a USB memory, an SD (registered trademark) card, a CD-ROM, a CD-R, or an IC card. Furthermore, all or part of these may be stored in a storage unit fixed to the monitoring unit 200 or the like, and the information may be rewritable at any time by wired or wireless communication from an external device.

[0096] By doing so, similarly to the person region identification unit 230, the algorithms and data used by the monitoring unit 200 can be updated quickly and easily, enabling more accurate condition determination and monitoring.

[0097] [iii] Operation / display device The operation / display device 310 includes an operation unit 320 that receives an instruction to start measurement by the person measurement device 10 and an instruction to set or change the judgment conditions of the monitoring unit 200, and a display unit 330 that displays the image captured by the imaging unit 110, the calculated number of people, the distance between people, and other values ​​on a screen. The operation unit 320 and the display unit 330 may be configured as separate devices, or may be configured as an integrated device that combines both. Examples of the latter include the notebook personal computer 410 of this embodiment, and also mobile terminals such as smartphones.

[0098] The display unit 330 is configured with a display such as a liquid crystal display, an organic electroluminescence display, an electronic paper display, etc. The operation unit 320 is configured with input devices such as a keyboard, a mouse, a touch pen, a touch panel, etc.

[0099] [iv] Alarm device The alarm device 340 has an audio output function such as a buzzer, siren, music, or a predetermined message. The alarm device 340 may turn on a warning light or output an alarm display either in synchronization with the display unit 330 or independently. The alarm device 340 can be used, for example, to notify relevant persons that a predetermined condition is no longer satisfied in relation to the number of people and the distance between people in the monitoring unit 200.

[0100] [b] Methods of Person Surveillance Next, a person monitoring method based on the person monitoring system 1 according to this embodiment will be described mainly with reference to FIG.

[0101] [i] Acquiring images and identifying areas and numbers of people First, the person measurement device 10 is started by operating the operation unit 320 of the operation / display device 310. As a result, the photographing unit 110 photographs the monitoring area S1, and the image information acquisition unit 220 of the information processing device 20 acquires a predetermined image 420 via the photographing unit interface 210 (step S801). Next, the person area identification unit 230 performs a predetermined process on the image 420 to identify a person area (step S802). In this embodiment, person areas 520a, 520b, and 520c are identified.

[0102] Furthermore, representative points 510a, 510b, and 510c are further specified as part of the person area. Meanwhile, from the result of specifying the person area and the representative points, the person number calculation unit 240 specifies the number of people (step S802). In this embodiment, the number of people is three. If the person area cannot be specified, it is determined that no person exists in the monitoring area S1, and the process returns to step S801 (step S803). Even if the person area can be specified, if the number of people is one, the distance between people cannot be measured, and the process also returns to step S801 (step S804).

[0103] [ii] Distance measurement from the depth measurement unit to the person area On the other hand, when a person area can be identified and the number of people is two or more, person direction identification unit 245 calculates the direction from imaging unit 110 to each of person areas 520a, 520b, and 520c from image 420. In this embodiment, the direction to each person area is calculated as the direction to each representative point 510a, 510b, and 510c.

[0104] Next, based on the directional information from the person direction identification unit 245, the person area distance calculation unit 270 instructs the depth measurement unit 120 to measure the distance to each person area in that direction. Then, the distance information acquisition unit 260 acquires the distance information via the depth measurement unit interface 250. The person area distance calculation unit 270 receives the information on the distance to each person area from the distance information acquisition unit 260 (step S805).

[0105] [iii] Calculation of spatial coordinates of the person area Next, the person area spatial coordinate calculation unit 280 calculates the spatial coordinates of each person area from the direction information from the person direction identification unit 245 and the distance information to each person area from the person area distance calculation unit 270 (step S806). In this embodiment, the spatial coordinates of the representative point of each person are calculated. Note that, as described above, if the depth measurement unit 120 can measure the shortest distance from the image plane of the imaging unit 110 or a plane corresponding to this to each person, the direction information from the person direction identification unit 245 is not necessary.

[0106] [iv] Calculating the distance between each person Furthermore, the person distance calculation unit 290 calculates the distance between the person areas of each person based on the information of the spatial coordinates of each person area from the person area spatial coordinate calculation unit 280 (step S807). In this embodiment, the distance between the representative points of each person is calculated.

[0107] [v] Determination of whether or not a specified condition is met Next, the monitoring unit 200 judges whether or not the people present in the monitoring area S1 satisfy a predetermined condition based on the information on the number of people from the person number calculation unit 240 and the information on the distance between the person areas of each person from the person distance calculation unit 290 (step S808). If the predetermined condition is satisfied, for example, a display to that effect, that is, an OK display is given on the display unit 330 of the operation / display device 310 (step S809). However, if the predetermined condition is satisfied, the display on the display unit 330 may be omitted.

[0108] On the other hand, if the predetermined conditions are not met, for example, the display unit 330 displays a message to that effect, i.e., a "NG" message, and if necessary, the alarm device 340 may display an alarm or warning (step S810). After that, regardless of whether the predetermined conditions are met or not, the monitoring state continues, and the process returns to step S801. However, if an operation to stop monitoring is performed from the operation unit 320, the monitoring process ends (step S811).

[0109] [c] Person monitoring system according to the first embodiment As described above, the person monitoring system 1 according to the first embodiment monitors the number of people and the distance between people present in a predetermined monitoring area S1. The person monitoring system 1 includes a person measurement device 10 that includes an image capture unit 110 that captures an image 420 of a person, and a depth measurement unit 120 that measures the distance to an object. The person monitoring system 1 also includes an information processing device 20 that includes the following components:

[0110] That is, the information processing device 20 includes a person region identification unit 230 that identifies person regions 520a, 520b, and 520c corresponding to the people 500a, 500b, and 500c from the image 420, and a person number calculation unit 240 that calculates the number of people from the information on the person regions. Also, the information processing device 20 includes a person region distance calculation unit 270 that calculates first distances Da, Db, and Dc that are distances from the depth measurement unit 120 to the person region for each person, from the information on the person region and information on the distance measured by the depth measurement unit 120.

[0111] In addition, the system includes a person-to-person distance calculation unit 290 that calculates second distances Dab, Dac, and Dbc between the people when the number of people is two or more. Furthermore, the system includes a monitoring unit 200 that determines whether information on the number of people and information on the second distances satisfy a predetermined condition.

[0112] By providing the person monitoring system 1 with such a configuration, it becomes possible to monitor the number of people and the distance between people quickly and accurately with as simple a configuration as possible, regardless of whether the people in the specified monitoring area S1 are stationary or moving. Both the imaging unit 110 that captures images of people in the monitoring area S1 and the depth measurement unit 120 that measures the distance to the person area are capable of capturing images and measuring distances regardless of the state of the people.

[0113] In addition, the photographing unit 110 and the depth measuring unit 120, which are person measuring devices, can be placed in approximately the same position, and information on the direction to each person as seen from the photographing unit 110 can be directly instructed as information on the direction in which the depth measuring unit 120 should measure the distance. Even if correction is necessary, the influence of the correction can be minimized because the two are close to each other.

[0114] Furthermore, although distance measurement within the monitoring area S1 by the depth measurement unit 120 can be performed over the entire area, by measuring only the distance in the required direction based on the result of identifying the person area by the imaging unit 110, the measurement load on the depth measurement unit 120 can be reduced and the measurement time can be shortened.

[0115] In this way, only the person area is identified from the image captured by the photographing unit 110, and the depth measurement unit 120 performs distance measurement limited to the direction of the identified person area, thereby making the most of the characteristics of both functions and enabling the number of people to be identified quickly and effectively and the distance between people to be calculated.

[0116] Furthermore, by further providing the information processing device 20 with a person direction identification unit 245 that identifies the direction from the imaging unit 110 toward the person area from the image 420, it is possible to quantify the direction to each person as seen from the imaging unit 110, and it becomes easy for the depth measurement unit 120 to reflect this information in information on the direction in which the distance should be measured. As a result, it is possible to further reduce the measurement load of the depth measurement unit 120 and to further speed up the measurement time. Note that, as described above, if the depth measurement unit 120 can measure the shortest distance from the image plane of the imaging unit 110 or a plane corresponding to the image plane to each person, the person direction identification unit 245 is not necessary.

[0117] Moreover, the information processing device 20 further includes a person area spatial coordinate calculation unit 280 that calculates the spatial coordinates of people in the monitoring area S1, which makes it easier for the person distance calculation unit 290 to calculate the second distance, which is the distance between each person. Meanwhile, when the monitoring unit 200 determines that a predetermined condition is not satisfied, the information processing device 20 further includes an operation / display device 310 as a display device that displays that fact and an alarm device 340 that issues an audio alarm to that effect. This makes it possible to quickly notify or warn relevant parties when a predetermined condition is not satisfied, and to encourage them to take corrective measures.

[0118] In addition, the person area identification unit 230 is equipped with a learning model that has undergone machine learning to identify a person from the image 420 and identify the person area of ​​the person, thereby making it possible to identify the person area from the image 420 more quickly and accurately.

[0119] 2. Second embodiment Next, a second embodiment of the person monitoring system of the present disclosure will be described. Fig. 8 is a schematic diagram illustrating a person monitoring system 1a according to the second embodiment. The person monitoring system 1a differs from the person monitoring system 1 of the first embodiment in that the functions of the information processing device 20 are provided in a management server 610 connected to a communication network 600. In addition, an operation / display device 310 and an alarm device 340 are also provided as terminal devices 620, such as a notebook personal computer or a mobile terminal, connected to the communication network 600, and the person measurement device 10 is also connected to the communication network 600.

[0120] In this manner, in this embodiment, the information processing device 20, the operation / display device 310, and the alarm device 340 are placed in a location that is not necessarily close to the monitoring area in which the human measurement device 10 is placed, and monitoring and display of results, etc. can be performed by remote operation via the communication network 600.

[0121] For this reason, there is no need to associate one-to-one between the person measurement device 10 monitoring a specified monitoring area and the information processing device 20, etc., and, for example, a single management server 610 can acquire information from multiple person measurement devices 10 placed in multiple monitoring areas, thereby monitoring multiple monitoring areas in parallel.

[0122] This reduces the cost of monitoring each monitored area, and also reduces the labor and costs involved in maintenance work, since updates to the algorithms and various data of the person area identification unit 230 and the monitoring unit 200 can be completed by performing operations only on the management server 610. Furthermore, by managing such important processing algorithms and data in the cloud, the risk of important information leaking can be reduced.

[0123] 3. Third embodiment Next, a third embodiment of a person monitoring system of the present disclosure will be described. Fig. 9 is a schematic perspective view illustrating a person monitoring system 1b according to the third embodiment. Fig. 10 is a configuration diagram showing the configuration of the person monitoring system 1b. The person monitoring system 1b differs from the person monitoring system 1 of the first embodiment in that, as a person measurement device 10a, in addition to the photographing unit 110 and the depth measurement unit 120, it further includes a carbon dioxide concentration measurement unit 130 that measures the carbon dioxide concentration in the air inside the monitoring area S2.

[0124] The information processing device 20a of the person monitoring system 1b further includes a carbon dioxide concentration measurement unit interface 296 corresponding to the carbon dioxide concentration measurement unit 130 and a carbon dioxide concentration information acquisition unit 297.

[0125] 9, the carbon dioxide concentration measurement unit 130 may be placed at one predetermined location in the monitoring area S2, or multiple units may be placed, for example, at the four corners of the monitoring area S2. By placing multiple carbon dioxide concentration measurement units 130 in the monitoring area S2 in this way, the relationship between the location of each person and the carbon dioxide concentration can be more precisely determined, enabling accurate judgment.

[0126] The carbon dioxide concentration measurement unit 130 can be configured as an optical type that detects carbon dioxide using the absorption characteristics of infrared light, such as a non-dispersive infrared (NDIR) detection method. It can also be configured as various types such as an electrochemical type that detects carbon dioxide using an electrochemical reaction, or a semiconductor type that detects carbon dioxide using a semiconductor such as tin oxide. In particular, when the monitored area is a conference room or the like where the air is less polluted by dust, it is preferable to use a carbon dioxide concentration measurement unit of the NDIR detection method, which is excellent in measurement accuracy and cost.

[0127] The carbon dioxide concentration measurement unit interface 296 converts the output from the carbon dioxide concentration measurement unit 130 into a measured value of the carbon dioxide concentration in the air within the actual monitoring area S2. Furthermore, the carbon dioxide concentration information acquisition unit 297 acquires information on the measured carbon dioxide concentration value converted by the carbon dioxide concentration measurement unit interface 296. These components can also be realized by the same hardware or software configuration as the information processing device 20 described above.

[0128] Moreover, unlike the monitoring unit 200, the monitoring unit 200a acquires information on the carbon dioxide concentration measurement value in the monitoring area S2 from the carbon dioxide concentration information acquisition unit 297 described above, in addition to the information on the number of people from the person number calculation unit 240 and the information on the distance between people from the person distance calculation unit 290. Then, based on this information, it is determined whether or not a predetermined condition is satisfied. For example, the concentration ranges are set as ranks A, B, and C in order from the lowest concentration according to the carbon dioxide concentration in the monitoring area S2.

[0129] Here, in the case of rank A, the upper limit of the allowable range of the number of people is expanded in the larger direction, and the lower limit of the allowable range of the distance between people is set wider in the shorter direction. Also, in the case of rank C, the upper limit of the allowable range of the number of people is reduced in the smaller direction, and the lower limit of the allowable range of the distance between people is reduced in the longer direction. In this way, by changing the allowable range of the number of people and the distance between people according to the carbon dioxide concentration state in the monitoring area S2, it becomes possible to monitor people while taking into account the degree of ventilation of the air in the monitoring area S2 and the degree of sealing of the monitoring area S2.

[0130] In particular, when the purpose of monitoring the monitoring area S2 is to prevent infectious diseases related to the degree of crowding of people and the degree of sealing of the monitoring area, more effective person monitoring is possible. Also, an allowable range may be set only for the carbon dioxide concentration, regardless of the number of people or the distance between people.

[0131] The person monitoring method based on person monitoring system 1b according to this embodiment will be described focusing on the differences from embodiment 1. Person monitoring system 1b may measure the carbon dioxide concentration in monitoring area S2 by carbon dioxide concentration measurement unit 130 after or before specifying the person area and the number of people in step S802 in Fig. 7.

[0132] If the monitoring unit 200a determines that this does not satisfy the predetermined condition, the process proceeds to step S810, and the display unit 330 may display NG, or the alarm device 340 may issue a warning. In addition, when determining in step S808 whether the number of people and the distance between people satisfy the predetermined condition, the monitoring unit 200a may also make the determination by taking into account information on the carbon dioxide concentration.

[0133] 4. Fourth embodiment Next, a fourth embodiment of the person monitoring system of the present disclosure will be described. Fig. 11(a) is a schematic perspective view for explaining a person monitoring system 1c according to the fourth embodiment. Fig. 11(b) is a diagram showing an image 421 captured by the image capture unit 110 of the monitoring area S3 shown in Fig. 11(a).

[0134] Based on an image 421 captured by the imaging unit 110 of the person measurement device 10, the person monitoring system 1c calculates the inter-person distance between two people 500d and 500e identified from the image 421 through steps similar to those of the person monitoring system 1 of the first embodiment. Specifically, the distance Dde between representative points 510d and 510e of each person is calculated as the distance between person areas 520d and 520e.

[0135] Here, in the monitoring area S3, a partition 430 is placed between two people 500d and 500e. In this way, when a non-human obstruction 430 is placed between people, even if the distance between the people does not satisfy a predetermined condition, that is, even if the people are closer than the allowable range, it may be preferable to exclude the information on the distance and make a judgment.

[0136] In this case, the information on the distance is not taken into consideration, and it is determined that the predetermined condition is met based on information on the distance between other people, etc. For example, such a determination may be effective when the purpose of the person monitoring system is to prevent droplet infection.

[0137] In this manner, in the present embodiment, even if the monitoring unit determines that the condition of the person-to-person distance does not satisfy the predetermined condition in the first embodiment, a different determination is made when some obstruction 430 other than a person is detected between the people. That is, by excluding information on the person-to-person distance in which the obstruction 430 is detected, and determining whether or not the predetermined condition is satisfied based on information on other person-to-person distances, etc., a more accurate determination suited to the purpose of monitoring a specific person becomes possible.

[0138] Various methods are used to determine whether or not there is an obstruction between the people. For example, in the image 421, the person area identification unit 230 identifies representative points as part of the person areas 520d and 520e, and also identifies representative points 510d and 510e. In addition, the person area distance calculation unit 270 calculates distances Dd and De from the depth measurement unit 120 to the representative points 510d and 510e.

[0139] Here, the information processing device 20 instructs the depth measurement unit 120 and the distance information acquisition unit 260 to measure and calculate the distance from the depth measurement unit 120 to a specified point on the line segment connecting both representative points of the image 421, for example, using an inter-person obstruction confirmation unit (not shown).

[0140] Here, the inter-person obstruction confirmation unit can determine that an obstruction exists between the two people if the distance value from the depth measurement unit 120 to a predetermined location is smaller than both Dd and De. For example, on the line segment connecting the representative points 510d and 510e in the image 421, there is an obstruction point 530 in the area of ​​the obstruction 430 whose distance from the depth measurement unit 120 is smaller than both Dd and De.

[0141] The method of checking the presence of an obstruction between people is not limited to this, and for example, an image that can be considered as an obstruction may be registered in advance as a reference pattern, and whether or not something similar to this exists between people may be searched for by successive comparison type pattern matching. Alternatively, a learning model that has been machine-learned to identify a predetermined obstruction from a captured image and specify the area of ​​the obstruction may be provided.

[0142] In other words, the inter-person obstruction confirmation unit may be equipped with a learning model that has undergone machine learning to identify people from a captured image, specify the person's area, and identify obstructions 430 that exist between people and specify the area of ​​the obstructions 430.

[0143] For example, the inter-person obstruction confirmation unit includes a learning model generated as a result of machine learning using a pattern matching or deep learning method with a specific obstruction image having a candidate obstruction area as training data. The learning model is configured to input specific image information and output information on the obstruction area identified in the image.

[0144] The distance to the obstruction may be calculated by specifying the obstruction area using the above-mentioned method and then determining a representative point.

[0145] 5. Fifth embodiment Next, a fifth embodiment of the person monitoring system of the present disclosure will be described. The person monitoring system of this embodiment further includes an identification unit (not shown) that identifies each person corresponding to the person area identified by the person area identification unit 230.

[0146] The information processing device 20 also has a memory unit 295, and the information processing device 20 stores information on the second distance corresponding to each person in the memory unit 295 in association with information on the time when the depth measurement unit 120 measured the distance corresponding to the first distance.

[0147] That is, in the first embodiment, a unique person ID is assigned to each of the persons 500a, 500b, and 500c whose person regions are identified from the image 420, and the persons are linked to the same persons identified from other images repeatedly captured by the image capturing unit 110. This makes it possible to monitor the inter-person distance over time for specific persons having the same person ID.

[0148] The person monitoring system of this embodiment stores a predetermined information table in the storage unit 295. An example of such an information table 700 is shown in Fig. 12. For the "person ID," a person ID that can uniquely identify the person is generated by a person ID identification unit (not shown) based on the person area of ​​each person identified by the person area identification unit 230, and if a person ID has not yet been assigned to the person, the person ID identification unit (not shown) generates a person ID that can uniquely identify the person and registers it in the information table 700.

[0149] The person ID identification unit then records the number of people and the inter-person distance linked to the person ID in the information table 700. For example, for a person with a person ID of "001", the shooting date and time of the image for which the number of people and the inter-person distance were calculated are recorded in the "Date" column and the "Time" column of "Date and Time" in the information table 700. Furthermore, information on the number of people at that time and the inter-person distance between the person and other people based on the person is recorded in the same row as the person ID in the information table 700.

[0150] In addition, the time when the depth measurement unit 120 measures the distance corresponding to the first distance is not limited to the time when the depth measurement unit 120 actually outputs the measurement result of a specified distance, but also includes a time slightly before or after the actual measurement time, taking into account the time required for processing such as identifying the person area or representative point, correcting and converting the measurement value, etc.

[0151] For example, the predetermined conditions determined by the monitoring unit 200 are that the allowable range for the number of people is 5 or less, and the allowable range for the distance between people is 1 meter or more. In this case, it is determined that the person with the person ID "001" satisfies the predetermined conditions, but the people with the person IDs "002" and "003" do not satisfy the predetermined conditions.

[0152] The distance between person ID "001" and person ID "002" is "3.56m", and the distance between person ID "001" and person ID "003" is "2.87m", both of which are within the allowable range. On the other hand, the distance between person ID "002" and person ID "003" is "0.97m", which is outside the allowable range.

[0153] In this manner, by recording information in the information table 700 on an ongoing basis, it becomes possible to look back at the history of a particular person identified by a person ID and examine at what time in the past the number of people or the distance between people did not satisfy the specified conditions.

[0154] Note that various methods including the following methods can be used as an identification means for identifying each person corresponding to a person region identified by the person region identification unit 230. For example, the person region identification unit 230 may be provided with a biometric authentication algorithm including face recognition as hardware or software in addition to identifying the person region, and may identify an individual through predetermined feature extraction. In this case, the person region identification unit 230 equipped with a biometric authentication function serves as the identification means.

[0155] In addition, when the people who may be present in the monitored area are limited, such as limited to specific employees, images of candidate people may be registered in advance as reference patterns, and the person whose person area has been identified may be identified by pattern matching with the reference pattern. In this case, too, the person area identification unit 230 equipped with a pattern matching function serves as the identification means.

[0156] Alternatively, the person area identification unit 230 may be provided with a learning model that has performed machine learning to determine whether a person corresponding to a person area is the same as a person previously identified or a different person.

[0157] For example, the person region identification unit 230 is provided with a learning model generated as a result of machine learning using a pattern matching or deep learning technique with multiple different images of the same person as training data. In this case as well, the person region identification unit 230 provided with the learning model generated by machine learning serves as the identification means.

[0158] At this time, the person area identification unit 230 identifies a person from the captured image, identifies the person's person area, and performs machine learning to determine whether or not the person is a person to whom a person ID has been assigned in the past. For example, the person area identification unit 230 is equipped with a learning model that is generated as a result of machine learning using a pattern matching or deep learning method with images of multiple candidate people as training data.

[0159] This learning model inputs image information including specific people, and if the person identified in the image is a person to whom a person ID has been assigned in the past, it outputs information about that assigned person ID, and if the identified person is a person to whom a person ID has not been assigned in the past, it outputs this fact.

[0160] On the other hand, if the people who may be present in the monitored area are limited, information from contactless IC cards or other devices carried by each person can be read at any time by a reader / writer located within the monitored area, and the person in question can be identified from the unique information recorded on the contactless IC card.

[0161] Also, each person in the monitored area may wear work clothes, a helmet, or the like, with an identification code such as a two-dimensional barcode showing unique information, and the person may be identified by photographing and decoding the image by the photographing unit 110. In this case, the reader / writer that reads information from a non-contact IC card, etc., and the person area identifying unit 230 that decodes the two-dimensional barcode from the acquired image serve as the identification means.

[0162] The control processing and other processing of the information processing device according to each embodiment of the present disclosure described above can also be achieved by the information processing device reading and executing program code constituting software for realizing each function of the embodiment.

[0163] Therefore, the program code itself and the storage medium on which the program code is recorded also constitute the present disclosure. Examples of storage media on which the program code is recorded include, but are not limited to, a hard disk, a USB memory, an SD (registered trademark) card, a CD-ROM, a CD-R, and an IC card.

[0164] The person monitoring system of the present disclosure can be configured as another embodiment by combining some or all of the aspects of the first to fifth embodiments as long as no contradiction occurs, and these are naturally included in the present disclosure. For example, a person monitoring system that combines the second and third embodiments, includes a carbon dioxide concentration measurement unit, and has an information processing device as a management server, also falls under this category.

[0165] Also included is a person monitoring system that combines the second, fourth and fifth embodiments, performs monitoring taking into account obstructions between people, has an identification means for identifying each person, and has an information processing device as a management server. [Explanation of symbols]

[0166] 1, 1a, 1b People Monitoring System 10, 10a Person measuring device 20, 20a Information processing device 110 Filming Department 120 Depth measurement section 130 Carbon dioxide concentration measurement unit 200, 200a monitoring section 210 Camera Interface 220 Image information acquisition unit 230 Human area identification part 240 Number of people calculation section 245 Person direction identification part 250 Depth Measuring Unit Interface 260 Distance information acquisition section 270 Person area distance calculation unit 280 Human region spatial coordinate calculation unit 290 Person Distance Calculation Unit 295 Memory section 296 Carbon Dioxide Concentration Measurement Unit Interface 297 Carbon Dioxide Concentration Information Acquisition Unit 310 Operation / display device 320 Control unit 330 Display section 340 Alarm device 400 racks 410 Personal Computers 420, 421 images 430 Shield 500a, 500b, 500c, 500d, 500e People 510a, 510b, 510c, 510d, 510e Representative points 520a, 520b, 520c, 520d, 520e People area 530 Shield point 600 Communication Network 610 Management Server 620 Terminal Equipment 700 Information Table

Claims

1. A person monitoring system for monitoring the number of people and the distance between people within a predetermined monitoring area, comprising: An image capturing unit that captures an image of the person; A person measuring device including a depth measuring unit that measures a distance to an object; and a person area identifying unit that identifies a person area corresponding to the person from the image; a person number calculation unit that calculates the number of people from the information of the person area; a person area distance calculation unit that calculates a first distance, which is a distance from the depth measurement unit to the person area for each person, based on information about the person area and information about the distance measured by the depth measurement unit; a person-to-person distance calculation unit that calculates a second distance between the people when the number of people is two or more; a monitoring unit that determines whether the information on the number of people and the information on the second distance satisfy a predetermined condition, The human measurement device further includes a carbon dioxide concentration measurement unit for measuring a carbon dioxide concentration in the air within the monitoring area, The monitoring unit determines whether or not information on the carbon dioxide concentration measured by the carbon dioxide concentration measuring unit satisfies the predetermined condition, in addition to the information on the number of people and the information on the second distance.

2. A person monitoring system for monitoring the number of people and the distance between people within a predetermined monitoring area, comprising: An image capturing unit that captures an image of the person; A person measuring device including a depth measuring unit that measures a distance to an object; and a person area identifying unit that identifies a person area corresponding to the person from the image; a person number calculation unit that calculates the number of people from the information of the person area; a person area distance calculation unit that calculates a first distance, which is a distance from the depth measurement unit to the person area for each person, based on information about the person area and information about the distance measured by the depth measurement unit; a person-to-person distance calculation unit that calculates a second distance between the people when the number of people is two or more; a monitoring unit that determines whether the information on the number of people and the information on the second distance satisfy a predetermined condition, When the number of persons is two or more, and the person area identification unit has identified a first person area corresponding to a first person and a second person area corresponding to a second person as the person areas corresponding to the persons, A person monitoring system in which, when it is determined from the distance information measured by the depth measurement unit that there is an obstruction other than a person between the first person area and the second person area, the monitoring unit excludes information on the distance between the first person area and the second person area from the second distance information and determines whether the specified condition is satisfied.

3. A person monitoring system for monitoring the number of people and the distance between people within a predetermined monitoring area, comprising: An image capturing unit that captures an image of the person; A person measuring device including a depth measuring unit that measures a distance to an object; and a person area identifying unit that identifies a person area corresponding to the person from the image; a person number calculation unit that calculates the number of people from the information of the person area; a person area distance calculation unit that calculates a first distance, which is a distance from the depth measurement unit to the person area for each person, based on information about the person area and information about the distance measured by the depth measurement unit; a person-to-person distance calculation unit that calculates a second distance between the people when the number of people is two or more; a monitoring unit that determines whether the information on the number of people and the information on the second distance satisfy a predetermined condition, an identification unit configured to identify each of the people corresponding to the person areas identified by the person area identification unit, The information processing device further includes a storage unit, The information processing device stores information on the second distance corresponding to each of the persons in the memory unit in association with information on the time when the depth measurement unit measured the distance corresponding to the first distance.

4. 4. The person monitoring system according to claim 1, wherein the information processing device further comprises a person area spatial coordinate calculation unit that calculates spatial coordinates of the person in the monitoring area, and the person distance calculation unit calculates the second distance based on information of the spatial coordinates.

5. 5. The human monitoring system according to claim 1, further comprising a display device that displays a message indicating that the monitoring unit does not satisfy the predetermined condition, or an alarm device that issues an audio warning to that effect, when the monitoring unit determines that the predetermined condition is not satisfied.

6. The person monitoring system according to claim 1 , wherein the person area identification unit is provided with a learning model that has undergone machine learning to identify the person from the image and identify the person area of ​​the person.

Images