Image analysis device and monitoring system

Abstract

To allow for grasping the state where people in a monitoring space cannot be correctly captured in a captured image.SOLUTION: An image analysis device 10 disclosed herein comprises people information acquisition unit configured to analyze a captured image of inside a vehicle cabin, designated as a monitoring space, acquired by a camera to acquire information on people captured in the captured image, or vehicle occupant information on vehicle occupants. The imager analysis device 10 also comprises a hidden state detection unit 40 configured to detect a hidden state where multiple occupants appear on top of each other in the captured image.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an image analysis device and a monitoring system. [Background technology]

[0002] Conventionally, there is an image analysis device that acquires information about people appearing in a captured image by analyzing an image of a monitored space captured by a camera. For example, Patent Document 1 discloses a configuration in which the cabin of a vehicle is used as a monitored space and the posture and physique of occupants appearing in the captured image by the camera are detected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-104680 Summary of the Invention [Problem to be solved by the invention]

[0004] However, depending on the positional relationship between the camera and the person, the person in the monitored space may not be captured correctly in the captured image, which may hinder the acquisition of that information. [Means for solving the problem]

[0005] The image analysis device that solves the above problem includes a person information acquisition unit that acquires information about people appearing in the captured image by analyzing the captured image of the monitored space captured by the camera, and a hidden state detection unit that detects the occurrence of a hidden state in which multiple people appear overlapping in the captured image.

[0006] According to the above configuration, it is possible to detect an occlusion state occurring in a captured image and understand that a person in the monitored space is not properly captured in the captured image. In this case, for example, by refraining from analytically using the captured image in which the occlusion state occurred, it is possible to ensure highly accurate information acquisition.

[0007] An image analysis device that solves the above problem includes a movement prediction unit that predicts the movement of each person appearing in the captured image, and an overlap rate calculation unit that calculates the overlap rate of the image capture area for each person that is estimated to appear overlapping based on the movement prediction, and it is preferable that the hidden state detection unit detects and judges the hidden state based on a comparison between the overlap rate and an overlap judgment value.

[0008] According to the above configuration, it is possible to accurately detect the occurrence of an occlusion state in a captured image. Furthermore, there is an advantage that the occurrence of an occlusion state can be predicted in advance. This allows for more accurate information acquisition.

[0009] An image analysis device that solves the above problem preferably includes an entry / exit number measurement unit that measures the number of people entering and exiting the monitored space, and the hidden state detection unit performs a detection judgment of the hidden state based on the difference between the total number of people in the monitored space identified by measuring the number of people entering and exiting and the detected number of people captured in the captured image.

[0010] In other words, if there is no occlusion in the captured image, the number of people detected in the captured image will be equal to the total number of people in the monitored space. Therefore, with the above configuration, it is possible to easily detect occlusion in the captured image with a simple configuration.

[0011] In an image analysis device that solves the above problem, it is preferable that the hidden state detection unit determines that the hidden state has occurred when the person in the captured image is detected at a hidden position close to the camera.

[0012] That is, a person in a hidden position close to the camera may obstruct the camera's field of view, thereby obscuring another person in the monitored space, creating an obscured state in the captured image. In this case, it is impossible to determine whether the obscured state actually occurs. In light of this, the above configuration assumes that an obscured state has occurred. In this case, for example, by refraining from analytically using the captured image in which the obscured state has occurred, it is possible to ensure highly accurate information acquisition.

[0013] An image analysis device that solves the above problem preferably includes an image ratio calculation unit that calculates the proportion of the person's image capture area to the entire captured image as an image ratio, and a camera proximity position determination unit that determines that the person is captured in the captured image at the hidden position when the person is detected with the image ratio equal to or greater than a predetermined proximity determination value.

[0014] According to the above configuration, a person who appears in a captured image at a hidden position close to the camera can be detected with a simple configuration. It is preferable that an image analysis device that solves the above problem includes a skeleton point detection unit that detects the skeleton points of the person included in the captured image, and an abnormality detection unit that detects abnormalities that have occurred in the monitored space based on the information about the person obtained by detecting the skeleton points.

[0015] That is, by detecting the skeleton points, it is possible to acquire physical information such as a person's posture and physique with high accuracy. This allows for highly accurate detection and determination of abnormalities in the monitored space based on the acquired information about the person. However, when an occlusion state occurs in the captured image, the detection state of the skeleton points also deteriorates. As a result, there is a possibility that the abnormality detection and determination cannot be performed with high accuracy. Therefore, by applying the detection and determination of the occlusion state shown in any of the above configurations to such a configuration, more significant effects can be obtained.

[0016] It is preferable that the image analysis device for solving the above problem includes a judgment output unit that, when the occurrence of the hidden state is detected, executes a judgment output regarding the detection state of the skeleton points in the captured image in which the occurrence of the hidden state is detected as the detection output of the hidden state.

[0017] According to the above configuration, it is possible to correctly grasp the detection state of skeleton points that has changed due to the occurrence of an occlusion state, and thereby to appropriately use the captured image in which the occlusion state has occurred.

[0018] In the image analysis device for solving the above problem, it is preferable that the monitored space is a passenger compartment of a vehicle, and the person is an occupant of the vehicle. According to the above configuration, highly accurate information about the occupants in the vehicle compartment can be obtained.

[0019] A surveillance system that solves the above problem includes any of the image analysis devices described above. [Effects of the Invention]

[0020] According to the present invention, it is possible to know when a person in a monitored space is not properly captured in a captured image. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a perspective view of a vehicle to which a monitoring system is applied. [Figure 2] 1 is an explanatory diagram of an occupant in a vehicle cabin and a camera that photographs the occupant. [Figure 3] FIG. 2 is an explanatory diagram of a vehicle interior as seen from above. [Figure 4] FIG. 1 is a block diagram showing a schematic configuration of an image analysis device. [Figure 5] FIG. 1 is an explanatory diagram showing human skeletal points. [Figure 6] FIG. 1 is a schematic configuration diagram of a monitoring system. [Figure 7] FIG. 2 is a block diagram showing a schematic configuration of a hidden state detection unit. [Figure 8]FIG. 10 is an explanatory diagram of occupant movement prediction and overlap rate calculation. [Figure 9] 10 is a flowchart showing a processing procedure for detecting a hidden state based on an overlap rate determination. [Figure 10] FIG. [Figure 11] An explanatory diagram of disembarkation detection. [Figure 12] 10 is a flowchart showing a processing procedure for measuring the number of passengers getting on and off. [Figure 13] 10 is a flowchart showing a processing procedure for detecting a hidden state based on a difference in the number of people determination. [Figure 14] An illustration of a passenger appearing in an image captured from a hidden position close to the camera. [Figure 15] 10 is a flowchart showing a processing procedure for detecting a hidden state based on a camera proximity position determination. [Figure 16] 10 is a flowchart showing a processing procedure for hidden state detection and anomaly detection. [Figure 17] 10 is a flowchart showing a processing procedure for outputting a determination regarding the detection state of a skeleton point. DETAILED DESCRIPTION OF THE INVENTION

[0022] An embodiment of an image analysis device and a monitoring system will be described below with reference to the drawings. As shown in FIGS. 1 to 3, a vehicle 1 of this embodiment has a vehicle body 2 that is substantially in the shape of a rectangular box and extends in the fore-and-aft direction of the vehicle. A door opening 3 is provided on the side of the vehicle body 2, through which passengers can get in and out. This door opening 3 is provided with a pair of sliding doors 4, 4 that open and close in opposite directions in the fore-and-aft direction of the vehicle. An occupant 5 of the vehicle 1 gets into the vehicle 1 in a "seated position" in which they are seated in a seat 7 provided in a passenger compartment 6, or in a "standing position" by using, for example, a strap or handrail (not shown).

[0023] The vehicle 1 of this embodiment is also provided with a camera 8 that photographs the interior of the passenger compartment 6. In the vehicle 1 of this embodiment, the camera 8 is provided in the vicinity of the ceiling 9 near a corner 6fa at a front position of the passenger compartment 6. Note that the camera 8 may be, for example, an infrared camera. The camera 8 of this embodiment is configured to photograph the occupant 5 of the vehicle 1 from a predetermined direction set in the passenger compartment 6.

[0024] 4, in the vehicle 1 of this embodiment, a captured image Vd of the interior of the vehicle cabin 6 captured by the camera 8 is input to an image analysis device 10. Furthermore, the image analysis device 10 has a function of monitoring the state of the interior of the vehicle cabin 6 captured in the captured image Vd by analyzing the captured image Vd. As a result, a monitoring system 15 is constructed in the vehicle 1 of this embodiment, in which the vehicle cabin 6 captured by the camera 8 serves as a monitored space 11.

[0025] More specifically, the image analysis device 10 of this embodiment includes an image analysis unit 20 and a person recognition unit 21 that recognizes a person H in the vehicle interior 6 that appears in the captured image Vd, i.e., an occupant 5 of the vehicle 1, based on the results of image analysis by the image analysis unit 20. In the image analysis device 10 of this embodiment, the person recognition unit 21 executes a recognition process for the person H by using an inference model generated by machine learning. The image analysis device 10 of this embodiment also includes an abnormality detection unit 22 that monitors the occupant 5 of the vehicle 1 recognized thereby, and detects any abnormality that has occurred in the vehicle interior 6 captured by the camera 8.

[0026] 4 and 5, the image analyzing device 10 of this embodiment includes a skeleton point detection unit 23 that detects skeleton points SP of a person H included in a captured image Vd. That is, the skeleton points SP are unique points that characterize the body of the person H, such as joints and points on the body surface, and include, for example, the head, neck, shoulders, armpits, elbows, wrists, fingers, waist, hip joints, buttocks, knees, and ankles. In the image analyzing device 10 of this embodiment, the skeleton point detection unit 23 also performs a process of detecting the skeleton points SP by using an inference model generated by machine learning.

[0027] 4, the image analyzing device 10 of this embodiment also includes a feature amount calculation unit 24 that calculates feature amounts Vsp based on the detection of the skeleton points SP. Specifically, in the image analyzing device 10 of this embodiment, the feature amount calculation unit 24 calculates feature amounts Vsp of the person H appearing in the captured image Vd based on the two-dimensional coordinate positions of the skeleton points SP in the captured image Vd. Furthermore, the feature amount calculation unit 24 calculates the feature amounts Vsp of the person H appearing in the captured image Vd based on the body dimensions indicated by the multiple skeleton points SP, such as the shoulder width of the occupant 5. The image analyzing device 10 of this embodiment also includes a person information acquisition unit 25 that acquires information Ih of the person H recognized in the monitored space 11 captured by the camera 8, based on the feature amounts Vsp of the person H obtained by this series of analysis processes.

[0028] More specifically, the person information acquisition unit 25 of this embodiment includes a posture determination unit 26 that determines the posture of the person H appearing in the captured image Vd. The posture determination unit 26 of this embodiment inputs the feature amounts Vsp of the person H acquired from the feature amount calculation unit 24 into an inference model generated by machine learning. Then, based on the posture determination probability value thus obtained, the posture of the person H appearing in the captured image Vd inside the vehicle interior 6 is determined.

[0029] Specifically, the posture determination unit 26 of this embodiment includes a standing position determination probability value calculation unit 26a that calculates the probability that the posture of the occupant 5 who is the target of posture determination is a "standing position." The posture determination unit 26 also includes a sitting position determination probability value calculation unit 26b that calculates the probability that the posture of the target occupant 5 is a "sitting position." The posture determination unit 26 of this embodiment also includes a fall determination probability value calculation unit 26c that calculates the probability that the posture of the target occupant 5 is a "falling position."

[0030] That is, in the posture determination unit 26 of this embodiment, the standing position determination probability value calculation unit 26a calculates a standing position determination probability value XA, the sitting position determination probability value calculation unit 26b calculates a sitting position determination probability value XB, and the fall determination probability value calculation unit 26c calculates a fall determination probability value XC as posture determination probability values. Furthermore, the posture determination unit 26 of this embodiment calculates the posture determination probability values ​​so that the sum of the standing position determination probability value XA, the sitting position determination probability value XB, and the fall determination probability value XC becomes "1.0." This enables the posture determination unit 26 of this embodiment to consistently determine the posture of the occupant 5 based on the posture determination probability values.

[0031] The standing position discrimination probability value XA calculated by the standing position discrimination probability value calculation unit 26a of this embodiment is further divided into the probability that the occupant 5 in that "standing position" is in a "moving state," a "stationary state," and a "state using a strap, handrail, or the like." The posture determination unit 26 of this embodiment is thus configured to be able to subdivide and discriminate the "standing position."

[0032] In the image analyzing device 10 of this embodiment, when the posture determining unit 26 determines that the occupant 5 of the vehicle 1 has fallen, the abnormality detecting unit 22 determines that an abnormality has occurred in the vehicle interior 6 shown in the captured image Vd of the camera 8. The monitoring system 15 of this embodiment is thus configured to ensure safety inside the vehicle interior 6.

[0033] In the image analyzing device 10 of this embodiment, the person information acquiring unit 25 is provided with, in addition to the posture determining unit 26, an attribute determining unit 27 that determines the attributes of the person H appearing in the photographed image Vd, a physique determining unit 28 that determines the physique, etc. This enables the image analyzing device 10 of this embodiment to accurately detect the state of the person H appearing in the photographed image Vd.

[0034] 6, the monitoring system 15 of this embodiment is formed by interconnecting a plurality of information processing devices 30 arranged inside and outside the vehicle 1 via an information communication network (not shown). Specifically, the image analyzing device 10 of this embodiment is configured so that the image analysis processing is distributed between an on-board information processing device 30a mounted on the vehicle 1 and an off-vehicle information processing device 30b constituting a cloud server 31. The monitoring system 15 of this embodiment is thereby configured to ensure excellent mountability on the vehicle 1 by reducing the calculation load on the information processing device 30a mounted on the vehicle 1.

[0035] Furthermore, the monitoring system 15 of this embodiment is configured so that when an abnormality occurs, the captured image Vd of the interior of the vehicle compartment 6 captured by the camera 8 can be confirmed by a manager 35 outside the vehicle, such as an operator 33 stationed at the operation center 32 of the vehicle 1. This ensures high reliability and safety in the monitoring system 15 of this embodiment.

[0036] (Hidden state detection) Next, the hidden state detection function implemented in the image analysis device 10 of this embodiment will be described.

[0037] 4, the image analysis device 10 of this embodiment includes an obscured state detection unit 40 that detects the occurrence of a so-called obscured state in which multiple people H overlap in a captured image Vd, potentially hindering the acquisition of information Ih based on the image analysis. That is, when multiple occupants 5 are riding in the vehicle 1, an occupant 5 located farther from the camera 8 may be obscured by a closer occupant 5 in the captured image Vd of the vehicle interior 6 captured by the camera 8. As a result, the presence of the occupant 5 at the back who is obscured by the occupant 5 at the front may not be recognized, or the detection of the skeleton point SP may be incomplete, making it impossible to accurately acquire the occupant information Ich.

[0038] In consideration of this, in the image analyzing device 10 of this embodiment, the occurrence of such an obscured state is detected by the obscured state detection unit 40. The monitoring system 15 of this embodiment is configured to thereby be able to grasp that the occupant 5 in the vehicle compartment 6 is not properly captured in the captured image Vd.

[0039] 7, the hidden state detection unit 40 of this embodiment includes a first hidden state determination unit 41, a first hidden state determination unit 42, and a third hidden state determination unit 43. The first hidden state determination unit 41, the second hidden state determination unit 42, and the third hidden state determination unit 43 are configured to determine the hidden state occurring in the captured image Vd of the camera 8, respectively, using different methods.

[0040] (Overlap rate determination) First, the detection and determination of the hidden state based on the overlap rate determination executed by the first hidden state determination unit 41 will be described.

[0041] As shown in FIG. 8 , in the hidden state detection unit 40 of this embodiment, the first hidden state determination unit 41 performs movement prediction for a person H captured in an image Vd captured by the camera 8. Based on the movement prediction, the first hidden state determination unit 41 determines whether or not there is an image capture area 50 of the person H that is estimated to be "overlapping" after a predetermined future time. The "predetermined future time" may be, for example, the timing when the next image analysis is scheduled to be performed. Furthermore, the first hidden state determination unit 41 calculates an overlap rate α for the image capture area 50 of the person H that is overlapping. The first hidden state determination unit 41 of this embodiment then performs detection and determination of a hidden state occurring in the captured image Vd based on a comparison between the overlap rate α and a predetermined overlap determination value αth.

[0042] More specifically, in the image analysis device 10 of this embodiment, the first hidden state determination unit 41 calculates the overlap rate α by using the upper body of the occupant 5 shown in the captured image Vd, specifically, the captured image 60 of the torso, as the captured image 50 of the person H. Specifically, the first hidden state determination unit 41 of this embodiment calculates the direction and amount of movement of the center point 60x of the captured image areas 61, 62 of the torso detected in the captured image Vd at the previous and current analysis times. Furthermore, the first hidden state determination unit 41 compares the sizes of these captured image areas 61, 62. The first hidden state determination unit 41 of this embodiment is configured to thereby predict the position and size of the captured image area 63 shown in the captured image Vd at a predetermined future time.

[0043] For example, FIG. 8 illustrates a situation in which an occupant 5b in a standing position walks across the rear seat area A1 at a position closer to the camera 8 than an occupant 5a in a seated position seated on the seat 7. In this case, the imaging area 50a in the captured image Vd of the seated occupant 5a hardly moves. On the other hand, the imaging area 50b of the standing occupant 5b is predicted to move from the lower right to the upper left in the captured image Vd of FIG. 8 as the occupant 5b walks. Furthermore, based on this predicted movement, an overlapping area 55 in the captured image Vd, where the imaging areas 50a and 50b of the occupants 5a and 5b overlap, is estimated. The first obscured state determination unit 41 of this embodiment is configured to calculate the proportion of the overlapping area 55 in the imaging area 50a of the distant occupant 5a who is predicted to be obscured by the standing occupant 5b who is closer to the camera 8, as the overlapping rate α.

[0044] That is, as shown in FIG. 9 , the first hidden state determination unit 41 of this embodiment detects the imaging area 50 of each occupant 5 captured in the captured image Vd (step 101). Next, the first hidden state determination unit 41 executes movement prediction for each of these occupants 5 (step 102). Subsequently, the first hidden state determination unit 41 estimates an overlapping area 55 formed in the imaging area 50 of each occupant 5 based on this movement prediction, and calculates the overlapping rate α (step 103). Then, the first hidden state determination unit 41 of this embodiment is configured to determine that an obscured state has occurred in the captured image Vd if this overlapping rate α is equal to or greater than a predetermined overlapping rate αth (α≧αth, step 104: YES) (step 105).

[0045] (Judgment of difference in number of people) Next, the detection and determination of the hidden state based on the number of people difference determination executed by the second hidden state determination unit 42 will be described.

[0046] 10 and 11, in the hidden state detection unit 40 of this embodiment, the second hidden state determination unit 42, based on an analysis of the captured image Vd, counts the number of people H entering and exiting the vehicle compartment 6 as the monitored space 11 shown in the captured image Vd, i.e., the number of people getting on and off the vehicle 1. The second hidden state determination unit 42 of this embodiment is configured to identify the total number of people in the vehicle compartment 6 based on the measurement of the number of people getting on and off.

[0047] 1 and 3, in the monitoring system 15 of this embodiment, the cabin 6 of the vehicle 1 captured by the camera 8 is defined with a rear seat area A1, a front seat area A2, and a middle seat area A3, which are arranged in a generally U-shape. The cabin 6 of the vehicle 1 is also defined with a floor area A4, which is surrounded by the rear seat area A1, the front seat area A2, and the middle seat area A3 and allows the occupant 5 to board in a standing position. The cabin 6 of the vehicle 1 is also defined with a boarding / exiting area A5, in which the occupant 5 is prohibited from remaining, near the sliding doors 4, 4 that open and close the door opening 3. When an occupant 5 is detected in the boarding / exiting area A5, the second hidden state determination unit 42 of this embodiment determines that the occupant 5 is a occupant 5 who has boarded the vehicle 1 or a occupant 5 who is exiting the vehicle 1.

[0048] 10 and 11, the second hidden state determination unit 42 of this embodiment identifies the movement direction of an occupant 5 located in the getting-in / out area A5. Furthermore, if the movement direction of the occupant 5 is a direction from the door opening 3 toward the interior of the vehicle compartment 6 (see FIG. 10, left side in the figure), the second hidden state determination unit 42 determines that the occupant 5 has gotten into the vehicle 1. Then, if the movement direction of the occupant 5 is a direction from the interior of the vehicle compartment 6 toward the door opening 3 (see FIG. 11, right side in the figure), the second hidden state determination unit 42 determines that the occupant 5 will get out of the vehicle 1.

[0049] In the image analysis device 10 of this embodiment, the second hidden state determination unit 42 determines whether the occupant 5 is getting in or out of the vehicle 1 after converting the captured image Vd of the vehicle interior 6 captured by the camera 8 into a top view as shown in Fig. 3. This enables the second hidden state determination unit 42 of this embodiment to accurately measure the number of people N getting in and out of the vehicle 1.

[0050] Furthermore, the second hidden state determination unit 42 of this embodiment counts the number of passengers getting on and off the vehicle 1 by adding "+1" when one passenger 5 getting on the vehicle 1 is detected, and by adding "-1" when one passenger 5 getting off the vehicle 1 is detected. This makes it possible for the second hidden state determination unit 42 of this embodiment to identify the total number of passengers 5 located in the vehicle compartment 6 based on the count of the number of passengers getting on and off.

[0051] 12, the second hidden state determination unit 42 of this embodiment converts the captured image Vd of the vehicle interior 6 captured by the camera 8 into a top view (step 201), and determines whether an occupant 5 is detected in the boarding / alighting area A5 in front of the door opening 3 (step 202). Next, if an occupant 5 is detected in the boarding / alighting area A5 (step 202: YES), the second hidden state determination unit 42 determines whether the occupant 5 will get on or off the vehicle 1 (steps 203 and 204). Furthermore, if the second hidden state determination unit 42 detects an occupant 5 getting on the vehicle 1 (step 203: YES), it adds "1" to the number N of people getting on and off the vehicle 1 that it is measuring (N = N + 1, step 205). Furthermore, when the second hidden state determination unit 42 detects an occupant 5 getting off the vehicle 1 (step 204: YES), it subtracts "1" from the measured number of people N getting on and off the vehicle 1 (N=N-1, step 206). Then, the second hidden state determination unit 42 of this embodiment identifies the thus-calculated number of people N getting on and off as the total number Na of occupants 5 located in the vehicle compartment 6 (Na=N, step 207).

[0052] Furthermore, the second hidden state determination unit 42 of this embodiment calculates the number of detected occupants 5 Nd appearing in the captured image Vd of the vehicle interior 6 as the number of detected people H appearing in the captured image Vd based on the detection of the skeleton points SP as described above. Specifically, when the second hidden state determination unit 42 of this embodiment detects an occupant 5 from which a major skeleton point SP used for the posture determination or the like can be extracted in the captured image Vd of the vehicle interior 6 captured by the camera 8, the second hidden state determination unit 42 adds the occupant 5 to the number of detected people Nd in the captured image Vd. Furthermore, the second hidden state determination unit 42 compares the total number of people Na in the vehicle interior 6 identified by measuring the number of people getting in and out N as described above with the detected number of people Nd of occupants 5 appearing in the captured image Vd. The second hidden state determination unit 42 of this embodiment is configured to detect and determine whether a hidden state has occurred in the captured image Vd based on the difference between the total number of people Na and the detected number of people Nd.

[0053] 13, the second hidden state determination unit 42 of this embodiment determines the total number of people Na in the vehicle cabin 6 based on the measurement of the number of people getting on and off N (step 301), and then calculates the number of detected occupants Nd of occupants 5 appearing in the captured image Vd (step 302). Next, the second hidden state determination unit 42 calculates a difference value δ between the total number of people Na in the vehicle cabin 6 determined based on the measurement of the number of people getting on and off N and the number of detected occupants Nd of occupants 5 appearing in the captured image Vd (δ=Na-Nd, step 303). Then, if the difference value δ is greater than 0 (δ>0, step 304: YES), the second hidden state determination unit 42 of this embodiment is configured to determine that an obscured state has occurred in the captured image Vd (step 305).

[0054] (Camera proximity detection) Next, the detection and determination of the hidden state based on the camera approach position determination executed by the third hidden state determination unit 43 will be described.

[0055] 14, when an occupant 5 in the vehicle interior 6 stands close to a camera 8, the occupant 5 may block the field of view of the camera 8. As a result, the occupant 5 close to the camera 8 may hide other occupants 5 located in the vehicle interior 6, resulting in an obscured state in the captured image Vd.

[0056] In consideration of this, in the hidden state detection unit 40 of this embodiment, the third hidden state determination unit 43 executes detection determination of the occupant 5 appearing in the captured image Vd at the hidden position Px close to the camera 8. Then, when the third hidden state determination unit 43 of this embodiment detects the occupant 5 appearing in the captured image Vd at the hidden position Px, it is configured to determine that a hidden state has occurred in this captured image Vd.

[0057] 14 and 15, the third hidden state determination unit 43 of this embodiment calculates the size of the image capture area 70 of the occupant 5 shown in the captured image Vd (step 401). Next, the third hidden state determination unit 43 calculates the proportion of the image capture area 70 of the occupant 5 to the entire captured image Vd as an image ratio β (step 402). Subsequently, the third hidden state determination unit 43 compares this image ratio β with a predetermined proximity determination value βth (step 403). Note that in the second hidden state determination unit 42 of this embodiment, the processes of steps 401 to 403 are performed on all occupants 5 shown in the captured image Vd. Furthermore, when an occupant 5 having an image ratio β equal to or greater than the proximity determination value βth is detected (β≧βth, step 403: YES), the third hidden state determination unit 43 determines that this occupant 5 is captured in the captured image Vd at a hidden position Px close to the camera 8 (step 404).The third hidden state determination unit 43 of this embodiment is configured to determine that a hidden state has occurred in the captured image Vd as a result (step 405).

[0058] (Decision output regarding skeleton point detection status) Next, a description will be given of the determination output regarding the detection state of the skeleton points SP in the photographed image Vd in which the occurrence of an occlusion state is detected.

[0059] The monitoring system 15 of this embodiment is configured so that even when the occurrence of a hidden state is detected as described above, a manager 35 outside the vehicle, such as an operator 33 stationed at the operation center 32 of the vehicle 1, can check the captured image Vd of the cabin 6 captured by the camera 8. In addition, the monitoring system 15 of this embodiment is configured so that at this time, the detected output of the hidden state by the hidden state detection unit 40 is delivered to the operation center 32 where the operator 33 is stationed, together with the captured image Vd of the cabin 6.

[0060] More specifically, as shown in FIG. 7, the hidden state detection unit 40 of this embodiment is provided with a judgment output unit 80 that, when detecting the occurrence of a hidden state, outputs a judgment about the detection state of the skeleton point SP in the captured image Vd in which the occurrence of the hidden state is detected.

[0061] Specifically, when the determination output unit 80 of this embodiment detects an obscured state occurring in the photographed image Vd based on the overlap rate determination performed by the first obscured state determination unit 41, it outputs the detection state of the skeleton points SP using the photographed image Vd as "indetermined" as its detection output. Furthermore, when the determination output unit 80 detects an obscured state occurring in the photographed image Vd based on the number of people difference determination performed by the second obscured state determination unit 42, it also outputs the detection state of the skeleton points SP using this photographed image Vd as "indetermined" as its detection output. Furthermore, when the determination output unit 80 of this embodiment determines that an obscured state has occurred in the photographed image Vd based on the camera proximity position determination performed by the third obscured state determination unit 43, it outputs the detection state of the skeleton points SP using this photographed image Vd as "undeterminable" as its detection output.

[0062] That is, the output of "indetermined" by the hidden state detection unit 40 indicates that the presence of an undetected occupant 5 whose skeleton point SP has not been detected has reduced the accuracy of acquiring the occupant information Ich based on the detection of the skeleton point SP. The output of "undeterminable" by the hidden state detection unit 40 indicates that it is not possible to determine whether the skeleton point SP is undetected, even through the overlap rate determination performed by the first hidden state determination unit 41 and the number difference determination performed by the second hidden state determination unit 42. Furthermore, in the monitoring system 15 of this embodiment, the output of "undeterminable" is treated as a highly urgent "occurrence of an abnormality," similar to the abnormality detection output by the abnormality detection unit 22. The monitoring system 15 of this embodiment is thus configured so that the manager 35 outside the vehicle can check the captured image Vd of the vehicle interior 6 while referring to the detection state of the skeleton point SP, i.e., the accuracy of the occupant information Ich acquired by image analysis.

[0063] 16, when the image analyzing device 10 of this embodiment acquires a captured image Vd of the vehicle interior 6 (step 501), it performs image analysis to detect skeleton points SP of the occupant 5 appearing in the captured image Vd (step 502). Next, the image analyzing device 10 performs a hidden state determination using the hidden state detection unit 40 (step 503). If the hidden state determination detects an hidden state in the captured image Vd (step 504: YES), it executes detection output control to deliver a detection output of this hidden state together with the captured image Vd to the manager 35 outside the vehicle (step 505).

[0064] 17, in the image analyzing device 10 of this embodiment, the hidden state detection unit 40 first executes a detection determination of a hidden state by its third hidden state determination unit 43, i.e., a camera proximity position determination (step 601). Then, in this embodiment, if the hidden state detection unit 40 detects an occupant 5 appearing in the captured image Vd at a hidden position Px close to the camera 8 (step 602: YES), its determination output unit 80 outputs "unable to determine" (step 603).

[0065] Furthermore, if the hidden state detection unit 40 does not detect an occupant 5 at a hidden position Px close to the camera 8 (step 602: NO), the first hidden state determination unit 41 executes a hidden state detection determination, i.e., an overlap rate determination (step 604). Subsequently, the hidden state detection unit 40 executes a hidden state detection determination, i.e., a number-of-persons difference determination, by the second hidden state determination unit 42 (step 605). Furthermore, if the hidden state in the captured image Vd is detected by these overlap rate determination and number-of-persons difference determination (step 606: YES), the hidden state detection unit 40 determines that there is an occupant 5 in the vehicle interior 6 whose skeleton point SP has not yet been detected (step 607). Then, in the image analyzing device 10 of this embodiment, the determination output unit 80 of the hidden state detection unit 40 outputs "indeterminate" as the hidden state detection output (step 608).

[0066] 16, when the image analyzing device 10 does not detect an occlusion state in the captured image Vd in the occlusion state determination (step 504: NO), the abnormality detection unit 22 performs an abnormality detection determination for the vehicle interior 6 shown in the captured image Vd (step 506). That is, as described above, in the image analyzing device 10 of this embodiment, the abnormality detection determination performed by the abnormality detection unit 22 is based on the posture determination of the occupant 5 performed by the posture determination unit 26 on the basis of the detection of the skeleton points SP, specifically, the detection of a fallen posture. Then, in the image analyzing device 10 of this embodiment, when an abnormality in the vehicle interior 6 is detected as a result of this (step 507: YES), the abnormality detection unit 22 delivers an abnormality detection output together with the captured image Vd to the manager 35 outside the vehicle (step 508).

[0067] Next, the operation of this embodiment will be described. That is, in the image analysis device 10 of this embodiment, for each occupant 5 shown in the captured image Vd of the vehicle interior 6, an overlap rate α of the captured image area 50 estimated based on the predicted movement of the occupant 5 is calculated. Also, a difference value δ between the total number of people Na in the vehicle interior 6 identified by measuring the number of people N getting in and out and the detected number of occupants 5 Nd shown in the captured image Vd is calculated. Furthermore, at a hidden position Px close to the camera 8, a detection determination is made as to whether or not an occupant 5 is shown in the captured image Vd. Then, based on the results of these overlap rate determinations, number of people difference determinations, and camera proximity position determinations, the occurrence of a hidden state in which multiple occupants 5 are overlapping in the captured image Vd is detected.

[0068] Next, the effects of this embodiment will be described. (1) The image analysis device 10 has a person information acquisition unit 25 that analyzes the captured image Vd of the vehicle interior 6 of the vehicle 1 captured by the camera 8 as the monitored space 11 to acquire information Ih of the person H captured in the captured image Vd, i.e., occupant information Ich of the occupant 5. The image analysis device 10 also has an obscured state detection unit 40 that detects the occurrence of an obscured state in which multiple occupants 5 are captured overlapping each other in the captured image Vd.

[0069] According to the above configuration, it is possible to detect an occlusion state occurring in the captured image Vd and to grasp that the occupant 5 in the vehicle interior 6 is not properly captured in the captured image Vd. In this case, for example, by refraining from analytically using the captured image Vd in which the occlusion state has occurred, it is possible to ensure highly accurate information acquisition.

[0070] (2) The first hidden state determination unit 41 provided in the hidden state detection unit 40 functions as a movement prediction unit 90a that predicts the movement of each occupant 5 appearing in the captured image Vd. The first hidden state determination unit 41 also functions as an overlap rate calculation unit 90b that calculates an overlap rate α for the captured image region 50 of each occupant 5 that is estimated to appear overlapped in the captured image Vd based on the movement prediction. The first hidden state determination unit 41 then performs detection and determination of the hidden state based on a comparison between the overlap rate α and an overlap determination value αth.

[0071] According to the above configuration, it is possible to accurately detect the occurrence of an occlusion state in the captured image Vd. Furthermore, there is an advantage that it is possible to predict the occurrence of an occlusion state in advance. This allows for more accurate information acquisition.

[0072] (3) The second hidden state determination unit 42 provided in the hidden state detection unit 40 functions as an entry / exit number measurement unit 90c that measures the number N of occupants 5, which is the number of people H entering and exiting the monitored space 11. The second hidden state determination unit 42 performs detection and determination of the hidden state based on the difference between the total number Na of people in the vehicle compartment 6 identified by measuring the number N of people entering and exiting, and the detected number Nd of occupants 5 captured in the captured image Vd.

[0073] That is, when no occlusion state occurs in the captured image Vd, the detected number Nd of occupants 5 appearing in this captured image Vd is equal to the total number Na of people in the vehicle compartment 6. Therefore, according to the above configuration, an occlusion state occurring in the captured image Vd can be easily detected with a simple configuration.

[0074] (4) The third hidden state determination unit 43 provided in the hidden state detection unit 40 determines that a hidden state has occurred in the captured image Vd when an occupant 5 is detected in the captured image Vd at a hidden position Px close to the camera 8.

[0075] That is, an occupant 5 appearing in the captured image Vd at a hiding position Px close to the camera 8 may block the field of view of the camera 8, thereby hiding another occupant 5 located in the vehicle interior 6, potentially creating an obscured state in the captured image Vd. In this case, it is impossible to determine whether or not the obscured state has actually occurred. In light of this, the obscured state is considered to have occurred as in the above configuration. In this case, for example, by refraining from analytically using the captured image Vd in which the obscured state has occurred, it is possible to ensure highly accurate information acquisition.

[0076] (5) The third hidden state determination unit 43 functions as an image ratio calculation unit 90d that calculates the ratio of the captured image Vd of the image capture area 70 of the occupant 5 to the entire captured image Vd as an image ratio β. The third hidden state determination unit 43 also functions as a camera proximity position determination unit 90e that, when a detected occupant 5 has an image ratio β equal to or greater than a predetermined proximity determination value βth, determines that the occupant 5 is captured in the captured image Vd at the hidden position Px. This makes it possible to detect the occupant 5 captured in the captured image Vd at the hidden position Px close to the camera 8 with a simple configuration.

[0077] (6) The image analysis device 10 includes a skeleton point detection unit 23 that detects skeleton points SP of the occupant 5 included in the captured image Vd. The image analysis device 10 also includes an abnormality detection unit 22 that detects an abnormality that has occurred in the vehicle interior 6 based on the occupant information Ich acquired by detecting the skeleton points SP.

[0078] That is, by detecting the skeleton points SP, it is possible to accurately acquire physical occupant information Ich, such as the posture and physique of the occupant 5. As a result, it is possible to perform highly accurate abnormality detection and judgment for the vehicle interior 6 in which the occupant 5 is riding, based on the acquired occupant information Ich. However, when an obscured state occurs in the captured image Vd, the detection state of the skeleton points SP also deteriorates. As a result, there is a possibility that the abnormality detection and judgment cannot be performed accurately. Therefore, by applying the detection and judgment for the obscured state shown in (1) to (5) above to such a configuration, more significant effects can be obtained.

[0079] (7) The hidden state detection unit 40 includes a judgment output unit 80 that, when the occurrence of a hidden state is detected, outputs a judgment regarding the detection state of the skeleton point SP in the captured image Vd in which the occurrence of the hidden state is detected, as the detection output of the hidden state.

[0080] According to the above configuration, it is possible to correctly grasp the detection state of the skeleton points SP that has changed due to the occurrence of an occlusion state, and thereby to appropriately utilize the captured image Vd in which the occlusion state has occurred.

[0081] The above embodiment can be modified as follows: The above embodiment and the following modifications can be combined with each other within the scope of technical compatibility.

[0082] In the above embodiment, an infrared camera is used as the camera 8, but the type of camera may be changed as desired. For example, a visible light camera or the like may be used. In the above embodiment, the imaging area 60 of the torso part is used as the imaging area 50 of the person H to calculate the overlap rate α, but the range of the imaging area 50 used to calculate the overlap rate α may be changed arbitrarily, for example, to include the head. Furthermore, the future time for which movement prediction and overlap estimation are performed may also be changed arbitrarily. Furthermore, the specific method of movement prediction may also be changed arbitrarily.

[0083] In the above embodiment, the determination of whether the occupants 5 are getting on or off is performed by converting the captured image Vd into a top-view image, but top-view conversion is not necessarily required. For example, if the total number of people Na in the vehicle cabin 6 can be obtained by a method other than image analysis, such as from boarding reservation information, that value may be used.

[0084] In the above embodiment, the detected number Nd of occupants 5 appearing in the captured image Vd is calculated based on the detection of skeleton points SP, specifically, the possibility of extracting major skeleton points SP. However, this does not necessarily have to be based on the detection of skeleton points SP. The number of occupants 5 recognized in the captured image Vd using other methods may also be used as the detected number Nd. Furthermore, the difference value δ between the total number Na of occupants in the vehicle cabin 6 and the detected number Nd of occupants 5 does not necessarily have to be calculated, and a configuration in which a match determination is simply performed may also be used.

[0085] In the above embodiment, the image ratio β is the ratio of the captured image Vd of the captured area 70 of the occupant 5 to the entire captured image Vd. An occupant 5 having an image ratio β equal to or greater than a predetermined proximity determination value βth is determined to be captured in the captured image Vd at the hidden position Px. However, this is not limiting, and the method for detecting and determining an occupant 5 captured in the captured image Vd at the hidden position Px may be changed as desired. That is, it is sufficient to determine that an occupant 5 in the vehicle compartment 6 is standing close to the camera 8 and thus blocking the view of the camera 8. The identified position of the occupant 5 is the hidden position Px. For example, the image ratio β does not necessarily have to be calculated. Instead, the camera proximity position may be determined based on a combination of the size of the captured image 70 and the use status of vehicle fixtures 1, such as a strap or handrail, that indicate that the occupant 5 is close to the camera 8.

[0086] In the above embodiment, the posture of the occupant 5 is determined based on the detection of the skeleton point SP. Then, by detecting the fallen posture, an abnormality in the vehicle interior 6 shown in the captured image Vd is detected. However, this is not limited to this, and the abnormality detection determination may be performed using other occupant information Ich acquired by image analysis of the captured image Vd. Furthermore, the present invention may be applied to a configuration in which the occupant information Ich is acquired without relying on the detection of the skeleton point SP. The acquired occupant information Ich may be used for purposes other than the abnormality detection determination.

[0087] In the above embodiment, the monitoring system 15 is formed by interconnecting multiple information processing devices 30 installed inside and outside the vehicle 1 via an information communication network (not shown). The image analysis device 10 performs its image analysis processing in a distributed manner between an on-board information processing device 30a installed in the vehicle 1 and an off-board information processing device 30b constituting a cloud server 31. However, the system configuration of the monitoring system 15 is not limited to this, and may be changed as desired. For example, the image analysis device 10 may be implemented in the on-board information processing device 30a installed in the vehicle 1. The off-board information processing device 30b constituting the image analysis device 10 may be located in the vehicle operation center 32 of the vehicle 1 where an operator 33 serving as a manager 35 is stationed.

[0088] Furthermore, when an abnormality occurs or a hidden state is detected, the captured image Vd inside the vehicle compartment 6 that the manager 35 checks may be configured so that the captured image Vd captured by the camera 8 is constantly distributed to the manager 35 outside the vehicle, or may be configured so that the image is distributed only when an event occurs.

[0089] In the above embodiment, the monitoring system 15 is embodied as the vehicle interior 6 captured by the camera 8 of the vehicle 1, with the monitored space 11 being the interior of a building. However, the present invention is not limited to this, and the monitored space 11 may be configured as the interior of a building. Furthermore, the monitored space 11 may be configured to be set outdoors, for example. [Explanation of symbols]

[0090] 1...Vehicle 5...Crew 6…Vehicle compartment 8...Camera 11…Surveillance space 25…Person Information Acquisition Department 40...Hidden state detection unit Vd...Captured image H…person Ih…information Ich...Crew information

Claims

1. a person information acquisition unit that acquires information about people appearing in a captured image by analyzing the captured image of the monitored space captured by the camera; an obscured state detection unit that detects the occurrence of an obscured state in which a plurality of the people are overlapped in the captured image; an entry / exit number counting unit that counts the number of people entering and exiting the monitored space; The image analysis device is characterized in that the hidden state detection unit performs the detection determination of the hidden state based on the difference between the total number of people in the monitored space identified by measuring the number of people entering and exiting and the detected number of people appearing in the captured image.

2. a person information acquisition unit that acquires information about people appearing in a captured image by analyzing the captured image of the monitored space captured by the camera; an obscured state detection unit that detects the occurrence of an obscured state in which a plurality of the people are overlapped in the captured image; the hiding state detection unit determines that the hiding state has occurred when the person in the captured image is detected at a hiding position close to the camera; An image analysis device characterized by the above.

3. 3. The image analysis device according to claim 1, a movement prediction unit that predicts the movement of each person appearing in the captured image; an overlap rate calculation unit that calculates an overlap rate of the image capture areas of the people estimated to be overlapped based on the movement prediction, The image analyzing device according to claim 1, wherein the hidden state detection unit detects and determines the hidden state based on a comparison between the overlap rate and an overlap determination value.

4. 3. The image analysis device according to claim 2, an image ratio calculation unit that calculates an image ratio, which is a ratio of an image area of ​​the person to the entire captured image; a camera proximity position determination unit that determines that the person is in the captured image at the hidden position when the person has the image ratio equal to or greater than a predetermined proximity determination value; An image analysis device comprising:

5. The image analysis device according to any one of claims 1 to 4, a skeleton point detection unit that detects skeleton points of the person included in the captured image; an abnormality detection unit that detects abnormalities that have occurred in the monitored space based on the information about the person obtained by detecting the skeleton points.

6. 6. The image analysis device according to claim 5, an output unit that, when the occurrence of the hidden state is detected, executes a determination output regarding the detection state of the skeleton points in the captured image in which the occurrence of the hidden state is detected, as a detection output of the hidden state.

7. The image analysis device according to any one of claims 1 to 6, the monitored space is a vehicle cabin, and the person is an occupant of the vehicle; An image analysis device characterized by the above.

8. A monitoring system comprising the image analysis device according to any one of claims 1 to 7.

Images