Monitoring system and monitoring method

Abstract

To provide a monitoring system and a monitoring method which facilitate grasping a situation in which a state different from a normal state has occurred.SOLUTION: A monitoring system according to an embodiment comprises an imaging device, a measurement device and a monitoring device. The imaging device captures an image within a space to be monitored. The measurement device measures a state of a person included in the image. The monitoring device generates notification information in a case where the person included in the image is in a predetermined state, on the basis of the measurement.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a monitoring system and a monitoring method. [Background technology]

[0002] There is an increasing demand for systems that use camera footage to monitor people's behavior. Such camera-based systems have the advantage of being able to record accidents and crimes that occur within the camera's range, allowing for post-event review. However, in spaces with many people, such as on trains, it can be difficult to assess the situation based on images alone. This can make it difficult to grasp unusual situations, such as accidents and crimes, in real time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6487371 Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, the problem that the invention aims to solve is to provide a monitoring system and a monitoring method that can more easily grasp situations in which an abnormal state has occurred. [Means for solving the problem]

[0005] A monitoring system according to an embodiment includes an image capturing device, a measurement device, and a monitoring device. The image capturing device captures an image of a space to be monitored. The measurement device measures the state of a person in the image. The monitoring device generates notification information based on the measurement when the person in the image is in a predetermined state. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a monitoring system. [Figure 2] FIG. 1 is a block diagram showing an example of the configuration of a monitoring system. [Figure 3] FIG. 1 is a top view showing an example of the arrangement of multiple cameras and multiple transmitters and receivers in a vehicle. [Figure 4] FIG. 4 is a diagram showing an example of a screen displayed on a display device by the monitoring device. [Figure 5] 10A and 10B are diagrams showing an example of an irradiation range of a transmitter / receiver in a measurement device and an example of a specified area. [Figure 6] 10 is a flowchart showing an example of processing by the imaging device. [Figure 7] 10 is a flowchart showing an example of processing by the measurement device. [Figure 8] 10 is a flowchart showing an example of processing by a monitoring device. [Figure 9] FIG. 10 is a top view showing another example of the arrangement of multiple cameras and multiple transmitter / receivers in a vehicle. [Figure 10] FIG. 10 is a block diagram showing an example of the configuration of a monitoring system according to a second embodiment. [Figure 11] 10 is a flowchart showing an example of processing by the scream detection device. [Figure 12] FIG. 10 is a block diagram showing the configuration of a monitoring system according to a third embodiment. [Figure 13] FIG. 11 is a top view showing an example of an arrangement inside a vehicle according to a third embodiment. [Figure 14] FIG. 2 is a diagram showing an example of an image captured by a camera. [Figure 15] FIG. 11 is a diagram showing an example of a screen displayed on a display device by a monitoring device according to a third embodiment. [Figure 16] 10 is a flowchart showing the processing of the scream detection device according to the third embodiment. [Figure 17] 10 is a flowchart showing an example of processing by a monitoring device according to the third embodiment. [Figure 18] FIG. 10 is a diagram showing a plurality of arrangement patterns. [Figure 19] FIG. 10 is a diagram showing an example in which a sound collecting unit is further disposed at an intermediate position in the traveling direction of the vehicle. [Figure 20] 1A and 1B are diagrams showing examples of vehicle layouts in which seats face the direction of travel or the opposite direction to the direction of travel. [Figure 21]FIG. 10 is a diagram showing an example in which a camera and a sound collecting unit are arranged in the center of a vehicle in the longitudinal direction. [Figure 22] FIG. 10 is a diagram showing an example of the arrangement of cameras, transmitter / receivers, and voice collectors in an elevator hall. DETAILED DESCRIPTION OF THE INVENTION

[0007] A monitoring system and a monitoring method according to an embodiment of the present invention will be described in detail below with reference to the drawings. Note that the embodiment described below is an example of an embodiment of the present invention, and the present invention should not be construed as being limited to these embodiments. Furthermore, in the drawings referred to in this embodiment, identical parts or parts having similar functions are given the same or similar reference numerals, and repeated explanations thereof may be omitted. Furthermore, for convenience of explanation, the dimensional ratios of the drawings may differ from the actual ratios, and some components may be omitted from the drawings.

[0008] (First embodiment) FIG. 1 is a diagram showing the schematic configuration of a monitoring system 1 according to this embodiment. As shown in FIG. 1, the monitoring system 1 is a system capable of acquiring information on the status of a person in a monitored space, and includes an image capturing device 11, a first measuring device 12, and a monitoring device 13. The monitoring system 1 is installed, for example, inside a passenger car such as a train or bus, or in an elevator hall. That is, the monitoring system 1 is installed, for example, on the ceiling of a railway vehicle, the ceiling of a bus, or the ceiling of an elevator hall, and is a device capable of acquiring status information from above downward. The image capturing device 11, the first measuring device 12, and the monitoring device 13 may be installed on the ceiling using separate fixtures. Alternatively, they may be fixed together using a pair of fixtures and installed on the ceiling. The structure of the fixture is arbitrary.

[0009] An example of the configuration of the monitoring system 1 will now be described with reference to Figs. 2 and 3. Fig. 2 is a block diagram showing an example of the configuration of the monitoring system 1. As shown in Fig. 2, the image capturing device 11 is configured to have, for example, a CPU. This image capturing device 11 is a device that can capture an image of a person and estimate the age, and has a person information storage unit 110, multiple cameras 111a and 111b, a person detection unit 112, an age estimation unit 113, and a communication unit 134. Details of the image capturing device 11 will be described later.

[0010] The first measuring device 12 is configured to have, for example, a CPU. The first measuring device 12 is a device that can measure, in a non-contact manner, the heart rate and respiratory rate as status information of a person in an image captured by the imaging device 11, and has a plurality of transmitting / receiving units 121, an amplifier 122, and a processing unit 123.

[0011] The monitoring device 13 is configured to have, for example, a CPU. The monitoring device 13 is a device capable of determining the state of a person, and has a memory unit 131, a determination unit 132, a firing unit 133, and a communication unit 134. The monitoring system 1 according to this embodiment has, but is not limited to, multiple cameras 1111a, 1111b, and 1111c and multiple transmitter / receivers 121. For example, the monitoring system 1 may be configured to have a single camera 111 and a single transmitter / receiver 121. The first measuring device 12 and the monitoring device 13 will be described in detail later.

[0012] 2 also shows a display device 200. The display device 200 is placed, for example, in the driver's cab of a train or the like, or in a central monitoring room. The display device 200 communicates with the monitoring device 13 via wire or wirelessly, and includes, for example, a monitor 201 and a speaker 202. The display device 200 displays an image on the monitor 201 in accordance with an image signal transmitted from the monitoring device 13, and produces sound from the speaker 202 in accordance with the transmitted sound signal.

[0013] The person information storage unit 110 of the image capturing device 11 is realized by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, etc. The person information storage unit 110 stores, for example, information on normal pulse rate ranges by age or pulse rate ranges in an abnormal state. For example, the pulse rate ranges in an abnormal state are divided into a low pulse rate range and a high pulse rate range.

[0014] 3 is a top view showing an example of the arrangement of multiple cameras 111a, b and multiple transmitter / receivers 121 inside vehicle 200. Line L200 indicates the middle of the opposing opening and closing doors. That is, vehicle 200 has, for example, eight opening and closing doors. Line L202 indicates the center line of vehicle 200 in the traveling direction, that is, the center line of vehicle 200 in the longitudinal direction. Seats 202 are arranged inside vehicle 200. Multiple cameras 111a, b are cameras capable of capturing video.

[0015] As shown in FIG. 3 , cameras 111a and 111b of image capture device 11 are disposed on the short sides of vehicle 200. That is, cameras 111a and 111b are disposed at the end of vehicle 200 where it is connected to other vehicles. As a result, camera 111a mainly captures images of the interior of vehicle 200 from the left end to the right, and camera 111b mainly captures images of the interior of vehicle 200 from the right end to the left. By disposing cameras 111a and 111b at both ends of vehicle 200 where it is connected to other vehicles, it is possible to capture images of passengers from both the left and right sides, and to capture images of at least half of the passengers' faces. As can be seen from this, disposing cameras 111a and 111b at the end of vehicle 200 where it is connected to other vehicles allows a wide depth in the direction of travel to be captured, thereby reducing the number of cameras 111a and 111b to be installed. Furthermore, in image capture device 11, the coordinates of images captured by cameras 111a and 111b are previously associated with coordinates within vehicle 200, such as planar coordinates. Furthermore, images captured by the multiple cameras 111a and 111b are stored in the person information storage unit 114 in association with the camera that captured the image and the time of capture.

[0016] 3, each of the multiple transmitter / receivers 121 is disposed at an intersection of line L200 and line L202. This allows the multiple transmitter / receivers 121 to measure almost all passengers entering and exiting through the opening and closing doors from above diagonally downward. In other words, the multiple transmitter / receivers 121 are disposed at positions where they can measure passengers entering and exiting through the opening and closing doors.

[0017] FIG. 4 is a diagram showing an example of a screen displayed on the display device 200 by the monitoring device 13. Screen M10 indicates the screen of the monitor 201 (see FIG. 2), screen M12 within screen M10 is a top view of the vehicle 200, and marks mh10 and mh20 indicate the positions of persons h10 and h20 in which some abnormality has been detected. Images captured by cameras 111a and 111b are alternately displayed on screen M14 within screen M10. Areas ag10 and ag20 indicate the areas of persons h10 and h20 detected by the person detection unit 112, which will be described later. Furthermore, areas f10 and f20 indicate face areas within areas ag10 and ag20 detected by the person detection unit 112, which will be described later.

[0018] As shown in FIG. 2 again, the person detection unit 112 of the image capture device 11 detects person areas ag10 and ag20 and face areas f10 and f20 within the person areas ag10 and ag20 from each image stored in the person information storage unit 114 (see M14 in FIG. 4). A general algorithm can be used to detect the person areas ag10 and ag20 and the face areas f10 and f20. For example, the contours of moving objects are extracted from time-series difference images, and the person areas are extracted by labeling and color separation processes based on the extracted contours. Furthermore, the face areas f10 and f20 are extracted by, for example, recognizing the color and shape within the person areas ag10 and ag20.

[0019] Furthermore, the person detection unit 112 associates the coordinates of the detected person's areas ag10, ag20 and the facial areas f10, f20 with planar coordinates within the vehicle 200. In this case, the image areas captured by the multiple cameras 111a may overlap with the image areas captured by the camera 111b. Therefore, the detected person's areas ag10, ag20 and the facial areas f10, f20 may also overlap. The person detection unit 112 then associates the coordinates of the detected person's areas ag10, ag20 and the facial areas f10, f20, and the planar coordinates within the vehicle 200 with the captured image, and stores them in the person information storage unit 110. That is, the position information generated by the person detection unit 112 includes, for example, the coordinates of the circumscribing rectangle of the detected person's area, the position coordinates of the center of gravity of the circumscribing rectangle, the coordinates of the circumscribing rectangle of the detected facial area, and the position coordinates of the center of gravity.

[0020] The age estimation unit 113 of the imaging device 11 performs age estimation processing on the image of the face region detected by the person detection unit 112. A general algorithm can be used for this age estimation processing. For example, the age estimation unit 113 includes an image processing unit that generates multiple feature amounts related to wrinkles and the like for the image of the face region, and a neural network unit that receives the multiple feature amounts as input and outputs age. The age estimation unit 113 further associates the age estimated by the neural network unit with the captured image stored in the person information storage unit 114 and stores it. The age estimation unit 113 also supplies position information, estimated age, and image capture time information of the detected person regions ag10 and ag20 to the storage unit 131 of the monitoring device 13.

[0021] The age estimation unit 113 according to this embodiment uses a so-called neural network for recognition, but is not limited to this. For example, the age estimation unit 113 may compare the image of the identified face region with images of face regions stored in the person information storage unit 114, and estimate the age of the identified person image. This age estimation method, for example, compares the similarity of the images, and determines the age of the image with the closest similarity as the age of the identified person.

[0022] Note that part or all of the person detection unit 112 and the age estimation unit 113 may be realized by hardware, which is a circuit board such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc. In other words, the person detection unit 112 and the age estimation unit 113 may be configured by electronic circuits.

[0023] 5 is a diagram schematically illustrating an example of an irradiation range As10 of the transmitter / receiver 121 in the first measuring device 12 and an identified area As12. The area As12 is, for example, the chest area of ​​a person h10 identified by the processing unit 123.

[0024] 5, and again as shown in FIG. 2, the transceiver unit 121 of the first measuring device 12 can measure the distance to an object and the position of the object using a pair of two transceivers. For example, the processing unit 123 measures the position of, for example, a person h10 by the principle of triangulation using the two distances to the object of the two transceivers and the distance between the two transceivers.

[0025] The transceiver 121 generates a modulated wave (transmission wave) and transmits it to the irradiation range As10. For example, this transmission wave is a millimeter wave, and a frequency-continuous modulation (FMCW) method is used. The transceiver 121 then receives a reflected wave reflected from, for example, a person h10. The transceiver 121 mixes the transmission wave and the reception wave to generate an IF (intermediate frequency) signal and supplies it to the amplifier 122. The amplifier 122 amplifies the IF (intermediate frequency) signal, converts it into a digital signal in an A / D converter, and supplies it to the processing unit 123. Furthermore, the transceiver 121 sweeps and irradiates the modulated frequency of the irradiated radio wave, thereby irradiating radio waves with different phases to various targets and making it possible to distinguish reflections. In this case, the two transceivers of the transceiver 121 synchronize the modulation of the irradiated radio wave.

[0026] The processing unit 123 performs general signal processing based on the digital signals to generate a distance for each object, and then generates the position of the object using the distance for each object from the two transmitting and receiving units.

[0027] Furthermore, the processing unit 123 generates a time series of distance fluctuations to the measurement target as amplitudes of the measurement target based on the digital signal. Then, the processing unit 123 calculates the time series fluctuations of the amplitudes of the measurement target as frequencies. Note that the amplitude generation process can be performed using general processes (see, for example, Patent Document 1).

[0028] In the processing of the processing unit 123, the amplitude is basically 0 when there is no change in the distance between devices for a stationary object such as a machine. On the other hand, in the case of a human body, the distance between devices fluctuates due to contractions of blood vessels and the heart (heart rate) and changes in skin due to breathing, so the calculated amplitude fluctuates over time. Then, the processing unit 123 calculates the frequency corresponding to the heart rate / respiratory rate based on periodic information of the amplitude that fluctuates over time. Furthermore, for example, the processing unit 123 can use so-called Doppler FFT to measure the frequency and calculate information on the frequency corresponding to the heart rate / respiratory rate based on the periodicity of the change in the velocity of the object.

[0029] The processing unit 123 determines that the measurement target is a person if the generated frequency is within the range of the heart rate / respiratory rate. Through this processing, the chest region of the identified person h10 is As12 (see FIG. 5). The processing unit 123 supplies the monitoring device 13 with position information of the region As12 of the identified person h10, information on the frequency corresponding to the heart rate / respiratory rate, and the measurement time. The position information generated by the processing unit 123 includes, for example, the coordinates of a circumscribing rectangle of the chest region of the detected person and the position coordinates of the center of gravity of the circumscribing rectangle.

[0030] The storage unit 131 of the monitoring device 13 is realized by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, etc. As described above, the storage unit 131 stores the position information, age information, and time information of the person detected by the image capturing device 11, and the position information, heart rate, respiratory rate, and time information of the person within the measurement range calculated by the first measuring device 12. The storage unit 131 stores, for example, information on the standard heart rate range by age, the heart rate range in an abnormal state, the standard respiratory rate range, the respiratory rate range in an abnormal state, etc.

[0031] The determination unit 132 determines whether or not there is something wrong with a person within the monitoring range. More specifically, the determination unit 132 acquires information on the standard range of heart rate and respiratory rate corresponding to the age information of the person detected by the image capture device 11 from the storage unit 131. Then, the determination unit 132 acquires from the storage unit 131 the heart rate and respiratory rate associated with the area As12 corresponding to the position information (area ag10) of the people h10 and h20 detected by the image capture device 11. Then, the determination unit 132 determines that the person h10 is in an abnormal state if at least one of the heart rate and respiratory rate associated with the area As12 is not within the standard range.

[0032] If an abnormality is detected, the determination unit 132 reads the captured images of the persons h10 and h20 from the person information storage unit 110, writes the area of ​​the person h10 in the image, and outputs the image together with notification information indicating the reason for the abnormality determination, such as "abnormal heart rate" or "abnormal breathing," to the firing unit 133. That is, if at least one of the heart rate and breathing rate of the person h10 captured by the imaging device 11 is not within a standard range, the determination unit 132 generates notification information including information on the person's condition, such as "abnormal heart rate" or "abnormal breathing."

[0033] Furthermore, when there is something wrong with the person h10, that is, when the person h10 is in a predetermined state, the determination unit 132 writes the position coordinates of the person h10 as marks mh10 and mh20 on the plan view of the train 200, and outputs the marks as notification information to the firing unit 133. For example, while the people h10 and h20 are continuously being photographed as a video, the determination unit 132 writes the position coordinates of the people h10 and h20 as marks mh10 and mh20 on the plan view of the train 200, and continues to output the notification information to the firing unit 133. As described above, in this embodiment, person condition information such as "abnormal heartbeat" and "abnormal breathing" is included in the notification information. For example, the determination unit 132 generates, as notification information, an image signal including an image written as marks mh10 and mh20 on a plan view of the train 200 (see M12 in FIG. 4), and an image captured by the cameras 111a and 111b (see M14 in FIG. 4) in which person areas ag10 and ag20, face areas f10 and f20, and the person's condition such as "abnormal heartbeat" and "abnormal breathing" are written. At the same time, the determination unit 132 generates, as notification information, an audio signal including the person's condition information such as "abnormal heartbeat" and "abnormal breathing".

[0034] The firing unit 133 has a speaker. When the firing unit 133 receives notification information, it fires a warning sound corresponding to the notification information. This makes it possible to notify people around the monitoring device 13 that an abnormality has occurred. Note that the audio signal may be configured to be output only once to the same person h10. Note that the firing unit 133 can change the content of the warning sound according to status information associated with the status of an object included in the notification information. For example, the warning sound may be fired as "abnormal heartbeat" or "abnormal breathing."

[0035] Furthermore, the firing unit 133 transmits notification information including the audio signal and image signal supplied from the determination unit 132 to the display device 200. As described above, this audio signal is a signal including information indicating that an abnormality has occurred. As a result, the monitor 202 of the display device 200 displays information about the vehicle having an abnormality (see M12 and 14) as shown in FIG. 4. Furthermore, the speaker 203 of the display device 200 emits a warning sound corresponding to the audio signal indicating the person's condition, such as "abnormal heart rate" or "abnormal breathing." In this way, when an abnormality has occurred, the determination unit 132 causes the display device 200 to display information about the vehicle having an abnormality (see M12 and 14 in FIG. 4) together with a warning sound via the firing unit 133 as shown in FIG. 4. This allows the person monitoring the display device 200 to easily determine the condition of the person inside the vehicle 200.

[0036] Furthermore, the determination unit 132 and firing unit 133 may be realized by hardware, such as a circuit board, such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). Furthermore, the image capture device 11, the first measuring device 12, and the monitoring device 13 may share a CPU, a memory unit, etc. Alternatively, the image capture device 11, the first measuring device 12, and the monitoring device 13 may be configured by sharing a single circuit board, such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). The above is an explanation of an example configuration of the monitoring system 1, and below is an explanation of a processing example.

[0037] 6 is a flowchart showing an example of processing by the imaging device 11. The cameras 111a and 111b capture images of a range diagonally downward from their installation positions or a range in the horizontal direction, and store the images in the object information storage unit 114 (step S11). Each captured image is associated with the time of capture.

[0038] Next, the person detection unit 112 detects a person area and a face area within this person area from each image stored in the person information storage unit 114. When a person is detected, the person detection unit 112 generates the coordinates of a rectangular area including the person and the coordinates of the face area (step S12).

[0039] Next, the person detection unit 112 calculates the two-dimensional coordinates of the center of gravity (intersection of the diagonals) of the rectangle representing the person (step S13). The calculated coordinates indicate the position of the identified person on the image and become the reference coordinates of marks mh10, mh20 (see FIG. 4), etc.

[0040] Next, the age estimation unit 113 generates a feature based on the image corresponding to the coordinates of the face area, and estimates the age based on the feature (step S14). The age estimation unit 113 transmits the position coordinates of the identified person and the estimated age to the storage unit 131 (step S15). The image capturing device 11 performs and updates these processes at regular intervals.

[0041] 7 is a flowchart showing an example of processing by the first measuring device 12. In step S21, the transmitting / receiving unit 121 radiates radio waves downward relative to the installation direction and receives radio waves reflected within the irradiation range (step S21). The transmitting / receiving unit 121 mixes the transmitted wave and the received wave to generate an IF (intermediate frequency) signal and supplies it to the amplifier 122. In addition, by radiating the radiated radio waves by sweeping the modulated frequency, radio waves with different phases can be radiated to various targets, making it possible to distinguish reflections.

[0042] Next, the amplifier 122 amplifies the IF (intermediate frequency) signal, converts it into a digital signal in the A / D conversion section, and supplies it to the processing section 123 (step S22).

[0043] Next, the processing unit 123 calculates the time-series fluctuation of the distance value to the measurement target as the amplitude of the reflecting target based on the digital signal (step S23). Subsequently, the processing unit 123 calculates the frequency of the time-series amplitude fluctuation, and if the generated frequency is within the range of the heart rate / respiratory rate, determines that the measurement target is a person (step S24). That is, the processing unit 123 determines that the target is a person if the calculated amplitude is not 0 and the frequency is within the range of the heart rate / respiratory rate of the person.

[0044] Next, the processing unit 123 calculates the position of the object determined to be a person using the distance of each object from the two transmitting and receiving units. In this case, the processing unit 123 detects the entire continuous area that fluctuates with the same period as, for example, the chest area (step S24).

[0045] Next, the processing unit 123 supplies information on the position and area of ​​the object determined to be a person, and frequencies corresponding to the heart rate and respiratory rate, to the storage unit 131 of the monitoring device 13. The first measuring device 12 performs and updates these processes at regular intervals.

[0046] 8 is a flowchart showing an example of processing by the monitoring device 13. Here, the processing up to issuing an alarm will be described. The determination unit 132 reads out the position information of the person identified from the image captured by the imaging device 11 and the position information of the person calculated by the first measuring device 12, both of which are stored in the memory unit 131 (step S31).

[0047] The determination unit 132 compares the respective position coordinates, searches for a combination of corresponding position coordinate values, and associates the heart rate / respiratory rate calculated by the first measuring device 12 with the person image captured by the photographing device 11 (step S32).

[0048] Next, the determination unit 132 reads out the standard heart rate / respiratory rate from the storage unit 131 based on the estimated age of the associated person image (step S33). The standard heart rate / respiratory rate of the human body varies depending on age, for example, the heart rate of an infant (1 to 3 years old) is 120 to 140, and that of an adolescent is 70 to 90, etc.

[0049] Next, the determination unit 132 compares the heart rate / respiratory rate of the person calculated by the first measuring device 12 with the standard heart rate / respiratory rate for a certain period of time, and determines whether they are within the set threshold range (step S34). If the determination result is determined to be normal (Yes in step S34), the determination unit 132 repeats the process from step S31.

[0050] On the other hand, if the determination result is determined to be abnormal (No in step S34), the determination unit 132 transmits notification information including the status information to the firing unit 133 and fires a shot to notify the abnormality. At the same time, the position information of the person, the time, etc. are stored in the storage unit 131.

[0051] In this way, by monitoring the position and heart rate / respiratory rate of a person within the measurement range, it is possible to quickly determine where on the image a person experiencing an abnormality in physical condition is located. This allows for early response. In addition, since the gun is fired at the same time, it is possible to notify people in the monitoring room, driver's cab, and surrounding areas such as the monitoring device 13 that an abnormality has occurred. In particular, it is possible to notify the monitor that an abnormality has occurred along with image information via the display device 200 installed in the monitoring room, driver's cab, etc. This allows the monitor to quickly decide what to do depending on the situation, such as stopping the train or opening and closing the doors.

[0052] FIG. 9 is a top view showing another example of the arrangement of multiple cameras 111a, 111b, and 111c and multiple transmitters / receivers 121 within a vehicle 200. In FIG. 9, the image capturing device 11 differs from the example of FIG. 3 in that it includes multiple cameras 111c. The cameras 111c are, for example, cameras with fisheye lenses and are arranged at the intersections of lines L200 and L202. This allows the multiple cameras 111c to capture images of almost all passengers entering and exiting through the opening and closing doors from above. In other words, the multiple cameras 111c are installed in positions where they can capture images of almost all passengers entering and exiting through the opening and closing doors from above. Each camera 111c is assigned an imaging area A200. In this embodiment, coordinates within an image captured by the camera 111c are associated in advance with planar coordinates within the imaging area A200. The images captured by the multiple cameras 111a, 111b, and 111c are associated with the camera that captured them and the time of capture, and are stored in the person information storage unit 114. Furthermore, the images taken by the multiple cameras 111c are directly below the vehicle, and overlap with the measurement ranges of the multiple transmitter / receivers 121, making it possible to more accurately associate people. This makes it possible to capture images of passengers in more detail even when the number of passengers in the vehicle 200 increases, further improving the determination accuracy of the monitoring system 1.

[0053] As described above, according to this embodiment, the image capturing device 11 captures an image of the space to be monitored, the first measuring device 12 measures the state of a person in the image, and the monitoring device 13 generates notification information when the person in the image is in a predetermined state. This uses the results of the state measurement of the person in the image in addition to the image information, making it possible to generate notification information according to the state of the person in the image with higher accuracy.

[0054] (Second embodiment) The monitoring system 1 according to the second embodiment differs from the monitoring system 1 according to the first embodiment in that it further includes a scream detection device (second measuring device) 14. The differences from the monitoring system 1 according to the first embodiment will be described below.

[0055] Fig. 10 is a block diagram showing an example of the configuration of a monitoring system 1 according to the second embodiment. As shown in Fig. 10, the monitoring system 1 according to the second embodiment further includes a scream detection device 14 in addition to an imaging device 11, a first measuring device 12, and a monitoring device 13. Note that the first measuring device 12 and the scream detection device (second measuring device) 14 according to this embodiment correspond to the measuring devices.

[0056] The scream detection device 14 has a sound collection unit 141, a sound information storage unit 142, a scream detection unit 143, and a second firing unit 144. The sound collection unit 141 is, for example, a microphone. The microphone may be directional or omnidirectional. Multiple microphones may be installed. The sound collection unit 141 records the collected voice of a person and surrounding sounds.

[0057] The audio information storage unit 142 is realized by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, etc. The audio information storage unit 142 stores sound data of various types of screams that have been recorded in the past.

[0058] The scream detection unit 143 detects a predetermined sound, for example, a scream, by comparing the collected sound data with the scream data in the audio information storage unit 142. A scream is, for example, a sound uttered by a person to notify the person of danger or crisis.

[0059] The scream detection unit 143 stores the time when the scream was detected and the position of the sound collection unit 141 where the scream was collected in the memory unit 131 of the monitoring device 13, and also supplies this to the determination unit 132 of the monitoring device 13. Note that the scream detection unit 143 may be realized by hardware that is a circuit board such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0060] When the scream detection unit 143 detects a scream, the second firing unit 144 transmits an audio signal indicating that a scream has been detected to the display device 200. The second firing unit 144 also has a speaker, and can emit an audio signal indicating that a scream has been detected using this speaker. Furthermore, the determination unit 132 of the monitoring device 13 may display, on the monitor 202, an image M14 (see FIG. 4) of the vehicle that includes the position of the audio collection unit 141 that collected the scream. This allows the person monitoring the display device 200 to more easily grasp the situation within the monitored space.

[0061] 11 is a flowchart showing an example of processing by the scream detection device 14. The sound collection unit 141 records sounds around the installation location (step S41). The recorded sounds are digitized using a processing method such as PCM (Pulse Code Modulation) or DSD (Direct Stream Digital), and transmitted to the scream detection unit 143.

[0062] Next, the scream detection unit 143 calculates a power spectrum from the digitized sound data and compares the calculated power spectrum with the sound data stored in the audio information storage unit 142 (step S42). Subsequently, the scream detection unit 143 determines whether the recorded sound data is a scream (step S43). If the scream data stored in the audio information storage unit 142 is similar to the recorded sound data, the scream detection unit 143 determines that the sound data is a scream (Yes in step S43). Subsequently, the scream detection unit 143 transmits a firing command to the second firing unit 144, and the second firing unit 144 fires an alarm (step S44).

[0063] On the other hand, if the scream data stored in the audio information storage unit 142 and the recorded sound data are not similar, the scream detection unit 143 determines that it is not a scream (No in step S43). If the scream detection unit 143 determines that it is not a scream, it repeats the process from step S41. Furthermore, based on the detection result of the scream detection unit 143, the determination unit 132 of the monitoring device 13 includes, for example, "scream" as status information of the person emitting the predetermined sound in the notification information.

[0064] In this way, by detecting the screams of a person within the measurement range, it is possible to quickly determine that an abnormality has occurred in the surrounding area, enabling early response. In addition, by firing a gun at the same time, it is possible to notify the surrounding area that an abnormality has occurred.

[0065] As described above, according to this embodiment, when the scream detection device (second measurement device) 14 detects a predetermined sound (for example, a scream), status information is generated that associates the person in the image with the state in which the person is making the predetermined sound. This makes it possible to generate notification information using the status information of the person making the predetermined sound in addition to the image information.

[0066] (Third embodiment) The monitoring system 1 according to the third embodiment includes a scream source location identifying unit 145 that identifies the location of the sound source. The monitoring system 1 according to the second embodiment differs from the monitoring system 1 according to the second embodiment in that it further comprises the following. The differences from the monitoring system 1 according to the second embodiment will be described below.

[0067] Fig. 12 is a block diagram showing the configuration of a monitoring system 1 according to the third embodiment. As shown in Fig. 12, the scream detection device 14 according to the third embodiment further includes a scream generation position identification unit 145 that identifies the position of the person who has generated the scream.

[0068] FIG. 13 is a top view showing an example of the arrangement of multiple cameras 111a and 111b, multiple transmitter / receivers 121, and multiple sound collection units 141 inside a vehicle 200 according to the third embodiment. Line L200 indicates the middle of the opposing opening and closing doors. That is, the vehicle 200 has, for example, eight opening and closing doors. Line L202 indicates the horizontal center line of the vehicle 200. As shown in FIG. 13, two sound collection units 141a and 141b, and two sound collection units 141c and 141d are arranged in pairs, respectively.

[0069] Each of the multiple transmitter / receivers 121 is disposed at the intersection of line L200 and line L202, and also at an intermediate portion thereof, thereby enabling more detailed measurement of almost all passengers in vehicle 200 from above diagonally downward.

[0070] Fig. 14 is a diagram showing an example of an image captured by camera 111b. This image includes line L13 passing through person h10 in Fig. 13, and was captured from a direction substantially facing line L13. Sound areas Av10 to Av16 are schematic examples of areas identified by scream generation position identification unit 145.

[0071] FIG. 15 is a diagram showing an example of a screen displayed on the display device 200 by the monitoring device 13 according to the third embodiment. Screen M10 shows the screen of the monitor 201 (see FIG. 12), screen M12 is a top view of the vehicle 200, and mark mh10 indicates the position of person h10 who uttered the scream. Images captured by cameras 111a and 111b are alternately displayed on screen M14. Area ag10 indicates the area of ​​person h10 detected by the person detection unit 112. Area f10 indicates the face area within area ag10 detected by the person detection unit 112.

[0072] 12, scream generation position identification unit 145 calculates the distance between sound collection unit 141a and sound collection unit 141b from the time difference between the screams recorded by sound collection units 141a, b, 141c, and d (see FIG. 13) and the corresponding sound data, and estimates the position of the sound source using the principle of triangulation. Similarly, scream generation position identification unit 145 calculates the distance between sound collection unit 141c and sound collection unit 141d, and estimates the position of the sound source using the principle of triangulation. More specifically, scream detection unit 143 performs scream recognition processing on the recorded sounds of sound collection units 141a, b, 141c, and d (see FIG. 13), and supplies the detection times corresponding to each of sound collection units 141a, b, 141c, and d to scream generation position identification unit 145.

[0073] The scream generation position identifying unit 145 identifies the planar coordinates of the sound source in the vehicle 200 according to the principle of triangulation using, for example, the distance at which a difference in detection time occurs between the sound collecting unit 141a and the sound collecting unit 141b, and the distance between the sound collecting unit 141a and the sound collecting unit 141b. Similarly, the scream generation position identifying unit 145 identifies the planar coordinates of the sound source in the vehicle 200 according to the principle of triangulation using the distance at which a difference in detection time occurs between the sound collecting unit 141c and the sound collecting unit 141d, and the distance between the sound collecting unit 141c and the sound collecting unit 141d. The scream generation position identifying unit 145 supplies the position coordinates and time of the sound source to the memory unit 131 of the monitoring device 13.

[0074] Based on the position coordinates of the sound source, the determination unit 132 of the monitoring device 13 selects one sound area Av13 from among the sound areas Av10 to Av16 to which the position coordinates of the sound source belong. Then, the determination unit 132 identifies a person h10 having coordinates close to the position coordinates of the sound source from the selected sound area Av13.

[0075] The determination unit 132 reads the captured image of the identified person h10 from the person information storage unit 110, writes the area of ​​the person h10 in the image, and outputs the image as notification information to the firing unit 133 together with status information such as the reason for determining the abnormality, for example, "scream," "abnormal heart rate," or "abnormal breathing." Furthermore, if an abnormality is detected, the determination unit 132 writes the position coordinates of the person h10 as a mark mh10 on the plan view of the train 200, and outputs the image signal to the firing unit 133. For example, while the person h10 is continuously being photographed as a video, the determination unit 132 writes the position coordinates of the person h10 as a mark mh10 on the plan view of the train 200, and continues to output the image signal to the firing unit 133.

[0076] The firing unit 133 transmits an image signal containing the image information and an audio signal supplied from the determination unit 132 to the display device 200. This audio signal is a signal containing status information indicating the occurrence of an abnormality, as described above. Thus, when an abnormality is detected, the determination unit 132 causes the display device 200 to display information about the vehicle with the abnormality along with an audible warning via the firing unit 133, as shown in FIG. 15 . In this case, the firing unit 133 and the second firing unit 144 may emit an audible warning. Firing a gun in this manner can also notify the monitoring device 13 and people around the display device 200 that an abnormality has occurred. Note that the audio signal can be configured to be output only once to the same person h10, for example. Furthermore, the monitor monitoring the display device 200 can easily grasp the situation inside the vehicle 200 from the image, allowing them to easily make decisions based on the situation, such as stopping the train or opening and closing the doors. Furthermore, if an abnormality occurs, it is only necessary to check the image of the location where the abnormality occurred, so there is no need to constantly monitor images inside the entire vehicle, and during normal operation, other operations, such as driving operations, are prevented from being hindered.

[0077] When images from both cameras 111a and 111b are available, determination unit 132 may preferentially display the image from camera 111b, which is closer to line L13 (see FIG. 13), on display device 200. For example, when displaying images in chronological order, four images from camera 111b may be displayed consecutively, followed by one image from camera 111a. This makes it easier to grasp the situation.

[0078] Fig. 16 is a flowchart showing the processing of the scream detection device 14 according to the third embodiment. Steps S41 to S43 perform the same processing as in Fig. 11. If it is determined that a scream has occurred (Yes in step S43), the scream occurrence position identification unit 145 identifies the coordinates of the scream source in the vehicle 200 by the principle of triangulation using, for example, the distance at which the detection time difference occurs between the sound collection unit 141a and the sound collection unit 141b, and the distance between the sound collection unit 141a and the sound collection unit 141b (step S54).

[0079] Scream occurrence position identification unit 145 transmits information on the calculated scream occurrence coordinates to storage unit 131 of monitoring device 13. Then, scream detection unit 142 transmits a firing command to second firing unit 144, which then fires an alarm to warn of danger (step S55). Furthermore, determination unit 132 of monitoring device 13 causes display device 200, via firing unit 133, to emit a sound including status information and display image information, etc.

[0080] 17 is a flowchart showing an example of processing by the monitoring device 13 according to the third embodiment. The determination unit 132 reads out, from the storage unit 131, the position coordinates of the person on the image calculated by the image capture device 11 and the scream occurrence position information calculated by the scream detection device 14 (step S61).

[0081] Next, the determination unit 132 searches for a combination that is closest to the position coordinates of the person on the image calculated by the image capture device 11 and the scream occurrence position calculated by the scream detection device 14, and if a matching combination is found, it determines that the person has screamed (step S62). Subsequently, the determination unit 132 transmits the position information and time of the person who emitted the scream to the storage unit 131 (step S63). Then, the determination unit 132 transmits a firing command to the firing unit 133, and the firing unit 133 fires a shot to warn of danger (step S64).

[0082] In this way, by monitoring the location of people within the measurement range and whether or not they are screaming, it is possible to quickly determine where on the image the person who is screaming is located, enabling early response.In addition, since a shot is fired at the same time, it is also possible to notify the user of any abnormalities or dangers in the surrounding area.

[0083] Although a detailed explanation will be omitted, it is also possible to link the results of the scream detection device 14 and the first measuring device 12 in addition to the results of the imaging device 11. This makes it possible to identify (location, person, condition) a person who is crying out in distress due to a change in their physical condition, for example, and thus makes it possible to respond to a variety of situations.

[0084] 18 to 21, other examples of the arrangement of the camera 111, the transmitter / receiver 121, and the voice collecting unit 141 will be described. Fig. 18 shows pattern P18 in which the voice collecting unit 141 is arranged in the door section, and pattern P19 in which the voice collecting unit 141 is arranged at the intersection of line L200 and line L202.

[0085] In pattern P18, the camera 111 and the transmitter / receiver 121 are placed in the center of the ceiling in the passageway near the entrance / exit door, and the image is taken directly below, allowing for more accurate correspondence with the sound source position. Similarly, by generating millimeter waves in the same range as the camera 111, it becomes possible to more accurately correspond the subject area of ​​the image with the chest area of ​​the heartbeat source. In addition, since the sound collector 141 is placed at the end, it is possible to suppress the input of reflected waves.

[0086] On the other hand, pattern P19 differs from pattern P18 in that the sound collection unit 141 is also placed in the center of the ceiling. In pattern P19, the sound collection area of ​​the sound collection unit 141, the imaging area of ​​the camera 111, and the millimeter wave generation range of the transmitter / receiver 121 can be made equivalent, allowing for more accurate correspondence between the measurement targets. In addition, because the sound collection unit 141 is placed in the center of the ceiling, direct sound can be collected more efficiently.

[0087] FIG. 19 is a diagram showing an example in which a sound collection unit 141 is further arranged in the middle in the traveling direction of the vehicle 200. This differs from the arrangement pattern in FIG. 13 in that a sound collection unit 141 is further arranged in the middle. Since the sound collection unit 141 is arranged in the center of the ceiling, direct sound can be collected more efficiently. Furthermore, the sound collected by the sound collection units 141 at the ends can be supplemented by the sound collection units 141 arranged in the center of the ceiling, making it possible to identify the position of the sound source with higher accuracy.

[0088] Fig. 20 is a diagram showing an example of the layout of vehicle 200 in which seats face in the direction of travel or opposite to the direction of travel. That is, Fig. 20 is a diagram showing an example of the layout when there are no opening and closing doors on the side of vehicle 200 along the direction of travel, such as in a Shinkansen train. Camera 151 is a camera installed on the door side. Image capturing device 11 can also refer to images from camera 151.

[0089] The camera 111c is disposed in the center of the vehicle 200. The camera 111c disposed in the center can supplement the image of the camera 111am disposed near the door. Furthermore, the sound collection units 141a-f are disposed near the doors between the vehicles and in the center of the vehicle 200. The sound collection units 141e and 141f disposed in the center can supplement the sound collection by the sound collection units 141a-d disposed near the doors. Furthermore, because the cameras 111a, 111b, and 111c, the transmitting / receiving unit 121, and the sound collection units 141a-141f are disposed symmetrically with respect to the traveling direction of the vehicle 200, fluctuations in measurement accuracy are suppressed even if the traveling direction changes.

[0090] FIG. 21 is a diagram showing an example in which the camera 111 and the voice collecting unit 141 are arranged in the longitudinal center of the vehicle 200. The camera 111e near the door is a camera with a fisheye lens, and the camera 111f is a camera with a normal lens. Furthermore, since the four cameras 111e and 111f are arranged above the aisle, the capturing range of the captured images is approximately 5 m apart, making it possible to capture images of seated people with higher accuracy. Furthermore, since the aisle camera 111f can capture images directly below, if the position of the sound source can be identified using the voice collecting unit 141, the accuracy of matching can be improved.

[0091] The monitoring system 1 according to this embodiment has been described using the vehicle 200 as an example, but is not limited to this. For example, the monitoring system 1 can also be placed in a monitored space where an unspecified number of people enter and exit, such as an elevator hall.

[0092] 22 is a diagram showing an example of the arrangement of cameras 111a, b, transmitter / receiver 121, and sound collectors 141a-d in an elevator hall 300. The arrangement of cameras 111a, b, and sound collectors 141a-d is the same as that shown in FIG. 13, etc., and the monitoring system 1 can capture images of people in the elevator hall 300, identify their areas, and obtain location information of people who have screamed or made other noises. In addition, because the transmitter / receiver 121 is placed in front of the elevator doors, the heart rates of people waiting for the elevator 302 can be measured with higher accuracy.

[0093] As described above, according to this embodiment, when the scream detection device (second measurement device) 14 detects a predetermined sound (for example, a scream), the scream generation position identification unit 145 identifies the position of the sound source. This makes it possible to generate status information that associates the position of the sound source with the position of a person in an image. Therefore, the status information makes it possible to confirm the status of the sound source from the image.

[0094] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0095] 1...surveillance system, 11...imaging device, 12...first measuring device, 13...surveillance device, 14...scream detection device (second measuring device), 111a, b, c, e, f...camera, 112...person detection unit, 113...age estimation unit, 114...person information storage unit, 121...transmitting / receiving unit, 122...amplifier, 123...processing unit, 131...storage unit, 132...determination unit, 133...gun firing unit, 141a, b, c, d...audio collection unit, 142...scream detection unit, 143...audio information storage unit, 144...second gun firing unit, 145...scream generation location identification unit.

Claims

1. an image capturing device that captures an image associated with coordinates in a space to be monitored, and outputs, based on the image, position information of a person area in the image and an estimated age of the person in the person area; a measurement device that outputs measurement area information of the person and at least one of a heart rate and a respiratory rate of the person associated with the measurement area information based on the received signal; A monitoring device; With The monitoring device a storage unit that stores position information of the person area output by the image capturing device, an estimated age of the person, and at least one of a heart rate and a respiratory rate of the person associated with the measurement area information of the person; a determination unit that determines, using the information stored in the storage unit, that at least one of a heart rate and a respiratory rate of the person associated with the measurement area information corresponding to position information of the person area is abnormal when the heart rate and the respiratory rate are not within a standard range for the estimated age of the person; A monitoring system having:

2. The monitoring system according to claim 1 , wherein when the monitoring device determines that an abnormality has occurred, the monitoring device generates, as notification information, information relating to the position of the person in the image and a state of the person.

3. The surveillance system according to claim 2 , wherein the image capturing device includes an age estimation unit that estimates an estimated age of the person in the image.

4. the age estimation unit estimates the estimated age based on image information within the person area in the image, The monitoring system according to claim 3 , wherein the monitoring device generates the notification information by associating the position information of the person area with the measurement area information.

5. The measurement device detects a predetermined sound, The monitoring system according to claim 1 , wherein the monitoring device generates notification information when the predetermined sound is detected.

6. the predetermined sound is a scream that notifies a person of danger or crisis, The monitoring system according to claim 5 , wherein when the monitoring device detects the scream, the monitoring device generates the notification information that associates the person in the image with a state in which the person is emitting the predetermined sound.

7. The monitoring system according to claim 6 , wherein the monitoring device generates a sound in response to the notification information.

8. The monitoring system according to claim 7 , wherein the monitoring device, when determining that an abnormality has occurred, causes a display device to display the image including the person.

9. The monitoring system according to claim 8 , wherein the notification information includes coordinate information within a vehicle or an elevator hall that indicates the position of the person.

10. The monitoring system according to claim 9 , wherein the image capturing device includes a camera that captures the image inside the vehicle or the elevator hall.

11. The monitoring system according to claim 10 , wherein the camera is disposed at an end of the vehicle in a traveling direction.

12. The monitoring system according to claim 10 , wherein the camera is disposed within a range capable of capturing an image of an opening and closing door of the vehicle.

13. 13. The surveillance system according to claim 11 or 12, wherein the camera captures images through a fisheye lens.

14. The measuring device is a transmitting / receiving unit that transmits and receives predetermined radio waves; a processing unit that generates a fluctuation period of the distance to the person as at least one of a heart rate and a respiratory rate of the person based on the predetermined radio wave, The monitoring system according to claim 9, wherein the transmitter / receiver is arranged within a range in which the radio waves can be transmitted to a door of the vehicle and within a range in which the radio waves can be transmitted to a door of an elevator in an elevator hall.

15. The measuring device is a voice collection unit that collects voice; a detection unit that detects a predetermined sound from the sound; an identification unit that identifies the position of a sound source that is emitting the sound, The monitoring system according to claim 9 , wherein the sound collecting unit is disposed at least one of an end of the vehicle in a traveling direction and an end of an elevator hall.

Images