Monitoring device, monitoring method, and program

Abstract

Provided is novel technology by which the severity of an abnormal situation that has occurred can be known. A monitoring device (1) comprises: a position acquisition unit (2) that acquires the position of occurrence of an abnormal situation in an area being monitored; an analysis unit (3) that analyzes the condition of a crowd in the vicinity of the position of occurrence of the abnormal situation on the basis of video data from a camera for filming the area being monitored; and a severity estimation unit (4) that estimates the severity of the abnormal situation on the basis of the analysis results.

Description

[Technical field]

[0001] The present disclosure relates to a monitoring device, a monitoring system, a monitoring method, and a non-transitory computer-readable medium having a program stored thereon. [Background technology]

[0002] In recent years, crimes such as terrorism, assault, and molestation have been increasing in public places such as on city streets, at stations, and on trains, but at the same time, unmanned surveillance is becoming more common due to a labor shortage, and human surveillance is not able to keep up. To compensate for this, surveillance methods have been devised that install security cameras and microphones and use programs to analyze captured images and sounds to detect abnormalities (for example, Patent Document 1).

[0003] Generally, when detecting an abnormality from a video, video data from a surveillance camera is collected via a network and analyzed by a computer, as described in Patent Document 1. In video analysis, video features that may be linked to danger, such as a facial image of a specific person, abnormal behavior of one or more people, or an object left behind in a specific location, are registered in advance, and the presence of the features is detected.

[0004] In addition to video, anomaly detection is also performed using sound, as in Patent Document 1. There are voice recognition, which recognizes and analyzes the contents of a person's speech, and acoustic analysis, which analyzes sounds other than voice, but neither of these requires significant computer resources. For this reason, even an embedded central processing unit (CPU) such as those installed in smartphones is fully capable of analyzing in real time.

[0005] Detecting the occurrence of abnormalities by analyzing sound is also effective for unexpected abnormalities, because it is a universal law of nature that people who encounter an abnormality will scream or yell, and that abnormalities will produce loud abnormal sounds such as bursts, explosions, gunshots, and shattered glass.

[0006] In addition, sound has the property of diffusing in all directions (360 degrees), propagating even in the dark, and going around obstacles in the way. For this reason, sound surveillance is not limited by the field of view, direction, or lighting as with cameras, and abnormal sounds occurring in the dark or behind objects are not overlooked, making it an excellent choice for surveillance.

[0007] Furthermore, when collecting sound using multiple microphones, as disclosed in Patent Document 2, the position of the sound source can be estimated from the difference in arrival time of the sound from the sound source to each microphone, and the sound pressure difference due to diffusion and attenuation of the sound.

[0008] Moreover, Patent Document 3 discloses a technology called gaze estimation, which estimates the gaze direction from a human face image.

[0009] Furthermore, Patent Document 4 discloses a technique for recognizing facial expressions from human face images, called facial expression recognition. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] JP 2013-131153 A [Patent Document 2] Special Publication No. 2013-545382 [Patent Document 3] JP 2021-61048 A [Patent Document 4] International Publication No. 2019 / 102619 Summary of the Invention [Problem to be solved by the invention]

[0011] However, in the case of anomaly detection by sound, it is possible to know that there is a high possibility that some kind of abnormality has occurred, but it is not possible to know the seriousness of the situation. Anomaly detection by sound is similar to when a person closes their eyes and listens carefully, and although it is possible to recognize that there is a high possibility that an abnormality has occurred from sounds such as screams and popping sounds, it is not possible to grasp the situation in more detail than that. Therefore, it is difficult to grasp the seriousness of the situation from the sound, such as whether security guards should be dispatched immediately or whether it is a minor abnormality that can be checked by waiting until the next day.

[0012] Therefore, one of the objectives that the embodiments disclosed in this specification aim to achieve is to provide a novel technology that can inform the severity of an abnormal situation that has occurred. [Means for solving the problem]

[0013] A monitoring device according to a first aspect of the present disclosure includes: A position acquisition means for acquiring a position where an abnormality has occurred in a monitored area; an analysis means for analyzing the state of the crowd around the location where the abnormal situation has occurred based on image data from a camera that photographs the area to be monitored; a severity estimation means for estimating the severity of the abnormal situation based on a result of the analysis; has.

[0014] A monitoring system according to a second aspect of the present disclosure includes: A camera for photographing the area to be monitored; A sensor that detects sound or heat generated in a monitored area; Surveillance equipment and Equipped with The monitoring device includes: a position acquisition means for acquiring a position where an abnormality has occurred in the monitoring target area by estimating a source of the sound or heat detected by the sensor; an analysis means for analyzing the state of the crowd around the location where the abnormal situation has occurred based on the video data from the camera; a severity estimation means for estimating the severity of the abnormal situation based on a result of the analysis; has.

[0015] In a monitoring method according to a third aspect of the present disclosure, Acquire the location of the abnormality in the monitored area, Based on image data from a camera capturing the monitored area, the state of the crowd around the location where the abnormal situation occurred is analyzed; Based on the results of the analysis, a severity of the abnormal situation is estimated.

[0016] A program according to the fourth aspect of the present disclosure includes: a position acquisition step of acquiring a position where an abnormality has occurred in a monitoring target area; an analysis step of analyzing the state of the crowd around the location where the abnormal situation has occurred based on image data from a camera photographing the monitored area; a severity estimation step of estimating the severity of the abnormal situation based on a result of the analysis; to be executed by the computer. Effect of the Invention

[0017] According to the present disclosure, a novel technique can be provided that allows one to know the seriousness of an abnormal situation that has occurred. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of a monitoring device according to an overview of an embodiment. [Diagram 2] 1 is a flowchart showing an example of an operation flow of a monitoring device according to an overview of an embodiment. [Diagram 3] FIG. 1 is a schematic diagram illustrating an example of a configuration of a monitoring system according to an embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of a functional configuration of an acoustic sensor. [Diagram 5] 2 is a block diagram illustrating an example of a functional configuration of an analysis server. [Figure 6] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a computer. [Figure 7] 1 is a flowchart illustrating an example of an operation flow of the monitoring system according to an embodiment. [Figure 8] 8 is a flowchart showing an example of the process flow in step S104 in the flowchart shown in FIG. 7. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] <Outline of the embodiment> Before describing the details of the embodiment, an overview of the embodiment will be described. Fig. 1 is a block diagram showing an example of the configuration of a monitoring device 1 according to the overview of the embodiment. As shown in Fig. 1, the monitoring device 1 has a position acquisition unit 2, an analysis unit 3, and a severity estimation unit 4, and is a device for monitoring a predetermined monitoring target area.

[0020] The position acquisition unit 2 acquires the position where the abnormality occurred in the monitoring target area. The position acquisition unit 2 may acquire information indicating the position where the abnormality occurred by any method. For example, the position acquisition unit 2 may acquire the position where the abnormality occurred by estimating the position where the abnormality occurred based on any information, or may acquire the position where the abnormality occurred by receiving input information on the position from a user or another device.

[0021] The analysis unit 3 analyzes the state of the crowd around (surrounding) the position where the abnormality occurred based on the video data of the camera that captures the monitored area. Here, the crowd around the position where the abnormality occurred does not mean, for example, people at the position where the abnormality occurred, but people who are far from the position where the abnormality occurred and are in the vicinity. For example, it corresponds to people who are at least one meter away from the position where the abnormality occurred and within a five meter radius. In other words, the crowd around the position where the abnormality occurred can be defined as people who are at least a first predetermined distance away from the position where the abnormality occurred and within a second predetermined distance from the position where the abnormality occurred. In addition, the state of the crowd specifically refers to a state that appears in the appearance of the people that make up the crowd, and may be, for example, the line of sight of the people or the facial expression of the people. In this way, the analysis unit 3 does not analyze the situation of the position where the abnormality occurred or the facial features and behavior of the people at the position where the abnormality occurred from the camera video, but analyzes the state of the crowd around the position where the abnormality occurred.

[0022] The seriousness estimation unit 4 estimates the seriousness of the abnormal situation based on the analysis result of the analysis unit 3. In general, the reaction of the crowd around the point where the abnormal situation occurs changes depending on the seriousness of the abnormal situation. For example, the higher the seriousness, the more the gazes of the crowd are focused on the point where the abnormal situation occurs, and the more the crowd shows unpleasant expressions. In this way, the seriousness estimation unit 4 of the monitoring device 1 estimates the seriousness of the abnormal situation by utilizing such universal natural laws that are generally observed in animals in response to abnormal situations.

[0023] 2 is a flowchart showing an example of the flow of operations of the monitoring device 1 according to the outline of the embodiment. Hereinafter, an example of the flow of operations of the monitoring device 1 will be described with reference to FIG.

[0024] First, in step S11, the position acquisition unit 2 acquires the position where the abnormal situation has occurred in the monitored area. Next, in step S12, the analysis unit 3 analyzes the state of the crowd around the position where the abnormal situation has occurred, based on the image data of the camera capturing the monitored area. Next, in step S13, the seriousness estimation unit 4 estimates the seriousness of the abnormal situation based on the result of the analysis in step S12.

[0025] The above describes the monitoring device 1 according to the outline of the embodiment. As described above, the monitoring device 1 makes it possible to know the severity of an abnormal situation that has occurred.

[0026] <Details of the embodiment> Next, the details of the embodiment will be described. 3 is a schematic diagram showing an example of the configuration of a monitoring system 10 according to an embodiment. In this embodiment, the monitoring system 10 includes an analysis server 100, a monitoring camera 200, and an acoustic sensor 300. The monitoring system 10 is a system for monitoring a predetermined monitoring target area 90. The monitoring target area 90 is any area where monitoring is performed, and is, for example, an area where the public may be present, such as a train station, an airport, a stadium, or a public facility.

[0027] The surveillance camera 200 is a camera installed to capture images of the monitored area 90. The surveillance camera 200 captures images of the monitored area 90 and generates video data. The surveillance camera 200 is installed in an appropriate position that allows monitoring of the entire monitored area 90. Note that a plurality of surveillance cameras 200 may be installed to monitor the entire monitored area 90.

[0028] In this embodiment, the acoustic sensor 300 is provided at various locations within the area 90 to be monitored. Specifically, the acoustic sensor 300 is provided at intervals of, for example, about 10 to 20 meters. The acoustic sensor 300 collects and analyzes sounds in the area 90 to be monitored. Specifically, the acoustic sensor 300 is a device that is composed of a microphone, a sound device, a CPU, and the like, and senses sounds. The acoustic sensor 300 collects surrounding sounds with a microphone, converts the sounds into digital signals with a sound device, and then performs acoustic analysis with the CPU. In this acoustic analysis, abnormal sounds such as screams, shouts, explosions, bursting sounds, and glass shattering sounds are detected. The acoustic sensor 300 may be equipped with a voice recognition function. In this case, more advanced analysis is possible, such as recognizing the contents of speech such as shouts and estimating the severity of an abnormal situation.

[0029] In this embodiment, the acoustic sensors 300 are installed at various locations in the monitoring area 90 at intervals of about 10 to 20 meters so that the multiple acoustic sensors 300 can detect any abnormal sound that occurs anywhere in the area. In general, noise in public facilities is about 60 decibels, while screams and shouts are about 80 to 100 decibels, and explosions and popping sounds are 120 decibels or more. However, for example, if the location where the sound is generated is 10 meters away, an abnormal sound that was 100 decibels near the sound source will attenuate to 80 decibels. Here, if the distance from the sound source to the acoustic sensor 300 is too far, it becomes difficult to distinguish the attenuated abnormal sound from the background noise of about 60 decibels at the location of the acoustic sensor 300. For this reason, in this embodiment, the acoustic sensors 300 are arranged at the intervals described above. Note that the distance between the multiple acoustic sensors 300 that can detect the same abnormal sound depends on the background noise level and the performance of each acoustic sensor 300, so there is no restriction to a placement distance of 10 to 20 meters.

[0030] The analysis server 100 is a server for analyzing data obtained by the surveillance camera 200 and the acoustic sensor 300, and has the functions of the surveillance device 1 shown in FIG. 1. The analysis server 100 receives analysis results from the acoustic sensor 300, and acquires video data from the surveillance camera 200 as necessary to analyze the video. The analysis server 100 and the surveillance camera 200 are communicatively connected via a network 500. Similarly, the analysis server 100 and the acoustic sensor 300 are communicatively connected via the network 500. The network 500 is a network that transmits communication between the surveillance camera 200, the acoustic sensor 300, and the analysis server 100, and may be a wired network or a wireless network.

[0031] Fig. 4 is a block diagram showing an example of the functional configuration of the acoustic sensor 300. Fig. 5 is a block diagram showing an example of the functional configuration of the analysis server 100.

[0032] As shown in FIG. 4, the acoustic sensor 300 includes an abnormality detection unit 301 and an abnormality determination unit 302. The abnormality detection unit 301 detects the occurrence of an abnormal situation in the monitored area 90 based on the sound detected by the acoustic sensor 300. The abnormality detection unit 301 detects the occurrence of an abnormal situation by, for example, determining whether or not the sound detected by the acoustic sensor 300 corresponds to a predetermined abnormal sound. That is, when the sound detected by the acoustic sensor 300 corresponds to a predetermined abnormal sound, the abnormality detection unit 301 determines that an abnormal situation has occurred in the monitored area 90. In addition, in this embodiment, when the abnormality detection unit 301 determines that an abnormal situation has occurred, it calculates a score indicating the degree of abnormality. For example, the abnormality detection unit 301 may calculate a higher score as the abnormal sound becomes louder, may calculate a score according to the type of abnormal sound, or may calculate a score by a combination of these.

[0033] When the occurrence of an abnormality is detected, the abnormality determination unit 302 determines whether or not a response to the abnormality is necessary. For example, the abnormality determination unit 302 performs this determination by comparing the score calculated by the abnormality detection unit 301 with a preset threshold value. That is, when the calculated score is equal to or less than the threshold value, the abnormality determination unit 302 determines that a response to the detected abnormality is unnecessary. In this case, no further processing is performed in the monitoring system 10. In contrast, when it is determined that a response to the abnormality is unnecessary, the acoustic sensor 300 notifies the analysis server 100 of the occurrence of the abnormality. This notification processing may be performed as processing of the abnormality detection unit 301. When the acoustic sensor 300 notifies the analysis server 100 of the occurrence of the abnormality, the analysis server 100 performs processing to be described later. In this way, in this embodiment, whether or not the processing of the analysis server 100 is performed is determined according to the determination result of the abnormality determination unit 302, but the processing of the analysis server 100 may be performed regardless of the determination result of the abnormality determination unit 302. That is, in all cases where the anomaly detection unit 301 detects the occurrence of an abnormal situation, the processing of the analysis server 100 may be performed. That is, the determination processing by the anomaly determination unit 302 may be omitted.

[0034] As shown in FIG. 5, the analysis server 100 includes a sound source position estimation unit 101, a video acquisition unit 102, a human detection unit 103, a crowd extraction unit 104, a gaze estimation unit 105, a facial expression recognition unit 106, a seriousness estimation unit 107, a seriousness determination unit 108, and a signal output unit 109.

[0035] The sound source position estimation unit 101 estimates the source of a sound detected by an acoustic sensor 300 installed in the monitored area 90, thereby estimating the location of the occurrence of an abnormal situation. Specifically, when the analysis server 100 is notified of the occurrence of an abnormal situation from the multiple acoustic sensors 300, the sound source position estimation unit 101 collects acoustic data on the abnormal sound, for example, from the multiple acoustic sensors 300. Then, the sound source position estimation unit 101 performs a known sound source position estimation process disclosed in, for example, Patent Document 2, and estimates the source position of the abnormal sound, that is, the location of the occurrence of the abnormal situation. The sound source position estimation unit 101 corresponds to the position acquisition unit 2 in FIG. 1. That is, in this embodiment, the location of the occurrence of an abnormal situation is acquired by estimating the source of the sound.

[0036] When the sound source position estimation unit 101 estimates the occurrence position of the abnormal situation, the video acquisition unit 102 acquires video data from the surveillance camera 200 capturing the estimated position. Note that, for example, information indicating which area each surveillance camera 200 is capturing is stored in advance in the analysis server 100, and the video acquisition unit 102 compares this information with the estimated position to identify the surveillance camera 200 capturing the estimated position.

[0037] The human detection unit 103 analyzes the video data acquired by the video acquisition unit 102 and detects people (whole body images and faces of people). Specifically, the human detection unit 103 inputs each frame of the video data into a multi-layer neural network trained by deep learning, and detects people appearing in the image of each frame.

[0038] The crowd extraction unit 104 extracts a crowd around the position where the abnormality occurred from the video data acquired by the video acquisition unit 102. That is, the crowd extraction unit 104 extracts people who are away from the position where the abnormality occurred and are in the vicinity. The crowd extraction unit 104 extracts people who belong to the crowd from among the people detected by the human detection unit 103. For example, the crowd extraction unit 104 detects the ground shown in the video data by image recognition processing, and estimates the position of the person detected by the human detection unit 103 in the monitored area 90 by identifying the position where the person's feet are in contact with the ground. Also, for example, the crowd extraction unit 104 estimates the position of the person with the face in the monitored area 90 by identifying the intersection of a straight line extending vertically downward from the position of the face detected by the human detection unit 103 and the ground. Also, the crowd extraction unit 104 may estimate the position of the person based on the size of the face shown in the video data. Then, the crowd extraction unit 104 extracts a crowd based on the distance between the estimated position of the person detected by the person detection unit 103 and the abnormality occurrence position estimated by the sound source position estimation unit 101. Specifically, the crowd extraction unit 104 extracts, for example, people who are more than 1 meter away and within 5 meters from the abnormality occurrence position as a crowd around the abnormality occurrence position.

[0039] The gaze estimation unit 105 estimates the gaze of each person constituting the crowd around the position where the abnormal situation occurred. That is, the gaze estimation unit 105 estimates the gaze of the person extracted as a crowd by the crowd extraction unit 104. The gaze estimation unit 105 estimates the gaze by performing a known gaze estimation process on the video data. For example, the gaze estimation unit 105 may estimate the gaze by performing the process disclosed in Patent Document 3 on the face image. Also, for example, for a person who turns the back of his / her head toward the surveillance camera 200, the gaze estimation unit 105 may estimate the gaze from the direction of the head reflected in the image. Also, the gaze estimation unit 105 may calculate the reliability (estimation accuracy) of the estimated gaze based on the number of pixels of the face and eyeball parts.

[0040] The facial expression recognition unit 106 recognizes the facial expressions of each person constituting the crowd around the position where the abnormal situation has occurred. That is, the facial expression recognition unit 106 recognizes the facial expressions of the people extracted as the crowd by the crowd extraction unit 104. The facial expression recognition unit 106 recognizes the facial expressions by performing a known facial expression recognition process on the video data. For example, the facial expression recognition unit 106 may recognize the facial expressions by performing the process disclosed in Patent Document 4 on the face image. In particular, the facial expression recognition unit 106 determines whether or not the facial expression shown on the face of the person is a predetermined facial expression. Here, the predetermined facial expression is specifically an unpleasant facial expression. In the case where a score value related to joy, anger, sadness, and happiness such as a smile degree or an anger degree is obtained as the recognition result of the facial expression, the facial expression recognition unit 106 may determine that the facial expression of the person is an unpleasant facial expression when the score value of the smile degree is equal to or less than a reference value or when the score value of the anger degree is equal to or more than a reference value. In this way, the facial expression recognition unit 106 determines whether or not the facial expressions of the crowd correspond to the facial expressions shown on a person who recognizes an abnormal situation. Furthermore, the facial expression recognition unit 106 may calculate the reliability (recognition accuracy) of the recognized facial expression based on the number of people in the crowd whose faces were captured or the number of pixels of each face.

[0041] The seriousness estimation unit 107 estimates the seriousness of the abnormal situation based on the processing results of the gaze estimation unit 105 and the facial expression recognition unit 106. Specifically, the seriousness estimation unit 107 estimates the seriousness of the abnormal situation based on the processing results of the gaze estimation unit 105 as follows. The seriousness estimation unit 107 estimates the seriousness of the abnormal situation from the number of people in the extracted crowd who are looking in the direction of the abnormal situation occurrence position, or the ratio of the number of people who are looking in the direction of the abnormal situation occurrence position to the number of people in the crowd. For example, the seriousness estimation unit 107 estimates that the more people who are looking in the direction of the abnormal situation occurrence position, the higher the seriousness. Similarly, the seriousness estimation unit 107 estimates that the more people who are looking in the direction of the abnormal situation occurrence position, the higher the seriousness. Note that the seriousness estimation unit 107 may calculate the reliability of the estimated seriousness of the abnormal situation based on the reliability of the gaze estimation result of each person.

[0042] Furthermore, the seriousness estimation unit 107 estimates the seriousness of the abnormal situation based on the processing result of the facial expression recognition unit 106 as follows. The seriousness estimation unit 107 estimates the seriousness of the abnormal situation from the number of people whose recognized facial expressions correspond to a predetermined facial expression, or the ratio of the number of people whose recognized facial expressions correspond to a predetermined facial expression to the number of people in the crowd. For example, the seriousness estimation unit 107 estimates that the more people whose recognized facial expressions correspond to a predetermined facial expression, the higher the seriousness. Similarly, the seriousness estimation unit 107 estimates that the more people whose recognized facial expressions correspond to a predetermined facial expression, the higher the seriousness. Furthermore, the seriousness estimation unit 107 may multiply an emotion score value such as a smile level or an anger level by a correlation coefficient that indicates the correlation between an unpleasant facial expression when seeing an abnormal situation and the emotion score value, as the seriousness. In this case, the seriousness estimation unit 107 may calculate the average of each seriousness calculated as described above from the facial expressions of each person constituting the extracted crowd, and estimate the seriousness of the emergency situation indicated by the entire crowd. The seriousness estimation unit 107 may calculate the reliability of the estimated seriousness of the abnormal situation based on the reliability of the facial expression recognition results of each person.

[0043] The seriousness estimation unit 107 may use either the seriousness estimated based on the processing result of the gaze estimation unit 105 or the seriousness estimated based on the processing result of the facial expression recognition unit 106, but in this embodiment, the seriousness estimation unit 107 integrates the seriousness estimated from the gaze of the extracted crowd and the seriousness estimated from the facial expression of the extracted crowd. For example, the seriousness estimation unit 107 may use the average value of the seriousness estimated based on the processing result of the gaze estimation unit 105 and the seriousness estimated based on the processing result of the facial expression recognition unit 106 as the final seriousness, or may calculate the final seriousness using the seriousness and reliability of both. For example, the seriousness estimation unit 107 may use the reliability of the seriousness based on the gaze estimation and the reliability of the seriousness based on the facial expression recognition as weights to calculate a weighted average of these seriousnesses. Note that this is merely an example of calculating the seriousness using the reliability, and the seriousness may be calculated by other methods. For example, known statistics may be used, and the overall severity may be calculated by Bayesian estimation based on the reliability of each individual person.

[0044] The seriousness determination unit 108 determines whether or not a response is required for the abnormal situation that has occurred. Specifically, the seriousness determination unit 108 determines whether or not the seriousness finally estimated by the seriousness estimation unit 107 is equal to or greater than a predetermined threshold. If the seriousness is equal to or greater than the predetermined threshold, the seriousness determination unit 108 determines that a response is required for the abnormal situation that has occurred, and if not, determines that no response is required.

[0045] The signal output unit 109 outputs a predetermined signal for responding to the abnormal situation when the seriousness determination unit 108 determines that a response is necessary to the abnormal situation that has occurred. That is, the signal output unit 109 outputs a predetermined signal when the seriousness is equal to or greater than a predetermined threshold. This predetermined signal may be a signal for issuing a predetermined instruction to another program (another device) or a person. For example, the predetermined signal may be a signal for issuing an alarm lamp and an alarm sound in a security guard's room or the like, or a message for instructing a security guard or the like to respond to the abnormal situation. In addition, the predetermined signal may be a signal for flashing a warning light near the location where the abnormal situation has occurred in order to suppress criminal acts, or a signal for outputting an alarm to encourage people in the vicinity of the location where the abnormal situation has occurred to evacuate.

[0046] The functions shown in Fig. 4 and the functions shown in Fig. 5 may be realized, for example, by a computer 50 as shown in Fig. 6. Fig. 6 is a schematic diagram showing an example of a hardware configuration of the computer 50. As shown in Fig. 6, the computer 50 includes a network interface 51, a memory 52, and a processor 53.

[0047] The network interface 51 is used to communicate with any other device and may include, for example, a network interface card (NIC).

[0048] The memory 52 is configured, for example, by a combination of a volatile memory and a non-volatile memory. The memory 52 is used to store a program including one or more instructions executed by the processor 53, data used for various processes, and the like.

[0049] The processor 53 reads out a program from the memory 52 and executes it to perform processing of each component shown in Fig. 4 or Fig. 5. The processor 53 may be, for example, a microprocessor, a microprocessor unit (MPU), a central processing unit (CPU), etc. The processor 53 may include multiple processors.

[0050] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0051] Next, the flow of operation of the monitoring system 10 will be described. FIG. 7 is a flowchart showing an example of the flow of operation of the monitoring system 10. FIG. 8 is a flowchart showing an example of the process flow in step S104 in the flowchart shown in FIG. 7. Hereinafter, an example of the flow of operation of the monitoring system 10 will be described with reference to FIGS. 7 and 8. In this embodiment, steps S101 and S102 are executed as the process of the acoustic sensor 300, and the process from step S103 onwards is executed as the process of the analysis server 100.

[0052] In step S101 , the abnormality detection unit 301 detects the occurrence of an abnormal situation in the monitored area 90 based on the sound detected by the acoustic sensor 300 .

[0053] Next, in step S102, the abnormality determination unit 302 determines whether or not a response is required for the abnormal situation that has occurred. If it is determined that a response is not required for the abnormal situation that has occurred (Yes in step S102), the process returns to step S101, and if not (No in step S102), the process proceeds to step S103.

[0054] In step S103, the sound source position estimation unit 101 estimates the source of the sound, thereby estimating the occurrence position of the abnormal situation.

[0055] Next, in step S104, the severity of the abnormal situation is estimated by video analysis. In this way, the video analysis process is not executed in normal times, but is executed only when an abnormal situation occurs. In other words, the analysis process using the video of the surveillance camera 200 is executed when the occurrence of an abnormal situation is detected, but is not executed before the occurrence of the abnormal situation is detected. When the video of the surveillance camera is analyzed in real time to detect the occurrence of an abnormal situation, as in the technology shown in Patent Document 1, a large amount of computer resources are required. In contrast, in this embodiment, as described above, the video analysis process is not executed in normal times, but is executed only when an abnormal situation occurs. Therefore, according to this embodiment, the use of computer resources can be suppressed.

[0056] The process of step S104 will be specifically described with reference to FIG. First, in step S201, in order to analyze the video, the video acquisition unit 102 acquires video data from the monitoring camera 200 capturing the location where the abnormality occurred, among all the monitoring cameras 200 installed in the monitored area 90. Therefore, the analysis process is performed only on the video data from the monitoring camera 200 (the monitoring camera 200 in the vicinity of the location of the sound source) capturing the area including the location where the abnormality occurred, among the multiple monitoring cameras 200. Then, as described above, the occurrence of an abnormality is detected not by analyzing the video, but by detecting the sound. For these reasons, in this embodiment, the video analysis process can be reduced. Therefore, according to this embodiment, the use of computer resources can be further suppressed.

[0057] Next, in step S202, the human detection unit 103 analyzes the acquired video data and detects a person (a whole body image of a person and a face of a person).

[0058] Next, in step S203, the crowd extraction unit 104 extracts people who make up the crowd around the position where the abnormal situation has occurred from among the detected people.

[0059] After step S203, processing regarding the gaze (steps S204 and S205) and processing regarding facial expressions (steps S206 and S207) are performed in parallel. Note that the processing regarding the gaze and the processing regarding facial expressions may be performed in sequence rather than in parallel.

[0060] In step S204, the gaze estimation unit 105 performs a gaze estimation process for the crowd around the position where the abnormal situation has occurred. Then, in step S205, the seriousness estimation unit 107 performs a process of estimating the seriousness of the abnormal situation based on the processing result of the line of sight estimation unit 105.

[0061] In step S206, the facial expression recognition unit 106 performs facial expression recognition processing on the crowd around the position where the abnormal situation has occurred. Then, in step S207, the seriousness estimation unit 107 performs a process of estimating the seriousness of the abnormal situation based on the processing result of the facial expression recognition unit .

[0062] After steps S205 and S207, the process proceeds to step S208. In step S208, the seriousness estimation unit 107 calculates a final seriousness by integrating the seriousness estimated based on the processing result of the gaze estimation unit 105 and the seriousness estimated based on the processing result of the facial expression recognition unit 106. After step S208, the process proceeds to step S105 shown in FIG. 7.

[0063] In step S105, the seriousness determination unit 108 determines whether the seriousness estimated in step S104 is equal to or greater than a predetermined threshold. If the seriousness is less than the predetermined threshold (No in step S105), the process returns to step S101, and if the seriousness is equal to or greater than the predetermined threshold (Yes in step S105), the process proceeds to step S106.

[0064] In step S106, the signal output unit 109 outputs a predetermined signal for dealing with the abnormal situation. After step S106, the process returns to step S101.

[0065] As described above, the monitoring system 10 according to the embodiment makes it possible to know the severity of an abnormal situation that has occurred.

[0066] Incidentally, when analyzing a video to detect the occurrence of an abnormality, it is necessary to define in advance the features of the video corresponding to the abnormality. That is, in order to detect the occurrence of an abnormality from a video, it is necessary to define in advance the features of the video for various abnormalities and prepare a program for analysis (for example, a program that generates a classifier by machine learning). However, in the real world, the facial features, belongings, and behaviors of crime suspects and victims are diverse, and various crimes and accidents occur. For this reason, unless some prerequisite is added, it is difficult to define in advance the features of the video corresponding to the abnormality, and the method of detecting the occurrence of an abnormality from a video lacks practicality. For example, Patent Document 1 illustrates the pre-registration of a facial image of a specific person, but since the facial images of all people who cause an unexpected abnormality are not collected in advance, the use of anomaly detection using facial images or facial features as video features is limited. Patent Document 1 also illustrates the pre-registration of abnormal behavior of a single or multiple people, but for example, there is little difference between a video of multiple people gathering and fighting and a video of multiple people drinking and getting excited and making noise, making it difficult to detect the occurrence of an abnormality from the video. In video analysis, it is difficult to perform an appropriate analysis without certain assumptions.

[0067] In contrast, in this embodiment, the occurrence of an abnormal situation is detected by a method other than video analysis. The analysis of the crowd captured in the video is performed on the premise that the occurrence of an abnormal situation has already been detected. For example, when a street musician or a busker is performing on the roadside, there are scenes in which the gaze of the crowd around a certain person is focused on that person even though no abnormal situation has occurred. In addition, there are scenes in which the surrounding crowd has unpleasant expressions even though no abnormal situation has occurred, such as when a politician who is not supported by the citizens is making a speech on the street. Therefore, it is not possible to determine that an abnormal situation has occurred simply by analyzing the gaze and expressions of the crowd. In contrast, when there is a premise that the occurrence of an abnormal situation such as a criminal act or an accident has been detected by some method, the gaze and expressions function effectively as video features that measure the seriousness of the abnormal situation. This is because the behavior of a crowd facing an abnormal situation often follows universal natural laws that are generally seen in animals. In this way, according to this embodiment, a practical surveillance system can be provided.

[0068] In addition, in this embodiment, the analysis of the state of the crowd includes an analysis of whether the gaze of the crowd is directed toward the location where the abnormality has occurred. This is based on the natural law that when an abnormality such as a criminal act or an accident occurs, the crowd will have doubts about what is happening there, whether rescue is necessary, whether they are in danger, etc., and will often turn their gaze toward the location where the abnormality has occurred. As disclosed in Patent Document 3 and other documents, a technology has already been established for estimating the gaze direction from a video captured from a somewhat distant location, such as a surveillance camera video. Therefore, it is possible to perform an analysis of whether the gaze of the crowd is directed toward the location where the abnormality has occurred with high accuracy using existing technology.

[0069] In addition, in this embodiment, the analysis of the state of the crowd includes an analysis of whether the facial expressions of the crowd are unpleasant. This is based on the natural law that when an abnormal situation such as a criminal act or an accident occurs, the crowd often feels that it is unpleasant, and their smiles disappear and they frown, making them look unpleasant. As disclosed in Patent Document 4 and the like, a technology has already been established for recognizing human facial expressions from images captured from a relatively distant location, such as surveillance camera images, and estimating emotions such as the degree of smile or anger from the facial expressions. Therefore, it is possible to perform an analysis of whether the facial expressions of the crowd are unpleasant with high accuracy using existing technology.

[0070] In addition, in this embodiment, the occurrence of an abnormality is detected by sound. As described above, sound has excellent characteristics suitable for monitoring, and by using sound, even unexpected abnormalities can be detected with high accuracy. Although there is a problem in that the seriousness of the situation cannot be determined when detecting an abnormality by sound, in this embodiment, the seriousness is estimated by analyzing the state of the crowd around the position where the abnormality has occurred. Therefore, in this embodiment, by combining abnormality detection by sound and estimation of the seriousness by video of the crowd, the difficulty of detecting the occurrence of an abnormality by video and the difficulty of judging the seriousness of an abnormality by sound are overcome.

[0071] Furthermore, in this embodiment, the location of an abnormal situation is estimated from sound. Since the sound source location can be identified from the difference in arrival time of sound at multiple microphones or the difference in sound pressure, the location of an abnormal situation can be easily estimated. As described above, sound is also suitable for detecting an abnormal situation, so that by detecting sound, it is possible to detect an abnormal situation and estimate its location. Therefore, by using sound to detect an abnormal situation and estimate its location in combination, sound detection can be effectively utilized.

[0072] Furthermore, according to this embodiment, as described above, the video analysis process is not executed under normal circumstances, but is executed only when an abnormal situation occurs. Therefore, according to this embodiment, the use of computer resources can be suppressed. And, as described above, among the multiple surveillance cameras 200, the analysis process is executed only on the video data from the surveillance camera 200 that captures the area including the position where the abnormal situation occurred. Therefore, according to this embodiment, the use of computer resources can be further suppressed.

[0073] <Modifications of the embodiment> In the above-described embodiment, the acoustic sensor 300 is disposed, and the acoustic sensor 300 includes the anomaly detection unit 301 and the anomaly determination unit 302. However, instead of such a configuration, the monitoring system may be configured as follows. That is, instead of the acoustic sensor 300, a microphone may be disposed in the monitored area 90, and a sound signal collected by the microphone may be transmitted to the analysis server 100, which may perform acoustic analysis and voice recognition. In other words, among the components of the acoustic sensor 300, at least the microphone may be disposed in the monitored area 90, and the other components may not be disposed in the monitored area 90. In this way, the processes of the anomaly detection unit 301 and the anomaly determination unit 302 described above may be realized by the analysis server 100.

[0074] Also, the acoustic sensor 300 in FIG. 3 can be replaced with another sensor. For example, if the abnormality to be monitored, such as the use of a gun or a bomb, generates high heat, a sensor that detects high temperatures, such as an infrared sensor or an infrared camera, can be used. In the case of an infrared camera, the location of high heat can be estimated from an image without disposing a large number of sensors. These can also be used in combination with an acoustic sensor, and can be used differently depending on the installation location. Therefore, the occurrence of an abnormality can be detected based on sound or heat detected by a sensor installed in the monitored area, or the location of the abnormality can be obtained by estimating the source of sound or heat detected by a sensor installed in the monitored area.

[0075] The monitoring method shown in the above-mentioned embodiment may be implemented as a monitoring program and sold. In this case, the user can install it on any hardware and use it, improving convenience. The monitoring method shown in the above-mentioned embodiment may be implemented as a monitoring device. In this case, the user can use the above-mentioned monitoring method without the trouble of preparing hardware and installing a program by himself, improving convenience. The monitoring method shown in the above-mentioned embodiment may be implemented as a system composed of multiple devices. In this case, the user can use the above-mentioned monitoring method without the trouble of combining and adjusting multiple devices by himself, improving convenience.

[0076] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

[0077] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) A position acquisition means for acquiring a position where an abnormality has occurred in a monitored area; an analysis means for analyzing the state of the crowd around the location where the abnormal situation has occurred based on image data from a camera that photographs the area to be monitored; a severity estimation means for estimating the severity of the abnormal situation based on a result of the analysis; A monitoring device having the above configuration. (Appendix 2) The analysis means, as an analysis of the state of the crowd, estimates the line of sight of each person constituting the crowd and analyzes the number of people whose line of sight is directed in the direction of the position where the abnormal situation has occurred, or the ratio of the number of people whose line of sight is directed in the direction of the position where the abnormal situation has occurred to the number of people in the crowd. 2. A monitoring device as described in claim 1. (Appendix 3) The analysis means recognizes the facial expression of each person constituting the crowd as an analysis of the state of the crowd, and analyzes the number of people whose recognized facial expressions correspond to a predetermined facial expression, or the ratio of the number of people whose recognized facial expressions correspond to the predetermined facial expression to the number of people in the crowd. 3. The monitoring device according to claim 1 or 2. (Appendix 4) The location acquisition means acquires the location of the abnormal situation by estimating the source of sound or heat detected by a sensor installed in the monitored area. 4. A monitoring device according to any one of claims 1 to 3. (Appendix 5) The analysis process by the analysis means is executed when the occurrence of the abnormal situation is detected, and is not executed before the occurrence of the abnormal situation is detected. 5. A monitoring device according to any one of claims 1 to 4. (Appendix 6) The monitoring system further includes an abnormality detection unit that detects the occurrence of the abnormal situation based on sound or heat detected by a sensor installed in the monitoring area. 6. A monitoring device as described in claim 5. (Appendix 7) The analysis means performs an analysis process only on the video data of a camera that captures an area including the location where the abnormal situation has occurred, among the plurality of cameras. 7. A monitoring device according to any one of claims 1 to 6. (Appendix 8) a severity determination means for determining whether the severity is equal to or greater than a predetermined threshold; a signal output means for outputting a predetermined signal when the severity is equal to or greater than a predetermined threshold; 8. The monitoring device according to claim 1, further comprising: (Appendix 9) A camera for photographing the area to be monitored; A sensor that detects sound or heat generated in a monitored area; Surveillance equipment and Equipped with The monitoring device includes: a position acquisition means for acquiring a position where an abnormality has occurred in the monitoring target area by estimating a source of the sound or heat detected by the sensor; an analysis means for analyzing the state of the crowd around the location where the abnormal situation has occurred based on the video data from the camera; a severity estimation means for estimating the severity of the abnormal situation based on a result of the analysis; having Surveillance system. (Appendix 10) The analysis means, as an analysis of the state of the crowd, estimates the line of sight of each person constituting the crowd and analyzes the number of people whose line of sight is directed in the direction of the position where the abnormal situation has occurred, or the ratio of the number of people whose line of sight is directed in the direction of the position where the abnormal situation has occurred to the number of people in the crowd. 10. The monitoring system according to claim 9. (Appendix 11) The analysis means recognizes the facial expression of each person constituting the crowd as an analysis of the state of the crowd, and analyzes the number of people whose recognized facial expressions correspond to a predetermined facial expression, or the ratio of the number of people whose recognized facial expressions correspond to the predetermined facial expression to the number of people in the crowd. 11. The monitoring system of claim 9 or 10. (Appendix 12) Acquire the location of the abnormality in the monitored area, Based on image data from a camera capturing the monitored area, the state of the crowd around the location where the abnormal situation occurred is analyzed; Estimating the severity of the abnormal situation based on the results of the analysis. Monitoring method. (Appendix 13) a position acquisition step of acquiring a position where an abnormality has occurred in a monitoring target area; an analysis step of analyzing the state of the crowd around the location where the abnormal situation has occurred based on image data from a camera photographing the monitored area; a severity estimation step of estimating the severity of the abnormal situation based on a result of the analysis; A non-transitory computer-readable medium on which a program for causing a computer to execute the program is stored. [Explanation of symbols]

[0078] 1 Monitoring device 2 Position acquisition part 3 Analysis Department 4 Severity Estimation Section 10. Surveillance System 50 Computer 51 Network Interface 52 Memory 53 Processors 90 Surveillance Areas 100 Analytics Servers 101 Sound source position estimation section 102 Video acquisition unit 103 Human detection unit 104 Crowd Extraction Unit 105 Gaze estimation section 106 Facial expression recognition unit 107 Severity Estimation Department 108 Severity Judgment Department 109 Signal output section 200 Surveillance Cameras 300 Acoustic Sensor 301 Anomaly detection unit 302 Abnormality determination section 500 Network

Claims

1. An abnormality detection means for detecting the occurrence of an abnormal situation based on sound or heat detected by a sensor installed in a monitored area; a position acquisition means for acquiring a position where the abnormal situation has occurred in the monitored area; an analysis means for analyzing the state of the crowd around the location where the abnormal situation has occurred by performing an analysis process on the image data of a camera that photographs the area to be monitored; a severity estimation means for estimating the severity of the abnormal situation based on a result of the analysis; having The analysis process by the analysis means is executed when the occurrence of the abnormal situation is detected based on the sound or the heat, and is not executed before the occurrence of the abnormal situation is detected. monitoring equipment.

2. The analysis means, as an analysis of the state of the crowd, estimates the line of sight of each person constituting the crowd and analyzes the number of people whose line of sight is directed in the direction of the position where the abnormal situation has occurred, or the ratio of the number of people whose line of sight is directed in the direction of the position where the abnormal situation has occurred to the number of people in the crowd. The monitoring device according to claim 1.

3. The analysis means recognizes the facial expression of each person constituting the crowd as an analysis of the state of the crowd, and analyzes the number of people whose recognized facial expressions correspond to a predetermined facial expression, or the ratio of the number of people whose recognized facial expressions correspond to the predetermined facial expression to the number of people in the crowd.

3. A monitoring device according to claim 1 or 2.

4. The location acquisition means acquires the location of the occurrence of the abnormal situation by estimating the source of the sound or heat detected by the sensor installed in the monitored area. A monitoring device according to any one of claims 1 to 3.

5. The analysis means performs the analysis process only on the video data of a camera that captures an area including the location where the abnormal situation has occurred, among the plurality of cameras. A monitoring device according to any one of claims 1 to 4.

6. a severity determination means for determining whether the severity is equal to or greater than a predetermined threshold; a signal output means for outputting a predetermined signal when the severity is equal to or greater than a predetermined threshold; 6. The monitoring device according to claim 1, further comprising:

7. Detecting the occurrence of an abnormality based on sound or heat detected by a sensor installed in a monitored area; acquiring a location where the abnormal situation has occurred in the monitored area; Analyzing the state of the crowd around the location where the abnormality occurred by performing an analysis process on the image data of the camera photographing the monitored area; Estimating the severity of the abnormal situation based on the results of the analysis; The analysis process is executed when the occurrence of the abnormal situation is detected based on the sound or the heat, and is not executed before the occurrence of the abnormal situation is detected. Monitoring method.

8. An abnormality detection step of detecting the occurrence of an abnormal situation based on sound or heat detected by a sensor installed in a monitored area; a position acquisition step of acquiring a position where the abnormal situation has occurred in the monitored area; an analysis step of analyzing the state of the crowd around the location where the abnormal situation has occurred by performing an analysis process on the image data of the camera photographing the area to be monitored; a severity estimation step of estimating the severity of the abnormal situation based on a result of the analysis; on the computer, The analysis process is executed when the occurrence of the abnormal situation is detected based on the sound or the heat, and is not executed before the occurrence of the abnormal situation is detected. program.

Images