Monitoring device, monitoring method, and program

Abstract

Provided is a novel technology with which the occurrence of an abnormal situation can be detected and the abnormal situation can be appropriately ascertained. A monitoring device (1) comprises: a voice acquisition unit (2) that acquires prescribed speech spoken by a person due to the occurrence of an abnormal situation in a monitoring target area; a person identification unit (3) that identifies the person who spoke the prescribed speech, on the basis of a feature obtained from the prescribed speech; an analysis unit (4) that searches for the identified person in the images from a camera which images the monitoring target area, and that analyzes an expression or action of the person; and an abnormal situation evaluation unit (5) that evaluates the abnormal situation in the monitoring target area, on the basis of the analysis results.

Description

[Technical field]

[0001] The present disclosure relates to a monitoring device, a monitoring system, a monitoring method, and a non-transitory computer-readable medium having a program stored thereon. [Background technology]

[0002] In recent years, there has been an increase in crimes such as robberies targeting stores operated by one person. To prevent this, an increasing number of stores are installing surveillance cameras and outsourcing monitoring to security companies. However, security companies that have contracts with many customers do not always watch the footage from each surveillance camera, and the reality is that they cannot see it unless an emergency call is made using an emergency button. In addition, there are cases where employees are unable to press the emergency button, for example, because they are being threatened by a robber. For this reason, there are limitations to the method of having people monitor surveillance camera footage.

[0003] To solve this problem, an intelligent monitoring method has been proposed in which a computer monitors images from security cameras. For example, Patent Document 1 discloses a monitoring method in which not only security cameras but also microphones are installed, and the captured images and sounds are analyzed by a program to detect the occurrence of an abnormal situation.

[0004] Generally, when detecting an abnormality from a video, video data from a surveillance camera is collected via a network and analyzed by a computer, as described in Patent Document 1. In video analysis, video features that may be linked to danger, such as a facial image of a specific person, abnormal behavior of one or more people, or an object left behind in a specific location, are registered in advance, and the presence of the features is detected.

[0005] In addition to video, anomaly detection is also performed using sound, as in Patent Document 1. There are voice recognition, which recognizes and analyzes the contents of a person's speech, and acoustic analysis, which analyzes sounds other than voice, but neither of these requires significant computer resources. For this reason, even an embedded central processing unit (CPU) such as those installed in smartphones is fully capable of analyzing in real time.

[0006] Detecting the occurrence of abnormal situations by analyzing sound is also effective for unexpected abnormal situations, because it is a universal law of nature that people who encounter an abnormal situation will scream or yell.

[0007] In addition, sound has the property of diffusing in all directions (360 degrees), propagating even in the dark, and going around obstacles in the way. For this reason, sound surveillance is not limited by the field of view, direction, or lighting as with cameras, and it has the excellent characteristic of being suitable for surveillance, as it will not miss even screams or loud voices that occur in the dark or in the shadows.

[0008] Furthermore, when collecting sound using multiple microphones, as disclosed in Patent Document 2, the position of the sound source can be estimated from the difference in arrival time of the sound from the sound source to each microphone, and the sound pressure difference due to diffusion and attenuation of the sound.

[0009] Furthermore, Patent Document 3 discloses a technology for estimating a posture from the joint positions of a person captured in an image. By applying this technology to video, a person's actions can be estimated from the movements of their arms and hands.

[0010] Furthermore, Patent Document 4 discloses a technique for recognizing facial expressions from human face images, called facial expression recognition. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] JP 2013-131153 A [Patent Document 2] Special Publication No. 2013-545382 [Patent Document 3] Patent Publication No. 2021-086322 [Patent Document 4] International Publication No. 2019 / 102619 Summary of the Invention [Problem to be solved by the invention]

[0012] It is difficult to detect the occurrence of unexpected abnormal events by focusing only on the features obtained from the video. In other words, it is difficult to predefine video features that can detect the occurrence of abnormal events from video features alone. When analyzing video to detect the occurrence of abnormal events, it is necessary to predefine video features corresponding to each abnormality. In other words, to detect the occurrence of abnormal events from video, video features must be predefined for various abnormal events and an analysis program (e.g., a program that generates a classifier by machine learning) must be prepared. However, in the real world, the facial features, belongings, and behavior of suspects and victims of crimes are diverse, and various crimes and accidents occur. For this reason, unless some prerequisite is added, it is difficult to predefine video features corresponding to abnormal events, and a method of detecting the occurrence of abnormal events from video alone lacks practicality.

[0013] For example, Patent Document 1 exemplifies the pre-registration of facial images of specific people, but since facial images of all people who may cause unexpected abnormal situations are not collected in advance, the use of anomaly detection using facial images or facial features as video features is limited. Patent Document 1 also exemplifies the pre-registration of abnormal behavior of a single or multiple people, but for example, there is little difference between the behavior of receiving payment from a customer and handing over change from the cash register and the behavior of handing over cash from the cash register after being threatened by a robber. For this reason, it is difficult to determine abnormal behavior from the video features of the person involved.

[0014] On the other hand, as mentioned above, detecting the occurrence of an abnormality by analyzing sound is also effective for unexpected abnormalities, but sound analysis alone cannot evaluate whether or not a response is necessary for the detected abnormality.

[0015] Therefore, one of the objectives that the embodiments disclosed in this specification aim to achieve is to provide a novel technology that can detect the occurrence of an abnormal situation and properly grasp the abnormal situation. [Means for solving the problem]

[0016] A monitoring device according to a first aspect of the present disclosure includes: a voice acquisition means for acquiring a predetermined voice uttered by a person in response to the occurrence of an abnormality in a monitored area; A person identification means for identifying a person who has uttered the predetermined voice based on features obtained from the predetermined voice; An analysis means for searching for the identified person from an image captured by a camera capturing the area to be monitored and analyzing a facial expression or a movement of the person; an abnormal situation evaluation means for evaluating the abnormal situation in the monitored area based on the result of the analysis; has.

[0017] A monitoring system according to a second aspect of the present disclosure includes: A camera for photographing the area to be monitored; A sensor that detects sounds generated in a monitored area; Surveillance equipment and Equipped with The monitoring device includes: a voice acquisition means for acquiring from the sensor a predetermined voice uttered by a person in response to the occurrence of an abnormality in the area to be monitored; A person identification means for identifying a person who has uttered the predetermined voice based on features obtained from the predetermined voice; An analysis means for searching for the identified person from the image of the camera and analyzing the facial expression or movement of the person; an abnormal situation evaluation means for evaluating the abnormal situation in the monitored area based on the result of the analysis; has.

[0018] In a monitoring method according to a third aspect of the present disclosure, Acquire a predetermined voice uttered by a person when an abnormality occurs in a monitored area, Identifying a person who uttered the predetermined voice based on characteristics obtained from the predetermined voice; Searching for the identified person from a video of a camera capturing the monitored area and analyzing the facial expression or movement of the person; Based on the results of the analysis, the abnormal situation in the monitored area is evaluated.

[0019] A program according to the fourth aspect of the present disclosure includes: a voice acquisition step of acquiring a predetermined voice uttered by a person in response to an occurrence of an abnormality in a monitored area; a person identification step of identifying a person who uttered the predetermined voice based on features obtained from the predetermined voice; An analysis step of searching for the identified person from an image of a camera capturing the monitored area and analyzing a facial expression or a movement of the person; an abnormal situation evaluation step of evaluating the abnormal situation in the monitored area based on a result of the analysis; to be executed by the computer. Effect of the Invention

[0020] According to the present disclosure, it is possible to provide a novel technique that can detect the occurrence of an abnormal situation and appropriately grasp the abnormal situation. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of a monitoring device according to an overview of an embodiment. [Diagram 2] 1 is a flowchart showing an example of an operation flow of a monitoring device according to an overview of an embodiment. [Diagram 3] FIG. 1 is a schematic diagram illustrating an example of a configuration of a monitoring system according to an embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of a functional configuration of an acoustic sensor. [Diagram 5] 2 is a block diagram illustrating an example of a functional configuration of an analysis server. [Figure 6] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a computer. [Figure 7] 1 is a flowchart illustrating an example of an operation flow of the monitoring system according to an embodiment. [Figure 8] 8 is a flowchart showing an example of the process flow in step S107 in the flowchart shown in FIG. 7. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] <Outline of the embodiment> Before describing the details of the embodiment, an overview of the embodiment will be described. Fig. 1 is a block diagram showing an example of the configuration of a monitoring device 1 according to the overview of the embodiment. As shown in Fig. 1, the monitoring device 1 has a voice acquisition unit 2, a person identification unit 3, an analysis unit 4, and an abnormal situation evaluation unit 5, and is a device for monitoring a predetermined monitoring target area.

[0023] The voice acquisition unit 2 acquires a predetermined voice uttered by a person when an abnormal situation occurs in the monitored area. Here, the predetermined voice is a voice uttered by a person when he or she encounters an abnormal situation, such as a scream or a shout. The voice acquisition unit 2 acquires, for example, the scream or the shout collected by a microphone installed in the monitored area.

[0024] The person identification unit 3 identifies the person who uttered the predetermined voice based on the features obtained from the predetermined voice acquired by the voice acquisition unit 2. For example, the person identification unit 3 identifies which person, among the people whose voice features are registered, the person who uttered the predetermined voice corresponds to based on the features obtained from the predetermined voice and the voice features of people registered in advance.

[0025] The analysis unit 4 searches for the person identified by the person identification unit 3 from the video of the camera capturing the monitored area, and analyzes the facial expression or movement of the person. For example, the analysis unit 4 analyzes whether the facial expression of the person searched for in the video is a predetermined facial expression. Here, the predetermined facial expression is an expression that appears when a person encounters an abnormal situation, specifically, for example, a frightened facial expression or an angry facial expression. In addition, for example, the analysis unit 4 analyzes whether the movement of the person searched for in the video is a predetermined movement. Here, the predetermined movement may be, for example, a series of movements or a gesture performed by a person who encounters an abnormal situation. Note that the analysis unit 4 may perform only one of the facial expression analysis and the movement analysis, or may perform both.

[0026] The abnormal situation evaluation unit 5 evaluates an abnormal situation in the monitored area based on the result of the analysis by the analysis unit 4. For example, the abnormal situation evaluation unit 5 calculates an index (e.g., a score) for determining whether or not the abnormal situation requires a response. The abnormal situation evaluation unit 5 may also determine whether or not the abnormal situation requires a response based on the index.

[0027] 2 is a flowchart showing an example of the flow of operations of the monitoring device 1 according to the outline of the embodiment. Hereinafter, an example of the flow of operations of the monitoring device 1 will be described with reference to FIG.

[0028] First, in step S11, the voice acquisition unit 2 acquires a predetermined voice uttered by a person when an abnormal situation occurs in a monitored area. Next, in step S12, the person identification unit 3 identifies the person who uttered the predetermined voice, based on features obtained from the predetermined voice acquired by the voice acquisition unit 2. Next, in step S13, the analysis unit 4 searches for the person identified by the person identification unit 3 from the video of the camera capturing the monitored area, and analyzes the facial expression or movement of the person. Next, in step S14, the abnormal situation evaluation section 5 evaluates the abnormal situation in the monitored area based on the result of the analysis by the analysis section 4.

[0029] The monitoring device 1 according to the embodiment has been described above. According to the monitoring device 1, processing is performed using audio and video, and thus it is possible to detect the occurrence of an abnormal situation and appropriately grasp the abnormal situation.

[0030] <Details of the embodiment> Next, the details of the embodiment will be described. 3 is a schematic diagram showing an example of the configuration of a monitoring system 10 according to an embodiment. In this embodiment, the monitoring system 10 includes an analysis server 100, a monitoring camera 200, and an acoustic sensor 300. The monitoring system 10 is a system for monitoring a predetermined monitoring target area 90. The monitoring target area 90 is, for example, a store or a financial institution, but is not limited thereto, and may be any area where monitoring is performed.

[0031] The surveillance camera 200 is a camera installed to capture images of the monitored area 90. The surveillance camera 200 captures images of the monitored area 90 and generates video data. The surveillance camera 200 is installed in an appropriate position that allows monitoring of the entire monitored area 90. Note that a plurality of surveillance cameras 200 may be installed to monitor the entire monitored area 90.

[0032] In this embodiment, the acoustic sensor 300 is provided at various locations within the monitored area 90. Specifically, the acoustic sensor 300 is provided at intervals of, for example, about 10 to 20 meters. The acoustic sensor 300 collects and analyzes sounds in the monitored area 90. Specifically, the acoustic sensor 300 is a device that is composed of a microphone, a sound device, a CPU, and the like, and senses sounds. The acoustic sensor 300 collects surrounding sounds with a microphone, converts the sounds into digital signals with a sound device, and then performs acoustic analysis with a CPU. In this acoustic analysis, for example, abnormal sounds such as screams and shouts are detected. The acoustic sensor 300 may be equipped with a voice recognition function. In that case, more advanced analysis is possible, such as recognizing the contents of speech such as shouts and estimating the severity of an abnormal situation.

[0033] In this embodiment, the acoustic sensors 300 are installed at various locations in the monitoring area 90 at intervals of about 10 to 20 meters so that the multiple acoustic sensors 300 can detect any abnormal sound that occurs anywhere in the area. Generally, noise in a store or the like is about 60 decibels, while screams and shouts have a volume of about 80 to 100 decibels. However, for example, if the sound source is 10 meters away from the source, the abnormal sound that was 100 decibels near the sound source will attenuate to 80 decibels. Here, if the distance from the sound source to the acoustic sensor 300 is too far, it becomes difficult to distinguish the attenuated abnormal sound from the background noise of about 60 decibels at the position of the acoustic sensor 300. For this reason, in this embodiment, the acoustic sensors 300 are arranged at intervals as described above. Note that, no matter how far apart the multiple acoustic sensors 300 are, the ability of the multiple acoustic sensors 300 to detect the same abnormal sound depends on the background noise level and the performance of each acoustic sensor 300, so there is no restriction of 10 to 20 meters for the arrangement.

[0034] The analysis server 100 is a server for analyzing data obtained by the surveillance camera 200 and the acoustic sensor 300, and has the functions of the surveillance device 1 shown in FIG. 1. The analysis server 100 receives analysis results from the acoustic sensor 300, and acquires video data from the surveillance camera 200 as necessary to analyze the video. The analysis server 100 and the surveillance camera 200 are communicatively connected via a network 500. Similarly, the analysis server 100 and the acoustic sensor 300 are communicatively connected via the network 500. The network 500 is a network that transmits communication between the surveillance camera 200, the acoustic sensor 300, and the analysis server 100, and may be a wired network or a wireless network.

[0035] Fig. 4 is a block diagram showing an example of the functional configuration of the acoustic sensor 300. Fig. 5 is a block diagram showing an example of the functional configuration of the analysis server 100.

[0036] As shown in FIG. 4, the acoustic sensor 300 includes an abnormality detection unit 301 and a primary determination unit 302.

[0037] The abnormality detection unit 301 detects the occurrence of an abnormal situation in the monitored area 90 based on the sound detected by the acoustic sensor 300. The abnormality detection unit 301 detects the occurrence of an abnormal situation by, for example, determining whether or not the sound detected by the acoustic sensor 300 corresponds to a predetermined abnormal sound (specifically, for example, a sound such as a scream or a shout). That is, when the sound detected by the acoustic sensor 300 corresponds to a predetermined abnormal sound, the abnormality detection unit 301 determines that an abnormal situation has occurred in the monitored area 90. In addition, in this embodiment, when the abnormality detection unit 301 determines that an abnormal situation has occurred, it calculates a score indicating the degree of abnormality. For example, the abnormality detection unit 301 may calculate a higher score as the volume of the sound increases.

[0038] When the occurrence of an abnormal situation is detected, the primary determination unit 302 determines whether or not a response to this abnormal situation is unnecessary. For example, the primary determination unit 302 makes this determination by comparing the score calculated by the abnormality detection unit 301 with a threshold value set in advance. That is, when the calculated score is equal to or less than the threshold value, the primary determination unit 302 determines that a response to the detected abnormal situation is unnecessary. In this case, no further processing is performed in the monitoring system 10. On the other hand, when it is determined that a response to the abnormal situation is not unnecessary, the acoustic sensor 300 notifies the analysis server 100 of the occurrence of the abnormal situation. Note that this notification processing may be performed as processing by the abnormality detection unit 301.

[0039] When the acoustic sensor 300 notifies the analysis server 100 of the occurrence of an abnormal situation, the analysis server 100 performs processing described below. As described above, in this embodiment, whether or not the analysis server 100 performs processing is determined depending on the determination result of the primary determination unit 302, but the analysis server 100 may perform processing regardless of the determination result of the primary determination unit 302. In other words, the analysis server 100 may perform processing in all cases where the abnormality detection unit 301 detects the occurrence of an abnormal situation. In other words, the determination processing by the primary determination unit 302 may be omitted.

[0040] As shown in FIG. 5, the analysis server 100 includes an audio acquisition unit 101, a person identification unit 102, a sound source position estimation unit 103, a video acquisition unit 104, a person search unit 105, a facial expression recognition unit 106, an action recognition unit 107, a facial expression score calculation unit 108, an action score calculation unit 109, a secondary judgment unit 110, a signal output unit 111, an audio feature memory unit 121, an appearance feature memory unit 122, an abnormal behavior memory unit 123, and a gesture memory unit 124.

[0041] The voice acquisition unit 101 acquires a predetermined voice uttered by a person due to the occurrence of an abnormal situation in the monitored area 90. Specifically, the voice acquisition unit 101 acquires a predetermined voice (scream or shout) detected by the acoustic sensor 300 from the acoustic sensor 300. Specifically, the voice acquisition unit 101 acquires the voice from the acoustic sensor 300 when the acoustic sensor 300 notifies the analysis server 100 of the occurrence of an abnormal situation.

[0042] The person identification unit 102 identifies the person who uttered the predetermined voice based on features obtained from the predetermined voice acquired by the voice acquisition unit 101. In this embodiment, the person identification unit 102 compares the features of the voice stored in the voice feature storage unit 121 with the features obtained from the predetermined voice acquired by the voice acquisition unit 101 to identify the person who uttered the predetermined voice.

[0043] The voice feature storage unit 121 is a database that stores, for each person (e.g., an employee) who may be present in the monitored area 90, the identification information of the person and the voice feature of the person in association with each other. The person identification unit 102 compares the voice features to identify which person who has uttered a predetermined voice corresponds to among the people whose voice features are registered. Note that voice features that are used include, but are not limited to, the base frequency of formants and fluctuations associated with opening and closing of vocal cords. In order to identify a person, the person identification unit 102 performs a predetermined voice analysis process on the voice acquired by the voice acquisition unit 101 to extract features.

[0044] The person identification unit 102 does not necessarily have to identify one person as the person corresponding to the predetermined voice acquired by the voice acquisition unit 101. When the voice acquisition unit 101 acquires voices of multiple people, the person identification unit 102 may identify each of the multiple people. Furthermore, it is not necessarily required to identify one person for one person's voice acquired by the voice acquisition unit 101. For example, when multiple people having similar voice characteristics are registered, the person identification unit 102 may identify multiple candidates who uttered the predetermined voice.

[0045] The sound source position estimation unit 103 estimates the source of the sound detected by the acoustic sensor 300 installed in the monitored area 90, thereby estimating the location of the occurrence of the abnormal situation. Specifically, when the analysis server 100 is notified of the occurrence of an abnormal situation from the multiple acoustic sensors 300, the sound source position estimation unit 103 performs a known sound source position estimation process, for example, disclosed in Patent Document 2, on the voice data collected from the multiple acoustic sensors 300. That is, for example, the sound source position estimation unit 103 may estimate the sound source position of the voice from the arrival time difference of the voice to microphones installed at multiple positions in the monitored area 90, the sound pressure difference due to the diffusion and attenuation of the sound, and the like. In this way, the sound source position estimation unit 103 estimates the sound source position of the predetermined voice, that is, the occurrence location of the abnormal situation.

[0046] When the sound source position estimation unit 103 estimates the occurrence position of the abnormal situation, the video acquisition unit 104 acquires video data from the surveillance camera 200 capturing the estimated position. Note that, for example, information indicating which area each surveillance camera 200 is capturing is stored in advance in the analysis server 100, and the video acquisition unit 104 compares this information with the estimated position to identify the surveillance camera 200 capturing the estimated position.

[0047] The person search unit 105 searches for a person who has emitted an abnormal sound from a video image in the vicinity of the position where the abnormality occurred. That is, the person search unit 105 searches for a person identified by the person identification unit 102 from the video image acquired by the video acquisition unit 104. Note that, when the person identification unit 102 identifies multiple people, the person search unit 105 performs a search process on these multiple people. In this embodiment, the person search unit 105 searches for a person identified by the person identification unit 102 from the video image by comparing the appearance features of the person stored in the appearance feature storage unit 122 with the appearance features of the person extracted from the video image acquired by the video acquisition unit 104.

[0048] The appearance feature storage unit 122 is a database that stores, for each person (e.g., an employee) who may be present in the monitoring area 90, that is, for each person whose voice feature is registered in the voice feature storage unit 121, the identification information of the person and the appearance feature of the person in association with each other. Specifically, the person search unit 105 detects people from the video, extracts the appearance feature of each person, and searches for the person identified by the person identification unit 102 by comparing the appearance feature with the appearance features of people previously registered in the appearance feature storage unit 122. Here, the appearance feature may be a facial feature, a feature of clothing or a hat, or a code (such as a barcode or a two-dimensional code) printed on an ID card worn by an employee. In other words, the appearance feature may be an appearance feature that can be obtained from the video and is different for each person. In order to search for a person, the person search unit 105 performs a predetermined image analysis process from the video acquired by the video acquisition unit 104 to extract features. When the person search unit 105 searches for a person, it adds an annotation to the video data that identifies the searched person within the video.

[0049] The facial expression recognition unit 106 recognizes the facial expression of the person identified by the person identification unit 102, that is, the person searched for by the person search unit 105. Specifically, for example, the facial expression recognition unit 106 performs a known facial expression recognition process on the video data to which the above-mentioned annotations are added to recognize facial expressions (facial expressions that express psychological states such as calm, laughter, anger, and fear). For example, the facial expression recognition unit 106 may recognize facial expressions by performing the process disclosed in Patent Document 4 on a face image. In particular, the facial expression recognition unit 106 analyzes whether or not the facial expression shown on the face of the person is a predetermined facial expression. Here, the predetermined facial expression is, for example, a frightened facial expression or an angry facial expression. For example, it is assumed that the person who emitted the abnormal voice was a store clerk. In this case, when a store clerk who would normally serve customers with a smile encounters an abnormal situation such as a robbery, the smile disappears and the facial expression changes to a frightened facial expression. Therefore, by detecting such facial expressions, that is, the psychological state that causes such facial expressions, the abnormal situation can be grasped in more detail.

[0050] The action recognition unit 107 recognizes the action of the person identified by the person identification unit 102, that is, the person searched for by the person search unit 105. Specifically, for example, the action recognition unit 107 recognizes the action by performing a known action recognition process on the video data to which the above-mentioned annotations are added. For example, the action recognition unit 107 identifies the movement and posture of the arms, hands, and legs by tracking the joint positions of the person using the technology disclosed in Patent Document 3 for the image. In particular, the action recognition unit 107 analyzes whether the action of the person is a predetermined action. Here, the predetermined action is a pre-registered action, and may be, for example, a series of actions performed by a person who encounters an abnormal situation, or a gesture (pose).

[0051] In this embodiment, when the action recognition unit 107 recognizes a person's action, it compares the recognized action with a series of actions stored in the abnormal action storage unit 123, thereby determining whether or not the person has performed an action that a person who encounters an abnormal situation would perform. That is, the action recognition unit 107 analyzes whether or not the recognized person's action is similar to a predefined series of actions. When the similarity between the two is equal to or greater than a predetermined threshold, the action recognition unit 107 may determine that the predefined series of actions has actually been performed.

[0052] The abnormal behavior storage unit 123 is a database that stores information representing a series of actions taken by a person who encounters an abnormal situation. The series of actions registered in the abnormal behavior storage unit 123 may be one or more. For example, in a case where the monitored area 90 is a store, when a store clerk is attacked by a robber, the store clerk may take money out of the cash register and hand it over to the robber. Therefore, as a series of actions taken by a person who encounters an abnormal situation, the abnormal behavior storage unit 123 may store information representing actions in which the arm of the person to be judged (the store clerk who made the abnormal sound) moves toward the cash register, takes out something with his or her hand, and hands it over to the person in front of him or her.

[0053] Furthermore, in this embodiment, when the action recognition unit 107 recognizes a person's action, it compares the recognized action with gestures stored in the gesture storage unit 124 to determine whether the person has performed an action that a person who encounters an abnormal situation would perform. That is, the action recognition unit 107 analyzes whether the recognized action (gesture) of the person is similar to a predefined gesture. If the similarity between the two is equal to or greater than a predetermined threshold, the action recognition unit 107 may determine that the predefined gesture has actually been performed.

[0054] The gesture storage unit 124 is a database that stores information representing gestures that employees have been trained to make in advance when they encounter an abnormal situation. The gestures registered in the gesture storage unit 124 may be one or more. For example, if the monitored area 90 is a store, employees such as store clerks are given an agreement or training in advance, such as "if attacked by a robber, shout loudly, make a gesture of stretching your left hand up, and follow the robber's request." In this case, information representing a gesture of stretching your left hand up is stored in the abnormal behavior storage unit 123 in advance. It is preferable that the gesture to be registered is a gesture that is rarely made by employees in their normal behavior, is a natural movement when they encounter an abnormal situation, and is easy to detect by video analysis. In this way, by registering gestures in advance, an abnormal situation that requires a response can be reliably detected by video analysis.

[0055] The facial expression score calculation unit 108 calculates a score value for the facial expression recognized by the facial expression recognition unit 106. The facial expression score calculation unit 108 calculates a score that quantifies the degree of abnormality of an abnormal facial expression that does not normally occur, such as anger or fear. For example, the facial expression score calculation unit 108 outputs a larger value as the recognized facial expression expresses greater anger or greater fear. Note that, when a score value relating to joy, anger, sadness, or happiness, such as a smile level or anger level, is obtained as the facial expression recognition result, the facial expression score calculation unit 108 may output the score value using a score value of a predetermined facial expression, such as anger level, obtained by the recognition process.

[0056] The action score calculation unit 109 calculates a score value for the action recognized by the action recognition unit 107. In this embodiment, the action score calculation unit 109 calculates a score that quantifies the degree of abnormality for the action stored in the abnormal action storage unit 123. For example, the action score calculation unit 109 outputs a larger value as the similarity between the recognized series of actions and the action stored in the abnormal action storage unit 123 increases. The action score calculation unit 109 may calculate different score values ​​depending on which of the predefined actions the recognized action corresponds to. Note that, in this embodiment, the action score calculation unit 109 calculates a score that quantifies the degree of abnormality for the action stored in the abnormal action storage unit 123, but may also calculate a score value for gestures in the same manner.

[0057] The secondary determination unit 110 determines whether or not a response is required for the abnormal situation that has occurred. Specifically, the secondary determination unit 110 determines whether or not a response is required by using the score values ​​calculated by the facial expression score calculation unit 108 and the action score calculation unit 109 and the determination result of whether or not a gesture defined in the gesture storage unit 124 has been performed. The secondary determination unit 110 uses these as inputs and determines whether or not a response is required according to a predetermined determination logic. Note that the secondary determination unit 110 may perform the determination by using only a part of these inputs. For example, the secondary determination unit 110 may determine that a response is required for the abnormal situation when the score value calculated by the facial expression score calculation unit 108 exceeds a first threshold value. In addition, the secondary determination unit 110 may determine that a response is required for the abnormal situation when the score value calculated by the action score calculation unit 109 exceeds a second threshold value. In addition, the secondary determination unit 110 may determine that a response is required for the abnormal situation when the sum of the two calculated score values ​​exceeds a third threshold value. Furthermore, the secondary determination unit 110 may determine that a response to an abnormal situation is necessary when a predefined gesture is performed. Furthermore, the secondary determination unit 110 may change the value of the above-mentioned threshold depending on whether or not a predefined gesture is performed. That is, when a predefined gesture is performed, a lower threshold may be used than when a predefined gesture is not performed. Note that the above-mentioned determination logic is merely an example, and the secondary determination unit 110 may use any determination logic. Thus, in this embodiment, the secondary determination unit 110 evaluates an abnormal situation in the monitored area 90 based on the result of the video analysis.

[0058] The signal output unit 111 outputs a predetermined signal for responding to the abnormal situation when the secondary judgment unit 110 judges that a response is necessary to the abnormal situation that has occurred. That is, the signal output unit 111 outputs a predetermined signal when the evaluation of the abnormal situation satisfies a predetermined criterion. This predetermined signal may be a signal for giving a predetermined instruction to another program (another device) or a person. For example, the predetermined signal may be a signal for turning on an alarm lamp and an alarm sound in a security guard's room or the like, or a message for instructing a security guard or the like to respond to the abnormal situation. In addition, the predetermined signal may be a signal for flashing a warning light near the location where the abnormal situation has occurred in order to suppress criminal acts, or a signal for outputting an alarm to encourage people in the vicinity of the location where the abnormal situation has occurred to evacuate.

[0059] The functions shown in Fig. 4 and the functions shown in Fig. 5 may be realized, for example, by a computer 50 as shown in Fig. 6. Fig. 6 is a schematic diagram showing an example of a hardware configuration of the computer 50. As shown in Fig. 6, the computer 50 includes a network interface 51, a memory 52, and a processor 53.

[0060] The network interface 51 is used to communicate with any other device and may include, for example, a network interface card (NIC).

[0061] The memory 52 is configured, for example, by a combination of a volatile memory and a non-volatile memory. The memory 52 is used to store a program including one or more instructions executed by the processor 53, data used for various processes, and the like.

[0062] The processor 53 reads out a program from the memory 52 and executes it to perform processing of each component shown in Fig. 4 or Fig. 5. The processor 53 may be, for example, a microprocessor, a microprocessor unit (MPU), a central processing unit (CPU), etc. The processor 53 may include multiple processors.

[0063] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0064] Next, a description will be given of the flow of operations of the monitoring system 10. Fig. 7 is a flowchart showing an example of the flow of operations of the monitoring system 10. Fig. 8 is a flowchart showing an example of the process flow in step S107 in the flowchart shown in Fig. 7.

[0065] 7 and 8, an example of the operation flow of the monitoring system 10 will be described below. In this embodiment, steps S101 and S102 are executed as processing by the acoustic sensor 300, and steps S103 and onward are executed as processing by the analysis server 100.

[0066] In step S101 , the abnormality detection unit 301 detects the occurrence of an abnormal situation in the monitored area 90 based on the sound detected by the acoustic sensor 300 .

[0067] Next, in step S102, the primary judgment unit 302 judges whether or not a response is required for the abnormal situation that has occurred. If it is judged that a response is not required for the abnormal situation that has occurred (Yes in step S102), the process returns to step S101, and if not (No in step S102), the process proceeds to step S103.

[0068] In step S103, the voice acquisition unit 101 acquires a predetermined voice from the acoustic sensor 300, and the person identification unit 102 identifies the person who uttered the predetermined voice based on features obtained from the predetermined voice acquired by the voice acquisition unit 101.

[0069] Next, in step S104, the sound source position estimating unit 103 estimates the sound source position of the predetermined sound (the position where the abnormal situation has occurred) based on the output of the acoustic sensor 300.

[0070] Next, in step S105, for video analysis, the video acquisition unit 104 acquires video data from the surveillance camera 200 capturing the location where the abnormality occurred, among all the surveillance cameras 200 installed in the monitored area 90. For this reason, analysis processing is performed only on the video data from the surveillance camera 200 capturing the area including the location where the abnormality occurred (the area including the sound source location), among the multiple surveillance cameras 200.

[0071] Also, the analysis process may be performed only on a partial image that is a part of an image constituting a video and that includes the sound source position. In other words, the analysis process may be performed only on a partial image that corresponds to a part of the area, not on the image of the entire shooting area of ​​the surveillance camera 200 that shoots the area including the sound source position.

[0072] In addition, in this embodiment, the video analysis process is not executed under normal circumstances, but is executed only when an abnormal situation occurs. In other words, the analysis process using the video of the surveillance camera 200 is executed when the occurrence of an abnormal situation is detected (specifically, when a predetermined sound is detected), but is not executed before the occurrence of an abnormal situation is detected (before the predetermined sound is detected).

[0073] Next, in step S106, the person search unit 105 searches for the person identified by the person identification unit 102 from the video acquired in step S105 based on the appearance features of the person.

[0074] Next, in step S107, a video analysis is performed on the searched person. The process of step S107 will be specifically described with reference to Fig. 8. In the video analysis, first, the processes of steps S201 and S203 are performed. Note that step S201 and the subsequent processes, and step S203 and the subsequent processes are performed in parallel, for example, but may also be performed sequentially.

[0075] In step S201, the facial expression recognition unit 106 recognizes the facial expression of the person searched for in step S 106. After step S201, in step S202, the facial expression score calculation unit 108 calculates a score value for an abnormal facial expression based on the recognition result in step S201.

[0076] On the other hand, in step S203, the action recognition unit 107 recognizes the action of the person searched for in step S106. After step S203, in step S204, the action recognition unit 107 checks whether or not a gesture stored in the gesture storage unit 124 has been detected. Also, after step S203, in step S205, the action score calculation unit 109 calculates a score value related to the action stored in the abnormal behavior storage unit 123 based on the recognition result in step S203.

[0077] When the processes of steps S202, S204, and S205 are completed, the process proceeds to step S108 shown in FIG.

[0078] In step S108, the secondary judgment unit 110 judges whether or not a response is required for the abnormal situation that has occurred based on the processing result of step S107. If it is judged that a response is not required for the abnormal situation that has occurred (Yes in step S108), the process returns to step S101, and if not (No in step S108), the process proceeds to step S109.

[0079] In step S109, the signal output unit 111 outputs a predetermined signal for dealing with the abnormal situation. This makes it possible to deal with the abnormal situation. After step S109, the process returns to step S101.

[0080] The embodiment has been described above. As described above, the monitoring system 10 performs processing using audio and video, thereby making it possible to detect the occurrence of an abnormal situation and appropriately grasp the abnormal situation.

[0081] In particular, in the monitoring system 10, the occurrence of an abnormal situation is first detected from an abnormal sound emitted by a person, and the person who emitted the abnormal sound is identified from the characteristics of the sound. Then, the monitoring system 10 performs detailed confirmation processing of the occurrence of an abnormal situation by analyzing the facial expression and behavior of the person who emitted the abnormal sound based on the video. Thus, in this embodiment, analysis of the video is performed together with the detection of the abnormal sound. The reason is that since there are many different types of crimes and accidents, it is difficult to define the video characteristics for unexpected abnormal situations in advance unless some precondition is added. In other words, when the precondition of "a person who emitted an abnormal sound" is added, it is easy to confirm the occurrence of an abnormal situation from the facial expression and behavior of the person in question shown in the video. In other words, by having such a precondition, for example, it is easy to distinguish between the action of receiving money from a customer and handing over change from the cash register and the action of handing over cash from the cash register after being threatened by a robber.

[0082] In addition, the processing using sound and video has the following advantages. Although the detection of the occurrence of an abnormal situation by analyzing sound is effective for unexpected abnormal situations, it is difficult to evaluate whether or not a response is necessary for the detected abnormal situation by only analyzing sound. The detection of an abnormal situation by sound is similar to when a person closes his eyes and listens carefully. Although it is possible to recognize that an abnormal situation is highly likely to have occurred by detecting a scream or a cry, it is not possible to grasp the situation in more detail. Therefore, it is difficult to grasp the details of the abnormal situation from the sound, such as whether a security guard should be dispatched immediately or whether it is a minor abnormality that can be confirmed by waiting until the next day. In contrast, by adding a video analysis of the facial expression and behavior of the person who emitted the abnormal sound, it is possible to evaluate the abnormal situation in detail. In this way, in this embodiment, the occurrence of an abnormal situation is first detected from the abnormal sound emitted by a person, the person who emitted the abnormal sound is identified, and then the facial expression and behavior are analyzed from the video of the person, thereby realizing a multimodal analysis using sound and video.

[0083] Also, as described above, the analysis process of the video in the analysis server 100 may be performed only on the video in the vicinity of the sound source position of the abnormal sound. That is, among the videos of the multiple surveillance cameras 200, the analysis may be performed only on the video of the surveillance camera 200 capturing the position estimated to be the sound source position. Also, the analysis may be performed only on a partial image including the position estimated to be the sound source position, which is cut out from the video of one surveillance camera 200. A large amount of computer resources are required to analyze the video in real time. However, in this embodiment, the use of computer resources can be suppressed by analyzing only the video in the vicinity of the sound source position. Also, according to this embodiment, as described above, the video analysis process is not performed in normal times, but is performed only when an abnormal situation is detected by sound. Therefore, according to this embodiment, the use of computer resources can be further suppressed.

[0084] <Modifications of the embodiment> In the above-described embodiment, the acoustic sensor 300 is disposed, and the acoustic sensor 300 includes the abnormality detection unit 301 and the primary determination unit 302. However, instead of such a configuration, the monitoring system may be configured as follows. That is, instead of the acoustic sensor 300, a microphone may be disposed in the monitored area 90, and a sound signal collected by the microphone may be transmitted to the analysis server 100, which may perform acoustic analysis and voice recognition. In other words, among the components of the acoustic sensor 300, at least the microphone may be disposed in the monitored area 90, and the other components may not be disposed in the monitored area 90. In this way, the processes of the abnormality detection unit 301 and the primary determination unit 302 described above may be realized by the analysis server 100.

[0085] The monitoring method shown in the above-mentioned embodiment may be implemented as a monitoring program and sold. In this case, the user can install it on any hardware and use it, improving convenience. The monitoring method shown in the above-mentioned embodiment may be implemented as a monitoring device. In this case, the user can use the above-mentioned monitoring method without the trouble of preparing hardware and installing a program by himself, improving convenience. The monitoring method shown in the above-mentioned embodiment may be implemented as a system composed of multiple devices. In this case, the user can use the above-mentioned monitoring method without the trouble of combining and adjusting multiple devices by himself, improving convenience.

[0086] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

[0087] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) a voice acquisition means for acquiring a predetermined voice uttered by a person in response to the occurrence of an abnormality in a monitored area; A person identification means for identifying a person who has uttered the predetermined voice based on features obtained from the predetermined voice; An analysis means for searching for the identified person from an image captured by a camera capturing the area to be monitored and analyzing a facial expression or a movement of the person; an abnormal situation evaluation means for evaluating the abnormal situation in the monitored area based on the result of the analysis; A monitoring device having the above configuration. (Appendix 2) The analysis means analyzes whether the facial expression of the person is a predetermined facial expression. 2. A monitoring device as described in claim 1. (Appendix 3) The analysis means analyzes whether the motion of the person is similar to a predefined series of motions. 3. The monitoring device according to claim 1 or 2. (Appendix 4) The analyzing means analyzes whether the movement of the person is similar to a predefined gesture. 4. A monitoring device according to any one of claims 1 to 3. (Appendix 5) The analysis process by the analysis means is executed when the predetermined voice is detected, and is not executed before the predetermined voice is detected. 5. A monitoring device according to any one of claims 1 to 4. (Appendix 6) The method further comprises: The analysis means performs an analysis process only on the video data of a camera that captures an area including the sound source position among the plurality of cameras. 6. A monitoring device according to any one of claims 1 to 5. (Appendix 7) The analysis means performs analysis processing only on a partial image including the sound source position within an image constituting the video. 7. A monitoring device as described in claim 6. (Appendix 8) The system further includes a signal output means for outputting a predetermined signal when the evaluation of the abnormal situation satisfies a predetermined criterion. 8. A monitoring device according to any one of claims 1 to 7. (Appendix 9) A camera for photographing the area to be monitored; A sensor that detects sounds generated in a monitored area; Surveillance equipment and Equipped with The monitoring device includes: a voice acquisition means for acquiring from the sensor a predetermined voice uttered by a person in response to the occurrence of an abnormality in the area to be monitored; A person identification means for identifying a person who has uttered the predetermined voice based on features obtained from the predetermined voice; An analysis means for searching for the identified person from the image of the camera and analyzing the facial expression or movement of the person; an abnormal situation evaluation means for evaluating the abnormal situation in the monitored area based on the result of the analysis; having Surveillance system. (Appendix 10) The analysis means analyzes whether the facial expression of the person is a predetermined facial expression. 10. The monitoring system according to claim 9. (Appendix 11) The analysis means analyzes whether the motion of the person is similar to a predefined series of motions. 11. The monitoring system of claim 9 or 10. (Appendix 12) The analyzing means analyzes whether the movement of the person is similar to a predefined gesture. 12. A monitoring system according to any one of claims 9 to 11. (Appendix 13) Acquire a predetermined voice uttered by a person when an abnormality occurs in a monitored area, Identifying a person who uttered the predetermined voice based on characteristics obtained from the predetermined voice; Searching for the identified person from a video of a camera capturing the monitored area and analyzing the facial expression or movement of the person; Evaluating the abnormal situation in the monitored area based on the results of the analysis. Monitoring method. (Appendix 14) a voice acquisition step of acquiring a predetermined voice uttered by a person in response to an occurrence of an abnormality in a monitored area; a person identification step of identifying a person who uttered the predetermined voice based on features obtained from the predetermined voice; An analysis step of searching for the identified person from an image of a camera capturing the monitored area and analyzing a facial expression or a movement of the person; an abnormal situation evaluation step of evaluating the abnormal situation in the monitored area based on a result of the analysis; A non-transitory computer-readable medium on which a program for causing a computer to execute the program is stored. [Explanation of symbols]

[0088] 1 Monitoring device 2. Voice Acquisition Unit 3 Person identification section 4 Analysis Department 5. Abnormal Situation Assessment Department 10. Surveillance System 50 Computer 51 Network Interface 52 Memory 53 Processors 90 Surveillance Areas 100 Analytics Servers 101 Voice Acquisition Unit 102 Person identification part 103 Sound source position estimation section 104 Video Acquisition Unit 105 Person Search Department 106 Facial expression recognition unit 107 Motion recognition section 108 Facial Expression Score Calculation Unit 109 Operation score calculation unit 110 Secondary judgment section 111 Signal output section 121 Voice feature memory unit 122 Appearance feature memory unit 123 Abnormal behavior memory unit 124 Gesture Memory Unit 200 Surveillance Cameras 300 Acoustic Sensor 301 Anomaly detection unit 302 Primary judgment part 500 Network

Claims

1. a voice acquisition means for acquiring a predetermined voice uttered by a person in response to the occurrence of an abnormality in a monitored area; a person identification means for identifying a person who has uttered the predetermined voice by comparing a feature obtained from the predetermined voice with a feature of a voice of a person registered in advance; an analysis means for searching for the identified person from an image captured by a camera capturing the area to be monitored using preregistered external features of the person, and analyzing the facial expression or movement of the person; an abnormal situation evaluation means for evaluating the abnormal situation in the monitored area based on the result of the analysis; A monitoring device having the above configuration.

2. The analysis means analyzes whether the facial expression of the person is a predetermined facial expression. The monitoring device according to claim 1.

3. The analysis means analyzes whether the motion of the person is similar to a predefined series of motions.

3. A monitoring device according to claim 1 or 2.

4. The analyzing means analyzes whether the movement of the person is similar to a predefined gesture. A monitoring device according to any one of claims 1 to 3.

5. The analysis process by the analysis means is executed when the predetermined voice is detected, and is not executed before the predetermined voice is detected. A monitoring device according to any one of claims 1 to 4.

6. The method further comprises: The analysis means performs an analysis process only on the video data of a camera that captures an area including the sound source position among the plurality of cameras. A monitoring device according to any one of claims 1 to 5.

7. The analysis means performs analysis processing only on a partial image including the sound source position within an image constituting the video.

7. The monitoring device according to claim 6.

8. The system further includes a signal output means for outputting a predetermined signal when the evaluation of the abnormal situation satisfies a predetermined criterion. A monitoring device according to any one of claims 1 to 7.

9. Acquire a predetermined voice uttered by a person when an abnormality occurs in a monitored area, Identifying a person who uttered the predetermined voice by comparing characteristics obtained from the predetermined voice with characteristics of voices of persons registered in advance; Searching for the identified person from the image of a camera capturing the monitored area using pre-registered external features of the person, and analyzing the facial expression or movement of the person; Evaluating the abnormal situation in the monitored area based on the results of the analysis. Monitoring method.

10. a voice acquisition step of acquiring a predetermined voice uttered by a person in response to an occurrence of an abnormality in a monitored area; a person identification step of identifying a person who uttered the predetermined voice by comparing a feature obtained from the predetermined voice with a feature of a voice of a person registered in advance; an analysis step of searching for the identified person from an image captured by a camera capturing the area to be monitored using pre-registered external features of the person, and analyzing the facial expression or movement of the person; an abnormal situation evaluation step of evaluating the abnormal situation in the monitored area based on a result of the analysis; A program that causes a computer to execute the following.

Images