Behavior detection system and behavior detection program

Abstract

An action sensing system 1 for identifying a person's actions from a surveillance image captured by using a surveillance camera 101 or the like and notifying a surveillance center 121, wherein a notification condition setting the necessity of notifying the surveillance center 121 is received for respective action classes (e.g., "striking", "falling over", "walking", "operating smartphone", etc.) under which the person's action included in the surveillance image falls. Notification conditions include the sustained duration of the action, congestion in the person's surroundings, a time slot where the surveillance image was captured, the person's visitation region in the surveillance image, an attribute of the person, an action / attribute of a person in the person's surroundings, and the like. The foregoing enables appropriate notification, according to the user's needs, of the results of action sensing by video surveillance.

Description

[Technical Field]

[0001] The present disclosure relates to a behavior detection system and a behavior detection program, and in particular to a technology for effectively notifying users of behavior detected by video surveillance. [Background technology]

[0002] In recent years, advances in video surveillance technology have made it possible to detect various events from video data.

[0003] For example, a care support system has been proposed that constantly films care recipients in care facilities, analyzes the obtained video using a machine learning model to detect whether the care recipient has taken a specific posture, and notifies caregivers of the care recipient's condition (see, for example, Patent Document 1). This technology can reduce the burden on caregivers who watch over care recipients, thereby reducing the effort required to watch over care recipients and enabling the provision of high-quality care.

[0004] However, depending on the type of behavior to be detected, it may not always be easy to properly determine whether or not that behavior has been performed from a single image. To address this problem, for example, a video surveillance device has been proposed that detects behavior by combining two images (see, for example, Patent Document 2). In this way, it becomes possible to properly detect even complex behavior that is difficult to detect from a single image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2017 / 026308 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-062131 [Non-patent literature]

[0006] [Non-Patent Document 1] Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", 14 Apr 2017, https: / / arxiv.org / pdf / 1611.08050.pdf. [Non-patent document 2] Sijie Yan, Yuanjun Xiong, Dahua Lin, "Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition", 25 Jan 2018, https: / / arxiv.org / pdf / 1801.07455.pdf. Summary of the Invention [Problem to be solved by the invention]

[0007] Even if actions appear similar on video, their importance to the monitor is not necessarily uniform, and it can be difficult to determine the importance of a given action from video alone. For example, the video surveillance device according to the above-mentioned conventional technology detects the action of unlocking a locked door (Fig. 21(a)), and then detects the action of entering the residence through that door (Fig. 21(b)), and determines that a burglar has entered the residence.

[0008] However, it is not only burglars who unlock doors and enter residences; residents of those residences also do the same. Furthermore, residents return home far more frequently than burglars. Therefore, if a video surveillance device notifies the user of such behavior every time it detects such behavior, it will not only notify when a burglar breaks in, but also when a resident returns home. This results in frequent and unnecessary notifications, significantly reducing user convenience and making the system impractical.

[0009] Furthermore, when using machine learning models for video surveillance, in order to detect behaviors that meet the needs of the user, it takes a huge amount of effort to prepare training data that meets the user's needs. There is also the problem that it is not necessarily easy to implement machine learning that can precisely distinguish between similar behaviors, such as the above-mentioned burglary and the resident returning home.

[0010] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a behavior detection system and a behavior detection program that can appropriately notify users of the results of behavior detection through video surveillance in accordance with their needs. [Means for solving the problem]

[0011] In order to achieve the above object, a behavior detection system according to one embodiment of the present disclosure includes: an identification means for identifying to which of predetermined behavior classes a behavior of a person included in an image corresponds using a machine learning model; a determination means for determining whether or not both a first notification condition that specifies, for each behavior class, whether or not a notification of the detection of a behavior that corresponds to the behavior class is necessary, and a second notification condition based on a reason other than the behavior class, are satisfied; and a notification means for notifying a predetermined notification destination of the detection of the behavior when the determination result is affirmative. The second notification condition includes at least one of the degree of congestion around the person, the positional relationship between the position of the person in the image and the specified range, the attribute of the person, the behavior class of people around the person, and the attribute of people around the person. It is characterized by:

[0012] In this case, when the image includes a plurality of people, the specifying means may specify, for each person, to which behavior class the behavior of that person falls.

[0013] The machine learning model may be a neural network, and preferably the neural network is a convolutional neural network.

[0014] The first notification condition may include a specification that notification is required when the behavior of the person corresponds to a combination of a plurality of behavior classes.

[0015] The second notification condition may be specified for each of the behavior classes.

[0016] The second notification condition may be specified in correspondence with a combination of the plurality of behavior classes.

[0017] The second notification condition may be a combination of a plurality of specifications for each of the behavior classes.

[0018] The second notification condition may be a combination of a plurality of specifications corresponding to the combination of the plurality of behavior classes.

[0019] The behavior class may also include at least one of hitting, kicking, pushing, climbing, crawling, falling, lying down, throwing, running, riding, walking, operating a smartphone, sitting, grasping, and talking.

[0021] The surroundings of the person may be a predetermined range that includes the person in the image.

[0022] Furthermore, the people around the person may be people who belong to a cluster that includes the person among a plurality of people included in the image.

[0023] The notification may also include information indicating which behavior class the behavior belongs to.

[0024] The notification may also include information indicating the importance of the notification set for each specification of whether or not notification is required in the second notification condition.

[0025] In addition, a behavior detection system according to another aspect of the present disclosure is characterized by comprising: an identification means for using a machine learning model to identify which of predetermined behavior classes the behavior of each of multiple people included in an image corresponds to; a determination means for determining whether or not both a first notification condition for specifying, for each behavior class, whether or not notification of the detection of behavior corresponding to the behavior class for one of the multiple people is required; and a second notification condition according to a combination of the behavior class for the one person and the behavior classes for the other people among the multiple people are satisfied; and a notification means for notifying a predetermined notification destination of the detection of the behavior if the determination result is positive. Furthermore, a behavior detection program according to an embodiment of the present disclosure causes a computer to execute the following steps: a specifying step of specifying to which of predetermined behavior classes the behavior of a person included in an image corresponds using a machine learning model; a determining step of determining whether or not both a first notification condition that specifies for each behavior class whether or not a notification of the detection of a behavior that corresponds to the behavior class is required and a second notification condition based on a reason other than the behavior class are satisfied; and a notifying step of notifying a predetermined notification destination of the detection of the behavior when the determination result is affirmative. and the second notification condition includes at least one of a degree of congestion around the person, a positional relationship between the position of the person in the image and a specified range, an attribute of the person, a behavior class of people around the person, and an attribute of people around the person. It is characterized by: [Effects of the Invention]

[0026] In this way, the second notification condition is used in addition to the first notification condition to determine whether or not a notification is necessary, so that the results of behavior detection by video monitoring can be appropriately notified according to the user's needs. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a diagram illustrating a main configuration of a behavior detection system 1 according to an embodiment of the present disclosure. [Figure 2] 1 is a block diagram illustrating the main configuration of a behavior detection system 1. FIG. [Figure 3] 1 is a block diagram illustrating the main functional configuration of a behavior detection system 1. FIG. [Figure 4] FIG. 2 is a diagram illustrating the main configuration of a machine learning model that uses a behavior identification unit 301 of the behavior detection system 1. [Figure 5] 10(a) to 10(c) are diagrams illustrating examples of surveillance images received by the behavior detection system 1 and detection results of people included in the surveillance images using a machine learning model. [Figure 6] FIG. 6 is a diagram illustrating an example of an "action class" tab 600 on a "notification condition setting" screen 6 displayed for setting an "action class" among notification conditions. [Figure 7] FIG. 7 is a diagram illustrating an example of an "action duration tab" 700 on a "notification condition setting" screen 6 displayed for setting the "action duration" of the notification conditions. [Figure 8] 10A and 10B are diagrams illustrating a notification operation when an action duration is set. [Figure 9] FIG. 9 is a diagram illustrating an example of a "Congestion level tab" 900 on a "Notification condition setting" screen 6 displayed for setting the "congestion level" of the notification conditions. [Figure 10] FIG. 10 is a diagram illustrating an example of a "time period tab" 1000 on a "notification condition setting" screen 6 displayed for setting the "time period" among the notification conditions. [Figure 11] FIG. 11 is a diagram illustrating an example of a "stay in specific area" tab 1100 on a "setting notification conditions" screen 6 displayed for setting "stay in specific area" among the notification conditions. [Figure 12] 12 is a table illustrating the data structure of a notification condition table 1200. [Figure 13] 10 is a flowchart illustrating the operation of the behavior detection system 1. [Figure 14] (a) to (c) are figures explaining the "Notification Condition Setting" screen 14 according to a modified example of the present disclosure, and in particular show the screen display and procedure for selecting "Surveillance Camera." [Figure 15] 10(a) to 10(c) are diagrams illustrating the "Notification Condition Setting" screen 14 according to a modified example of the present disclosure, and in particular show the screen display and procedure for specifying the "Behavior Class." [Figure 16] (a) and (b) are figures explaining the "Notification Condition Setting" screen 14 relating to a modified example of the present disclosure, and in particular show the screen display and procedure for specifying "Notification Conditions" corresponding to "Behavior Classes." [Figure 17] (a) and (b) are figures explaining the "Notification Condition Setting" screen 14 relating to a modified example of the present disclosure, and in particular show the screen display and procedure for specifying "Notification Conditions" corresponding to "Behavior Classes." [Figure 18] (a) to (c) are figures explaining the "Notification Condition Setting" screen 14 relating to a modified example of the present disclosure, and in particular show the screen display and procedure for specifying a combination of "behavior classes." [Figure 19]1 is a diagram illustrating a system configuration in a nursing care facility to which a behavior detection system 1 according to the present disclosure is applied. [Figure 20] 10A and 10B are diagrams illustrating examples of detection results of joint information according to a modification of the present disclosure, in the order of the time at which the monitoring images from which the detection was made were captured. [Figure 21] 1A and 1B are diagrams illustrating the operation of a video surveillance device according to the prior art, in which (a) shows the action of unlocking a door, and (b) shows the action of entering a residence. BEST MODE FOR CARRYING OUT THE INVENTION

[0028] Hereinafter, an embodiment of a behavior detection system according to the present disclosure will be described with reference to the drawings. [1] Configuration of behavior detection system First, the configuration of the behavior detection system according to this embodiment will be described.

[0029] As shown in FIG. 1, behavior detection system 1 according to this embodiment is a so-called cloud system that sequentially acquires video images (hereinafter referred to as "monitoring images") captured by surveillance cameras 101, 102, and 103 installed on the streets. The surveillance images captured by surveillance cameras 101 and the like include, for example, people 111, 112, 113, and 114. Behavior detection system 1 detects people 111 and the like and their behavior from the acquired surveillance images, sorts the detection results as described below, and then notifies monitoring center 121 of the detection results. Note that behavior detection system 1 may be installed within monitoring center 121 instead of a cloud system.

[0030] As shown in Fig. 2, the behavior detection system 1 comprises a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, etc., connected so that they can communicate with each other via an internal bus 207. The CPU 201 reads a boot program from the ROM 202 to start up, and reads and executes an OS (Operating System), application programs, etc. from a HDD 204 using the RAM 203 as a working storage area.

[0031] A NIC (Network Interface Card) 205 executes processing for communicating with the monitoring cameras 101 and the monitoring center 121. A timer 206 executes timing processing required for the CPU 201 to execute the OS, application programs, and the like.

[0032] FIG. 3 is a block diagram showing the functional configuration of the behavior detection system 1. As shown in FIG. 3, the behavior detection system 1 includes a behavior identification unit 301, a notification condition determination unit 302, and a notification processing unit 303. When the behavior identification unit 301 acquires a surveillance image from the surveillance camera 101 or the like, it uses a machine learning model to recognize a person from the surveillance image and identifies which of multiple classes the person's behavior corresponds to. When the behavior identification unit 301 recognizes a person, the notification condition determination unit 302 determines whether the notification condition specified by the monitoring center 121 is satisfied. When the notification condition is satisfied, the notification processing unit 303 notifies the monitoring center 121. [2] Behavior identification section 301 The behavior identification unit 301 according to this embodiment periodically acquires surveillance images and their capture dates and times from the surveillance cameras 101, etc., and recognizes people and their behaviors from the surveillance images using a convolutional neural network (CNN) as a machine learning model. Specifically, as shown in Fig. 4, the convolutional layer 401 of the CNN 400 is used to extract features of a surveillance image 411, and the output of the convolutional layer 401 is input to a fully connected layer (output layer) 402, thereby identifying the position and behavior of the person.

[0033] This convolutional layer 401 detects characteristic patterns and various characteristic combinations of patterns in the surveillance image 411. The output layer 402 identifies the position and size of a bounding box surrounding a person included in the surveillance image 411 from the output data of the convolutional layer 401, and the class to which the person's behavior corresponds (hereinafter referred to as "behavior class").

[0034] CNN400 uses images of people performing various actions as training data, and performs machine learning to determine the position and size of the bounding box that represents the area of ​​the person included in the training data image, as well as the behavior class to which the person's behavior corresponds. CNN400 outputs the probability that the person's behavior corresponds to each of the pre-specified behavior classes.

[0035] In this embodiment, behavior classes include "hitting (hitting someone)," "falling (falling down)," "walking (walking)," and "smartphone operation (looking at smartphone)," and it is determined which behavior class each person's behavior falls into. The number of behavior classes detected for each person may be two or more. For example, if the probability that CNN 400 outputs each behavior class is equal to or greater than a preset threshold, it can be determined that the behavior of that person falls into that behavior class.

[0036] When the threshold is 30%, if CNN400 outputs a behavior class with a probability of 75%, the behavior of the person is determined to fall into that behavior class with that 75% probability.Also, when the threshold is 30%, if CNN400 outputs two behavior classes with a probability of 40%, the behavior of the person is determined to fall into both of the behavior classes with that 40% probability.Furthermore, when the threshold is 30%, if CNN400 outputs three behavior classes with a probability of 33%, the behavior of the person is determined to fall into all of the behavior classes with that 33% probability.

[0037] For example, in the example of Figure 5(a), two people 111 and 112 are included in a surveillance image 500, and bounding boxes 501 and 502 indicating the positions of people 111 and 112 are detected. For each of the bounding boxes 501 and 502, the CNN 400 identifies "hit" as the class to which the actions of the people in question belong. In addition, the upper right corner of the surveillance image 500 displays "23:55" as the time of capture.

[0038] In the example of Figure 5(a), the date of photographing is attached as separate data to the surveillance image 500 and is not displayed, but the date of photographing may be displayed along with the time of photographing. The time of photographing may also be attached as separate data to the surveillance image 500 and not displayed. In particular, when detecting a person included in the surveillance image 500 or identifying the behavior of the detected person, it is desirable to omit displaying the date of photographing and the time of photographing from the surveillance image 500. The same applies to Figures 5(b) and 5(c).

[0039] In the example of FIG. 5(b), a person 113 is included in a surveillance image 510, and a bounding box 511 indicating the position of the person 113 is detected. The CNN 400 identifies "fall" as the class to which the person's behavior belongs for the bounding box 511. The time of capture, "09:43," is displayed in the upper right corner of the surveillance image 510.

[0040] In the example of FIG. 5(c), a person 114 is included in a surveillance image 520, and a bounding box 521 indicating the position of the person 114 is detected. The CNN 400 identifies two classes, "walking" and "smartphone operation," as classes to which the person's behavior belongs for the bounding box 521. The shooting time, "13:18," is displayed in the upper right corner of the surveillance image 510.

[0041] In this way, the behavior identification unit 301 receives an input of a surveillance image 411 and outputs the position and size of a bounding box for each person included in the surveillance image 411, as well as the class to which the behavior of that person belongs. At this time, it also outputs the identifier of the surveillance camera that captured the surveillance image 411 and the date and time that the surveillance image 411 was captured.

[0042] The behavior identification unit 301 may acquire surveillance images by requesting the surveillance camera 101 or the like to acquire the surveillance images, or may acquire the surveillance images by having the surveillance camera 101 or the like voluntarily transmit the surveillance images to the behavior identification unit 301. [3] Notification condition determination unit 302 The notification condition determination unit 302 will be explained by dividing the processing into two steps: setting the notification condition and determining the notification condition. (3-1) Setting notification conditions The notification condition determination unit 302 receives, from the monitoring center 121, notification conditions for determining whether or not to notify the monitoring center 121 when the behavior identification unit 301 detects a person. In this embodiment, an example will be described in which the monitoring center 121 accesses the behavior detection system 1 via the web, causing a notification condition setting screen to be displayed on the browser of the monitoring center 121 and allowing the user to input the notification conditions, but it goes without saying that the present disclosure is not limited to this, and instead, notification conditions may be set in other ways. (3-1-1) Behavioral Class 6 is a diagram illustrating an example of the notification condition setting screen 6, and particularly shows the "Behavior Class" tab 600. As shown in FIG. 6, the "Behavior Class" tab 600 accepts the class of behavior for which notification should be sent when detected and the designation of the surveillance camera that will capture the behavior. In the example of FIG. 6, two check boxes for the behavior class, "Walking" and "Smartphone Operation," are checked among the check boxes 601.

[0043] Furthermore, in the list box 602 for specifying the surveillance camera, the surveillance camera 103 is specified. Therefore, unless both the behavior classes of "walking" and "smartphone operation" are detected for each person included in the surveillance image captured by the surveillance camera 103, even if only one of "walking" or "smartphone operation" is detected, no notification is sent to the monitoring center 121. If other notification conditions are also specified, the monitoring center 121 is notified only when those notification conditions are also satisfied.

[0044] Regarding notification conditions other than those for the behavior classes described below, the specification may be accepted for all behavior classes in common, or may be accepted for each behavior class. The user may be allowed to select which method to use. (3-1-2) Action duration 7, the behavior duration tab 700 allows the user to set the duration of the behavior so that the monitoring center 121 is notified when the behavior of the class specified in the "behavior class" tab 600 continues for a predetermined period of time. For this reason, the behavior duration tab 700 displays a check box for specifying the duration. In addition, the upper right area of ​​the behavior duration tab 700 displays a character string indicating which monitoring camera detected which behavior class the behavior duration is set for.

[0045] 7, the duration of the "fall" state is set to "3 to 4 minutes" when a "fall" is detected by the monitoring camera 102. Therefore, if a person who has "fallen" is detected continuously for 3 minutes or more in the monitoring image captured by the monitoring camera 102, the monitoring center 121 is notified of this.

[0046] As shown in FIG. 8, a person 113 detected in a surveillance image 800 taken at time T0 has “fallen” and is located within a bounding box 801 in the surveillance image 800.

[0047] The person 113 who has "fallen" is also detected in surveillance images 810, 820, and 830 taken at times T1, T2, and T3, and the bounding boxes 811, 821, and 831 surrounding the person 113 match the bounding box 801 surrounding the person 113 taken at time T0. Therefore, it is estimated that the person 113 has not moved after falling.

[0048] However, since the time elapsed from time T0 to times T1, T2, and T3 is all shorter than the duration (threshold) set in the behavior duration tab 700, it is not possible to determine with sufficient certainty whether the person 113 has fallen but is safe, or whether he or she is in danger, such as injured. For this reason, no notification is sent to the monitoring center 121.

[0049] The person 113 who has "fallen" is also detected in the surveillance image 840 captured at time T4, and the bounding box 841 surrounding the person 113 matches the bounding boxes 801, 811, 821, and 831 surrounding the person 113 captured in images captured at times T0, T1, T2, and T3. Furthermore, since the time elapsed from time T0 to time T4 is longer than the duration (threshold) set in the behavior duration tab 700, there is a risk that the person 113 may be in danger, such as being injured. For this reason, a notification is sent to the monitoring center 121.

[0050] 7 illustrates an example of the behavior duration tab 700 in which the duration is specified using a check box, but it goes without saying that the present disclosure is not limited to this, and instead of or in addition to this, a field for specifying the duration numerically may be provided. Furthermore, it goes without saying that the duration specification may be accepted using other GUI (Graphical User Interface) components, etc. (3-1-3) Congestion level 9, the congestion level tab 900 allows the user to specify the number of people around the detected person, thereby accepting the specification of the congestion level that needs to be notified to the monitoring center 121. In the example of FIG. 9, if the behavior of the detected person falls into two behavior classes, walking and smartphone operation, and 12 or more people are detected within 8 meters of the detected person, it is determined that the area around the detected person is crowded, and since using a smartphone while walking is dangerous, a notification is sent to the monitoring center 121.

[0051] Instead of the distance from the detected person, all people included in the surveillance image may be detected, the position of each person may be identified in the surveillance image, and clustering may be performed according to the distribution of the people's positions, with the cluster containing the detected person being considered as people surrounding that person, and the congestion level may be determined from the number of people. In this case, there is no need for the user to input the distance to identify the range around the person, improving user convenience.

[0052] Furthermore, when detecting the number of people included in a surveillance image, the bounding box surrounding each person may be detected and the number of bounding boxes may be used as the number of people. Alternatively, the number of people may be determined by detecting the skeleton of each person.

[0053] 9 illustrates a congestion level tab 900 in which the number of people in the vicinity and the distance to the vicinity are specified using check boxes, but it goes without saying that the present disclosure is not limited to this, and instead of or in addition to this, fields may be provided in which the number of people in the vicinity and the distance to the vicinity are numerically specified. Furthermore, it goes without saying that the number of people in the vicinity and the distance to the vicinity may be specified using other GUI components, etc. (3-1-4) Time Zone The time period tab 1000 is a screen for inputting judgment conditions (notification conditions) for determining whether or not a notification is required to the monitoring center 121 depending on the time period when a monitoring image including a detected person was captured. In the example of Fig. 10, four consecutive time periods are specified, and if the behavior class of the detected person is "beating," a notification is sent to the monitoring center if the monitoring image was captured between 6:00 PM and 6:00 AM.

[0054] Furthermore, instead of specifying the time itself, it is also possible to specify a time period that may vary depending on the season, such as "daytime" or "nighttime," or a time period that may vary depending on the day of the week, such as "rush hour." When accepting the specification of multiple time periods, it goes without saying that the multiple time periods do not have to be consecutive, and it is also possible to accept the specification of multiple intermittent time periods. Furthermore, it goes without saying that the specification of time periods may also be accepted using other GUI components, etc. (3-1-5) Stay in a specific area The specific area stay tab 1100 is a screen for inputting a judgment condition (notification condition) for determining whether or not a notification is required to the monitoring center 121 depending on whether or not the detected person is located in a specific area in the monitoring image. In the example of Fig. 11, if the behavior of the detected person 114 falls into two behavior classes, "walking" and "smartphone operation," and the person 114 is located within a specific area 1102 in the monitoring image 1101, a notification is sent to the monitoring center 121.

[0055] 11, for the convenience of the user, when specifying a specific area 1102 in a surveillance image 1101 captured by a surveillance camera 103, the surveillance image 1101 is displayed as an auxiliary image, but the surveillance image 1101 does not have to be displayed. Also, in the example of FIG. 11, a case where the specific area 1102 is rectangular is illustrated, but the specific area may have a shape other than a rectangle, such as a circle or an ellipse.

[0056] If the specific area 1102 is rectangular, the specific area 102 can be specified by, for example, the smallest X coordinate (X1) and Y coordinate (Y1) of the four vertices of the rectangle, and the size of the rectangle in the X and Y directions (Xsize, Ysize). Needless to say, the specific area 102 may also be specified by other methods.

[0057] 11 illustrates an example in which only one specific area 1102 is specified, but it goes without saying that the present disclosure is not limited to this, and it is also possible to accept the specification of two or more specific areas 1102. It goes without saying that it is also possible to accept the specification of the specific area 1102 using a method other than the above, such as accepting input of coordinate values ​​in a monitoring image. (3-1-6) Other Other notification conditions may include, for example, person attributes. Examples of person attributes include gender, age, physique, posture, hairstyle, and clothing. Person attributes may be detected using, for example, a machine learning model. In this case, a bounding box surrounding the person in question may be extracted from the surveillance image and input to the machine learning model.

[0058] Furthermore, when referring to people around the person of interest as in the above-mentioned congestion degree, it may be determined whether to take into consideration the activity class to which the activity of the surrounding people corresponds or the attributes of the surrounding people when determining the congestion degree by referring to the activity class to which the activity of the person of interest corresponds, and if the activity class is "sitting", it may be possible to make it unnecessary to notify the monitoring center 121.

[0059] By accepting the designation as described above, a notification condition table 1200 such as the one shown in Fig. 12 is created. In the notification condition table 1200 shown in Fig. 12, one behavior class to be notified to the monitoring center 121 is designated for each monitoring camera number, but it goes without saying that two or more behavior classes may be designated, and the notification conditions other than the behavior class may or may not be the same for the two or more behavior classes.

[0060] In addition, in this embodiment, an example is described in which a notification is sent to the monitoring center 121 when all notification conditions corresponding to the behavior class for each surveillance camera number are met (when the AND of all notification conditions is met), but it goes without saying that the present disclosure is not limited to this, and a logical expression for the notification condition may be specified that includes not only AND but also OR and NOT. (3-2) Notification condition determination The notification condition determination unit 302 determines whether the notification condition is satisfied for each person identified by the behavior identification unit 301.

[0061] For example, if the behavior identification unit 301 identifies the behavior class of the person 111 as "hitting" for the surveillance image 500 (FIG. 5(a)) received from the surveillance camera 101, the notification condition determination unit 302 refers to the column in the notification condition table 1200 where the surveillance camera number is 101 and the behavior class is "hitting." In the notification condition table 1200, the corresponding notification conditions are specified as a congestion level of "two or more people within one email" from the person of interest and a time period of "6 PM to 6 AM."

[0062] In the surveillance image 500, two people, 111 and 112, are located within a one-meter range of the person 111, and the notification condition relating to the congestion level, "two or more people within one email," is met. In addition, since the image was taken at "23:55," the notification condition relating to the time period, "from 18:00 to 6:00," is also met. Therefore, the notification condition determination unit 302 determines that the notification condition is met.

[0063] When the behavior identification unit 301 identifies the behavior class of the person 113 as "fall" for the surveillance image 510 (FIG. 5(b)) received from the surveillance camera 102, the notification point condition determination unit 302 refers to the column in the notification condition table 1200 where the surveillance camera number is 102 and the behavior class is "fall." In the notification condition table 1200, the behavior duration is specified as "3 minutes or more" as the corresponding notification condition.

[0064] For this reason, the notification condition determination unit 302 records the surveillance image 510, the shooting time "09:43" attached to the surveillance image 510, the position and size of the bounding box 511 surrounding the person 113 identified by the behavior identification unit 301, and the behavior class corresponding to the behavior of the person 113 in the HDD 204.

[0065] Subsequently, if a person is detected in a surveillance image received from the same surveillance camera 102, surrounded by a bounding box of the same size and position as the bounding box 511, and the behavior class is "falling," the shooting time "09:43" of the surveillance image 510 recorded on the HDD 204 is compared with the shooting time of the newly received surveillance image to calculate the behavior duration.

[0066] If the calculated behavior duration corresponds to the behavior duration "3 minutes or more" specified in the notification condition table 1200, the notification condition determination unit 302 determines that the notification condition is met. As a result, the notification processing unit 303 notifies the monitoring center 121. If the calculated behavior duration is less than 3 minutes, the notification condition is not met, and the monitoring center 121 is not notified, and the recording of the monitoring image 510, etc. in the HDD 204 is maintained.

[0067] If no person is detected in a surveillance image received from the same surveillance camera 102, if a person is detected but the position or size of the bounding box surrounding the person differs from the position or size of the previously detected bounding box 511, or if the behavior class of the person is not "falling," it is determined that the behavior of "falling" by person 113 is not continuing, and so the record of the surveillance image 510, etc. is deleted from HDD 204, or the record of the surveillance image 510, etc. is excluded from the determination of behavior duration. As a result, the "falling" of person 113 included in the surveillance image 510 is excluded from the monitoring target for behavior duration.

[0068] When the behavior identification unit 301 identifies the behavior class of the person 114 as "walking" and "smartphone operation" for the surveillance image 520 (FIG. 5(c)) received from the surveillance camera 103, the notification point condition determination unit 302 refers to the column in the notification condition table 1200 where the surveillance camera number is 103 and the behavior class is "walking" and "smartphone operation." In the notification condition table 1200, the corresponding notification conditions are specified as behavior duration of "3 minutes or more," congestion level of "12 or more people within 8 meters," and specific area stay of "(X1, Y1, Xsize, Ysize)."

[0069] In the monitoring image 520, the bounding box 521 surrounding the person 114 is included in the range (X1, Y1, Xsize, Ysize) of the specific area specified in the "Dwell in specific area" column of the notification condition table 1200. On the other hand, the only person included in the monitoring image 520 is person 114, and the notification condition of "12 or more people within 8 meters" specified in the "Congestion level" column of the notification condition table 1200 is not met.

[0070] Therefore, even if "3 minutes or more" is specified in the action duration column of the notification condition table 1200, unlike the case of the monitoring image 510, the monitoring image 520 and the shooting time "13:18" attached to the monitoring image 520 are not recorded in the HDD 204. Therefore, the notification condition determination unit 302 does not determine that the notification condition has been satisfied when the monitoring image 520 is taken, and the notification processing unit 303 does not send a notification to the monitoring center 121. [4] Notification processing unit 303 When the notification condition determination unit 302 determines that the notification conditions specified in the notification condition table 1200 are satisfied, the notification processing unit 303 notifies the monitoring center 121. In other words, the notification processing unit 303 transmits to the monitoring center 121 a signal indicating the presence or absence of an alarm that notifies that a specific behavior has been detected in the monitoring image.

[0071] The notification from the notification processing unit 303 to the monitoring center 121 preferably includes one or more of the following: identification information of the surveillance camera, surveillance image data, the time and date the surveillance image was taken, the behavior class identified by the behavior identification unit 301, the position and size in the surveillance image of the bounding box surrounding the person who performed the behavior corresponding to the behavior class, and information regarding which notification condition specified in the notification condition table 1200 was satisfied and how.

[0072] Furthermore, in the notification from the notification processing unit 303 to the monitoring center 121, the notification processing unit 303 may include information indicating the importance of the alarm notifying that a specific behavior has been detected in the monitoring image. For example, as shown in Fig. 12, the importance of the alarm may be determined by providing a column for specifying the importance in the notification condition table 1200, and the notification processing unit 303 may refer to this column, or the notification condition determination unit 302 may refer to this column and notify the notification processing unit 303 of the importance, so that the notification processing unit 303 may notify the monitoring center 121 of the importance. [5] Operation of Behavior Detection System 1 Next, the operation of the behavior detection system 1 will be described.

[0073] 13, when the behavior detection system 1 acquires a monitoring image from a monitoring camera (S1301: YES), the behavior identification unit 301 checks whether a person is included in the monitoring image. If a person is detected in the monitoring image (1302: YES), the system executes the processes from step S1303 to step S1309 for each person detected in the monitoring image.

[0074] That is, the behavior identification unit 301 identifies the behavior class to which the behavior of the person corresponds (S1304). As described above, the behavior class to which the behavior corresponds is identified using the CNN 400. Next, the notification condition determination unit 302 refers to the notification condition table 1200 to identify the notification conditions corresponding to the combination of the surveillance camera number and the behavior class (S1305), and checks whether all the identified notification conditions are satisfied (S1306).

[0075] If all the notification conditions are met (S1307: YES), the monitoring center 121 is notified that the behavior has been detected (S1308). If the notification conditions are not met (S1307: NO), the monitoring center 121 is not notified, and the processes from step S1303 to step S1309 are executed for another person detected in the monitoring image.

[0076] Once the processes from step S1303 to step S1309 have been completed for all persons detected from the surveillance image, the process returns to step S1301 and the above-described processes are repeated.

[0077] In this way, the results of behavior detection by video monitoring using surveillance images can be appropriately notified according to the user's needs by referring to the notification condition table. [6] Variation The present disclosure has been described above based on the embodiments, but it goes without saying that the present disclosure is not limited to the above-described embodiments, and the following modified examples can be implemented. (6-1) In the above embodiment, the case where CNN400 is used as a machine learning model has been described as an example. However, it goes without saying that the present disclosure is not limited to this, and a neural network other than CNN may be used, or a machine learning model other than a neural network may be used. (6-2) In the above embodiment, the behavior classes detected by the behavior identification unit 301 have been described as “hitting,” “falling,” “walking,” and “smartphone operation” as examples. However, it goes without saying that the present disclosure is not limited to these. Instead of or in addition to these, for example, behavior classes such as “kicking,” “pushing,” “climbing,” “crawling,” “lying down,” “throwing,” “running,” “riding,” “sitting,” “grasping,” and “talking” may be provided to detect the behavior of people included in surveillance images. (6-3) In the above embodiment, an example has been described in which importance is specified for each activity class in the notification condition table 1200. However, the present disclosure is not limited to this example, and the following may be used instead. An importance may be specified for each notification condition, such as "activity duration" or "congestion level." Furthermore, this importance may be specified according to the settings of the notification condition.

[0078] For example, in the case of "action duration," if the duration is between 30 and 60 seconds, the importance may be set to medium, and if it exceeds 60 seconds, the importance may be set to high. Also, instead of the three levels of high, medium, and low, the importance may be set to finer levels, such as numbers from 0 to 1. In such a case, when notifying the monitoring center 121, the importance of each notification condition may also be notified to the monitoring center 121. If the importance differs between notification conditions, the notification condition with the highest importance may be notified to the monitoring center 121.

[0079] In addition, the importance of each notification condition may be multiplied by a weight corresponding to each notification condition, and then the weighted importance values ​​may be added together, and the resulting total value may be notified to the monitoring center 121 as the importance of the notification. (6-4) In the above embodiment, the example was described in which the notification conditions were determined for each person included in the surveillance image. However, it goes without saying that the present disclosure is not limited to this, and the following may be used instead.

[0080] For example, if a surveillance image contains multiple people, it may be possible to determine whether or not to notify the monitoring center 121 depending on the combination of behavior classes corresponding to the behaviors of the multiple people. For example, if a surveillance image shows a person carrying a bag and running, and another person running around that person, it is possible that a theft has occurred, so it is desirable to notify the monitoring center 121, check the situation at the scene, analyze the surveillance image that recorded the situation of the crime, and identify the culprit, etc.

[0081] Furthermore, if the surveillance image contains multiple people and the behavioral class to which these people's actions correspond is "driving," there is a possibility that an emergency situation has occurred, so it is desirable to notify the surveillance center 121 to check the situation on site and take measures to deal with the situation.

[0082] Furthermore, if the attribute of a person included in a surveillance image taken at night is "child" and the behavior class to which the person belongs is "walking," if no other people are detected around the person in the surveillance image or if only people with an attribute other than "adult" are detected, it is possible that the "child" has become lost or has become involved in a crime, and therefore it is desirable to notify the surveillance center 121.

[0083] In this way, the necessity of notification may be determined by correlating the behaviors of multiple people included in one surveillance image. Furthermore, the necessity of notification may be determined by correlating the behavior of a person detected using a surveillance image with the behavior of a person detected using another surveillance image. These other surveillance images may be surveillance images captured at a different date and time from the date and time at which the original surveillance image was captured, using the surveillance camera that captured the original surveillance image.

[0084] Furthermore, a monitoring image taken by a monitoring camera other than the one that took the original monitoring image may be combined with the original monitoring image to determine whether or not a notification is required. In this way, it is possible to more accurately identify complex situations and determine whether or not a notification is required than when determining whether or not a notification is required based only on the behavior class to which the behavior of one person included in one monitoring image corresponds and the notification conditions corresponding to that behavior class. (6-5) In the above embodiment, as illustrated in Figures 6 to 11, an example was described in which notification conditions were set by selecting a tab and clicking a check box. However, it goes without saying that the present disclosure is not limited to this, and the following may be used instead.

[0085] 14(a), for example, the "Notification Condition Settings" screen 14 displays a "Select Surveillance Camera" pull-down menu 1401. The "Select Surveillance Camera" pull-down menu 1401 is a pull-down menu that allows the behavior detection system 1 to select a surveillance camera that can send a notification to the monitoring center 120 by referring to surveillance images.

[0086] When the cursor 1411 is placed over the "Select Surveillance Camera" pull-down menu 1401 and clicked, a list of surveillance cameras 1402 is displayed, as shown in Fig. 14(b). When the cursor 1411 is placed over the list of surveillance cameras 1402, the name of the surveillance camera the cursor is placed over is highlighted, indicating that it can be selected by clicking. In the example of Fig. 14(b), "surveillance camera 102" is highlighted among the surveillance camera names, and clicking in this state allows the surveillance camera with that name to be selected.

[0087] If there are many selectable surveillance camera names, it is not possible to display all of them at once, so only some of the surveillance camera names are displayed as a list, as shown in Fig. 14(b). Also, by operating the scroll bar 1403, the surveillance camera names can be moved up and down to display other surveillance camera names in a list.

[0088] When a surveillance camera is selected, the character string displayed in the "Select Surveillance Camera" pull-down menu 1401 changes to the name of the surveillance camera selected from "Select Surveillance Camera," as shown in Fig. 14(c). In the example of Fig. 14(c), the display changes to the name of the surveillance camera, "Survival Camera 102."

[0089] When a surveillance camera ("surveillance camera 102" in Fig. 15(a)) is selected and you click within the "notification condition settings" screen 14, a click menu 1501 is displayed, as shown in Fig. 15(b). The click menu 1501 displays a list of behavior classes that can be detected as the behavior of people included in surveillance images captured by the selected surveillance camera.

[0090] In the example of Figure 15(b), the behavior classes displayed are "hit," "fall," "walk," and "smartphone operation." Also, the behavior class "fall" that the cursor 1411 is over is highlighted. Clicking on a highlighted behavior class selects that behavior class. Then, the click menu 1501 disappears, and the name of the selected behavior class is displayed.

[0091] In the example of Figure 15(c), the character string "fall" is displayed as the name of the behavior class. This character string is surrounded by a rounded rectangle 1502, which indicates that it is the name of the behavior class. Needless to say, the name of the behavior class may be displayed in a manner other than by surrounding it with a rounded rectangle 1502.

[0092] Next, by placing cursor 1411 over the name of an activity class and clicking, a click menu 1601 is displayed for specifying the notification conditions when the activity of a person included in a surveillance image captured by the surveillance camera corresponds to the activity class, as shown in Fig. 16(a). In the example of Fig. 16(a), the notification conditions "activity duration," "congestion level," "time period," "stay in specific area," and "person attributes" are listed in click menu 1601, and of these notification conditions, "congestion level," which cursor 1411 is over, is displayed in reverse video.

[0093] If you click in this state, "Congestion" is selected from the notification conditions, and as shown in Figure 16(b), a leader line 1511 is drawn from the rounded rectangle 1502 displaying "Fall," and the character string 1512 reading "Congestion" is displayed at the end of this leader line 1511. By repeating the same operation, another notification condition can be selected and displayed.

[0094] 17(a), in addition to the lead line 1511, a lead line 1701 is drawn, and a character string 1702 saying "behavior duration" is displayed at the end of the lead line 1701. In this way, the relationship between the behavior class and the notification condition can be interactively specified in a graph structure.

[0095] On the "Notification Condition Settings" screen 14, when a character string indicating a notification condition is clicked, a screen for setting the notification condition is displayed. For example, as shown in Fig. 17(b), when the character string 1702 "behavior duration" is clicked, the behavior duration tab 700 as shown in Fig. 7 is displayed. In this case, the character string "fall" is displayed in the behavior class column of the behavior duration tab 700 in accordance with the character string "fall" enclosed in the rounded rectangle 1502 that is the origin of the leader line 1701 leading to the character string 1702 "behavior duration".

[0096] Additionally, since "Monitoring Camera 102" is displayed in the "Select Surveillance Camera" pull-down menu 1401, the character string "Monitoring Camera 102" is displayed in the Surveillance Camera field of the behavior duration tab 700. This allows the user to set detailed behavior duration conditions as notification conditions when the behavior of a person included in a surveillance image captured by the surveillance camera 102 corresponds to the behavior class "falling."

[0097] As described above, according to the above embodiment and this modification, notification conditions can be interactively specified using check boxes, pull-down menus, value specification, screen area specification, graph structure, and the like.

[0098] Note that when combining multiple activity classes, the following method may be used. That is, by selecting the activity class names in the same manner as in FIG. 15(b) described above, the names of the activity classes to be combined may be displayed on the "Notification Condition Settings" screen, and then these activity classes may be combined. In the example of FIG. 18(a), the activity classes to be combined, "Walking" and "Smartphone Operation," are each displayed on the "Notification Condition Settings" screen 18.

[0099] 18(b), a rectangle 1803 enclosing the activity classes "Walking" and "Smartphone Operation" is displayed by performing a drag operation from the top left (position of cursor 1812) to the bottom right (position of cursor 1813) of the activity class "Walking" on the "Notification Condition Settings" screen 18. During the drag operation, the range of the activity classes to be combined is undetermined, so the rectangle 1803 is displayed with a dashed line, and the start point of the drag operation (position of cursor 1812) is fixed, but the end point is undetermined.

[0100] After that, when the drag operation is completed, a rectangle 1804 is displayed in a solid line at a position and size corresponding to the rounded rectangles 1801 and 1802 that respectively enclose the behavior classes "walking" and "smartphone operation" that were enclosed by the rectangle 1803 at that time (Figure 18(c)). The dashed rectangle 1803 is erased. After that, when setting notification conditions, by clicking on the rectangle 1804, candidate notification conditions for the combination of "walking" and "smartphone operation" are displayed, as in the example of Figure 16(a).

[0101] In this way, according to this modification, the notification conditions can be set by intuitive operations even for combinations of behavior classes, thereby improving user convenience. (6-6) In the above embodiment, an example was given in which surveillance images taken using a surveillance camera 101 or the like installed on the street were used. However, it goes without saying that the present disclosure is not limited to this, and surveillance images taken by a surveillance camera installed in a location other than the street may be used instead.

[0102] For example, the behavior of the care recipient may be monitored using surveillance images captured by a surveillance camera installed in a nursing facility. As shown in Fig. 19, the monitoring support system MS is a system for supporting caregivers in monitoring the care recipients in a nursing facility, and is configured by connecting mobile terminals TA installed in the rooms RM of each person to be monitored Ob, the nurse stations ST, and each caregiver NS so that they can communicate with each other via a communication network NW.

[0103] A sensor box SB (SB-1 to SB-4) is installed in each room RM (RM-1 to RM-4) of the person being monitored Ob (Ob-1: Mr. A to SB-4: Mr. D). The sensor box SB is equipped with a camera and captures images of the inside of the room RM. When the person being monitored Ob is in the room, the image captured by the sensor box SB will include the image of the person being monitored Ob.

[0104] The sensor box SB inputs the captured surveillance image into a machine learning model such as CNN, and outputs a bounding box surrounding the monitored person Ob in the surveillance image and the behavior class to which the monitored person Ob's behavior corresponds.

[0105] The sensor box SB is configured to determine whether or not notifications are required for each behavior class, and when it detects behavior that corresponds to a behavior class that requires notification, it sends to the management server SV information such as the surveillance image, the position and size of the bounding box for the monitored person Ob contained in the surveillance image, the behavior class to which the behavior of the monitored person Ob corresponds, and the date and time the surveillance image was taken.

[0106] The sensor box SB may detect, for example, the behavior of the monitored person Ob going to sleep, waking up, falling out of bed, wandering around, falling outside the bed, and the like.

[0107] When the management server SV receives information such as surveillance images from the sensor box SB, it checks whether the notification conditions set for each sensor box SB and for each action class are met. These notification conditions may also include the destination of notification when the notification conditions are met.

[0108] The fixed terminal device SP is used to set notification conditions in the sensor box SB and the management server SV. The notification conditions set in the sensor box SB are, for example, the behavior classes that require notification, as described above. The notification conditions set in the management server SV include the time period and the range of measurement values ​​such as body temperature, blood pressure, and pulse rate measured by a biometric device.

[0109] The biometric device performs biometric measurements while attached to the person being watched over Ob, and can notify the sensor box SB and management server SV of the obtained measurement values. The biometric device may be attached to the person being watched over Ob at all times and perform biometric measurements at pre-set times, or may be detached as needed and perform biometric measurements only when attached. When the biometric device is detached from the person being watched over Ob, it may notify the sensor box SB, etc.

[0110] In this modified example, when the notification conditions are met, the management server SV sends a notification to the mobile terminal TA carried by the caregiver NS, which is the notification destination set in accordance with the notification conditions. The mobile terminal TA to which the notification is sent may be one or more. In an emergency, the notification may be sent to all mobile terminals TA. In addition, the importance of the notification may be set in accordance with the notification conditions, and the type and volume of the ringtone output by the mobile terminal TA to which the notification is sent may be changed depending on the importance.

[0111] Care recipients housed in nursing facilities may behave in a variety of ways depending on their physical condition and other factors, but even in such cases, by applying the present disclosure, appropriate notifications can be made according to the condition of the care recipient, thereby reducing the burden on caregivers and providing high-quality care. (6-7) In the above embodiment, the case where the surveillance camera 101 and the behavior detection system 1 are separate has been described as an example. However, the present disclosure is not limited to this, and the following may be used instead. For example, a behavior detection system may be built into each surveillance camera. Furthermore, the behavior detection system built into a surveillance camera may send notifications to multiple notification destinations, and in this case, the notification conditions may be different for each notification destination. (6-8) In the above embodiment, the case where the notification destination of the behavior detection system 1 is the monitoring center 121 has been described as an example, but it goes without saying that the present disclosure is not limited to this, and notification may be sent to another notification destination instead of or in addition to the monitoring center 121. In this case, the notification destination may be a mobile communication device such as a specimen terminal. (6-9) In the above embodiment, the position and size of a bounding box surrounding a person in a surveillance image and the behavior class to which the person's behavior corresponds are used as an example. However, the present disclosure is not limited to this, and the following may also be used. For example, the identity of a person may be detected between surveillance images taken at different dates and times by the same surveillance camera. In this way, even if the positions and sizes of the bounding boxes match, if the people surrounded by the bounding boxes differ between the surveillance images, false detection of the duration of the behavior can be prevented. (6-10) In the above modified example, a case has been described in which a convolutional neural network, CNN400, is used to identify people included in a surveillance image and the behavioral classes to which their behavior corresponds. However, it goes without saying that the present disclosure is not limited to this, and the following may be used instead.

[0112] For example, joint information of a person may be extracted from a surveillance image, and the behavior of the person may be recognized using the obtained joint information.

[0113] There are two methods for extracting joint information: top-down and bottom-up. In the top-down method, each person in the surveillance image is first identified, and then the joints for each identified person are extracted. In the bottom-up method, joints are extracted directly from the surveillance image without identifying the people in the image, and then the extracted joints are associated with each other to identify each person's joints.

[0114] As a method for extracting joint information, for example, OpenPose (see Non-Patent Document 1), which employs a bottom-up approach, can be used. OpenPose extracts joint information from images using a neural network. OpenPose can use the Caffe (Convolutional Architecture for Fast Feature Embedding) model published by Carnegie Mellon University.

[0115] Furthermore, as a method for recognizing human behavior from joint information, for example, ST-GCN (Spatial Temporal Graph Convolutional Networks) can be used (see Non-Patent Document 2). In this case, video images are used as monitoring images, and joint information is extracted from the video images using OpenPose. ST-GCN estimates the behavior class to which each person's behavior corresponds from the joint information in the video images.

[0116] 20, joints 2003, 2005, etc. are extracted as joint information 2000, 2010, and 2020 from a series of surveillance images constituting a video, and a connection relationship 2004 between the joints 2003 and 2005, etc. is sequentially identified from their positional relationships. For example, the joint information 2000 includes two people 2001 and 2002.

[0117] Similarly, joint information 2010 includes people 2011 and 2012, and joint information 2020 includes people 2021 and 2022, each of which specifies a joint and its connection relationship. From such joint information, an action class (e.g., "hit") corresponding to the action of the two people is estimated.

[0118] The joint information 2000, 2010, and 2020 are images that represent the positions of each joint in the monitoring image and their connection relationships. When processed by a computer, the joint information, such as the coordinate values ​​of each joint and an ID that represents the connection relationship, is stored in a recording medium such as RAM 203 or HDD 204.

[0119] By using the behavioral class estimated for each person in this way, it is possible to determine whether or not a notification is necessary, in the same way as in the above embodiment. Therefore, it is possible to appropriately notify the user of the results of behavior detection by video monitoring in accordance with their needs.

[0120] It goes without saying that the method for extracting joint information from a surveillance image is not limited to OpenPose, and a method other than OpenPose may be used, and furthermore, a method other than the bottom-up method may be used. It goes without saying that the method for estimating an action class from joint information is not limited to ST-GCN, and a method other than ST-GCN may be used.

[0121] Furthermore, it goes without saying that the machine learning model used to extract joint information from surveillance images and the machine learning model used to recognize human behavior from joint information are not limited to convolutional neural networks, and neural networks other than convolutional neural networks may be used, or machine learning models other than neural networks may be used. (6-11) In the above embodiment, the case of detecting a person's behavior has been described as an example, but it goes without saying that the present disclosure is not limited to this, and the following may be used instead of a person's behavior.

[0122] For example, when using a surveillance camera to detect the behavior of vermin such as bears, wild boars, or monkeys through video, a warning sound may be emitted to scare off the vermin depending on the time of year (such as the growing season of agricultural crops.) Also, when detecting the behavior of machinery such as automobiles through video, if a car stopped at an intersection is detected and does not move off within a predetermined time after the green light of the traffic signal has turned on, a traffic control center may be notified, as there may be something wrong with the driver or vehicle. (6-12) As described above, the behavior detection system 1 according to the present disclosure is a computer system including a microprocessor and a memory. The memory may store a computer program, and the microprocessor may operate according to the computer program.

[0123] Here, a computer program is a combination of multiple instruction codes that indicate instructions to a computer to achieve a specified function.

[0124] The computer program may also be recorded on a computer-readable recording medium, such as a flexible disk, a hard disk, an optical disk, or a semiconductor memory.

[0125] In addition, computer programs may be transmitted via wired or wireless telecommunications lines, networks such as the Internet, data broadcasting, etc.

[0126] The present invention may also be a method used by the behavior detection system 1 by executing the computer program. (6-13) The above-described embodiments and modifications may be combined with each other. [Industrial Applicability]

[0127] The behavior detection system and behavior detection program according to the present disclosure are useful as a technology for effectively notifying behavior detected by video surveillance. [Explanation of symbols]

[0128] 1.……………………………………Behavior detection system 101, 102, 103...Surveillance cameras 121…………………………………Monitoring Center 301…………………………………………Behavior Specification Department 302………………………………Notification condition determination section 303...................................................................Notification processing unit 400…………………………………Convolutional Neural Network 401…………………………………Convolutional layer 402....................................................................Output layer 411, 500, 510, 520...Surveillance images 501, 502, 511, 521...Bounding Box 6, 14, 18………………………"Notification Condition Settings" screen 600…………………………………”Action Class” tab 700…………………………………"Action Duration" tab 900…………………………………”Crowd Level” tab 1000………………………………"Time Zone" tab 1100………………………………"Stay in a specific area" tab 1200………………………………Notification condition table

Claims

1. An identification means for identifying which of predetermined behavior classes a behavior of a person included in an image corresponds to using a machine learning model; a determination means for determining whether or not both a first notification condition, which specifies for each behavior class whether or not a notification of the detection of a behavior corresponding to the behavior class is necessary, and a second notification condition, which is based on a reason other than the behavior class, are satisfied; a notification means for notifying a predetermined notification destination of the detection of the behavior when the determination result is affirmative, The second notification condition includes at least one of a degree of congestion around the person, a positional relationship between the position of the person in the image and a specified range, an attribute of the person, a behavior class of people around the person, and an attribute of people around the person. A behavior detection system characterized by:

2. The behavior detection system according to claim 1, characterized in that, when the image contains multiple people, the identification means identifies, for each person, which behavior class the behavior of that person belongs to.

3. The machine learning model is a neural network.

3. The behavior detection system according to claim 1 or 2.

4. The neural network is a convolutional neural network. The behavior detection system according to claim 3 .

5. The first notification condition includes a specification that notification is required when the behavior of the person corresponds to a combination of a plurality of behavior classes.

5. The behavior detection system according to claim 1, wherein the behavior detection system is a system for detecting a behavior of a user.

6. The second notification condition is specified for each of the behavior classes.

5. The behavior detection system according to claim 1, wherein the behavior detection system is a system for detecting a behavior of a user.

7. 6. The behavior detection system according to claim 5, wherein the second notification condition is specified in correspondence with a combination of the plurality of behavior classes.

8. The second notification condition is a combination of a plurality of specifications for each of the behavior classes. The behavior detection system according to claim 6 .

9. The second notification condition is a combination of a plurality of specifications corresponding to the combination of the plurality of behavior classes. The behavior detection system according to claim 8 .

10. The behavior class includes at least one of hitting, kicking, pushing, climbing, crawling forward, falling, lying down, throwing, running, riding, walking, operating a smartphone, sitting, grasping, and talking.

10. The behavior detection system according to claim 1.

11. The surroundings of the person are within a predetermined range that includes the person in the image. The behavior detection system according to claim 1 .

12. The people around the person are people who belong to a cluster including the person among the multiple people included in the image. The behavior detection system according to claim 1 or 11.

13. The notification includes information indicating which behavior class the behavior belongs to.

13. The behavior detection system according to claim 1.

14. The notification includes information indicating the importance of the notification set for each designation of whether or not notification is required in the second notification condition.

14. The behavior detection system according to claim 1.

15. An identification means for identifying, for each of a plurality of people included in an image, to which of a predetermined behavioral class the behavior of that person belongs, using a machine learning model; a determination means for determining whether or not both a first notification condition, which specifies for each behavior class whether or not notification of detection of behavior corresponding to the behavior class for one of the plurality of persons is required, and a second notification condition according to a combination of the behavior class for the one person and the behavior classes for the other persons of the plurality of persons, are satisfied; and a notification means for notifying a predetermined notification destination of the detection of the behavior when the determination result is affirmative. A behavior detection system characterized by:

16. an identifying step of identifying, using a machine learning model, which of a predetermined behavior classes the behavior of a person included in an image corresponds to; a determination step of determining whether or not both a first notification condition that specifies, for each behavior class, whether or not a notification of the detection of a behavior corresponding to the behavior class is necessary, and a second notification condition based on a reason other than the behavior class are satisfied; a notification step of notifying a predetermined notification destination of the detection of the behavior when the determination result is affirmative, The second notification condition includes at least one of a degree of congestion around the person, a positional relationship between the position of the person in the image and a specified range, an attribute of the person, a behavior class of people around the person, and an attribute of people around the person. A behavior detection program characterized by:

Images