Monitoring information processing device, monitoring information processing method, and monitoring information processing program

Abstract

To provide a technique of controlling notification according to a detected person.SOLUTION: According to one embodiment of the present invention, a monitoring information processing device stores label data including one label or more according to features of a person, and notification object data including the features of the person as a plurality of notification objects; detects the person based on image data acquired by each of a plurality of sensor devices arranged in a distributed manner; determines label information according to the features of the person that are detected based on the label data; decides whether or not the detected person is a notification object, based on the notification object data and the determined label information; and controls notification given by a notification mode according to the determined label information, in response to determining that the detected person is the notification object.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] An embodiment of the present invention relates to a surveillance information processing device, a surveillance information processing method, and a surveillance information processing program used in a system in which a plurality of surveillance cameras are distributed in a surveillance area, for example. [Background technology]

[0002] For example, in large stores such as shopping malls, office buildings, train stations, and other facilities used by many people, a system is known in which multiple surveillance cameras are distributed throughout the surveillance area and the system detects suspicious individuals and other people based on the video information obtained by these surveillance cameras.

[0003] In such a system, information about a detected person is displayed on a screen so that a monitor or the like can see it. As a technology for displaying information about a detected person, a technology for displaying an image or various information received from an authentication device has been proposed (see, for example, Patent Document 1). Patent Document 1 proposes a technology for outputting a predetermined voice guidance or buzzer sound based on the control of the authentication device in addition to displaying information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6151582 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 merely outputs a predetermined voice guidance or a buzzer sound. Therefore, it is difficult for a monitor to understand the contents of the detection just by listening to the predetermined voice guidance or the buzzer sound. Furthermore, as the number of detections increases, it becomes more difficult to understand the contents of the detection.

[0006] The present invention has been made in light of the above-mentioned circumstances, and aims to provide a technique for controlling notification in accordance with a person to be detected. [Means for solving the problem]

[0007] In order to solve the above problems, one aspect of the present invention is an apparatus or method for performing information processing related to surveillance based on image data acquired by each of a plurality of distributed sensor devices, detecting a person based on the image data, determining label information corresponding to characteristics of the detected person based on label data including one or more labels corresponding to the characteristics of the person stored in a memory unit, determining whether or not the detected person is a notification target based on notification target data including characteristics of a plurality of people that are notification targets stored in the memory unit and the determined label information, and in response to the detected person being determined to be a notification target, controlling notification in a notification manner corresponding to the determined label information. Effect of the Invention

[0008] According to one aspect of the present invention, it is possible to control notification depending on the person to be detected. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a monitoring system including a monitoring information processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a Web server device used as a monitoring information processing device according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a block diagram showing an example of the software configuration of a Web server device used as a monitoring information processing device according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating an example of the label table. [Diagram 5] FIG. 5 is a flowchart showing the procedure and contents of the notification process executed by the Web server device. [Figure 6] FIG. 6 is a diagram showing an example of characteristics of a person to be notified. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [One embodiment] (Configuration example) (1) System FIG. 1 is a diagram showing the overall configuration of a system including a monitoring information processing device according to an embodiment of the present invention. For example, multiple surveillance cameras C1-Cn are distributed in the corridors and sales floors of large stores such as shopping malls and department stores. The surveillance cameras are an example of a sensor device. The surveillance cameras C1-Cn are attached to, for example, a ceiling or a wall, and output image data captured in their respective imaging ranges. Hereinafter, image data captured by each of the multiple surveillance cameras C1-Cn distributed in the same manner is also referred to as captured image data. An area such as a large store where multiple surveillance cameras C1-Cn are distributed in the same manner is an example of a monitored area.

[0012] The surveillance cameras C1 to Cn are provided with video analysis engines VE1 to VEn, respectively. The video analysis engines VE1 to VEn extract image data of a person having image features similar to those of a surveillance target from captured image data acquired by the corresponding surveillance cameras C1 to Cn. The surveillance target is an example of a person. Hereinafter, the image data of a person extracted from captured image data by each of the video analysis engines VE1 to VEn is also referred to as person image data. The person image data includes at least one of face image data and whole body image data. The person image data is also an example of captured image data. For example, the image features of the surveillance target are obtained from a registered image based on the registered image data of the surveillance target. The registered image data includes at least one of face image data and whole body image data.

[0013] The registered image data of the monitored person is image data used for comparison with the person image data to detect the monitored person. The registered image data of the monitored person is image data showing the monitored person registered in advance. The registered image data of the monitored person may be image data showing the monitored person different from the captured image data previously acquired from a PC by the monitor or the facility manager. The monitor or the facility manager is an example of a user. The registered image data of the monitored person may be image data showing the monitored person selected in advance from the captured image data in the past by the monitor or the manager. The registered image data of the monitored person may be image data showing the monitored person selected from the captured image data acquired in real time by the monitor or the manager. Each of the video analysis engines VE1 to VEn calculates the similarity of the person image based on the person image data to the registered image based on the registered image data based on the image feature amount of the monitored person.

[0014] Each of the video analysis engines VE1 to VEn generates a video analysis result. The video analysis result includes person image data. The video analysis result includes a person ID that identifies a monitored person having image features similar to those of a person shown in the person image data. The person ID is an ID given to each person. The person ID may be set appropriately. The video analysis result includes a camera ID that identifies a surveillance camera among the surveillance cameras C1 to Cn that acquired the person image data. The video analysis result includes a similarity of a person image based on the person image data to a registered image based on the registered image data.

[0015] Each of the video analysis engines VE1 to VEn can also extract people with various characteristics from the captured image data, in addition to the surveillance target, by analyzing the captured image data.

[0016] The various features include, but are not limited to, features based on appearance, behavior, and symptoms. The various features include the state of a physically disabled person. For example, the state of a physically disabled person is, but is not limited to, a visually impaired person who uses a cane, a visually impaired person who is accompanied by a guide dog, a hearing impaired person who is accompanied by a hearing dog, a person in a wheelchair, and a person in a wheelchair who is accompanied by a service dog. The various features include the state of a person who is in poor health. For example, the state of a person who is in poor health is, but is not limited to, a person who has fallen, a person who is crouching, and a person who is walking unsteadily. The various features include a state of a lost child. A state of a lost child is, but is not limited to, a child alone. The various features include a state of a symptom. For example, the state of a symptom is, but is not limited to, a fever, a cough, a sweat, and a heart rate. Fever, cough, sweat, and heart rate are intended to be states that are higher or higher than a standard. Each state included in the various features corresponds to the contents of a label managed in a management table 33 described later. A label is information indicating a person's characteristics.

[0017] In this example, the surveillance cameras C1 to Cn may include various sensors such as a thermo camera. Each of the video analysis engines VE1 to VEn may use sensing data of various sensors such as a thermo camera when extracting a person who is mainly in a symptomatic state. In this example, each of the video analysis engines VE1 to VEn does not require registered image data. In this example, the video analysis result includes person image data related to a person with various characteristics. The video analysis result includes a person ID that identifies a person appearing in the person image data. The person ID may be set arbitrarily. Here, an example is described in which a person appearing in the person image data is tracked using a person ID, but the video analysis result may include an arbitrary ID that tracks a person appearing in the person image data instead of the person ID. The video analysis result includes a camera ID that identifies a surveillance camera that has acquired the person image data among the surveillance cameras C1 to Cn. The video analysis result includes information on characteristics related to a person appearing in the person image data. The information on characteristics is information on the contents of one or more labels corresponding to one or more conditions such as a visually impaired person using a cane, a person who has fallen, one child, and a fever.

[0018] It is also possible that the video analysis engines VE1 to VEn do not have to be arranged one-to-one with the surveillance cameras C1 to Cn, but rather a smaller number of video analysis engines are arranged for multiple cameras, and the video data from the multiple surveillance cameras is processed collectively by the smaller number of video analysis engines.

[0019] The system of one embodiment includes a Web server device SV used as a surveillance information processing device. The Web server device SV is a device that performs surveillance-related information processing based on human image data acquired by each of the distributed surveillance cameras C1 to Cn. The video analysis engines VE1 to VEn are capable of data communication with the Web server device SV via a network NW. The video analysis engines VE1 to VEn transmit video analysis results to the Web server device SV via the network NW. The network NW is, for example, a wired LAN (Local Area Network) or a wireless LAN, but may be any other network.

[0020] In addition, the Web server device SV may be equipped with video analysis engines VE1 to VEn or one video analysis engine, and the video analysis engines VE1 to VEn or one video analysis engine of the Web server device SV may receive video data from each of the surveillance cameras C1 to Cn via the network NW and analyze the received video data.

[0021] The system of one embodiment includes an output device MT and an administrator terminal OT. The output device MT is connected to a Web server device SV. The output device MT is a device having a function of displaying information output from the Web server device SV. The output device MT is also a device that outputs information output from the Web server device SV as audio. The output device MT may be a wearable device or smart glasses. The output device MT is an example of a display unit, an audio output unit, or an output unit. An observer checks the information output by the output device MT. The administrator terminal OT is connected to the Web server device SV. The administrator terminal OT is a terminal into which an administrator inputs various settings for the Web server device SV.

[0022] (2) Web server device SV FIG. 2 is a block diagram showing an example of the hardware configuration of the Web server device SV. The Web server device SV includes a control unit 1, a program storage unit 2, a data storage unit 3, an input / output I / F (interface) 4, and a communication I / F 5. The elements constituting the Web server device SV are connected to each other via a bus 6.

[0023] The control unit 1 corresponds to the central part of the Web server device SV. The control unit 1 has a hardware processor such as a central processing unit (CPU). The control unit 1 has a ROM (Read Only Memory) as a non-volatile memory area. The control unit 1 has a RAM (Random Access Memory) as a volatile memory area.

[0024] The program storage unit 2 is configured with a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) that can be written to and read from as needed as a storage medium. The program storage unit 2 stores programs necessary for executing various control processes. For example, the program storage unit 2 stores a monitoring information processing program.

[0025] The data storage unit 3 is configured with a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read from at any time. The data storage unit 3 stores management data 31, detection history data 32, a management table 33, a detection history table 34, a camera information table 35, a label table 36, and a notification target table 37.

[0026] The management data 31 includes registered image data of each person. The registered image data can be used not only by the Web server device SV but also by the video analysis engines VE1 to VEn. The registered image data is associated with the person ID of the person to be monitored. The management data 31 can be updated based on the addition or deletion of registered image data.

[0027] The detection history data 32 includes detection image data of each person. The detection image data is person image data of a person detected by the control unit 1 as described later among the person image data. The detection image data is associated with a person ID. The detection history data 32 can be updated based on the addition or deletion of detection image data.

[0028] The management table 33 manages records for each person ID in which the person ID and the person name are associated with each other. The person name is a name such as the person's name. The person name can be set appropriately. The label is information indicating the characteristics of the person. The management table 33 can be updated based on updates to the data constituting the record. The management table 33 can be updated based on addition or deletion of a record.

[0029] The detection history table 34 manages records for each detection ID in which the detection ID, person ID, detection camera ID, detection date and time, etc. are associated with each other. The detection ID is an ID given to each detection image data. The detection ID can be set as appropriate. The detection camera ID is an ID that identifies the monitoring camera that acquired the detection image data. The detection camera ID corresponds to the camera ID. The camera ID is an ID that is given to each of the monitoring cameras C1 to Cn. The camera ID can be set as appropriate. The detection date and time is the date and time when a person is detected by the control unit 1 based on the person image data, as described later. The date and time when a person is detected by the control unit 1 based on the person image data corresponds to the date and time when the person image data is captured by any one of the monitoring cameras C1 to Cn. The detection date and time is the date and time for each detection. The detection history table 34 can be updated based on updates to the data that constitutes the record. The detection history table 34 can be updated based on addition or deletion of a record.

[0030] The camera information table 35 manages records for each camera ID in which the camera ID, camera name, restricted area information, etc. are associated with each other. The camera name is a name related to the surveillance camera. For example, the camera name is the name of the installation location of the surveillance camera, etc. The camera name can be set as appropriate. The restricted area information is information indicating that the surveillance camera is installed in a restricted area. The restricted area information can be set as appropriate by an administrator. The camera information table 35 can be updated based on updates to the data that constitutes the record. The camera information table 35 can be updated based on addition or deletion of a record.

[0031] The label table 36 includes one or more labels according to the characteristics of a person. The label table 36 includes one or more labels in each of one or more label groups. The label groups are groups for each item according to type. The label group includes one label according to the characteristics of a person according to the item, or two or more labels according to different characteristics of a person according to the item. By including one or more labels in each of one or more label groups according to type in the label table 36, it becomes possible to determine the characteristics of a person detected by the control unit 1 described later from various perspectives.

[0032] The label table 36 includes at least one of one or more label groups including one or more labels assigned to each person and one or more label groups including one or more labels according to the state of the person. Hereinafter, the label group including one or more labels assigned to each person is also referred to as a first label group. The label group including one or more labels according to the state of the person is also referred to as a second label group. When the label table 36 includes the first label group, it becomes possible to appropriately determine the characteristics of the person detected by the control unit 1 described later according to the characteristics assigned to each person in advance. When the label table 36 includes the second label group, it becomes possible to dynamically determine the characteristics of the person detected by the control unit 1 described later according to the state of the person in real time.

[0033] For example, the label table 36 includes one or more label groups according to the type in each of one or more categories. The categories correspond to the purpose of monitoring. For example, the categories include, but are not limited to, crime prevention, marketing, compassion, and infection control. The label table 36 may include only one category, or may include multiple categories. The label table 36 may include only one label group in each category, or may include multiple label groups. The label table 36 may include only one or more first label groups in each category. Alternatively, the label table 36 may include only one or more second label groups in each category. Alternatively, the label table 36 may include one or more first label groups and one or more second label groups in each category.

[0034] The label table 36 includes, for each of one or more first label groups, a person ID associated with each of one or more labels included in the first label group. Each of the one or more labels included in the first label group is associated with one or more person IDs. The one or more person IDs associated with each of the one or more labels included in the first label group may be set appropriately by an administrator. Here, an example will be described in which the label table 36 manages information in which each of the one or more labels included in the first label group is associated with one or more person IDs, but this is not limiting. The management table 33 may manage information in which each of the one or more labels included in the first label group is associated with one or more person IDs.

[0035] The categories, label groups, and labels can be set appropriately by an administrator. The label table 36 can be updated based on the settings of the categories, label groups, and labels. The label table 36 is an example of label data.

[0036] The notification target table 37 includes the characteristics of a plurality of people who are notification targets. A person who is a notification target is a person who is a notification target in response to detection. Notification is to notify a monitor. The notification mode includes a notification method and a notification destination. For example, the notification method includes at least one of visual notification, auditory notification, and tactile notification, but is not limited to these. The notification method may be a notification method that can be perceived by a person. The visual notification is, but is not limited to, various displays and messages such as icons of browser tabs on the display screen of the output device MT. The visual notification may include notifications by email, chat, etc. The auditory notification is, but is not limited to, audio output such as sounds and messages from the output device MT. The tactile notification is output of tactile stimuli such as vibrations from a sensor or a contact device provided in the output device MT. The notification destination is, for example, a destination of email, a chat ID, etc.

[0037] The characteristics of a person to be notified are set based on one or more labels included in the label table 36. In one example, the characteristics of a person to be notified are set based on one label in one label group out of one or more label groups included in the label table 36. The one label group may be a first label group or a second label group. Setting the characteristics of a person to be notified based on one label makes it possible to set a wide range of people to be notified. Setting a wide range of people to be notified makes it possible to increase the number of people determined to be people to be notified.

[0038] In another example, the characteristics of a person to be notified are set based on a combination of one or more labels in at least two label groups among one or more label groups included in the label table 36. The at least two label groups may be in the same division, or may span two or more different divisions. The at least two label groups may be composed of only the first label group, or may be composed of only the second label group. The at least two label groups may be composed of the first label group and the second label group. Setting the characteristics of a person to be notified based on a combination of multiple labels makes it possible to narrow the range of the notification target. Setting the range of the notification target to be narrow makes it possible to realize notification limited to necessary people.

[0039] The characteristics of a person to be notified include characteristics associated with different durations. The duration is the time during which the same person continues to have the same characteristic. The duration may be a point in time or a predetermined time. The length of the predetermined time may be set appropriately. The characteristics of a person to be notified may include different characteristics depending on the duration. For example, a characteristic of a person to be notified that is set based on one or more labels may be a target for a duration of a point in time but not a target for a duration of a predetermined time. The same is true vice versa.

[0040] The characteristics of a person to be notified include characteristics associated with the duration of stay in the monitoring area. In this example, the characteristics of a person to be notified in the first detection of a person by the control unit 1 described later are different from the characteristics of a person to be notified in the long-term detection of a person by the control unit 1. The first detection is the first detection in a day in the monitoring area. The first detection is an example of a detection in a state where the duration of stay in the monitoring area is short. The first detection is an example of a current detection. The characteristics of a person to be notified in the first detection are an example of a characteristic of a person to be notified at a single point in time whose duration of stay is longer than the long-term determination reference time in a day. The long-term determination reference time is a reference time for determining a long stay. For example, the long-term determination reference time is 20 minutes, but is not limited thereto. The long-term detection is an example of a detection in a state where the duration of stay in the monitoring area is long.

[0041] The characteristics of a person who is to be notified may include different characteristics depending on the duration of stay in the monitoring area. For example, the characteristics of a person who is to be notified, which are set based on one or more labels, are the subject of notification in the first detection but not in the long-term detection. The same is true vice versa. The characteristics of a person who needs to be notified in the first detection and the characteristics of a person who needs to be notified in the long-term detection are different, but the characteristics of a person who is to be notified can be appropriately set depending on the duration of stay.

[0042] The notification target table 37 includes notification modes associated with the characteristics of a person who is a notification target. The notification mode may differ for each of the person's characteristics, such as by displaying or outputting a different sound for each person's characteristics. For example, the different display may be a different blinking pattern of an icon or a different message. The different sound output may be a different sound pattern or a different message. This is to allow the monitor to easily identify the characteristics of the person who has been detected. The monitor or administrator can arbitrarily set an appropriate notification mode according to the characteristics of the person who is a notification target, such as setting a more characteristic notification mode for the characteristics of a specific person who is a notification target.

[0043] The characteristics of the people who are the notification targets and the notification manner can be appropriately set by an administrator. The notification target table 37 can be updated based on the settings of the characteristics of the people who are the notification targets and the notification manner. The notification target table 37 is an example of notification target data.

[0044] The input / output I / F 4 is configured with an interface that connects to the output device MT and the administrator terminal OT. The communication I / F 5 transmits data between the video analysis engines VE1 to VEn using a communication protocol defined by the network NW under the control of the control unit 1. For example, the communication I / F 5 is configured by an interface compatible with a wired LAN or a wireless LAN.

[0045] FIG. 3 is a block diagram showing an example of the software configuration of the Web server device SV. The control unit 1 includes a detection unit 11, an output unit 12, a determination unit 13, a decision unit 14, a judgment unit 15, and a notification control unit 16. Each unit is realized by causing a hardware processor of the control unit 1 to execute a monitoring information processing program stored in the program storage unit 2. Each unit can also be said to be provided by a hardware processor. Each unit can also be said to be a function. Note that each unit described below can be read as the control unit 1 or the hardware processor.

[0046] The detection unit 11 detects a person based on the person image data. For example, the detection unit 11 acquires a video analysis result from each of the video analysis engines VE1 to VEn. The detection unit 11 detects a person appearing in the person image data based on the person image data included in the video analysis result. If the video analysis result includes a similarity related to a person to be monitored, the detection unit 11 compares the similarity included in the video analysis result with a threshold value. The threshold value is a reference value for determining whether or not a person appearing in the person image data is to be regarded as a person to be monitored. The threshold value can be set appropriately. If the similarity is equal to or greater than the threshold value, the detection unit 11 detects the person appearing in the person image data as a person to be monitored. The detection unit 11 may detect the same person only once. The detection unit 11 may detect the same person multiple times at different times. The detection unit 11 stores the person image data including the detected person as detected image data in the detection history data 32 in association with a person ID included in the video analysis result.

[0047] The detection unit 11 acquires the detection date and time based on the detection. Hereinafter, the date and time when the detection unit 11 first detects a person is also referred to as the first detection date and time or the first detection date and time. The first detection corresponds to the first detection. The date and time when the detection unit 11 last detects the same person is also referred to as the latest detection date and time. The detection unit 11 adds a record to the detection history table 34 in which the detection ID, the person ID, the detection camera ID, and the detection date and time are associated with each other based on the detection of the person. The detection unit 11 uses the person ID included in the video analysis result. The detection unit 11 uses the camera ID included in the video analysis result as the detection camera ID. The detection unit 11 uses the acquired detection date and time. The detection unit 11 adds a record to the detection history table 34 every time it detects a person.

[0048] The detection unit 11 may send an instruction to output the detection result to the output unit 12 every time a person is detected. Each time the detection unit 11 detects the same person for the second or subsequent times, it acquires the first detection date and time associated with the person ID of the person from the detection history table 34. Each time the detection unit 11 detects the same person for the second or subsequent times, it may send the person ID, the first detection date and time, and the latest detection date and time to the determination unit 13. The detection unit 11 may send the person ID, the camera ID, and the feature information included in the video analysis result for each detection to the determination unit 14. Note that the feature information may be generated by the detection unit 11 based on the person image data instead of the video analysis engines VE1 to VEn.

[0049] The detection unit 11 can also detect that the detected person has the same characteristic for a predetermined period of time based on the video analysis result. The detection unit 11 may detect that the detected person has the same characteristic for a predetermined period of time based on the video analysis result of one surveillance camera. The detection unit 11 may detect that the detected person has the same characteristic for a predetermined period of time based on the video analysis result of multiple surveillance cameras.

[0050] The output unit 12 outputs the detection result for each of the multiple people detected by the detection unit 11 to a detection alert output area included in the display screen of the output device MT. The detection result is information related to the detection. Based on the first detection of a person, the output unit 12 outputs a new detection result related to this person to the detection alert output area. The output unit 12 outputs the detection results for each of the multiple people to the detection alert output area, arranging them in order of the first detection date and time of each of the multiple people.

[0051] An example of information included in the detection result will be described. The detection result may include a person ID. The output unit 12 may acquire the person ID from the detection unit 11. The detection result may include a person name. The output unit 12 may acquire the person name from the management table 33 based on the person ID. The detection result may include a registered image based on registered image data. The output unit 12 may acquire the registered image data from the management data 31 based on the person ID. The detection result may include a detection image based on detection image data related to the detection at the latest detection date and time. The output unit 12 may acquire the detection image data related to the detection at the latest detection date and time from the detection unit 11. The detection result may include a similarity. The output unit 12 may acquire the similarity from the detection unit 11. The detection result may include information on the history of the detection. The information on the history of the detection may include information on the detection at the first detection date and time. The information on the detection at the first detection date and time includes the first detection date and time and the camera name of the surveillance camera that acquired the detection image data related to the detection at the first detection date and time. The output unit 12 may acquire the first detection date and time from the detection history table 34 based on the person ID. The output unit 12 may acquire, based on the person ID, from the detection history table 34, the detection camera ID of the surveillance camera that acquired the detection image data related to the detection at the first detection date and time. The output unit 12 may acquire, based on the detection camera ID, from the camera information table 35, the camera name of the surveillance camera that acquired the detection image data related to the detection at the first detection date and time. The information on the detection history may include information on the detection at the latest detection date and time. The information on the detection at the latest detection date and time includes the latest detection date and time and the camera name of the surveillance camera that acquired the detection image data related to the detection at the latest detection date and time. The output unit 12 may acquire the latest detection date and time from the detection unit 11. The output unit 12 acquires the camera ID included in the video analysis result from the detection unit 11. The output unit 12 may acquire, based on the camera ID, from the camera information table 35, the camera name of the surveillance camera that acquired the detection image data related to the detection at the latest detection date and time. The information on the detection history may include information on some or all of the detection dates and times between the first detection date and time and the latest detection date and time. The detection result may include the number of detections.The output unit 12 can obtain the number of detections based on the number of records associated with the same person ID in the detection history table .

[0052] The output unit 12 outputs the detection result regarding the person determined as a long-time resident to a long-time resident alert output area included in the display screen of the output device MT based on the long-time resident determination result by the determination unit 13 described later. The detection result output to the long-time resident alert output area is the same as the detection result output to the detection alert output area.

[0053] The determination unit 13 determines whether the interval from the first detection date and time to each of the second and subsequent detection dates and times of a plurality of persons is equal to or longer than the long-term determination reference time. The determination unit 13 determines a person detected at an interval equal to or longer than the long-term determination reference time as a long-term resident in the monitoring area. The determination unit 13 sends the determination result of the long-term resident to the output unit 12.

[0054] The determination unit 14 determines label information according to the characteristics of the person detected by the detection unit 11 based on the label table 36. The label information is information indicating one or more labels. An example of the process performed by the determination unit 14 will be described later.

[0055] The determination unit 15 determines whether or not the person detected by the detection unit 11 is a notification target, based on the notification target table 37 and the label information determined by the determination unit 14. An example of the process performed by the determination unit 15 will be described later.

[0056] In response to the person detected by the detection unit 11 being determined by the determination unit 15 to be a notification target, the notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14. An example of processing by the notification control unit 16 will be described later.

[0057] An example of the configuration of the label table 36 will now be described. FIG. 4 is a diagram showing an example of the label table 36. As shown in FIG. The label table 36 includes categories relating to crime prevention, marketing, compassion, and infection control.

[0058] The label table 36 includes a label group related to crime prevention history in the crime prevention category. The label group related to crime prevention history includes multiple labels according to the risk of a person. The label group related to crime prevention history includes the labels danger, caution, and caution. Each of the labels danger, caution, and caution is associated with one or more person IDs by an administrator. The label group related to crime prevention history is an example of a first label group. Note that information associating each of the labels danger, caution, and caution with one or more person IDs may be managed in the management table 33.

[0059] The label table 36 includes a label group related to restricted area management in the category related to crime prevention. The label group related to restricted area management includes a plurality of labels according to whether or not the relevant person is allowed to enter the restricted area. The label group related to restricted area management includes each label for the relevant person and the non-relevant person. Each label for the relevant person and the non-relevant person is associated with one or more person IDs by the administrator. The label group related to restricted area management is an example of a first label group. Note that information associating each label for the relevant person and the non-relevant person with one or more person IDs may be managed in the management table 33.

[0060] The label table 36 includes a label group related to purchase amount in the marketing-related category. The label group related to purchase amount includes a plurality of labels corresponding to past purchase amounts. The label group related to purchase amount includes each of the labels VVIP, VIP, diamond, gold, silver, and bronze. Each of the labels VVIP, VIP, diamond, gold, silver, and bronze is associated with one or more person IDs by the administrator. The label group related to purchase amount is an example of a first label group. Note that information associating each of the labels VVIP, VIP, diamond, gold, silver, and bronze with one or more person IDs may be managed in the management table 33.

[0061] The label table 36 includes a label group related to visit frequency in the marketing-related category. The label group related to visit frequency includes a plurality of labels according to past visit frequencies. The label group related to visit frequency includes the labels of 30 times or more, 20 times or more, 10 times or more, 1 time or more, and 0 times (newcomer). Each of the labels of 30 times or more, 20 times or more, 10 times or more, and 1 time or more is associated with one or more person IDs by the administrator. The label of 0 times or more is not associated with a person ID, but is associated with a person ID other than the plurality of person IDs associated with each of the labels of 30 times or more, 20 times or more, 10 times or more, and 1 time or more. The label group related to purchase amount is an example of a first label group. Note that information associating each of the labels of 30 times or more, 20 times or more, 10 times or more, and 1 time or more with one or more person IDs may be managed in the management table 33.

[0062] The label table 36 includes a label group related to complaint frequency in the marketing category. The label group related to complaint frequency includes multiple labels according to past complaint frequencies. The label group related to complaint frequency includes the labels 10 or more, 5 or more, and 0. The labels 10 or more, 5 or more, and 0 are associated with one or more person IDs by the administrator. The label group related to complaint frequency is an example of a first label group. Note that information associating the labels 10 or more, 5 or more, and 0 with one or more person IDs may be managed in the management table 33.

[0063] The label table 36 includes a label group related to the number of visitors in the marketing-related category. The label group related to the number of visitors includes multiple labels corresponding to the current number of visitors. The label group related to the number of visitors includes each of the labels 1 person, 2 people, 3 people, and 4 or more people. The multiple labels included in the label group related to the number of visitors correspond to the state of the person to be detected, and therefore are not associated with one or more person IDs. The label group related to the number of visitors is an example of a second label group.

[0064] The label table 36 includes a label group for physically disabled people in the category related to "heartful". The label group for physically disabled people includes multiple labels according to the current state of the visitor. The label group for physically disabled people includes the labels of a visually impaired person who uses a cane, a visually impaired person accompanied by a guide dog, a hearing impaired person accompanied by a hearing dog, a person in a wheelchair, and a person in a wheelchair accompanied by a service dog. The multiple labels included in the label group for physically disabled people correspond to the state of the person to be detected, and therefore are not associated with one or more person IDs. The label group for physically disabled people is an example of a second label group.

[0065] The label table 36 includes a label group related to people who are in poor health in the heartfelt category. The label group related to people who are in poor health includes multiple labels according to the current state of the visitor. The label group related to people who are in poor health includes the labels of a person who has fallen, a person who is crouching, and a person who is staggering. The multiple labels included in the label group related to people who are in poor health correspond to the state of the person to be detected, and are therefore not associated with one or more person IDs. The label group related to people who are in poor health is an example of a second label group.

[0066] The label table 36 includes a label group related to a lost child in the heartfelt section. The label group related to a lost child includes one label corresponding to the current state of a visitor. The label group related to a lost child includes a label for one child. The labels included in the label group related to a lost child correspond to the state of the person being detected, and are therefore not associated with one or more person IDs. The label group related to a person who is not feeling well is an example of a second label group.

[0067] The label table 36 includes a label group related to symptoms in the category related to infection control. The label group related to symptoms includes multiple labels according to the current condition of the visitor. The label group related to symptoms includes the labels of fever, cough, sweat, and heart rate. The multiple labels included in the label group related to symptoms correspond to the condition of the person to be detected, and are therefore not associated with one or more person IDs. The label group related to people who are feeling unwell is an example of a second label group.

[0068] (Example of operation) Next, an example of the operation of the Web server device SV configured as above will be described. Fig. 5 is a flowchart showing the procedure and contents of the notification process executed by the Web server device SV. For the sake of simplicity, the flowchart in Fig. 5 shows the notification process based on one person detection, but in reality, the process shown in Fig. 5 is repeated every time a person is detected.

[0069] The detection unit 11 detects a person based on the person image data (step S1). In step S1, for example, the detection unit 11 detects a person appearing in the person image data based on the person image data as described above. The detection unit 11 may send information on the person ID, the camera ID, and features to the determination unit 14 for each detection.

[0070] The determination unit 14 determines label information according to the characteristics of the person detected by the detection unit 11 based on the label table 36 (step S2). In step S2, for example, the determination unit 14 receives information on the person ID, the camera ID, and the characteristics from the detection unit 11 each time the detection unit 11 detects a person.

[0071] The determination unit 14 refers to the person ID and acquires one or more labels corresponding to the person ID based on the label table 36 or the management table 33. The one or more labels acquired by the determination unit 14 with reference to the person ID correspond to the labels included in one or more first label groups. If the camera ID from the detection unit 11 is associated with restricted area information in the camera information table 35, the determination unit 14 acquires a label associated with the person ID from a label group related to restricted area management. The determination unit 14 can acquire one label from one first label group. The determination unit 14 may acquire multiple labels from one first label group depending on the items of the label group.

[0072] The determination unit 14 refers to the feature information and acquires one or more labels corresponding to the feature information based on the label table 36. The one or more labels acquired by the determination unit 14 with reference to the feature information correspond to the labels included in each of the one or more second label groups. The determination unit 14 can acquire one label from one second label group. The determination unit 14 may acquire multiple labels from one second label group depending on the items of the label group.

[0073] A case will be described where the determining unit 14 acquires one label. The determining unit 14 determines label information based on one label in one of the one or more label groups included in the label table 36. The one label may be one label acquired from the first label group, or one label acquired from the second label group.

[0074] A case where the determining unit 14 acquires a plurality of labels will be described. The determining unit 14 determines label information based on a combination of one or more labels in at least two label groups among the one or more label groups included in the label table 36. The at least two label groups may be only at least two first label groups. The at least two label groups may be only at least two second label groups. The at least two label groups may be both one or more first label groups and one or more second label groups. The at least two label groups may be in the same division, or may span two or more different divisions.

[0075] The determination unit 14 may determine the label information according to the characteristics of the person at any one detection timing. Alternatively, the determination unit 14 may determine the label information according to the characteristics of the person at a plurality of detection timings for the same person. In the latter case, the determination unit 14 may determine the label information based on a combination of one or more labels according to the characteristics of the person at the first detection timing and one or more labels according to the characteristics of the person at the second detection timing at a second detection timing after the first detection timing. For example, the determination unit 14 can determine the label information based on a combination of one or more labels in each of a label group related to the number of visitors and a label group related to a lost child. In this example, the label of the label group related to the number of visitors is a label according to the characteristics of the person at the first detection timing. The label of the label group related to a lost child is a label according to the characteristics of the person at the second detection timing.

[0076] The determination unit 14 determines label information based on the one or more labels obtained as described above. The determination unit 14 sends the determined label information to the judgment unit 15.

[0077] The determination unit 15 determines whether or not the person detected by the detection unit 11 is a notification target based on the notification target table 37 and the label information determined by the determination unit 14 (step S3). In step S3, for example, the determination unit 15 compares a combination of one or more labels indicated by the label information determined by the determination unit 14 with a combination of one or more labels set for each of the characteristics of the person as a notification target included in the notification target table 37. The determination unit 15 determines whether or not there is a characteristic of the person as a notification target to which one or more labels matching all of the one or more labels indicated by the label information are set in the notification target table 37. The determination unit 15 may determine whether or not there is a characteristic of the person as a notification target to which one or more labels matching some of the one or more labels indicated by the label information are set in the notification target table 37. Hereinafter, a characteristic of the person as a notification target to which one or more labels matching all or some of the one or more labels indicated by the label information are set is also referred to as a characteristic of the person as a notification target corresponding to the label information. When there is no characteristic of a person who is a notification target corresponding to the label information, the determination unit 15 determines that the person detected by the detection unit 11 is not a notification target. When there is a characteristic of a person who is a notification target corresponding to the label information, the determination unit 15 determines that the person detected by the detection unit 11 is a notification target. The determination unit 15 sends the characteristic of the person who is a notification target corresponding to the label information to the notification control unit 16.

[0078] In response to the judgment by the judgment unit 15 that the person detected by the detection unit 11 is not a notification target (step S4, NO), the processing ends. In response to the judgment by the judgment unit 15 that the person detected by the detection unit 11 is a notification target (step S4, YES), the notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14 (step S5). In step S5, for example, the notification control unit 16 determines the notification mode associated with the characteristics of the person who is a notification target corresponding to the label information based on the notification target table 37. The notification mode associated with the characteristics of the person who is a notification target corresponding to the label information is an example of a notification mode according to the label information.

[0079] The notification control unit 16 controls notification of a person detected by the detection unit 11 in a determined notification mode. For example, the notification control unit 16 displays an icon or a message on the output device MT based on the notification mode determined according to the characteristics of the detected person. Alternatively or in addition to this, for example, the notification control unit 16 outputs a sound or a message from the output device MT based on the notification mode determined according to the characteristics of the detected person. Note that the notification control unit 16 may control the display of the detection result of the person detected by the detection unit 11, which is output to the detection alert output area or the long stay alert output area, based on the determined notification mode. In this example, the notification control unit 16 may control the display of an icon or a message in the detection result, or the display of the detection result, such as lighting or blinking.

[0080] FIG. 6 is a diagram showing an example of characteristics of a person to be notified. The notification target table 37 includes characteristics of people who are to be notified based on the “danger” label from the label group related to the crime prevention history included in the label table 36. These characteristics are characteristics of people who are to be notified upon initial detection. In this example, the determination unit 14 determines label information based on the "danger" label according to the characteristics of the person detected for the first time by the detection unit 11. The judgment unit 15 determines that the person has the characteristics of a person to be notified, which is set with the "danger" label indicated by the label information in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14. This allows the monitor to know in real time when a high-risk individual is approaching and take measures to prevent an incident from occurring.

[0081] The notification target table 37 includes characteristics of people who are the notification targets set based on the “VVIP” label from among the label groups related to purchase amounts included in the label table 36. These characteristics are characteristics of people who are the notification targets upon initial detection. In this example, the determination unit 14 determines label information based on the "VVIP" label according to the characteristics of the person detected for the first time by the detection unit 11. The judgment unit 15 determines that the person has the characteristics of a person to be notified, which is set with the "VVIP" label indicated by the label information in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14. This allows the monitor to grasp in real time the presence of people with a strong desire to buy, and to take measures to proactively sell products.

[0082] The notification target table 37 includes the characteristics of the person who is the notification target, which are set based on the “fall” label from the label group related to physically disabled people included in the label table 36. This characteristic is a characteristic of the person who is the notification target at a certain point in time. This characteristic is not limited to the characteristics of the person who is the notification target at the first detection. In this example, the determination unit 14 determines label information based on the "fall" label according to the characteristics of a person detected by the detection unit 11 at any one time. The judgment unit 15 determines that the person has the characteristics of a person to be notified, for which the "fall" label indicated by the label information is set in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14. This allows the observer to detect the person who has fallen in real time and take action such as providing rescue.

[0083] The notification target table 37 includes characteristics of the person who is the notification target, which are set based on a combination of the label "one child" from the label group related to lost children and the label "two people" from the label group related to the number of visitors included in the label table 36. These characteristics are characteristics of the person who is the notification target at a certain point in time. These characteristics are not limited to the characteristics of the person who is the notification target at the first detection. In this example, the determination unit 14 determines label information based on the "one child" label and the "two children" label according to the characteristics of a person detected by the detection unit 11 at any one time. The judgment unit 15 judges that the person has characteristics that make him or her a notification target, and that the combination of the "one child" label and the "two children" label indicated by the label information is set in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14. This allows the monitor to identify lost people in real time and take appropriate action, such as providing protection for them.

[0084] The notification target table 37 includes characteristics of people who are notification targets set based on the “non-related person” label from the label group related to the restricted area management included in the label table 36. These characteristics are characteristics of people who are notification targets in long-term detection. In this example, the determination unit 14 determines label information based on the "non-related person" label according to the characteristics of a person detected by the detection unit 11 and determined as a long-time resident by the determination unit 13. The determination unit 15 determines that the person has the characteristics of a person who is a notification target with the "non-related person" label set as indicated by the label information in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14. This allows supervisors to grasp in real time whether someone is staying in a restricted area for an extended period of time and take appropriate action, such as sending out patrols.

[0085] The notification target table 37 includes characteristics of people who are notification targets set based on a combination of the label "30 times or more" from the label group related to the frequency of visits and the label "10 times or more" from the label group related to the frequency of complaints included in the label table 36. These characteristics are characteristics of people who are notification targets in long-term detection. In this example, the determination unit 14 determines label information based on the "30 times or more" label and the "10 times or more" label according to the characteristics of a person who is detected by the detection unit 11 and determined as a long-time resident by the determination unit 13. The determination unit 15 determines that the person has the characteristics of a person who is a notification target set with a combination of the "30 times or more" label and the "10 times or more" label indicated by the label information in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14. This allows the monitor to grasp in real time the long-term stay of a person who is of interest for marketing purposes and to take appropriate measures.

[0086] The notification target table 37 includes characteristics of people who are the notification targets, which are set based on the “person in a wheelchair” label from the label group related to physically disabled people included in the label table 36. These characteristics are characteristics of people who are the notification targets in long-term detection. In this example, the determination unit 14 determines label information based on the "wheelchair person" label according to the characteristics of a person who is detected by the detection unit 11 and determined by the determination unit 13 as a long-time resident. The determination unit 15 determines that the person has the characteristics of a person who is a notification target set with the "wheelchair person" label indicated by the label information in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14. This allows the supervisor to grasp in real time if a disabled person is staying for an extended period of time and take action such as providing assistance.

[0087] The notification target table 37 includes characteristics of people who are notification targets that are set based on a combination of the “person crouching” label from a label group related to people who are in poor physical condition and the “cough” label from a label group related to infection control measures included in the label table 36. These characteristics are characteristics of people who are notification targets in long-term detection. In this example, the determination unit 14 determines label information based on the "person crouching" label and the "coughing" label according to the characteristics of the person detected by the detection unit 11 and determined by the determination unit 13 as a long-time resident. The determination unit 15 determines that the person has the characteristics of a person who is a notification target set with a combination of the "person crouching" label and the "coughing" label indicated by the label information in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14. This allows the monitor to grasp in real time if an unwell person is staying for a long period of time and take action such as providing rescue.

[0088] As described above, the characteristics of the person to be notified can be set appropriately. The characteristics of the person to be notified may be set not only for the purpose of immediate response as described above, but also for future use. For example, by setting the characteristics of the person to be notified in relation to marketing, data such as the number of newcomers visiting and people in various states who stay for a long time can be collected. This data can be used for marketing such as sales strategy planning.

[0089] According to one embodiment, the Web server device SV can control the notification according to the detected person. This allows the monitor to not only know the occurrence of a new detection, but also to easily understand what characteristics the newly detected person has. The monitor can immediately take appropriate measures against the detected person.

[0090] The monitoring information processing device may be realized by one device as explained in the above example, or may be realized by a plurality of devices with distributed functions.

[0091] The program may be transferred in a state where it is stored in an apparatus, or in a state where it is not stored in an apparatus. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The recording medium may be in any form, such as a CD-ROM or a memory card, as long as it is capable of storing the program and is readable by a computer.

[0092] Although the embodiment of the present invention has been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted.

[0093] In short, this invention is not limited to the above-mentioned embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0094] SV: Web server device OT: Administrator terminal MT: Output device NW…Network C1~Cn…Surveillance cameras VE1~VEn...Video analysis engine 1...Control unit 2...Program memory section 3. Data storage section 4...Input / output interface 5...Communication I / F 6…Bus 11…Detection unit 12...Output section 13…Judgment section 14. Decision making section 15... Judgment section 16...Notification control unit 31...Management data 32…Detection history data 33...Management table 34…Detection history table 35…Camera information table 36…Label table 37…Notification table

Claims

1. A surveillance information processing device that processes surveillance-related information based on image data acquired by each of a plurality of distributed sensor devices, A storage unit that stores label data including one or more labels corresponding to characteristics of a person and notification target data including characteristics of a plurality of people who are notification targets; A detection unit that detects a person based on the image data; an output unit that outputs a detection result relating to each of the plurality of people detected by the detection unit to a detection alert output area included in a display screen of a display unit; a determination unit that determines label information according to characteristics of the person detected by the detection unit based on the label data; a determination unit that determines whether or not the person detected by the detection unit is a notification target based on the notification target data and the label information determined by the determination unit; a notification control unit that controls a display of the detection result of the person detected by the detection unit, which is output to the detection alert output area, based on a notification mode corresponding to the label information determined by the determination unit, in response to the person detected by the detection unit being determined to be a notification target by the determination unit; and A monitoring information processing device comprising:

2. The monitoring information processing device according to claim 1 , wherein the label data includes one or more labels in each of one or more groups according to type.

3. The surveillance information processing device according to claim 2 , wherein the label data includes at least one of one or more groups including one or more labels assigned to each person and one or more groups including one or more labels according to a state of the person.

4. the notification target data includes a characteristic of a person based on one label in one of the one or more groups; the determination unit determines the label information based on one label in one group of the one or more groups. The monitoring information processing device according to claim 2 .

5. the notification target data includes characteristics of a person based on a combination of one or more labels in each of at least two groups among the one or more groups; the determination unit determines the label information based on a combination of one or more labels in each of at least two groups among the one or more groups. The monitoring information processing device according to claim 2 .

6. The monitoring information processing apparatus according to claim 1 , wherein the characteristics of the plurality of persons to be notified include characteristics that vary depending on a stay time in a monitoring area in which the plurality of sensor devices are distributed.

7. the notification subject data includes notification modes associated with the characteristics of the persons who are the plurality of notification subjects, The notification control unit determines a notification mode corresponding to the label information based on the notification target data. The monitoring information processing device according to claim 1 .

8. A surveillance information processing method executed by a surveillance information processing device that processes surveillance-related information based on image data acquired by each of a plurality of distributed sensor devices, comprising: detecting a person based on the image data; outputting a detection result for each of the plurality of detected persons to a detection alert output area included in a display screen of a display unit; determining label information corresponding to the characteristics of the detected person based on label data including one or more labels corresponding to the characteristics of the person stored in a storage unit; determining whether a detected person is a notification target based on notification target data including characteristics of a plurality of notification targets stored in the storage unit and the determined label information; a step of controlling a display of the detection result of the detected person output to the detection alert output area based on a notification mode corresponding to the determined label information in response to the detected person being determined to be a notification target; A monitoring information processing method comprising:

9. 8. A monitoring information processing program for causing a processor included in the monitoring information processing device to execute processing of each of the units included in the monitoring information processing device according to claim 1.

Images