Monitoring device, monitoring method, and program

Abstract

A monitoring device (102) comprises: a detection unit (103) that detects a person P who took a predetermined action, on the basis of an image capturing an inspection area R; and an output unit (104) that outputs detection information relating to the detected person P. The predetermined action includes, for example, at least one of a first action determined by a relationship with an article, a second action determined in relation to the line of sight of the person, a third action determined in relation to movement (including staying) of the person, a fourth action determined in relation to entering / exiting of a predetermined area, and the like.

Description

[Technical Field]

[0001] The present invention relates to a monitoring device. , Supervised Viewing methods and programs. [Background technology]

[0002] The processing device described in Patent Document 1 includes an image analysis means, a registration management means, and a change determination means. The image analysis means extracts multiple types of feature amounts of a person detected from an image. The registration management means determines, based on the extracted feature amounts, whether data of the detected person is stored in a memory unit that stores feature amounts for each of multiple people. If it is determined that data of the detected person is stored, the change determination means determines whether or not there has been a change in the appearance of the detected person based on the feature amounts stored in the memory unit and the extracted feature amounts.

[0003] The image search device described in Patent Document 2 includes a posture estimation unit, a feature extraction unit, an image database, a query generation unit, and an image search unit. The posture estimation unit recognizes posture information of a search target, which is composed of multiple feature points, from an input image. The feature extraction unit extracts features from the posture information and the input image. The image database stores the features in association with the input image. The query generation unit generates a search query from posture information specified by a user. The image search unit searches the image database for images containing similar postures according to the search query.

[0004] The video monitoring device described in Patent Document 3 has an imaging unit, a video processing unit, a gaze feature calculation unit, an information recording unit, and a notification unit. The video processing unit detects people from images captured by the imaging unit and extracts gaze direction information of the people. The gaze feature calculation unit calculates gaze feature values ​​from the gaze direction information for each person. The information recording unit records the images obtained from the imaging unit, the gaze direction information for each person, and the gaze feature values. The notification unit obtains information about the behavior of the captured people from the gaze feature values ​​recorded in the information recording unit and notifies the user.

[0005] Patent Document 4 describes that the stress index of an entrant is calculated from biometric data detected by a biometric sensor in the immigration inspection area, and if the stress index is higher than a predetermined reference value, the entrant is presumed to be a suspicious person. It also describes that the biometric data can include vital data such as brain waves, cerebral blood flow, pulse waves, blood pressure, respiratory rate, body temperature, and sweat rate. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-160883 [Patent Document 2] Japanese Patent Application Publication No. 2019-091138 [Patent Document 3] Japanese Patent Application Laid-Open No. 2007-006427 [Patent Document 4] Japanese Patent Application Publication No. 2018-037075 Summary of the Invention [Problem to be solved by the invention]

[0007] In general, inspection areas such as customs are places that require particular vigilance in order to crack down on illegal activities. On the other hand, inspection areas are also difficult to monitor because they are often crowded with people taking off and landing and there are many people carrying large baggage.

[0008] However, Patent Documents 1 to 3 do not disclose any technology for monitoring the inspection site, and therefore the technologies described in Patent Documents 1 to 3 may not be able to effectively support monitoring at the inspection site.

[0009] In Patent Document 4, if the stress index calculated from the biometric data of an entrant is below a predetermined reference value, there is a risk that the entrant will not be detected as a suspicious person who requires control, etc. Therefore, even the technology described in Patent Document 4 may not be able to effectively support surveillance at inspection sites.

[0010] In view of the above-mentioned problems, an example of an object of the present invention is to provide a monitoring device, a monitoring system, a monitoring method, and a program that solve the risk of not being able to effectively support monitoring at inspection sites. [Means for solving the problem]

[0011] According to one aspect of the present invention, a detection means for detecting a person who has performed a predetermined action based on an image of the inspection site; and an output means for outputting detection information relating to the detected person. 、 the predetermined behavior includes a first behavior defined in relation to the luggage included in the image; the first behavior includes being within a first range from the luggage and maintaining a first posture for a first time or more; the first range from the luggage is a range that touches the luggage, The first posture is at least one of a squatting posture and a crouching posture. A monitoring device is provided.

[0013] According to one aspect of the present invention, The computer Based on images taken at the inspection site, it detects people who have performed predetermined actions, outputting detection information relating to the detected person. fruit, the predetermined behavior includes a first behavior defined in relation to the luggage included in the image; the first behavior includes being within a first range from the luggage and maintaining a first posture for a first time or more; the first range from the luggage is a range that touches the luggage, The first posture is at least one of a squatting posture and a crouching posture. A monitoring method is provided.

[0014] According to one aspect of the present invention, On the computer, Based on images taken at the inspection site, it detects people who have performed predetermined actions, outputting detection information relating to the detected person. 、 the predetermined behavior includes a first behavior defined in relation to the luggage included in the image; the first behavior includes being within a first range from the luggage and maintaining a first posture for a first time or more; the first range from the luggage is a range that touches the luggage, The first posture is at least one of a squatting posture and a crouching posture. Programs are offered. [Effects of the Invention]

[0015] According to the present invention, it is possible to effectively support surveillance at inspection sites. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram showing an overview of a monitoring system according to a first embodiment of the present invention. [Figure 2] 4 is a flowchart showing an outline of a monitoring process according to the first embodiment of the present invention. [Figure 3] 1 is a diagram showing an example of the configuration of a monitoring system according to a first embodiment, together with a diagram showing an example of an inspection site as viewed from above. [Figure 4] FIG. 2 is a diagram illustrating an example of the functional configuration of a detection unit according to the first embodiment. [Figure 5] 1 is a diagram illustrating an example of the physical configuration of a monitoring device according to a first embodiment of the present invention. [Figure 6] 6 is a flowchart showing a detailed example of a detection process according to the first embodiment. [Figure 7] 10 is a flowchart showing a detailed example of an output process according to the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a configuration of detection information. [Figure 9] FIG. 10 is a diagram illustrating an example of detection information displayed on a display unit. [Figure 10] FIG. 10 is a diagram illustrating an example of the configuration of a monitoring system according to a second modification. [Figure 11] FIG. 10 is a diagram illustrating an example of the configuration of a monitoring system according to a second embodiment of the present invention. [Figure 12] FIG. 10 is a diagram illustrating an example of the functional configuration of a detection unit according to a second embodiment. [Figure 13] 10 is a flowchart illustrating an example of a detection process according to the second embodiment. [Figure 14]FIG. 10 is a diagram illustrating an example of the configuration of a monitoring system according to a third embodiment of the present invention. [Figure 15] FIG. 10 is a diagram illustrating an example of the functional configuration of a detection unit according to a third embodiment. [Figure 16] 11 is a flowchart illustrating an example of a detection process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0018] <Embodiment 1> (overview) FIG. 1 is a diagram showing an overview of a monitoring system 100 according to a first embodiment of the present invention. The monitoring system 100 includes an imaging device 101 and a monitoring device 102 .

[0019] The imaging device 101 generates image information including an image of the inspection site in response to capturing an image of the inspection site.

[0020] The monitoring device 102 includes a detection unit 103 and an output unit 104 . The detection unit 103 detects a person who has performed a predetermined action based on an image of the inspection site. The output unit 104 outputs detection information relating to the detected person.

[0021] According to the monitoring system 100, it is possible to effectively support monitoring at an inspection site. According to the monitoring method, it is possible to effectively support monitoring at an inspection site.

[0022] FIG. 2 is a flowchart showing an outline of the monitoring process according to the first embodiment of the present invention. The detection unit 103 detects a person who has performed a predetermined action based on an image captured at the inspection site (step S101). The output unit 104 outputs detection information related to the detected person (step S102).

[0023] This monitoring process makes it possible to effectively support monitoring at the inspection site.

[0024] A detailed example of the monitoring system according to the first embodiment will be described below. (detail) 3 is a diagram showing an example of the configuration of the monitoring system 100 according to the first embodiment of the present invention, together with a diagram showing an example of an inspection site R viewed from above. The monitoring system 100 is a system for monitoring the inspection site R.

[0025] Inspection area R is, for example, a customs inspection area. Figure 3 shows three people Pa, Pb, and Pc standing around an entry desk T set up at inspection area R. Near people Pa, Pb, and Pc are luggage La, Lb, and Lc, respectively. Near person Pa is a smartphone SP.

[0026] Hereinafter, when there is no particular distinction between persons Pa, Pb, and Pc, any one of persons Pa, Pb, and Pc will also be referred to as "person P." In other words, "person P" is a person present at inspection area R. When there is no particular distinction between luggage La, Lb, and Lc, any one of luggage La, Lb, and Lc will also be referred to as "baggage L." In other words, "baggage L" is luggage present at inspection area R.

[0027] As described above, the monitoring system 100 includes the image capturing device 101 and the monitoring device 102.

[0028] The image capturing device 101 and the monitoring device 102 are connected to each other via a network N. The network N is a communication network configured using wired or wireless connections or a combination of these. Therefore, the image capturing device 101 and the monitoring device 102 can transmit and receive information, data, etc. to and from each other via the network N.

[0029] The photographing device 101 is a device for photographing the inspection site R. The photographing device 101 is a camera or the like. The photographing device 101 photographs the inspection site R. In response to photographing the inspection site R, the photographing device 101 generates image information including an image of the inspection site R. The photographing device 101 transmits the image information to the monitoring device 102 via the network N.

[0030] The image capturing device 101 may capture images continuously. In this case, the image capturing device 101 may transmit image information including a video (moving image) made up of a plurality of images to the monitoring device 102 in real time.

[0031] It is desirable that the image capturing device 101 captures an image of the entire inspection site R. In order to capture an image of the entire inspection site R, the surveillance system 100 may be provided with a plurality of image capturing devices 101.

[0032] (Functional configuration of monitoring device 102) The monitoring device 102 is a device for monitoring the inspection site R. In detail, the monitoring device 102 includes a detection unit 103, an output unit 104, a display unit 105, an image storage unit 106, and a detection storage unit 107.

[0033] The detection unit 103 detects a person P who has performed a predetermined action based on an image of the inspection site R included in the image information. Here, "action" includes not only movement but also standing still.

[0034] In detail, the detection unit 103 includes an image acquisition unit 108, an image analysis unit 109, and an analysis control unit 110, as shown in an example of the functional configuration of FIG.

[0035] The image acquisition unit 108 acquires image information from the image capturing device 101 via the network N. The image acquisition unit 108 can acquire image information including video (moving images) from the image capturing device 101 in real time.

[0036] The image analysis unit 109 analyzes the image of the inspection site R included in the image information.

[0037] In detail, the image analysis unit 109 has one or more analysis functions that perform processing for analyzing an image (analysis processing). The analysis processing performed by the analysis functions provided in the image analysis unit 109 includes one or more of (1) object detection processing, (2) face analysis processing, (3) human figure analysis processing, (4) posture analysis processing, (5) behavior analysis processing, (6) appearance attribute analysis processing, (7) gradient feature analysis processing, (8) color feature analysis processing, and (9) movement line analysis processing.

[0038] (1) Object detection processing can detect objects from an image. Object detection processing can also determine the position of an object within an image. An example of a model that can be applied to object detection processing is YOLO (You Only Look Once). Object detection processing can detect, for example, a person P, luggage L, a smartphone SP, etc.

[0039] Here, "object" includes people and things, and the same applies hereinafter.

[0040] (2) Facial analysis processing can detect human faces from images. Facial analysis processing can extract the features of detected faces (facial feature amounts) and classify (classify) detected faces. Facial detection processing can also determine the position of faces within an image. Facial detection processing can also determine the identity of people detected from different images based on the similarity between the facial feature amounts of people detected from different images.

[0041] (3) Humanoid analysis processing can extract the physical features of people in an image (for example, values ​​that indicate overall characteristics such as whether they are fat or thin, height, and clothing), and classify (category) people in an image. Humanoid feature detection processing can also identify the position of a person in an image. Humanoid feature detection processing can also determine the identity of people detected from different images based on the physical features of people in different images.

[0042] (4) In posture analysis processing, for example, joint points of a person are detected from an image, and a stick figure model is constructed by connecting the joint points. Then, in posture analysis processing, information on the stick figure model is used to estimate the posture of the person, extract feature values ​​(posture feature values) of the estimated posture, and classify (classify) the people included in the image. For example, in posture analysis processing, the identity of people detected from different images can be determined based on the posture feature values ​​of people included in different images. In posture analysis processing, for example, postures such as a crouching posture, a squatting posture, a standing posture, and a talking posture are estimated from the image, and posture feature values ​​are extracted. A talking posture is a posture when talking using a calling device such as a smartphone SP, and the same applies hereinafter.

[0043] For pose analysis processing, for example, the technology disclosed in Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields," The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 7291-7299 can be applied.

[0044] (5) In the behavior analysis process, information on the stick figure model, changes in posture, etc. are used to estimate human movements, extract the feature values ​​of human movements (movement feature values), and classify (classify) people in the image. In the behavior detection process, information on the stick figure model can also be used to estimate a person's height and identify the person's position in the image.

[0045] (6) Appearance attribute analysis processing can recognize appearance attributes associated with people. In appearance attribute analysis processing, it is possible to extract features (appearance attribute features) related to the recognized appearance attributes and classify (classify) people included in images. Appearance attributes are attributes of appearance, and include, for example, one or more of clothing color, shoe color, hairstyle, and whether or not a hat, tie, or glasses is worn.

[0046] (7) Gradient feature analysis processing can determine the gradient feature of an image. For example, technologies such as SIFT, SURF, RIFF, ORB, BRISK, CARD, and HOG can be applied to the gradient feature detection processing.

[0047] (8) Color feature analysis processing can detect objects from an image. Color feature analysis processing can extract color features (color features) of detected objects and classify (classify) the detected objects. Color features are, for example, color histograms. For example, when color feature analysis processing detects a person P, luggage L, and smartphone SP contained in an image, it can classify them into classes of person P, luggage L, and smartphone SP.

[0048] (9) In the flow line analysis process, for example, the result of determining the identity of a person in any of the above analysis processes (2) to (6) can be used to determine the flow line (trajectory of movement) of a person. In more detail, for example, by connecting people who are determined to be the same in different images in a time series, the flow line of that person can be determined. In the flow line analysis process, for example, it is also possible to determine the flow line across multiple images taken of different areas, such as images taken of adjacent areas.

[0049] In each of the analysis processes (1) to (9), the results of other analysis processes may be used as appropriate.

[0050] The analysis control unit 110 controls the image analysis unit 109 and acquires the results of the analysis by the image analysis unit 109. Then, the analysis control unit 110 detects a person P who has performed a predetermined behavior based on the results of the analysis by the image analysis unit 109. The predetermined behavior includes one or more of the following first to fourth behaviors.

[0051] The first behavior is a behavior determined in relation to an object (e.g., baggage L, smartphone SP) included in an image captured of the inspection site R. For example, the analysis control unit 110 detects a person P performing the first behavior using an object detection process, a behavior detection process, or the like.

[0052] The second behavior is a behavior determined in relation to the line of sight of person P included in an image captured of the inspection site R. For example, the analysis control unit 110 detects person P performing the second behavior using face detection processing, behavior detection processing (e.g., shoulder direction), etc. In order to detect the second behavior, the face detection processing may detect the positions and movements of the iris and pupil (so-called black part of the eye) of the face, and obtain the feature amounts of these.

[0053] The second action may be an action defined in relation to the line of sight toward one or both of a predetermined person and object, such as an employee at the inspection site R or a search dog. In this case, the second action may further use one or more of object detection processing, humanoid feature detection processing, appearance attribute detection processing, gradient feature detection processing, color feature detection processing, etc. to detect the person P performing the second action.

[0054] The third behavior is a behavior determined regarding the movement of person P included in an image captured of inspection site R. Here, "movement" includes not only movement accompanied by a change in position but also staying in a fixed place. For example, the analysis control unit 110 detects person P performing the third behavior using a flow line analysis process or the like.

[0055] The third behavior may be a behavior defined regarding movement toward one or both of a predetermined person and object, such as an officer at the inspection site R, a search dog, or a writing desk T. In this case, the analysis control unit 110 may further use one or more of object detection processing, humanoid feature detection processing, appearance attribute detection processing, gradient feature detection processing, color feature detection processing, etc. to detect one or both of the predetermined person and object, thereby detecting a person P performing the third behavior.

[0056] The fourth behavior is a behavior defined regarding the entry and exit of a predetermined area (e.g., inspection site R, restroom) of a person included in an image captured of inspection site R. For example, the analysis control unit 110 detects person P performing the fourth behavior using a flow line analysis process or the like.

[0057] To identify the entrance / exit of the predetermined area, the analysis control unit 110 may further use, for example, object detection processing, gradient feature detection processing, or color feature detection processing to detect the person P performing the fourth behavior. Alternatively, the analysis control unit 110 may store in advance information indicating the location of an entrance / exit of a toilet or the like, and further use this information to detect the person P performing the fourth behavior.

[0058] The first to fourth actions will be described in detail later.

[0059] Furthermore, the analysis control unit 110 may perform processing to track the detected person P. In this case, for example, the analysis control unit 110 may acquire the flow line of the detected person P using a flow line analysis process or the like.

[0060] The output unit 104 outputs the detection information (see FIG. 8) relating to the person P detected by the analysis control unit 110.

[0061] The detection information includes, for example, person identification information for identifying the detected person P.

[0062] The person identification information may be information indicating the position (e.g., current position) of the detected person P. In this case, the person identification information may further include at least one of an image of the inspection site R, a map of the inspection site R, etc., and may be information in which a mark indicating the detected person P is attached to at least one of the image, map, etc. of the inspection site R. The image of the inspection site R included in the person identification information is preferably an image taken of the inspection site R, and more preferably an image in which the marked person P is detected.

[0063] The person identification information may include the results of tracking the detected person P. In this case, the person identification information may further include an image of the inspection site R, a map of the inspection site R, etc., and may be information in which the movement path of the detected person P is attached to the image, map, etc. of the inspection site R. The image of the inspection site R included in the person identification information is preferably an image taken of the inspection site R, more preferably an image in which person P with the movement path attached is detected, and even more preferably the latest image in which person P with the movement path attached is detected.

[0064] The person identification information may be information indicating the appearance (clothing, hairstyle, etc.) of the detected person P.

[0065] The detection information may include, for example, the reason why the person P identified using the person identification information was detected. The reason for detection is information indicating the content of a predetermined behavior that caused the person P to be detected.

[0066] The display unit 105 displays various types of information. For example, the display unit 105 acquires detection information from the output unit 104 and displays the detection information. The display unit 105 may be a terminal device (not shown) that is connected to the monitoring device 102 via the network N and is held by a staff member (particularly, a security guard) at the inspection site R.

[0067] The image storage unit 106 is a storage unit for storing images acquired by the detection unit 103 (image acquisition unit 108).

[0068] The detection storage unit 107 is a storage unit for storing detection information.

[0069] Up to now, the functional configuration of the monitoring system 100 according to the first embodiment has been mainly explained. From here, the physical configuration of the monitoring system 100 according to this embodiment will be explained.

[0070] <<Physical Configuration of Monitoring System 100>> The monitoring system 100 is physically composed of an image capturing device 101 and a monitoring device 102 connected via a network N. The image capturing device 101 and the monitoring device 102 are each composed of a single, physically separate device.

[0071] The monitoring device 102 may be physically composed of multiple devices connected via an appropriate communication line such as a network N. The image capturing device 101 and the monitoring device 102 may be physically composed of a single device. When the monitoring system 100 includes multiple image capturing devices 101, for example, one or more of the image capturing devices 101 may include at least a part of the monitoring device 102.

[0072] The monitoring device 102 is physically, for example, a general-purpose computer.

[0073] In detail, for example, the monitoring device 102 physically includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, a network interface 1050, an input interface 1060, and an output interface 1070, as shown in FIG.

[0074] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, network interface 1050, input interface 1060, and output interface 1070. However, the method of connecting the processor 1020 and other components to each other is not limited to bus connection.

[0075] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0076] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0077] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules for realizing the functions of the monitoring device 102. The processor 1020 reads each of these program modules into the memory 1030 and executes them, thereby realizing the function corresponding to that program module.

[0078] The network interface 1050 is an interface for connecting the monitoring device 102 to the network N.

[0079] The input interface 1060 is an interface such as a touch panel, keyboard, or mouse that allows the user to input information.

[0080] The output interface 1070 is a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like, which serves as an interface for presenting information to the user. The output interface 1070 constitutes the display unit 105. The output interface 1070 may be built into the monitoring device 102 or may be provided externally to the monitoring device 102.

[0081] Up to this point, the explanation has been mainly given of the physical configuration of the monitoring system 100 according to embodiment 1. From here, the operation of the monitoring system 100 according to this embodiment will be explained.

[0082] (Operation of monitoring system 100) The monitoring device 102 executes a monitoring process (see FIG. 2). The monitoring process is a process for monitoring the inspection site R. The monitoring device 102 starts the monitoring process when, for example, a start instruction is received from a user. The monitoring device 102 ends the monitoring process when, for example, a stop instruction is received from a user.

[0083] The detection process (step S101) and the output process (step S102) will be described in detail below, using an example in which all of the first to fourth behaviors are detected.

[0084] FIG. 6 is a flowchart showing a detailed example of the detection process (step S101) according to this embodiment.

[0085] The image acquisition unit acquires image information from the photographing device 101 (step S101a). The image acquisition unit stores the image information in the image storage unit .

[0086] In order to detect the person P who has performed the first behavior, the analysis control unit 110 causes the image analysis unit 109 to analyze the image included in the image information acquired in step S101a. As a result, the analysis control unit 110 detects the person P who has performed the first behavior defined in relation to an object included in the image, based on the image of the inspection site R (step S101b).

[0087] The object defined by the first action is, for example, luggage L or a smartphone SP. The smartphone SP is an example of a communication device that is a device used for making calls. The communication device is not limited to the smartphone SP, but may be a mobile phone, a headset, etc. A headset is a device equipped with headphones, earphones, etc. and a microphone.

[0088] (Example of the first action) For example, the first action may include being within a first range from the load L and in a first position.

[0089] The first range is a range that is determined in advance as appropriate. The first range may be, for example, within a predetermined distance from the baggage L, or within a range that can be touched by the baggage L. The first range may be, for example, within a predetermined distance from a specific part of person P, such as the hand or waist, or within a range that can be touched by the specific part. These distances may be defined as distances in real space (for example, 10 centimeters) or as distances in an image (for example, 20 pixels). Furthermore, the first range is not limited to a spherical shape and may be defined as an appropriate shape.

[0090] The first posture is a posture that is appropriately determined in advance, and is, for example, at least one of a crouching posture, a squatting posture, etc. The crouching posture can be detected based on detecting that the waist and legs are bent. The squatting posture can be detected based on detecting that the legs are bent and the waist is lower than in an upright position.

[0091] The first behavior may be a first time or more that continues within a first range from the baggage L and in a first posture. The first time is a time that is appropriately determined in advance, such as three minutes.

[0092] In general, it can be difficult to detect whether baggage L is being opened or closed from an image at inspection area R due to factors such as the fact that inspection area R is often crowded with aircraft taking off and landing, and there are many people P carrying baggage L. Also, there is a risk that it may be overlooked through manual monitoring alone. By detecting person P who has performed the first example of behavior described here, it is possible to assist in the detection of person P who is opening or closing baggage L at inspection area R.

[0093] (Another example of the first action) Furthermore, for example, the first behavior may include being in a posture (call posture) in which a call is made using a smartphone SP or the like. The call posture may be detected, for example, based on detecting the posture of the arm holding a call device such as a smartphone SP close to the ear or mouth.

[0094] The first behavior may include maintaining a talking posture for a second period of time or more, which is a predetermined period of time, such as one minute.

[0095] The first behavior may further include holding the smartphone SP within a second range from the face. The second range is a range that is determined in advance as appropriate. The second range may be, for example, within a predetermined distance from the face, a range that can touch the face, or the like. This distance may be defined as a distance in real space (e.g., several centimeters) or as a distance in the image (e.g., 10 pixels). Furthermore, the first range is not limited to a spherical shape and may be defined as an appropriate shape.

[0096] Furthermore, for example, the first behavior may include talking in a conversation posture. Talking may be detected, for example, by detecting mouth movement using face detection processing. In this case, the first behavior may be talking while there is no person P in the vicinity (i.e., within a predetermined range from the person P). Furthermore, the first behavior may include talking for a predetermined period of time or longer.

[0097] In general, it can be difficult to detect person P making a phone call at inspection site R, for reasons such as the fact that inspection site R is often crowded with aircraft taking off and landing. Also, there is a risk that the person may be overlooked through manual monitoring alone. By detecting person P who has performed another example of the first behavior described here, it is possible to assist in the detection of person P making a phone call at inspection site R.

[0098] In order to detect person P who has performed the second behavior, the analysis control unit 110 causes the image analysis unit 109 to analyze the image included in the image information acquired in step S101a. As a result, based on the image of the inspection site R, the analysis control unit 110 detects person P who has performed the second behavior determined in relation to the line of sight of the person included in the image (step S101c).

[0099] (Example of the second action) For example, the second behavior may include changing the direction of gaze continuously for a third period of time or more, or intermittently at a first frequency or more. The third period of time is a predetermined period of time, such as 15 seconds. The first frequency is a predetermined period of time, such as twice every 10 seconds. The second behavior may also include intermittently changing the direction of gaze at the first frequency or more for a predetermined period of time or more.

[0100] In general, it can be difficult to detect a restless person P who is constantly moving their eyes at inspection site R, for reasons such as the fact that inspection site R is often crowded with aircraft taking off and landing. There is also a risk that the person P may be overlooked through manual monitoring alone. By detecting person P who has engaged in the second example of behavior described here, it is possible to assist in the detection of a restless person P who is constantly moving their eyes at inspection site R.

[0101] (Another example of the second action) Furthermore, for example, the second behavior may include a behavior related to gaze toward a predetermined target. The second behavior may be directing gaze toward the predetermined target continuously for a fourth period of time or more, or intermittently at a second frequency or more.

[0102] The predetermined target here may be one or both of a person and an object, such as a staff member at inspection site R (particularly, a security guard), a search dog, etc.

[0103] In many cases, officers and investigative dogs wear uniforms and are equipped with predetermined equipment. In such cases, the analysis control unit 110 can detect predetermined targets such as officers and investigative dogs by distinguishing them from other people P and objects using one or more of object detection processing, appearance attribute detection processing, color feature detection processing, etc.

[0104] If a predetermined target such as an employee or a search dog has a device (not shown) that transmits information for identifying its location, the analysis control unit 110 may identify the location of the predetermined target based on information from the device. Then, the analysis control unit 110 may use the location to detect the predetermined target from the image.

[0105] The fourth time period is a predetermined time period, for example, 15 seconds. The second frequency is a predetermined time period, for example, three times every 15 seconds. The second behavior is 2 The gaze direction may be directed toward a predetermined target intermittently at a frequency of at least 100% for a predetermined period of time or longer.

[0106] In general, because inspection site R is often crowded with aircraft taking off and landing, it can be difficult to detect a person P who is looking at a predetermined target and is concerned about that target. Also, there is a risk that the person P may be overlooked through manual monitoring alone. By detecting person P who has performed another example of the second behavior described here, it is possible to assist in the detection of person P who is concerned about a predetermined target.

[0107] In order to detect person P who has performed the third behavior, the analysis control unit 110 causes the image analysis unit 109 to analyze the image included in the image information acquired in step S101a. As a result, based on the image of the inspection site R, the analysis control unit 110 detects person P who has performed the third behavior defined regarding the movement of the person included in the image (step S101d).

[0108] (Example of the third action) For example, the third behavior may include staying in the first area for a fifth time or more. The third behavior may include staying in the first area for a fifth time or more. The third behavior may include staying in the first area in a constant posture for a fifth time or more.

[0109] Staying means stopping at a certain point, moving within a certain area, etc. The first area is, for example, a predetermined range from a reference location, an inspection site R, etc. The reference location is, for example, a location where the person P was at a certain time, or a location where a writing desk T is installed. The fifth time is a time that is determined appropriately in advance, and may be determined appropriately depending on the first area. The certain posture is at least one of a crouching posture, a squatting posture, etc.

[0110] In general, because inspection area R is often crowded with aircraft taking off and landing, it can be difficult to detect a person P who stays in the first area for a long time, such as the inspection area R, a certain location within the inspection area R, or around the writing table T. It can also be difficult to detect a person P who maintains a constant posture while staying there. There is a risk that the person P may be overlooked through manual monitoring alone. Detecting a person P who has performed the third example of behavior described here can assist in the detection of a person P who stays in the first area for a long time or a person P who maintains a constant posture while staying there.

[0111] (Another example of the third action) Furthermore, for example, the third behavior may include a behavior in which a person included in an image taken of the inspection site R avoids a predetermined target. As described above, the predetermined target may be one or both of a person and an object, such as a staff member at the inspection site R (particularly, it may be a security guard), a search dog, etc.

[0112] The avoidance behavior is, for example, a behavior of reversing the movement direction within a third range from a predetermined target. The third range is a range that is determined appropriately in advance. The third range is, for example, a range within a predetermined distance. This distance may be defined as a distance in real space (for example, 1 m to 2 m) or as a distance in an image (for example, 100 to 200 pixels). 3 The range is not limited to a sphere and may be defined as an appropriate shape. By detecting the person P who has taken such avoidance action, it is possible to detect the person P who has entered the field of view of a predetermined target and is retreating.

[0113] Furthermore, for example, the avoidance behavior is a behavior in which, when the predicted position based on the moving direction of the person P falls within a fourth range from a predetermined target, the person P changes the moving direction so as to move out of the fourth range. The fourth range is a range that is determined appropriately in advance. This distance may be defined as a distance in real space (e.g., 1 m to 2 m) or as a distance in the image (e.g., 100 to 200 pixels). Furthermore, the first range is not limited to a spherical shape and may be determined as an appropriate shape. By detecting the person P who has performed such an avoidance behavior, it is possible to detect the person P who moves so as not to approach the predetermined target.

[0114] In general, because inspection site R is often crowded with aircraft taking off and landing, it can be difficult to detect person P who is avoiding predetermined targets, such as inspection site R staff and search dogs. Also, there is a risk that such a person may be overlooked through manual surveillance alone. By detecting person P who has engaged in another example of the third behavior described here, it is possible to assist in the detection of person P who is avoiding predetermined targets.

[0115] In order to detect person P who has performed the fourth behavior, the analysis control unit 110 causes the image analysis unit 109 to analyze the image included in the image information acquired in step S101a. As a result, based on the image of the inspection site R, the analysis control unit 110 detects person P who has performed the fourth behavior defined regarding the person included in the image entering or leaving a predetermined area (step S101e).

[0116] The predetermined area in the fourth behavior may be an area that is appropriately determined in advance, for example, a toilet in the inspection site R. The predetermined area may also be the inspection site R or the like.

[0117] (Example of the fourth action) For example, the fourth behavior may include a person included in an image taken of the inspection site R not leaving a predetermined area for a sixth time or more after entering the predetermined area. The sixth time is a time that is appropriately determined in advance, for example, 3 to 5 minutes.

[0118] In general, because inspection site R is often crowded with aircraft taking off and landing, it can be difficult to detect person P who stays in a predetermined area, such as a restroom, within inspection site R for a long time. Also, there is a risk that the person P may be overlooked through manual monitoring alone. By detecting person P who has performed the fourth example of behavior described here, it is possible to assist in the detection of person P who stays in a predetermined area for a long time.

[0119] (Another example of the fourth action) Furthermore, for example, the fourth behavior may include a behavior in which a person included in an image captured of the inspection site R changes their appearance in a predetermined area. The appearance may be, for example, the clothing, belongings, etc. of the person P.

[0120] In this case, the analysis control unit 110 may detect, for example, a person P whose appearance changes between when he enters a predetermined area and when he leaves the predetermined area, as a person who has performed an action that changes his appearance.

[0121] In detail, the analysis control unit 110 holds information on the facial feature amounts and appearance of the person P when he / she enters a predetermined area (for example, appearance attribute feature amounts, humanoid feature amounts, detected object). When a person P with matching facial feature amounts emerges from the predetermined area, the analysis control unit 110 compares the appearance of the person P contained in the held information with the appearance of the person P emerging from the predetermined area.

[0122] If the comparison shows that the appearances do not match, the analysis control unit 110 may determine that the appearance of the person P has changed, and detect the person P as a person P who has taken an action that changes their appearance.If the comparison shows that the appearances match, the analysis control unit 110 may determine that the appearance of the person P has not changed, and may not detect the person P as a person P who has taken an action that changes their appearance.

[0123] Here, "match" means to substantially match. In other words, "match" does not only mean to be completely identical, but also includes cases where the two differ within a predetermined range.

[0124] More specifically, for example, when detecting a change in clothing in a restroom, the analysis control unit 110 holds information about the facial feature amounts and clothing of person P when he / she enters the restroom (for example, appearance attribute feature amounts, human figure feature amounts). The analysis control unit 110 acquires information about the facial feature amounts and clothing of person P when he / she leaves the restroom.

[0125] The analysis control unit 110 compares the facial features of person P entering the toilet with person P leaving the toilet. Based on the comparison result, the analysis control unit 110 determines the identity of person P. Then, for person P determined to be the same, if the information about the person's clothing when entering the toilet and when leaving the toilet does not match, the analysis control unit 110 detects that the person's clothing has changed in the toilet.

[0126] Furthermore, for example, when detecting a change in baggage L in a restroom, the analysis control unit 110 holds the facial feature amounts of person P when they enter the restroom and information about the baggage L (for example, the detection results of the object detection process, appearance attribute feature amounts, and human figure feature amounts). The analysis control unit 110 acquires the facial feature amounts of person P when they leave the restroom and information about the baggage L.

[0127] The analysis control unit 110 compares at least one of the facial feature amounts, appearance attribute feature amounts, and human figure feature amounts between a person P entering the toilet and a person P leaving the toilet. Based on the comparison result, the analysis control unit 110 determines the identity of the person P. Then, for a person P determined to be the same, if the information about the baggage L when entering the toilet and when leaving the toilet does not match, the analysis control unit 110 detects that the baggage L has changed in the toilet.

[0128] A person P whose appearance changes between when he or she enters a predetermined area and when he or she leaves the predetermined area can usually be assumed to have performed an action that changes his or her appearance in the predetermined area. Therefore, by detecting a person P whose appearance changes between when he or she enters a predetermined area and when he or she leaves the predetermined area, it is possible to detect a person P who has performed an action that changes his or her appearance in the predetermined area.

[0129] Furthermore, for example, the analysis control unit 110 may retain entry information regarding a person P who has entered a predetermined area. Then, the analysis control unit 110 may detect, among the people P who leave the predetermined area, a person P who is not included in the entry information, as a person P who has performed an action that changes their appearance.

[0130] Of the persons P coming out of a predetermined area, those not included in the entrance information can be estimated to be persons P whose appearance has changed to such an extent that they cannot be determined to be the same person from the image when they enter and exit the predetermined area. Therefore, by detecting the persons P not included in the entrance information among the persons P coming out of a predetermined area, it is possible to detect persons P who have performed an action that changes their appearance in the predetermined area.

[0131] In general, it can be difficult to detect a person P whose appearance has changed in a predetermined area at an inspection site R, for reasons such as the fact that the site is often crowded with aircraft taking off and landing. There is also a risk that the person P may be overlooked through manual surveillance alone. In particular, if the person P's appearance has changed to the extent that it is no longer possible to determine that the person P is the same person from an image, there is a high possibility that the person P will be overlooked. By detecting a person P who has performed another example of the fourth behavior described here, it is possible to assist in the detection of a person P who has performed a behavior that changes their appearance in a predetermined area.

[0132] The analysis control unit 110 performs processing to track the person P detected in at least one of steps S101b to S101e (step S101f).

[0133] In detail, the analysis control unit 110 assigns a person ID to each person P included in the image in order to identify the person P. For the person P detected in at least one of steps S101b to S101e, the analysis control unit 110 holds tracking information for tracking the person P. The tracking information includes, for example, the person ID of the detected person P and at least one of image features (for example, facial features) used for tracking.

[0134] Then, based on the tracking information, the analysis control unit 110 determines whether the detected person P is included in the image included in the image information acquired in step S101a. If the analysis control unit 110 determines that the detected person P is included, the analysis control unit 110 causes the image analysis unit 109 to create a movement line of the detected person P using a movement line analysis process, for example, and stores the movement line in association with the person ID in the tracking information. This allows the analysis control unit 110 to track the detected person P using the movement line after being detected in at least one of steps S101b to S101e, even after any of the first to fourth actions has ended.

[0135] After executing step S101f, the analysis control unit 110 ends the detection process (step S101) and returns to the monitoring process (see FIG. 2).

[0136] FIG. 7 is a flowchart showing a detailed example of the output process (step S102) according to this embodiment.

[0137] The output unit 104 generates detection information 111 about the person P detected in at least one of steps S101b to S101e based on the results of steps S101b to S101f executed by the analysis control unit 110 (step S102a). The output unit 104 stores the detection information in the detection storage unit 107.

[0138] FIG. 8 is a diagram showing an example of the configuration of the detection information 111. The detection information 111 is information that associates person identification information for identifying the detected person P with the reason why the person P was detected. The person identification information includes the person ID of the person P, a map and appearance of the inspection site R, and the time. The map includes a mark indicating the location of the person P at the associated time. The map also includes the movement path of the person P from when the person P was detected to the associated time. The map includes the location where the person P was detected.

[0139] The output unit 104 outputs the detection information 111 generated in step S102a (step S102b). In this embodiment, the output unit 104 outputs the detection information 111 to the display unit 105, and causes the display unit 105 to display the detection information 111.

[0140] 9 is a diagram showing an example of the detection information 111 displayed on the display unit 105. In the example shown in FIG. 3, person Pa at position X1 was detected because he was in a talking position using smartphone SP, and then moved. The figure shows an example in which the detection information 111 includes a plan view of the inspection site R (a map of the inspection site R viewed from above), and a mark indicating the position of person P, his / her movement, and the reason for detection are superimposed on the plan view.

[0141] The flow line indicates the trajectory of the movement of person Pa. Fig. 9 shows an example in which a dotted square is added around person Pa as a mark indicating the position (for example, current location) of person Pa. Fig. 9 also shows an example in which "phone call" as the reason for detecting person Pa is associated with an image of person Pa (in the example of Fig. 9, a mark indicating the position of person Pa).

[0142] Since the detection information 111 includes a mark indicating the position of the person Pa, the user can easily grasp the position of the person Pa using the display unit 105. Furthermore, by grasping the position of the person Pa, it becomes easy to visually track the person Pa. Therefore, it becomes possible to effectively support surveillance at the inspection site R.

[0143] Since the detection information 111 includes the flow line, the user can easily understand the flow line of the person Pa using the display unit 105. Therefore, it is possible to effectively support the surveillance at the inspection site R.

[0144] Since the detection information 111 includes the reason for detection, the user can easily understand the reason why the person Pa was detected using the display unit 105. Therefore, it is possible to effectively support surveillance at the inspection site R.

[0145] Since the detection information 111 includes images of person P before and after a change in appearance, the user can view the images before and after the change in appearance using the display unit 105. This makes it easier for the user to understand the facial features of person P and to find person P at the inspection site R. Therefore, it is possible to effectively support surveillance at the inspection site R. Here, the images before and after a change in appearance of person P may be either moving images or still images including person P.

[0146] Note that the mark indicating the position of person P is not limited to this, and the shape of the mark may be changed as appropriate. Also, the output unit 104 may use, as a mark, a blinking area of ​​person P in the image. Also, the display unit may always display the reason for detection, or may display it temporarily.

[0147] An example of temporary display is displaying for a predetermined time when the user places the cursor over the area where person Pa is displayed in response to a user operation, or when the user places the cursor and clicks. Another example of temporary display is displaying while the user places the cursor over the area where person Pa is displayed in response to a user operation. Yet another example of temporary display is displaying for a predetermined time after tapping on the area when a touch panel is provided on the display unit 105.

[0148] After executing step S102b, the output unit 104 ends the output process (step S102) and returns to the monitoring process (see FIG. 2). Then, the detection unit 103 executes step S101 again.

[0149] So far, the first embodiment of the present invention has been described.

[0150] According to this embodiment, the monitoring device 102 includes a detection unit 103 and an output unit 104. The detection unit 103 detects a person who has performed a predetermined action based on an image captured of the inspection site R. The output unit 104 outputs detection information related to the detected person.

[0151] In this way, the monitoring device 102 processes the image captured at the inspection site R to detect the person P who has performed a predetermined action, and can therefore quickly and appropriately detect the person P who requires attention. This makes it possible to effectively support the monitoring of the inspection site R.

[0152] According to this embodiment, the predetermined actions include the first to fourth actions described above. By including each of the first to fourth actions in the predetermined actions, it is possible to detect person P by processing an image of the inspection site R based on actions generally performed by people who require caution, and it is possible to quickly and appropriately detect person P who requires caution. Therefore, by including each of the first to fourth actions in the predetermined actions, it is possible to effectively support surveillance at the inspection site R. By including multiple of the first to fourth actions in the predetermined actions, it is possible to more effectively support surveillance at the inspection site R. Furthermore, by including all of the first to fourth actions in the predetermined actions as in this embodiment, it is possible to even more effectively support surveillance at the inspection site R.

[0153] (Variation 1) The information included in the detection information 111 is not limited to the information described in embodiment 1. The detection information 111 may include the time of detection in addition to the information exemplified in Fig. 8, or may include some of these.

[0154] Furthermore, the detection information 111 may include images (for example, a face image or a whole-body image) of the person P in whom the fourth behavior has been detected when the person P enters a predetermined area and when the person P leaves the predetermined area. In other words, the detection information 111 may include images of the person P before and after the appearance of the person P changes when the person P performs a behavior that changes the appearance.

[0155] When the detection information 111 includes images of person P before and after a change in appearance, it is desirable that the images are images of the front of person P. To that end, the surveillance system 100 may include multiple image capturing devices 101 installed at the entrances and exits of a predetermined area so that person P entering and exiting the area can be photographed from the front. Furthermore, when multiple entrances and exits are provided in the predetermined area, the surveillance system 100 may include one or more image capturing devices 101 for photographing each of the entrances and exits.

[0156] Since the detection information 111 includes images of the person P before and after the appearance change, the user can view the images before and after the appearance change using the display unit 105. This makes it easier for the user to understand the facial features of the person P. Therefore, it is possible to effectively support surveillance at the inspection site R.

[0157] (Variation 2) Fig. 10 is a diagram showing an example of the configuration of a monitoring system 100 according to Modification 2. The monitoring system 100 according to this modification includes a plurality of image capturing devices 101 similar to the image capturing device 101 according to Embodiment 1. Except for this point, the monitoring system 100 according to this modification may be configured similarly to the monitoring system 100 according to Embodiment 1. That is, Fig. 10 illustrates an example in which the monitoring system 100 includes a plurality of image capturing devices 101.

[0158] Even when the monitoring system 100 includes a plurality of image capturing devices 101, the monitoring device 102 may have the same functions and execute the same processes as in the first embodiment. This modification also achieves the same effects as the first embodiment.

[0159] <Embodiment 2> In the first embodiment, the analysis control unit 110 determines (verifies) whether all persons P indicated by person IDs included in the tracking information are included in the images included in the image information. Therefore, as the number of person IDs included in the tracking information increases, the processing load on the analysis control unit 110 for tracking persons P increases.

[0160] Here, normally, person P who leaves inspection site R does not return to inspection site R. Therefore, there is little need to execute the process for tracking person P who leaves inspection site R (step S101f). In embodiment 2, an example will be described in which the analysis control unit 110 is not made to execute the process for tracking person P who leaves inspection site R (step S101f). In order to simplify the explanation, in this embodiment, differences from embodiment 1 will be mainly explained, and explanations that overlap with embodiment 1 will be omitted as appropriate.

[0161] FIG. 11 is a diagram showing an example of the configuration of a monitoring system 200 according to the second embodiment of the present invention.

[0162] Book The monitoring system 200 according to the embodiment includes a monitoring device 202 that replaces the monitoring device 102 according to the first embodiment. Except for this point, the monitoring system 200 according to the embodiment may be configured similarly to the monitoring system 100 according to the first embodiment.

[0163] The monitoring device 202 includes a detection unit 203 that replaces the detection unit 103 according to the first embodiment. Except for this point, the monitoring device 202 according to this embodiment may be configured similarly to the monitoring device 102 according to the first embodiment.

[0164] 12 shows an example of the functional configuration of the detection unit 203, which includes an analysis control unit 210 instead of the analysis control unit 110 of the first embodiment. Except for this point, the detection unit 203 of this embodiment may be configured similarly to the detection unit 103 of the first embodiment.

[0165] The analysis control unit 210 uses the image analysis unit 109 to detect that the detected person P has left the inspection site R. When it is detected that the detected person P has left the inspection site R, the analysis control unit 210 erases information (tracking information) for tracking the person P who has left the inspection site R. Except for this point, the analysis control unit 210 according to this embodiment may be configured similarly to the analysis control unit 110 according to the first embodiment.

[0166] The monitoring system 200 according to this embodiment may be physically configured in the same manner as the monitoring system 100 according to the first embodiment.

[0167] Fig. 13 is a flowchart showing an example of the detection process according to this embodiment. Fig. 13 shows only the parts of the detection process according to this embodiment that are different from the detection process (step S101) according to the first embodiment. As shown in the figure, the monitoring process according to this embodiment includes steps S201 and S202 between steps S101e and S101f. Except for this point, the monitoring process according to this embodiment may be similar to the monitoring process according to the first embodiment.

[0168] After executing step S101e, the analysis control unit 210 determines whether the detected person P has left the inspection site R based on the tracking information it holds (step S201). If it is determined that the person P has not left the inspection site R (step S201; No), the analysis control unit 210 executes step S101f.

[0169] When it is determined that the person P has left the inspection site R (step S201; Yes), the analysis control unit 210 erases the tracking information for tracking the person P who has left the inspection site R (step S202).

[0170] Subsequently, the analysis control unit 210 executes a tracking process (step S101f).

[0171] In the detection process according to this embodiment, in step S202 before the tracking process (step S101f) is executed, the tracking information of the person P who has left the inspection site R can be erased. Therefore, the analysis control unit 110 can be prevented from executing the process for tracking the person P (step S101f).

[0172] So far, the second embodiment of the present invention has been described.

[0173] According to this embodiment, when it is detected that the detected person P has left the inspection site R, the detection unit 203 (analysis control unit 210) erases the information for tracking the person P. As a result, as described above, after the detected person P has left the inspection site R, it is possible to prevent the analysis control unit 110 from substantially executing the process for tracking the person P (step S101f).

[0174] In the detection process according to this embodiment, the analysis control unit 210 holds the tracking information while the person P is in the inspection site R, and can use this tracking information to track the detected person P. On the other hand, after the detected person P leaves the inspection site R, it becomes extremely difficult for the analysis control unit 110 to execute the process for tracking the person P (step S101f). Therefore, the processing load on the analysis control unit 110 can be reduced compared to when tracking information about the detected person P is held even after the detected person P leaves the inspection site R. In other words, it is possible to reduce the processing load for tracking the detected person P while still enabling tracking of the detected person P. Therefore, it is possible to effectively support monitoring at the inspection site R while reducing the processing load on the monitoring device 202.

[0175] <Embodiment 3> In the first embodiment, it has been described as if one or more analysis functions are provided and the analysis processes are always executed. However, the image analysis unit 109 may be provided with multiple analysis functions, and it may be possible to switch which of the analysis processes is used to analyze the image of the inspection site R. In this embodiment, for the sake of brevity, differences from the first embodiment will be mainly described, and explanations that overlap with the first embodiment will be omitted as appropriate.

[0176] FIG. 14 is a diagram showing an example of the configuration of a monitoring system 300 according to the third embodiment of the present invention.

[0177] Book The monitoring system 300 according to the embodiment includes a monitoring device 302 that replaces the monitoring device 102 according to the first embodiment. Except for this point, the monitoring system 300 according to the embodiment may be configured similarly to the monitoring system 100 according to the first embodiment.

[0178] The monitoring device 302 includes a detection unit 303 that replaces the detection unit 103 according to the first embodiment. Except for this point, the monitoring device 302 according to this embodiment may be configured similarly to the monitoring device 102 according to the first embodiment.

[0179] 15 shows an example of the functional configuration of the detection unit 303, which includes an analysis control unit 310 instead of the analysis control unit 110 of the first embodiment. Except for this point, the detection unit 303 of this embodiment may be configured similarly to the detection unit 103 of the first embodiment.

[0180] The analysis control unit 310 switches which of multiple analysis processes for processing images captured at the inspection site R to use to process the images captured at the inspection site R, depending on the situation at the inspection site R.

[0181] In particular, the analysis control unit 310 switches the analysis process used to process the image of the inspection site R depending on whether the situation at the inspection site R satisfies a predetermined standard.

[0182] As mentioned above, the analysis process consists of (1) object detection process, (2) face analysis process, and (3) human Type analysis (4) posture analysis processing, (5) behavior analysis processing, (6) appearance attribute analysis processing, (7) gradient feature analysis processing, (8) color feature analysis processing, and (9) movement line analysis processing.

[0183] The situation of the inspection site R is, for example, the degree of congestion at the inspection site R. The degree of congestion may be the overall degree of congestion at the inspection site R, or the local degree of congestion in a part of the inspection site R. The overall degree of congestion may be, for example, the total number of people P present at the inspection site R. The local degree of congestion may be, for example, the density of an area in the inspection site R where people P are particularly concentrated, or the number of people P present in a predetermined area of ​​the inspection site R (for example, a predetermined range from the writing table T).

[0184] The predetermined criterion is, for example, that the congestion level is less than a predetermined threshold. For example, when the congestion level is equal to or greater than the threshold, the analysis control unit 310 processes the image of the inspection site R using all of the multiple analysis processes. For example, when the congestion level is less than the threshold, the analysis control unit 310 processes the image of the inspection site R using some of the multiple analysis processes.

[0185] In this case, the analysis control unit 310, for example, compares the degree of congestion at the inspection site R with a threshold value TH, and based on the comparison result, controls which of multiple analysis processes for processing images of the inspection site R to use to process the images of the inspection site R.

[0186] The monitoring system 300 according to this embodiment may be physically configured in the same manner as the monitoring system 100 according to the first embodiment.

[0187] The monitoring device 302 according to this embodiment executes a monitoring process including a detection process (step S301) instead of the detection process (S101) according to embodiment 1. Except for this point, the monitoring process according to this embodiment may be similar to the monitoring process according to embodiment 1.

[0188] FIG. 16 is a flowchart showing an example of the detection process (S301) according to this embodiment. FIG. 16 shows only the parts of the detection process according to this embodiment that are different from the detection process (step S101) according to the first embodiment. As shown in the figure, the detection process (S301) according to this embodiment includes steps S301a to S301c before step S101a. Except for this point, the detection process (S301) according to this embodiment may be similar to the detection process (S101) according to the first embodiment.

[0189] The figure shows an example of a flowchart when the status of inspection site R is its congestion level. Also, an example of a flowchart is shown when the predetermined criterion is that the congestion level is less than a predetermined threshold.

[0190] The analysis control unit 310 determines whether the degree of congestion at the inspection site R is smaller than a predetermined threshold value TH (step S301a).

[0191] If it is determined that the congestion level is smaller than the threshold value TH (step S301a; Yes), the analysis control unit 310 controls the image analysis unit 109 to use some of the multiple analysis processes executed by the image analysis unit 109 (step S301b).

[0192] As a result of executing step S301b, the image analysis unit 109 teeth, Some of the multiple analysis processes are used to analyze the image captured of the inspection site R. Here, which of the multiple analysis processes is to be used may be determined as appropriate.

[0193] The plurality of analysis processes executed by the image analysis unit 109 are, in other words, the plurality of analysis functions that the image analysis unit 109 has.

[0194] If it is determined that the congestion level is not smaller than the threshold value TH (step S301a; No), the analysis control unit 310 controls the image analysis unit 109 to use all of the multiple analysis processes executed by the image analysis unit 109 (step S301c).

[0195] After executing step S301b or S301c, the analysis control unit 310 executes the processes from step S101a onward according to the first embodiment.

[0196] In this embodiment, before executing the processing from step S101a onwards, the analysis control unit 310 executes step S301b or S301c depending on the situation of the inspection site R, and switches the analysis processing used to process the image captured of the inspection site R.

[0197] In the example of this embodiment, when the congestion level is less than the threshold value TH, the analysis processing used to process the image of the inspection site R can be reduced compared to when the congestion level is greater than or equal to the threshold value TH.

[0198] When the degree of congestion is low, it is easy to visually check the inspection site R, so even if the accuracy of the analysis processing is relatively low, it is possible to support monitoring at the inspection site R. When the degree of congestion is high, it is difficult to visually check the inspection site R, so high-accuracy analysis processing is expected. In addition, by reducing the analysis processing used to process images captured at the inspection site R, it is possible to reduce the processing load on the monitoring device 302.

[0199] In this way, the detection process (S301) according to this embodiment makes it possible to effectively support monitoring at the inspection site R while reducing the processing load on the monitoring device 302.

[0200] So far, the third embodiment of the present invention has been described.

[0201] According to this embodiment, the detection unit 303 (analysis control unit 310) switches which of multiple analysis processes to use to process images captured at the inspection site R, depending on the situation at the inspection site R.

[0202] As a result, as described above, it is possible to reduce the processing load on the monitoring device 302 while maintaining the accuracy of the analysis processing required depending on the situation at the inspection site R. Therefore, it is possible to effectively support monitoring at the inspection site R while reducing the processing load on the monitoring device 302.

[0203] (Variation 3) In the third embodiment, the analysis control unit 310 may acquire aviation information including, for example, at least one of the takeoff time and landing time of an airplane. For example, the analysis control unit 310 may acquire the aviation information based on a user input, or may acquire it from an external device (not shown).

[0204] Then, based on the aviation information, the analysis control unit 310 may switch which of multiple analysis processes for processing images of the inspection site R to use to process the images of the inspection site R.

[0205] In more detail, for example, the analysis control unit 310 may predict the situation (e.g., the degree of congestion) of the inspection site R based on aviation information. In this case, the analysis control unit 310 may switch which of a plurality of analysis processes for processing images of the inspection site R to use for processing the images of the inspection site R based on the prediction result.

[0206] In this example, too, the analysis control unit 310 can switch which of multiple analysis processes for processing images of the inspection site R to use to process images of the inspection site R, depending on the situation at the inspection site R.

[0207] Furthermore, for example, the analysis control unit 310 may switch which of a plurality of analysis processes for processing an image of inspection site R is to be used to process the image of inspection site R, based on whether or not it is a predetermined time from the time included in the aviation information. Generally, inspection site R becomes congested as airplanes take off and land.

[0208] Therefore, in this example, the analysis control unit 310 can switch which of multiple analysis processes for processing images taken of the inspection site R to use to process images taken of the inspection site R, depending on the situation at the inspection site R.

[0209] In this modification, the detection unit 303 (analysis control unit 310) switches which of a plurality of analysis processes for processing an image of the inspection site R is to be used to process the image of the inspection site R, depending on the situation of the inspection site R. Therefore, the same effects as those of the third embodiment are achieved.

[0210] Although the embodiments and modifications of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0211] In addition, although the flowcharts used in the above description show multiple steps (processes) in a sequential order, the order of steps performed in each embodiment is not limited to the order shown. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of content. Furthermore, the above-described embodiments and variations can be combined as long as the content is not contradictory.

[0212] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0213] 1. a detection means for detecting a person who has performed a predetermined action based on an image of the inspection site; and an output means for outputting detection information relating to the detected person. monitoring equipment. 2. The predetermined behavior includes a first behavior defined in relation to an object included in the image. 10. The monitoring device described in Appendix 1. 3. The object is luggage, The first action includes being within a first range from the load and in a first position. 10. The monitoring device described in Appendix 2. 4. The first range from the luggage is a range that touches the luggage. 10. The monitoring device described in Appendix 3. 5. The first posture is at least one of a squatting posture and a crouching posture. 5. The monitoring device of claim 3 or 4. 6. The first behavior includes the first posture being maintained for a first period of time or more. 6. The monitoring device of any one of appendixes 3 to 5. 7. The object is a communication device, The first behavior includes being in a position to make a call using the communication device. 7. The monitoring device of any one of claims 2 to 6. 8. The first behavior includes the state of talking being continued for a second period of time or more. 7. The monitoring device described in Appendix 7. 9. The first action further includes holding the communication device at a second range from the face. 9. The monitoring device of claim 7 or 8. 10. The second range from the face is a range that touches the face. 10. The monitoring device described in claim 9. 11. The first behavior includes talking. 11. The monitoring device of any one of clauses 7 to 10. 12. The predetermined behavior includes a second behavior determined with respect to a line of sight of a person included in the image. 12. The monitoring device of any one of claims 1 to 11. 13. The second behavior includes changing the direction of gaze continuously for a third period of time or more, or intermittently at a first frequency or more. 13. The monitoring device of claim 12. 14. The second behavior includes a behavior related to a line of sight toward a predetermined target. 14. The monitoring device of claim 12 or 13. 15. The second behavior is directing one's gaze toward the predetermined target continuously for a fourth period of time or more, or intermittently at a second frequency or more. 15. The monitoring device of claim 14. 16. The predetermined behavior includes a third behavior defined regarding the movement of the person included in the image. 16. The monitoring device of any one of claims 1 to 15. 17. The third behavior includes staying in the first area for a fifth time or more. 17. The monitoring device of claim 16. 18. The first area is a predetermined range from the inspection site or the reference location. 18. The monitoring device of claim 17. 19. The third behavior includes staying in a constant posture in the first area for the fifth period of time or more. 19. The monitoring device of claim 17 or 18. 20. The third behavior includes a behavior in which the person included in the image avoids a predetermined target. 20. The monitoring device of any one of clauses 16 to 19. twenty one. The avoidance behavior includes at least one of a behavior of reversing a moving direction within a third range from the predetermined target, and a behavior of changing a moving direction to move outside a fourth range when a predicted position based on the moving direction of the person falls within a fourth range from the predetermined target. 21. The monitoring device of claim 20. twenty two. The predetermined actions include a fourth action defined regarding the person included in the image entering or exiting a predetermined area. 22. The monitoring device of any one of claims 1 to 21. twenty three. The fourth behavior includes the person included in the image not leaving the predetermined area for a sixth time or more after entering the predetermined area. 23. The monitoring device of claim 22. twenty four. The fourth behavior includes a behavior in which a person included in the image changes their appearance in the predetermined area. 24. The monitoring device of claim 22 or 23. twenty five. The detection means detects a person whose appearance changes between when entering the predetermined area and when leaving the predetermined area as a person who has performed an action that changes the appearance. 25. The monitoring device of claim 24. 26. The detection means holds entry information regarding persons who have entered the predetermined area, and detects persons who exit the predetermined area but are not included in the entry information as persons who have performed the behavior that changes their appearance. 26. The monitoring device of claim 24 or 25. 27. The detection information includes images of a person who has performed an action that changes the appearance, before and after the appearance changes. 27. The monitoring device of any one of clauses 24 to 26. 28. The detection means further performs processing to track the detected person, The detection information includes a result of tracking the detected person. 28. The monitoring device of any one of claims 1 to 27. 29. When the detection means detects that the detected person has left the inspection site, the detection means erases information for tracking the person who has left the inspection site. 29. The monitoring device of any one of clauses 1 to 28. 30. The detection means switches which of a plurality of analysis processes for processing the image is to be used to process the image depending on the situation of the inspection site. 30. The monitoring device of any one of claims 1 to 29. 31. A monitoring device according to any one of claims 1 to 30; an imaging device that generates image information including an image of the inspection site in response to imaging the inspection site; Surveillance system. 32. The computer Detecting a person who has performed a predetermined action based on an image of the inspection site; outputting detection information regarding the detected person. Monitoring method. 33. On the computer, Detecting a person who has performed a predetermined action based on an image of the inspection site; and outputting detection information relating to the detected person. [Explanation of symbols]

[0214] 100, 200, 300 monitoring systems 101 Imaging equipment 102,202,302 Monitoring equipment 103,203,303 Detection unit 104 Output section 105 Display section 106 Image storage unit 107 Detection memory unit 108 Image acquisition unit 109 Image Analysis Unit 110,210,310 Analysis control section 111 Detection Information P,Pa,Pb,Pc person L, La, Lb, Lc luggage R Inspection Station SP Smartphone T Writing desk

Claims

1. a detection means for detecting a person who has performed a predetermined action based on an image of the inspection site; an output means for outputting detection information relating to the detected person; the predetermined action includes a first action defined in relation to the luggage included in the image; the first behavior includes being within a first range from the luggage and maintaining a first posture for a first time or more; the first range from the luggage is a range that touches the luggage, The first posture is at least one of a squatting posture and a crouching posture. monitoring equipment.

2. The predetermined action includes an action determined in relation to a communication device included in the image, The predetermined behavior includes at least one of being in a posture for talking using the communication device and being in the posture for talking for a second period of time or more. The monitoring device of claim 1 .

3. The predetermined behavior includes a second behavior determined with respect to a line of sight of a person included in the image. The monitoring device according to claim 1 or 2.

4. The second behavior includes changing the direction of gaze continuously for a third period of time or more, or intermittently at a first frequency or more. The monitoring device according to claim 3.

5. the predetermined behavior includes a third behavior defined regarding movement of a person included in the image; The third behavior includes a behavior in which the person included in the image avoids a predetermined target. A monitoring device according to any one of claims 1 to 4.

6. The avoidance behavior includes at least one of a behavior of reversing a moving direction within a third range from the predetermined target, and a behavior of changing a moving direction to move outside a fourth range when a predicted position based on the moving direction of the person falls within a fourth range from the predetermined target. The monitoring device according to claim 5.

7. the predetermined behavior includes a fourth behavior defined regarding a person included in the image entering or exiting a predetermined area; the fourth behavior includes a behavior in which a person included in the image changes their appearance in the predetermined area; The detection means detects a person whose appearance changes between when entering the predetermined area and when leaving the predetermined area as a person who has performed an action that changes the appearance. A monitoring device according to any one of claims 1 to 6.

8. The computer Based on images taken at the inspection site, it detects people who have performed predetermined actions, outputting detection information relating to the detected person; the predetermined action includes a first action defined in relation to the luggage included in the image; the first behavior includes being within a first range from the luggage and maintaining a first posture for a first time or more; the first range from the luggage is a range that touches the luggage, The first posture is at least one of a squatting posture and a crouching posture. Monitoring method.

9. On the computer, Based on images taken at the inspection site, it detects people who have performed predetermined actions, outputting detection information relating to the detected person; the predetermined action includes a first action defined in relation to the luggage included in the image; the first behavior includes being within a first range from the luggage and maintaining a first posture for a first time or more; the first range from the luggage is a range that touches the luggage, The program, wherein the first posture is at least one of a squatting posture and a crouching posture.

Images