Monitoring device, monitoring method, and program

Abstract

To allow for easily changing a condition for detection on a user side while utilizing an image analysis system or the like using AI (Artificial Intelligence).SOLUTION: An object information acquisition unit 22 acquires object information indicating a position of an object detected from each of a plurality of pieces of image data. A determination unit 23 determines whether the object information acquired by the object information acquisition unit 22 with respect to each of the plurality of pieces of image data satisfies a determination condition about movement of the object defined by a detection pattern provided so as to be editable. A notification unit 25 makes a notification if the determination unit 23 determines that the determination condition is satisfied.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a technique for monitoring a moving state using image data. [Background technology]

[0002] In pools or the sea, lifeguards monitor to make sure no one is drowning. On roads, police officers or contracted companies monitor to make sure no one is parking on the street. Such monitoring work takes time and effort if done visually. There is also the risk that monitoring will not be thorough.

[0003] Instead of direct visual monitoring, people may visually monitor image data obtained by a surveillance camera. In recent years, monitoring by analyzing image data obtained by a surveillance camera using an image analysis system using AI has been considered. AI stands for Artificial Intelligence. For example, Patent Document 1 describes a method of determining whether or not workers at a construction site are wearing equipment such as helmets and gloves through image analysis. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-043932 Summary of the Invention [Problem to be solved by the invention]

[0005] The detection conditions for detecting a monitored object vary depending on the site. The detection conditions for detecting a monitored object also vary depending on factors such as the installation location of the camera used to acquire image data. Depending on the site, the installation location of the camera may change. This may result in the need to change the detection conditions. With an image analysis system that uses AI, it is difficult for users to change the detection conditions themselves; to change the detection conditions, users must request the vendor to modify the image analysis system. Having an image analysis system modified takes time and money, and is inconvenient for users. The present disclosure aims to enable users to easily change detection conditions while using an image analysis system that uses AI. [Means for solving the problem]

[0006] A monitoring device according to the present disclosure includes: an object information acquisition unit that acquires object information indicating the position of an object detected from each of the plurality of image data; a determination unit that determines whether the object information acquired for each of the plurality of image data by the object information acquisition unit satisfies a determination condition for object movement defined by an editable detection pattern; a notification unit that issues a notification when the determination unit determines that the detection pattern is satisfied; Equipped with. [Effects of the Invention]

[0007] In the present disclosure, a notification is issued when a determination condition for object movement defined by an editable detection pattern is met. Because the detection pattern is set to be editable, the user can easily change the determination condition, which is the detection condition. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a configuration diagram of a monitoring system 100 according to a first embodiment. [Figure 2] 1 is a configuration diagram of a monitoring device 10 according to a first embodiment. [Figure 3] FIG. 3 is an explanatory diagram of a setting file 31 according to the first embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a detection pattern master 32 according to the first embodiment. [Figure 5] FIG. 3 is an explanatory diagram of a result file 33 according to the first embodiment. [Figure 6] 3 is a flowchart showing the flow of operations of the monitoring device 10 according to the first embodiment. [Figure 7] FIG. 3 is an explanatory diagram of a measurement process according to the first embodiment. [Figure 8] FIG. 3 is an explanatory diagram of a measurement process according to the first embodiment. [Figure 9] FIG. 3 is an explanatory diagram of a movement determination process according to the first embodiment. [Figure 10] FIG. 3 is an explanatory diagram of a movement determination process according to the first embodiment. [Figure 11] FIG. 10 is a supplementary explanatory diagram of the measurement process in the first embodiment. [Figure 12] FIG. 10 is an explanatory diagram of a period determination process according to the first modification. [Figure 13] FIG. 10 is an explanatory diagram of a measurement process according to Modification 2. [Figure 14] FIG. 20 is an explanatory diagram of a detection pattern master 32 according to a sixth modification. [Figure 15] 13 is a flowchart showing the flow of operations of the monitoring device 10 according to the sixth modification. [Figure 16] FIG. 20 is an explanatory diagram of a detection pattern master 32 according to a seventh modification. DETAILED DESCRIPTION OF THE INVENTION

[0009] Embodiment 1 In the first embodiment, an example of monitoring whether or not someone is drowning in a pool will be described. However, the monitoring is not limited to whether or not someone is drowning in a pool, and other monitoring can also be performed, such as monitoring whether or not someone is parking on the street.

[0010] ***Configuration Description*** The configuration of a monitoring system 100 according to the first embodiment will be described with reference to FIG. The monitoring system 100 comprises a monitoring device 10, one or more cameras 50, and one or more alarm devices 60. The monitoring device 10 is connected to each of the cameras 50 and each of the alarm devices 60 via a network 70. The monitoring device 10 is a computer that detects unsafe conditions and issues notifications. Each camera 50 is a surveillance camera installed to capture images of the area being monitored. In the first embodiment, each camera 50 is a surveillance camera installed to capture images of the underwater world of the pool. Each camera 50 may be installed so as to be embedded in the bottom or side of the pool, but here it is assumed that it is installed on the lane rope. Since the camera 50 is installed on the lane rope, it captures images of the underwater world from near the water surface. The alarm 60 is a device that notifies lifeguards and the like by sound and light. In the first embodiment, the alarm 60 is a device that notifies that someone is drowning.

[0011] The configuration of the monitoring device 10 according to the first embodiment will be described with reference to FIG. The monitoring device 10 is a computer. The monitoring device 10 includes the following hardware components: a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware components via signal lines and controls the other hardware components.

[0012] The processor 11 is an IC that performs processing. IC stands for Integrated Circuit. Specific examples of the processor 11 include a CPU, a DSP, and a GPU. CPU stands for Central Processing Unit. DSP stands for Digital Signal Processor. GPU stands for Graphics Processing Unit.

[0013] The memory 12 is a storage device that temporarily stores data. Specific examples of the memory 12 include SRAM and DRAM. SRAM stands for Static Random Access Memory. DRAM stands for Dynamic Random Access Memory.

[0014] The storage 13 is a storage device that stores data. Specific examples of the storage 13 include an SSD or an HDD. SSD stands for Solid State Drive. HDD stands for Hard Disk Drive. The storage 13 may also be a portable recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. SD stands for Secure Digital. DVD stands for Digital Versatile Disk.

[0015] The communication interface 14 is an interface for communicating with external devices. Specific examples of the communication interface 14 include Ethernet (registered trademark), USB, and HDMI (registered trademark) ports. USB stands for Universal Serial Bus. HDMI stands for High-Definition Multimedia Interface.

[0016] The monitoring device 10 includes, as functional components, a video acquisition unit 21, an object information acquisition unit 22, a determination unit 23, a measurement unit 24, a notification unit 25, and an operation unit 26. The functions of each functional component of the monitoring device 10 are realized by software. The storage 13 stores programs that realize the functions of each functional component of the monitoring device 10. These programs are loaded into the memory 12 by the processor 11 and executed by the processor 11. In this way, the functions of each functional component of the monitoring device 10 are realized.

[0017] The storage 13 stores a setting file 31, a detection pattern master 32, and a result file 33.

[0018] 2 shows only one processor 11. However, there may be a plurality of processors 11, and the plurality of processors 11 may cooperate to execute programs that realize the respective functions.

[0019] ***Explanation of Operation*** The operation of the monitoring device 10 according to the first embodiment will be described with reference to FIGS. The operation procedure of the monitoring device 10 according to the embodiment 1 corresponds to the monitoring method according to the embodiment 1. Furthermore, the program that realizes the operation of the monitoring device 10 according to the embodiment 1 corresponds to the monitoring program according to the embodiment 1.

[0020] The setting file 31 according to the first embodiment will be described with reference to FIG. The setting file 31 contains basic information about the monitoring device 10. The setting file 31 includes a video switching flag, a threshold number of times, a video saving folder, a video saving pattern, a video saving time, a video saving period, the number of video saving frames, the number of cameras, setting information, a video file, a warning sound, and a reference size. The video switching flag is a flag that switches the source of video data between the camera 50 and an external storage device. The threshold number is a reference number of times for determining whether to issue a notification. The video storage folder is a folder where video data is stored. The video storage pattern is a condition for saving video data. The video storage time is the time of video data saved as one video file. The video storage period is the period for which saved video data is retained. The number of video storage frames is the frame rate of the video data to be saved. The number of cameras is the number of cameras 50 included in the surveillance system 100. The setting information includes the camera name and the connection URL to which the camera 50 connects, set for each camera 50. The video file is video data captured by each camera. The warning audio is audio that is output when it is determined that an area is unsafe, etc. One or more warning audios are set, and a warning audio number is assigned to each. The reference size is the size of a reference object and is used in distance measurement, which will be described later. The reference size is set for each type of object.

[0021] The detection pattern master 32 according to the first embodiment will be described with reference to FIG. One or more detection patterns indicating the conditions for detecting an object for which a warning is issued are set in the detection pattern master 32. Each detection pattern includes a pattern number, a camera number, a warning sound number, and a judgment condition. The pattern number is the identification number of the detection pattern. The camera number is the identification number of the camera 50 to which the detection pattern corresponds. In other words, a detection pattern is set for each camera 50. The warning sound number is the identification number of the warning sound corresponding to the detection pattern. The judgment condition is a condition for determining whether or not the detection pattern is met. Multiple detection pattern masters 32 can be set for each camera 50. One detection pattern master 32 can also be shared by multiple cameras 50. The correspondence between cameras 50 and detection pattern masters 32 can be one-to-one, one-to-many, many-to-one, or many-to-many.

[0022] The determination conditions include one or more detected objects, an area, a detection time, and a movement distance. Each detection object is a condition for an object to be detected. A type of object to be detected is set for each detection object. The type of object indicates what the object to be detected is. The type of object is set to, for example, a person or a vehicle. In the first embodiment, a drowning person is detected, so the type of object is a person. The area is a condition for the region in which the detection target object is detected. Whether the condition is met is determined based on whether or not there is detection in the region indicated by the area. If the area is not set, whether or not the condition is met is determined based on whether or not there is detection in the entire shooting region of the camera 50. For example, the shooting region of the camera 50 is divided into multiple regions according to the distance from the camera 50, and an area is set for each region. The detection time is a condition regarding the time period during which the state of not moving continues. Whether the condition is met is determined based on whether the state of not moving continues for the detection time or longer. The movement distance is a condition for determining whether or not an object has moved. In a pair of two image data sets adjacent in time series, if the distance between the object detected in each of the two image data sets is within the movement distance, it is determined that the object has not moved during the period in which the two image data sets in that pair were acquired.

[0023] Here, the operation unit 26 accepts editing of the detection pattern master 32 as necessary. For example, when a new judgment condition is set, the operation unit 26 accepts addition of a detection pattern indicating the judgment condition. The operation unit 26 may also accept changes to the definition of an existing detection pattern.

[0024] The result file 33 according to the first embodiment will be described with reference to FIG. Information when a target for issuing a warning is detected is set in the result file 33. The result file 33 includes the camera number, the detection date and time, the pattern number, the warning audio number, and the video file. The camera number is the camera number of the camera 50 that captured the video data of the detection source of the target for which a warning is issued. The detection date and time is the date and time when the target for which a warning is issued was detected. The pattern number is the pattern number of the detection pattern in which the target for which a warning is issued was detected. The warning audio number is the warning audio number of the warning audio corresponding to the detection pattern indicated by the pattern number. The video file is the video file of the video data in which the target for which a warning is issued was detected.

[0025] The flow of operations of the monitoring device 10 according to the first embodiment will be described with reference to FIG. The processes from step S101 to step S110 are executed for each camera 50 as the target camera 50.

[0026] (Step S101: Video acquisition process) The video acquisition unit 21 acquires image data that is a new frame of video data captured by the target camera 50 in accordance with the setting file 31. This image data is called target image data. Specifically, the video acquisition unit 21 sets the source of the video data to either the target camera 50 or an external storage device, according to the video switching flag in the setting file 31. The video acquisition unit 21 acquires image data from the set source of the video data. Note that video acquisition unit 21 may not acquire all frame image data acquired according to the frame rate of camera 50, but may acquire only a portion of the image data acquired by camera 50 at regular intervals. For example, if camera 50 acquires image data at 30 FPS, video acquisition unit 21 may acquire only one piece of image data every 0.5 seconds. FPS stands for Frames Per Second.

[0027] (Step S102: Image Storage Process) The video acquisition unit 21 stores the target image data acquired in step S101 in the storage 13 in accordance with the setting file 31. Specifically, when the conditions indicated by the video saving pattern in the setting file 31 are met, the video acquisition unit 21 stores the image data in the destination folder indicated by the video saving folder in the setting file 31. At this time, the video acquisition unit 21 stores the image data while dividing it into different video files for each time indicated by the video saving time in the setting file 31. Furthermore, the video acquisition unit 21 stores the image data while thinning it out as necessary so as to achieve the frame rate indicated by the number of video saving frames in the setting file 31.

[0028] (Step S103: Object information acquisition process) The object information acquisition unit 22 acquires object information indicating the type and position of one or more objects detected from the image data of the object acquired in step S101. Specifically, the object information acquisition unit 22 inputs image data of the target into an object detection model. The object detection model is a model that detects objects of a detection target type using AI technology. The object detection model is, for example, a model constructed using CNN. CNN stands for Convolutional Neural Network. The object detection model detects objects of the detection target type from the input image data and outputs object information indicating the type and position of each detected object. The object detection model outputs a detection frame surrounding the object as the position. The object information acquisition unit 22 acquires the object information indicating the type and position of each object output from the object detection model. In the first embodiment, the detected object is a person, but since it is particularly desired to detect a person drowning in a pool, the object detection model can be a trained model that has been trained to treat the entire body or part of the body of a person swimming or diving as a person. In this way, the object detection model can be a trained model that has been trained according to the detected object. The object information acquisition unit 22 associates the acquired object information with the image data being processed, and stores the image data and the object information in the memory 12. In the following description, the term "image data" refers to image data including object information.

[0029] (Step S104: Area object extraction process) The determination unit 23 determines whether the position of the object indicated by the object information acquired in step S103 is included within the region indicated by the area in the target detection pattern. The determination unit 23 extracts, from the objects indicated by the object information, objects whose position is included within the region indicated by the area in the target detection pattern. If the area in the target detection pattern has not been set, the determination unit 23 extracts all objects indicated by the object information acquired in step S103.

[0030] (Step S105: Object determination process) The determination unit 23 determines whether or not the type of object indicated by the object information for the object extracted in step S104 matches the type of detected object in the target detection pattern. If the type of object indicated by the object information matches the type of detected object in the target detection pattern, determination unit 23 proceeds to step S106. On the other hand, if the type of detected object does not match, determination unit 23 deletes the image data stored in memory 12 in step S103 and then returns the process to step S101. In the first embodiment, if the type of object indicated by the object information is a person, determination unit 23 proceeds to step S106, and otherwise ends the process.

[0031] (Step S106: Measurement process) The determination unit 23 sets a pair of two image data, the target image data stored in step S103 and the immediately previous image data stored in step S103 of the previous process, as the target pair. That is, the determination unit 23 sets a pair of two image data that are adjacent in time series as the target pair. The determination unit 23 instructs the measurement unit 24 to measure the distance between the positions indicated by the object information acquired for each of the two image data in the target pair. The measurement unit 24 measures the distance between the positions indicated by the object information acquired for each of the two image data in the target pair. Specifically, the measurement unit 24 measures the distance between an object extracted in step S104 from the object information acquired for the target image data and determined in step S105 to match the type of detected object in the target detection pattern, and an object extracted in step S104 from the object information acquired for the immediately previous image data and determined in step S105 to match the type of detected object in the target detection pattern. Here, it is assumed that one object is extracted from both image data in step S104. In the first embodiment, as shown in FIG. 7, the determination unit 23 measures the distance between each of the four corner points of the detection frame of each of the two image data as the distance between the objects in the two image data. That is, the determination unit 23 measures the distance between the top left points of the detection frame, the distance between the top right points of the detection frame, the distance between the bottom left points of the detection frame, and the distance between the bottom right points of the detection frame. As shown in FIG. 8, the distance Z between two points is the Euclidean distance between the two points. In FIG. 8, the coordinates (X1, Y1) of the top left point of the detection frame of object A in image data X are (2, 17), and the coordinates (X2, Y2) of the top left point of the detection frame of object A in image data Y are (7, 19). Therefore, distance Z=√{(X2-X1) 2 +(Y2-Y1) 2}=√{(7-2) 2 +(19-17) 2}=√{5 2 +2 2}=√{25+4}=√29=5.385...

[0032] (Step S107: Movement determination process) The determination unit 23 compares the distance measured in step S106 with the movement distance in the target detection pattern to determine whether the object is moving. If all of the distances measured for each of the four corner points are within the movement distance, the determination unit 23 determines that the object did not move during the period in which the two image data in the target pair were acquired. In other words, the determination unit 23 determines that the object did not move during the target period from the time the previous image data was acquired to the time the target image data was acquired. On the other hand, if even one of the distances measured for each of the four corner points exceeds the movement distance, the determination unit 23 determines that the object moved during the target period. The determination unit 23 writes movement determination data in memory 12 that associates the target period with whether or not the object moved. If the determination unit 23 determines that the user has not moved, the process proceeds to step S108. On the other hand, if the determination unit 23 determines that the user has moved, the process returns to step S101.

[0033] Let's say the movement distance in the detection pattern is 50px, where px means pixel. At this time, assume that the position of object A in image data X, which is the previous image data, and the position of object A in image data Y, which is the target image data, have been identified, as shown in Figure 9. In this case, the distances between the four corner points are 36px for the top left point, 38px for the top right point, 45px for the bottom left point, and 40px for the bottom right point. In this case, the distances measured for each of the four corner points are all within the movement distance, so it is determined that the object did not move between the time when image data X was acquired and the time when image data Y was acquired.

[0034] Assume that the movement distance in the detection pattern is 50px. At this time, assume that the position of object A in image data X, which is the previous image data, and the position of object A in image data Y, which is the target image data, have been identified, as shown in Figure 10. In this case, the distances between the four corner points are 51px for the top left point, 45px for the top right point, 49px for the bottom left point, and 40px for the bottom right point. In this case, the distance measured for the upper left point exceeds the movement distance, so it is determined that the object moved between the time when image data X was acquired and the time when image data Y was acquired.

[0035] If it is difficult to determine movement in the determination of step S107, the images set as the target pair in step S106 may be image data from several images prior to the previous image data, rather than the previous image data. This may make it easier to determine movement in accordance with human movement. In this case, if the determination of step S105 is YES, this can be achieved by retaining the image data stored in memory 12 in step S103. To make it easier to detect that an object is not moving, the determination unit 23 may be configured to determine that an object is moving during the period in which two image data sets in the target pair were acquired if, for example, two or more of the distances measured for each of the four corner points exceed the movement distance. With this configuration, it is determined that an object is moving based on the result that a predetermined number of points or more of the four corners have moved, making it easier to detect a person drowning in a pool. In this way, the condition for determining that an object is moving can be set arbitrarily regarding the relationship between the distance measured for each of the four corner points and the movement distance.

[0036] (Step S108: Period Determination Process) The determination unit 23 determines whether or not a state of no movement for the detection time or longer in the target detection pattern continues. Specifically, the determination unit 23 reads movement determination data for the past detection time from the memory 12. In the first embodiment, if all of the movement determination data for the past detection time indicates no movement, the determination unit 23 determines that a state of no movement for the detection time or longer continues. On the other hand, if at least a portion of the movement determination data for the past detection time indicates movement, the determination unit 23 determines that a state of no movement for the detection time or longer has not continued. If the determination unit 23 determines that the state of no movement has continued for the detection time or more, the process proceeds to step S109. On the other hand, if the determination unit 23 determines that the state of no movement has not continued for the detection time or more, the process returns to step S101.

[0037] (Step S109: Result storage process) The determination unit 23 stores the detection result in the result file 33. Specifically, the determination unit 23 writes the camera number of the target camera 50, the current date and time, the pattern number of the target detection pattern, the warning sound number of the warning sound corresponding to the target detection pattern, and the identification information of the video file into which the image data was stored in step S102 to the result file 33. The determination unit 21 erases the movement determination data written to the memory 12.

[0038] (Step S110: Notification process) The notification unit 25 notifies that a state that satisfies the determination condition has occurred. Specifically, the notification unit 25 outputs an audio warning corresponding to the target detection pattern to the alarm device 60. This causes the alarm device 60 to light up and output an audio warning from the alarm device 60. In the first embodiment, a notification is given that a person is drowning.

[0039] With reference to FIG. 11, a method for measuring distance in the measurement process (step S107 in FIG. 6) in the first embodiment will be additionally described. The measurement unit 24 measures the distance between two objects, that is, the object detected in the target image data and the object detected in the immediately previous image data. Here, the distance is, for example, the number of pixels. In the first embodiment, the distance between the two objects is the distance between each of the four corner points of the detection frame in each of the two image data. FIG. 11 shows, as an example, a case where the number of pixels between the upper left point of the detection frame of each of the two objects is measured. Here, it is assumed that the distance between the two objects is 40 pixels.

[0040] Next, the measurement unit 24 corrects the measured distance between objects using the ratio between the size of the object detected from the target image data and the reference size corresponding to the type of detected object. Here, the reference size of the detected object in the target detection pattern is assumed to be 30 pixels wide and 40 pixels high. Meanwhile, the size of the detected object is assumed to be 40 pixels wide and 55 pixels high. Furthermore, the distance between the upper left points of the detection frame is assumed to be 40 pixels. First, the measurement unit 24 calculates the ratio as follows. (1) Diagonal distance of the standard size = √(30 2 +40 2 )=50 (2) The diagonal distance of the detected object size = √(40 2 +55 2 )=68.00... (3) Ratio=68.00 / 50=1.36... Next, the measurement unit 24 corrects the distance between the objects in the two image data using the ratio: (4) Corrected distance = 40 / 1.36 = 29.41 Based on this calculation, the distance between the two upper left points of the detection frame is corrected from 40 pixels to 29.41 pixels. The corrected distances for the other three corner points are calculated in a similar manner. In the movement determination process (step S107 in FIG. 6), the corrected distance is compared with the movement distance to determine whether or not movement has occurred.

[0041] Here, the ratio is calculated from the reference size of the detected object and the size of the object detected in the target image data. However, the ratio may also be calculated from the reference size of the detected object and the size of the object detected in the previous image data. Furthermore, the ratio used for correction may be calculated using the ratio between the reference size of the detected object and the size of the object detected in the target image data and the ratio between the reference size of the detected object and the size of the object detected in the previous image data. For example, the average value of the ratio between the reference size of the detected object and the size of the object detected in the target image data and the ratio between the reference size of the detected object and the size of the object detected in the previous image data may be used as the ratio used for correction.

[0042] ***Effects of the First Embodiment*** As described above, the monitoring device 10 according to the first embodiment issues a notification when the determination conditions for object movement defined by the editable detection pattern are met. There are cases where the detection conditions must be changed due to, for example, a change in the installation position of the camera for acquiring image data. In the monitoring device 10 according to the first embodiment, the detection pattern is set to be editable, so that the user can easily change the determination conditions, which are the detection conditions.

[0043] Furthermore, the monitoring device 10 according to the first embodiment measures the distance between each of the four corners of the detection frame, treating the image data of the object and the immediately preceding image data as a pair, and determines that the object is moving if any of the distances exceeds the movement distance. With this determination method, if the object is a person, it can determine that the object is moving if part of a hand or part of a foot is moving. The criteria for determining whether an object is moving can be set arbitrarily based on the relationship between the distance measured at each of the four corners and the distance traveled. This configuration allows for settings that take into account the ease of detecting a drowning person in a pool and the importance of preventing false positives.

[0044] ***Other Configurations*** <Variation 1> In the first embodiment, in the period determination process (step S108 in FIG. 6), if all of the movement determination data for the past detection time indicates no movement, the determination unit 23 determines that the state of no movement for the detection time or more has continued. Also, if at least a portion of the movement determination data for the past detection time indicates movement, the determination unit 23 determines that the state of no movement for the detection time or more has not continued. There is a possibility that the position of an object may be detected as being temporarily off due to a false detection. If such a false detection occurs, it will be impossible to make an appropriate judgment. Therefore, the processing may be changed as follows. As shown in FIG. 12 , a reference ratio is added to the judgment conditions in the detection pattern master 32. Then, when movement judgment data equal to or greater than the reference ratio among the movement judgment data for the past detection time indicates no movement, the judgment unit 23 judges that the state of no movement for the detection time or longer has continued. That is, when the judgment unit 23 determines that no movement has occurred for pairs of two image data sets for the past detection time at a ratio equal to or greater than the reference ratio, the judgment unit 23 judges that the state of no movement for the detection time or longer has continued. Furthermore, when the judgment unit 23 determines that only movement judgment data less than the reference ratio among the movement judgment data for the past detection time indicates no movement, the judgment unit 23 judges that the state of no movement for the detection time or longer has not continued. That is, when the judgment unit 23 determines that only pairs of two image data sets less than the reference ratio have not moved among the past detection time, the judgment unit 23 judges that the state of no movement for the detection time or longer has not continued.

[0045] <Variation 2> In the first embodiment, it is assumed that there is one object extracted from the image data in step S104 and determined in step S105 to match the type of detected object in the target detection pattern. In other words, it is assumed that only one object is captured in the area indicated by the area in the target detection pattern. However, there are cases where multiple objects are extracted from one image data. In other words, there are cases where multiple objects are captured in the area indicated by the area in the target detection pattern. In this case, when measuring distance in the measurement process (step S106 in FIG. 6), it is necessary to measure the distance between the same objects after associating the same objects between the object in the previous image data and the object in the target image data. Therefore, as shown in FIG. 13, the measurement unit 24 identifies, for each of the multiple objects in the target image data, the object that is closest to the multiple objects in the previous image data as the same object. Here, the measurement unit 24 uses, for example, only the distance to one of the four corner points to identify the object that is closest as the same object. In FIG. 13, only the distance to the upper left point is used. Then, the measurement unit 24 measures the distance between the same object and the identified object. That is, the measurement unit 24 measures the distance between the same object and the identified object for each of the four corner points.

[0046] Note that when the closest object is identified as the same object, there is a possibility that, for multiple objects in the target image data, one object in the previous image data may be identified as the same object. In this case, the measurement unit 24 may re-identify the same object using not only the distance between one of the four corner points but also the distance between the other points. Alternatively, the measurement unit 24 may re-identify the same object using the feature amount of the object. Nevertheless, there may be cases where one object in the previous image data is identified as the same object for multiple objects in the target image data. For example, suppose the target image data contains objects A, B, C, and D, and the previous image data contains objects V, W, X, and Y. In this case, suppose objects A, B, and C are identified as the same object as object V, and object D is identified as the same object as object Y. In this case, the measurement unit 24 determines that object D is the same object as object Y. Furthermore, the measurement unit 24 identifies the object A, B, and C in the target image data that is most likely to be recognized as the same object as object V in the previous image data. The measurement unit 24 determines that the identified object (assumed to be object A) and object V in the previous image data are the same object. Then, the measurement unit 24 re-identifies the same object for objects B and C other than identified object A among objects A, B, and C in the target image data, using the distance between them and objects W and X other than object V among the multiple objects in the previous image data.

[0047] <Variation 3> When many objects are captured in the area indicated by the area in the target detection pattern, it is difficult to identify the same object using the method described in Modification 2. However, when detecting a drowning person as in the example described in Embodiment 1, if there are many people nearby, detection is considered unnecessary. This is because if there are many people near the drowning person, it is considered that the people nearby will notice that the person is drowning even if the lifeguard does not notice. In other words, if the people nearby notice that the person is drowning, it is considered unnecessary to force detection, even though the possibility of false detection is high. Therefore, a reference number is added to the setting file 31 or the detection pattern master 32. Then, if the number of objects extracted from the image data in step S104 and determined to match the type of detection object in the target detection pattern in step S105 exceeds the reference number, the determination unit 23 may stop determining whether the detection pattern is satisfied. In other words, if the number of objects extracted from the image data in step S104 and determined to match the type of detection object in the target detection pattern in step S105 exceeds the reference number, the determination unit 23 may end processing for the target detection pattern. Here, for example, the reference number is set to 4. If the reference number is 4, then if there is one drowning person, there will be three people around. If there are three people around, there is a high probability that they will notice that there is a drowning person.

[0048] <Variation 4> In the first embodiment, a drowning person is detected using image data acquired by a camera 50 that is installed on a lane rope and captures images of the underwater world from near the water surface. Because the images are taken from near the water surface, the upper parts of a person standing, such as the face and shoulders, are not captured. Also, since a person swimming is usually near the water surface, only a small part of the person submerged in water is captured. Therefore, if a person who has drowned is to be detected, it is desirable to avoid detecting people who are standing or swimming. Therefore, the object information acquisition unit 22 may acquire object information only about people whose upper bodies, including faces and shoulders, are shown. In other words, the upper bodies of people may be set as detection targets in the object detection model. This prevents the acquisition of object information about unnecessarily many people, making it possible to properly detect drowning people.

[0049] <Variation 5> In the first embodiment, the monitoring device 10 identifies an object that has not moved continuously for the detection time. However, the monitoring device 10 may detect an object that has been moving continuously for the detection time. In this case, if the determination unit 23 determines in the movement determination process (step S107 in FIG. 6) that the object has been moving, the process proceeds to step S108, and if the determination unit 23 determines that the object has not been moving, the process for the target detection pattern ends. In the period determination process (step S108 in FIG. 6), the determination unit 23 determines whether the state of movement has continued for more than the detection time in the target detection pattern.

[0050] <Variation 6> In the sixth modification, as shown in FIG. 14, a detection time interval (seconds) and a number of checks are added to the judgment conditions in the detection pattern master 32. The monitoring device 10 of the sixth modification then executes the processing shown in FIG. 15. At the start of step S101, the object information acquisition unit 22 refers to the detection pattern master 32, and the object information acquisition unit 22 and the judgment unit 23 repeat the processing of detecting an object within the area of ​​the detection pattern from the target image data in steps S101 to S104′ based on the detection time interval in the detection pattern master 32. The processing contents of steps S101 to S103 are the same as those described in the first embodiment, but in the processing of step S104′, the judgment unit 23 stores the number of times the same object has been detected. In step S105′, if the judgment unit 23 determines in the processing of step S104 that the same object has been detected a number of times equal to or greater than the number of checks of the detection pattern master 32, the processing proceeds to step S106. The processing from step S106 onwards is as explained in the first embodiment, but the image data to be compared in the measurement in step S106 is the image data acquired in the initial step S103. If the determination unit 23 determines in step S105' that the same object has not been detected the number of times checked, the process returns to step S101. For example, if the detection time interval is set to 10 seconds and the number of checks is set to 15 in the detection pattern master 32, the object information acquisition unit 22 and the determination unit repeat the processes from step S101 to step S104' for 10 seconds, and the determination unit 23 determines whether a person has been detected 15 or more times in step S105'. If the determination unit 23 determines in step S105' that a person has been detected 15 or more times, the process proceeds to step S106. If the determination unit 23 determines in step S105' that the number of times a person has been detected is less than 15, the process returns to step S101.

[0051] In the sixth modification, in step S104′, the object information acquisition unit 22 is configured to detect an object at least the check count within the detection time interval of the detection pattern master 32. The object information acquisition unit 22 is an object detection model, and this configuration has the effect of suppressing false detections by the object detection model.

[0052] <Variation 7> In Modification 7, as shown in Fig. 16, the number of detections and the number of checks are added to the judgment conditions in the detection pattern master 32. The processing flow will be explained using Fig. 15, which was used in Modification 6. In monitoring device 10 of Modification 7, in step S104', object information acquisition unit 22 repeats processing to detect an object within the area in the detection pattern based on the number of detections in the detection pattern master 32. When the processing of step S104' has been completed for the number of detections, if it is determined in step S105' that the same object has been detected a number of times equal to or greater than the number of checks in the detection pattern master 32, the processing proceeds to step S106. In the same manner as in the sixth modification, the seventh modification repeats the detection process for the detected object the number of times of detection, and when the object is detected the number of times of check, that is, when the ratio of the number of checks to the number of detections reaches a certain value or more, it is determined that the detected object has been detected. This configuration has the effect of suppressing false detections by the object information acquisition unit 22, which is the object detection model.

[0053] <Variation 8> In the first embodiment, the result storage process is performed in step S109, and the notification process is performed in step S110. However, the result storage process may be performed after the notification process. Furthermore, the notification process and the result storage process may be performed simultaneously.

[0054] <Variation 9> In the first embodiment, each functional component is realized by software. However, as a modification 9, each functional component may be realized by hardware. The differences between this modification 9 and the first embodiment will be described below.

[0055] When each functional component is realized by hardware, the monitoring device 10 includes an electronic circuit 15 instead of the processor 11, memory 12, and storage 13. The electronic circuit 15 is a dedicated circuit for realizing the functions of each functional component, the memory 12, and the storage 13.

[0056] The electronic circuit 15 may be a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. Each functional component may be realized by one electronic circuit 15, or each functional component may be realized by distributing it among a plurality of electronic circuits 15.

[0057] <Modification 10> As a tenth modification, some of the functional components may be realized by hardware, and other functional components may be realized by software.

[0058] The processor 11, memory 12, storage 13, and electronic circuit 15 are collectively referred to as a processing circuit. In other words, the functions of the respective functional components are realized by the processing circuit.

[0059] Furthermore, the term "unit" in the above description may be read as a "circuit," "step," "procedure," "process," or "processing circuit."

[0060] Various aspects of the present disclosure are summarized below as appendices. (Appendix 1) an object information acquisition unit that acquires object information indicating the position of an object detected from each of the plurality of image data; a determination unit that determines whether the object information acquired for each of the plurality of image data by the object information acquisition unit satisfies a determination condition for object movement defined by an editable detection pattern; a notification unit that issues a notification when the determination unit determines that the detection pattern is satisfied; A monitoring device comprising: (Appendix 2) The detection pattern is set using an editable detection time, and the determination condition is that the object has not moved continuously for at least the detection time. 10. The monitoring device described in Appendix 1. (Appendix 3) the detection pattern has a determination condition defined using a moving distance that is set in an editable manner; The determination unit targets pairs of two image data sets that are adjacent in time series among the plurality of image data sets, and determines that the object did not move during the period in which the two image data sets in the target pair were acquired if the distance between the positions indicated by the object information acquired for each of the two image data sets in the target pair is within the movement distance. 10. The monitoring device described in Appendix 2. (Appendix 4) the detection pattern has a reference ratio that is set in an editable manner, and the determination condition is determined using the reference ratio; The determination unit determines that the object has not moved continuously for at least the detection time when it is determined that no movement has occurred for at least the reference ratio of pairs of two image data sets during the detection time. 10. The monitoring device described in Appendix 3. (Appendix 5) When the number of the objects detected from one image data exceeds a reference number, the determination unit stops determining whether the detection pattern is satisfied. 5. The monitoring device of claim 1. (Appendix 6) the plurality of image data are acquired by a camera provided to photograph underwater; the object is a person, The determination unit determines that a person is drowning when the detection pattern is satisfied. 6. The monitoring device of any one of claims 1 to 5. (Appendix 7) The camera is attached to a lane rope of the pool and is configured to photograph underwater images of the pool, The object information acquisition unit acquires object information about a person whose upper body is at least shown. 6. A monitoring device as described in Appendix 6. (Appendix 8) The computer acquires object information indicating the position of the detected object from each of the plurality of image data; a computer determines whether the object information acquired for each of the plurality of image data satisfies a determination condition for object movement defined by an editable detection pattern; A monitoring method in which a computer issues a notification when it is determined that the detection pattern is satisfied. (Appendix 9) an object information acquisition process for acquiring object information indicating the position of an object detected from each of the plurality of image data; a determination process for determining whether the object information acquired for each of the plurality of image data by the object information acquisition process satisfies a determination condition for object movement defined by an editable detection pattern; a notification process for issuing a notification when it is determined by the determination process that the detection pattern is satisfied; A monitoring program that causes a computer to function as a monitoring device that performs the following:

[0061] The embodiments and modifications of the present disclosure have been described above. Some of these embodiments and modifications may be combined and implemented. Also, one or more of them may be implemented partially. Note that the present disclosure is not limited to the above embodiments and modifications, and various modifications are possible as needed. [Explanation of symbols]

[0062] 100 Surveillance system, 10 Surveillance device, 11 Processor, 12 Memory, 13 Storage, 14 Communication interface, 21 Video acquisition unit, 22 Object information acquisition unit, 23 Judgment unit, 24 Measurement unit, 25 Notification unit, 26 Operation unit, 31 Setting file, 32 Detection pattern master, 33 Result file, 34 Related pattern master, 50 Camera, 60 Alarm.

Claims

1. an object information acquisition unit that acquires object information indicating the position of an object detected from each of the plurality of image data; a determination unit that determines whether the object information acquired for each of the plurality of image data by the object information acquisition unit satisfies a determination condition for object movement defined by an editable detection pattern; a notification unit that issues a notification when the determination unit determines that the detection pattern is satisfied; A monitoring device comprising:

2. the detection pattern has a determination condition defined using an editable detection time; The determination unit determines that the determination condition is met when the user has not moved continuously for the detection time or longer. The monitoring device of claim 1 .

3. the detection pattern has a determination condition defined using a moving distance that is set in an editable manner; The determination unit targets pairs of two image data sets that are adjacent in time series among the plurality of image data sets, and determines that the object did not move during a period in which the two image data sets in the target pair were acquired if the distance between positions indicated by the object information acquired for each of the two image data sets in the target pair is within the movement distance. The monitoring device according to claim 2 .

4. the detection pattern has a determination condition defined using an editable reference ratio; The determination unit determines that the object has not moved continuously for at least the detection time when it is determined that no movement has occurred for at least the reference ratio of pairs of two image data sets during the detection time. The monitoring device according to claim 3.

5. When the number of the objects detected from one image data exceeds a reference number, the determination unit stops determining whether the detection pattern is satisfied. The monitoring device of claim 1 .

6. the plurality of image data are acquired by a camera provided to photograph underwater; the object is a person, The determination unit determines that a person is drowning when the detection pattern is satisfied. The monitoring device of claim 1 .

7. The camera is attached to a lane rope of the pool and is configured to photograph underwater images of the pool, The object information acquisition unit acquires object information about a person whose upper body is at least shown. The monitoring device according to claim 6.

8. The computer acquires object information indicating the position of the detected object from each of the plurality of image data; a computer determines whether the object information acquired for each of the plurality of image data satisfies a determination condition for object movement defined by an editable detection pattern; A monitoring method in which a computer issues a notification when it is determined that the detection pattern is satisfied.

9. an object information acquisition process for acquiring object information indicating the position of an object detected from each of the plurality of image data; a determination process for determining whether the object information acquired for each of the plurality of image data by the object information acquisition process satisfies a determination condition for object movement defined by an editable detection pattern; a notification process for issuing a notification when it is determined by the determination process that the detection pattern is satisfied; A monitoring program that causes a computer to function as a monitoring device that performs the following:

Images