Monitoring device, monitoring method, and program

Abstract

In order to provide a monitoring device which makes it possible to accurately identify a moving body detected in a video, this monitoring device comprises: an acquisition unit that acquires video data composed of frames associated with angle-of-view information which includes the imaging direction and imaging magnification of a camera that images a monitoring target region; an identification unit that adjusts the coordinate representation of a rectangular region which includes a moving body extracted from the frames, and identifies the moving body in accordance with the movement of the rectangular region for which the coordinate representation has been adjusted; and an output unit that outputs identification information which includes the identification results for the identified moving body.

Description

Monitoring device, monitoring method, and program

[0001] The present disclosure relates to a monitoring device, a monitoring method, and a program.

[0002] There is a need to monitor a flying object flying in the sky. For example, by using an image recognition model, a flying object included in an image captured at a resolution of a certain level or higher can be detected. Patent Document 1 discloses a device for determining the type of an object included in an input image. The device of Patent Document 1 detects an object included in an image captured by a predetermined imaging device. The device of Patent Document 1 identifies a rectangular area including the object based on the movement vector of pixels forming the detected object and a predetermined likelihood. The device of Patent Document 1 acquires information related to the movement history of the identified rectangular area as movement history information. The device of Patent Document 1 determines the type of the object detected in the detection step based on the movement history information.

[0003] Japanese Patent No. 6678706

[0004] In a scenario of monitoring a flying object, it is necessary to widen the angle of view of a camera in order to monitor a vast airspace. When the angle of view of the camera is widened, an image of a distant flying object may not be obtained at a sufficient resolution. Further, in order not to lose sight of a flying object moving away, the angle of view must be frequently changed and tracking must be continued, and the same object included in temporally continuous images may appear as completely different pixels. In the method of Patent Document 1, an optical flow and a likelihood are calculated for each pixel constituting images of a plurality of consecutive frames, and pixels with movement are extracted based on the optical flow. Therefore, in the method of Patent Document 1, there is a possibility that the same object included in temporally continuous images may be regarded as different ones.

[0005] An object of the present disclosure is to provide a monitoring device, a monitoring method, and a program capable of accurately identifying the type of a moving object detected in a video.

[0006] A monitoring device according to one aspect of the present disclosure includes: an acquisition unit that acquires video data composed of frames associated with angle of view information including the shooting direction and shooting magnification of a camera that photographs a target area to be monitored; an identification unit that adjusts the coordinate representation of a rectangular area including a moving object extracted from the frame and identifies the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted; and an output unit that outputs identification information including the identification result of the identified moving object.

[0007] In a monitoring method according to one aspect of the present disclosure, a computer acquires video data composed of multiple frames associated with angle of view information including the shooting direction and magnification of a camera that photographs the area to be monitored, adjusts the coordinate representation of a rectangular area containing a moving object extracted from the frames, identifies the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted, and outputs identification information including the identification result of the identified moving object.

[0008] A program according to one aspect of the present disclosure causes a computer to perform the following processes: acquire video data composed of a plurality of frames associated with angle of view information including the shooting direction and magnification of a camera that photographs a monitored area; adjust the coordinate representation of a rectangular area containing a moving object extracted from the frames, identify the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted; and output identification information including the identification result of the identified moving object.

[0009] This disclosure makes it possible to provide a monitoring device, a monitoring method, and a program that can accurately identify moving objects detected in video footage.

[0010] This is a conceptual diagram showing an example of monitoring a monitored area using the monitoring device in this disclosure. This is a block diagram showing an example of the configuration of the monitoring device in this disclosure. This is a conceptual diagram showing an example of a rectangular area extracted from a frame containing a moving object detected by the monitoring device in this disclosure. This is a conceptual diagram showing an example of a rectangular area whose coordinate representation has been adjusted by the monitoring device in this disclosure. This is a conceptual diagram showing an example of the movement of a rectangular area tracked by the monitoring device in this disclosure. This is a conceptual diagram to explain an example of identification of a moving object by the monitoring device in this disclosure. This is a conceptual diagram showing an example of the display of the identification result identified by the monitoring device in this disclosure. This is a conceptual diagram showing an example of the display of the identification result identified by the monitoring device in this disclosure. This is a flowchart showing an example of the operation of the monitoring device in this disclosure. This is a flowchart showing an example of the identification process in this disclosure. This is a flowchart showing an example of the identification process in a modified version of this disclosure. This is a conceptual diagram showing an example of the display of the identification result identified by the monitoring device in a modified version of this disclosure. This is a conceptual diagram showing an example of monitoring a monitored area using the monitoring device in this disclosure. This is a block diagram showing an example of the configuration of the monitoring device in this disclosure. This is a conceptual diagram showing an example of camera control by the monitoring device in this disclosure. This is a conceptual diagram showing an example of monitoring a monitored area using the monitoring device in this disclosure. This is a conceptual diagram showing an example of camera control by the monitoring device in this disclosure. This is a conceptual diagram showing an example of monitoring a monitored area using the monitoring device in this disclosure. This is a conceptual diagram showing an example of displaying identification results identified by the monitoring device in this disclosure. This is a conceptual diagram showing an example of displaying identification results identified by the monitoring device in this disclosure. This is a conceptual diagram showing an example of displaying identification results identified by the monitoring device in this disclosure. This is a flowchart showing an example of the operation of the monitoring device in this disclosure. This is a flowchart showing an example of identification processing in this disclosure. This is a block diagram showing an example of the configuration of the monitoring device in this disclosure. This is a block diagram showing an example of the operation of the monitoring device in this disclosure. This is a block diagram showing an example of the hardware configuration that performs control and processing in this disclosure.

[0011] The embodiments for carrying out this disclosure will be described below with reference to the drawings. In this disclosure, the drawings used in the description of each embodiment are associated with one or more embodiments. Also, the elements included in each drawing may correspond to one or more embodiments. The embodiments described below have technically preferred limitations for carrying out this disclosure, but the scope of the disclosure is not limited thereto. In all the drawings used in the description of the embodiments below, the same parts are denoted by the same reference numerals unless there is a specific reason not to. In the embodiments below, repeated descriptions of similar configurations and operations may be omitted. The direction of the arrows in the drawings is an example of the flow of signals, data, etc., and does not limit the flow of signals, data, etc.

[0012] (First Embodiment) First, the monitoring device in the first embodiment will be described with reference to the drawings. In this embodiment, an example of monitoring a target area will be described. The monitoring device of this embodiment acquires images captured by a camera that photographs the target area. For example, the monitoring device of this embodiment is used for real-time analysis of images captured by a camera. The monitoring device of this embodiment may also be used for analysis of images captured in the past. The method of this embodiment can be applied not only to airspace but also to land and sea areas. For the sake of explanation, some of the drawings used in the description of the following embodiments are shown to show the type of moving object. In this embodiment, the type of moving object detected from the actual image cannot be identified from the image alone. In cases where the distance to the moving object is close or the size of the moving object is sufficiently large, it may be possible to identify the type of moving object from the actual image. Examples of such cases will be described in the modified examples.

[0013] (Configuration) Figure 1 is a conceptual diagram showing an example of monitoring a target area using the monitoring device in this disclosure. Camera 100 is positioned to have a clear view of the target area. Camera 100 captures images of the target area. The images consist of multiple frames that are continuous in time. In Figure 1, the range corresponding to the frames captured by camera 100 is shown by a rectangle. Camera 100 transmits video data, including the captured images, to the monitoring device 10. The video data includes the time of capture and field of view information for each frame that makes up the images. The field of view information includes the shooting direction and magnification of camera 100. For example, the field of view information can be configured to be acquired by a gyro sensor or the like implemented in camera 100. Using the field of view information, the relative coordinate system of the rectangular area extracted from the frame can be converted to an absolute coordinate system. In this embodiment, the absolute coordinate system is assumed to be a two-dimensional coordinate system. When considering the degrees of freedom of pan, tilt, and zoom, the assumed absolute coordinate system is a coordinate system that excludes the radius of a spherical coordinate system with the position of camera 100 as the origin. The video data transmitted to the monitoring device 10 is used to identify moving objects (flying objects) flying within the monitored area.

[0014] Camera 100 has the ability to remotely control the shooting direction and magnification. Camera 100 also has a communication function for transmitting video data. For example, camera 100 is implemented as a PTZ (Pan Tilt Zoom) camera. A PTZ camera allows for remote control of panning in the horizontal plane, tilting in the vertical plane, and zooming. The field of view of the frame captured by camera 100 is changed by adjusting the pan, tilt, and zoom. The adjustment of pan, tilt, and zoom may be performed manually by a monitor remotely operating camera 100, or it may be achieved by the camera 100's automatic tracking function. Alternatively, the adjustment of pan, tilt, and zoom may be configured to be performed by the monitoring device 10.

[0015] The monitoring device 10 acquires video data transmitted from the camera 100 via a network such as the Internet. The monitoring device 10 detects moving objects from the frames that make up the video data. The monitoring device 10 extracts a rectangular area containing the moving object. The monitoring device 10 adjusts the coordinate representation of the rectangular area containing the moving object by converting the coordinate system of the rectangular area to an absolute coordinate system using the field of view information. The monitoring device 10 identifies the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted. The method of identifying the moving object by the monitoring device 10 will be described later. The monitoring device 10 outputs identification information including the identification result of the identified moving object. The use of the identification information output from the monitoring device 10 is not limited. For example, the identification information is displayed on the screen of a terminal device used by an organization that is contracted to monitor the area under surveillance.

[0016] For example, the monitoring device 10 may be implemented in the cloud or on a server. The monitoring device 10 may also be implemented in a terminal device used by an organization that undertakes the monitoring of a target area. Alternatively, the functions of the monitoring device 10 may be mounted on a camera 100 to form an IoT (Internet of Things) device. Furthermore, the monitoring device 10 may be configured to use distance, speed, and altitude data measured using radar or the like, in addition to the video captured by the camera 100. Using data measured using radar or the like enables more accurate and robust identification of moving objects.

[0017] [Monitoring Device] Next, the details of the configuration of the monitoring device in this disclosure will be described with reference to the drawings. Figure 2 is a block diagram showing an example of the configuration of the monitoring device in this disclosure. The monitoring device 10 comprises an acquisition unit 11, an identification unit 13, and an output unit 15. The identification unit 13 has a detection unit 131, an adjustment unit 132, a tracking unit 133, an extraction unit 135, and a moving object identification unit 136.

[0018] The acquisition unit 11 acquires video data transmitted from the camera 100 via a network such as the Internet. The video data consists of multiple frames. Each of the multiple frames contains field of view information at the time the frame was captured. The field of view information includes the shooting direction and magnification of the camera 100.

[0019] The detection unit 131 detects moving objects from each of the multiple frames that make up the acquired video data. The detection unit 131 detects moving objects in accordance with changes between multiple temporally consecutive frames. For example, the detection unit 131 detects moving objects using methods such as image difference methods, motion vector analysis methods, machine learning methods, segmentation methods, and feature point extraction methods. These methods detect moving objects by accumulating motion in multiple temporally consecutive frames. These methods may be used individually or in combination.

[0020] Image subtraction methods include background subtraction and interframe subtraction. Background subtraction detects moving objects by taking the difference between a static background image and the current frame. Background subtraction can be used when the background is static. Interframe subtraction calculates the difference between consecutive frames to detect changing regions as moving objects. Interframe subtraction can also be used when the background is dynamic. Motion vector analysis methods include optical flow. Optical flow estimates the motion of a moving object by calculating the movement of pixels between consecutive frames. According to optical flow, information on the velocity and direction of the moving object can be obtained. Machine learning methods include techniques using convolutional neural networks. For example, machine learning methods include YOLO (You Only Look Once), SSD (Single Shot Detector), and Faster R-CNN (Region-based Convolutional Neural Networks). In machine learning methods, patterns of moving objects that have been trained in advance are detected. According to machine learning methods, complex backgrounds and diverse moving objects can be detected. Segmentation-based methods detect moving objects by segmenting the frame and analyzing the changes between segments. Segmentation-based methods are effective when the boundary between the background and the moving object can be clearly distinguished. Feature point extraction methods use algorithms such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features) to extract characteristic points in the image and detect moving objects by analyzing the movement of these points. Motion detection algorithms include MHI (Motion History Image) and MEI (Motion Energy Image).

[0021] Furthermore, the detection unit 131 extracts a rectangular region containing the moving object from the frame in which the moving object was detected. The detection unit 131 extracts a rectangular region for each moving object contained in the frame. The detection unit 131 assigns an identifier to each extracted rectangular region. The detection unit 131 associates rectangular regions containing the same moving object with each other. For example, the detection unit 131 may assign the same identifier to rectangular regions containing the same moving object, or assign identifiers that are consecutive in time.

[0022] Figure 3 is a conceptual diagram showing an example of a rectangular region extracted from a frame containing a moving object detected by the monitoring device in this disclosure. In the example in Figure 3, rectangular regions extracted from multiple frames constituting video footage of the same moving object are arranged in chronological order. The multiple rectangular regions are assigned identifiers such as t1, t2, t3, t4, t5, t6, t7, ... in chronological order. The multiple rectangular regions are captured at various angles of view depending on the movement of the moving object. Therefore, the angle of view information for each rectangular region is different from one another.

[0023] The adjustment unit 132 uses the field of view information for each of the multiple frames to convert the coordinate system of the rectangular area containing the detected moving object into an absolute coordinate system. For example, the absolute coordinate system is a spherical coordinate system with the camera 100 as the origin. The absolute coordinate system is not limited to a spherical coordinate system, as long as the position of the moving object can be uniquely identified with respect to the position of the camera 100.

[0024] Figure 4 is a conceptual diagram showing an example of a rectangular region whose coordinate representation has been adjusted by the monitoring device in this disclosure. In the example in Figure 4, the rectangular regions whose coordinate representations have been adjusted are arranged in chronological order. Multiple rectangular regions have been adjusted to the same field of view. Therefore, the field of view information of each rectangular region is the same.

[0025] The tracking unit 133 tracks the movement of a rectangular region transformed into an absolute coordinate system. For example, the tracking unit 133 tracks the movement of the rectangular region using methods such as the Kalman filter, extended Kalman filter, particle filter, mean shift method, and CAM Shift (Continuously Adaptive Mean Shift). The Kalman filter provides optimal estimation for linear systems, is computationally efficient, and is suitable for real-time processing. The extended Kalman filter can be applied to nonlinear problems. The particle filter is a probabilistic estimation method using the Monte Carlo method and can handle nonlinear and non-Gaussian cases. The mean shift method is a tracking method that uses the color and brightness distribution of the object, is computationally efficient, and can handle partial occlusion. CAM Shift can adapt to changes in the size of the object. For example, the tracking unit 133 tracks the movement of the rectangular region using methods such as optical flow, feature point tracking, correlation filter tracking, deep learning, and composite tracking methods. Optical flow is a method that tracks the movement of pixels in a frame and provides dense motion information. Feature point tracking is a method that uses feature points for tracking and is well-suited to handling rotation and scale changes. Examples of feature point tracking include SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF). Correlation filter tracking is a method that utilizes efficient computation in the frequency domain and is fast and highly accurate. Deep learning-based tracking is a learning-based method using convolutional neural networks and is well-suited to handling complex changes and backgrounds. The tracking unit 133 may be configured to track the movement of a rectangular region by combining multiple methods. By combining multiple methods, the advantages of each method can compensate for the disadvantages of each method.

[0026] Figure 5 is a conceptual diagram showing an example of the movement of a rectangular area tracked by the monitoring device in this disclosure. In the example in Figure 5, a solid curve is shown representing the movement of a rectangular area with adjusted coordinate representation. The solid curve is marked with an arrowhead indicating the direction of travel of the moving object. In the solid arrow in Figure 5, the difference in the field of view change from frame to frame is canceled out. In this embodiment, since the change in the field of view of the frame corresponding to the tracking of the moving object is canceled out, the original movement of the moving object can be tracked.

[0027] The extraction unit 135 extracts motion features corresponding to the movement of the tracked rectangular region. The extraction unit 135 extracts image features extracted from the rectangular region and trajectory features extracted from the trajectory corresponding to the movement of the rectangular region as motion features. Image features are features extracted from an image using a machine learning model. For example, image features are a sequence of rectangular regions in the frames that make up the video. Trajectory features include the curvature of the trajectory with respect to the center point of the rectangular region, changes in velocity, acceleration, etc. The movement of a flying object includes a flight pattern characterized by the shape and changes in the trajectory. The extraction unit 135 extracts trajectory features and image features in units of one or a small number of specific frames. For example, the extraction unit 135 extracts motion features by fusing the trajectory features and image features extracted in units of specific frames. For example, the extraction unit 135 may be configured to extract features obtained by accumulating the trajectory features and image features fusing in units of specific frames in the time direction as motion features. In other words, the extraction unit 135 extracts a feature quantity as a motion feature quantity by fusing image feature quantities and trajectory feature quantities extracted from multiple temporally consecutive frames.

[0028] For example, the extraction unit 135 may be configured to extract velocity profiles, periodic features, and optical flow statistics. A velocity profile is a profile showing the change in velocity within a rectangular area. Different types of flying objects exhibit distinctive velocity change patterns. Periodic features are obtained by analyzing the periodicity of motion using Fourier transforms. The flapping wings of birds and insects exhibit periodic motion characteristics. Optical flow statistics include average velocity, velocity variance, and the main direction of motion. By using optical flow statistics, the direction of motion and velocity distribution of a flying object can be captured. For example, the extraction unit 135 may be configured to extract spatiotemporal features, shape change features, statistical moments, motion history image features, and relative motion features. Spatiotemporal features include HOG3D (Histogram of Oriented Gradients 3 Dimension) and MoSIFT (Motion Scale-Invariant Feature Transform). Spatiotemporal features allow for the simultaneous capture of both the temporal changes and spatial distribution of a moving object's motion, enabling detailed analysis of the movement patterns of flying objects. Shape change features include changes in aspect ratio and area rate within a rectangular region. The movement of a bird's wings and the attitude changes of a drone exhibit characteristic shape changes during flight. Statistical moments include the mean, variance, skewness, and kurtosis of the motion vector within a rectangular region. Statistical moments allow for the quantification of the distribution characteristics of motion that appear in response to the movements of different types of flying objects. Motion history image features indicate the characteristics of the direction and magnitude of motion within a rectangular region. Motion history image features allow for the visual capture of the trajectory and patterns of a moving object's motion. Relative motion features indicate the characteristics of motion relative to the background and other objects. Relative motion features allow for the identification of relative motion to the background and other objects.

[0029] The mobile object identification unit 136 identifies a mobile object using the extracted motion features. For example, the mobile object identification unit 136 uses a machine learning model to identify a mobile object according to its motion features. The machine learning model is a model that has been trained to output a category included in the mobile object to be identified, in response to the input of motion features.

[0030] Figure 6 is a conceptual diagram illustrating an example of the identification of a moving object by a monitoring device in this disclosure. The moving object identification unit 136 inputs motion features extracted from the movement of a rectangular area into a machine learning model 160. The machine learning model 160 outputs the type (class) of the moving object in response to the input motion features. The moving object identification unit 136 outputs the type (class) of the moving object output from the machine learning model 160 as the identification result.

[0031] The video data used to train machine learning models is captured under various weather conditions, time of day, and season. For example, the video data is captured by surveillance cameras that film from above. Each frame that makes up the video data to be trained contains at least one moving object. Frames containing moving objects are used as training data. When training a machine learning model based on the difference between the background and the moving object, frames that do not contain moving objects are also used for training. For example, the training data is a dataset in which motion features related to the movement of regions containing moving objects detected in each frame of the video data are associated with the classes of moving objects corresponding to those movements. The motion features used to identify moving objects are set according to the type of moving object to be identified.

[0032] For example, the machine learning model is a model trained using a CNN (Convolutional Neural Network) based method. For example, the machine learning model is a model trained using a PCA (Principal Component Analysis) method. For example, the machine learning model is a model trained using a VAE (Variational Autoencoder). For example, the machine learning model is a model trained using a conditional GAN ​​(Generative Adversarial Networks) method. For example, the machine learning model may be configured to convert trajectory features and image features into features fused in units of one or a small number of frames, and then process that sequence with a time-series model. The above methods are just examples and do not limit the methods used to train a machine learning model.

[0033] The classes of mobile objects (flying objects) to be identified include unmanned aerial vehicles (drones), helicopters, aircraft, balloons, birds, etc. For example, when the object to be identified is an animal such as a bird, the detection tendency differs depending on the weather and season. Therefore, information representing the weather and season may be used in identifying the class of mobile objects. The classes of mobile objects may include objects other than those listed above. The classes of mobile objects to be identified are set according to the type of area being monitored. For example, when the area being monitored is land, the classes of mobile objects include vehicles, unmanned mobile objects, animals, people, etc. For example, when the area being monitored is water, the classes of mobile objects include ships, unmanned mobile objects on water, fish, etc.

[0034] The output unit 15 outputs identification information, including the identification result of the identified mobile object. The use of the identification information output from the output unit 15 is not limited. For example, the identification information may be displayed on the screen of a terminal device used by an organization contracted to monitor the monitored area.

[0035] Figure 7 is a conceptual diagram showing an example of the display of identification results identified by the monitoring device in this disclosure. The screen of the terminal device 180, used by the administrator responsible for monitoring the monitored area, displays information indicating the movement of a moving object detected in the monitored area. The information indicating the movement of the moving object displays a curve showing the trajectory of the moving object and a rectangular area containing the moving object. The curve is marked with an arrowhead indicating the direction of the moving object's movement. A rectangular area is displayed around the moving object. The most recent rectangular area is clearly indicated with a solid line. The administrator who views the information displayed on the screen of the terminal device 180 can intuitively grasp the movement of the moving object detected in the monitored area.

[0036] Figure 8 is a conceptual diagram showing an example of the display of identification results identified by the monitoring device in this disclosure. The screen of the terminal device 180, used by the administrator responsible for monitoring the monitored area, displays information indicating the movement of a moving object detected in the monitored area. The information indicating the movement of the moving object displays a curve showing the trajectory of the moving object and a rectangular area containing the moving object. The curve is marked with an arrowhead indicating the direction of travel of the moving object. A rectangular area is displayed around the moving object. The most recent rectangular area is clearly indicated by a solid line. The class of the identified moving object is displayed in the most recent rectangular area indicated by a solid line. In the example of Figure 8, the type of moving object detected in the monitored area is a drone. The administrator who views the information displayed on the screen of the terminal device 180 can recognize that a drone has been detected in the monitored area. The administrator can also intuitively grasp the movement of the drone detected in the monitored area. For example, the type of identified moving object may be displayed provisionally. In that case, in addition to the type of identified moving object, a numerical value representing the probability or certainty of that type may also be displayed.

[0037] Figure 9 is a conceptual diagram showing an example of the display of identification results identified by the monitoring device in this disclosure. The screen of the terminal device 180, used by the administrator responsible for monitoring the monitored area, displays information indicating the movement of a moving object detected in the monitored area. The information indicating the movement of the moving object displays a curve showing the trajectory of the moving object and a rectangular area containing the moving object. The curve is marked with an arrowhead indicating the direction of travel of the moving object. A rectangular area is displayed around the moving object. The most recent rectangular area is clearly indicated by a solid line. The class of the identified moving object is displayed in the most recent rectangular area indicated by a solid line. In the example of Figure 9, the type of moving object detected in the monitored area is a drone. In addition, the screen of the terminal device 180 displays the text information, "A drone has been detected." The administrator who views the information displayed on the screen of the terminal device 180 can recognize that a drone has been detected in the monitored area. Furthermore, the administrator can intuitively grasp the movement of the drone detected in the monitored area. For example, the type of identified moving object may be displayed provisionally. In that case, the system may be configured to display not only the type of identified moving object, but also a numerical value representing the probability or certainty that it is that type.

[0038] (Operation) Next, an example of the operation of the monitoring device in this disclosure will be described with reference to the drawings. Figure 10 is a flowchart of an example of the operation of the monitoring device in this disclosure. In the explanation of the process according to the flowchart in Figure 10, the monitoring device 10 will be the main operator. For example, the process according to the flowchart in Figure 10 is realized by a processor executing a program stored in the memory installed in a computer (not shown) on which the monitoring device 10 is implemented.

[0039] In Figure 10, first, the monitoring device 10 acquires video data including field of view information (step S11).

[0040] Next, the monitoring device 10 performs an identification process (step S12). Details of the identification process in step S12 will be described later.

[0041] Next, the monitoring device 10 outputs identification information including the identification result of the identified moving object (step S13).

[0042] If the process is to be continued (Yes in step S14), the process returns to step S11. If the process is not to be continued (No in step S14), the process along the flowchart of FIG. 10 ends. The condition for continuing the process is set arbitrarily.

[0043] 〔Identification Process〕Next, an example of the identification process by the monitoring device in the present disclosure will be described with reference to the drawings. FIG. 11 is a flowchart showing an example of the identification process in the present disclosure. In the description of the process along the flowchart of FIG. 11, the components of the identification unit 13 included in the monitoring device 10 are taken as the operation main body. The operation main body of the process along the flowchart of FIG. 11 may be the monitoring device 10 or the identification unit 13. For example, the process along the flowchart of FIG. 11 is realized by the processor executing a program stored in a memory mounted on a computer (not shown) on which the monitoring device 10 is mounted.

[0044] In FIG. 11, first, the detection unit 131 detects a moving object from each of a plurality of frames constituting the monitoring target video (step S121).

[0045] Next, the adjustment unit 132 converts the coordinate system of the rectangular area including the moving object into an absolute coordinate system using the angle-of-view information for each frame (step S122).

[0046] Next, the tracking unit 133 tracks the rectangular area converted into the absolute coordinate system (step S123).

[0047] Next, the extraction unit 135 extracts a motion feature amount according to the motion of the rectangular area converted into the absolute coordinate system (step S124).

[0048] Next, the moving object identification unit 136 identifies the moving object using the extracted motion feature amount (step S125). After step S125, the process proceeds to step S13 of FIG. 10.

[0049] (Modifications) Next, modifications of this embodiment will be described with reference to the drawings. In these modifications, moving objects that can be identified from the frames constituting the video are identified without using motion features. Below, an example of the identification process in these modifications will be described. The processes other than the identification process are the same as the processes in the flowchart of Figure 10.

[0050] Figure 12 is a flowchart showing an example of identification processing in a modified version of the present disclosure. In describing the processing according to the flowchart in Figure 12, the components of the identification unit 13 included in the monitoring device 10 are considered the main operating components. The main operating components of the processing according to the flowchart in Figure 12 may be the monitoring device 10 or the identification unit 13. For example, the processing according to the flowchart in Figure 12 is realized by a processor executing a program stored in the memory of a computer (not shown) on which the monitoring device 10 is implemented.

[0051] In Figure 12, first, the detection unit 131 detects a moving object from each of the multiple frames that make up the monitored video (step S151).

[0052] If a moving object can be identified from individual frames (Yes in step S152), the moving object identification unit 136 identifies the moving object without using motion features (step S156). For example, if the resolution corresponding to the size of the rectangular area in the frame is greater than or equal to a preset threshold, the moving object identification unit 136 identifies the moving object from the frame. For example, the moving object identification unit 136 is configured to identify the moving object using a machine learning model trained to identify moving objects from images. For example, the moving object identification unit 136 may be configured to identify the moving object based on the shape and color of the objects contained in the image.

[0053] If the moving object cannot be identified from individual frames (No in step S152), the adjustment unit 132 uses the field of view information for each frame to convert the coordinate system of the rectangular area containing the moving object to an absolute coordinate system (step S153).

[0054] Next, the tracking unit 133 tracks the rectangular region containing the moving object that has been transformed into an absolute coordinate system (step S154).

[0055] Next, the extraction unit 135 extracts motion features according to the movement of the rectangular region converted to an absolute coordinate system (step S155).

[0056] Next, the moving object identification unit 136 identifies the moving object using the extracted motion features (step S156). After step S156, the process proceeds to step S13 in Figure 10.

[0057] Figure 13 is a conceptual diagram showing an example of the display of identification results identified by a monitoring device in a modified version of the present disclosure. The screen of the terminal device 180, used by the administrator responsible for monitoring the monitored area, displays information indicating a moving object detected in the monitored area. In the example of Figure 13, the type of moving object detected in the monitored area is an airplane. The screen of the terminal device 180 also displays the text information, "An airplane has been detected." The administrator who views the information displayed on the screen of the terminal device 180 can recognize that an airplane has been detected in the monitored area. According to this modified version, the processing for identifying moving objects can be reduced when motion features do not need to be used.

[0058] As described above, the monitoring device of this embodiment comprises an acquisition unit, an identification unit, and an output unit. The identification unit includes a detection unit, an adjustment unit, a tracking unit, an extraction unit, and a moving object identification unit. The acquisition unit acquires video data composed of frames associated with angle of view information, including the shooting direction and magnification of a camera that photographs the area to be monitored. The detection unit extracts a rectangular area containing a moving object from each of the multiple frames that constitute the video data. The adjustment unit adjusts the coordinate representation of the rectangular area using the angle of view information. For example, the adjustment unit adjusts the coordinate representation of the rectangular area by converting the coordinate system of the rectangular area containing the detected moving object to an absolute coordinate system using the angle of view information for each of the multiple frames. The tracking unit tracks the movement of the rectangular area whose coordinate representation has been adjusted. The extraction unit extracts motion features from the movement of the tracked rectangular area. The moving object identification unit identifies the moving object using the extracted motion features. The output unit outputs identification information including the identification result of the identified moving object.

[0059] In this embodiment, the coordinate representation of the rectangular region containing the moving object is adjusted using the field of view information for each frame. The movement of the rectangular region with the adjusted coordinate representation accurately represents the movement of the moving object. Therefore, according to this embodiment, moving objects detected in the video can be identified with high accuracy.

[0060] In one embodiment of this system, the extraction unit extracts a feature quantity as a motion feature quantity that includes image feature quantities extracted from a rectangular region and trajectory feature quantities extracted from a trajectory corresponding to the movement of the rectangular region. For example, the extraction unit extracts a feature quantity as a motion feature quantity that is a fusion of image feature quantities and trajectory feature quantities extracted from multiple temporally consecutive frames. According to this embodiment, by using a motion feature quantity that includes image feature quantities and trajectory feature quantities, moving objects detected in the video can be identified with greater accuracy.

[0061] In one embodiment of this system, the mobile object identification unit inputs motion features extracted from the movement of a rectangular area with adjusted coordinate representations to a machine learning model that outputs a class of mobile objects in response to the input of motion features. The mobile object identification unit outputs the class of mobile objects output by the machine learning model as the mobile object identification result. According to this embodiment, the type of mobile object can be identified by inputting motion features corresponding to the movement of a rectangular area with adjusted coordinate representations to the machine learning model.

[0062] In one embodiment of this system, if a moving object can be identified from a frame, the moving object identification unit identifies the moving object from the frame. For example, if the resolution corresponding to the size of the rectangular area in the frame is greater than or equal to a preset threshold, the moving object identification unit identifies the moving object from the frame. If a moving object cannot be identified from individual frames, the moving object identification unit identifies the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted. In this embodiment, if a moving object can be identified from a frame, the moving object is identified from the frame regardless of the movement of the rectangular area whose coordinate representation has been adjusted. Therefore, according to this embodiment, the processing required when a moving object can be identified from a frame can be reduced.

[0063] In one embodiment of this design, the output unit displays information representing the class of the identified mobile object on the screen of the terminal device. According to this design, the class of the mobile object detected in the monitored area can be recognized by viewing the screen of the terminal device.

[0064] (Second Embodiment) Next, the monitoring device in the second embodiment will be described with reference to the drawings. The monitoring device in this embodiment differs from the first embodiment in that it controls the camera that photographs the area to be monitored according to the tracking status of the detected moving object.

[0065] (Configuration) Figure 14 is a conceptual diagram showing an example of monitoring a target area using the monitoring device in this disclosure. The camera 200 has the same configuration as the camera 100 in the first embodiment. The camera 200 is positioned to have a clear view of the target area. The camera 200 captures images of the target area. The images consist of multiple frames that are continuous in time. In Figure 14, rectangles indicate the ranges corresponding to the frames captured by the camera 200. The camera 200 transmits the image data, including the captured images, to the monitoring device 20. The image data includes the time of capture and field of view information for each frame that makes up the image. The field of view information includes the shooting direction and magnification of the camera 200. The image data transmitted to the monitoring device 20 is used to identify moving objects (flying objects) flying over the target area. The camera 200 also receives control signals from the monitoring device 20. The camera 200 adjusts shooting conditions such as pan, tilt, and zoom according to the control signals. The field of view of camera 200 is changed by adjusting the shooting conditions in response to the control signal.

[0066] The monitoring device 20 acquires video data transmitted from the camera 200 via a network such as the Internet. The monitoring device 20 detects moving objects from the frames that make up the video data. The monitoring device 20 extracts a rectangular area containing the moving object. The monitoring device 20 adjusts the coordinate representation of the rectangular area containing the moving object by converting the coordinate system of the rectangular area to an absolute coordinate system using the field of view information. The monitoring device 20 identifies the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted. The method of identifying the moving object by the monitoring device 20 is the same as in the first embodiment. The monitoring device 20 outputs identification information including the identification result of the identified moving object. The use of the identification information output from the monitoring device 20 is not limited. For example, the identification information is displayed on the screen of a terminal device used by an organization that is contracted to monitor the area under surveillance. The monitoring device 20 also predicts the position of the moving object at the next control timing of the camera 200 according to the movement of the rectangular area whose coordinate representation has been adjusted. The monitoring device 20 transmits a control signal to the camera 200 that includes shooting conditions for photographing the predicted position.

[0067] For example, the monitoring device 20 may be implemented in the cloud or on a server. The monitoring device 20 may also be implemented in a terminal device used by an organization that is contracted to monitor the area to be monitored. Alternatively, the functions of the monitoring device 20 may be incorporated into a camera 200 to form an IoT (Internet of Things) device.

[0068] [Monitoring Device] Next, the details of the configuration of the monitoring device in this disclosure will be described with reference to the drawings. Figure 15 is a block diagram showing an example of the configuration of the monitoring device in this disclosure. The monitoring device 20 comprises an acquisition unit 21, an identification unit 23, an output unit 25, and a transmission unit 27. The identification unit 23 has a detection unit 231, an adjustment unit 232, a tracking unit 233, an extraction unit 235, a moving object identification unit 236, and a prediction unit 237. The identification unit 23 differs from the identification unit 13 of the first embodiment in that it has a prediction unit 237.

[0069] The acquisition unit 21 has the same configuration as the acquisition unit 11 of the first embodiment. The acquisition unit 21 acquires video data transmitted from the camera 200 via a network such as the Internet. The video data consists of multiple frames. Each of the multiple frames contains field of view information at the time the frame was captured. The field of view information includes the shooting direction and shooting magnification of the camera 200.

[0070] The detection unit 231 has the same configuration as the detection unit 131 of the first embodiment. The detection unit 231 detects moving objects from each of the multiple frames that make up the acquired video data. The detection unit 231 detects moving objects in accordance with changes between multiple temporally consecutive frames. For example, the detection unit 231 detects moving objects using methods such as image difference method, motion vector analysis method, machine learning method, segmentation method, and feature point extraction method. These methods detect moving objects by accumulating motion in multiple temporally consecutive frames. These methods may be used individually or in combination.

[0071] Furthermore, the detection unit 231 extracts a rectangular region containing the moving object from the frame in which the moving object was detected. The detection unit 231 extracts a rectangular region for each moving object contained in the frame. The detection unit 231 assigns an identifier to each extracted rectangular region. The detection unit 231 associates rectangular regions containing the same moving object with each other. For example, the detection unit 231 may assign the same identifier to rectangular regions containing the same moving object, or assign identifiers that are consecutive in time.

[0072] The adjustment unit 232 has the same configuration as the adjustment unit 132 of the first embodiment. The adjustment unit 232 uses the field of view information for each of the multiple frames to convert the coordinate system of the rectangular area containing the extracted moving object into an absolute coordinate system. For example, the absolute coordinate system is a geocentric Cartesian coordinate system (World Geodetic System) with the Earth's center of mass as the origin. The absolute coordinate system is not limited to the World Geodetic System as long as the position of the moving object can be uniquely identified with respect to the Earth.

[0073] The tracking unit 233 has the same configuration as the tracking unit 133 of the first embodiment. The tracking unit 233 tracks the movement of a rectangular region converted to an absolute coordinate system. For example, the tracking unit 233 tracks the movement of the rectangular region using methods such as Kalman filters, extended Kalman filters, particle filters, mean shift methods, and CAM Shift (Continuously Adaptive Mean Shift). For example, the tracking unit 233 tracks the movement of the rectangular region using methods such as optical flow, feature point tracking, correlation filter tracking, deep learning, and composite tracking methods. The tracking unit 233 may be configured to track the movement of the rectangular region by combining multiple methods.

[0074] The extraction unit 235 has the same configuration as the extraction unit 135 of the first embodiment. The extraction unit 235 extracts motion features corresponding to the movement of the tracked rectangular region. For example, the extraction unit 235 extracts trajectory features, velocity profiles, periodicity features, and optical flow statistics. For example, the extraction unit 235 extracts spatiotemporal features, shape change features, statistical moments, motion history image features, and relative motion features.

[0075] The mobile object identification unit 236 has the same configuration as the mobile object identification unit 136 of the first embodiment. The mobile object identification unit 236 identifies a mobile object using extracted motion features. For example, the mobile object identification unit 236 identifies a mobile object according to the motion features using a machine learning model. The machine learning model is a model that has been trained to output a category included in the mobile object to be identified, in response to the input of motion features.

[0076] The prediction unit 237 acquires time-series data of absolute coordinates indicating the position of the moving object from the tracking unit 233. Using the time-series data of absolute coordinates indicating the position of the moving object, the prediction unit 237 predicts the position of the moving object at the next control timing of the camera 200. The prediction unit 237 generates a control signal for capturing the position of the moving object at the next control timing of the camera 200. The prediction unit 237 may also be configured to generate a control signal for setting the shooting conditions of the camera 200 during a predetermined time period.

[0077] For example, the prediction unit 237 predicts the position of a moving object using a Kalman filter, linear regression, polynomial fitting, particle filter, LSTM (Long Short-Term Memory), or a physical model. When using a Kalman filter, the prediction unit 237 predicts the next position from the past position and velocity of the moving object. When using linear regression, the prediction unit 237 creates a linear model from the past position data of the moving object. When using polynomial fitting, the prediction unit 237 approximates the past trajectory of the moving object with a polynomial. When using a particle filter, the prediction unit 237 processes a number of hypotheses (particles) sequentially and predicts the next position of the moving object from their statistics. When using LSTM (Long Short-Term Memory), the prediction unit 237 predicts the next position of the moving object by learning the past position data of the moving object and recognizing time-series patterns. When using a physical model, the prediction unit 237 predicts the next position of the moving object using equations of motion. The method used by the prediction unit 237 may be configured to be selected according to the characteristics of the motion of the identified moving object. Furthermore, the prediction unit 237 may be configured to predict the position of the moving object by combining multiple methods.

[0078] The output unit 25 outputs identification information, including the identification result of the identified mobile object. The use of the identification information output from the output unit 25 is not limited. For example, the identification information may be displayed on the screen of a terminal device used by an organization contracted to monitor the monitored area.

[0079] The transmitting unit 27 transmits the control signal generated by the prediction unit 237 to the camera 200. The control signal transmitted from the transmitting unit 27 is received by the camera 200.

[0080] Figure 16 is a conceptual diagram showing an example of camera control by the monitoring device in this disclosure. In Figure 16, the range corresponding to the frame captured by the camera 200 at shooting timing t1 is shown by a rectangle (solid line). Also in Figure 16, the range including the predicted position of the moving object at the next shooting timing t2, as predicted by the monitoring device 20, is shown by a rectangle (dashed line). The camera 200 receives a control signal transmitted from the monitoring device 20. The control signal includes the shooting direction and magnification of the camera 200 at shooting timing t2. The camera 200 sets the shooting direction and magnification according to the control signal and takes a picture of the sky at shooting timing t2.

[0081] Figure 17 is a conceptual diagram showing an example of monitoring a target area using the monitoring device described in this disclosure. The shooting direction and magnification of the camera 200 are adjusted by control signals transmitted from the monitoring device 20. In Figure 17, the range corresponding to the frame captured by the camera 200 at shooting timing t2 is shown by a rectangle (solid line). The camera 200 transmits video data, including the captured video, to the monitoring device 20. The video data includes the shooting time and field of view information for each frame that makes up the video. The field of view information includes the shooting direction and magnification of the camera 200. The video data transmitted to the monitoring device 20 is used to identify moving objects (flying objects) flying in the target area.

[0082] Figure 18 is a conceptual diagram showing an example of camera control by a monitoring device in this disclosure. In Figure 18, the range corresponding to the frame captured by the camera 200 at shooting timing t1 is shown by the monitoring device 20 as a rectangle (dashed line). Also in Figure 18, the range corresponding to the frame captured by the camera 200 at shooting timing t2 is shown as a rectangle (solid line). Furthermore, in Figure 18, the range including the predicted position of the moving object at the next shooting timing t3, as predicted by the monitoring device 20, is shown as a rectangle (dashed line). The camera 200 receives a control signal transmitted from the monitoring device 20. The control signal includes the shooting direction and magnification of the camera 200 at shooting timing t3. The camera 200 sets the shooting direction and magnification according to the control signal and takes a picture of the sky at shooting timing t3.

[0083] Figure 19 is a conceptual diagram showing an example of monitoring a target area using the monitoring device described herein. The shooting direction and magnification of the camera 200 are adjusted by control signals transmitted from the monitoring device 20. In Figure 19, the range corresponding to the frame captured by the camera 200 at shooting timing t3 is shown by a rectangle (solid line). The camera 200 transmits video data, including the captured video, to the monitoring device 20. The video data includes the shooting time and field of view information for each frame that makes up the video. The field of view information includes the shooting direction and magnification of the camera 200. The video data transmitted to the monitoring device 20 is used to identify moving objects (flying objects) flying over the target area.

[0084] Figure 20 is a conceptual diagram showing an example of the display of identification results identified by the monitoring device in this disclosure. The screen of the terminal device 280, used by the administrator responsible for monitoring the monitored area, displays information indicating the movement of a moving object detected in the monitored area. The information indicating the movement of the moving object displays a curve showing the trajectory of the moving object and a rectangular area containing the moving object. The curve is marked with an arrowhead indicating the direction of travel of the moving object. A rectangular area is displayed around the moving object. The most recent rectangular area is clearly indicated by a solid line. The class of the identified moving object is displayed in the most recent rectangular area indicated by a solid line. In the example of Figure 20, the type of moving object detected in the monitored area is a drone. The screen of the terminal device 280 also displays a rectangular area including the predicted position of the moving object. The administrator who views the information displayed on the screen of the terminal device 280 can recognize that a drone has been detected in the monitored area. The administrator can also intuitively grasp the movement and predicted position of the drone detected in the monitored area.

[0085] Figure 21 is a conceptual diagram showing an example of the display of identification results identified by the monitoring device in this disclosure. The screen of the terminal device 280, used by the administrator responsible for monitoring the monitored area, displays information indicating the movement of a moving object detected in the monitored area. The information indicating the movement of the moving object displays a curve showing the trajectory of the moving object and a rectangular area containing the moving object. The curve is marked with an arrowhead indicating the direction of travel of the moving object. A rectangular area is displayed around the moving object. The most recent rectangular area is clearly indicated by a solid line. The class of the identified moving object is displayed in the most recent rectangular area clearly indicated by a solid line. In the example of Figure 21, the type of moving object detected in the monitored area is a drone. The screen of the terminal device 280 displays a rectangular area including the predicted position of the moving object. The administrator who views the information displayed on the screen of the terminal device 280 can recognize that a drone has been detected in the monitored area. The administrator can also intuitively grasp the movement and predicted position of the drone detected in the monitored area.

[0086] In Figure 21, the screen of the terminal device 280 displays the text message, "A suspicious drone has been detected." The screen of the terminal device 280 also displays the text message, "Do you want to transmit jamming signals at the next predicted location?" Furthermore, the screen of the terminal device 280 displays a button for executing the action contained in the text message. In other words, the screen of the terminal device 280 displays a component that accepts the execution of an action against the moving object at the next control timing of the camera 200. For example, when the button is clicked, the action contained in the text message is executed. An administrator who views the information displayed on the screen of the terminal device 280 can recognize that a suspicious drone has been detected in the monitored area. The administrator can also recognize that it is possible to transmit jamming signals against the suspicious drone. The administrator decides whether or not to transmit jamming signals against the suspicious drone and clicks the button if necessary. With this configuration, it is possible to prevent the suspicious drone from entering the managed area.

[0087] Figure 22 is a conceptual diagram showing an example of the display of identification results identified by the monitoring device in this disclosure. The screen of the terminal device 280, used by the administrator responsible for monitoring the monitored area, displays information indicating the movement of a moving object detected in the monitored area. The information indicating the movement of the moving object displays a curve showing the trajectory of the moving object and a rectangular area containing the moving object. The curve is marked with an arrowhead indicating the direction of travel of the moving object. A rectangular area is displayed around the moving object. The most recent rectangular area is clearly indicated by a solid line. The class of the identified moving object is displayed in the most recent rectangular area indicated by a solid line. In the example in Figure 22, the type of moving object detected in the monitored area is a bird. The screen of the terminal device 280 displays a rectangular area including the predicted position of the moving object. The administrator who views the information displayed on the screen of the terminal device 280 can recognize that a bird has been detected in the monitored area. The administrator can also intuitively grasp the movement and predicted position of the bird detected in the monitored area.

[0088] In Figure 22, the screen of the terminal device 280 displays the text information, "Bird detected." The screen of the terminal device 280 also displays the text information, "Should we use sound to deter the bird at the next predicted location?" Furthermore, the screen of the terminal device 280 displays a button for executing the processing of the content contained in the text information. For example, when the button is clicked, the processing of the content contained in the text information is executed. An administrator who views the information displayed on the screen of the terminal device 280 can recognize that a bird has been detected in the monitored area. The administrator can also recognize that a deterrent sound will be emitted towards the bird. The administrator decides whether or not to emit a deterrent sound towards the bird and clicks the button as necessary. With this configuration, birds can be driven away from the managed area.

[0089] (Operation) Next, an example of the operation of the monitoring device in this disclosure will be described with reference to the drawings. Figure 23 is a flowchart of an example of the operation of the monitoring device in this disclosure. In the explanation of the process according to the flowchart in Figure 23, the monitoring device 20 will be the main operator. For example, the process according to the flowchart in Figure 23 is realized by a processor executing a program stored in the memory installed in a computer (not shown) on which the monitoring device 20 is implemented.

[0090] In Figure 23, first, the monitoring device 20 acquires video data including field of view information (step S21).

[0091] Next, the monitoring device 20 performs identification processing / control processing (step S22). Details of the identification processing / control processing in step S22 will be described later.

[0092] Next, the monitoring device 20 outputs a control signal to the camera 200 for taking a picture of the predicted position (step S23).

[0093] Next, the monitoring device 20 outputs identification information including the identification result of the identified moving object (step S24).

[0094] If processing is to continue (Yes in step S25), the process returns to step S21. If processing is not to continue (No in step S25), the process according to the flowchart in Figure 23 is terminated. The conditions for continuing processing can be set arbitrarily.

[0095] [Identification Processing] Next, an example of identification processing by the monitoring device in this disclosure will be described with reference to the drawings. Figure 24 is a flowchart of an example of identification processing in this disclosure. In the explanation of the processing according to the flowchart in Figure 24, the components of the identification unit 23 included in the monitoring device 20 will be considered the main operating entities. The main operating entities of the processing according to the flowchart in Figure 24 may be the monitoring device 20 or the identification unit 23. For example, the processing according to the flowchart in Figure 24 is realized by a processor executing a program stored in the memory installed in a computer (not shown) on which the monitoring device 20 is implemented.

[0096] In Figure 24, first, the detection unit 231 detects a moving object from each of the multiple frames that make up the monitored video (step S221).

[0097] Next, the adjustment unit 232 uses the field of view information for each frame to convert the coordinate system of the rectangular area including the moving object into an absolute coordinate system (step S222).

[0098] Next, the tracking unit 233 tracks the rectangular region converted to an absolute coordinate system (step S223). After step S223, the process branches into steps S224 and S226.

[0099] Following step S223, the prediction unit 237 predicts the position of the moving object at the next control timing according to the movement of the rectangular area (step S224).

[0100] Next, the prediction unit 237 generates a control signal for capturing the predicted position (step S225). After step S225, the process proceeds to step S23 in the flowchart of Figure 23.

[0101] Following step S223, the extraction unit 235 extracts motion features according to the movement of the rectangular region converted to an absolute coordinate system (step S226).

[0102] Next, the mobile object identification unit 236 identifies the mobile object using the extracted motion features (step S227). Following step S227, the process proceeds to step S24 in the flowchart of Figure 23.

[0103] The processes in steps S224 to S225 and steps S226 to S227 may be executed in parallel or sequentially. Alternatively, the processes in steps S224 to S225 and steps S226 to S227 may be configured so that only one of them is executed.

[0104] As described above, the monitoring device of this embodiment comprises an acquisition unit, an identification unit, and an output unit. The identification unit includes a detection unit, an adjustment unit, a tracking unit, an extraction unit, a moving object identification unit, and a prediction unit. The acquisition unit acquires video data composed of frames associated with angle of view information, including the shooting direction and magnification of a camera that photographs the area to be monitored. The detection unit extracts a rectangular area containing a moving object from each of the multiple frames that constitute the video data. The adjustment unit adjusts the coordinate representation of the rectangular area using the angle of view information. For example, the adjustment unit adjusts the coordinate representation of the rectangular area by converting the coordinate system of the rectangular area containing the detected moving object to an absolute coordinate system using the angle of view information for each of the multiple frames. The tracking unit tracks the movement of the rectangular area whose coordinate representation has been adjusted. The extraction unit extracts motion features from the movement of the tracked rectangular area. The moving object identification unit identifies the moving object using the extracted motion features. The output unit outputs identification information including the identification result of the identified moving object. The prediction unit predicts the position of the moving object at the next control timing of the camera using time-series data of the absolute coordinates of the moving object. The prediction unit generates a control signal to capture the predicted location of the moving object. The transmission unit transmits the generated control signal to the camera.

[0105] In this embodiment, the coordinate representation of a rectangular area containing a moving object is adjusted using the field of view information for each frame. The movement of the rectangular area with adjusted coordinate representation accurately represents the movement of the moving object. Therefore, according to this embodiment, moving objects detected in the video can be identified with high accuracy. Furthermore, according to this embodiment, the camera can be controlled to capture images at a predicted position using time-series data of the absolute coordinates of the moving object.

[0106] In one embodiment of this system, the output unit displays information representing the class of the identified moving object on the terminal device's screen. According to this embodiment, the class of the moving object detected in the monitored area can be recognized by viewing the terminal device's screen. Furthermore, the output unit displays a component on the terminal device's screen that accepts the execution of an action against the moving object at the next camera control timing. According to this embodiment, an action can be taken against the moving object detected in the monitored area.

[0107] (Third Embodiment) Next, the monitoring device in the third embodiment will be described with reference to the drawings. The monitoring device in this embodiment has a simplified configuration compared to the monitoring devices in the first and second embodiments. For example, the functions of the components of the monitoring device in this embodiment are realized by the functions of the components of the monitoring devices in the first and second embodiments.

[0108] (Configuration) Figure 25 is a block diagram showing an example of the configuration of a monitoring device in this disclosure. The monitoring device 30 comprises an acquisition unit 31, an identification unit 33, and an output unit 35. The acquisition unit 31 acquires video data composed of frames to which angle of view information, including the shooting direction and shooting magnification of a camera that photographs the area to be monitored, is associated. The identification unit 33 adjusts the coordinate representation of a rectangular area containing a moving object extracted from the frame. The identification unit 33 identifies the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted. The output unit 35 outputs identification information including the identification result of the identified moving object.

[0109] (Operation) Figure 26 is a flowchart showing an example of the operation of the monitoring device in this disclosure. In describing the process according to the flowchart in Figure 26, the monitoring device 30 will be the main operator. For example, the process according to the flowchart in Figure 26 is realized by a processor executing a program stored in the memory of a computer (not shown) on which the monitoring device 30 is implemented.

[0110] In Figure 26, first, the acquisition unit 31 acquires video data consisting of frames associated with angle of view information, including the shooting direction and magnification of the camera that photographs the area to be monitored (step S31).

[0111] The identification unit 33 adjusts the coordinate representation of the rectangular region containing the moving object extracted from the frame (step S32).

[0112] The identification unit 33 identifies the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted (step S33).

[0113] The output unit 35 outputs identification information including the identification result of the identified moving object (step S34).

[0114] In this embodiment, the coordinate representation of the rectangular region containing the moving object is adjusted using the field of view information for each frame. The movement of the rectangular region with the adjusted coordinate representation accurately represents the movement of the moving object. Therefore, according to this embodiment, moving objects detected in the video can be identified with high accuracy.

[0115] (Hardware) Next, the hardware configuration for performing the control and processing described in this disclosure will be explained with reference to the drawings. Figure 27 is a block diagram showing an example of a hardware configuration for performing the control and processing described in this disclosure. Here, an information processing device 90 (computer) is shown as an example of a hardware configuration. The information processing device in Figure 27 is an example configuration for performing the control and processing described in this disclosure and does not limit the scope of this disclosure.

[0116] As shown in Figure 27, the information processing device 90 includes a processor 91, memory 92, auxiliary storage device 93, input / output interface 95, and communication interface 96. In Figure 27, interface is abbreviated as I / F (Interface). The information processing device 90 may include at least one or more of the processor 91, memory 92, auxiliary storage device 93, input / output interface 95, and communication interface 96. The processor 91, memory 92, auxiliary storage device 93, input / output interface 95, and communication interface 96 are connected to each other via a bus 98 so that they can communicate data. In addition, the processor 91, memory 92, auxiliary storage device 93, and input / output interface 95 are connected to a network such as the Internet or an intranet via the communication interface 96.

[0117] The processor 91 loads a program (instruction) stored in an auxiliary storage device 93 or the like into memory 92. For example, the program is a software program for executing the control and processing described in this disclosure. The processor 91 executes the program loaded into memory 92. By executing the program, the processor 91 performs the control and processing described in this disclosure. The processor 91 may be composed of a single piece of hardware or of multiple pieces of hardware.

[0118] Memory 92 is a storage device having an area where a program is loaded. The processor 91 loads the program stored in the auxiliary storage device 93 or the like into memory 92. Memory 92 can be implemented using volatile memory such as DRAM (Dynamic Random Access Memory). Alternatively, non-volatile memory such as MRAM (Magnetoresistive Random Access Memory) may be used as memory 92. Memory 92 may be composed of a single piece of hardware or multiple pieces of hardware.

[0119] The auxiliary storage device 93 stores various data, such as programs. For example, the auxiliary storage device 93 can be implemented by a local disk such as a hard disk or flash memory. The auxiliary storage device 93 may be configured by a single piece of hardware or by multiple pieces of hardware. The auxiliary storage device 93 may also be configured as external hardware. It is also possible to configure the system to store various data in memory 92 and omit the auxiliary storage device 93.

[0120] The input / output interface 95 is an interface for connecting the information processing device 90 to peripheral devices based on standards and specifications. The communication interface 96 is an interface for connecting to external systems and devices via a network such as the Internet or an intranet, based on standards and specifications. The input / output interface 95 may be composed of a single piece of hardware or multiple pieces of hardware. The input / output interface 95 and the communication interface 96 may be common as interfaces for connecting to external devices.

[0121] The information processing device 90 may be connected to input devices such as a keyboard, mouse, or touch panel, as needed. These input devices are used to input information and settings. When a touch panel is used as an input device, the screen with touch panel functionality becomes the interface. The processor 91 and the input devices are connected via an input / output interface 95.

[0122] The information processing device 90 may be equipped with a display device for displaying information. If a display device is provided, the information processing device 90 is equipped with a display control device (not shown) for controlling the display of the display device. The information processing device 90 and the display device are connected via an input / output interface 95.

[0123] The information processing device 90 may be equipped with a drive device. The drive device mediates between the processor 91 and the recording medium (program recording medium) by reading data and programs stored on the recording medium and writing the processing results of the information processing device 90 to the recording medium. The information processing device 90 and the drive device are connected via an input / output interface 95.

[0124] The above is an example of a hardware configuration that enables the control and processing described in this disclosure. The hardware configuration in Figure 27 is an example of a hardware configuration that executes the control and processing described in this disclosure, and does not limit the scope of this disclosure. Programs that cause a computer to execute the control and processing described in this disclosure are also included in the scope of this disclosure.

[0125] A program recording medium that stores a program for performing the processing in this embodiment is also included in the scope of the present invention. For example, the program recording medium is a computer-readable, non-transient recording medium. The recording medium can be implemented as an optical recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc). The recording medium may also be implemented as a semiconductor recording medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) card. Furthermore, the recording medium may be implemented as a magnetic recording medium such as a flexible disk, or other recording media.

[0126] The components in this disclosure may be combined in any way. The components in this disclosure may be implemented by software. The components in this disclosure may be implemented by circuitry.

[0127] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0128] Some or all of the above embodiments may also be described as follows, but are not limited to the following. In the following appendices, the dependent items of each category may also be dependent on other categories. The descriptions included in the following appendices have significance as a basis for amendment. (Appendix 1) A monitoring device comprising: an acquisition unit that acquires video data composed of frames to which angle of view information including the shooting direction and shooting magnification of a camera that photographs a monitored area is associated; an identification unit that adjusts the coordinate representation of a rectangular area including a moving object extracted from the frame and identifies the type of the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted; and an output unit that outputs identification information including the identification result of the identified moving object. (Note 2) The monitoring device according to Note 1, wherein the identification unit comprises: a detection unit that extracts the rectangular region including the moving object from each of the plurality of frames constituting the video data; an adjustment unit that adjusts the coordinate representation of the rectangular region by converting the coordinate system of the detected rectangular region including the moving object to an absolute coordinate system using the angle of view information for each of the plurality of frames; a tracking unit that tracks the movement of the rectangular region whose coordinate representation has been adjusted; an extraction unit that extracts motion features from the tracked movement of the rectangular region; and a moving object identification unit that identifies the moving object using the extracted motion features. (Note 3) The monitoring device according to Note 2, wherein the extraction unit extracts a feature amount including image features extracted from the rectangular region and trajectory features extracted from the trajectory corresponding to the movement of the rectangular region as the motion features. (Note 4) The monitoring device according to Note 3, wherein the extraction unit extracts a feature amount obtained by fusing the image features and trajectory features extracted from a plurality of temporally consecutive frames as the motion features. (Note 5) The monitoring device according to Note 2, wherein the moving object identification unit inputs the motion features extracted from the movement of the rectangular region whose coordinate representation has been adjusted to a machine learning model that outputs the class of the moving object in response to the input of the motion features, and outputs the class of the moving object output from the machine learning model as the identification result of the moving object.(Note 6) The monitoring device according to Note 2, wherein the moving object identification unit identifies the moving object from the frame if the moving object can be identified from the frame, and identifies the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted if the moving object cannot be identified from the individual frames. (Note 7) The monitoring device according to Note 3, comprising: a prediction unit that predicts the position of the moving object at the next control timing of the camera using time-series data of the absolute coordinates of the moving object and generates a control signal for photographing the predicted position of the moving object; and a transmission unit that transmits the generated control signal to the camera. (Note 8) The monitoring device according to any one of Notes 1 to 7, wherein the output unit displays information representing the class of the identified moving object on the screen of a terminal device. (Note 9) A monitoring method comprising: a computer acquiring video data composed of multiple frames associated with angle of view information including the shooting direction and magnification of a camera that photographs a monitored area; adjusting the coordinate representation of a rectangular area containing a moving object extracted from the frames; identifying the type of the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted; and outputting identification information including the identification result of the identified moving object. (Note 10) A program that causes a computer to perform the following processes: acquiring video data composed of multiple frames associated with angle of view information including the shooting direction and magnification of a camera that photographs a monitored area; adjusting the coordinate representation of a rectangular area containing a moving object extracted from the frames; identifying the type of the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted; and outputting identification information including the identification result of the identified moving object. Furthermore, some or all of the configurations described in Notes 2 to 8, which are dependent on Note 1 above, may also be dependent on Notes 9 and 10 in the same dependent relationship as Notes 2 to 8. Furthermore, not limited to Appendix 1, Appendix 9, and Appendix 10, within the scope of the embodiments described above, some or all of the configurations described in the appendices can be made subordinate to various hardware, software, various recording means for recording software, or systems.This application claims priority based on Japanese Patent Application No. 2024-184117, filed on 18 October 2024, and incorporates all of its disclosures herein.

[0129] 10, 20, 30 Monitoring device 11, 21, 31 Acquisition unit 13, 23, 33 Identification unit 15, 25, 35 Output unit 27 Transmission unit 100, 200 Camera 131, 231 Detection unit 132, 232 Adjustment unit 133, 233 Tracking unit 135, 235 Extraction unit 136, 236 Mobile object identification unit 160 Machine learning model 180, 280 Terminal device 237 Prediction unit

Claims

1. A monitoring device comprising: an acquisition unit that acquires video data composed of frames associated with angle of view information including the shooting direction and shooting magnification of a camera that photographs a monitored area; an identification unit that adjusts the coordinate representation of a rectangular area including a moving object extracted from the frame and identifies the type of the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted; and an output unit that outputs identification information including the identification result of the identified moving object.

2. The monitoring device according to claim 1, wherein the identification unit comprises: a detection unit that extracts the rectangular region including the moving object from each of the plurality of frames constituting the video data; an adjustment unit that adjusts the coordinate representation of the rectangular region by converting the coordinate system of the detected rectangular region including the moving object to an absolute coordinate system using the angle of view information for each of the plurality of frames; a tracking unit that tracks the movement of the rectangular region whose coordinate representation has been adjusted; an extraction unit that extracts motion features from the tracked movement of the rectangular region; and a moving object identification unit that identifies the moving object using the extracted motion features.

3. The monitoring device according to claim 2, wherein the extraction unit extracts a feature amount including an image feature amount extracted from the rectangular region and a trajectory feature amount extracted from the trajectory corresponding to the movement of the rectangular region, as the motion feature amount.

4. The monitoring device according to claim 3, wherein the extraction unit extracts a feature quantity obtained by fusing the image feature quantity and the trajectory feature quantity extracted from a plurality of temporally consecutive frames as the motion feature quantity.

5. The monitoring device according to claim 2, wherein the moving object identification unit inputs the motion features extracted from the movement of the rectangular region whose coordinate representation has been adjusted to a machine learning model that outputs a class of the moving object in response to the input of the motion features, and outputs the class of the moving object output from the machine learning model as the identification result of the moving object.

6. The monitoring device according to claim 2, wherein the moving body identification unit identifies the moving body from the frame if the moving body can be identified from the frame, and identifies the moving body according to the movement of the rectangular region whose coordinate representation has been adjusted if the moving body cannot be identified from the individual frames.

7. The monitoring device according to claim 3, comprising: a prediction unit that predicts the position of the moving object at the next control timing of the camera using time-series data of the absolute coordinates of the moving object and generates a control signal for photographing the predicted position of the moving object; and a transmission unit that transmits the generated control signal to the camera.

8. The monitoring device according to any one of claims 1 to 7, wherein the output unit displays information representing the class of the identified mobile body on the screen of a terminal device.

9. A monitoring method comprising: a computer acquiring video data composed of multiple frames associated with angle of view information including the shooting direction and magnification of a camera that photographs a monitored area; adjusting the coordinate representation of a rectangular area containing a moving object extracted from the frames; identifying the type of the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted; and outputting identification information including the identification result of the identified moving object.

10. A program that causes a computer to perform the following processes: acquiring video data composed of multiple frames associated with angle of view information including the shooting direction and magnification of a camera that photographs a monitored area; adjusting the coordinate representation of a rectangular area containing a moving object extracted from the frames, identifying the type of the moving object according to the movement of the rectangular area whose coordinate representation has been adjusted; and outputting identification information including the identification result of the identified moving object.

Images