Video monitoring device, video monitoring method, and program

Abstract

To a video monitoring device in which restrictions on an area that can be monitored are small and which suppresses the occurrence of such a situation that a monitoring object is missed.SOLUTION: A video monitoring device 10 comprises: a data acquisition unit 11 which acquires image data of a captured image from each camera; a monitoring object detection unit 12 which detects a monitoring object from the image data; a coordinate calculation unit 13 which calculates coordinates in a real space of the monitoring object on the basis of information about the detected monitoring object and information about a camera that images the monitoring object; a movement destination prediction unit 14 which obtains the time displacement of the coordinates and predicts coordinates of the movement destination of the monitoring object on the basis of the obtained time displacement; and a camera control unit 15 which controls all of or partial visual axis direction of the camera on the basis of the predicted coordinates of the movement destination.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a video monitoring device and a video monitoring method for monitoring an object from a camera image, and further to a program for implementing these. [Background technology]

[0002] Conventionally, surveillance systems have been proposed that use images from cameras to monitor suspicious individuals, etc. In addition, surveillance systems use multiple surveillance cameras to enable wide-area surveillance (see Patent Documents 1 and 2).

[0003] Specifically, Patent Document 1 discloses a surveillance system that uses multiple surveillance cameras to track a surveillance target. The surveillance system disclosed in Patent Document 1 is composed of multiple surveillance cameras, a camera control device for each camera, and a control terminal for controlling the entire system.

[0004] In the surveillance system disclosed in Patent Document 1, one of multiple surveillance cameras is set as a master camera. An operator sends commands to the master camera's control device via a control terminal, directly controlling the master camera to capture images of the monitored object. The master camera's control device then measures the position of the monitored object and transmits position information indicating the measured position to the control devices of the other cameras. Upon receiving the position information, the control devices of the other cameras control the attitude of the corresponding cameras based on the position information to capture images of the monitored object.

[0005] According to the surveillance system disclosed in Patent Document 1, when an operator manually controls a master camera, other surveillance cameras can track the subject of surveillance through automatic control, making it possible to monitor a wide area.

[0006] Furthermore, Patent Document 2 discloses a monitoring system that automatically tracks a monitoring target using multiple monitoring cameras. The monitoring system disclosed in Patent Document 2 is composed of multiple monitoring cameras and a tracking control device.

[0007] In the surveillance system disclosed in Patent Document 2, each surveillance camera is positioned so that part of its shooting range overlaps with the shooting range of another adjacent surveillance camera. When a user specifies a monitoring target on an image from one of the surveillance cameras displayed on a display device, the tracking control device identifies an image from another surveillance camera that includes the specified monitoring target and switches the image displayed on the display device to the identified image. As a result, the specified monitoring target is tracked, and its image is continuously displayed on the display device. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-025003 [Patent Document 2] Patent Publication No. 2021-029028 Summary of the Invention [Problem to be solved by the invention]

[0009] The monitoring systems disclosed in the above-mentioned Patent Documents 1 and 2 are thought to enable wide-area monitoring, but these monitoring systems have the following problems.

[0010] First, in the monitoring system disclosed in Patent Document 1, an operator needs to manually control the master camera to track the monitoring target. Therefore, the monitoring system disclosed in Patent Document 1 has a problem in that if the operator loses sight of the monitoring target, it becomes difficult to track the monitoring target.

[0011] Furthermore, the surveillance system disclosed in Patent Document 2 has the problem that the shooting ranges of the multiple surveillance cameras must overlap with each other, which limits the areas that can be monitored to areas with few obstacles, such as airport runways.

[0012] An example of an object of the present invention is to provide a video monitoring device, a video monitoring method, and a program that have few restrictions on the area that can be monitored and that can reduce the occurrence of situations where the target of monitoring is lost. [Means for solving the problem]

[0013] In order to achieve the above object, a video monitoring device according to one aspect of the present invention comprises: a data acquisition unit that acquires image data of captured images from each of the plurality of cameras; a monitoring target detection unit that detects a monitoring target from the acquired image data; a coordinate calculation unit that calculates coordinates of the monitoring target in real space based on information about the monitoring target detected and information about a camera that captured the monitoring target; a destination prediction unit that calculates a time displacement of the calculated coordinates and predicts the coordinates of a destination of the monitoring target based on the calculated time displacement; a camera control unit that controls the visual axis directions of all or some of the plurality of cameras based on the predicted coordinates of the movement destination; The present invention is characterized by the following features.

[0014] In order to achieve the above object, a video monitoring method according to one aspect of the present invention comprises: a data acquisition step of acquiring image data of captured images from each of the plurality of cameras; a monitoring target detection step of detecting a monitoring target from the acquired image data; a coordinate calculation step of calculating coordinates of the monitoring target in real space based on information about the monitoring target detected and information about a camera that captured the monitoring target; a destination prediction step of determining a time displacement of the calculated coordinates and predicting a destination coordinate of the monitoring target based on the determined time displacement; a camera control step of controlling the visual axis directions of all or some of the plurality of cameras based on the predicted coordinates of the movement destination; The present invention is characterized by having the following.

[0015] Furthermore, in order to achieve the above object, a program according to one aspect of the present invention comprises: On the computer, a data acquisition step of acquiring image data of captured images from each of the plurality of cameras; a monitoring target detection step of detecting a monitoring target from the acquired image data; a coordinate calculation step of calculating coordinates of the monitoring target in real space based on information about the monitoring target detected and information about a camera that captured the monitoring target; a destination prediction step of determining a time displacement of the calculated coordinates and predicting a destination coordinate of the monitoring target based on the determined time displacement; a camera control step of controlling the visual axis directions of all or some of the plurality of cameras based on the predicted coordinates of the movement destination; The present invention is characterized in that the following is executed. [Effects of the Invention]

[0016] As described above, according to the present invention, there are fewer restrictions on the area that can be monitored, and it is possible to prevent the occurrence of a situation in which the target of monitoring is lost. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a video monitoring device according to an embodiment. [Figure 2] FIG. 2 is a diagram showing the overall configuration of a video monitoring system using the video monitoring device according to the embodiment. [Figure 3] FIG. 3 is a block diagram specifically showing the configuration of the video monitoring device according to the embodiment. [Figure 4] FIG. 4 is an explanatory diagram illustrating processing by the coordinate calculation unit according to the embodiment. [Figure 5] FIG. 5 is an explanatory diagram illustrating processing by the camera control unit in the embodiment. [Figure 6] FIG. 6 is an explanatory diagram illustrating the processing by the destination prediction unit according to the embodiment. [Figure 7] FIG. 7 is a diagram showing an example of an image captured by the camera in the example of FIG. [Figure 8] FIG. 8 is a flow diagram showing the process of detecting and tracking a monitoring target by the video monitoring device according to the embodiment. [Figure 9] FIG. 9 is a flowchart showing the movement prediction operation of the monitoring target by the video monitoring device according to the embodiment. [Figure 10] FIG. 10 is a flow diagram showing the operation of the camera in the embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of a computer that realizes a video monitoring device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] (Embodiment) Hereinafter, a video monitoring device, a video monitoring method, and a program according to an embodiment will be described with reference to FIGS.

[0019] [Device configuration] First, the schematic configuration of a video monitoring device according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the schematic configuration of a video monitoring device according to an embodiment.

[0020] In the embodiment shown in Fig. 1, video monitoring device 10 is a device for monitoring based on images from multiple cameras. As shown in Fig. 1, video monitoring device 10 includes data acquisition unit 11, monitoring target detection unit 12, coordinate calculation unit 13, destination prediction unit 14, and camera control unit 15.

[0021] The data acquisition unit 11 acquires image data of images captured by each of the multiple cameras 20. The monitoring target detection unit 12 detects the monitoring target from the acquired image data. The coordinate calculation unit 13 calculates the coordinates of the monitoring target in real space based on information about the detected monitoring target and information about the camera that captured the monitoring target (hereinafter referred to as "camera information").

[0022] The destination prediction unit 14 calculates the time displacement of the calculated coordinates and predicts the coordinates of the destination of the monitored object based on the calculated time displacement. The camera control unit 15 controls the shooting directions of all or some of the multiple cameras based on the predicted destination coordinates.

[0023] As described above, in the embodiment, the monitoring target is automatically detected, eliminating the need for an operator to manually track the monitoring target. Furthermore, in the embodiment, the coordinates of the destination of the detected monitoring target are predicted, eliminating the need for the imaging areas of adjacent cameras to overlap. Therefore, according to the embodiment, it is possible to reduce restrictions on the area that the video monitoring device 10 can monitor, and to prevent the monitoring target from being lost.

[0024] Next, the configuration and functions of the video monitoring device in the embodiment will be specifically explained using Figures 2 and 3. Figure 2 is a configuration diagram showing the overall configuration of a video monitoring system using the video monitoring device in the embodiment. Figure 3 is a block diagram specifically showing the configuration of the video monitoring device in the embodiment.

[0025] As shown in FIG. 2, a video monitoring device 10 according to an embodiment is connected to a plurality of cameras 20 so as to be able to communicate data with each other. The video monitoring device 10, together with the plurality of cameras 20, constitutes a monitoring system 100. The video monitoring device 10 detects a monitoring target (truck) 30 from image data output by the cameras 20 and predicts the coordinates of the destination of the detected monitoring target 30. Therefore, the monitoring system 100 can track and monitor the monitoring target 30 even if an obstruction 40 is present between the monitoring target 30 and the camera 20. The configurations of the video monitoring device 10 and the camera 20 will be described in detail below. Note that, in the example of FIG. 2, a truck is illustrated as the monitoring target 30, but the monitoring target 30 is not limited to a truck. The monitoring target 30 may be a moving object other than a truck, such as another type of automobile, a two-wheeled motor vehicle, a bicycle, a person, etc.

[0026] As shown in Fig. 3, camera 20 includes an imaging unit 21, a direction sensor 22, a GPS (Global Positioning System) sensor 23, a variable mechanism 24, a communication unit 25, and a control device 26. All cameras 20 have the same configuration, but Fig. 3 shows the configuration of only one camera 20, and the configurations of the other cameras 20 are omitted.

[0027] The imaging unit 21 is composed of a lens, a solid-state imaging element, a signal processing board, etc., and outputs image data of the captured subject on a frame-by-frame basis. In the imaging unit 21, light incident from the lens is converted into an electrical signal by the solid-state imaging element, and the electrical signal is further converted into image data by the signal processing board. Note that the wavelength range that can be captured by the solid-state imaging element does not need to be limited to the visible range, and may also be the infrared range. The wavelength range that can be captured is set appropriately depending on the application of the monitoring system 100, etc.

[0028] The orientation sensor 22 is a sensor that detects the visual axis direction of the camera 20, and outputs visual axis information indicating the detected visual axis direction. The visual axis is an axis along a normal line passing through the center of the solid-state image sensor, and the visual axis direction is determined by the angle (θ m0 ) and is synonymous with the shooting direction of the camera. The GPS sensor is a sensor that receives GPS signals from satellites and outputs position information that indicates the position of the camera 20 identified by the GPS signals. The variable mechanism 24 is made up of a support base that supports the camera 20, a mechanism for rotating the support base, and an electric motor for rotation, and is a mechanism that can change the shooting direction of the camera 20.

[0029] The communication unit 25 performs wired or wireless data communication with the video monitoring device 10. The communication unit 25 transmits image data, visual axis information, and position information to the video monitoring device 10 as camera information.

[0030] Furthermore, when a control signal is transmitted from the video monitoring device 10, the communication unit 25 receives the signal and sends the received control signal to the control device 26. The control signal includes a command for the camera control unit 15 of the video monitoring device 10 to change the visual axis direction of the camera 20. Upon receiving the control signal, the control device 26 operates the electric motor of the variable mechanism 24 based on the received control signal, thereby changing the visual axis direction of the camera 20.

[0031] In the embodiment, the visual axis direction of camera 20 may be fixed, in which case camera 20 may not be provided with variable mechanism 24 and control device 26. Also, in the embodiment, camera 20 may be mounted on a moving body, and its position may be variable.

[0032] As shown in FIG. 3, the video monitoring device 10 includes a data acquisition unit 11, a monitoring target detection unit 12, a coordinate calculation unit 13, a destination prediction unit 14, and a camera control unit 15, as well as a memory unit 16.

[0033] In this embodiment, data acquisition unit 11 acquires camera information, i.e., position information indicating the position of the camera and visual axis information indicating the visual axis direction of the camera, in addition to image data from each camera 20. Furthermore, data acquisition unit 11 stores the acquired image data and camera information in storage unit 16 for each camera 20.

[0034] In the embodiment, when the monitoring target detection unit 12 acquires image data from each camera 20, it detects the monitoring target by, for example, comparing the feature amount of the image identified in the image data with the feature amount of the monitoring target registered in advance. The monitoring target detection unit 12 can also detect the monitoring target by inputting the image data into a learning model in which the feature amount of the monitoring target has been machine-learned in advance. The feature amount used in the embodiment can be, for example, a SHIFT feature amount used in existing image processing.

[0035] In this embodiment, the coordinate calculation unit 13 uses the visual axis information of the camera 20 that photographed the monitored object to calculate the coordinates of the monitored object in a coordinate system with the camera 20 as the origin, and calculates the coordinates of the monitored object in real space from the calculated coordinates and the coordinates of the camera position specified by the position information.

[0036] Specifically, coordinate calculation unit 13 calculates the positional relationship between the monitored object and the camera that captured it, based on the two-dimensional coordinates of the monitored object on the image data in which the monitored object is detected and the visual axis information. Furthermore, coordinate calculation unit 13 calculates the distance from the camera that captured the monitored object to the monitored object, based on the size of the monitored object on the image data in which the monitored object is detected and the calculated positional relationship. Then, coordinate calculation unit 13 calculates the coordinates of the monitored object in a coordinate system (hereinafter referred to as the "camera coordinate system") with camera 20 that captured the monitored object as the origin, based on the calculated positional relationship and distance. Thereafter, coordinate calculation unit 13 calculates the coordinates of the monitored object in real space, based on the calculated coordinates in the camera coordinate system and position information of camera 20 that captured the monitored object.

[0037] In this embodiment, once the coordinates of the monitoring target are calculated by the coordinate calculation unit 13, the camera control unit 15 tracks the monitoring target using each camera 20. Specifically, the camera control unit 15 identifies the cameras 20 that are within a certain range from the monitoring target based on the position information of each camera 20. Then, the camera control unit 15 sends a control signal to the identified camera 20 to control the visual axis direction of the identified camera 20, thereby tracking the monitoring target.

[0038] In the embodiment, the destination prediction unit 14 predicts the coordinates of the destination of the monitored object from the time displacement of the coordinates of the monitored object under certain conditions, for example, when an obstruction 40 exists between the monitored object and one of the cameras 20 and the monitored object cannot be tracked.

[0039] The processing in the coordinate calculation unit 13 will be specifically described with reference to Fig. 4. Fig. 4 is an explanatory diagram for explaining the processing by the coordinate calculation unit in the embodiment.

[0040] As shown in FIG. 4, the coordinate calculation unit 13 identifies the camera m that captured the image of the monitored object from the image data in which the monitored object is detected, and calculates the position (x m , y m Then, the coordinate calculation unit 13 determines the coordinates (x A , y A ) is calculated.

[0041] Specifically, the coordinate calculation unit 13 calculates the actual size (width d A , height t A The actual size can be determined, for example, by creating a database in advance in which the actual size is registered for each monitoring target.

[0042] Next, the coordinate calculation unit 13 calculates the image size H of the monitored object 30 in the horizontal direction from the image data of the camera m. A and further, the distance H from the center of gravity of the monitored object 30 to the horizontal center of the image mAThen, the coordinate calculation unit 13 determines the horizontal image size H m and horizontal angle of view θ m and distance H mA Applying this to the following equation 1, the angle Δθ between the visual axis of the camera m and the axis passing through the center of gravity of the monitored object 30 from the center of the camera m is calculated. mA Calculate the horizontal image size H m and horizontal angle of view θ m is a known value.

[0043]

number

[0044] Next, the coordinate calculation unit 13 calculates the actual width d of the object to be monitored. A and the image size H of the monitored object 30 in the horizontal direction. A and the distance H from the center of gravity of the monitored object 30 to the horizontal center of the image. mA Applying this to the following equation 2, the horizontal distance D from the visual axis of the camera m to the monitored object 30 is calculated. mA Calculate.

[0045]

number

[0046] Next, the coordinate calculation unit 13 calculates an angle θ indicating the visual axis direction from the visual axis information acquired by the data acquisition unit 11. m0 Identify the specified angle θ m0 and the angle Δθ calculated earlier mA Substituting the above into Equation 3, the angle θ between the reference direction and the axis passing through the center of the camera m and the center of gravity of the monitored object 30 is mA The calculated angle θ mA indicates the positional relationship between the monitored object and camera m.

[0047]

number

[0048] Furthermore, the coordinate calculation unit 13 calculates the calculated angle θ mA and the horizontal distance D from the visual axis of the camera m to the monitored object 30. mA Applying this to Equation 4, the distance L from the camera m to the monitored object 30 is mA Calculate.

[0049]

number

[0050] Next, the coordinate calculation unit 13 calculates an angle θ mA and the distance L from the camera m to the monitored object 30 mA and are applied to Equation 5 and Equation 6. As a result, the coordinates of the monitored object in the coordinate system (camera coordinate system) with the camera m as the origin (Δx mA , Δy mA ) is calculated.

[0051]

number

[0052]

number

[0053] Then, the coordinate calculation unit 13 calculates the camera position (x m , y m ) and the coordinates of the monitored object in the camera coordinate system (Δx mA , Δy mA ) to Equation 7 and Equation 8, the coordinates (x A , y A ) is calculated.

[0054]

number

[0055]

number

[0056] The processing in the camera control unit 15 will be specifically described with reference to Fig. 5. Fig. 5 is an explanatory diagram illustrating the processing by the camera control unit in the embodiment.

[0057] 5, when the coordinates of the monitoring target 30 in real space are calculated by the coordinate calculation unit 13, the camera control unit 15 compares the calculated coordinates with the position information of the other cameras 20 to identify at least one camera 20 that is in the vicinity of the monitoring target 30. Specifically, the coordinate calculation unit 13 calculates the distance between the positions of the other cameras 20 and the monitoring target 30, and identifies a camera n where the calculated distance is equal to or less than a threshold. Note that if multiple cameras n are identified, the following process is performed for each identified camera n.

[0058] Next, the camera control unit 15 calculates the visual axis direction (photographing direction) for the identified camera n to track the monitoring target 30. Specifically, the camera control unit 15 calculates an angle θ indicating the visual axis direction from the visual axis information of the identified camera n. n0 and determine the position of camera n (x n , y n ) to identify the

[0059] Next, the camera control unit 15 calculates the coordinates (x A , y A ) and the position of camera n (x n , y n ) and Equation 9 and Equation 10 below to obtain the coordinates (Δx nA , Δy nA ) is calculated.

[0060]

number

[0061]

number

[0062] Next, the camera control unit 15 calculates the coordinates (Δx nA , Δy nA ) is applied to the following equation 11 to obtain the angle θ between the reference direction and the axis passing through the center of the camera n and the center of gravity of the monitored object 30. nA Calculate.

[0063]

number

[0064] Next, the camera control unit 15 uses the following equation 12 to calculate the calculated angle θ nA From the angle θ that indicates the visual axis direction of camera n n0 By subtracting the angle Δθ between the visual axis of the camera n and the axis passing through the center of the camera n and the center of gravity of the monitored object 30, nA The calculated angle Δθ nA is the visual axis direction of camera n for tracking the monitored object 30.

[0065]

number

[0066] Next, the camera control unit 15 tracks the monitoring target 30 using image data from the camera m that first detected the monitoring target 30. For the camera n, the camera control unit 15 also uses the calculated Δθ nA and instructs the camera to change the visual axis direction. After that, when the monitoring target 30 is detected by the image data from the camera n, the camera control unit 15 tracks the monitoring target using the image data from the camera n as well.

[0067] The tracking process of the monitoring target will be described in detail. First, for each camera 20 used for tracking, the camera control unit 15 compares the feature amount extracted from the current frame with the feature amount extracted from the previous frame for the monitoring target in the image to calculate the similarity. If the similarity is equal to or greater than a predetermined value, the camera control unit 15 determines that the monitoring target is the same in the current frame and the previous frame, and when a new frame is input, it executes the above-mentioned similarity calculation process again. Furthermore, when tracking processing is being performed, the coordinate calculation unit 13 calculates the coordinates of the monitoring target in real space every time image data is acquired.

[0068] On the other hand, if the similarity is less than a predetermined value, the camera control unit 15 determines that the monitoring target is not the same in the current frame and the previous frame. In this case, the camera control unit 15 instructs the destination prediction unit 14 to predict the coordinates of the monitoring target's destination, since this corresponds to a case where the monitoring target cannot be tracked due to the presence of an obstruction or the monitoring target being out of frame.

[0069] The processing in the destination prediction unit 14 will be specifically described with reference to Fig. 6 and Fig. 7. Fig. 6 is an explanatory diagram illustrating the processing by the destination prediction unit 14 in the embodiment. Fig. 7 is a diagram showing an example of an image captured by the camera in the example of Fig. 6. In the example of Fig. 6 and Fig. 7, it is assumed that the monitoring target 30 is detected up to the (f-1)th frame and the fth frame of the image data, but is not detected from the (f+1)th frame onwards.

[0070] As shown in Fig. 6 and Fig. 7, the monitoring target 30 is detected in the (f-1)th frame and the fth frame. Therefore, the movement destination prediction unit 14 calculates the time displacement of the coordinates of the monitoring target between these frames using the following equations 13 to 15, and calculates the velocity (v Ax , v Ay ) In addition, in equation 15, f m is the frame rate, which is a known value. m is the frame rate f m is the inverse of the time between frames.

[0071]

number

[0072]

number

[0073]

number

[0074] Next, the destination prediction unit 14 calculates the inter-frame time t m Using this, the start time t of the f+1th frame f and the start time t of the (f+α)th frame to be predicted. f+α Then, the destination prediction unit 14 calculates the start time t f and start time t f+α Using this, the time from when the monitored object 30 was last detected to the time to be predicted (t f+α -t f Then, the destination prediction unit 14 calculates the previously calculated speed (v Ax , v Ay ) and the calculated time (t f+α -t f ) are applied to equations 16 and 17 to obtain the coordinates of the destination of the monitored object 30, that is, the coordinates (x A,f+α , y A,f+α ) is calculated.

[0075]

number

[0076]

number

[0077] When the destination prediction unit 14 calculates the coordinates of the destination of the monitoring target, the camera control unit 15 calculates the visual axis direction of the camera m using the above-mentioned equations 9 to 12 so that the coordinates of the destination are captured by the camera m. Then, the destination prediction unit 14 transmits a control signal to the camera m to change the visual axis direction. As a result, the visual axis of the camera m is changed, and when the monitoring target 30 is detected from the image data of the camera m, the camera control unit 15 resumes tracking.

[0078] [Device operation] Next, the operation of the video monitoring device 10 in the embodiment will be described with reference to Figures 8 to 10. In the following description, Figures 1 to 7 will be referenced as appropriate. In the embodiment, a video monitoring method is implemented by operating the video monitoring device 10. Therefore, the description of the video monitoring method in the embodiment will be replaced by the following description of the operation of the video monitoring device 10.

[0079] First, the process of detecting a monitoring target and the process of tracking the detected monitoring target by the video monitoring device 10 will be described with reference to Fig. 8. Fig. 8 is a flow diagram showing the process of detecting and tracking a monitoring target by the video monitoring device in this embodiment.

[0080] As shown in FIG. 8, first, the data acquisition unit 11 acquires image data, position information indicating the position of the camera, and visual axis information indicating the visual axis direction of the camera from each of the cameras 20 (step A1).

[0081] Next, the monitoring target detection unit 12 compares, for each camera 20, the image features identified by the image data acquired in step A1 with the features of the monitoring targets registered in advance, and performs a monitoring target detection process (step A2).

[0082] Next, the monitoring target detection unit 12 determines whether or not a monitoring target has been detected from the image data of any of the cameras based on the result of the detection process in step A2 (step A3).

[0083] If the result of the determination in step A2 is that the monitoring target cannot be detected from the image data of any of the cameras 20, step A1 is executed again. On the other hand, if the result of the determination in step A2 is that the monitoring target can be detected from the image data of any of the cameras 20, step A4 is executed.

[0084] In step A4, the coordinate calculation unit 13 uses the visual axis information of the camera 20 that captured the image of the monitored object to calculate the coordinates of the monitored object in the camera coordinate system of this camera 20. The coordinate calculation unit 13 also calculates the coordinates of the monitored object in real space from the calculated coordinates of the monitored object in the camera coordinate system and the coordinates of the camera position specified by the position information of the camera 20.

[0085] Next, the camera control unit 15 compares the coordinates in real space of the monitored object 30 calculated in step A4 with the position information of the other cameras 20, and identifies at least one camera 20 that is in the vicinity of the monitored object 30 (step A5).

[0086] Next, the camera control unit 15 calculates the visual axis direction for tracking the monitoring target 30 for the camera 20 identified in step A5 (step A6). Furthermore, the camera control unit 15 transmits a control signal to the camera 20 identified in step A5 to instruct it to change the visual axis direction in order to notify it of the calculated visual axis direction (step A7).

[0087] Next, the camera control unit 15 executes a tracking process for the monitoring target 30 in each of the cameras 20 that captured the image of the monitoring target and the cameras 20 identified in step A5 (step A8). In step A8, the camera control unit 15 executes a tracking process for each camera 20, each time image data is acquired from each camera 20, until the monitoring target 30 is lost. In addition, the coordinate calculation unit 13 calculates the coordinates of the monitoring target in real space each time image data is acquired.

[0088] Specifically, in step A8, the camera control unit 15 compares, for each camera 20, the features extracted from the current frame with the features extracted from the previous frame, calculates the similarity, and determines whether the similarity is greater than or equal to a predetermined value.

[0089] If the similarity is equal to or greater than a predetermined value, the camera control unit 15 determines that the monitoring target 30 is being tracked, and when new image data is acquired, it calculates and determines the similarity again. On the other hand, if the similarity is less than the predetermined value, the camera control unit 15 determines that the monitoring target 30 has been lost, and the following process shown in Fig. 9 is executed for the corresponding camera 20.

[0090] Next, the process performed by the video monitoring device 10 when it becomes unable to track a detected monitoring target will be described with reference to Fig. 9. Fig. 9 is a flow diagram showing the movement prediction operation of the monitoring target by the video monitoring device in the embodiment. The movement prediction operation shown in Fig. 9 is executed when the camera control unit 15 determines that the monitoring target has been lost in any of the cameras 20.

[0091] As shown in FIG. 9, the destination prediction unit 14 first calculates the speed of the monitoring target 30 from the time displacement of the coordinates of the monitoring target (step B1).

[0092] Next, the destination prediction unit 14 calculates the time from when the monitoring target 30 was last detected until the time to be predicted (step B2).

[0093] Next, the destination prediction unit 14 calculates the coordinates of the destination of the monitoring target 30 using the latest coordinates in real space of the monitoring target 30, the speed calculated in step B1, and the time calculated in step B2 (step B3).

[0094] Next, camera control unit 15 calculates a new visual axis direction for camera 20 that has lost sight of monitoring target 30 based on the coordinates of the destination of monitoring target 30 so that camera 20 can capture an image at the coordinates of the destination of monitoring target 30 (step B4). Furthermore, camera control unit 15 transmits a control signal notifying change of the visual axis direction to camera 20 in order to notify the calculated visual axis direction (step B5).

[0095] By executing step B5, the visual axis of the camera 20 that has lost sight of the monitoring target 30 is changed, and when the monitoring target 30 is detected from the image data of this camera, the camera control unit 15 resumes tracking (step B6).

[0096] Next, the processing performed by camera 20 when a control signal is received will be described with reference to Fig. 10. Fig. 10 is a flow chart showing the operation of the camera in this embodiment.

[0097] 10, first, in camera 20, communication unit 25 receives a control signal transmitted from video monitoring device 10 (step C1). Communication unit 25 sends the received control signal to control device 26.

[0098] Next, upon receiving the control signal, the control device 26 operates the electric motor of the variable mechanism 24 based on the received control signal to change the visual axis direction of the camera 20 (step C2).

[0099] Next, if the visual axis direction has been changed, the control device 26 transmits the changed visual axis information to the video monitoring device 10 via the communication unit 25 (step C3).

[0100] [Effects of the embodiment] In this way, the video monitoring device 10 automatically detects and tracks the monitoring target 30 based on image data from each camera 20. Furthermore, if another camera 20 is present near the monitoring target 30, the video monitoring device 10 can also track the monitoring target 30 using the other camera 20. Furthermore, if the video monitoring device 10 loses sight of the monitoring target 30, it can predict the destination of the monitoring target 30 and track it based on the coordinates of the predicted destination. Therefore, according to the embodiment, it is possible to reduce restrictions on the area that the video monitoring device 10 can monitor and prevent the monitoring target from being lost.

[0101] [Variations] In the above example, a visible light camera is used as the camera 20, but in the embodiment, the camera 20 may be a camera having a function of identifying the depth to the subject. Examples of such a camera 20 include a TOF (Time Of Flight) camera and a LiDAR (Light Detection and Ranging) camera.

[0102] If the camera 20 has a function for identifying the depth to the subject, the data acquisition unit 11 further acquires depth information identifying the depth to the subject from each of the multiple cameras 20. Furthermore, in this case, the coordinate calculation unit 13 calculates the coordinates of the monitored target in a coordinate system with the camera that captured the monitored target as the origin, based on the two-dimensional coordinates of the monitored target on the image data in which the monitored target is detected and the depth information of the camera that captured the monitored target. Thereafter, the coordinate calculation unit 13 calculates the coordinates of the monitored target in real space from the calculated coordinates and the coordinates of the position of the camera 20 identified by the position information, as in the above example.

[0103] [program] The program in the embodiment may be any program that causes a computer to execute steps A1 to A8 shown in Fig. 8 and steps B1 to B6 shown in Fig. 9. By installing and executing this program in a computer, the video monitoring device 10 and video monitoring method in the present embodiment can be realized. In this case, the processor of the computer functions as a data acquisition unit 11, a monitoring target detection unit 12, a coordinate calculation unit 13, a destination prediction unit 14, and a camera control unit 15, and performs processing.

[0104] Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device. In the embodiment, the storage unit 16 may be realized by a storage device such as a hard disk provided in the computer, or may be realized by a storage device of another computer.

[0105] The program in the embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the data acquisition unit 11, the monitoring target detection unit 12, the coordinate calculation unit 13, the destination prediction unit 14, and the camera control unit 15.

[0106] [Physical configuration] Here, a computer that realizes the video monitoring device 10 by executing the program in the embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing an example of a computer that realizes the video monitoring device in the embodiment.

[0107] 11, a computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0108] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0109] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0110] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0111] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0112] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0113] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0114] The video monitoring device 10 in the embodiment can be realized by using hardware corresponding to each part, rather than a computer on which a program is installed. Furthermore, the video monitoring device 10 may be partially realized by a program and the remaining part by hardware.

[0115] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 15) described below, but are not limited to the following descriptions.

[0116] (Appendix 1) a data acquisition unit that acquires image data of captured images from each of the plurality of cameras; a monitoring target detection unit that detects a monitoring target from the acquired image data; a coordinate calculation unit that calculates coordinates of the monitoring target in real space based on information about the monitoring target detected and information about a camera that captured the monitoring target; a destination prediction unit that calculates a time displacement of the calculated coordinates and predicts the coordinates of a destination of the monitoring target based on the calculated time displacement; a camera control unit that controls the visual axis directions of all or some of the plurality of cameras based on the predicted coordinates of the movement destination; A video monitoring device comprising:

[0117] (Appendix 2) 10. The video surveillance device according to claim 1, the data acquisition unit further acquires, from each of the plurality of cameras, position information indicating a position of the camera and visual axis information indicating a visual axis direction of the camera; The coordinate calculation unit Using the visual axis information, calculate coordinates of the monitoring target in a coordinate system having an origin at the camera that photographed the monitoring target; calculating coordinates in real space of the monitoring target based on the calculated coordinates and the position information of the camera that captured the monitoring target; A video surveillance device characterized by:

[0118] (Appendix 3) 3. The video surveillance device according to claim 2, The coordinate calculation unit determining a positional relationship between the monitoring target and a camera that photographed the monitoring target from the two-dimensional coordinates of the monitoring target on the image data where the monitoring target was detected and the visual axis information; and calculating a distance from the camera that captured the image of the monitoring target to the monitoring target based on the size of the monitoring target on the image data in which the monitoring target was detected and the calculated positional relationship; calculating coordinates of the monitoring target in a coordinate system having an origin at the camera that captured the image of the monitoring target based on the determined positional relationship and distance; A video monitoring device characterized by

[0119] (Appendix 4) 3. The video surveillance device according to claim 2, the data acquisition unit further acquires depth information specifying a depth to a subject from each of the plurality of cameras; The coordinate calculation unit calculating coordinates of the monitoring target based on two-dimensional coordinates of the monitoring target on the image data in which the monitoring target is detected, the visual axis information, and the depth information of the camera that captured the monitoring target; A video monitoring device characterized by

[0120] (Appendix 5) 5. A video monitoring device according to any one of claims 1 to 4, each of the plurality of cameras includes a mechanism capable of changing the visual axis direction and a control device that controls the operation of the mechanism; the camera control unit notifies the control device of each of the plurality of cameras of the coordinates of the predicted destination; A video surveillance device characterized by:

[0121] (Appendix 6) a data acquisition step of acquiring image data of captured images from each of the plurality of cameras; a monitoring target detection step of detecting a monitoring target from the acquired image data; a coordinate calculation step of calculating coordinates of the monitoring target in real space based on information about the monitoring target detected and information about a camera that captured the monitoring target; a destination prediction step of determining a time displacement of the calculated coordinates and predicting a destination coordinate of the monitoring target based on the determined time displacement; a camera control step of controlling the visual axis directions of all or some of the plurality of cameras based on the predicted coordinates of the movement destination; A video monitoring method comprising:

[0122] (Appendix 7) 7. The video surveillance method according to claim 6, comprising: In the data acquisition step, position information indicating a position of each of the plurality of cameras and visual axis information indicating a visual axis direction of each of the cameras are further acquired, In the coordinate calculation step, Using the visual axis information, calculate coordinates of the monitoring target in a coordinate system having an origin at the camera that photographed the monitoring target; calculating coordinates in real space of the monitoring target based on the calculated coordinates and the position information of the camera that captured the monitoring target; A video monitoring method comprising:

[0123] (Appendix 8) 8. The video surveillance method according to claim 7, further comprising: In the coordinate calculation step, determining a positional relationship between the monitoring target and a camera that photographed the monitoring target from the two-dimensional coordinates of the monitoring target on the image data where the monitoring target was detected and the visual axis information; and calculating a distance from the camera that captured the image of the monitoring target to the monitoring target based on the size of the monitoring target on the image data in which the monitoring target was detected and the calculated positional relationship; calculating coordinates of the monitoring target in a coordinate system having an origin at the camera that captured the image of the monitoring target based on the determined positional relationship and distance; A video monitoring method comprising:

[0124] (Appendix 9) 8. The video surveillance method according to claim 7, further comprising: In the data acquisition step, depth information specifying a depth to a subject is further acquired from each of the plurality of cameras; In the coordinate calculation step, calculating coordinates of the monitoring target based on two-dimensional coordinates of the monitoring target on the image data in which the monitoring target is detected, the visual axis information, and the depth information of the camera that captured the monitoring target; A video monitoring method comprising:

[0125] (Appendix 10) A video monitoring method according to any one of Supplementary Notes 6 to 9, each of the plurality of cameras includes a mechanism capable of changing the visual axis direction and a control device that controls the operation of the mechanism; In the camera control step, the predicted coordinates of the movement destination are notified to the control device of each of the plurality of cameras. A video monitoring method comprising:

[0126] (Appendix 11) On the computer, a data acquisition step of acquiring image data of captured images from each of the plurality of cameras; a monitoring target detection step of detecting a monitoring target from the acquired image data; a coordinate calculation step of calculating coordinates of the monitoring target in real space based on information about the monitoring target detected and information about a camera that captured the monitoring target; a destination prediction step of determining a time displacement of the calculated coordinates and predicting a destination coordinate of the monitoring target based on the determined time displacement; a camera control step of controlling the visual axis directions of all or some of the plurality of cameras based on the predicted coordinates of the movement destination; A program that executes.

[0127] (Appendix 12) 12. The program of claim 11, In the data acquisition step, position information indicating a position of each of the plurality of cameras and visual axis information indicating a visual axis direction of each of the cameras are further acquired, In the coordinate calculation step, Using the visual axis information, calculate coordinates of the monitoring target in a coordinate system having an origin at the camera that photographed the monitoring target; calculating coordinates in real space of the monitoring target based on the calculated coordinates and the position information of the camera that captured the monitoring target; A program characterized by:

[0128] (Appendix 13) 13. The program of claim 12, In the coordinate calculation step, determining a positional relationship between the monitoring target and a camera that photographed the monitoring target from the two-dimensional coordinates of the monitoring target on the image data where the monitoring target was detected and the visual axis information; and calculating a distance from the camera that captured the image of the monitoring target to the monitoring target based on the size of the monitoring target on the image data in which the monitoring target was detected and the calculated positional relationship; calculating coordinates of the monitoring target in a coordinate system having an origin at the camera that captured the image of the monitoring target based on the determined positional relationship and distance; A program characterized by:

[0129] (Appendix 14) 13. The program of claim 12, In the data acquisition step, depth information specifying a depth to a subject is further acquired from each of the plurality of cameras; In the coordinate calculation step, calculating coordinates of the monitoring target based on two-dimensional coordinates of the monitoring target on the image data in which the monitoring target is detected, the visual axis information, and the depth information of the camera that captured the monitoring target; A program characterized by:

[0130] (Appendix 15) The program according to any one of Supplementary Notes 11 to 14, each of the plurality of cameras includes a mechanism capable of changing the visual axis direction and a control device that controls the operation of the mechanism; In the camera control step, the predicted coordinates of the movement destination are notified to the control device of each of the plurality of cameras. A program characterized by: [Industrial Applicability]

[0131] As described above, according to the present invention, there are fewer restrictions on the area that can be monitored, and it is possible to prevent the monitoring target from being lost. The present invention is useful for a system that performs monitoring using video. [Explanation of symbols]

[0132] 10. Video surveillance equipment 11 Data Acquisition Section 12 Monitoring target detection unit 13 Coordinate calculation unit 14 Destination Prediction Unit 15 Camera control unit 16 Memory section 20 Camera 21 Imaging unit 22 Orientation sensor 23 GPS sensor 24 Variable Mechanism 25 Communications Department 26 Control device 30 monitored 110 Computer 111 CPU 112 main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display Device 120 Recording Media 121 Bus

Claims

1. a data acquisition unit that acquires image data of captured images from each of the plurality of cameras; a monitoring target detection unit that detects a monitoring target from the acquired image data; a coordinate calculation unit that calculates coordinates of the monitoring target in real space based on information about the monitoring target detected and information about a camera that captured the monitoring target; a destination prediction unit that calculates a time displacement of the calculated coordinates and predicts coordinates of a destination of the monitoring target based on the calculated time displacement; a camera control unit that controls the visual axis directions of all or some of the plurality of cameras based on the predicted coordinates of the movement destination; Equipped with each of the plurality of cameras includes a mechanism capable of changing the visual axis direction and a control device that controls the operation of the mechanism; The camera control unit compares the coordinates of the monitored object in real space with the position information of cameras other than the camera that photographed the monitored object, identifies a camera whose distance to the monitored object is below a threshold, calculates the visual axis direction of the identified camera so that the identified camera will photograph the coordinates of the predicted destination of the monitored object, and sends a control signal to the control device of the identified camera to change the visual axis direction.

2. 2. The video monitoring device according to claim 1, the data acquisition unit further acquires, from each of the plurality of cameras, position information indicating a position of the camera and visual axis information indicating a visual axis direction of the camera; The coordinate calculation unit Using the visual axis information, calculate coordinates of the monitoring target in a coordinate system having the origin of the camera that photographed the monitoring target; calculating coordinates in real space of the monitoring target based on the calculated coordinates and the position information of the camera that captured the monitoring target; A video surveillance device characterized by:

3. 3. The video monitoring device according to claim 2, The coordinate calculation unit determining a positional relationship between the monitoring target and a camera that photographed the monitoring target from the two-dimensional coordinates of the monitoring target on the image data where the monitoring target was detected and the visual axis information; and calculating a distance from the camera that captured the image of the monitoring target to the monitoring target based on the size of the monitoring target on the image data in which the monitoring target was detected and the calculated positional relationship; calculating coordinates of the monitoring target in a coordinate system having an origin at the camera that captured the image of the monitoring target based on the determined positional relationship and distance; A video surveillance device characterized by:

4. 3. The video monitoring device according to claim 2, the data acquisition unit further acquires depth information specifying a depth to a subject from each of the plurality of cameras; The coordinate calculation unit calculating coordinates of the monitoring target based on two-dimensional coordinates of the monitoring target on the image data in which the monitoring target is detected, the visual axis information, and the depth information of the camera that captured the monitoring target; A video surveillance device characterized by:

5. a data acquisition step of acquiring image data of captured images from each of the plurality of cameras; a monitoring target detection step of detecting a monitoring target from the acquired image data; a coordinate calculation step of calculating coordinates of the monitoring target in real space based on information about the monitoring target detected and information about a camera that captured the monitoring target; a destination prediction step of determining a time displacement of the calculated coordinates and predicting a destination coordinate of the monitoring target based on the determined time displacement; a camera control step of controlling the visual axis directions of all or some of the plurality of cameras based on the predicted coordinates of the movement destination; and each of the plurality of cameras includes a mechanism capable of changing the visual axis direction and a control device that controls the operation of the mechanism; In the camera control step, the coordinates of the monitored object in real space are compared with position information of a camera other than the camera that photographed the monitored object, and a camera whose distance to the monitored object is equal to or less than a threshold is identified, the visual axis direction of the identified camera is calculated so that the identified camera photographs the coordinates of the predicted destination of the monitored object, and a control signal for changing the visual axis direction is transmitted to a control device of the identified camera. A video monitoring method comprising:

6. On the computer, a data acquisition step of acquiring image data of captured images from each of the plurality of cameras; a monitoring target detection step of detecting a monitoring target from the acquired image data; a coordinate calculation step of calculating coordinates of the monitoring target in real space based on information about the monitoring target detected and information about a camera that captured the monitoring target; a destination prediction step of determining a time displacement of the calculated coordinates and predicting a destination coordinate of the monitoring target based on the determined time displacement; a camera control step of controlling the visual axis directions of all or some of the plurality of cameras based on the predicted coordinates of the movement destination; Execute each of the plurality of cameras includes a mechanism capable of changing the visual axis direction and a control device that controls the operation of the mechanism; In the camera control step, the coordinates of the monitored object in real space are compared with position information of a camera other than the camera that photographed the monitored object, and a camera whose distance to the monitored object is equal to or less than a threshold is identified, the visual axis direction of the identified camera is calculated so that the identified camera photographs the coordinates of the predicted destination of the monitored object, and a control signal for changing the visual axis direction is transmitted to a control device of the identified camera. program.

Images