Moving Trajectory Information Processing Device, Moving Trajectory Information Processing Method, and Recording Medium

Abstract

A movement path information processing device according to the present invention is provided with a calculating means and a transformation means. The calculation means calculates position information, in images of a moving object, from a plurality of images in which the object has been continuously photographed using a camera which captures the object in photographed images, and movement path information which is a time series of orientation information of the camera. The transformation means calculates, from the abovementioned images, a plane projection transformation matrix to prescribed images, and uses the calculated plane projection transformation matrix to transform the movement path information into movement path information constituting a time series of position information in the prescribed images of the object.

Description

Technical Field

[0001] The present invention relates to a moving trajectory information processing apparatus, a moving trajectory information processing method, and a recording medium.

Background Art

[0002] An example of a conventional apparatus for calculating moving trajectory information is described in Patent Document 1. The conventional apparatus continuously captures a moving object such as a moving person while fixing the pan value, tilt value, and zoom value (PTZ value) of a camera. Further, at each time of shooting, the conventional apparatus extracts the position of the moving object from the image obtained by shooting with the camera, and converts the coordinate value of the object on the image into the coordinate value in the real space using camera calibration information (internal parameters and external parameters). The conventional apparatus uses the time series of the positions in the real space of the moving object obtained as described above as the moving trajectory information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a camera with fixed PTZ values, the field of view of the camera is limited. Therefore, for the purpose of monitoring a wider range, it is important to calculate the movement trajectory information of an object from a plurality of images continuously captured by the camera while capturing the object to be moved within the imaging range by changing the PTZ values. However, when the PTZ value of the camera changes, the camera calibration information of that camera changes. Also, in order to obtain the camera calibration information corresponding to the changed PTZ value, a calibration operation by an operator is required. Therefore, in a conventional device that directly converts the coordinate value of an object on the image into the coordinate value in the real space using the camera calibration information from each of the plurality of continuously captured images, it has been difficult to calculate the movement trajectory information from the plurality of images continuously captured by the camera that captures the object to be moved within the imaging range.

[0005] The present invention aims to provide a movement trajectory information processing device that solves the above-described problems.

Means for Solving the Problems

[0006] A movement trajectory information processing device according to one aspect of the present invention includes a calculating means for calculating movement trajectory information, which is a time series of the position information of an object on an image and the posture information of the camera, from a plurality of images continuously captured by a camera that captures an object to be moved within a captured image; a conversion means for calculating a planar projective transformation matrix from the image to a predetermined image, and converting the movement trajectory information using the calculated planar projective transformation matrix into movement trajectory information, which is a time series of the position information of the object on the predetermined image; A movement trajectory information processing device comprising the above. is configured as described above.

[0007] A movement trajectory information processing method according to another aspect of the present invention calculates movement trajectory information, which is a time series of the position information of an object on an image and the posture information of the camera, from a plurality of images continuously captured by a camera that captures an object to be moved within a captured image, calculates a planar projective transformation matrix from the image to a predetermined image, Converting the movement trajectory information into movement trajectory information which is a time series of the position information on the predetermined image of the object by using the calculated planar projective transformation matrix. It is configured as follows.

[0008] A computer-readable recording medium according to another aspect of the present invention causes a computer to perform a process of calculating movement trajectory information which is a time series of the position information on the image of the object and the posture information of the camera from a plurality of images obtained by continuously photographing the object with a camera that captures the moving object within the photographed image, a process of calculating a planar projective transformation matrix from the image to a predetermined image, and a process of converting the movement trajectory information into movement trajectory information which is a time series of the position information on the predetermined image of the object by using the calculated planar projective transformation matrix. It is configured to record a program for causing the above to be performed.

Advantages of the Invention

[0009] With the configuration described above, the present invention can calculate the movement trajectory information of the object from a plurality of images obtained by continuously photographing the object with a camera that captures the moving object within the photographed range.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0011] Next, embodiments of the present invention will be described in detail with reference to the drawings. In the description in the specification, although descriptions having similar meanings of "tracking" and "tailgating" coexist, "tracking" is mainly used to explain the configuration in the part with a strong relationship with people, and "tailgating" is mainly used to explain the configuration in the part with a strong relationship with devices, and it is only used separately for convenience. [First Embodiment] First, regarding the first embodiment of the present invention, for ease of understanding, the problems assumed by the first embodiment of the present invention will be described.

[0012] Detecting people from the images of surveillance cameras and performing tracking is useful for crime prevention, marketing, etc. In a camera with fixed PTZ values, the field of view of the camera is limited. By changing the PTZ values of the camera, the field of view of the camera can be expanded. However, even then, there is a limit to the field of view of a single camera. Therefore, consider a system in which multiple PTZ cameras are installed in a distributed manner in the surveillance area with a part of the camera field of view in common to track people over a wider range spatially and temporally. In such a tracking system, it is desirable that the system automatically recognizes that a person being traced by one camera has entered the field of view of another camera, and that the other camera can start tracing that person. In order to determine the person to take over the tracing from among many people present in the camera's field of view, the movement trajectory of the person can be utilized. That is, the tracking target is determined by collating the information on the movement trajectory of the person being traced by one camera (tracking target movement trajectory information) with the information on the movement trajectory of the people within the field of view of the other camera (collated movement trajectory information). In order to perform such collation, the tracking target movement trajectory information and the collated movement trajectory information need to be expressed in the same coordinate system.

[0013] Generally, for the coordinate systems representing the positions of people obtained by photographing with cameras, there are an image coordinate system and a camera coordinate system that are specific to each camera, and a world coordinate system that is common to multiple cameras. The image coordinate system is a two-dimensional coordinate system on the imaging device, and the position of a point on the image is usually represented in this image coordinate system. The camera coordinate system is a coordinate system determined by the image coordinate system and the internal parameters of the camera. Therefore, the image coordinate system of the camera and its camera coordinate system can be mutually converted using the internal parameters of the camera. The world coordinate system is a coordinate system determined by the camera calibration information composed of the internal parameters and external parameters of the camera and the image coordinate system. Therefore, the image coordinate system of the camera and the world coordinate system can be mutually converted using the camera calibration information of the camera.

[0014] The conventional device extracts the position of a person from the captured image in the image coordinate system, and uses the camera calibration information to convert the coordinate values of the extracted person in the image coordinate system into the coordinate values in the world coordinate system. However, when the PTZ values of the camera are changed to track the person, the camera calibration information changes. Also, it is difficult to obtain the camera calibration information corresponding to the changed PTZ values in real time. Therefore, it has been difficult for the conventional device to calculate the movement trajectory information while tracking the object with a camera having a tracking function for capturing the moving object within the shooting range. As a result, it has been difficult to collate the movement trajectories of a person in real time among a plurality of PTZ cameras.

[0015] This embodiment solves the above problems. Hereinafter, a tracking system 10 according to an embodiment to which the present invention is applied will be described in detail with reference to the drawings.

[0016] FIG. 1 is a schematic diagram showing a configuration example of a tracking system 10 according to an embodiment of the present invention. Referring to FIG. 1, the tracking system 10 is a system that detects and tracks a person on a passage, and includes two PTZ cameras 11 and 12 and a control device 20.

[0017] PTZ cameras (hereinafter simply referred to as cameras) 11 and 12 include a solid-state imaging device such as a CMOS sensor or a CCD sensor and a pan-tilt unit. The cameras 11 and 12 generate a plurality of images by continuously photographing a photographing range at a certain period, for example, according to a command from the control device 20. It is preferable that the photographing periods and photographing timings of the camera 11 and the camera 12 are the same or similar, but they may be different. The cameras 11 and 12 can change the photographing range by changing pan, tilt, and zoom according to a command from the control device 20. Here, pan means swinging the direction of the camera left and right, tilt means swinging the direction of the camera up and down, and zoom means changing the angle of view to telephoto or wide angle, respectively. The cameras 11 and 12 are installed dispersedly in the monitoring area so that a part of their photographing ranges overlaps. In the example shown in FIG. 1, the camera 11 monitors the monitoring areas 13A, 13B, and 13C set on the passage extending in the left-right direction of the paper surface. The camera 12 monitors the monitoring area 13C and the monitoring area 13D set on the passage extending in the up-down direction of the paper surface. The monitoring area 13C corresponds to the corner part of the passage. That is, the area at the corner of the passage is the common monitoring area 13C for the two cameras 11 and 12. In the passage, it is assumed that a person enters from the left side of the paper surface, passes through the corner of the passage, and exits from the lower side of the paper surface. The monitoring areas 13A to 13D are flat surfaces.

[0018] The control device 20 is connected to the cameras 11 and 12 by wire or wirelessly. The control device 20 calculates the camera calibration information (internal parameters and external parameters) of each camera when the cameras 11 and 12 are installed. The calculation of this camera calibration information is performed by setting the PTZ values of the cameras 11 and 12 to the reference PTZs of the respective cameras. For example, the reference PTZ of the camera 11 is set so as to have a camera field of view capable of photographing a person in the monitoring area 13A. Also, the reference PTZ of the camera 12 is set so as to have a camera field of view capable of photographing a person in the monitoring area 13C. The control device 20 monitors and displays on the screen display unit the images continuously captured by the camera 11 fixed to the reference PTZ. When the operator designates a person to be tracked on the monitored image, the control device 20 tracks the designated person while changing the PTZ value of the camera 11. Then, the control device 20 calculates in real time the information on the movement trajectory of the tracked person (tracking target movement trajectory information) in the image coordinate system of the camera 12.

[0019] On the other hand, based on the images continuously captured by the camera 12 fixed to the reference PTZ, the control device 20 calculates in real time the information on the movement trajectories of all the persons in the monitoring area 13C (collated movement trajectory information) in the image coordinate system of the camera 12. Then, the control device 20 collates in real time the tracking target movement trajectory information of the person being tracked by the camera 11 and the collated movement trajectory information of all the persons shown in the camera 12. Thereby, the control device 20 determines which person among the persons shown in the camera 12 is the person being tracked by the camera 11. Next, the control device 20 continues tracking with the camera 12 while capturing the determined person within the shooting range of the camera 12, and monitors and displays the image on the screen display unit.

[0020] The above is the outline of the tracking system 10. Subsequently, the control device 20 will be described in detail.

[0021] FIG. 2 is a block diagram showing an example of the control device 20. Referring to FIG. 2, the control device 20 includes a communication I / F (interface) unit 21, an operation input unit 22, a screen display unit 23, a storage unit 24, and an arithmetic processing unit 25.

[0022] The communication I / F unit 21 is composed of a data communication circuit and is configured to perform data communication with the cameras 11 and 12 and other external devices (not shown) by wire or wirelessly. The operation input unit 22 is composed of an operation input device such as a keyboard or a mouse, and is configured to detect an operator's operation and output it to the arithmetic processing unit 25. The screen display unit 23 is composed of a display device such as an LCD (Liquid Crystal Display), and is configured to display images obtained by photographing with the cameras 11 and 12.

[0023] The storage unit 24 is composed of one or more storage devices such as a hard disk or a memory, and is configured to store processing information and a program 241 necessary for various processes in the arithmetic processing unit 25. The program 241 is a program that realizes various processing units when read and executed by the arithmetic processing unit 25, and is pre-read from an external device (not shown) or a recording medium via a data input / output function such as the communication I / F unit 21 and stored in the storage unit 24. The main processing information stored in the storage unit 24 includes camera calibration information 242-1 and 242-2, reference images 243-1 and 243-2, image DBs (databases) 244-1 and 244-2, tracking target movement trajectory information 245, and collated object movement trajectory information 246.

[0024] The camera calibration information 242-1 is the camera calibration information of the camera 11 (internal parameters and external parameters of the camera 11). The camera calibration information 242-2 is the camera calibration information of the camera 12 (internal parameters and external parameters of the camera 12).

[0025] The reference image 243-1 is an image of the monitoring area 13A taken by the camera 11 set to the pan value, tilt value, and zoom value when obtaining the camera calibration information 242-1. In addition to the image, the reference image 243-1 includes the pan value, tilt value, and zoom value (the reference PTZ value of the camera 11) of the camera 11 at the time of taking the reference image, and the camera position (the position of the camera 11 in the world coordinate system). Also, the reference image 243-2 is an image of the monitoring area 13C taken by the camera 12 set to the pan value, tilt value, and zoom value when obtaining the camera calibration information 242-2. In addition to the image, the reference image 243-2 includes the pan value, tilt value, and zoom value (the reference PTZ value of the camera 12) at the time of taking the reference image, and the camera position (the position of the camera 11 in the world coordinate system).

[0026] The image DB 244-1 accumulates the time series of images taken by the camera 11. The individual images stored in the image DB 244-1 are added with the camera ID of the camera 11, the shooting time, and the PTZ value at the time of shooting. That is, the image data sent from the camera 11 to the arithmetic processing unit 25 through the communication I / F unit 21 includes the camera ID of the camera 11, the image taken by the camera 11, the time of shooting the image, and the attitude information (pan value, tilt value, zoom value) of the camera 11 at the time of shooting the image. The image DB 244-2 accumulates the time series of images taken by the camera 12. The individual images stored in the image DB 244-2 are added with the camera ID of the camera 12, the shooting time, and the PTZ value at the time of shooting.

[0027] The tracking target movement trajectory information 245 is information regarding the movement trajectory of a person to be tracked, which is calculated based on the image captured by the camera 11. FIG. 3 is a diagram showing a configuration example of the tracking target movement trajectory information 245. The tracking target movement trajectory information 245 in this example is composed of an entry 2451 constituted by a tracking target ID that uniquely identifies the tracking target, a camera ID of the camera 11, and association information, and a plurality of entries 2452 that correspond one-to-one to a plurality of images continuously captured by the camera 11. The plurality of entries 2452 are connected in a row in the order of shooting times by the association information in the entry 2451 and the association information in the entry 2452.

[0028] Each entry 2452 is composed of a shooting time, a person area, pose information, position information (image coordinate system of the camera 11), image features, position information (image coordinate system of the camera 12), a movement vector, a movement speed, an acceleration, and association information. The shooting time represents the shooting time of the corresponding image. The person area represents, for example, the circumscribed rectangle of the person area in the corresponding image. The pose information represents the pose of the camera 11 when the corresponding image is captured. The method for obtaining this pose information will be described later. The position information (image coordinate system of the camera 11) represents the coordinate value in the image coordinate system of the camera 11 of a point representing the person area in the same entry. As a point representing the person area, for example, the center of gravity of the person area can be considered, but it is not limited thereto, and it may be the center of gravity of the face, the feet, etc. The image features represent the feature amounts of the image extracted from the person area in the corresponding image. As the image features, the feature amounts of the face, the feature amounts of the clothing, the size of the person, etc. can be considered, but it is not limited thereto. The position information (image coordinate system of the camera 12) represents the position information in the image coordinate system of the camera 12 corresponding to the position information (image coordinate system of the camera 11) in the same entry. The movement vector represents the movement amount and the movement direction of the person to be tracked between the corresponding image and the adjacent image. The movement speed represents the movement speed of the person to be tracked. The acceleration represents the acceleration of the person to be tracked. The movement vector, the movement speed, and the acceleration are calculated based on, for example, the position information (image coordinate system of the camera 12) in a plurality of adjacent entries 2452. The types of data included in the entry 2452 are an example and are not limited to the above.

[0029] The collated movement trajectory information 246 is information regarding the movement trajectory of a person extracted from the image captured by the camera 12. The collated movement trajectory information 246 exists for each person extracted from the image captured by the camera 12. FIG. 4 is a diagram showing a configuration example of the collated movement trajectory information 246. The collated movement trajectory information 246 in this example is composed of an entry 2461 constituted by a person ID that uniquely identifies a person, a camera ID of the camera 12, and association information, and a plurality of entries 2462 that correspond one-to-one to a plurality of images continuously captured by the camera 12. The plurality of entries 2462 are connected in a row in the order of shooting time by the association information in the entry 2461 and the association information in the entry 2462.

[0030] Each individual entry 2462 is composed of a shooting time, a person area, posture information, position information (image coordinate system of the camera 12), image features, a movement vector, a movement speed, an acceleration, and association information. The shooting time represents the shooting time of the corresponding image. The person area represents, for example, the circumscribed rectangle of the person area in the corresponding image. The posture information represents the posture of the camera 12 when the corresponding image was taken. The position information (image coordinate system of the camera 12) represents the coordinate value in the image coordinate system of the camera 11 of a single point representing the person area in the same entry. The single point representing the person area is defined in the same way as the position information (image coordinate system of the camera 11) in the entry 2452 of FIG. 3. The image features represent the feature amount of the image extracted from the person area in the corresponding image. The movement vector represents the amount and direction of movement of the target person between the corresponding image and the adjacent image. The movement speed represents the movement speed of the target person. The acceleration represents the acceleration of the target person. The movement vector, the movement speed, and the acceleration are calculated based on, for example, the position information (image coordinate system of the camera 12) in a plurality of adjacent entries 2462. The types of data included in the entry 2462 are an example and are not limited to the above.

[0031] The arithmetic processing unit 25 has one or more microprocessors such as an MPU and its peripheral circuits, and is configured to read and execute the program 241 from the storage unit 24, so as to cooperate the above hardware and the program 241 to realize various processing units. The main processing units realized by the arithmetic processing unit 25 include a camera calibration unit 251 and two monitoring units 252 and 253.

[0032] The camera calibration unit 251 is configured to obtain the camera calibration information of the cameras 11 and 12 through interactive processing with the operator via the operation input unit 22 and the screen display unit 23. The camera calibration method to be used is not particularly limited.

[0033] The monitoring unit 252 is configured to detect and track the persons moving within the monitoring areas 13A to 13C using the camera 11. The monitoring unit 252 includes a detection unit 2521, a tracking unit 2522, and a coordinate conversion unit 2523.

[0034] The detection unit 2521 is configured to detect a specific person as a tracking target from all the persons existing within the monitoring area 13A based on the images continuously captured by the camera 11 set to the reference PTZ value.

[0035] The tracking unit 2522 calculates the information of the movement trajectory of the tracking target in the image coordinate system of the camera 11 while tracking the tracking target captured by the camera 11 so as to capture the tracking target at a predetermined position within the shooting range of the camera 11, and stores it in the storage unit 24 as the tracking target movement trajectory information 245. The above predetermined position can be the center of the shooting range, but is not limited thereto, and may be a position other than the center of the shooting range as long as it is the same in a plurality of images. Since the tracking unit 2522 changes the PTZ value of the camera 11 to track the tracking target, even if the tracking target moves from the monitoring area 13A to the monitoring area 13B or the monitoring area 13C, the tracking target can be captured at a predetermined position within the shooting range of the camera 11.

[0036] The coordinate conversion unit 2523 is configured to convert the position information (image coordinate system of camera 11) in the entry 2452 of the tracking target movement trajectory information 245 calculated by the tracking unit 2522 into the position information (image coordinate system of camera 12).

[0037] FIG. 5 is a schematic diagram for explaining an example of a method for converting the position information (image coordinate system of camera 11) into the position information (image coordinate system of camera 12). In FIG. 5, the image I0 is an image obtained by photographing with camera 12 set to the same PTZ value as when the camera calibration information 242-2 was calculated. The images I1, I2, and I3 are a plurality of images of camera 11 obtained by continuously photographing while changing the PTZ value.

[0038] For the conversion of the position information P1 on the image I1 into the image coordinate system of camera 12, the coordinate conversion unit 2523 first calculates a planar projective transformation matrix H1 from the image I1 to the image I0. Next, the coordinate conversion unit 2523 uses the calculated planar projective transformation matrix H1 to calculate the position information P'1 on the image I0 corresponding to the position information P1 on the image I1.

[0039] Generally, the planar projective transformation matrix from camera 11 that captures the same plane to camera 12 can be obtained if the translation and rotation between the cameras are known. The translation between camera 11 and camera 12 can be obtained from the installation positions of camera 11 and camera 12. Also, the rotation between the cameras can be obtained from the posture when camera 11 captures image I1 and the posture when camera 12 captures image I0. Therefore, the coordinate conversion unit 2523 acquires the installation position of camera 11 stored in the reference image 243-1 and the installation position of camera 12 stored in the reference image 243-2, respectively, and calculates the translation between camera 11 and camera 12. Also, the coordinate conversion unit 2523 acquires the posture information of camera 11 stored in the entry 2452 corresponding to image I1 and the posture information when camera 12 captures image I0 stored in the reference image 243-2, respectively, and calculates the rotation between camera 11 and camera 12. Then, the coordinate conversion unit 2523 calculates the planar projective transformation matrix H1 based on the translation and rotation between camera 11 and camera 12 thus obtained.

[0040] Note that the method for calculating the planar projective transformation matrix H1 is not limited to the above. For example, the coordinate conversion unit 2523 may obtain the corresponding point group between image I1 and image I0 using a known feature point extraction method, and calculate the planar projective transformation matrix H1 from the obtained corresponding point group. The corresponding point group between image I1 and image I0 is, in other words, a pair of a plurality of corresponding feature points between image I1 and image I0.

[0041] The coordinate conversion unit 2523 calculates the planar projective transformation matrix H2 from image I2 to image I0 by the same method as above, and uses this planar projective transformation matrix H1 to calculate the position information P'2 on image I0 corresponding to the position information P2 on image I2. Also, the coordinate conversion unit 2523 calculates the planar projective transformation matrix H3 from image I3 to image I0, and uses this planar projective transformation matrix H3 to calculate the position information P'3 on image I0 corresponding to the position information P3 on image I3. The time series of the position information P'1, position information P'2, and position information P'3 calculated in this way is the movement trajectory in the image coordinate system of the target camera 12 being tracked.

[0042] Note that the coordinate conversion unit 2523 may have a function of converting the time series of the positions on the image of the camera 12 obtained by the above conversion into the time series of the positions in the world coordinate system using the camera calibration information of the camera 12. This is used when, for example, collating the tracking target movement trajectory information 245 and the collated movement trajectory information 246 in the world coordinate system. Here, the coordinate conversion unit 2523 may use a plane (common map) where z = 0 (that is, height 0) as the world coordinate system. Compared with the configuration of converting a point on the image coordinate system into a three-dimensional world coordinate system, the conversion into a two-dimensional world coordinate system reduces the amount of calculation and error. However, the coordinate conversion unit 2523 may be configured to convert a point in the image coordinate system into a three-dimensional world coordinate system.

[0043] Referring to FIG. 2 again, the monitoring unit 253 is configured to detect a person being tracked by the camera 11 from among the persons in the monitoring area 13C using the camera 12, and track the detected person in the monitoring areas 13C and 13D. The monitoring unit 253 includes a collation unit 2531, a tracking unit 2532, and a coordinate conversion unit 2533.

[0044] The collation unit 2531 is configured to calculate, in the image coordinate system of the camera 12, information on the movement trajectory (collated movement trajectory information 246) for each person moving within the monitoring area 13C from a plurality of images of the monitoring area 13C continuously captured by the camera 12 set to the reference PTZ value.

[0045] The collation unit 2531 is also configured to collate the collated movement trajectory information 246 with the tracking target movement trajectory information 245 of the person being tracked by the monitoring unit 252. Further, the collation unit 2531 is configured to determine, based on the result of the collation, which person among the persons moving within the monitoring area 13C captured by the camera 12 is the person being tracked by the camera 11.

[0046] An example of a method for collating the tracking target movement trajectory information 245 and the collated movement trajectory information 246 in the collation unit 2531 will be described.

[0047] First, the matching unit 2531 targets the latest N frames of each trajectory for matching. Here, N is a predetermined value of 2 or more. Therefore, the matching unit 2531 extracts N entries 2452 in order from the last entry 2452 of the tracking target movement trajectory information 245, and uses these N extracted entries 2452 as the movement trajectory information of the tracking target. If there are not N entries 2452 in the tracking target movement trajectory information 245, the matching at that time is not performed. On the other hand, from each of one or more movement trajectory information 246 to be matched, the matching unit 2531 extracts N entries 2462 in order from the last entry 2462, and uses these N extracted entries 2462 as the movement trajectory information to be matched. The matching unit 2531 does not perform the matching at that time for the movement trajectory information 246 to be matched that does not have N entries 2462. If there is no movement trajectory information 246 to be matched that has N entries 2462, the matching at that time is not performed.

[0048] Next, the matching unit 2531 calculates the degree of shape matching and the degree of direction matching between the movement trajectory information of the tracking target and the movement trajectory information of each individual object to be matched.

[0049] In calculating the degree of shape matching, the matching unit 2531 first normalizes each movement trajectory with the center of gravity to eliminate the influence of the absolute position. Here, the movement trajectory of the tracking target is formed by connecting the position information (image coordinate system of camera 12) in the N entries 2452 that make up the movement trajectory information of the tracking target with line segments in the order of shooting time. Also, the movement trajectory to be matched is formed by connecting the position information (image coordinate system of camera 12) in the N entries 2462 that make up the movement trajectory information to be matched with line segments in the order of shooting time. Next, the matching unit 2531 obtains the error between the positions at the same time of the movement trajectories of the tracking target and the object to be matched using the least squares method or the like, and calculates a value that becomes higher as the error becomes smaller as the degree of shape matching.

[0050] Also, in calculating the degree of direction matching, the matching unit 2531 calculates a value that becomes higher as the total error between the movement vectors at the same time of the movement trajectories of the tracking target and the object to be matched becomes smaller as the degree of direction matching.

[0051] Next, the matching unit 2531 calculates a matching result from the shape matching degree and the orientation matching degree. For example, the matching unit 2531 may use, as the matching result, a value obtained by multiplying the shape matching degree and the orientation matching degree. Alternatively, the matching unit 2531 may use, as the matching result, a value obtained by adding the shape matching degree and the orientation matching degree, for example.

[0052] The above is an example of a method for the matching unit 2531 to match the tracking target movement trajectory information 245 and the matched movement trajectory information 246. However, the matching method is not limited to the above. For example, the matching unit 2531 may match the two pieces of movement trajectory information in consideration of the image features, movement speed, and acceleration included in the entry 2452 of the tracking target movement trajectory information 245 and the entry 2462 of the matched movement trajectory information 245.

[0053] Based on the result of matching the tracking target movement trajectory information 245 and one or more pieces of matched movement trajectory information 246 as described above, the matching unit 2531 is configured to determine which person among the persons moving within the monitoring area 13C photographed by the camera 12 is the person being tracked by the camera 11. For example, if the highest matching result is equal to or greater than a certain threshold value, the matching unit 2531 determines that the person related to the matched movement trajectory information 246 with the highest matching result is the person being tracked by the camera 11. On the other hand, for example, if the highest matching result is not equal to or greater than a certain threshold value, the matching unit 2531 determines that the person being tracked by the camera 11 does not exist among the persons photographed by the camera 12.

[0054] The tracking unit 2532 is configured to track with the camera 12 so as to capture the tracking target (the person being tracked by the camera 11 shown in the camera 12) within the shooting range. Since the tracking unit 2532 changes the PTZ value of the camera 12 to track the tracking target, even if the tracking target moves from the monitoring area 14C to the monitoring area 13D, the tracking target can be captured within the shooting range.

[0055] The coordinate conversion unit 2533 is configured to convert the collated moving trajectory information 246 calculated by the collation unit 2531 from the image coordinate system of camera 12 to the world coordinate system using the camera calibration information 242-2. That is, the coordinate conversion unit 2533 is configured to convert the position information in entry 2462 (image coordinate system of camera 12) to the position information in the world coordinate system using the camera calibration information 242-2 and add it to entry 2462. Here, the coordinate conversion unit 2533 uses a plane (common map) where z = 0 (i.e., height 0) as the world coordinate system, similar to the coordinate conversion unit 2523. This coordinate conversion unit 2533 is used when collating the collated moving trajectory information 246 and the tracked object moving trajectory information 245 in the world coordinate system and the like.

[0056] Subsequently, a method for calculating the attitude information to be set in each entry 2452 of the tracked object moving trajectory information 245 will be described.

[0057] An entry 2452 of the tracked object moving trajectory information 245 is newly generated every time a new image is acquired from camera 11. As described above, in addition to the image, the PTZ values of camera 11 when the image was taken are added to the image data sent from camera 11. Therefore, the first method for calculating the attitude information is that the monitoring unit 252 calculates the attitude information to be set in the entry based on the pan value and tilt value in the PTZ values added to the acquired image.

[0058] The second method for calculating the posture information is that the monitoring unit 252 calculates the motion vector of the entire image from each of the plurality of images acquired from the camera 11, and calculates the posture information of the camera 11 based on the calculated motion vector. The monitoring unit 252 may use any method for calculating the motion vector of the entire image. For example, the monitoring unit 252 may obtain a corresponding point group between images based on a plurality of feature points obtained by existing feature point extraction, and calculate the motion vector of the entire image based on the obtained corresponding point group. Alternatively, the monitoring unit 252 may obtain a corresponding point group between images based on each lattice point obtained by dividing the entire image into a mesh shape, and calculate the motion vector of the entire image based on the obtained corresponding point group. The frame interval between the two images for calculating the motion vector may be 1 or 2 or 3 or more. This second method is a method for pseudo-calculating a hardware-based camera posture change in an image-based manner.

[0059] Subsequently, the operation of the control device 20 will be described. First, the camera calibration performed before the system operation will be described.

[0060] The calibration unit 251 of the control device 20 calibrates the cameras 11 and 12 at an arbitrary time point before the system operation, such as when the cameras 11 and 12 are installed.

[0061] In the calibration of the camera 11, the calibration unit 251 first adjusts the PTZ values of the camera 11 so that the entire monitoring area 13A can be photographed. Next, the calibration unit 251 calculates the camera calibration information of the camera 11 using a predetermined camera calibration method through interactive processing with the operator through the operation input unit 22 and the screen display unit 23. The calibration unit 251 stores the calculated camera calibration information in the storage unit 24 as the camera calibration information 242-1. The calibration unit 251 also acquires the PTZ value of the camera 11 when the calibration is performed as the reference PTZ value. Further, the calibration unit 251 acquires an image obtained by photographing the area 13A with the camera 11 set to the reference PTZ value as the reference image. The calibration unit 251 adds the reference PTZ value to the acquired reference image and stores it in the storage unit 24 as the reference image 243-1.

[0062] In the calibration of camera 12, the calibration unit 251 first adjusts the PTZ values of camera 12 so that the entire monitoring area 13C (the common area with camera 11) can be photographed. Next, the calibration unit 251 calculates the camera calibration information of camera 12 using a predetermined camera calibration method through interactive processing with the operator via the operation input unit 22 and the screen display unit 23. The calibration unit 251 stores the calculated camera calibration information in the storage unit 24 as camera calibration information 242-2. Further, the calibration unit 251 acquires the PTZ value of camera 12 at the time of calibration as the reference PTZ value. Furthermore, the calibration unit 251 acquires an image obtained by photographing area 15 with camera 12 set to the reference PTZ value as the reference image. The calibration unit 251 adds the reference PTZ value to the acquired reference image and stores it in the storage unit 24 as the reference image 243-2.

[0063] Subsequently, the operation of the control device 20 during system operation will be described. FIG. 6 is a flowchart showing an example of the tracking process performed using camera 11. Hereinafter, with reference to FIG. 6, the tracking process performed by the control device 20 using camera 11 will be described.

[0064] First, the detection unit 2521 in the monitoring unit 252 of the control device 20 performs initialization (step S11). In this initialization, the detection unit 2521 sets the camera 11 to the reference PTZ value. In the initialization, the detection unit 2521 further clears all entries in the tracking target movement trajectory information 245. Next, the detection unit 2521 acquires an image of the monitoring area 13A taken by the camera 11 set to the reference PTZ value, saves it in the image DB244-1, and monitors and displays it on the screen display unit 23 (step S12). Next, the detection unit 2521 uses various methods such as pattern recognition and machine learning to detect all persons from the image saved this time (step S13). Next, the detection unit 2521 displays the detection result on the screen display unit 23 (step S14). For example, the detection unit 2521 generates an image in which the circumscribed rectangle for each person detected from the image is superimposed on the image and displays it on the screen display unit 23. Next, the detection unit 2521 determines whether a person to be tracked has been specified from the operation input unit 22 (step S15). The operator can specify the person to be tracked, for example, by performing an operation such as clicking on the rectangle of the person on the image of the camera 11 displayed on the screen display unit 23. If the person to be tracked is not specified, the detection unit 2521 returns to step S12 and repeats the same processing as the above-described processing. On the other hand, when the person to be tracked is specified, the detection unit 2521 updates the tracking target movement trajectory information 245 (step S16).

[0065] In the update of the tracking target movement trajectory information 245 in step S16, the detection unit 2521 sets the ID assigned to the tracking target person specified by the operator and the camera ID of the camera 11 in the entry 2451. Further, the detection unit 2521 secures one empty entry 2452, and associates the secured entry 2452 with the entry 2451 by setting association information in the secured entry 2452 and the entry 2451. Next, the detection unit 2521 focuses on the secured entry 2452, and sets the shooting time, the person area, the posture information, the position information (image coordinate system of the camera 11), and the image features in the entry 2452 being focused on, and sets the position information (image coordinate system of the camera 12), the movement vector, the movement speed, and the acceleration to NULL values.

[0066] Next, the coordinate conversion unit 2523 of the monitoring unit 252 performs coordinate conversion on the entry 2452 in the tracking target movement trajectory information 245 (step S17). In this step S17, as described with reference to FIG. 5, the coordinate conversion unit 2523 first calculates a planar projective transformation matrix from the image corresponding to the entry 2452 in the annotation to the reference image of camera 12. Next, the coordinate conversion unit 2523 uses the calculated planar projective transformation matrix to calculate the position information in the image coordinate system of camera 12 corresponding to the position information (image coordinate system of camera 11) set in the entry 2452 in the annotation. Next, the coordinate conversion unit 2523 sets the obtained position information (image coordinate system of camera 12) in the entry 25452 in the annotation. Also, in step S17, the coordinate conversion unit 2523 may further convert the position information (image coordinate system of camera 12) into position information in the world coordinate system using the camera calibration information 242-2 and add it to the entry 25452 in the annotation.

[0067] Next, the tracking unit 2522 of the monitoring unit 252 performs tracking control (step S18). For example, the tracking unit 2522 generates a command for controlling the PTZ values of camera 11 according to the position information (image coordinate system of camera 11) set in the entry 2452 in the annotation and transmits it to camera 11 through the communication I / F unit 21. At this time, for example, the tracking unit 2522 may adjust the pan and tilt so that the center of gravity of the circumscribed rectangle of the tracking target person represented by the position information (image coordinate system of camera 11) of the entry 2452 is displayed at the center of the image, and adjust the zoom so that the entire circumscribed rectangle is within a predetermined viewing angle. Camera 11 changes the imaging range by changing the pan, tilt, and zoom in response to the above command.

[0068] Next, the tracking unit 2522 acquires the image captured by the camera 11 set to the changed PTZ value, stores it in the image DB 243-1, and monitors and displays it on the screen display unit 23 (step S19). Next, the tracking unit 2522 detects the person to be tracked from the image saved this time using various methods such as pattern recognition and machine learning (step S20), generates an image with the circumscribed rectangle of the detected person superimposed on the image, and displays it on the screen display unit 23 (step S21). As a result, the operator can confirm the tracking status of the specified person in real time on the image displayed on the screen display unit 23.

[0069] Next, the tracking unit 2522 updates the tracking target movement trajectory information 245 (step S22). In updating the tracking target movement trajectory information 245 in step S22, the tracking unit 2522 first secures one empty entry 2452, sets the association information to the secured empty entry 2452 and the entry of interest 2452, and then moves the attention to the secured empty entry 2452. Next, the tracking unit 2522 sets the shooting time, the person area, the pose information, the position information (image coordinate system of the camera 11), and the image features to the newly focused entry 2452, and sets the position information (image coordinate system of the camera 12), the movement vector, the movement speed, and the acceleration to NULL values.

[0070] Next, the coordinate conversion unit 2523 of the monitoring unit 252 converts the position information (image coordinate system of camera 11) set in the entry 2452 of interest into the position information in the image coordinate system of camera 12 (step S23). The image corresponding to the entry 2452 of interest may have been captured by camera 11 with PTZ values different from those of the previous image. Therefore, as described with reference to FIG. 5, the coordinate conversion unit 2523 first calculates a planar projective transformation matrix from the image corresponding to the entry 2452 of interest to the reference image of camera 12. Next, the coordinate conversion unit 2523 uses the calculated planar projective transformation matrix to calculate the position information in the image coordinate system of camera 12 corresponding to the position information (image coordinate system of camera 11) set in the entry 2452 of interest. Next, the coordinate conversion unit 2523 sets the obtained position information (image coordinate system of camera 12) in the entry 25452 of interest. Also, in step S23, the coordinate conversion unit 2523 calculates a movement vector, a movement speed, and an acceleration based on the position information (image coordinate system of camera 12) of the entry 2452 of interest and the previous entry 2452, and sets them in the entry 2452 of interest. Further, in step S23, the coordinate conversion unit 2523 may convert the position information (image coordinate system of camera 12) into the position information in the world coordinate system using the camera calibration information 242-2 and add it to the entry 25452 of interest.

[0071] Next, the tracking unit 2522 of the monitoring unit 252 determines whether to end the tracking (step S24). For example, when the tracking unit 2522 detects that the person to be tracked has disappeared from the monitoring areas 13A, 13B, and 13C, it may determine to end the tracking. Alternatively, when the tracking unit 2522 detects that the person to be tracked is in a state where it cannot be tracked by camera 11, it may end the tracking. If the tracking unit 2522 does not determine to end the tracking, it returns to step S18 and repeats the same process as described above. As a result, the tracking of the person to be tracked by camera 11 continues, and accordingly, the tracking target movement trajectory information 245 is further updated.

[0072] When the trailing unit 2522 determines that the trailing has ended, it determines whether to end the monitoring (step S25). For example, when a monitoring end command is input from the operator through the operation input unit 22, the trailing unit 2522 determines to end the monitoring. If it is not determined to end the monitoring, the monitoring unit 252 returns to step S11 and repeats the same process as described above. If it is determined to end the monitoring, the monitoring unit 252 ends the process shown in FIG. 6.

[0073] FIG. 7 is a flowchart showing an example of the tracking process performed using the camera 12. Hereinafter, with reference to FIG. 7, the tracking process performed by the control device 20 using the camera 12 will be described.

[0074] First, the collation unit 2531 in the monitoring unit 253 of the control device 20 performs initialization (step S31). In this initialization, the collation unit 2531 sets the camera 12 to the reference PTZ value. Further, in the initialization, the collation unit 2531 also clears all entries of the collated movement trajectory information 246. Next, the collation unit 2531 acquires an image obtained by photographing the monitoring area 13C with the camera 12 set to the reference PTZ value, stores it in the image DB244-2, and displays it on the screen display unit 23 for monitoring (step S32). Next, the collation unit 2531 detects all persons from the image saved this time using various methods such as pattern recognition and machine learning (step S33). Next, the collation unit 2531 displays the detection result on the screen display unit 23 (step S34).

[0075] Next, the collation unit 2531 updates the collated movement trajectory information 245 based on the detection result of the person (step S35). Since all the persons detected from the image of the camera 12 obtained first after the initialization are persons detected for the first time, in step S35, the collation unit 2531 assigns one piece of collated movement trajectory information 246 to each detected person, and performs the following processing for each of the assigned collated movement trajectory information 246.

[0076] First, the matching unit 2531 sets the person ID assigned to the person detected in the entry 2461 of the collated movement trajectory information 246 and the camera ID of the camera 12. Next, the matching unit 2531 secures one empty entry 2462, sets information for mutually associating the secured empty entry 2462 and the entry 2461, and then focuses on the secured empty entry 2462. Next, the matching unit 2531 sets the shooting time, the person area, the posture information, the position information (image coordinate system of the camera 12), and the image features in the entry 2462 being focused on, and sets the movement vector, the movement speed, and the acceleration to NULL values.

[0077] Next, the coordinate conversion unit 2533 of the monitoring unit 253 performs the following processing on the entry 2462 being focused on (step S36). First, the coordinate conversion unit 2533 uses the camera calibration information 442-2 to convert the position information (image coordinate system of the camera 12) set in the entry 2462 being focused on into the position information in the world coordinate system. Next, the coordinate conversion unit 2533 newly adds and sets the position information in the world coordinate system obtained by the above conversion as the position information (world coordinate system) in the entry 2462 being focused on.

[0078] Next, the matching unit 2531 of the monitoring unit 253 collates the tracking target movement trajectory information 245 and one or more collated movement trajectory information 246 (step S37). This collation is performed in the image coordinate system of the camera 12 for the movement trajectory of the tracking target and the collated movement trajectory, but it may also be performed in the world coordinate system. The matching unit 2531 determines, by this collation, which of the persons moving within the monitoring area 13C photographed by the camera 12 is the person being tracked by the camera 11. Next, the matching unit 2531 determines whether or not the detection of the person being tracked by the camera 11 was successful (step S38). If the matching unit 2531 fails in the above detection, it returns to step S32 and repeats the same processing as described above. On the other hand, if the matching unit 2531 succeeds in the above detection, it transmits the person ID of the person determined to be the person being tracked by the camera 11 to the tracking unit 2532.

[0079] The trailing part 2532 focuses on the last entry 2462 of the collated movement trajectory information 246 in which the person ID received from the collation part 2532 is set in the entry 2461, and tracks the person to be tracked using the camera 12. For example, the trailing part 2532 generates a command for controlling the PTZ value of the camera 12 according to the position information (image coordinate system) set in the entry 2462 being focused on, and transmits it to the camera 12 through the communication I / F part 21. At this time, for example, the trailing part 2532 adjusts the pan and tilt so that the centroid of the circumscribed rectangle of the person to be tracked represented by the position information (image coordinate system of the camera 12) in the entry 2462 being focused on is displayed at the center of the image, and adjusts the zoom so that the entire circumscribed rectangle fits within a predetermined viewing angle. The camera 12 changes the imaging range by changing the pan, tilt, and zoom in response to the above command. Also, in step S39, the trailing part 2532 acquires the image captured by the camera 12 after the PTZ change, and performs processes such as detection of the person to be tracked and monitor display on the screen display part 23.

[0080] Furthermore, the trailing part 2532 determines whether to end the tracking (step S40). For example, when the trailing part 2532 detects that the person to be tracked no longer exists in the monitoring areas 13C and 13D, it may determine to end the tracking. Alternatively, when the trailing part 2532 detects that the person to be tracked has become in a state where it cannot be tracked by the camera 12, it may end the tracking. If the trailing part 2532 does not determine to end the tracking, it returns to step S38 and repeats the same process as the above-described process. As a result, the tracking of the person to be tracked by the camera 12 continues. If the trailing part 2532 determines to end the tracking, it determines whether to end the monitoring (step S41). For example, when the trailing part 2532 receives a monitoring end command from the operator through the operation input part 22, it determines to end the monitoring. If it is not determined to end the monitoring, the monitoring part 253 returns to step S31 and repeats the same process as the above-described process. If it is determined to end the monitoring, the monitoring part 253 ends the process shown in FIG. 7.

[0081] According to this embodiment, the tracking unit 2522 acquires the position information of the person to be tracked (image coordinate system of camera 11) from a plurality of images continuously captured while tracking with the camera 11 having a tracking function of capturing a moving person to be tracked within the shooting range, and calculates each entry 2452 of the tracking target movement trajectory information 245. Further, the coordinate conversion unit 2523 calculates a planar projective transformation matrix from the image corresponding to each entry 2452 obtained by shooting with the camera 11 to the reference image of the camera 12, and uses the calculated planar projective transformation matrix to convert the position information (camera 11 image coordinate system) of each entry 2452 of the tracking target movement trajectory information 245 into the position information in the image coordinate system of the camera 12. Unlike the camera calibration information, the above planar projective transformation matrix can be calculated without manual intervention. Therefore, according to this embodiment, it is possible to calculate the movement trajectory information, which is the time series of the position information in the image coordinate system of the camera 12 of the person, while tracking the person with the camera 11 having a tracking function of capturing the moving person within the shooting range. As a result, the movement trajectories of the person can be collated in real time between the camera 11 and the camera 12, and automatic collation of the moving object between the PTZ cameras becomes possible.

[0082] Subsequently, a modified example of this embodiment will be described.

[0083] A computer may be installed in the camera 11, and all or part of the functions of the monitoring unit 252 may be installed on the computer. Similarly, a computer may be installed in the camera 12, and all or part of the functions of the monitoring unit 253 may be installed on the computer. Further, the operation input unit 22, the screen display unit 23, and the storage unit 24 may be installed on a computer connected to the cameras 11 and 12 through a network.

[0084] The present invention may be applied to a system for detecting and tracking a person in a place other than a passage, for example, a store, a factory, a platform of a station, a ground, a stadium, or the like.

[0085] The object to be detected and tracked in the present invention may be any moving object other than a person, for example, an animal, an automobile, a walking robot, or the like.

[0086] The cameras that share a part of the camera field of view used in the present invention are not limited to two cameras, camera 11 and camera 12, and may be three or more cameras.

[0087] [Second Embodiment] Next, a second embodiment of the present invention will be described in detail with reference to the drawings.

[0088] Referring to FIG. 8, the moving trajectory information processing apparatus 30 according to the present embodiment includes a calculation means 31 and a conversion means 32.

[0089] The calculation means 31 is configured to calculate moving trajectory information, which is a time series of the position information of the object on the image and the attitude information of the camera, from a plurality of images in which the object is continuously photographed by a camera that captures the moving object in the photographed image. The calculation means 31 can be configured in the same manner as, for example, the trailing part 2522 in FIG. 2, but is not limited thereto.

[0090] The conversion means 32 is configured to calculate a planar projective transformation matrix from the image to a predetermined image, and use the calculated planar projective transformation matrix to convert the moving trajectory information into moving trajectory information that is a time series of the position information of the object on the predetermined image. The conversion means 32 can be configured in the same manner as, for example, the coordinate conversion unit 2523 in FIG. 2, but is not limited thereto.

[0091] The moving trajectory information processing apparatus 30 configured as described above operates as follows. That is, the calculation means 31 calculates moving trajectory information, which is a time series of the position information of the object on the image and the attitude information of the camera, from a plurality of images in which the object is continuously photographed by a camera that captures the moving object in the photographed image. Next, the conversion means 32 calculates a planar projective transformation matrix from the image to a predetermined image, and uses the calculated planar projective transformation matrix to convert the moving trajectory information into moving trajectory information that is a time series of the position information of the object on the predetermined image.

[0092] According to the moving trajectory information processing apparatus 30 configured and operating as described above, it is possible to calculate the moving trajectory information of the object from a plurality of images continuously captured by a camera that captures the moving object within the imaging range. The reason is that the above planar projective transformation matrix can be calculated without manual intervention, unlike the camera calibration information.

[0093] As described above, the present invention has been described with reference to the above embodiments. However, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0094] For example, in addition to the purpose of monitoring and tracking, the moving trajectory of the object calculated according to the present invention may be used for the purpose of performing personal authentication or detecting abnormal behavior based on the moving trajectory. Further, the camera used in the present invention does not necessarily have to have a tracking function. For example, the present invention is also applicable when the operator manually changes the posture of the camera so as to capture the moving object within the imaging range and calculates the moving trajectory information of the object from a plurality of images continuously captured while tracking the object.

Industrial Applicability

[0095] It can be used in a system that detects and tracks an object from a plurality of images continuously captured by a camera that captures a moving object such as a PTZ camera within the imaging range.

[0096] Some or all of the above embodiments may be described as follows in the following supplementary notes, but are not limited thereto. [Supplementary Note 1] Calculating means for calculating moving trajectory information, which is a time series of the position information of the object on the image and the posture information of the camera, from a plurality of images continuously captured by a camera that captures a moving object within the imaging image; Converting means for calculating a planar projective transformation matrix from the image to a predetermined image, and converting the moving trajectory information using the calculated planar projective transformation matrix into moving trajectory information, which is a time series of the position information of the object on the predetermined image of the object; A moving trajectory information processing apparatus comprising [Appendix 2] The calculating means calculates a motion vector of the entire image from each of the plurality of images, and calculates a time series of the posture information of the camera based on the calculated motion vector. The moving trajectory information processing apparatus according to Appendix 1. [Appendix 3] The predetermined image is an image captured by another camera installed so that a part of the camera field of view overlaps with the camera. The moving trajectory information processing apparatus according to Appendix 1 or 2. [Appendix 4] Collation means for collating the moving trajectory information obtained by conversion by the conversion means with other moving trajectory information which is a time series of position information of an object to be collated on the predetermined image calculated from a plurality of images continuously captured by the other camera is further provided. The moving trajectory information processing apparatus according to Appendix 3. [Appendix 5] The camera is a PTZ camera. The moving trajectory information processing apparatus according to any one of Appendices 1 to 4. [Appendix 6] Moving trajectory information is calculated from a plurality of images continuously captured by a camera that captures a moving object within a captured image, which is a time series of position information of the object on the image of the object and posture information of the camera. A planar projective transformation matrix from the image to a predetermined image is calculated. Using the calculated planar projective transformation matrix, the moving trajectory information is converted into moving trajectory information which is a time series of position information of the object on the predetermined image of the object. Moving trajectory information processing method. [Appendix 7] In the calculation of the time series of the posture information of the camera, a motion vector of the entire image is calculated from each of the plurality of images, and a time series of the posture information of the camera is calculated based on the calculated motion vector. The moving trajectory information processing method according to Appendix 6. [Appendix 8] The predetermined image is an image captured by another camera installed so that a part of the camera field of view overlaps with the camera. The moving trajectory information processing method according to Appendix 6 or 7. [Appendix 9] Furthermore, the moving trajectory information obtained by the conversion is collated with other moving trajectory information that is a time series of the position information of the object to be collated on the predetermined image calculated from a plurality of images continuously captured by the other camera. The moving trajectory information processing method according to Appendix 8. [Appendix 10] A computer A process of calculating moving trajectory information that is a time series of the position information of the object on the image of the object and the posture information of the camera from a plurality of images continuously captured by a camera that captures a moving object within a captured image, A process of calculating a planar projective transformation matrix from the image to a predetermined image, A process of converting the moving trajectory information into moving trajectory information that is a time series of the position information of the object on the predetermined image of the object using the calculated planar projective transformation matrix, A computer-readable recording medium recording a program for causing the above to be performed.

Explanation of Signs

[0097] 10 Tracking system 11, 12 PTZ camera 13A~13D Monitoring area 20 Control device 21 Communication I / F unit 22 Operation input unit 23 Screen display unit 24 Memory unit 25 Arithmetic processing unit 30 Moving trajectory information processing device 31 Calculation means 32 Conversion means 241 Program 242-1, 242-2 Camera calibration information 243-1, 243-2 Reference image 244-1, 244-2 Image DB 245 Tracking target movement trajectory information 246 Collated movement trajectory information 251 Camera calibration unit 252, 253 Monitoring units 2521 Detection unit 2522, 2532 Tracking parts 2523, 2533 Coordinate conversion parts 2531 Collation unit

Claims

1. Calculation means for calculating movement trajectory information including a time series of position information of the object in the image and a time series of the posture information of the camera from a plurality of images continuously captured by a camera that captures a moving object within a captured image; Using an image captured by another camera installed so that a part of the camera field of view overlaps with the camera as a predetermined image, calculating a planar projective transformation matrix from the image to the predetermined image, and using the calculated planar projective transformation matrix to convert the movement trajectory information into movement trajectory information including a time series of position information of the object on the predetermined image; A movement trajectory information processing apparatus comprising:

2. The calculation means calculates a motion vector of the entire image from each of the plurality of images, and calculates a time series of the posture information of the camera based on the calculated motion vector. The movement trajectory information processing apparatus according to Claim 1.

3. The apparatus further comprises collation means for collating the movement trajectory information obtained by conversion by the conversion means with other movement trajectory information which is a time series of position information of the object to be collated on the predetermined image calculated from a plurality of images continuously captured by the other camera. The movement trajectory information processing apparatus according to Claim 2.

4. The camera is a PTZ camera. The movement trajectory information processing apparatus according to any one of Claims 1 to 3.

5. Calculating movement trajectory information including a time series of position information of the object in the image and a time series of the posture information of the camera from a plurality of images continuously captured by a camera that captures a moving object within a captured image; Using an image captured by another camera installed so that a part of the camera field of view overlaps with the camera as a predetermined image, calculating a planar projective transformation matrix from the image to the predetermined image; Converting the movement trajectory information into movement trajectory information including a time series of position information of the object on the predetermined image using the calculated planar projective transformation matrix. A movement trajectory information processing method.

6. In the calculation of the time series of the posture information of the camera, a motion vector of the entire image is calculated from each of the plurality of images, and a time series of the posture information of the camera is calculated based on the calculated motion vector. The movement trajectory information processing method according to Claim 5.

7. Furthermore, collate the movement trajectory information obtained by the conversion with other movement trajectory information, which is a time series of the position information on the predetermined image of the object to be collated, calculated from a plurality of images continuously captured by the other camera. The movement trajectory information processing method according to claim 6.

8. On a computer, a process of calculating movement trajectory information including a time series of position information on the image of the object and a time series of the posture information of the camera from a plurality of images obtained by continuously capturing the object with a camera that captures the moving object within the captured image; a process of setting an image captured by another camera installed so that a part of the camera field of view overlaps with the camera as a predetermined image and calculating a planar projective transformation matrix from the image to the predetermined image; a process of converting the movement trajectory information into movement trajectory information including a time series of position information on the predetermined image of the object using the calculated planar projective transformation matrix; A program for causing the above to be performed.

Images