Mobile information processing device, system, program, and method

Abstract

A mobile object information processing device and the like capable of continuously tracking an unspecified number of mobile objects moving over a relatively wide area is provided. [Solution] This mobile object information processing device includes a mobile object detection unit that detects mobile objects from multiple frame images of an image signal, a coordinate conversion unit that determines coordinate information of the mobile object in a predetermined coordinate system based on position information on the image of the detected mobile object, a mobile object tracking unit that determines a correspondence between at least one mobile object detected from a first image signal obtained by imaging a first spatial region and at least one mobile object detected from a second image signal obtained by imaging a second spatial region based on the coordinate information of the mobile object in the predetermined coordinate system, and a tracking result processing unit that determines information regarding the position, etc. of at least one mobile object across multiple spatial regions based on coordinate information in the predetermined coordinate system of the mobile object determined to correspond by the mobile object tracking unit.

Description

[Technical Field]

[0001] The present invention relates to a mobile object information processing device and a mobile object information processing method for obtaining information about the position or movement path of a mobile object such as an automobile based on an image signal. Furthermore, the present invention relates to a mobile object information processing system including such a mobile object information processing device, and a mobile object information processing program used in the mobile object information processing device. In this application, the mobile object corresponds to an automobile including an automated guided vehicle or an autonomous vehicle, a vehicle including a motorcycle or a bicycle, a forklift, a self-propelled robot, a walking person, a drone, or the like. [Background technology]

[0002] As a technology for obtaining information about the path of a moving object, Patent Document 1 discloses a movement path prediction system that predicts the path of a moving object moving within a predetermined area such as an airport or a factory. A departure checkpoint, a destination checkpoint, and intermediate checkpoints are arranged in the predetermined area, and the moving object moves within the predetermined area from the departure checkpoint to the destination checkpoint via the intermediate checkpoints. Information about the movement path of the moving object is measured by a movement measurement system.

[0003] This movement path prediction system includes a measurement data acquisition unit that acquires movement path information from a movement measurement system, a map data input unit that acquires location information of multiple checkpoints, a workflow input unit that sets route information for multiple checkpoints through which a moving object will move, a movement path prediction model learning unit that machine-learns a movement path prediction model that calculates a first objective variable from a first feature amount, and a movement path prediction unit that simulates the path of the moving object based on the movement path prediction model.

[0004] According to Patent Document 1, for example, when simulating the movement of people within an airport after a layout change, the accuracy of the movement simulation can be improved by dividing the movement route information into multiple sections and machine learning a movement route prediction model for each divided movement route information.

[0005] Furthermore, Patent Document 2 discloses a position estimation system that aims to perform position estimation with good accuracy by using two-dimensional images captured by a monocular imaging device, without using a stereo camera, which is difficult to miniaturize and reduce the cost of hardware.

[0006] This position estimation system includes a monocular imaging device, a memory unit that stores a feature map that indicates a feature space in which real space is expressed by feature values ​​according to predetermined feature types, and a position estimation unit that estimates a position corresponding to the imaging device based on the result of comparing a reference image that indicates the feature values ​​of an image captured by the imaging device with a projection image that is generated according to a projection center set in the feature space of the feature map.

[0007] The accuracy of a 3D point cloud generated from images captured by a monocular imaging device is insufficient, resulting in a decrease in the accuracy of position estimation. In contrast, the position estimation system of Patent Document 2 uses a feature map, thereby eliminating the need to use a 3D point cloud. This makes it possible to estimate the self-position of a moving object with good accuracy, even when using a monocular imaging device. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2020-123128 A (paragraphs 0008 and 0143) [Patent Document 2] JP 2020-173617 A (paragraphs 0004-0006, 0012, 0025) DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0009] However, with the movement path prediction system of Patent Document 1, when measuring the movement paths of an unspecified number of automobiles passing through a relatively wide area for traffic volume surveys, for example, it becomes difficult to measure the movement path of an automobile if the automobile moves far away or enters a blind spot, making it difficult to continuously track the automobile. Also, the position estimation system of Patent Document 2 is a system for self-position estimation installed inside a moving body such as a vehicle or robot, and is not suitable for obtaining information on the position or movement path of a moving body using an imaging device installed outside the moving body.

[0010] In view of the above, a first object of the present invention is to provide a mobile object information processing device or method capable of continuously tracking an unspecified number of mobile objects moving over a relatively wide area. A second object of the present invention is to provide a mobile object information processing device or method capable of accurately determining information relating to the position or path of a mobile object using an imaging device disposed outside the mobile object. A third object of the present invention is to provide a mobile object information processing system including such a mobile object information processing device, or a mobile object information processing program or the like used in the mobile object information processing device. [Means for solving the problem]

[0011] In order to solve at least part of the above problems, a mobile object information processing device according to a first aspect of the present invention is a mobile object information processing device that obtains information about a mobile object based on an image signal obtained from at least one imaging device that captures an image of a spatial region and generates an image signal, and includes a mobile object detection unit that detects a mobile object from images of a plurality of frames of the image signal, a coordinate conversion unit that obtains coordinate information of the mobile object in a predetermined coordinate system based on position information on the image of the mobile object detected from the image signal, and a coordinate conversion unit that determines a correspondence relationship between at least one mobile object detected from a first image signal obtained by capturing an image of a first spatial region and at least one mobile object detected from a second image signal obtained by capturing an image of a second spatial region based on the coordinate information of the mobile object in the predetermined coordinate system. When detecting a moving object whose coordinate distance is within a predetermined range from the images of the first image signal frame and the second image signal frame that are closest in time of imaging among a plurality of frames of the first image signal and the second image signal obtained by imaging a first spatial region and a second spatial region that partially overlap each other, information on estimated coordinates calculated based on the position or coordinate of the moving object detected from the images of a predetermined number of other frames of the first or second image signal is used to determine the correspondence relationship of the moving object.The system includes a moving object tracking unit and a tracking result processing unit that determines information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple spatial regions based on coordinate information in the specified coordinate system of a moving object determined to be in correspondence by the moving object tracking unit.

[0012] In the mobile information processing device according to the second aspect of the present invention, ,before The moving object tracking unit Among a plurality of frames of a first image signal and a second image signal obtained by imaging a first spatial region and a second spatial region, respectively, which partially overlap each other, If the distance between the coordinates of two moving objects calculated from the images of the frame of the first image signal and the frame of the second image signal that are closest in time to capture the images is within a predetermined range, it is determined that the two moving objects correspond to each other.

[0013] A mobile object information processing system according to a first or second aspect of the present invention includes, in addition to a mobile object information processing device according to the first or second aspect of the present invention, a plurality of imaging devices including a first imaging device and a second imaging device that capture images of a first spatial region and a second spatial region, respectively, and generate a first image signal and a second image signal, respectively.

[0014] A moving object information processing program according to a first aspect of the present invention is a moving object information processing program used in a moving object information processing device that obtains information about a moving object based on an image signal obtained from at least one imaging device that images a spatial region and generates an image signal, the program comprising: a step (a) of detecting a moving object from images of a plurality of frames of the image signal; a step (b) of obtaining coordinate information of the moving object in a predetermined coordinate system based on position information on the image of the moving object detected from the image signal; and a step (c) of determining, based on the coordinate information of the moving object in the predetermined coordinate system, a correspondence relationship between at least one moving object detected from a first image signal obtained by imaging a first spatial region and at least one moving object detected from a second image signal obtained by imaging a second spatial region. When detecting a moving object whose coordinate distance is within a predetermined range from the images of the first image signal frame and the second image signal frame that are closest in time of imaging among a plurality of frames of the first image signal and the second image signal obtained by imaging a first spatial region and a second spatial region that partially overlap each other, information on estimated coordinates calculated based on the position or coordinate of the moving object detected from the images of a predetermined number of other frames of the first or second image signal is used to determine the correspondence relationship of the moving object.The CPU executes step (c) and step (d) of determining information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving body across multiple spatial regions based on coordinate information in the specified coordinate system of the moving body determined to correspond in step (c).

[0015] A moving object information processing method according to a first aspect of the present invention is a moving object information processing method for determining information about a moving object based on an image signal obtained from at least one imaging device that captures an image of a spatial region and generates an image signal, the method comprising the steps of: (a) detecting a moving object from images of a plurality of frames of the image signal; (b) determining coordinate information of the moving object in a predetermined coordinate system based on position information on the image of the moving object detected from the image signal; and (c) determining, based on the coordinate information of the moving object in the predetermined coordinate system, a correspondence relationship between at least one moving object detected from a first image signal obtained by capturing an image of a first spatial region and at least one moving object detected from a second image signal obtained by capturing an image of a second spatial region. When detecting a moving object whose coordinate distance is within a predetermined range from the images of the first image signal frame and the second image signal frame that are closest in time of imaging among a plurality of frames of the first image signal and the second image signal obtained by imaging a first spatial region and a second spatial region that partially overlap each other, information on estimated coordinates calculated based on the position or coordinate of the moving object detected from the images of a predetermined number of other frames of the first or second image signal is used to determine the correspondence relationship of the moving object. The method includes step (c), and step (d) of determining information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving body across multiple spatial regions based on coordinate information in the predetermined coordinate system of the moving body determined to correspond in step (c). [Effects of the Invention]

[0016] The first aspect of the present invention or second According to the viewpoint of the present invention, a moving object is detected from a plurality of frame images of an image signal obtained from at least one imaging device, coordinate information of the moving object in a predetermined coordinate system is obtained based on position information of the moving object in the image, and a correspondence relationship between the moving object detected in the first spatial region and the moving object detected in the second spatial region is determined based on the coordinate information of the moving object, thereby tracking the moving object across a plurality of spatial regions. This makes it possible to continuously track an unspecified number of moving objects moving in a relatively wide range, or to obtain information on the position or movement path of the moving object with high accuracy using an imaging device disposed outside the moving object.

[0017] Generally, when multiple imaging devices placed outside a moving body are used to capture images of multiple spatial regions and obtain multiple image signals, when the moving body crosses over the images captured by the imaging devices, a large error occurs between the position of the moving body predicted from one image and the position of the moving body detected from the other image, making it difficult to recognize that the moving bodies are the same.

[0018] In such cases and determining a correspondence between the moving object detected in the first spatial region and the moving object detected in the second spatial region based on the distance between the coordinates of the two moving objects calculated from the images of the nearest frames of two image signals obtained by capturing images of the first spatial region and the second spatial region, which partially overlap each other, over a plurality of frames. , complex It is possible to track moving objects over a number of spatial regions. If no moving object is detected within a predetermined distance between coordinates from the images of the closest frames of the two image signals, information on estimated coordinates obtained based on the position or coordinates of the moving object detected from images of a predetermined number of other frames is used to determine the correspondence between the moving objects. This allows the capture of an unspecified number of vehicles passing through a relatively wide area that cannot be captured by a single imaging device. Using multiple imaging devices By continuously tracking, it is also possible to monitor the number of cars passing through the road being surveyed in a traffic volume survey. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a schematic diagram illustrating an example of an overview of a mobile object information processing system according to an embodiment of the present invention; [Figure 2] 2 is a block diagram showing an example of the configuration of the mobile information processing device shown in FIG. 1. FIG. [Figure 3] FIG. 10 is a schematic diagram illustrating an example of a change in the position of a moving object over time. [Figure 4] 3 is a flowchart showing a mobile object information processing method according to the first embodiment of the present invention. [Figure 5] 1 is a schematic diagram for explaining the relationship between camera images obtained from a plurality of imaging devices and a map image in a predetermined coordinate system. [Figure 6]5 is a flowchart showing a specific example of the moving object tracking process in FIG. 4. [Figure 7] 1 is a schematic diagram showing an image in which a moving object designated on a map image is identified in camera image 1 and camera image 2. FIG. [Figure 8] 10 is a flowchart showing a mobile object information processing method according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. <Mobile Information Processing System> Fig. 1 is a schematic diagram illustrating an example of an overview of a mobile object information processing system according to an embodiment of the present invention. Fig. 1 shows a single automobile M as an example of a mobile object, but this system can obtain information about the positions or travel routes of multiple mobile objects.

[0021] For example, this system can obtain information on the positions or routes of an unspecified number of automobiles traveling over a relatively wide area, and can monitor the number of automobiles passing through the roads being surveyed in a traffic volume survey. Furthermore, this system can be widely applied to situations where information on the positions or routes of moving vehicles, forklifts, and self-propelled robots is required in factories, airports, and stores.

[0022] This system includes at least one imaging device that captures an image of a spatial region and generates an image signal (video signal), and a mobile object information processing device 20 that obtains information about the mobile object based on the image signal obtained from the at least one imaging device. As shown in Fig. 1, multiple imaging devices including a first imaging device 11, a second imaging device 12, ... may be used.

[0023] In the example shown in FIG. 1, a first imaging device 11 captures an image of a first spatial region A1 to generate a first image signal, and a second imaging device 12 captures an image of a second spatial region A2 to generate a second image signal. Here, the adjacent first spatial region A1 and second spatial region A2 partially overlap each other. The same applies below. These imaging devices may perform imaging operations in synchronization with each other. Alternatively, multiple image signals obtained by capturing multiple spatial regions may be generated based on an image signal obtained from a single imaging device.

[0024] For example, a plurality of imaging devices including a first imaging device 11, a second imaging device 12, etc. may be placed at approximately equal intervals along a road to be surveyed in order to monitor the number of cars passing through the road in a traffic volume survey. Note that an existing security camera may be used as each imaging device.

[0025] Alternatively, multiple imaging devices with different optical magnifications may be used. For example, a first imaging device 11 that uses a wide-angle lens or a standard lens to capture an image of a moving object at a relatively close distance and a second imaging device 12 that uses a telephoto lens to capture an image of a moving object at a relatively long distance may be arranged within a predetermined distance.

[0026] Each imaging device has a camera unit that captures an image of a subject with an image sensor through an optical system including at least one lens and generates a digital or analog image signal, and a communication unit that communicates wirelessly or wired with the mobile information processing device 20 and transmits the image signal generated by the camera unit to the mobile information processing device 20.

[0027] Alternatively, a recording device 10 may be used that records image signals generated by a camera unit. The recording device 10 may be installed inside each imaging device, or may be installed in a building or the like outside the plurality of imaging devices, as shown in Fig. 1. The image signals recorded in the recording device 10 are supplied to the mobile information processing device 20 via wireless or wired communication, or by using a portable recording medium.

[0028] The recording medium in the recording device 10 or the portable recording medium may be a hard disk, a flexible disk, an optical disk, a magneto-optical disk, a magnetic tape, an SSD (solid state drive), or various types of memory including a USB memory.

[0029] Communication between the first imaging device 11 etc. and the mobile information processing device 20, or communication between the first imaging device 11 etc. and the recording device 10, or communication between the recording device 10 and the mobile information processing device 20 may be performed via a network such as a wired LAN, a wireless LAN, an intranet, the Internet, or a mobile communication network.

[0030] Furthermore, when connecting them by wire, a system such as HDMI (High-Definition Multimedia Interface; registered trademark) or SDI (Serial Digital Interface) may be used.

[0031] The image signals generated by each imaging device or the image signals recorded in the recording device 10 are supplied as measurement information to the mobile object information processing device 20 together with incidental information including identification information of the imaging device or timing information relating to the time of imaging. At least a part of the incidental information may be included in the file name of the image signal or the measurement information.

[0032] The mobile object information processing device 20 obtains information about the mobile object based on an image signal obtained from at least one imaging device that captures an image of a spatial region and generates an image signal. For example, the mobile object information processing device 20 is configured as a server installed in a data center or a control center of a research company or a manufacturing company, or a PC (personal computer) and a display, etc.

[0033] <Mobile information processing device> Fig. 2 is a block diagram showing an example of the configuration of the mobile information processing device shown in Fig. 1. As shown in Fig. 2, the mobile information processing device 20 includes an operation unit 21, a display unit 22, an audio input / output unit 23, a communication circuit 24, an interface 25, a CPU (Central Processing Unit) 26, a cache memory 27, and a storage unit 28. The interface 25 to the storage unit 28 are connected to each other via a bus line. Note that some of the components shown in Fig. 1 or 2 may be omitted or modified, or other components may be added to the components shown in Fig. 1 or 2.

[0034] The operation unit 21 is composed of, for example, a keyboard, a mouse, etc., and is used to input various commands and data to the mobile information processing device 20. The display unit 22 includes, for example, an LCD (liquid crystal display), etc., and displays an operation screen, etc. The audio input / output unit 23 includes, for example, a microphone, an amplifier, a speaker, etc., and converts audio signals into electrical signals and converts electrical signals into audio signals.

[0035] The communication circuit 24 has the function of performing data communication with multiple imaging devices including a first imaging device 11, a second imaging device 12, etc., or with a recording device 10 that records image signals generated by these imaging devices.

[0036] The interface 25 is connected to the operation unit 21 to the communication circuit 24 and external devices such as a portable recording medium, and transmits various commands and data between them and the CPU 26. Data input to the CPU 26 via the interface 25 is written to the cache memory 27 or the storage unit 28.

[0037] The CPU 26 performs various calculations and data processing in accordance with various software (including a mobile object information processing program) stored in the storage unit 28. The cache memory 27 and the storage unit 28 store various data used to obtain information about the mobile object and tracking result information about the mobile object in multiple databases (DB).

[0038] The cache memory 27 is configured with RAM (random access memory), etc. As a recording medium (storage medium) in the storage unit 28, various types of memories including a hard disk, a flexible disk, an optical disk, a magneto-optical disk, a magnetic tape, an SSD, or a ROM (read only memory) can be used.

[0039] Here, the CPU 26 and the mobile object information processing program stored in the storage unit 28 configure an image signal acquisition unit 260, a mobile object detection unit 261, a mobile object position estimation unit 262, a coordinate conversion unit 263, a mobile object tracking unit 264, and a tracking result processing unit 265 as functional blocks.

[0040] The image signal acquisition unit 260 acquires measurement information including an image signal and additional information from at least one imaging device. For example, the image signal acquisition unit 260 may control the communication circuit 24 to receive measurement information from a plurality of imaging devices including a first imaging device 11, a second imaging device 12, etc., or from a recording device 10 that records image signals generated by these imaging devices.

[0041] Alternatively, the image signal acquiring unit 260 may acquire the measurement information from a portable recording medium on which the measurement information is recorded when the recording medium is connected to the interface 25. For example, the image signal acquiring unit 260 stores the measurement information acquired from the first imaging device 11 or the like in the image database of the cache memory 27, and stores the measurement information acquired from the recording device 10 or the portable recording medium in the image database of the storage unit 28.

[0042] The moving body detection unit 261 to the tracking result processing unit 265 obtain information on the position, movement route, movement direction, movement speed, movement time, etc. of at least one moving body based on the image signal included in the measurement information. For example, the position or movement route, etc. of the moving body on a predetermined map are obtained and displayed on the display unit 22.

[0043] <First operation example> Next, a first operation example of the mobile object information processing device shown in Fig. 2 will be described with reference to Fig. 1 to Fig. 7. Fig. 4 is a flowchart showing a mobile object information processing method according to the first embodiment of the present invention. Note that part of the processing shown in Fig. 4 may be omitted or changed, or other processing may be added to the processing shown in Fig. 4.

[0044] <Step S10> In step S10 of Fig. 4, the image signal acquisition unit 260 acquires measurement information including an image signal and accompanying information from the first imaging device 11 or the like, the recording device 10, or a portable recording medium shown in Fig. 1. The image signal acquisition unit 260 stores the acquired measurement information in the image database of the cache memory 27 or the storage unit 28. As a result, image signals of multiple frames obtained from the first imaging device 11 or the like are stored in the image database.

[0045] <Step S11> In step S11, the moving object detection unit 261 reads out the image signals collected by the image signal acquisition unit 260 in step S10 from the image database, and performs moving object detection processing on the image signals obtained from one imaging device to detect moving objects from multiple frame images of the image signals. As a result, position information of the moving object on the image is obtained for each frame of the image signal. This moving object detection processing can use object detection processing using AI (Artificial Intelligence), image recognition processing, etc.

[0046] For example, AI-based object detection processing involves dividing an image frame into many small segments, labeling each segment with information such as what is depicted, and associating a category with it. If an object with the same label or category is detected in two consecutive frames with a position shift within a specified distance, it is recognized as the same object having moved. This makes it possible to detect moving objects whose position changes across multiple frames of an image signal.

[0047] Alternatively, the moving object detection unit 261 may detect a moving object from multiple frames of images by performing a predetermined image recognition process on image signals of multiple frames obtained by a single imaging device. In this image recognition process, for example, an active appearance model can be used, which separates the image of the target object into shape and texture, and compresses the dimensions of each using principal component analysis, thereby making it possible to represent changes in the shape and texture of the target object with fewer parameters.

[0048] In the active appearance model, the shape vector of all feature points is expressed using the average shape vector calculated in advance from training data and the eigenvector matrix obtained by principal component analysis of the deviations from the average shape vector. In other words, the shape vector is expressed as the average shape vector plus the product of the eigenvector matrix and the shape parameters.

[0049] Similarly, the appearance vector, which is an arrangement of normalized texture brightness values, is expressed using a mean appearance vector calculated in advance from training data and an eigenvector matrix obtained by principal component analysis of deviations from the mean appearance vector. That is, the appearance vector is expressed as the mean appearance vector plus the product of the eigenvector matrix and the appearance parameters.

[0050] Shape and appearance parameters are parameters that represent variations from the average, and changing them can change the shape and appearance. Furthermore, because there is a correlation between shape and appearance, by further performing principal component analysis on the shape and appearance parameters, it is possible to express the shape vector and appearance vector using a low-dimensional parameter vector (local parameter) that controls both shape and appearance.

[0051] Next, parameters related to global changes in the location, size, and orientation of the target object in the image (movement parameters) are considered. In an active appearance model, model search involves changing the above parameters to generate images of the target object by locally and globally modifying the model, and then comparing the generated images with the input image to find the parameter values ​​that minimize the error.

[0052] If the error falls below a threshold, it may be determined that the input image contains a set of feature points that match the model being searched for, and the error minimization may be terminated. Conversely, if the error does not fall below a threshold, it may be determined that the input image does not contain a set of feature points that match the model being searched for.

[0053] In this embodiment, since the target object is limited to a moving object, differential image signals between multiple frames are calculated, and the target object is estimated based on the differential image signals. Data on its feature points may be used together with the training data or may be used instead of the training data to calculate the average shape vector and average appearance vector.

[0054] When the moving object detection unit 261 detects a target object in an image of one frame, it changes the above parameters to locally and globally change the model of the target object to generate an image of the target object in another frame, compares the generated image with the image of the other frame, and when the error is minimized or below a threshold, extracts multiple feature points corresponding to the image of the target object in the image of the other frame. This makes it possible to detect a moving object from images of multiple frames.

[0055] When several moving objects are detected by applying the above-described moving object detection process to multiple frames of image signals obtained by one imaging device, the moving object detection unit 261 associates the position information of the detected moving objects on the image with additional information (including identification information of the imaging device) and stores it in the moving object detection result database in the cache memory 27 as moving object detection result information.

[0056] <Step S12> Step S12 is performed using the moving object detection result information obtained in step S11 by performing moving object detection processing on the image signal obtained from one imaging device.

[0057] In step S12, the moving body position estimation unit 262 determines an estimated position of the first moving body at the time of another frame based on the position of the first moving body detected from images of a predetermined number of frames (e.g., two or more frames) out of multiple frames of an image signal obtained from one imaging device, and if the error between the position of the second moving body detected from the image of the other frame and the estimated position is within a predetermined range, it determines that the first moving body and the second moving body are the same, and saves the position information of the second moving body detected from the image of the other frame as position information on the image of the first moving body.

[0058] As an example, first, the moving object position estimation unit 262 reads moving object detection result information from the moving object detection result database in the cache memory 27, and when moving objects are detected from consecutive first and second frame images among multiple frames of an image signal obtained from one imaging device with their positions shifted within a predetermined distance, it determines that they are the same moving object (first moving object). The moving object position estimation unit 262 assigns a moving object ID (identification information) to the first moving object, associates the moving object ID and position information of the first moving object with accompanying information, and stores the associated moving object ID and position information of the first moving object in the moving object tracking information database in the cache memory 27 as moving object tracking information.

[0059] From then on, the moving object position estimation unit 262 reads the moving object tracking information of the first moving object from the moving object tracking information database, and calculates an estimated position of the first moving object at the time of the third frame following the second frame based on the position of the first moving object detected from the images of the first and second frames. The moving object position estimation unit 262 stores information on the estimated position of the first moving object at the time of the third frame in the moving object tracking information database.

[0060] Next, a moving object position comparison process is performed. The moving object position estimation unit 262 sequentially reads out moving object detection result information from the moving object detection result database, and if the error between the position of the second moving object detected from the image of the third frame and the estimated position is within a predetermined range, determines that the first moving object and the second moving object are the same, and saves the position information of the second moving object detected from the image of the third frame in the moving object tracking information database as position information on the image of the first moving object.

[0061] As a result, the information on the estimated position of the first moving object at the time of the third frame is rewritten with the position information of the first moving object, and the moving object tracking information database is updated. Furthermore, the moving object position estimation unit 262 deletes the used moving object detection result information from the moving object detection result database.

[0062] On the other hand, if no moving object is detected in the image of the third frame whose position error from the estimated position is within a predetermined range, the moving object position estimation unit 262 will not update the moving object tracking information database, and will use the estimated position information as the position information on the image of the first moving object at the time of the third frame.

[0063] Furthermore, if the error between the position of the second moving object detected from the image of the third frame and the estimated position exceeds a predetermined range, the moving object position estimation unit 262 determines that the first moving object is different from the second moving object. That is, the moving object position estimation unit 262 determines that a second moving object different from the first moving object has been discovered, assigns a moving object ID to the second moving object, associates the moving object ID and position information of the second moving object with the accompanying information, and stores the association result in the moving object tracking information database as moving object tracking information.

[0064] The above moving object position comparison process is repeated for the number of pieces of moving object detection result information obtained by the moving object detection process for the image signals obtained from one imaging device. Also, when m moving objects are detected from the images of the first group of frames of the image signals obtained from one imaging device (m is a natural number), the moving object position estimation process of step S12 is repeated for the m moving objects.

[0065] If n new moving objects are discovered in this process (n is a natural number), the moving object position estimation process in step S12 is repeated for the n new moving objects. The processes in steps S11 to S12 are repeated the same number of times as the number of imaging devices.

[0066] Alternatively, the moving object position estimation unit 262 may assign the same moving object ID to the first moving object and the second moving object when the error between the position of the second moving object detected from the image of the third frame and the estimated position is within a predetermined range. If a moving object ID has already been assigned to the first moving object, the moving object position estimation unit 262 may assign the same moving object ID as the moving object ID of the first moving object to the second moving object. In this way, the first moving object and the second moving object that are determined to be the same are linked to each other.

[0067] For the above estimation, an object position prediction method using a Kalman filter, or extrapolation or interpolation, which are statistical methods for estimating data in an unknown range, can be used.

[0068] A Kalman filter is a type of infinite impulse response filter that uses observations with errors to estimate or control the state of a dynamic system. It is used to estimate quantities that change over time (such as the position and speed of a moving object) from observations with discrete errors.

[0069] For example, a Kalman filter calculates a posterior estimate of the position of a moving object at time (t+1) by combining a prior estimate at time (t+1) calculated from a posterior estimate at time t with the observed value at time (t+1). The prior estimate at time (t+1) can be calculated by adding the movement vector of the moving object to the posterior estimate at time t.

[0070] Here, the movement vector of the moving object can be calculated as a difference vector between the position of the moving object at time t and the position of the moving object at time (t-1). The position of the moving object calculated by the moving object detection unit 261 can be used as the observed value. The two prior estimated values ​​immediately after the start of object position prediction can be the observed values ​​at the respective times.

[0071] The prior estimated value and the observed value are combined, for example, by weighted averaging. In this combination, the degree of importance attached to either the prior estimated value or the observed value is controlled using a parameter called the Kalman gain. Since it may not be possible to detect the same moving object in an image of one frame as the moving object detected in the image of the previous frame, the Kalman gain may be variable for each frame.

[0072] Fig. 3 is a schematic diagram illustrating an example of a change in the position of a moving object over time. The position of the automobile M shown in Fig. 1 is suitable for display on a two-dimensional map, but here, for ease of explanation, the position of the automobile M is displayed in one-dimensional coordinates. In Fig. 3, the vertical axis represents one-dimensional coordinate x, and the horizontal axis represents time t. Furthermore, frame numbers F1, F2, F3, ... are shown along the time axis as timing information related to the time points of imaging.

[0073] The first image signal and the second image signal are obtained by the first imaging device 11, the second imaging device 12, etc. shown in Fig. 1 capturing images of the first spatial region A1, the second spatial region A2, etc. over multiple frames. In Fig. 3, the coordinates of the automobile M calculated based on the first to third image signals are indicated by circles, squares, and triangles, respectively. The lines connecting these indicate the path (trajectory) of the automobile M.

[0074] However, since car M may be hidden behind an obstacle or change direction when viewed from the nearest imaging device, car M is not detected (marked with an *) in frame F3 of the first image signal and frame F7 of the second image signal.

[0075] In such a case, for each image signal, the moving body position estimation unit 262 may set the weighting coefficient of the observation value equal to or greater than the weighting coefficient of the pre-estimated value for frames in which the moving body detection unit 261 was able to detect the automobile M, and may set the weighting coefficient of the observation value smaller than the weighting coefficient of the pre-estimated value (for example, to zero) for frames in which the moving body detection unit 261 was unable to detect the automobile M.

[0076] In this way, by changing the weighting of the prior estimates and the observed values ​​for each frame, even if the vehicle M is not detected in the image of frame F3 of the first image signal, a posterior estimate of the position of the vehicle M at the time of frame F3 can be obtained. Also, even if the vehicle M is not detected in the image of frame F7 of the second image signal, a posterior estimate of the position of the vehicle M at the time of frame F7 can be obtained.

[0077] In addition to the object position prediction method using the Kalman filter described above, the moving body position estimation unit 262 may also extrapolate the position of the same moving body at the time of capturing the frame in which the same moving body was not detected by the moving body detection unit 261 based on the position of the same moving body detected from images of multiple frames preceding or following the frame in which the same moving body was not detected.

[0078] In addition, the moving body position estimation unit 262 may interpolate the position of the same moving body at the time of capturing the one frame based on the position of the same moving body detected from images of multiple frames surrounding one frame in which the same moving body was not detected by the moving body detection unit 261.

[0079] <Step S13> In step S13, the coordinate conversion unit 263 calculates the coordinate information of the moving body in a predetermined coordinate system based on the position information on the image of the moving body detected from the image signal (in the first embodiment, the position information calculated by the moving body position estimation unit 262 in step S12).

[0080] The predetermined coordinate system may be, for example, a one- or two-dimensional coordinate system that is nearly perpendicular to the vertical direction, a three-dimensional coordinate system with two axes that are nearly perpendicular to the vertical direction, a one- or two-dimensional coordinate system that is nearly parallel to the ground surface where one of the imaging devices is located, or a three-dimensional coordinate system with two axes that are nearly parallel to the ground surface. Alternatively, an absolute coordinate system obtained using a satellite positioning system such as GPS may be used.

[0081] If the coordinates of three or more points defining a predetermined surface area including a moving object in an image represented by an image signal are known, the position of the moving object on the image can be converted into the coordinates of the moving object in a predetermined coordinate system using a transformation matrix such as a perspective projection transformation matrix. The coordinate transformation definition used in the coordinate transformation process by the coordinate transformation unit 263 is stored in a coordinate transformation information database in the storage unit 28.

[0082] The above coordinate conversion process is repeated as many times as the number of pieces of moving object tracking information stored in the moving object tracking information database in the cache memory 27. The coordinate conversion unit 263 adds the coordinate information obtained by the coordinate conversion process to the moving object tracking information and stores it in the moving object tracking information database.

[0083] Fig. 5 is a schematic diagram illustrating the relationship between camera images acquired from multiple imaging devices and a map image in a predetermined coordinate system. In Fig. 5, (a) shows camera image 1 acquired from a first imaging device at a first time point, (b) shows camera image 2 acquired from a second imaging device at a second time point after a predetermined time (a predetermined number of frames) has elapsed since the first time point, and (c) shows a map image onto which camera image 1 and camera image 2 are projected.

[0084] For example, the camera image has 980 x 490 pixels, with the origin O (0,0) located in the upper left. The map image has 420 x 1390 pixels. In camera image 1, four reference points A (420,80), B (110,410), C (630,410), and D (720,80) are set on the road surface or on a reference plane within a predetermined distance from the road surface.

[0085] A first coordinate transformation definition is calculated and stored in the coordinate transformation information database in order to project four reference points on camera image 1 onto four reference points A' (120,140), B' (50,740), C' (300,740), and D' (380,140) on the map image. Similarly, four reference points are also set in camera image 2, and a second coordinate transformation definition is calculated and stored in the coordinate transformation information database in order to project the four reference points on camera image 2 onto the four reference points on the map image.

[0086] The coordinate conversion unit 263 reads out a plurality of coordinate conversion definitions including a first coordinate conversion definition and a second coordinate conversion definition from the coordinate conversion information database. The coordinate conversion unit 263 sequentially reads out moving object tracking information from the moving object tracking information database, and calculates the coordinates of the moving object on the map image based on the position of the moving object on the camera image using the coordinate conversion definition that is compatible with the imaging device identified by the moving object tracking information.

[0087] As a result, the position P1 (558,290) of the car traveling on the road in camera image 1 is converted to coordinates P1' (218,582) on the map image. Also, the position P2 (431,207) of the car traveling on the road in camera image 2 is converted to coordinates P2' (228,932) on the map image.

[0088] <Step S14> In step S14, the moving object tracking unit 264 performs tracking of the moving objects (moving object tracking processing) by determining correspondences between the moving objects detected from a plurality of image signals obtained by capturing images of a plurality of spatial regions, based on coordinate information of the moving objects in a predetermined coordinate system. This makes it possible to easily track the same moving object on a map image, even when a single camera image captures many moving objects and it is difficult to track the same moving object across multiple camera images.

[0089] For example, the moving object tracking unit 264 determines that the two moving objects correspond when the distance (error) between the coordinates of the two moving objects calculated from the images (images of corresponding frames) of the frame of the first image signal and the frame of the second image signal that are closest in time of capture among the frames in which the overlapping portion of the first spatial region and the second spatial region are captured is within a predetermined range.

[0090] Alternatively, when the coordinates of multiple moving objects are determined from an image of a corresponding frame of one image signal, the moving object tracking unit 264 may determine that the moving object having coordinates that have the smallest distance (error) within a predetermined range from the coordinates of the moving object determined from the image of the corresponding frame of the other image signal corresponds to the moving object whose coordinates are determined from the other image signal.

[0091] In addition, when a moving object whose coordinate distance is within a predetermined range is not detected from images (images of corresponding frames) of the frame of the first image signal and the frame of the second image signal that are closest in time of capture among frames in which the overlapping portion of the first spatial region and the second spatial region is captured, the moving object tracking unit 264 may use information on estimated coordinates (coordinates at the time of the corresponding frame) calculated based on the position or coordinates of the moving object detected from images of a predetermined number of other frames (e.g., two or more frames) of the first or second image signal to determine the correspondence relationship of the moving object.

[0092] In the example shown in Figure 3, the moving object tracking unit 264 determines a correspondence between at least one moving object detected from a first image signal obtained by imaging a first spatial region A1 and at least one moving object detected from a second image signal obtained by imaging a second spatial region A2, based on coordinate information of the moving object in a predetermined coordinate system.

[0093] For example, for frame F4 of the first image signal and frame F4 of the second image signal, which are captured at corresponding times among frames capturing the overlapping portion of the first spatial region A1 and the second spatial region A2, if the distance between the coordinates of the moving object (circle) obtained from the first image signal and the coordinates of the moving object (square) obtained from the second image signal is within a predetermined range, it is determined that the moving objects correspond to each other. The coordinates of the moving object in frame F4 may be the coordinates (circle) obtained from the first image signal, the coordinates (square) obtained from the second image signal, or an average value thereof.

[0094] Furthermore, for frame F7 of the second image signal and frame F7 of the third image signal, which are frames in which the overlapping portion of the second spatial region A2 and the third spatial region A3 are captured at corresponding capture times, no moving object was detected in the second image signal whose coordinates are within a predetermined range from the coordinates (△ mark) of the moving object determined from the third image signal, but the coordinates (* mark) of the moving object in frame F7 were estimated based on the position or coordinates (□ mark) of the moving object detected from frames F5 and F6 of the second image signal.

[0095] In such a case, if the distance between the estimated coordinates (marked *) obtained from the second image signal and the coordinates (marked △) of the moving object obtained from the third image signal is within a predetermined range, it is determined that the moving objects correspond to each other. The coordinates of the moving object in frame F7 are determined to be the coordinates (marked △) obtained from the third image signal. )of You can use it.

[0096] 6 is a flowchart showing a specific example of moving object tracking processing. As a premise, N pieces of moving object tracking information are stored in the moving object tracking information database of the cache memory 27 (N is an integer equal to or greater than 2), and the moving object tracking unit 264 refers to the N pieces of moving object tracking information using a variable (i) or a variable (j) (i or j=1, 2, . . . , N).

[0097] In step S141, the moving object tracking unit 264 starts a loop of the variable (i) by setting i = 0. In step S142, the moving object tracking unit 264 increments the variable (i) by 1 and reads the moving object tracking information M(i) from the moving object tracking information database in the cache memory 27.

[0098] In step S143, the moving object tracking unit 264 starts a loop of the variable (j) by setting j = 0. In step S144, the moving object tracking unit 264 increments the variable (j) by "1" and reads the moving object tracking information M(j) from the moving object tracking information database in the cache memory 27.

[0099] In step S145, the moving object tracking unit 264 determines whether the moving object tracking information M(i) and the moving object tracking information M(j) were obtained by the same imaging device. If the moving object tracking information M(i) and the moving object tracking information M(j) were obtained by the same imaging device (YES), the process proceeds to step S148, and if not (NO), the process proceeds to step S146.

[0100] In step S146, the moving object tracking unit 264 determines whether or not map coordinates (coordinate information in a predetermined coordinate system) exist in the moving object tracking information M(i) and the moving object tracking information M(j). Here, the existence of map coordinates means that the map coordinates stored in the moving object tracking information database in the cache memory 27 are not empty (for example, the coordinate data has not been erased).

[0101] If map coordinates exist in the moving object tracking information M(i) and the moving object tracking information M(j), the moving object tracking unit 264 determines whether the distance between the two map coordinates is within a predetermined range, thereby determining the correspondence between the moving object identified by the moving object tracking information M(i) and the moving object identified by the moving object tracking information M(j).

[0102] If it is determined in step S146 that the distance between the map coordinates of the moving object tracking information M(i) and the moving object tracking information M(j) is within a predetermined range (YES), the process proceeds to step S147; if not (NO), the process proceeds to step S148.

[0103] In step S147, the moving object tracking unit 264 determines that the moving object identified by the moving object tracking information M(i) and the moving object identified by the moving object tracking information M(j) correspond (are the same), and adds at least a part of the map coordinates of the moving object tracking information M(j) to part of the map coordinates of the moving object tracking information M(i). Thereafter, the moving object tracking unit 264 empties the map coordinates of the moving object tracking information M(j) that has been used and saves it (for example, erases the coordinate data of the moving object tracking information M(j)).

[0104] In step S148, the moving object tracking unit 264 determines whether the loop for the variable (j) has ended. If the loop for the variable (j) has not ended, the process returns to step S144, and if the loop for the variable (j) has ended, the process proceeds to step S149.

[0105] In step S149, the moving object tracking unit 264 determines whether the loop for the variable (i) has ended. If the loop for the variable (i) has not ended, the process returns to step S142, and when the loop for the variable (i) has ended, the moving object tracking process ends.

[0106] <Step S15> In step S15, the tracking result processing unit 265 obtains information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving body across multiple spatial regions based on coordinate information in a predetermined coordinate system of the moving body that was determined to correspond by the moving body tracking unit 264 in step S14.

[0107] For example, the tracking result processing unit 265 sequentially reads out moving object tracking information from the moving object tracking information database in the cache memory 27, selects moving object tracking information including non-empty map coordinates, and stores it in the tracking result information database in the storage unit 28. Furthermore, the tracking result processing unit 265 may use this to associate tracking result information obtained by the following tracking result processing with additional information including the moving object ID and store it in the tracking result information database.

[0108] The tracking result processing unit 265 may read map data representing a two-dimensional or three-dimensional map from the map database in the storage unit 28, generate an image signal for displaying the position or movement path, etc. of at least one moving object on the map, and display a map image including a surrounding map on the display unit 22. Alternatively, the tracking result processing unit 265 may control the communication circuit 24 to transmit information about the detected moving object to an external device via a network.

[0109] 3, the tracking result processing unit 265 obtains information about the series of positions, movement path, or movement direction of the automobile M based on the coordinate information of each of the automobile M detected in the first spatial region A1 to the third spatial region A3, etc. Furthermore, the tracking result processing unit 265 can obtain information about the movement speed or movement time of the automobile M, and further, the average movement speed or average movement time, etc., based on the difference in the coordinate information of the automobile M between two points, etc.

[0110] Furthermore, the tracking result processing unit 265 may measure the movement routes of multiple moving objects, for example, an unspecified number of automobiles, and based on the measurement, count the number of automobiles passing through a specific area, thereby making it possible to monitor the number of automobiles passing through a road subject to a traffic volume survey.

[0111] In addition, the tracking result processing unit 265 may calculate the inverse matrix of the transformation matrix used by the coordinate transformation unit 263 to obtain a coordinate inverse transformation definition, and when a moving object is specified on the map image, may use the coordinate inverse transformation definition to display on the display unit 22 an image representing the state in which the specified moving object is identified in the corresponding camera image.

[0112] That is, the tracking result processing unit 265 may inversely transform the coordinates of a moving object specified on a map image into positions on a plurality of camera images, select a camera image in which the inversely transformed position of the moving object exists within the effective pixels for each camera, generate an image signal representing an image including a landmark that identifies the moving object at the corresponding position in the selected camera image, and display the image on the display unit 22. Furthermore, the tracking result processing unit 265 may display the moving object ID of the identified moving object in the image in correspondence with the identified moving object.

[0113] 7 is a schematic diagram showing images in which a moving object specified on a map image is identified in camera image 1 and camera image 2. As shown in (a) of FIG. 7, the tracking result processing unit 265 inversely transforms the coordinates P1' (218, 582) of the moving object specified on the map image shown in FIG. 5 to calculate the position P1 (558, 290) in camera image 1. Furthermore, the tracking result processing unit 265 identifies the automobile located at position P1 (558, 290), and causes the display unit 22 to display an image including the moving object ID "0001" of the automobile.

[0114] 7(b), the tracking result processing unit 265 inversely transforms the coordinates P2′ (228,932) of the moving object specified on the map image shown in FIG. 5 to calculate the position P2 (431,207) in the camera image 2. Furthermore, the tracking result processing unit 265 identifies the vehicle located at the position P2 (431,207), and causes the display unit 22 to display an image including the moving object ID “0001” of the vehicle.

[0115] <Second operation example> Next, a second operation example of the mobile object information processing device shown in Fig. 2 will be described with reference to Fig. 1 to Fig. 8. Fig. 8 is a flowchart showing a mobile object information processing method according to the second embodiment of the present invention. In Fig. 8, the same steps as those in the first embodiment shown in Fig. 4 are given the same reference numerals and their description will be omitted.

[0116] In the second embodiment, instead of the moving object position estimation process on the camera image in the first embodiment (step S12 in FIG. 4), step S23 is performed to perform a moving object coordinate estimation process equivalent to the first embodiment in a predetermined coordinate system. Note that part of the process shown in FIG. 8 may be omitted or changed, or other processes may be added to the process shown in FIG. 8.

[0117] <Step S22> In step S22, the coordinate conversion unit 263 determines coordinate information of the moving object in a predetermined coordinate system based on position information on the image of the moving object detected from the image signal (in the second embodiment, position information determined by the moving object detection unit 261 in step S11). Details of the coordinate conversion process in step S22 are the same as those in the first embodiment (step S13 in FIG. 4). The coordinate conversion unit 263 adds the coordinate information determined by the coordinate conversion process to the moving object detection result information and stores it in the moving object detection result database.

[0118] <Step S23> In step S23, the moving object position estimation unit 262 performs moving object coordinate estimation processing using the coordinate information obtained in step S22 by the coordinate conversion unit 263. The moving object position estimation unit 262 obtains estimated coordinates of the first moving object at the time of another frame based on the coordinates of the first moving object detected from images of a predetermined number of frames (e.g., two or more frames) among the multiple frames of the image signal obtained from one imaging device, and when an error between the coordinates of the second moving object detected from the image of the other frame and the estimated coordinates is within a predetermined range, it determines that the first moving object and the second moving object are the same, and saves information on the coordinates of the second moving object detected from the image of the other frame as coordinate information of the first moving object.

[0119] As an example, first, the moving object position estimation unit 262 reads moving object detection result information from the moving object detection result database in the cache memory 27, and when moving objects are detected with coordinates shifted within a predetermined distance from images of a first frame and a second frame consecutively among a plurality of frames of an image signal obtained from one imaging device, it determines that they are the same moving object (first moving object). The moving object position estimation unit 262 assigns a moving object ID (identification information) to the first moving object, associates the moving object ID and coordinate information of the first moving object with accompanying information, and stores the association result as moving object tracking information in the moving object tracking information database in the cache memory 27.

[0120] From then on, the moving object position estimation unit 262 reads the moving object tracking information of the first moving object from the moving object tracking information database, and calculates the estimated coordinates of the first moving object at the time of the third frame following the second frame based on the coordinates of the first moving object detected from the images of the first and second frames. The moving object position estimation unit 262 stores the information of the estimated coordinates of the first moving object at the time of the third frame in the moving object tracking information database.

[0121] Next, a moving object coordinate comparison process is performed. The moving object position estimation unit 262 sequentially reads out moving object detection result information from the moving object detection result database, and if the error between the coordinates of the second moving object detected from the image of the third frame and the estimated coordinates is within a predetermined range, determines that the first moving object and the second moving object are the same, and saves the information of the coordinates of the second moving object detected from the image of the third frame as coordinate information of the first moving object in the moving object tracking information database.

[0122] As a result, the information on the estimated coordinates of the first moving object at the time of the third frame is rewritten with the coordinate information of the first moving object, and the moving object tracking information database is updated. Furthermore, the moving object position estimation unit 262 deletes the used moving object detection result information from the moving object detection result database.

[0123] On the other hand, if no moving object is detected in the image of the third frame whose coordinate error from the estimated coordinates is within a specified range, the moving object position estimation unit 262 will not update the moving object tracking information database, and will use the estimated coordinate information as the coordinate information of the first moving object at the time of the third frame.

[0124] Furthermore, if the error between the coordinates of the second moving object detected from the image of the third frame and the estimated coordinates exceeds a predetermined range, the moving object position estimation unit 262 determines that the first moving object is different from the second moving object. That is, the moving object position estimation unit 262 determines that a second moving object different from the first moving object has been discovered, assigns a moving object ID to the second moving object, associates the moving object ID and coordinate information of the second moving object with accompanying information, and stores the association result in the moving object tracking information database as moving object tracking information.

[0125] The above moving object coordinate comparison process is repeated for the number of pieces of moving object detection result information obtained by the moving object detection process and coordinate conversion process for the image signals obtained from one imaging device. Also, when m moving objects are detected from the images of the first group of frames of the image signals obtained from one imaging device (m is a natural number), the moving object coordinate estimation process of step S23 is repeated for the m moving objects.

[0126] If n new moving objects are discovered in this process (n is a natural number), the moving object coordinate estimation process in step S23 is repeated for the n new moving objects. Furthermore, the processes in steps S11 to S23 are repeated the same number of times as the number of imaging devices.

[0127] Alternatively, the moving object position estimation unit 262 may assign the same moving object ID to the first moving object and the second moving object when the error between the coordinates of the second moving object detected from the image of the third frame and the estimated coordinates is within a predetermined range. If a moving object ID has already been assigned to the first moving object, the moving object position estimation unit 262 may assign the same moving object ID as the moving object ID of the first moving object to the second moving object. As a result, the first moving object and the second moving object that are determined to be the same are linked to each other.

[0128] As in the first embodiment, the above estimation can use an object position prediction method using a Kalman filter, or extrapolation or interpolation, which is a statistical method for estimating data in an unknown range.

[0129] <Step S24> In step S24, the moving object tracking unit 264 performs tracking of the moving object (moving object tracking processing) by determining a correspondence relationship between the moving object detected from a plurality of image signals obtained by capturing images of a plurality of spatial regions, based on coordinate information of the moving object in a predetermined coordinate system. The detailed processing in step S24 is the same as that in the first embodiment (step S14 in FIG. 4).

[0130] <Step S25> In step S25, the tracking result processing unit 265 obtains information on the position, moving path, moving direction, moving speed, or moving time of at least one moving object across multiple spatial regions, based on the coordinate information in the predetermined coordinate system of the moving object determined to correspond by the moving object tracking unit 264 in step S24. The detailed processing in step S25 is the same as in the first embodiment (step S15 in FIG. 4).

[0131] <Effects of the embodiment> According to the first or second embodiment, a moving object is detected from a plurality of frame images of an image signal obtained from at least one imaging device, coordinate information of the moving object in a predetermined coordinate system is obtained based on position information of the moving object in the image, and a correspondence relationship between the moving object detected in the first spatial region and the moving object detected in the second spatial region is determined based on the coordinate information of the moving object, thereby tracking the moving object across a plurality of spatial regions. This makes it possible to continuously track an unspecified number of moving objects moving over a relatively wide range, or to obtain information on the position or movement path of the moving object with high accuracy using an imaging device disposed outside the moving object.

[0132] Generally, when multiple imaging devices placed outside a moving body are used to capture images of multiple spatial regions and obtain multiple image signals, when the moving body crosses over the images captured by the imaging devices, a large error occurs between the position of the moving body predicted from one image and the position of the moving body detected from the other image, making it difficult to recognize that the moving bodies are the same.

[0133] In such cases and determining a correspondence between the moving object detected in the first spatial region and the moving object detected in the second spatial region based on the distance between the coordinates of the two moving objects calculated from the images of the nearest frames of two image signals obtained by capturing images of the first spatial region and the second spatial region, which partially overlap each other, over a plurality of frames. , complex This allows tracking of moving objects across a large number of spatial regions. This makes it possible to capture an unspecified number of vehicles passing through a relatively wide area that cannot be captured by a single imaging device. Using multiple imaging devices By continuously tracking, it is also possible to monitor the number of cars passing through the road being surveyed in a traffic volume survey.

[0134] The present invention is not limited to the above-described embodiments, and many modifications within the technical spirit of the present invention are possible by those skilled in the art. For example, it is also possible to combine at least a part of the first embodiment with at least a part of the second embodiment. [Industrial Applicability]

[0135] The present invention can be used in a mobile object information processing device or the like that obtains information about the position or travel route of a mobile object such as an automobile based on an image signal. [Explanation of symbols]

[0136] 10...recording device, 11...first imaging device, 12...second imaging device, 20...mobile object information processing device, 21...operation unit, 22...display unit, 23...audio input / output unit, 24...communication circuit, 25...interface, 26...CPU, 260...image signal acquisition unit, 261...mobile object detection unit, 262...mobile object position estimation unit, 263...coordinate conversion unit, 264...mobile object tracking unit, 265...tracking result processing unit, 27...cache memory, 28...storage unit

Claims

1. A mobile object information processing device that obtains information about a mobile object based on an image signal obtained from at least one imaging device that captures an image of a spatial region and generates an image signal, comprising: a moving object detection unit that detects a moving object from a plurality of frame images of the image signal; a coordinate conversion unit that calculates coordinate information of the moving object in a predetermined coordinate system based on position information of the moving object on the image detected from the image signal; a moving object tracking unit that, when determining a correspondence between at least one moving object detected from a first image signal obtained by imaging a first spatial region and at least one moving object detected from a second image signal obtained by imaging a second spatial region based on coordinate information of the moving object in the predetermined coordinate system, uses information on estimated coordinates calculated based on the position or coordinates of the moving object detected from images of another predetermined number of frames of the first or second image signal, when a moving object whose coordinate distance is within a predetermined range is not detected from images of a frame of the first image signal and a frame of the second image signal that are closest in image time among a plurality of frames of the first image signal and a plurality of frames of the second image signal obtained by imaging the first spatial region and the second spatial region, respectively, that partially overlap each other, to determine the correspondence between the moving objects; a tracking result processing unit that determines information regarding a position, a moving path, a moving direction, a moving speed, or a moving time of at least one moving object across a plurality of spatial regions based on coordinate information in the predetermined coordinate system of the moving object that is determined to correspond by the moving object tracking unit; A mobile information processing device comprising:

2. A mobile object information processing device as described in claim 1, wherein the mobile object tracking unit determines that the two moving objects correspond to each other when the distance between the coordinates of the two moving objects calculated from the images of the frame of the first image signal and the frame of the second image signal that are closest in time of capture among multiple frames of a first image signal and a second image signal obtained by capturing images of a first spatial region and a second spatial region that partially overlap each other is within a predetermined range.

3. 2. The mobile object information processing device of claim 1, further comprising a mobile object position estimation unit that calculates an estimated position of a first moving object at the time of another frame based on the position of the first moving object detected from images of a predetermined number of frames out of the multiple frames of the image signal, and determines that the first moving object and the second moving object are the same if the error between the position of a second moving object detected from the images of the other frames and the estimated position is within a predetermined range, and saves information on the position of the second moving object detected from the images of the other frames as position information on the image of the first moving object.

4. 4. The mobile object information processing device of claim 3, wherein the mobile object position estimation unit uses the estimated position information as the position information of the first mobile object on the image of the other frame when no mobile object is detected in the image of the other frame whose position error from the estimated position is within a predetermined range.

5. 2. The mobile object information processing device according to claim 1, further comprising a mobile object position estimation unit that calculates estimated coordinates of a first moving object at the time of another frame based on the coordinates of the first moving object detected from images of a predetermined number of frames out of the multiple frames of the image signal, and determines that the first moving object and the second moving object are the same if an error between the coordinates of a second moving object detected from the images of the other frames and the estimated coordinates is within a predetermined range, and saves information on the coordinates of the second moving object detected from the images of the other frames as coordinate information of the first moving object.

6. 6. The mobile object information processing device of claim 5, wherein the mobile object position estimation unit uses the estimated coordinate information as coordinate information of the first mobile object at the time of the other frame when no mobile object is detected in the image of the other frame whose coordinate error from the estimated coordinates is within a predetermined range.

7. a plurality of imaging devices including a first imaging device and a second imaging device that capture images of a first spatial region and a second spatial region, respectively, to generate a first image signal and a second image signal; A mobile information processing device according to any one of claims 1 to 6; A mobile information processing system comprising:

8. 1. A mobile object information processing program used in a mobile object information processing device that obtains information about a mobile object based on an image signal obtained from at least one imaging device that captures an image of a spatial region and generates an image signal, comprising: a step (a) of detecting a moving object from a plurality of frame images of the image signal; a step (b) of calculating coordinate information of the moving object in a predetermined coordinate system based on position information of the moving object on the image detected from the image signal; a step (c) of determining, based on coordinate information of the moving object in the predetermined coordinate system, a correspondence between at least one moving object detected from a first image signal obtained by imaging a first spatial region and at least one moving object detected from a second image signal obtained by imaging a second spatial region, if no moving object whose coordinate distance is within a predetermined range is detected from images of a frame of the first image signal and a frame of the second image signal that are closest in time of imaging among a plurality of frames of the first image signal and the second image signal obtained by imaging the first spatial region and the second spatial region, respectively, that partially overlap each other, using information on estimated coordinates calculated based on the positions or coordinates of the moving object detected from images of a predetermined number of other frames of the first or second image signal to determine the correspondence between the moving objects; a step (d) of determining information on the position, movement path, movement direction, movement speed, or movement time of at least one moving object across a plurality of spatial regions based on coordinate information in the predetermined coordinate system of the moving object determined to correspond in the step (c); A mobile information processing program that causes a CPU to execute the above.

9. 1. A mobile object information processing method for obtaining information about a mobile object based on an image signal obtained from at least one imaging device that captures an image of a spatial region and generates an image signal, comprising: (a) detecting a moving object from a plurality of frame images of the image signal; (b) determining coordinate information of the moving object in a predetermined coordinate system based on position information of the moving object on the image detected from the image signal; a step (c) of using, when determining a correspondence relationship between at least one moving object detected from a first image signal obtained by imaging a first spatial region and at least one moving object detected from a second image signal obtained by imaging a second spatial region based on coordinate information of the moving object in the predetermined coordinate system, information on estimated coordinates calculated based on the positions or coordinates of the moving object detected from images of another predetermined number of frames of the first or second image signal, when a moving object whose coordinate distance is within a predetermined range is not detected from images of a frame of the first image signal and a frame of the second image signal that are closest in time of imaging among a plurality of frames of the first image signal and a plurality of frames of the second image signal obtained by imaging the first spatial region and the second spatial region, respectively, that partially overlap each other, to determine a correspondence relationship between the moving objects; a step (d) of determining information regarding a position, a moving path, a moving direction, a moving speed, or a moving time of at least one moving object across a plurality of spatial regions based on coordinate information in the predetermined coordinate system of the moving object determined to correspond in the step (c); A mobile information processing method comprising:

Images