Mobile information processing device, system, program, and method

Abstract

The present invention provides a mobile object information processing device, etc., capable of continuously tracking an unspecified number of moving objects that travel over a relatively wide area. [Solution] This mobile object information processing device includes: a mobile object detection unit that detects a mobile object from images of multiple frames of an image signal; a coordinate transformation unit that obtains coordinate information of the mobile object in a predetermined coordinate system based on the position information of the detected mobile object on the image; a mobile object tracking unit that determines the correspondence between at least one mobile object detected from a first image signal obtained by imaging a first spatial region and at least one mobile object detected from a second image signal obtained by imaging a second spatial region, based on the coordinate information of the mobile object in the predetermined coordinate system; and a tracking result processing unit that obtains information regarding the position of at least one mobile object across multiple spatial regions based on the coordinate information of the mobile object determined to correspond by the mobile object tracking unit in a predetermined coordinate system.

Description

Technical Field

[0001] The present invention relates to a moving body information processing apparatus and a moving body information processing method for obtaining information such as the position or moving route of a moving body such as an automobile based on an image signal. Further, the present invention relates to a moving body information processing system including such a moving body information processing apparatus, and a moving body information processing program used in the moving body information processing apparatus. In the present application, examples of the moving body include automobiles including automated guided vehicles and autonomous driving vehicles, vehicles including motorcycles and bicycles, forklifts, self-propelled robots, walking humans, or drones.

Background Art

[0002] As a technique for obtaining information regarding the route of a moving body, Patent Document 1 discloses a moving route prediction system for predicting the route of a moving body moving within a predetermined area such as an airport or a factory. In the predetermined area, a departure checkpoint, a destination checkpoint, and intermediate checkpoints are arranged, and the moving body moves within the predetermined area from the departure checkpoint to the destination checkpoint via the intermediate checkpoints. Information on the moving route of the moving body is measured by a movement measurement system.

[0003] This moving route prediction system includes a measurement data acquisition unit that acquires information on the moving route from the movement measurement system, a map data input unit that acquires position information of a plurality of checkpoints, a workflow input unit that sets route information of a plurality of checkpoints through which the moving body moves, a moving line prediction model learning unit that machine-learns a moving line prediction model for calculating a first objective variable from a first feature amount, and a moving route prediction unit that simulates the route of the moving body based on the moving line prediction model.

[0004] According to Patent Document 1, for example, when performing a person's moving line simulation in an airport after layout change, by dividing the information on the moving route into a plurality of sections and machine-learning the moving line prediction model for each of the divided moving route information, the accuracy of the moving line simulation can be improved.

[0005] Furthermore, Patent Document 2 discloses a position estimation system that aims to perform position estimation with good accuracy using two-dimensional images captured by a monocular imaging device, without using a stereo camera, which is difficult to miniaturize and reduce the cost of.

[0006] This position estimation system comprises a monocular imaging device, a storage unit that stores a feature map showing a feature space in which real space is represented by feature quantities of a predetermined feature type, and a position estimation unit that estimates the position corresponding to the imaging device based on the result of comparing a reference image showing the feature quantities of an image captured by the imaging device with a projected image generated according to a projection center set in the feature space of the feature map.

[0007] The accuracy of 3D point clouds generated from images captured by monocular imaging devices is insufficient, resulting in reduced accuracy in position estimation. In contrast, the position estimation system described in Patent Document 2 eliminates the need for 3D point clouds by utilizing feature maps. This makes it possible to estimate the self-position of a moving object with good accuracy even when using a monocular imaging device. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2020-123128 (paragraphs 0008, 0143) [Patent Document 2] Japanese Patent Publication No. 2020-173617 (paragraphs 0004-0006, 0012, 0025) [Disclosure of the Invention] [Problems that the invention aims to solve]

[0009] However, in the movement path prediction system of Patent Document 1, for example, when measuring the movement paths of an unspecified number of vehicles traveling over a relatively wide area for traffic volume surveys, it becomes difficult to measure the movement path when the vehicles move far away or enter blind spots, making it difficult to continuously track the vehicles. Furthermore, the position estimation system of Patent Document 2 is a self-position estimation system installed inside a moving object such as a vehicle or robot, and is not suitable for obtaining information about the position or movement path of a moving object using an imaging device placed outside the moving object.

[0010] Therefore, in view of the above, the first object of the present invention is to provide a mobile object information processing device or method capable of continuously tracking an unspecified number of moving objects moving over a relatively wide area. The second object of the present invention is to provide a mobile object information processing device or method capable of obtaining information regarding the position or movement path of a moving object with good accuracy using an imaging device placed outside the moving object. Furthermore, the third object of the present invention is to provide a mobile object information processing system equipped with such a mobile object information processing device, or a mobile object information processing program used in a mobile object information processing device. [Means for solving the problem]

[0011] To solve at least some of the above problems, a mobile object information processing device according to a first aspect of the present invention is a mobile object information processing device that obtains information about a mobile object based on an image signal obtained from at least one imaging device that images a spatial region and generates an image signal, and comprises: a mobile object detection unit that detects a mobile object from images of multiple frames of the image signal; a coordinate transformation unit that obtains coordinate information of a mobile object in a predetermined coordinate system based on the position information of the mobile object on the image detected from the image signal; a mobile object tracking unit that determines a correspondence between at least one mobile object detected from a first image signal obtained by imaging a first spatial region and at least one mobile object detected from a second image signal obtained by imaging a second spatial region, based on the coordinate information of the mobile object in the predetermined coordinate system; and a tracking result processing unit that obtains information about the position, movement path, movement direction, movement speed, or movement time of at least one mobile object spanning multiple spatial regions based on the coordinate information of the mobile object determined to be corresponding by the mobile object tracking unit in the predetermined coordinate system.

[0012] In a mobile object information processing device according to a second aspect of the present invention, the first image signal and the second image signal are obtained by imaging a first spatial region and a second spatial region that partially overlap each other over multiple frames, and the mobile object tracking unit determines that the two mobile objects correspond to each other when the distance between the coordinates of the two mobile objects, determined from the images of the frame of the first image signal and the frame of the second image signal, which are closest in time of imaging, is within a predetermined range.

[0013] A mobile information processing system according to the first or second aspect of the present invention comprises, in addition to the mobile information processing device according to the first or second aspect of the present invention, a plurality of imaging devices including a first imaging device and a second imaging device that capture images of a first spatial region and a second spatial region, respectively, and generate a first image signal and a second image signal, respectively.

[0014] A mobile object information processing program according to a first aspect of the present invention is a mobile object information processing program used in a mobile object information processing device that obtains information about a mobile object based on an image signal obtained from at least one imaging device that images a spatial region and generates an image signal, and causes the CPU to execute the following steps: (a) detecting a mobile object from images of multiple frames of the image signal; (b) obtaining coordinate information of the mobile object in a predetermined coordinate system based on the position information of the mobile object on the image detected from the image signal; (c) determining a correspondence between at least one mobile object detected from a first image signal obtained by imaging a first spatial region and at least one mobile object detected from a second image signal obtained by imaging a second spatial region, based on the coordinate information of the mobile object in the predetermined coordinate system; and (d) obtaining information about the position, movement path, movement direction, movement speed, or movement time of at least one mobile object spanning multiple spatial regions based on the coordinate information of the mobile object determined to correspond in step (c).

[0015] A method for processing information about a moving object according to a first aspect of the present invention is a method for processing information about a moving object based on an image signal obtained from at least one imaging device that images a spatial region and generates an image signal, comprising: (a) detecting a moving object from images of multiple frames of the image signal; (b) determining coordinate information of the moving object in a predetermined coordinate system based on the position information of the moving object on the image detected from the image signal; (c) determining a correspondence between at least one moving object detected from a first image signal obtained by imaging a first spatial region and at least one moving object detected from a second image signal obtained by imaging a second spatial region, based on the coordinate information of the moving object in the predetermined coordinate system; and (d) determining information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object spanning multiple spatial regions based on the coordinate information of the moving objects determined to correspond in step (c) in the predetermined coordinate system. [Effects of the Invention]

[0016] According to a first aspect of the present invention, a moving object is detected from multiple frames of image signals obtained from at least one imaging device, coordinate information of the moving object in a predetermined coordinate system is determined based on the position information of the moving object on the image, and the correspondence between the moving object detected in a first spatial region and the moving object detected in a second spatial region is determined based on the coordinate information of the moving object, thereby tracking of the moving object across multiple spatial regions. This makes it possible to continuously track an unspecified number of moving objects moving over a relatively wide area, or to obtain information regarding the position or movement path of the moving object with good accuracy using an imaging device placed outside the moving object.

[0017] Generally, when multiple imaging devices placed outside a moving object are used to image multiple spatial regions and obtain multiple image signals, a large error occurs between the position of the moving object predicted from one image and the position of the moving object detected from the other image when the moving object crosses over the images obtained by those imaging devices, making it difficult to recognize that those moving objects are the same.

[0018] According to a second aspect of the present invention, by determining the correspondence between a moving object detected in the first spatial region and a moving object detected in the second spatial region based on the distance between the coordinates of two moving objects obtained from the image of the closest frame of two image signals obtained by imaging a first spatial region and a second spatial region that partially overlap each other over multiple frames, it is possible to track a moving object across multiple spatial regions imaged by multiple imaging devices. This makes it possible to continuously track an unspecified number of vehicles traveling over a relatively wide area that cannot be captured by a single imaging device, and to monitor the number of vehicles traveling on the road being surveyed in traffic volume surveys. [Brief explanation of the drawing]

[0019] [Figure 1] This is a schematic diagram illustrating an example of a mobile information processing system according to one embodiment of the present invention. [Figure 2] It is a block diagram showing a configuration example of a mobile body information processing apparatus shown in FIG. 1. [Figure 3] It is a schematic diagram exemplarily showing changes in the position of a mobile body over time. [Figure 4] It is a flowchart showing a mobile body information processing method according to a first embodiment of the present invention. [Figure 5] It is a schematic diagram for explaining the relationship between a camera image obtained from a plurality of imaging devices and a map image in a predetermined coordinate system. [Figure 6] It is a flowchart showing a specific example of the mobile body tracking process in FIG. 4. [Figure 7] It is a schematic diagram showing an image in which a mobile body specified on a map image is identified in camera image 1 and camera image 2. [Figure 8] It is a flowchart showing a mobile body information processing method according to a second embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. <Mobile Body Information Processing System> FIG. 1 is a schematic diagram exemplarily showing an overview of a mobile body information processing system according to an embodiment of the present invention. In FIG. 1, one automobile M is shown as an example of a mobile body, but this system can obtain information regarding the positions or movement routes of a plurality of mobile bodies.

[0021] For example, this system can obtain information regarding the positions or movement routes of an unspecified number of automobiles passing through a relatively wide area, and monitor the number of automobiles passing through a road to be surveyed in a traffic volume survey. Further, this system can be widely applied when obtaining information regarding the positions or movement routes of moving vehicles, forklifts, or autonomous robots in factories, airports, or stores.

[0022] This system includes at least one imaging device that captures a spatial region and generates an image signal (motion signal), and a mobile object information processing device 20 that obtains information about a moving object based on the image signal obtained from at least one imaging device. As shown in Figure 1, a plurality of imaging devices, including a first imaging device 11, a second imaging device 12, ..., may be used.

[0023] In the example shown in Figure 1, the first imaging device 11 captures the first spatial region A1 and generates a first image signal, and the second imaging device 12 captures the second spatial region A2 and generates a second image signal. Here, the adjacent first spatial region A1 and second spatial region A2 partially overlap each other. The same applies hereafter. These imaging devices may perform their imaging operations in synchronous motion. Alternatively, multiple image signals may be generated by capturing multiple spatial regions based on an image signal obtained from one imaging device.

[0024] For example, multiple imaging devices, including the first imaging device 11, the second imaging device 12, etc., may be arranged at approximately equal intervals along a road to be surveyed in order to monitor the number of vehicles passing through the road in a traffic volume survey. Existing security cameras may be used as each of the imaging devices.

[0025] Alternatively, multiple imaging devices with different optical system magnifications may be used. For example, a first imaging device 11 that images a relatively close-range moving object using a wide-angle lens or a standard lens, and a second imaging device 12 that images a relatively long-range moving object using a telephoto lens, may be arranged within a predetermined distance from each other.

[0026] Each imaging device includes a camera unit that captures an image of a subject with an image sensor via an optical system including at least one lens and generates a digital or analog image signal, and a communication unit that communicates wirelessly or via a wired connection with the mobile information processing device 20 and transmits the image signal generated by the camera unit to the mobile information processing device 20.

[0027] Alternatively, a recording device 10 that records the image signals generated by the camera unit may be used. The recording device 10 may be mounted inside each imaging device, or it may be located in a building or the like outside of the multiple imaging devices, as shown in Figure 1. The image signals recorded in the recording device 10 are supplied to the mobile information processing device 20 by wireless or wired communication, or using a portable recording medium.

[0028] The recording medium in the recording device 10, or the portable recording medium, can be a hard disk, flexible disk, optical disk, magneto-optical disk, magnetic tape, SSD (solid state drive), or various types of memory including USB memory.

[0029] Communication between the first imaging device 11, etc. and the mobile information processing device 20, or communication between the first imaging device 11, etc. and the recording device 10, or communication between the recording device 10 and the mobile information processing device 20 may be conducted via a network such as a wired LAN, wireless LAN, intranet, internet, or mobile communication network.

[0030] Furthermore, when connecting them via a wired connection, methods such as HDMI (High-Definition Multimedia Interface; registered trademark) or SDI (Serial Digital Interface) may be used.

[0031] The image signals generated by each imaging device, or the image signals recorded in the recording device 10, are supplied to the mobile information processing device 20 as measurement information, along with supplementary information including identification information of the imaging device or timing information related to the time of imaging. At least a portion of the supplementary information may be included in the file name of the image signal or measurement information.

[0032] The mobile object information processing device 20 obtains information about a mobile object based on an image signal obtained from at least one imaging device that captures a spatial area and generates an image signal. For example, the mobile object information processing device 20 consists of a server installed in a data center or a control center of a research company or manufacturing company, or a PC (personal computer) and a display, etc.

[0033] <Mobile Information Processing Device> Figure 2 is a block diagram showing an example configuration of the mobile information processing device shown in Figure 1. As shown in Figure 2, the mobile information processing device 20 includes an operation unit 21, a display unit 22, an audio input / output unit 23, a communication circuit 24, an interface 25, a CPU (Central Processing Unit) 26, a cache memory 27, and a storage unit 28. The interface 25 to the storage unit 28 are connected to each other via a bus line. Note that some of the components shown in Figure 1 or Figure 2 may be omitted or changed, or other components may be added to the components shown in Figure 1 or Figure 2.

[0034] The operation unit 21 consists of, for example, a keyboard and mouse, and is used to input various commands and data to the mobile information processing device 20. The display unit 22 includes, for example, an LCD (liquid crystal display), and displays the operation screen, etc. The audio input / output unit 23 includes, for example, a microphone, amplifier, and speaker, and converts audio signals into electrical signals and electrical signals into audio signals.

[0035] The communication circuit 24 has the function of performing data communication with a plurality of imaging devices, including the first imaging device 11, the second imaging device 12, ..., or with a recording device 10 that records image signals generated by those imaging devices.

[0036] Interface 25 is connected to the operation unit 21, the communication circuit 24, and external devices such as portable recording media, and transmits various commands and data between them and the CPU 26. Data input to the CPU 26 via interface 25 is written to the cache memory 27 or storage unit 28.

[0037] The CPU 26 performs various calculations and data processing according to the various software (including mobile information processing programs) stored in the storage unit 28. The cache memory 27 and storage unit 28 store various data used to obtain information about mobile objects and tracking results information of mobile objects in multiple databases (DBs).

[0038] The cache memory 27 is composed of RAM (Random Access Memory) or the like. Various types of memory, including hard disks, flexible disks, optical disks, magneto-optical disks, magnetic tapes, SSDs, or ROM (Read-Only Memory), can be used as recording media (storage media) in the storage unit 28.

[0039] Here, the CPU 26 and the mobile object information processing program stored in the storage unit 28 configure the image signal acquisition unit 260, the mobile object detection unit 261, the mobile object position estimation unit 262, the coordinate transformation unit 263, the mobile object tracking unit 264, and the tracking result processing unit 265 as functional blocks.

[0040] The image signal acquisition unit 260 acquires measurement information, including image signals and associated information, from at least one imaging device. For example, the image signal acquisition unit 260 may control the communication circuit 24 to receive measurement information from a plurality of imaging devices, including a first imaging device 11, a second imaging device 12, ..., or from a recording device 10 that records image signals generated by those imaging devices.

[0041] Alternatively, the image signal acquisition unit 260 may acquire measurement information from a portable recording medium on which measurement information is recorded when the recording medium is connected to the interface 25. For example, the image signal acquisition unit 260 stores measurement information acquired from the first imaging device 11, etc., in the image database of the cache memory 27, and stores measurement information acquired from the recording device 10 or the portable recording medium in the image database of the storage unit 28.

[0042] The moving object detection unit 261 to the tracking result processing unit 265 obtain information such as the position, movement path, movement direction, movement speed, or movement time of at least one moving object based on the image signal included in the measurement information. For example, the position or movement path of the moving object on a predetermined map is obtained and displayed on the display unit 22.

[0043] <Example of operation 1> Next, a first example of operation of the mobile information processing device shown in Figure 2 will be described with reference to Figures 1 to 7. Figure 4 is a flowchart of the mobile information processing method according to the first embodiment of the present invention. Note that some of the processes shown in Figure 4 may be omitted or modified, or other processes may be added to the processes shown in Figure 4.

[0044] <Step S10> In step S10 of Figure 4, the image signal acquisition unit 260 acquires measurement information, including image signals and associated information, from the first imaging device 11 shown in Figure 1, the recording device 10, or a portable recording medium. The image signal acquisition unit 260 stores the acquired measurement information in the cache memory 27 or the image database of the storage unit 28. As a result, multiple frames of image signals obtained from the first imaging device 11 are stored in the image database.

[0045] <Step S11> In step S11, the moving object detection unit 261 reads the image signals collected by the image signal acquisition unit 260 in step S10 from the image database and performs moving object detection processing on the image signals obtained from one imaging device, thereby detecting moving objects from multiple frames of the image signal. As a result, the position information of the moving object on the image is obtained for each frame of the image signal. This moving object detection processing can utilize AI (Artificial Intelligence) for object detection processing or image recognition processing.

[0046] AI-based object detection processing, for example, divides a single frame of an image into many small segments, and labels each segment, indicating what is depicted, or associates it with a category. If an object with the same label or category is detected in two consecutive frames, but within a predetermined distance from each other, it is recognized as the same object that has moved. This makes it possible to detect moving objects whose position changes across multiple frames of an image signal.

[0047] Alternatively, the moving object detection unit 261 may detect a moving object from multiple frames of images by applying predetermined image recognition processing to the image signals of multiple frames obtained by a single imaging device. In this image recognition processing, for example, an active appearance model can be used, which separates the image of the target object into shape and texture, and reduces the dimensionality of each by principal component analysis, thereby enabling the representation of changes in the shape and texture of the target object with fewer parameters.

[0048] In an active appearance model, the shape vector, which represents all feature points, is expressed using the mean shape vector obtained from the training data and the eigenvector matrix obtained by principal component analysis of the deviations from the mean shape vector. In other words, the shape vector is expressed as the mean shape vector plus the product of the eigenvector matrix and the shape parameters.

[0049] Similarly, the appearance vector, which is a sequence of normalized texture luminance values, is represented using the average appearance vector obtained from the training data and the eigenvector matrix obtained by principal component analysis of the deviations from the average appearance vector. In other words, the appearance vector is expressed as the average appearance vector plus the product of the eigenvector matrix and the appearance parameter.

[0050] Shape parameters and appearance parameters are parameters that represent the change from the mean, and by changing them, the shape and appearance can be changed. Furthermore, since there is a correlation between shape and appearance, by further performing principal component analysis on the shape parameters and appearance parameters, it is possible to represent the shape vector and appearance vector using low-dimensional parameter vectors (local parameters) that control both shape and appearance.

[0051] Next, parameters related to the global changes in where, what size, and in what orientation the target object exists within the image (movement parameters) are considered. In the active appearance model, model exploration involves changing the above parameters to locally and globally change the model, generating an image of the target object, comparing the generated image with the input image, and finding the parameter values ​​that minimize the error.

[0052] If the error falls below the threshold, it may be determined that a group of feature points matching the model being searched exists in the input image, and the error minimization process may be terminated. On the other hand, if the error does not fall below the threshold, it may be determined that a group of feature points matching the model being searched does not exist in the input image.

[0053] In this embodiment, since the target object is limited to a moving object, the difference image signal between multiple frames is obtained, the target object is estimated based on the difference image signal, and the data of its feature points is used together with the training data, or used in place of the training data, to obtain the average shape vector and average appearance vector.

[0054] When the moving object detection unit 261 detects a target object in one frame of an image, it changes the above-mentioned parameters to locally and globally modify the model of the target object and generate images of the target object in other frames. It then compares the generated images with the images in other frames, and if the error is minimized or falls below a threshold, it extracts multiple feature points corresponding to the image of the target object in the images of other frames. This makes it possible to detect moving objects from images of multiple frames.

[0055] When several moving objects are detected by applying the moving object detection process described above to multiple frames of image signals obtained by a single imaging device, the moving object detection unit 261 associates the image position information of the detected moving objects with supplementary information (including the identification information of the imaging device) and stores it as moving object detection result information in the moving object detection result database of the cache memory 27.

[0056] <Step S12> Step S12 is performed using the moving object detection result information obtained in step S11 by applying moving object detection processing to the image signal obtained from one imaging device.

[0057] In step S12, the moving object position estimation unit 262 determines the estimated position of the first moving object at the time of the other frames based on the position of the first moving object detected from the images of a predetermined number of frames (for example, two or more frames) among multiple frames of image signals obtained from one imaging device. If the error between the position of the second moving object detected from the images of the other frames and the estimated position is within a predetermined range, the unit determines that the first moving object and the second moving object are the same and saves the position information of the second moving object detected from the images of the other frames as the position information of the first moving object on the image.

[0058] As an example, first, the moving object position estimation unit 262 reads moving object detection result information from the moving object detection result database in the cache memory 27. If a moving object is detected in consecutive first and second frames of multiple frames of an image signal obtained from a single imaging device, shifting its position within a predetermined distance, the unit determines that they are the same moving object (first moving object). The moving object position estimation unit 262 assigns a moving object ID (identification information) to the first moving object, associates the moving object ID and position information of the first moving object with the associated information, and stores it as moving object tracking information in the moving object tracking information database in the cache memory 27.

[0059] Next, the mobile object position estimation unit 262 reads the mobile object tracking information for the first mobile object from the mobile object tracking information database and, based on the position of the first mobile object detected from the images of the first and second frames, determines the estimated position of the first mobile object at the time of the third frame following the second frame. The mobile object position estimation unit 262 stores the information of the estimated position of the first mobile object at the time of the third frame in the mobile object tracking information database.

[0060] Next, a mobile object position comparison process is performed. The mobile object position estimation unit 262 sequentially reads mobile object detection result information from the mobile object detection result database, and if the error between the position of the second mobile object detected from the image of the third frame and the estimated position is within a predetermined range, it determines that the first mobile object and the second mobile object are the same, and saves the position information of the second mobile object detected from the image of the third frame as the position information of the first mobile object in the mobile object tracking information database.

[0061] As a result, the information of the estimated position of the first mobile object at the time of the third frame is overwritten with the position information of the first mobile object, and the mobile object tracking information database is updated. Furthermore, the mobile object position estimation unit 262 deletes the used mobile object detection result information from the mobile object detection result database.

[0062] On the other hand, if no moving object is detected in the third frame image whose positional error with the estimated position is within a predetermined range, the moving object position estimation unit 262 does not update the moving object tracking information database, and therefore uses the estimated position information as the positional information of the first moving object on the image at the time of the third frame.

[0063] Furthermore, if the error between the position of the second moving object detected from the image of the third frame and the estimated position exceeds a predetermined range, the moving object position estimation unit 262 determines that the first moving object and the second moving object are different. That is, the moving object position estimation unit 262 determines that a second moving object different from the first moving object has been found, assigns a moving object ID to the second moving object, associates the moving object ID and position information of the second moving object with the associated information, and stores it in the moving object tracking information database as moving object tracking information.

[0064] The above moving object position comparison process is repeated for each moving object detection result obtained by the moving object detection process on the image signal obtained from one imaging device. Furthermore, if m moving objects are detected from the first group of frames of the image signal obtained from one imaging device (where m is a natural number), the moving object position estimation process in step S12 is repeated for each of the m moving objects.

[0065] If n new moving objects are discovered during this process (where n is a natural number), the moving object position estimation process in step S12 is repeated for each of the n additional moving objects. Furthermore, the processes in steps S11 to S12 are repeated as many times as there are imaging devices.

[0066] Alternatively, the mobile object position estimation unit 262 may assign the same mobile object ID to the first mobile object and the second mobile object if the error between the position of the second mobile object detected from the image of the third frame and the estimated position is within a predetermined range. If the first mobile object has already been assigned a mobile object ID, the mobile object position estimation unit 262 may assign the same mobile object ID to the second mobile object as to the first mobile object. In this way, the first mobile object and the second mobile object, which have been determined to be the same, are linked to each other.

[0067] The above estimation can utilize methods such as the Kalman filter for predicting object positions, or extrapolation or interpolation, which are statistical techniques for estimating data in unknown ranges.

[0068] A Kalman filter is a type of infinite impulse response filter used to estimate or control the state of a dynamic system using error-laden observations. It is used to estimate constantly changing quantities (for example, the position and velocity of a moving object) from discrete, error-laden observations.

[0069] For example, the Kalman filter calculates a prior estimate of the object's position at time (t+1) by combining the prior estimate at time (t+1), which is calculated from the posterior estimate at time t, with the observed value at time (t+1). The prior estimate at time (t+1) can be obtained by adding the object's movement vector to the posterior estimate at time t.

[0070] Here, the motion vector of the moving object can be determined as the difference vector between the position of the moving object at time t and the position of the moving object at time (t-1). Furthermore, the position of the moving object determined by the moving object detection unit 261 can be used as the observed value. Note that the observed values ​​at each time can be used as the two prior estimates immediately after the start of object position prediction.

[0071] A weighted average, for example, is used to combine the pre-estimated values ​​and observed values. In this combination, the weight given to each of the pre-estimated and observed values ​​is controlled by a parameter called the Kalman gain. Since it is not always possible to detect the same moving object in one frame as in the previous frame, the Kalman gain may be varied from frame to frame.

[0072] Figure 3 is a schematic diagram illustrating the change in the position of a moving object over time. While the position of the car M shown in Figure 1 is best represented on a two-dimensional map, for simplicity of explanation, the position of the car M is shown here using one-dimensional coordinates. In Figure 3, the vertical axis represents the one-dimensional coordinate x, and the horizontal axis represents time t. Additionally, frame numbers F1, F2, F3, ... are shown along the time axis as timing information related to the imaging point.

[0073] As shown in Figure 1, the first imaging device 11, the second imaging device 12, etc., capture images of the first spatial region A1, the second spatial region A2, etc., over multiple frames, respectively, thereby obtaining the first image signal, the second image signal, etc. In Figure 3, the coordinates of the automobile M, determined based on the first to third image signals, are shown by circles, squares, and triangles, respectively. The lines connecting them represent the movement path (trajectory) of the automobile M.

[0074] However, since car M may be hidden behind an obstacle or change direction from the perspective of the nearest imaging device, car M is not detected in frame F3 of the first image signal and frame F7 of the second image signal (*).

[0075] In such cases, the moving object position estimation unit 262 may, with respect to each image signal, make the weighting coefficient of the observed value equal to or greater than the weighting coefficient of the prior estimate for frames in which the moving object detection unit 261 was able to detect the automobile M, and make the weighting coefficient of the observed value smaller than the weighting coefficient of the prior estimate (for example, to zero) for frames in which the moving object detection unit 261 was unable to detect the automobile M.

[0076] In this way, by changing the weighting of the pre-estimated value and the observed value for each frame, it is possible to obtain a post-mortem estimate of the position of car M at frame F3 of the first image signal, even if car M is not detected in frame F3. Similarly, even if car M is not detected in frame F7 of the second image signal, it is possible to obtain a post-mortem estimate of the position of car M at frame F7.

[0077] In addition to the object position prediction method using the Kalman filter described above, the moving object position estimation unit 262 may also extrapolate the position of the same moving object at the time of imaging of a single frame, based on the position of the same moving object detected from images of multiple frames preceding or following a single frame in which the same moving object was not detected by the moving object detection unit 261.

[0078] Furthermore, the moving object position estimation unit 262 may also determine the position of the same moving object at the time of imaging of the aforementioned single frame by interpolation, based on the position of the same moving object detected from multiple frames that surround a single frame in which the same moving object was not detected by the moving object detection unit 261.

[0079] <Step S13> In step S13, the coordinate transformation unit 263 determines the coordinate information of the moving object in a predetermined coordinate system based on the image position information of the moving object detected from the image signal (in the first embodiment, the position information obtained by the moving object position estimation unit 262 in step S12).

[0080] The specified coordinate system is, for example, a one- or two-dimensional coordinate system that is substantially orthogonal to the vertical direction, a three-dimensional coordinate system with two axes substantially orthogonal to the vertical direction, a one- or two-dimensional coordinate system substantially parallel to the ground surface at the location where any imaging device is placed, or a three-dimensional coordinate system with two axes substantially parallel to that ground surface. Alternatively, an absolute coordinate system obtained using a satellite positioning system such as GPS may be used.

[0081] If the coordinates of three or more points defining a predetermined surface region including a moving object in an image represented by an image signal are known, the position of the moving object in the image can be transformed into the coordinates of the moving object in a predetermined coordinate system using a transformation matrix such as a perspective projection transformation matrix. The coordinate transformation definitions used for coordinate transformation processing by the coordinate transformation unit 263 are stored in the coordinate transformation information database of the storage unit 28.

[0082] The above coordinate transformation process is repeated for each mobile object tracking information stored in the mobile object tracking information database of the cache memory 27. The coordinate transformation unit 263 adds the coordinate information obtained by the coordinate transformation process to the mobile object tracking information and stores it in the mobile object tracking information database.

[0083] Figure 5 is a schematic diagram illustrating the relationship between camera images obtained from multiple imaging devices and a map image in a predetermined coordinate system. In Figure 5, (a) shows camera image 1 obtained from the first imaging device at a first time point in time, and (b) shows camera image 2 obtained from the second imaging device at a second time point in time, after a predetermined time (a predetermined number of frames) has elapsed from the first time point. (c) shows a map image onto which camera images 1 and 2 are projected.

[0084] For example, the camera image has 980 x 490 pixels, with the origin O(0,0) located in the upper left. The map image has 420 x 1390 pixels. In camera image 1, four reference points A(420,80), B(110,410), C(630,410), and D(720,80) are set on the road surface or on a reference plane within a predetermined distance from the road surface.

[0085] A first coordinate transformation definition is calculated and stored in the coordinate transformation information database in order to project four reference points on camera image 1 onto four reference points A'(120,140), B'(50,740), C'(300,740), and D'(380,140) on the map image. Similarly, four reference points are set in camera image 2, and a second coordinate transformation definition is calculated and stored in the coordinate transformation information database in order to project the four reference points on camera image 2 onto the four reference points on the map image.

[0086] The coordinate transformation unit 263 reads multiple coordinate transformation definitions, including a first coordinate transformation definition and a second coordinate transformation definition, from the coordinate transformation information database. The coordinate transformation unit 263 sequentially reads mobile object tracking information from the mobile object tracking information database and calculates the coordinates of the mobile object on the map image based on the position of the mobile object on the camera image, using the coordinate transformation definition that is suitable for the imaging device identified by the mobile object tracking information.

[0087] As a result, the position P1 (558,290) of a car traveling on the road in camera image 1 is converted to coordinates P1' (218,582) on the map image. Similarly, the position P2 (431,207) of a car traveling on the road in camera image 2 is converted to coordinates P2' (228,932) on the map image.

[0088] <Step S14> In step S14, the moving object tracking unit 264 performs moving object tracking (moving object tracking processing) by determining the correspondence between the moving object detected from multiple image signals obtained by imaging multiple spatial regions based on the coordinate information of the moving object in a predetermined coordinate system. As a result, even when multiple moving objects are captured in a single camera image and it is difficult to track the same moving object across multiple camera images, it becomes possible to easily track the same moving object on the map image.

[0089] For example, the moving object tracking unit 264 determines that two moving objects correspond if the distance (error) between the coordinates of the two moving objects, calculated from the images of the first image signal frame and the second image signal frame (corresponding frames) that are closest in time of acquisition within the overlapping portion of the first and second spatial regions captured, is within a predetermined range.

[0090] Alternatively, if the coordinates of multiple moving objects are obtained from the corresponding frame of one image signal, the moving object tracking unit 264 may determine that the moving object whose coordinates are obtained from the other image signal corresponds to the moving object whose coordinates were obtained from the other image signal, and which has the smallest distance (error) between that coordinate and the coordinate of the moving object obtained from the corresponding frame of the other image signal within a predetermined range.

[0091] Furthermore, if the moving object tracking unit 264 does not detect a moving object whose coordinate distance is within a predetermined range from the image of the first image signal frame and the second image signal frame (corresponding frame image) that are closest in time of imaging among the frames in which the overlapping portion of the first spatial region and the second spatial region is captured, it may use information on estimated coordinates (coordinates at the time of the corresponding frame) obtained based on the position or coordinates of the moving object detected from images of a predetermined number of other frames (for example, two or more frames) of the first or second image signal to determine the correspondence relationship of the moving object.

[0092] In the example shown in Figure 3, the moving object tracking unit 264 determines the correspondence between at least one moving object detected from a first image signal obtained by imaging a first spatial region A1 and at least one moving object detected from a second image signal obtained by imaging a second spatial region A2, based on the coordinate information of the moving object in a predetermined coordinate system.

[0093] For example, in a frame in which the overlapping portion of the first spatial region A1 and the second spatial region A2 is captured, if the distance between the coordinates of the moving object obtained from the first image signal (marked with a circle) and the coordinates of the moving object obtained from the second image signal (marked with a square) is within a predetermined range, then it is determined that the moving objects correspond. The coordinates of the moving object in frame F4 may be the coordinates obtained from the first image signal (marked with a circle), the coordinates obtained from the second image signal (marked with a square), or their average value, etc.

[0094] Furthermore, in the frame in which the overlapping portion of the second spatial region A2 and the third spatial region A3 was captured, for frame F7 of the second image signal and frame F7 of the third image signal corresponding to the time of capture, no moving objects with coordinates within a predetermined range were detected in the second image signal, but the coordinates of the moving objects in frame F7 (marked with *) were estimated based on the position or coordinates of the moving objects detected in frames F5 and F6 of the second image signal (marked with □).

[0095] In such cases, if the distance between the estimated coordinates (marked with *) obtained from the second image signal and the coordinates of the moving object (marked with △) obtained from the third image signal is within a predetermined range, it is determined that the moving objects correspond to each other. The coordinates of the moving object in frame F7 may be those obtained from the third image signal (marked with △).

[0096] Figure 6 is a flowchart illustrating a specific example of the mobile object tracking process. As a premise, the mobile object tracking information database in the cache memory 27 stores N mobile object tracking information (where N is an integer greater than or equal to 2), and the mobile object tracking unit 264 references the N mobile object tracking information using variable (i) or variable (j) (i or j = 1, 2, ..., N).

[0097] In step S141, the mobile object tracking unit 264 starts a loop of variable (i) by setting i=0. In step S142, the mobile object tracking unit 264 increments variable (i) by "1" and reads the mobile object tracking information M(i) from the mobile object tracking information database in the cache memory 27.

[0098] In step S143, the mobile object tracking unit 264 starts a loop of the variable (j) by setting j=0. In step S144, the mobile object tracking unit 264 increments the variable (j) by "1" and reads the mobile object tracking information M(j) from the mobile object tracking information database in the cache memory 27.

[0099] In step S145, the mobile object tracking unit 264 determines whether the mobile object tracking information M(i) and the mobile object tracking information M(j) were obtained by the same imaging device. If the mobile object tracking information M(i) and the mobile object tracking information M(j) were obtained by the same imaging device (YES), the process proceeds to step S148; otherwise (NO), the process proceeds to step S146.

[0100] In step S146, the mobile object tracking unit 264 determines whether or not map coordinates (coordinate information in a predetermined coordinate system) exist in the mobile object tracking information M(i) and mobile object tracking information M(j). Here, the existence of map coordinates means that the map coordinates stored in the mobile object tracking information database of the cache memory 27 are not empty (for example, the coordinate data has not been erased).

[0101] If map coordinates exist in the mobile object tracking information M(i) and mobile object tracking information M(j), the mobile object tracking unit 264 determines whether the distance between the two map coordinates is within a predetermined range. This determines the correspondence between the mobile object identified by the mobile object tracking information M(i) and the mobile object identified by the mobile object tracking information M(j).

[0102] If it is determined in step S146 that the distance between the map coordinates of the moving object tracking information M(i) and the moving object tracking information M(j) is within a predetermined range (YES), the process proceeds to step S147; otherwise (NO), the process proceeds to step S148.

[0103] In step S147, the mobile object tracking unit 264 determines that the mobile object identified by the mobile object tracking information M(i) corresponds to (is the same as) the mobile object identified by the mobile object tracking information M(j), and adds at least a portion of the map coordinates of the mobile object tracking information M(j) as part of the map coordinates of the mobile object tracking information M(i). Subsequently, the mobile object tracking unit 264 clears the map coordinates of the used mobile object tracking information M(j) and saves it (for example, by deleting the coordinate data of the mobile object tracking information M(j)).

[0104] In step S148, the moving object tracking unit 264 determines whether the loop for variable (j) has finished. If the loop for variable (j) has not finished, the process returns to step S144. If the loop for variable (j) has finished, the process proceeds to step S149.

[0105] In step S149, the mobile object tracking unit 264 determines whether the loop for variable (i) has finished. If the loop for variable (i) has not finished, the process returns to step S142. If the loop for variable (i) has finished, the mobile object tracking process ends.

[0106] <Step S15> In step S15, the tracking result processing unit 265 obtains information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple spatial regions, based on the coordinate information in a predetermined coordinate system of the moving object that the moving object tracking unit 264 determined to be corresponding to in step S14.

[0107] For example, the tracking result processing unit 265 sequentially reads mobile tracking information from the mobile tracking information database in the cache memory 27, selects mobile tracking information that includes non-empty map coordinates, and saves it to the tracking result information database in the storage unit 28. Furthermore, the tracking result processing unit 265 may use this to store the tracking result information obtained by the following tracking result processing, associated with supplementary information including the mobile ID, in the tracking result information database.

[0108] The tracking result processing unit 265 may read map data representing a two-dimensional or three-dimensional map from the map database in the storage unit 28, generate an image signal to display the position or movement path of at least one moving object on the map, and display the map image including the surrounding map on the display unit 22. Alternatively, the tracking result processing unit 265 may control the communication circuit 24 to transmit information about the detected moving object to an external device via a network.

[0109] Referring to Figure 3, the tracking result processing unit 265 obtains information regarding the position, movement path, or direction of movement of the vehicle M based on the coordinate information of the vehicle M detected in the first spatial region A1 to the third spatial region A3, etc. Furthermore, the tracking result processing unit 265 can obtain information regarding the vehicle M's movement speed or movement time, as well as the average movement speed or average movement time, based on the difference in the coordinate information of the vehicle M between two points.

[0110] Furthermore, the tracking result processing unit 265 may measure the movement routes of multiple moving objects, such as an unspecified number of automobiles, and based on that, count the number of automobiles passing through a specific area. This makes it possible to monitor the number of automobiles passing through the roads being surveyed in a traffic volume survey.

[0111] Furthermore, the tracking result processing unit 265 may calculate the inverse matrix of the transformation matrix used by the coordinate transformation unit 263 to obtain the inverse coordinate transformation definition, and when a moving object is specified on the map image, it may use the inverse coordinate transformation definition to display an image on the display unit 22 that represents the state in which the specified moving object is identified in the corresponding camera image.

[0112] In other words, the tracking result processing unit 265 may inversely transform the coordinates of a moving object specified on the map image to a position on multiple camera images, select a camera image in which the inversely transformed position of the moving object is within the effective pixels for each camera, generate an image signal representing an image that includes a marker to identify the moving object at the corresponding position in the selected camera image, and display that image on the display unit 22. Furthermore, the tracking result processing unit 265 may also display the moving object ID of the identified moving object in that image.

[0113] Figure 7 is a schematic diagram showing images in which a moving object specified on the map image is identified in camera image 1 and camera image 2. As shown in Figure 7(a), the tracking result processing unit 265 performs an inverse transformation on the coordinates P1'(218,582) of the moving object specified on the map image shown in Figure 5 to calculate the position P1(558,290) in camera image 1. Furthermore, the tracking result processing unit 265 identifies the car at position P1(558,290) and displays an image containing the car's moving object ID "0001" on the display unit 22.

[0114] Furthermore, as shown in Figure 7(b), the tracking result processing unit 265 performs an inverse transformation on the coordinates P2'(228,932) of the moving object specified on the map image shown in Figure 5 to calculate the position P2(431,207) in the camera image 2. In addition, the tracking result processing unit 265 identifies the car at position P2(431,207) and displays an image containing the moving object ID "0001" of that car on the display unit 22.

[0115] <Second example of operation> Next, a second example of operation of the mobile information processing device shown in Figure 2 will be described with reference to Figures 1 to 8. Figure 8 is a flowchart of the mobile information processing method according to the second embodiment of the present invention. In Figure 8, the same reference numerals are used for the same steps as in the first embodiment shown in Figure 4, and their explanation is omitted.

[0116] In the second embodiment, instead of the moving object position estimation process on the camera image in the first embodiment (step S12 in Figure 4), step S23 is performed, which is equivalent to the moving object coordinate estimation process in a predetermined coordinate system. Note that some of the processes shown in Figure 8 may be omitted or modified, or other processes may be added to the processes shown in Figure 8.

[0117] <Step S22> In step S22, the coordinate transformation unit 263 determines the coordinate information of the moving object in a predetermined coordinate system based on the image position information of the moving object detected from the image signal (in the second embodiment, the position information obtained by the moving object detection unit 261 in step S11). The details of the coordinate transformation process in step S22 are the same as in the first embodiment (step S13 in Figure 4). The coordinate transformation unit 263 adds the coordinate information obtained by the coordinate transformation process to the moving object detection result information and stores it in the moving object detection result database.

[0118] <Step S23> In step S23, the moving object position estimation unit 262 performs a moving object coordinate estimation process using the coordinate information obtained by the coordinate transformation unit 263 in step S22. The moving object position estimation unit 262 determines the estimated coordinates of the first moving object at the time of other frames based on the coordinates of the first moving object detected from images of a predetermined number of frames (for example, two or more frames) out of multiple frames of image signals obtained from one imaging device. If the error between the estimated coordinates and the coordinates of the second moving object detected from the images of the other frames is within a predetermined range, the unit determines that the first moving object and the second moving object are the same and saves the coordinate information of the second moving object detected from the images of the other frames as the coordinate information of the first moving object.

[0119] As an example, first, the moving object position estimation unit 262 reads moving object detection result information from the moving object detection result database in the cache memory 27. If a moving object is detected in consecutive first and second frames of multiple frames of an image signal obtained from a single imaging device, with the coordinates shifted within a predetermined distance, the unit determines that they are the same moving object (first moving object). The moving object position estimation unit 262 assigns a moving object ID (identification information) to the first moving object, associates the moving object ID and coordinate information of the first moving object with the associated information, and stores it as moving object tracking information in the moving object tracking information database in the cache memory 27.

[0120] Next, the mobile object position estimation unit 262 reads the mobile object tracking information for the first mobile object from the mobile object tracking information database and, based on the coordinates of the first mobile object detected from the images of the first and second frames, determines the estimated coordinates of the first mobile object at the time of the third frame following the second frame. The mobile object position estimation unit 262 stores the information of the estimated coordinates of the first mobile object at the time of the third frame in the mobile object tracking information database.

[0121] Next, a mobile object coordinate comparison process is performed. The mobile object position estimation unit 262 sequentially reads mobile object detection result information from the mobile object detection result database, and if the error between the coordinates of the second mobile object detected from the image of the third frame and the estimated coordinates is within a predetermined range, it determines that the first mobile object and the second mobile object are the same, and saves the coordinate information of the second mobile object detected from the image of the third frame as the coordinate information of the first mobile object in the mobile object tracking information database.

[0122] As a result, the estimated coordinate information of the first moving object at the time of the third frame is overwritten with the coordinate information of the first moving object, and the moving object tracking information database is updated. Furthermore, the moving object position estimation unit 262 deletes the used moving object detection result information from the moving object detection result database.

[0123] On the other hand, if no moving object is detected in the third frame image whose coordinate error with the estimated coordinates is within a predetermined range, the moving object position estimation unit 262 does not update the moving object tracking information database, and therefore uses the estimated coordinate information as the coordinate information of the first moving object at the time of the third frame.

[0124] Furthermore, if the error between the coordinates of the second moving object detected from the image of the third frame and the estimated coordinates exceeds a predetermined range, the moving object position estimation unit 262 determines that the first moving object and the second moving object are different. That is, the moving object position estimation unit 262 determines that a second moving object different from the first moving object has been found, assigns a moving object ID to the second moving object, associates the moving object ID and coordinate information of the second moving object with the associated information, and stores it in the moving object tracking information database as moving object tracking information.

[0125] The above moving object coordinate comparison process is repeated for each moving object detection result obtained by the moving object detection process and coordinate transformation process on the image signal obtained from one imaging device. Furthermore, if m moving objects are detected from the first group of frames of the image signal obtained from one imaging device (where m is a natural number), the moving object coordinate estimation process in step S23 is repeated for each of the m moving objects.

[0126] If n new moving objects are discovered during this process (where n is a natural number), the moving object coordinate estimation process in step S23 is repeated for each of the n additional moving objects. Furthermore, the processes from steps S11 to S23 are repeated for each imaging device.

[0127] Alternatively, the mobile object position estimation unit 262 may assign the same mobile object ID to the first mobile object and the second mobile object if the error between the coordinates of the second mobile object detected from the image of the third frame and the estimated coordinates is within a predetermined range. If the first mobile object has already been assigned a mobile object ID, the mobile object position estimation unit 262 may assign the same mobile object ID to the second mobile object as to the first mobile object. In this way, the first mobile object and the second mobile object, which have been determined to be the same, are linked to each other.

[0128] Similar to the first embodiment, the above estimation can be performed using methods such as object position prediction using a Kalman filter, or statistical methods for estimating data in an unknown range, such as extrapolation or interpolation.

[0129] <Step S24> In step S24, the moving object tracking unit 264 performs moving object tracking (moving object tracking process) by determining the correspondence of the moving object detected from multiple image signals obtained by imaging multiple spatial regions based on the coordinate information of the moving object in a predetermined coordinate system. The detailed processing in step S24 is the same as in the first embodiment (step S14 in Figure 4).

[0130] <Step S25> In step S25, the tracking result processing unit 265 obtains information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple spatial regions, based on the coordinate information in a predetermined coordinate system of the moving object that the moving object tracking unit 264 determined to correspond to in step S24. The detailed processing in step S25 is the same as in the first embodiment (step S15 in Figure 4).

[0131] <Effects of the Embodiment> According to the first or second embodiment, a moving object is detected from multiple frames of image signals obtained from at least one imaging device, coordinate information of the moving object in a predetermined coordinate system is obtained based on the position information of the moving object on the image, and the correspondence between the moving object detected in a first spatial region and the moving object detected in a second spatial region is determined based on the coordinate information of the moving object, thereby tracking of the moving object across multiple spatial regions. This makes it possible to continuously track an unspecified number of moving objects moving over a relatively wide area, or to obtain information regarding the position or movement path of the moving object with good accuracy using an imaging device placed outside the moving object.

[0132] Generally, when multiple imaging devices placed outside a moving object are used to image multiple spatial regions and obtain multiple image signals, a large error occurs between the position of the moving object predicted from one image and the position of the moving object detected from the other image when the moving object crosses over the images obtained by those imaging devices, making it difficult to recognize that those moving objects are the same.

[0133] According to the first or second embodiment, by determining the correspondence between a moving object detected in the first spatial region and a moving object detected in the second spatial region based on the distance between the coordinates of two moving objects obtained from the image of the closest frame of two image signals obtained by imaging a first spatial region and a second spatial region that partially overlap each other over multiple frames, it is possible to track moving objects across multiple spatial regions imaged by multiple imaging devices. This makes it possible to continuously track an unspecified number of vehicles traveling over a relatively wide area that cannot be captured by a single imaging device, and to monitor the number of vehicles traveling on the road being surveyed in traffic volume surveys.

[0134] The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the invention by those who have ordinary skill in the art. For example, it is possible to implement at least a part of the first embodiment and at least a part of the second embodiment in combination. [Industrial applicability]

[0135] The present invention can be used in a mobile information processing device that obtains information such as the position or travel path of a moving object, such as an automobile, based on an image signal. [Explanation of Symbols]

[0136] 10... Recording device, 11... First imaging device, 12... Second imaging device, 20... Mobile object information processing device, 21... Operation unit, 22... Display unit, 23... Audio input / output unit, 24... Communication circuit, 25... Interface, 26... CPU, 260... Image signal acquisition unit, 261... Mobile object detection unit, 262... Mobile object position estimation unit, 263... Coordinate transformation unit, 264... Mobile object tracking unit, 265... Tracking result processing unit, 27... Cache memory, 28... Storage unit

Claims

1. A mobile object information processing device that obtains information about a moving object based on an image signal obtained from at least one imaging device that captures a spatial region and generates an image signal, A moving object detection unit that detects a moving object from images of multiple frames of the aforementioned image signal, A coordinate transformation unit that determines the coordinate information of a moving object in a predetermined coordinate system based on the position information of the moving object on the image detected from the image signal, A moving object tracking unit determines, based on the coordinate information of the moving object in the predetermined coordinate system, the correspondence between at least one moving object detected from a first image signal obtained by imaging a first spatial region and at least one moving object detected from a second image signal obtained by imaging a second spatial region, A tracking result processing unit obtains information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple spatial regions, based on the coordinate information of the moving object in the predetermined coordinate system determined to correspond to the moving object by the moving object tracking unit. A mobile information processing device equipped with the following features.

2. The first image signal and the second image signal are obtained by imaging a first spatial region and a second spatial region, each partially overlapping, over multiple frames. The mobile object information processing device according to claim 1, wherein the mobile object tracking unit determines that the two mobile objects correspond when the distance between the coordinates of the two mobile objects, obtained from the images of the first image signal frame and the second image signal frame, respectively, which are captured at the closest time of imaging within the overlapping portion of the first spatial region and the second spatial region, is within a predetermined range.

3. The first image signal and the second image signal are obtained by imaging a first spatial region and a second spatial region, each partially overlapping, over multiple frames. The mobile object information processing device according to claim 1, wherein if the mobile object tracking unit does not detect a mobile object whose coordinate distance is within a predetermined range from the images of the frame of the first image signal and the frame of the second image signal that are closest in time of imaging among the frames in which the overlapping portion of the first spatial region and the second spatial region is captured, it uses information of estimated coordinates obtained based on the position or coordinates of the mobile object detected from a predetermined number of other frames of the first or second image signal to determine the correspondence of the mobile object.

4. The mobile object information processing device according to claim 1, further comprising a mobile object position estimation unit that determines the estimated position of the first mobile object at the time of another frame based on the position of the first mobile object detected from a predetermined number of frames among a plurality of frames of the image signal, determines that the first mobile object and the second mobile object are the same if the error between the position of the second mobile object detected from the images of the other frames and the estimated position is within a predetermined range, and stores the position information of the second mobile object detected from the images of the other frames as position information of the first mobile object on the image.

5. The mobile object information processing device according to claim 4, wherein if the mobile object position estimation unit does not detect a mobile object in the image of the other frame whose positional error with the estimated position is within a predetermined range, it uses the information of the estimated position as the positional information of the first mobile object on the image at the time of the other frame.

6. The mobile object information processing device according to claim 1, further comprising a mobile object position estimation unit that determines the estimated coordinates of the first mobile object at the time of another frame based on the coordinates of the first mobile object detected from a predetermined number of frames among a plurality of frames of the image signal, and determines that the first mobile object and the second mobile object are the same if the error between the coordinates of the second mobile object detected from the images of the other frames and the estimated coordinates is within a predetermined range, and stores the coordinate information of the second mobile object detected from the images of the other frames as coordinate information of the first mobile object.

7. The mobile object information processing device according to claim 6, wherein if the mobile object position estimation unit does not detect a mobile object in the image of the other frame whose coordinate error with the estimated coordinates is within a predetermined range, it uses the estimated coordinate information as the coordinate information of the first mobile object at the time of the other frame.

8. A plurality of imaging devices, including a first imaging device and a second imaging device, which image a first spatial region and a second spatial region, respectively, and generate a first image signal and a second image signal, respectively. A mobile information processing device according to any one of claims 1 to 7, A mobile information processing system equipped with the following features.

9. A mobile object information processing program used in a mobile object information processing device that obtains information about a moving object based on an image signal obtained from at least one imaging device that captures a spatial region and generates an image signal, A procedure (a) for detecting a moving object from multiple frames of the aforementioned image signal, A procedure (b) for determining the coordinate information of a moving object in a predetermined coordinate system based on the position information of the moving object on the image detected from the image signal, A procedure (c) for determining a correspondence between at least one moving object detected from a first image signal obtained by imaging a first spatial region and at least one moving object detected from a second image signal obtained by imaging a second spatial region, based on the coordinate information of the moving object in the predetermined coordinate system, Step (d) is a procedure to obtain information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple spatial regions, based on the coordinate information of the moving object in the predetermined coordinate system determined to be corresponding in step (c). A mobile information processing program that instructs the CPU to execute.

10. A method for processing information about a moving object, which obtains information about a moving object based on an image signal obtained from at least one imaging device that captures a spatial region and generates an image signal, Step (a) of detecting a moving object from images of multiple frames of the aforementioned image signal, (b) A step of determining the coordinate information of a moving object in a predetermined coordinate system based on the position information of the moving object on the image detected from the image signal, (c) A step of determining a correspondence between at least one moving object detected from a first image signal obtained by imaging a first spatial region and at least one moving object detected from a second image signal obtained by imaging a second spatial region, based on the coordinate information of the moving object in the predetermined coordinate system. Step (d) involves obtaining information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple spatial regions, based on the coordinate information of the moving object in the predetermined coordinate system determined to correspond in step (c). A mobile information processing method comprising the following:

Images