Mobile information processing device, system, program, and method

Abstract

A mobile object information processing device and the like capable of continuously tracking a mobile object moving between the vicinity and distance of an imaging device is provided. [Solution] This moving object information processing device includes an image processing unit that sets a first region of interest and a second region of interest that have overlapping portions within the image of each frame of an image signal; a moving object detection unit that detects moving objects from the first and second regions of interest; a coordinate conversion unit that determines coordinate information of the moving object in a predetermined coordinate system based on position information of the detected moving object on the image; a moving object tracking unit that determines a correspondence between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest based on the coordinate information of the moving object; and a tracking result processing unit that determines information regarding the position, etc. of moving objects across multiple regions of interest based on the coordinate information of moving objects determined to correspond.

Description

[Technical Field]

[0001] The present invention relates to a mobile object information processing device and a mobile object information processing method for obtaining information on the position, movement path, movement direction, movement speed, or movement time of a mobile object such as an automobile based on an image signal. Furthermore, the present invention relates to a mobile object information processing system including such a mobile object information processing device, and a mobile object information processing program used in the mobile object information processing device. In this application, the mobile object corresponds to an automobile including an automated guided vehicle or an autonomous vehicle, a vehicle including a motorcycle or a bicycle, a forklift, a self-propelled robot, or a walking person. [Background technology]

[0002] As a technology for detecting the positions of vehicles traveling on roads, for example, preventive safety systems have been developed to prevent accidents before they occur in order to reduce the number of fatalities and injuries caused by traffic accidents. Such preventive safety systems are activated when there is a high possibility of an accident occurring, and alert the driver when there is a possibility of a collision in front of the vehicle, or automatically brake when a collision becomes unavoidable to mitigate injuries to the occupants.

[0003] Patent Document 1 discloses a vehicle detection method that can be implemented inexpensively because it uses a monocular camera installed on a vehicle and can detect various types of vehicles because it uses the edges on both ends of the vehicle. However, because the appearance of a vehicle varies depending on the distance, high detection accuracy cannot be achieved simply by applying the same processing regardless of whether the vehicle is near or far. For example, because resolution decreases at long distances, highly discriminative features cannot be captured, resulting in reduced detection accuracy.

[0004] Patent Document 2 discloses an external environment recognition device that performs object detection appropriately regardless of distance when analyzing an image captured by an in-vehicle camera. This external environment recognition device includes a processing area setting unit that sets a first area and a second area for object detection within the image, and first and second object detection units that perform object detection in the set first and second areas, respectively. When the first object detection unit performs object detection, it uses only the object pattern, and when the second object detection unit performs object detection, it uses both the object pattern and the background pattern of the object pattern. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2005-156199 A (paragraphs 0002-0008) [Patent Document 2] JP 2013-061919 A ​​(paragraphs 0002-0009, 0014-0015, Figure 1A) DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0006] In Patent Document 2, object detection is performed in a processing area set within an image, and by narrowing the target area for object detection from the entire image to a relatively narrow processing area, it is possible to reduce the number of false detections that may occur during object detection. However, referring to Figure 1A of Patent Document 2, since the nearby processing area 8 and the distant processing area 9 are separated, when the object to be detected moves between the vicinity and the distance of the vehicle-mounted camera, it is difficult to continuously track the moving object.

[0007] Furthermore, the background pattern in Patent Document 2 refers to patterns other than the object pattern to be detected (vehicle rear pattern 13 in FIG. 1A) in the processing area for object detection, and includes unclear patterns such as background pattern 14 of object 12 in the distance. Therefore, even if both the object pattern and the background pattern of the object pattern are used, the detection accuracy of objects in the distance area is not necessarily improved.

[0008] Furthermore, when using an in-vehicle camera to detect moving objects in front of the vehicle, situations may arise where the camera is unable to detect an object in its blind spot, making a collision unavoidable. In addition, there is also the problem that adding an in-vehicle camera to a user's vehicle and maintaining it imposes a burden on the user.

[0009] In view of the above, a first object of the present invention is to provide a mobile object information processing device, program, method, etc., that can continuously track a mobile object moving between the proximity and distance of an imaging device while reducing the number of false detections that may occur during mobile object detection. A second object of the present invention is to improve the detection accuracy of a mobile object in a distant area in such a mobile object information processing device, etc.

[0010] Furthermore, a third object of the present invention is to provide a mobile object information processing system that can prevent traffic accidents by using such a mobile object information processing device, etc. A fourth object of the present invention is to realize such a mobile object information processing system by using any imaging device other than an in-vehicle camera that has a limited imaging range. [Means for solving the problem]

[0011] In order to solve at least part of the above problems, a mobile object information processing device according to a first aspect of the present invention is a mobile object information processing device that obtains information about a mobile object based on an image signal obtained from at least one imaging device that images a spatial region and generates an image signal, and includes: an image processing unit that sets a first region of interest and a second region of interest that have overlapping portions within an image of each frame of the image signal; a mobile object detection unit that detects a mobile object from the first and second regions of interest in images of multiple frames of the image signal; and a coordinate conversion unit that obtains coordinate information of the mobile object in a predetermined coordinate system based on position information on the image of the mobile object detected from the first and second regions of interest. a moving body position estimation unit that, for each region of interest, (1) calculates an estimated position of the first moving body at the time of another frame based on the position of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the position of the second moving body detected from the images of the other frames and the estimated position is within a predetermined range, determines that the first moving body and the second moving body are the same and saves information on the position of the second moving body as position information on the image of the first moving body; or (2) calculates estimated coordinates of the first moving body at the time of another frame based on the coordinates of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the coordinates of the second moving body detected from the images of the other frames and the estimated coordinates is within a predetermined range, determines that the first moving body and the second moving body are the same and saves information on the coordinates of the second moving body as coordinate information of the first moving body; The system includes a moving object tracking unit that determines a correspondence between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest based on coordinate information of the moving object in the specified coordinate system, and a tracking result processing unit that obtains information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple regions of interest based on coordinate information in the specified coordinate system of the moving object determined to be corresponding by the moving object tracking unit.

[0012] In a mobile object information processing device according to a second aspect of the present invention, the image processing unit sets, within an image of each frame of the image signal, a first region of interest corresponding to a first spatial region that is relatively close to the imaging device within the spatial region imaged by the imaging device, and a second region of interest corresponding to a second spatial region that is relatively far from the imaging device, and the moving object detection unit resamples image data representing the image in at least the second region of interest to extract a predetermined number of pixels to be subjected to the moving object detection process, thereby reducing the rate of reduction in the number of pixels compared to resampling image data representing the entire image.

[0013] Furthermore, a moving object information processing program according to a first aspect of the present invention is a moving object information processing program used in a moving object information processing device that obtains information about a moving object based on an image signal obtained from at least one imaging device that images a spatial region and generates an image signal, the program comprising: (a) a step of setting a first region of interest and a second region of interest having overlapping portions within an image of each frame of the image signal; (b) a step of detecting a moving object from the first and second regions of interest in images of multiple frames of the image signal; and (c) a step of obtaining coordinate information of the moving object in a predetermined coordinate system based on position information on the image of the moving object detected from the first and second regions of interest. For each region of interest, (1) prior to step (c), an estimated position of the first moving body at the time of another frame is calculated based on the position of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the position of the second moving body detected from the images of the other frames and the estimated position is within a predetermined range, the first moving body and the second moving body are determined to be the same, and information on the position of the second moving body is saved as position information on the image of the first moving body; or (2) following step (c), a step (d) is performed, in which an estimated coordinate of the first moving body at the time of another frame is calculated based on the coordinate of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the coordinate of the second moving body detected from the images of the other frames and the estimated coordinate is within a predetermined range, the first moving body and the second moving body are determined to be the same, and information on the coordinate of the second moving body is saved as coordinate information of the first moving body; a step of determining a correspondence relationship between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest based on coordinate information of the moving object in the predetermined coordinate system ( e ) and procedure ( e ) based on the coordinate information in the predetermined coordinate system of the moving object determined to correspond in the step (a), the step (b) of obtaining information on the position, the moving path, the moving direction, the moving speed, or the moving time of at least one moving object across a plurality of regions of interest. f ) and have the CPU execute it.

[0014] Furthermore, a moving object information processing method according to a first aspect of the present invention is a moving object information processing method for obtaining information about a moving object based on an image signal obtained from at least one imaging device that captures an image of a spatial region and generates an image signal, the method comprising: (a) setting a first region of interest and a second region of interest having overlapping portions within an image of each frame of the image signal; (b) detecting a moving object from the first and second regions of interest in the images of multiple frames of the image signal; and (c) obtaining coordinate information of the moving object in a predetermined coordinate system based on position information on the image of the moving object detected from the first and second regions of interest. For each region of interest, (1) prior to step (c), an estimated position of the first moving body at the time of another frame is calculated based on the position of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the position of the second moving body detected from the images of the other frames and the estimated position is within a predetermined range, the first moving body and the second moving body are determined to be the same, and information on the position of the second moving body is saved as position information on the image of the first moving body; or (2) following step (c), a step (d) is performed, in which an estimated coordinate of the first moving body at the time of another frame is calculated based on the coordinate of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the coordinate of the second moving body detected from the images of the other frames and the estimated coordinate is within a predetermined range, the first moving body and the second moving body are determined to be the same, and information on the coordinate of the second moving body is saved as coordinate information of the first moving body;determining a correspondence relationship between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest based on coordinate information of the moving object in the predetermined coordinate system ( e ) and step ( e (2) determining information about the position, movement path, movement direction, movement speed, or movement time of at least one moving object across a plurality of regions of interest based on coordinate information in the predetermined coordinate system of the moving object determined to correspond in ( f ) and

[0015] A mobile object information processing system according to a third aspect of the present invention may further include, in addition to the mobile object information processing device according to any one of the aspects of the present invention, at least one imaging device that captures an image of a spatial region and generates an image signal, and a mobile object information receiving device, a self-location detection device, a mobile object information providing device, or a driving control device mounted on the automobile. Here, the mobile object information processing device further includes a communication circuit that transmits information determined by the tracking result processing unit to the automobile, the mobile object information receiving device receives information transmitted from the communication circuit of the mobile object information processing device, the self-location detection device detects the position of the automobile, and the mobile object information providing device provides information about other mobile objects in the form of electronic data, images, or audio based on the information received by the mobile object information receiving device and the detection result of the self-location detection device. [Effects of the Invention]

[0016] According to a first aspect of the present invention, a first region of interest and a second region of interest having overlapping portions are set in an image of each frame of an image signal obtained from at least one imaging device, a moving object is detected from the regions of interest, coordinate information of the moving object in a predetermined coordinate system is obtained based on position information of the moving object in the image, and a correspondence relationship between the moving object detected from the first region of interest and the moving object detected from the second region of interest is determined based on the coordinate information of the moving object, thereby tracking the moving object across multiple regions of interest (moving object tracking process).This makes it possible to narrow the target region for moving object detection from the entire image to a relatively narrow region of interest, reducing the number of false detections that may occur during moving object detection, and continuously tracking a moving object moving between the vicinity and distance of the imaging device.

[0017] According to a second aspect of the present invention, an image processing unit sets a first region of interest corresponding to a region near the imaging device and a second region of interest corresponding to a region far from the imaging device within an image of each frame of an image signal, and a moving object detection unit resamples image data representing the image in at least the second region of interest to extract a predetermined number of pixels to be subjected to moving object detection processing, thereby making it possible to suppress the rate of reduction in the number of pixels compared to resampling image data representing the entire image. As a result, the resolution of the image portion corresponding to the far region can be improved, thereby improving the accuracy of moving object detection.

[0018] Furthermore, according to a third aspect of the present invention, a vehicle receives information transmitted from a communication circuit of a mobile object information processing device according to any one of the aspects of the present invention, detects the position of the vehicle, and provides information about other mobile objects in the form of telegram data, images, or audio based on the received information and the detection result of the position of the vehicle. Therefore, a mobile object information processing system can be provided that can prevent traffic accidents by using at least one imaging device placed on the road or the like to acquire images of other mobile objects in blind spots that cannot be captured by the vehicle's on-board camera, and transmitting information about the positions of the other mobile objects to the vehicle. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a schematic diagram showing a first configuration example of a mobile object information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram showing a second configuration example of a mobile object information processing system according to an embodiment of the present invention. [Figure 3] 3 is a block diagram showing an example of the configuration of the mobile information processing device shown in FIG. 1 or 2. FIG. [Figure 4] 3 is a flowchart showing a mobile object information processing method according to the first embodiment of the present invention. [Figure 5] FIG. 1 is a schematic diagram showing an example of a plurality of regions of interest set in a camera image obtained from an imaging device. [Figure 6] FIG. 10 is a schematic diagram illustrating an example of a change in the position of a moving object over time. [Figure 7] 2 is a schematic diagram for explaining the relationship between a camera image obtained from an imaging device and a map image in a predetermined coordinate system. FIG. [Figure 8] 10 is a flowchart illustrating a specific example of a moving object tracking process. [Figure 9] 10 is a schematic diagram showing an image in which a moving object designated on a map image is identified in a camera image. FIG. [Figure 10] 10 is a flowchart showing a mobile object information processing method according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the same reference numerals are used to designate the same components, and redundant description will be omitted. <Mobile Information Processing System 1> FIG. 1 is a schematic diagram showing a first configuration example of a mobile object information processing system according to an embodiment of the present invention.

[0021] As shown in Fig. 1, this system may include at least one imaging device 10 that captures an image of a spatial region and generates an image signal (video signal), and a mobile object information processing device 20 that obtains information about the mobile object based on the image signal obtained from the imaging device 10. Fig. 1 shows automobiles MA and MB as examples of mobile objects, but this system is capable of obtaining information about the position or travel route of at least one mobile object.

[0022] For example, this system can monitor the number of cars passing through a road being surveyed in a traffic volume survey by obtaining information about the positions or routes of an unspecified number of cars passing between the vicinity and distant locations of the imaging device 10. Alternatively, this system can be widely applied to situations where information about the positions or routes of moving vehicles, forklifts, and self-propelled robots is required in factories, airports, and stores.

[0023] In this system, an imaging device 10 captures an image of a spatial region A0 and generates an image signal. For example, the imaging device 10 may be fixedly installed on a predetermined road to detect the position or path of a moving object traveling on the road. Note that an existing security camera may also be used as the imaging device 10.

[0024] The imaging device 10 has a camera unit that captures an image of a subject with an image sensor via an optical system including at least one lens and generates a digital or analog image signal, and a communication unit that communicates wirelessly or wired with the mobile information processing device 20 and transmits the image signal generated by the camera unit to the mobile information processing device 20.

[0025] Alternatively, a recording device 10a that records an image signal generated by a camera unit may be used. The recording device 10a may be installed inside the imaging device 10, or may be located in a building or the like outside the imaging device 10 as shown in Fig. 1. The image signal recorded in the recording device 10a is supplied to the mobile information processing device 20 by wireless or wired communication, or by using a portable recording medium.

[0026] The recording medium in the recording device 10a or the portable recording medium may be a hard disk, a flexible disk, an optical disk, a magneto-optical disk, a magnetic tape, an SSD (solid state drive), or various types of memory including a USB memory.

[0027] Communication between the imaging device 10 and the mobile information processing device 20, communication between the imaging device 10 and the recording device 10a, or communication between the recording device 10a and the mobile information processing device 20 may be performed via various networks such as a wired LAN, a wireless LAN, an intranet, the Internet, or a mobile communication network.

[0028] The image signal generated by the imaging device 10 or the image signal recorded in the recording device 10a is supplied as measurement data to the mobile object information processing device 20 together with incidental information including identification information of the imaging device or timing information relating to the time of imaging. At least a part of the incidental information may be included in the image signal or in the file name of the image signal or measurement data.

[0029] The mobile object information processing device 20 obtains information regarding the position, movement route, movement direction, movement speed, movement time, etc. of the mobile object based on an image signal obtained from the imaging device 10 that captures an image of a spatial region and generates an image signal. For example, the mobile object information processing device 20 is configured by a server installed in a data center or a control center of a research company or a manufacturing company, or a PC (personal computer) and a display, etc.

[0030] <Mobile Information Processing System 2> 2 is a schematic diagram showing a second configuration example of a mobile object information processing system according to an embodiment of the present invention. In this system, in order to prevent traffic accidents, information about mobile objects such as automobiles and pedestrians traveling on roads (for example, automobile MB shown in FIG. 1) is obtained by a mobile object information processing device 20, and the information is provided to the driver of automobile MA.

[0031] The imaging device 10 is placed in a position and orientation that allows it to capture, for example, an image of the area ahead of or a blind spot area of ​​the automobile MA for which moving object information is provided. A blind spot area refers to an area that cannot be captured by the onboard camera of the automobile MA due to an obstruction such as a building or an oncoming vehicle when turning right, or an area that is blocked from the view of the driver of the automobile MA. If there are multiple blind spot areas, multiple imaging devices 10 may be placed.

[0032] 2, immediacy is required for the mobile object information processing device 20. In order to provide mobile object information in real time, it is desirable to reduce transmission delays by directly connecting the imaging device 10 and the mobile object information processing device 20. For example, an edge computer installed near the road on which the imaging device 10 is installed is used as the mobile object information processing device 20.

[0033] When connecting the imaging device 10 and the mobile information processing device 20 etc. via a wired connection, a method such as HDMI (High-Definition Multimedia Interface; registered trademark) or SDI (Serial Digital Interface) may be used.

[0034] The automobile MA is equipped with, for example, a mobile object information receiving device 31, a self-position detecting device 32, a mobile object information providing device 33, and a driving control device 34. The mobile object information receiving device 31 receives information transmitted from the communication circuit 24 of the mobile object information processing device 20, and obtains information regarding the position, movement route, movement direction, movement speed, movement time, etc. of at least one mobile object. Communication between the mobile object information processing device 20 and the mobile object information receiving device 31 may be performed via various networks such as a wireless LAN, an intranet, the Internet, or a mobile communication network.

[0035] The self-location detection device 32 detects the position of the automobile MA. For example, the self-location detection device 32 includes a receiver that complies with a positioning method such as GNSS (Global Navigation Satellite System), RTK (Real Time Kinematic), CLAS (Centimeter Level Augmentation Service), or SLAS (Submeter Level Augmentation Service), and calculates the current position of the automobile MA based on the radio waves received by the receiver.

[0036] Alternatively, the self-location detection device 32 may detect the position of the automobile MA using SLAM (Simultaneous Localization and Mapping) technology, which simultaneously identifies the self-location and grasps the structure of the surrounding environment. SLAM technology mainly includes LiDAR SLAM, which uses a laser sensor called LiDAR (Light Detection and Ranging), and Visual SLAM, which uses a camera as a sensor.

[0037] In addition, to make it easier to detect the position of the automobile MA, AR (Augmented Reality) markers, AR tags, QR codes (registered trademark), checkerboards, etc. may be placed at key points along the road on which the automobile MA travels.

[0038] The mobile object information providing device 33 provides information about other mobile objects passing on the road in the form of electronic data, images, or audio, based on the information received by the mobile object information receiving device 31 and the detection results of the self-position detecting device 32. The information about other mobile objects includes information that there is another mobile object near the automobile MA, information that there is a possibility of collision with another mobile object, etc.

[0039] For this purpose, the mobile object information providing device 33 supplies, for example, an image signal to a display unit (such as a display) and an audio signal to an audio output unit (such as a speaker). The display unit and the audio output unit may be configured as part of the mobile object information providing device 33, or may share the display unit and the audio output unit that are pre-installed in the automobile MA.

[0040] The driving control device 34 performs control operations related to the driving of the automobile MA when it determines that there is a high possibility of the automobile MA colliding with another moving object, using the telegram data provided by the mobile object information providing device 33. In this case, the driving control device 34 may control the accelerator / brake system 35 and the steering system 36 of the automobile MA.

[0041] <Mobile information processing device> Fig. 3 is a block diagram showing an example of the configuration of the mobile information processing device shown in Fig. 1 or 2. As shown in Fig. 3, the mobile information processing device 20 includes an operation unit 21, a display unit 22, an audio input / output unit 23, a communication circuit 24, an interface 25, a CPU (Central Processing Unit) 26, a cache memory 27, and a storage unit 28. The interface 25 to the storage unit 28 are connected to one another via a bus line. Note that some of the components shown in Figs. 1 to 3 may be omitted or modified, or other components may be added to the components shown in Figs. 1 to 3.

[0042] The operation unit 21 is composed of, for example, a keyboard, a mouse, etc., and is used to input various commands and data to the mobile information processing device 20. The display unit 22 includes, for example, an LCD (liquid crystal display), etc., and displays an operation screen, etc. The audio input / output unit 23 includes, for example, a microphone, an amplifier, a speaker, etc., and converts audio signals into electrical signals and converts electrical signals into audio signals.

[0043] In the second configuration example shown in FIG. 2, when an edge computer is used as the mobile information processing device 20, an HMI (Human Machine Interface) or a GUI (Graphical User Interface) may be used instead of the display unit 22 or the voice input / output unit 23.

[0044] The communication circuit 24 has a function of performing data communication with the imaging device 10, the recording device 10a, or the mobile information receiving device 31. The interface 25 is connected to the operation unit 21 to the communication circuit 24 and external devices such as a portable recording medium, and transmits various commands and data between them and the CPU 26. Furthermore, data input to the CPU 26 via the interface 25 is written to the cache memory 27 or the storage unit 28.

[0045] The CPU 26 performs various calculations and data processing in accordance with various software (including a mobile object information processing program) stored in the storage unit 28. The cache memory 27 and the storage unit 28 store various data used to obtain information about the mobile object and tracking result information about the mobile object in multiple databases (DB).

[0046] The cache memory 27 is configured with RAM (random access memory), etc. As a recording medium (storage medium) in the storage unit 28, various types of memories including a hard disk, a flexible disk, an optical disk, a magneto-optical disk, a magnetic tape, an SSD, or a ROM (read only memory) can be used.

[0047] Here, the CPU 26 and the mobile object information processing program stored in the storage unit 28 configure an image signal acquisition unit 260, an image processing unit 261, a mobile object detection unit 262, a mobile object position estimation unit 263, a coordinate conversion unit 264, a mobile object tracking unit 265, and a tracking result processing unit 266 as functional blocks.

[0048] The image signal acquisition unit 260 acquires measurement data including an image signal and accompanying information from the imaging device 10. For example, the image signal acquisition unit 260 may control the communication circuit 24 to receive the measurement data from the imaging device 10, or to receive the measurement data from a recording device 10a that records the image signal generated by the imaging device 10.

[0049] Alternatively, the image signal acquiring unit 260 may acquire the measurement data from a portable recording medium on which the measurement data is recorded when the recording medium is connected to the interface 25. For example, the image signal acquiring unit 260 stores the measurement data acquired from the imaging device 10 in the image database of the cache memory 27, and stores the measurement data acquired from the recording device 10a or the portable recording medium in the image database of the storage unit 28.

[0050] The moving body detection unit 262 to the tracking result processing unit 266 obtain information on the position, moving route, moving direction, moving speed, or moving time of at least one moving body based on the image signal included in the measurement data. For example, the position or moving route of the moving body on a predetermined map is obtained and displayed on the display unit 22.

[0051] <First operation example> Next, a first operation example of the mobile object information processing device shown in Fig. 3 will be described with reference to Fig. 1 to Fig. 9. Fig. 4 is a flowchart showing a mobile object information processing method according to the first embodiment of the present invention. Note that part of the processing shown in Fig. 4 may be omitted or changed, or other processing may be added to the processing shown in Fig. 4.

[0052] <Step S10> In step S10 of Fig. 4, the image signal acquisition unit 260 acquires measurement data including image signals and accompanying information from the imaging device 10, the recording device 10a, or a portable recording medium shown in Fig. 1. The image signal acquisition unit 260 stores the acquired measurement data in the image database of the cache memory 27 or the storage unit 28. As a result, image signals of multiple frames obtained from the imaging device 10 are stored in the image database.

[0053] <Step S11> In step S11, the image processing unit 261 reads out the image signals collected by the image signal acquisition unit 260 in step S10 from the image database. The image processing unit 261 sets multiple regions of interest having overlapping portions within the image of each frame (camera image) of the image signal obtained from the imaging device 10, in accordance with an operation by an operator using the operation unit 21 or region information previously stored in the storage unit 28 corresponding to the position and orientation of the imaging device 10.

[0054] For example, the image processing unit 261 cuts out image data of the set region of interest from each frame of the image signal, and supplies the extracted image data to the moving object detection unit 262. When two regions of interest are set in one frame of the camera image, two frames of region of interest image data are extracted from one frame of the image signal.

[0055] Fig. 5 is a schematic diagram showing an example of multiple regions of interest set in a camera image obtained from an imaging device. Fig. 5 shows camera images of frames FA and FB obtained by imaging device 10 capturing an image of a car MB traveling on a road from a distance toward the vicinity of imaging device 10. The origin O(0,0) on the image is located in the upper left corner of the camera image.

[0056] For example, image processing unit 261 sets a first region of interest A1 and a second region of interest A2 that have overlapping portions within an image (camera image) of each frame of an image signal obtained from imaging device 10. Note that if a rectangular region of interest is used, the regions can be easily set.

[0057] The first region of interest A1 corresponds to a first spatial region (nearby region) that is relatively close to the imaging device 10 within the spatial region A0 (FIG. 1) imaged by the imaging device 10. On the other hand, the second region of interest A2 corresponds to a second spatial region (far region) that is relatively far from the imaging device 10, and may have a smaller size (fewer pixels) than the first region of interest A1.

[0058] If imaging device 10 is positioned at a predetermined height facing diagonally downward and the angle is set so that imaging device 10 can capture images of the road surface from the near area to the far area, the first region of interest A1 corresponding to the near area will be located relatively lower in the camera image, and the second region of interest A2 corresponding to the far area will be located relatively higher. That is, the center of gravity of second region of interest A2 will be located higher than the center of gravity of first region of interest A1 in the camera image.

[0059] 5 is obtained by imaging an automobile MB with imaging device 10 at a first time point, and automobile MB is located within a second region of interest A2. Frame FB is obtained by imaging an automobile MB with imaging device 10 at a second time point after a predetermined time (a predetermined number of frames) has elapsed since the first time point, and automobile MB is located within a first region of interest A1.

[0060] <Step S12> In step S12, the moving object detection unit 262 performs moving object detection processing on the image signal representing each region of interest set by the image processing unit 261 in step S11. For example, the moving object detection unit 262 performs moving object detection processing on each region of interest image data supplied from the image processing unit 261. As a result, the moving object detection unit 262 detects moving objects from multiple regions of interest in multiple frame images of the image signal. As a result, position information of the moving object on the image is obtained in each frame of the camera image.

[0061] Generally, when capturing images of both near and far areas in space using a single imaging device, objects appear small, particularly in the image portion corresponding to the far area, which can result in a loss of detail, a decrease in image resolution, and reduced accuracy in detecting moving objects compared to when multiple imaging devices are placed along a road.

[0062] Furthermore, for example, in moving object detection processing using AI (Artificial Intelligence), the size of the matrix of input pixels (number of pixels) is fixed, so the input image must be resampled before the moving object detection processing is performed. In this case, downsampling is often performed, and since downsampling is performed uniformly on the input image, object information may be lost due to the reduced number of pixels, especially in images of distant areas consisting of few pixels.

[0063] 5, a second region of interest A2 is set to capture an image portion corresponding to a distant spatial region, and the moving object detection unit 262 resamples image data representing at least the image in the second region of interest A2 to extract a predetermined number of pixels to be subjected to the moving object detection process. This reduces the rate of pixel reduction compared to resampling image data representing the entire image. As a result, the resolution of the image portion corresponding to the distant region is improved, thereby improving the accuracy of moving object detection. A similar effect can also be achieved by resampling image data representing the image in the first region of interest A1, which captures an image portion corresponding to a nearby spatial region.

[0064] In particular, when the second region of interest A2 has a smaller size (fewer number of pixels) than the first region of interest A1, the moving object detection unit 262 resamples image data representing the image in the first region of interest A1 and image data representing the image in the second region of interest A2 in order to extract a predetermined number of pixels to be subjected to the moving object detection process, thereby making it possible to reduce the rate of reduction in the number of pixels in the second region of interest A2 to be less than the rate of reduction in the number of pixels in the first region of interest A1.

[0065] Alternatively, in order to extract a predetermined number of pixels to be subjected to the moving object detection process, the moving object detection unit 262 may increase the number of pixels in the second region of interest A2 by upsampling (oversampling) image data representing at least the image in the second region of interest A2, thereby improving the resolution of the image in the second region of interest A2 capturing the image portion corresponding to the distant region, thereby improving the accuracy of detecting the moving object.

[0066] Here, the number of pixels (horizontal, vertical) in the entire image is (M0, N0), the number of pixels (horizontal, vertical) in the first region of interest A1 is (M1, N1), and the number of pixels (horizontal, vertical) in the second region of interest A2 is (M2, N2). The number of pixels after resampling is (Mr0, Nr0), (Mr1, Nr1), and (Mr2, Nr2), respectively.

[0067] The pixel count reduction rate P0 when image data representing the entire image is resampled is expressed as P0 = (M0 x N0 - Mr0 x Nr0) / (M0 x N0). The pixel count reduction rate P1 when image data representing the image in the first region of interest A1 is resampled is expressed as P1 = (M1 x N1 - Mr1 x Nr1) / (M1 x N1), and the pixel count reduction rate P2 when image data representing the image in the second region of interest A2 is resampled is expressed as P2 = (M2 x N2 - Mr2 x Nr2) / (M2 x N2).

[0068] In this case, it is desirable to make P0>P1>P2 by resampling image data representing the image in each of the first region of interest A1 and the second region of interest A2, rather than resampling image data representing the entire image. Note that when the number of pixels is increased by upsampling, P1<0 or P2<0 in the above formula. In this application, "reducing the rate of reduction in the number of pixels" means reducing the value of the rate of reduction, and when the number of pixels is increased, the rate of reduction takes a negative value.

[0069] The first image resampling technique in this embodiment is to interpolate new pixels between original pixels of the input image in the case of upsampling. For example, the average value of K original pixels (K is an integer equal to or greater than 2) surrounding one new pixel may be used as the value of the new pixel. Even if the resolution of the original camera image is limited, by applying an appropriate interpolation process, it is possible to improve the image resolution in the region of interest and increase the accuracy of detecting moving objects.

[0070] The second technique involves resampling the input image to a ratio different from its aspect ratio. In other words, the horizontal and vertical directions of the input image are resampled at different ratios. This allows for the region of interest to be a rectangle of any aspect ratio, but allows for moving object detection processing on an image with a nearly constant number of pixels (number of horizontal pixels x number of vertical pixels). The third technique involves copying nearby pixel values ​​in uniform regions of the input image and resampling only in complex regions. This reduces calculation time.

[0071] In this embodiment, the target area for moving object detection can be narrowed down from the entire camera image to a relatively narrow region of interest, and moving object detection processing can be performed on the region of interest, which makes it possible to reduce the number of false detections that can occur during moving object detection and continuously track moving objects that move between close and far away from the image capture device 10. This moving object detection processing can use AI object detection processing, image recognition processing, etc.

[0072] For example, AI-based object detection processing involves dividing an image in each region of interest into many small segments, labeling each segment with information about what is in it, and associating a category with it. If an object with the same label or category is detected in two consecutive frames with a position shift within a specified distance, it is recognized as the same object having moved. This makes it possible to detect moving objects whose position changes across multiple frames of an image signal.

[0073] Alternatively, the moving object detection unit 262 may detect a moving object from a region of interest in multiple frames of images by performing a predetermined image recognition process on the image signals of multiple frames representing each region of interest. In this image recognition process, for example, an active appearance model can be used, which separates the image of the target object into shape and texture and compresses the dimensions of each by principal component analysis, thereby making it possible to represent changes in the shape and texture of the target object with fewer parameters.

[0074] In the active appearance model, the shape vector of all feature points is expressed using the average shape vector calculated in advance from training data and the eigenvector matrix obtained by principal component analysis of the deviations from the average shape vector. In other words, the shape vector is expressed as the average shape vector plus the product of the eigenvector matrix and the shape parameters.

[0075] Similarly, the appearance vector, which is an arrangement of normalized texture brightness values, is expressed using a mean appearance vector calculated in advance from training data and an eigenvector matrix obtained by principal component analysis of deviations from the mean appearance vector. That is, the appearance vector is expressed as the mean appearance vector plus the product of the eigenvector matrix and the appearance parameters.

[0076] Shape and appearance parameters are parameters that represent variations from the average, and changing them can change the shape and appearance. Furthermore, because there is a correlation between shape and appearance, by further performing principal component analysis on the shape and appearance parameters, it is possible to express the shape vector and appearance vector using a low-dimensional parameter vector (local parameter) that controls both shape and appearance.

[0077] Next, parameters related to global changes in the location, size, and orientation of the target object in the image (movement parameters) are considered. In an active appearance model, model search involves changing the above parameters to generate images of the target object by locally and globally modifying the model, and then comparing the generated images with the input image to find the parameter values ​​that minimize the error.

[0078] If the error falls below a threshold, it may be determined that the input image contains a set of feature points that match the model being searched for, and the error minimization may be terminated. Conversely, if the error does not fall below a threshold, it may be determined that the input image does not contain a set of feature points that match the model being searched for.

[0079] In this embodiment, since the target object is limited to a moving object, differential image signals between multiple frames are calculated, and the target object is estimated based on the differential image signals. Data on its feature points may be used together with the training data or may be used instead of the training data to calculate the average shape vector and average appearance vector.

[0080] When the moving object detection unit 262 detects a target object in a region of interest in an image of one frame, it changes the above parameters to locally and globally change the model of the target object to generate an image of the target object in another frame, compares the generated image with the image of the other frame, and when the error is minimized or below a threshold, extracts multiple feature points corresponding to the image of the target object in the image of the other frame. This makes it possible to detect a moving object from the region of interest in the images of multiple frames.

[0081] After the moving object detection process is completed, the moving object detection unit 262 may assign a bounding box to each detected moving object. A bounding box is a partial region that surrounds an object in an image, and is generally used to determine the location of a specific object in an image and classify the object.

[0082] Classification is a method of classifying objects by type. For example, if an image shows a car as a moving object, the moving object is classified as a "passenger car," "truck," "bus," etc. It is also possible to calculate the probability that the moving object is a "passenger car," "truck," "bus," etc.

[0083] Bounding boxes can be used to identify the location of a specific object in an image, and play a very important role in a wide range of fields, such as autonomous driving systems and security camera systems. Furthermore, using rectangular bounding boxes makes image annotation a simpler process than segmentation (pixel-by-pixel labeling).

[0084] When several moving objects are detected by applying the above-described moving object detection process to multiple frames of image signals representing the region of interest, the moving object detection unit 262 associates the position information of the detected moving objects on the image (which may include a bounding box) with additional information and stores it in the moving object detection result database in the cache memory 27 as moving object detection result information.

[0085] <Step S13> Step S13 is performed using the moving object detection result information obtained by performing the moving object detection process on the image signal in step S12.

[0086] In step S13, the moving body position estimation unit 263 determines, for each region of interest, an estimated position of the first moving body at the time of another frame based on the position of the first moving body detected from images of a predetermined number of frames (e.g., two or more frames) among the multiple frames of the image signal, and if the error between the position of the second moving body detected from the image of the other frame and the estimated position is within a predetermined range, it determines that the first moving body and the second moving body are the same, and saves the position information of the second moving body detected from the image of the other frame as position information on the image of the first moving body.

[0087] As an example, first, the moving object position estimation unit 263 reads moving object detection result information from the moving object detection result database in the cache memory 27, and when moving objects are detected from images of consecutive first and second frames among the multiple frames of the image signal with their positions shifted within a predetermined distance, it determines that they are the same moving object (first moving object). The moving object position estimation unit 263 assigns a moving object ID (identification information) to the first moving object, associates the moving object ID and position information of the first moving object with accompanying information, and stores the associated moving object ID and position information of the first moving object in the moving object tracking information database in the cache memory 27 as moving object tracking information.

[0088] From then on, the moving object position estimation unit 263 reads the moving object tracking information of the first moving object from the moving object tracking information database, and calculates an estimated position of the first moving object at the time of the third frame following the second frame based on the position of the first moving object detected from the images of the first and second frames. The moving object position estimation unit 263 stores information on the estimated position of the first moving object at the time of the third frame in the moving object tracking information database.

[0089] Next, a moving object position comparison process is performed. The moving object position estimation unit 263 sequentially reads out moving object detection result information from the moving object detection result database, and if the error between the position of the second moving object detected from the image of the third frame and the estimated position is within a predetermined range, determines that the first moving object and the second moving object are the same, and saves the information of the position of the second moving object detected from the image of the third frame in the moving object tracking information database as position information on the image of the first moving object.

[0090] As a result, the information on the estimated position of the first moving object at the time of the third frame is rewritten with the position information of the first moving object, and the moving object tracking information database is updated. Furthermore, the moving object position estimation unit 263 deletes the used moving object detection result information from the moving object detection result database.

[0091] On the other hand, if no moving object is detected in the image of the third frame whose position error from the estimated position is within a specified range, the moving object position estimation unit 263 will not update the moving object tracking information database, and will use the estimated position information as the position information on the image of the first moving object at the time of the third frame.

[0092] Furthermore, if the error between the position of the second moving object detected from the image of the third frame and the estimated position exceeds a predetermined range, the moving object position estimation unit 263 determines that the first moving object and the second moving object are different. That is, the moving object position estimation unit 263 determines that a second moving object different from the first moving object has been discovered, assigns a new moving object ID to the second moving object, associates the moving object ID and position information of the second moving object with the accompanying information, and stores the association result in the moving object tracking information database as moving object tracking information.

[0093] The above moving object position comparison process is repeated the number of times equal to the number of pieces of moving object detection result information obtained by the moving object detection process for the image signals obtained from the imaging device 10. Furthermore, when m moving objects are detected from the first group of frames of the image signals obtained from the imaging device 10 (m is a natural number), the moving object position estimation process of step S13 is repeated for the m moving objects.

[0094] If n new moving objects are discovered in the process (n is a natural number), the moving object position estimation process of step S13 is repeated for the n new moving objects. Furthermore, the processes of steps S12 to S13 are repeated the number of times equal to the number of regions of interest set by the image processing unit 261.

[0095] Alternatively, the moving object position estimation unit 263 may assign the same moving object ID to the first moving object and the second moving object when the error between the position of the second moving object detected from the image of the third frame and the estimated position is within a predetermined range. If a moving object ID has already been assigned to the first moving object, the moving object position estimation unit 263 may assign the same moving object ID as the moving object ID of the first moving object to the second moving object. In this way, the first moving object and the second moving object that are determined to be the same are linked to each other.

[0096] For the above estimation, an object position prediction method using a Kalman filter, or extrapolation or interpolation, which are statistical methods for estimating data in an unknown range, can be used.

[0097] A Kalman filter is a type of infinite impulse response filter that uses observations with errors to estimate or control the state of a dynamic system. It is used to estimate quantities that change over time (such as the position and speed of a moving object) from observations with discrete errors.

[0098] For example, a Kalman filter calculates a posterior estimate of the position of a moving object at time (t+1) by combining a prior estimate at time (t+1) calculated from a posterior estimate at time t with the observed value at time (t+1). The prior estimate at time (t+1) can be calculated by adding the movement vector of the moving object to the posterior estimate at time t.

[0099] Here, the movement vector of the moving object can be calculated as a difference vector between the position of the moving object at time t and the position of the moving object at time (t-1). The position of the moving object calculated by the moving object detection unit 262 can be used as the observed value. The two prior estimated values ​​immediately after the start of object position prediction can be the observed values ​​at the respective times.

[0100] The prior estimated value and the observed value are combined, for example, by weighted averaging. In this combination, the degree of importance attached to either the prior estimated value or the observed value is controlled using a parameter called the Kalman gain. Since it may not be possible to detect the same moving object in an image of one frame as the moving object detected in the image of the previous frame, the Kalman gain may be variable for each frame.

[0101] Fig. 6 is a schematic diagram illustrating an example of a change in the position of a moving object over time. The position of the automobile MB shown in Fig. 1 and other figures is suitable for display on a two-dimensional map, but here, for ease of explanation, the position of the automobile MB is displayed in one-dimensional coordinates. In Fig. 6, the vertical axis represents one-dimensional coordinate x, and the horizontal axis represents time t. Furthermore, frame numbers F1, F2, F3, ... are shown along the time axis as timing information related to the time of imaging.

[0102] 6, the coordinates of the automobile MB detected from the second region of interest A2 and the first region of interest A1 in the images obtained by imaging the spatial region A0 (FIG. 1) over multiple frames by the imaging device 10 shown in FIG. 1 are indicated by circles and squares, respectively. The line connecting these indicates the movement path (trajectory) of the automobile MB.

[0103] However, since the car MB may be hidden behind an obstacle or change direction when viewed from the imaging device 10, the car MB is not detected (marked with an *) in the second area of ​​interest A2 of the image of frame F3 and the first area of ​​interest A1 of the image of frame F9.

[0104] In such a case, the moving body position estimation unit 263 may, for each region of interest, set the weighting coefficient of the observation value equal to or greater than the weighting coefficient of the pre-estimated value for frames in which the moving body detection unit 262 was able to detect an automobile MB, and may set the weighting coefficient of the observation value smaller than the weighting coefficient of the pre-estimated value (for example, to zero) for frames in which the moving body detection unit 262 was unable to detect an automobile MB.

[0105] In this way, by varying the weighting of the prior estimates and the observed values ​​for each frame, it is possible to obtain a posteriori estimate of the position of the automobile MB at the time of frame F3 even if the automobile MB is not detected in the second region of interest A2 of the image of frame F3. Also, it is possible to obtain a posteriori estimate of the position of the automobile MB at the time of frame F9 even if the automobile MB is not detected in the first region of interest A1 of the image of frame F9.

[0106] In addition to the object position prediction method using the Kalman filter described above, the moving body position estimation unit 263 may also extrapolate the position of the same moving body at the time of capturing the above-mentioned one frame based on the position of the same moving body detected from the same region of interest in images of multiple frames preceding or following the one frame in which the same moving body was not detected by the moving body detection unit 262.

[0107] In addition, the moving body position estimation unit 263 may interpolate the position of the same moving body at the time of capturing the one frame based on the positions of the same moving body detected from the same region of interest in images of multiple frames surrounding one frame in which the same moving body was not detected by the moving body detection unit 262.

[0108] <Step S14> In step S14, the coordinate conversion unit 264 determines the coordinate information of the moving body in a predetermined coordinate system based on the position information on the image of the moving body detected from multiple regions of interest in the image represented by the image signal (in the first embodiment, the position information determined by the moving body position estimation unit 263 in step S13).

[0109] The predetermined coordinate system may be, for example, a one- or two-dimensional coordinate system that is nearly perpendicular to the vertical direction, a three-dimensional coordinate system with two axes that are nearly perpendicular to the vertical direction, a one- or two-dimensional coordinate system that is nearly parallel to the ground surface where one of the imaging devices is located, or a three-dimensional coordinate system with two axes that are nearly parallel to the ground surface. Alternatively, an absolute coordinate system obtained using a positioning method such as GNSS, RTK, CLAS, or SLAS may be used.

[0110] If the coordinates of three or more points defining a predetermined surface area including a moving object in an image represented by an image signal are known, the position of the moving object on the image can be converted into the coordinates of the moving object in a predetermined coordinate system using a transformation matrix such as a perspective projection transformation matrix. The coordinate transformation definition used in the coordinate transformation process by the coordinate transformation unit 264 is stored in a coordinate transformation information database in the storage unit 28.

[0111] The above coordinate conversion process is repeated as many times as the number of pieces of moving object tracking information stored in the moving object tracking information database in the cache memory 27. The coordinate conversion unit 264 adds the coordinate information obtained by the coordinate conversion process to the moving object tracking information and stores it in the moving object tracking information database.

[0112] Fig. 7 is a schematic diagram for explaining the relationship between a camera image obtained from an imaging device and a map image in a predetermined coordinate system. In Fig. 7, (a) shows the camera image of frame FA, (b) shows the camera image of frame FB, and (c) shows the map image onto which the camera images of frames FA and FB are projected.

[0113] For example, the camera image has 980 x 490 pixels, with the origin O(0,0) located in the upper left corner of the image. The map image has 420 x 1390 pixels. In the camera image, multiple reference points are set on the road surface or on a reference plane within a predetermined distance from the road surface.

[0114] 7, eight reference points A to H are set in the camera image so as to overlap with the four vertices of the first region of interest A1 and the four vertices of the second region of interest A2. In order to project these reference points onto the eight reference points A' to H' on the map image, a coordinate transformation definition is calculated and stored in a coordinate transformation information database.

[0115] Alternatively, to improve the accuracy of the coordinate transformation, a coordinate transformation definition may be calculated for each region of interest. In this case, for example, a first coordinate transformation definition is calculated for a first region of interest A1 to project four reference points A to D on the camera image onto four reference points A' to D' on the map image, and a second coordinate transformation definition is calculated for a second region of interest A2 to project four reference points E to H on the camera image onto four reference points E' to H' on the map image.

[0116] The coordinate conversion unit 264 reads out the coordinate conversion definition from the coordinate conversion information database. The coordinate conversion unit 264 also sequentially reads out the moving object tracking information from the moving object tracking information database, and calculates the coordinates of the moving object on the map image based on the position of the moving object on the camera image using the coordinate conversion definition specified by the moving object tracking information.

[0117] As a result, the position P1 (581,80) of the car traveling on the road in the camera image of frame FA is converted to coordinates P1' (256,269) on the map image. Also, the position P2 (544,329) of the car traveling on the road in the camera image of frame FB is converted to coordinates P2' (231,897) on the map image.

[0118] <Step S15> In step S15, the moving object tracking unit 265 performs tracking of the moving objects (moving object tracking process) by determining the correspondence between the moving objects detected from the multiple regions of interest based on the coordinate information of the moving objects in a predetermined coordinate system. This makes it possible to easily track the same moving object on the map image even when multiple moving objects are captured in one region of interest and it is difficult to track the same moving object across multiple regions of interest.

[0119] For example, the moving object tracking unit 265 determines that two moving objects correspond to each other when the distance (error) between the coordinates of two moving objects calculated from two regions of interest in an image of the same frame of the image signal is within a predetermined range.

[0120] Alternatively, when the coordinates of multiple moving objects are determined from one region of interest in an image of the same frame of the image signal, the moving object tracking unit 265 may determine that the moving object having coordinates whose distance (error) between it and the coordinates of the moving object determined from the other region of interest is the smallest within a predetermined range corresponds to the moving object whose coordinates are determined from the other region of interest.

[0121] In addition, when a moving object is not detected whose coordinate distance is within a predetermined range from two regions of interest in an image of the same frame of the image signal, the moving object tracking unit 265 may use information on estimated coordinates (coordinates at the time of the same frame) calculated based on the position or coordinates of the moving object detected from at least one region of interest in an image of another predetermined number of frames (e.g., two or more frames) of the image signal to determine the correspondence relationship of the moving object.

[0122] In the example shown in Figure 6, the moving object tracking unit 265 determines a correspondence between at least one moving object detected from a first region of interest A1 and at least one moving object detected from a second region of interest A2 based on coordinate information of the moving object in a predetermined coordinate system.

[0123] For example, in the image of frame F6, if the distance between the coordinates (squares) of a moving object determined from the first region of interest A1 and the coordinates (circles) of a moving object determined from the second region of interest A2 is within a predetermined range, it is determined that the moving objects correspond to each other. The coordinates of the moving object in frame F6 may be the coordinates (squares) determined from the first region of interest A1, the coordinates (circles) determined from the second region of interest A2, or an average value thereof.

[0124] On the other hand, if, in the image of frame 6, a moving object having coordinates within a predetermined range of the distance between the moving object coordinates determined from the first region of interest A1 and the moving object coordinates is not detected from the second region of interest A2, the coordinates of the moving object in frame F6 are estimated, for example, based on the position or coordinates of the moving object detected from the second region of interest A2 in the images of frames F4 and F5.

[0125] In such a case, if the distance between the coordinates (□) of the moving object obtained from the first region of interest A1 and the estimated coordinates obtained from the second spatial region A2 is within a predetermined range, it is determined that the moving objects correspond to each other. The coordinates (□) obtained from the first region of interest A1 may be used as the coordinates of the moving object in frame F6.

[0126] 8 is a flowchart showing a specific example of moving object tracking processing. As a premise, N pieces of moving object tracking information are stored in the moving object tracking information database of the cache memory 27 (N is an integer equal to or greater than 2), and the moving object tracking unit 265 refers to the N pieces of moving object tracking information using a variable (i) or a variable (j) (i or j=1, 2, . . . , N).

[0127] In step S151, the moving object tracking unit 265 starts a loop of the variable (i) by setting i = 0. In step S152, the moving object tracking unit 265 increments the variable (i) by 1 and reads the moving object tracking information M(i) from the moving object tracking information database in the cache memory 27.

[0128] In step S153, the moving object tracking unit 265 starts a loop of the variable (j) by setting j = 0. In step S154, the moving object tracking unit 265 increments the variable (j) by "1" and reads the moving object tracking information M(j) from the moving object tracking information database in the cache memory 27.

[0129] In step S155, the moving object tracking unit 265 determines whether the moving object tracking information M(i) and the moving object tracking information M(j) relate to different regions of interest in the image of the same frame. If the moving object tracking information M(i) and the moving object tracking information M(j) relate to different regions of interest in the image of the same frame (YES), the process proceeds to step S156; if not (NO), the process proceeds to step S158.

[0130] In step S156, the moving object tracking unit 265 determines whether or not map coordinates (coordinate information in a predetermined coordinate system) exist in the moving object tracking information M(i) and the moving object tracking information M(j). Here, the existence of map coordinates means that the map coordinates stored in the moving object tracking information database in the cache memory 27 are not empty (for example, the coordinate data has not been erased).

[0131] If map coordinates exist in the moving object tracking information M(i) and the moving object tracking information M(j), the moving object tracking unit 265 determines whether the distance between the two map coordinates is within a predetermined range, thereby determining the correspondence between the moving object identified by the moving object tracking information M(i) and the moving object identified by the moving object tracking information M(j).

[0132] If it is determined in step S156 that the distance between the map coordinates of the moving object tracking information M(i) and the moving object tracking information M(j) is within a predetermined range (YES), the process proceeds to step S157; if not (NO), the process proceeds to step S158.

[0133] In step S157, the moving object tracking unit 265 determines that the moving object identified by the moving object tracking information M(i) and the moving object identified by the moving object tracking information M(j) correspond (are the same), and adds at least a part of the map coordinates of the moving object tracking information M(j) to part of the map coordinates of the moving object tracking information M(i). Thereafter, the moving object tracking unit 265 empties the map coordinates of the moving object tracking information M(j) that has been used and saves it (for example, erases the coordinate data of the moving object tracking information M(j)).

[0134] In step S158, the moving object tracking unit 265 determines whether the loop for the variable (j) has ended. If the loop for the variable (j) has not ended, the process returns to step S154, and if the loop for the variable (j) has ended, the process proceeds to step S159.

[0135] In step S159, the moving object tracking unit 265 determines whether the loop for the variable (i) has ended. If the loop for the variable (i) has not ended, the process returns to step S152, and when the loop for the variable (i) has ended, the moving object tracking process ends.

[0136] <Step S16> In step S16, the tracking result processing unit 266 obtains information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving body across multiple regions of interest based on coordinate information in a specified coordinate system of the moving body that was determined to correspond by the moving body tracking unit 265 in step S15.

[0137] For example, the tracking result processing unit 266 sequentially reads out moving object tracking information from the moving object tracking information database in the cache memory 27, selects moving object tracking information including non-empty map coordinates, and stores it in the tracking result information database in the storage unit 28. Furthermore, the tracking result processing unit 266 may use this to associate tracking result information obtained by the following tracking result processing with additional information including the moving object ID and store it in the tracking result information database.

[0138] The tracking result processing unit 266 may read map data representing a two-dimensional or three-dimensional map from the map database in the storage unit 28, generate an image signal for displaying the position, etc. of at least one moving object on the map, and display a map image in which the position, movement path, movement direction, movement speed, movement time, etc. of the at least one moving object are specified on the display unit 22. Alternatively, the tracking result processing unit 266 may control the communication circuit 24 to transmit information about the detected moving object to an external device via a network.

[0139] 6, the tracking result processing unit 266 obtains information about the series of positions, movement path, or movement direction of the automobile MB based on the coordinate information of the automobile MB detected in the first region of interest A1 and the second region of interest A2. The tracking result processing unit 266 can also obtain information about the movement speed or movement time of the automobile MB, or the average movement speed or average movement time, based on the difference in the coordinate information of the automobile MB between two points, etc.

[0140] Furthermore, the tracking result processing unit 266 may measure the movement routes of multiple moving objects, for example, an unspecified number of automobiles, and based on the measurement, count the number of automobiles passing through a specific area, thereby making it possible to monitor the number of automobiles passing through a road subject to a traffic volume survey.

[0141] In addition, the tracking result processing unit 266 may calculate the inverse matrix of the transformation matrix used by the coordinate transformation unit 264 to obtain a coordinate inverse transformation definition, and when a moving object is specified on the map image, may use the coordinate inverse transformation definition to display on the display unit 22 an image representing the specified moving object identified in the corresponding region of interest.

[0142] That is, the tracking result processing unit 266 may inversely convert the coordinates of the moving object specified on the map image into a position on the camera image, select a frame in which the inversely converted position of the moving object exists in the image of the moving object in the above-mentioned area of ​​interest, generate an image signal representing an image including a landmark that identifies the moving object at the corresponding position in the camera image of the selected frame, and display the image on the display unit 22.

[0143] For example, the tracking result processing unit 266 displays an image including a moving object ID, a type of moving object, or position information in the camera image, corresponding to the moving object identified in the image, on the display unit 22. At this time, the bounding box determined by the moving object detection unit 262 may be used.

[0144] 9 is a schematic diagram showing an image in which a moving object specified on a map image is identified in a camera image. For example, when a moving object at coordinates P1' (256, 269) is specified on the map image shown in FIG. 7(c), the tracking result processing unit 266 inversely transforms the coordinates P1' (256, 269) of the specified moving object to calculate a position P1 (581, 80) in the camera image of frame FA. Furthermore, the tracking result processing unit 266 identifies the moving object at that position and causes the display unit 22 to display an image including the ID "01" of the identified moving object, the type of moving object "car" (passenger vehicle), or the position "P1 (581, 80)" in the camera image.

[0145] 7(c), when a moving object at coordinate P2' (231,897) is specified, the tracking result processing unit 266 inversely transforms the coordinate P2' (231,897) of the specified moving object to calculate a position P2 (544,329) in the camera image of frame FB. Furthermore, the tracking result processing unit 266 identifies the moving object at that position, and causes the display unit 22 to display an image including the ID "01" of the identified moving object, the type of moving object "car" (passenger vehicle), or the position "P2 (544,329)" in the camera image.

[0146] As another example, as shown in FIG. 2, the mobile object information processing device 20 can perform data communication with a mobile object information receiving device 31 mounted on an automobile MA. The automobile MA may be an autonomously driven bus, a passenger car, a truck, or the like. When the tracking result processing unit 266 obtains information (mobile object information) regarding the position, movement route, movement direction, movement speed, movement time, or the like of a mobile object (e.g., automobile MB shown in FIG. 1) traveling on a road, the tracking result processing unit 266 controls the communication circuit 24 to transmit the mobile object information to the automobile MA. The communication circuit 24 transmits the mobile object information obtained by the tracking result processing unit 266 to the automobile MA.

[0147] In the automobile MA, the mobile object information receiving device 31 receives the mobile object information transmitted from the communication circuit 24 of the mobile object information processing device 20, and the self-position detecting device 32 detects the position of the automobile MA and obtains coordinate information of the automobile MA in a predetermined coordinate system. The mobile object information providing device 33 may provide the driver with information in the form of images or audio about other mobile objects that may be in a collision with the automobile MA, based on the mobile object information received by the mobile object information receiving device 31 and the coordinate information of the automobile MA obtained by the self-position detecting device 32.

[0148] For example, the mobile object information providing device 33 compares the coordinates of the automobile MA obtained from the self-position detection device 32 with the coordinates of other mobile objects obtained from the mobile object information receiving device 31 in the same coordinate system to determine the distance between the automobile MA and the other mobile objects, and determines whether other mobile objects are present within a specified distance from the automobile MA.

[0149] When the mobile object information providing device 33 determines that another mobile object is present within a predetermined distance from the mobile object MA, it may control the display unit or audio output unit of the mobile object MA to display or announce information that another mobile object is present near the mobile object MA, or information that there is a possibility of a collision with another mobile object.

[0150] Furthermore, the mobile object information providing device 33 may provide the driving control device 34 with information on other mobile objects that may collide with the motor vehicle MA as electronic data based on the mobile object information received by the mobile object information receiving device 31 and the coordinate information of the motor vehicle MA determined by the self-location detecting device 32. The driving control device 34 performs control operations related to the driving of the motor vehicle MA using the electronic data provided by the mobile object information providing device 33. For example, the driving control device 34 calculates the possibility of the motor vehicle MA colliding with another motor vehicle, a pedestrian, etc., or the predicted time until the collision.

[0151] Based on the calculation results, the driving control device 34 may control the display unit or audio output unit of the vehicle MA to display or announce information regarding the possibility of collision or the estimated time until collision. Alternatively, the driving control device 34 may compare the risk level represented by the calculation results with at least one threshold value and perform various control operations according to the risk level.

[0152] For example, when the risk level exceeds a first threshold, the driving control device 34 may determine that the risk level is medium and control the audio output unit of the automobile MA to emit a warning sound to alert the driver. Alternatively, when the risk level exceeds a second threshold, the driving control device 34 may determine that the risk level is high and control the accelerator / brake system 35 to slow down or stop the automobile MA, or control the steering system 36 to divert the direction of travel of the automobile MA away from other automobiles, pedestrians, etc.

[0153] 1, as the automobile MA continues to move forward, the automobile MA may enter the imaging range of the imaging device 10. In such a case, it is desirable to associate the coordinate information of the moving body obtained from the moving body information receiving device 31 with the coordinate information of the automobile MA obtained from the self-position detecting device 32 so that the automobile MA imaged by the imaging device 10 is not recognized as another moving body.

[0154] For example, the mobile information providing device 33 may compare the coordinates of the mobile body MC obtained from the mobile information receiving device 31 with the coordinates of the automobile MA obtained from the self-position detection device 32 in the same coordinate system, and if the distance between them is within a predetermined range, determine that the mobile body MC and the automobile MA correspond (are the same mobile body).

[0155] Alternatively, the mobile object information providing device 33 may collect the coordinates of the mobile object MC obtained from the mobile object information receiving device 31 and the coordinates of the automobile MA obtained from the self-location detecting device 32 over a predetermined period of time, compare the moving distance and moving direction over the predetermined period of time, and determine whether they belong to the same mobile object. Even if the coordinate information obtained from the self-location detecting device 32 contains an error, the influence of the error can be reduced by comparing the moving distance and moving direction, which are the difference.

[0156] When the moving object MC captured by the imaging device 10 corresponds to the automobile MA, the moving object information providing device 33 can avoid providing erroneous information that another moving object is present near the automobile MA. In addition, the determination result of the moving object information providing device 33 is also supplied to the driving control device 34 as part of the telegram data, so that the driving control device 34 can avoid malfunctioning in the driving control of the automobile MA.

[0157] Furthermore, when it is determined that the moving object MC captured by the imaging device 10 corresponds to the automobile MA, the moving object information providing device 33 or the driving control device 34 may use the coordinate information of the moving object MC obtained from the moving object information receiving device 31 as the coordinate information of the automobile MA. This makes it possible to accurately grasp the relative positional relationship between the automobile MA and other moving objects when the coordinate information obtained from the self-position detecting device 32 contains an error.

[0158] According to the mobile object information processing system shown in Figure 2, a mobile object information processing system can be provided that can prevent traffic accidents by using at least one imaging device placed on the road or the like to obtain images of other mobile objects that are in blind spots that cannot be captured by the onboard camera of the automobile MA, and transmitting information regarding the positions of the other mobile objects to the automobile MA.

[0159] <Second operation example> Next, a second operation example of the mobile object information processing device shown in Fig. 3 will be described with reference to Fig. 1 to Fig. 10. Fig. 10 is a flowchart showing a mobile object information processing method according to the second embodiment of the present invention. In Fig. 10, the same steps as those in the first embodiment shown in Fig. 4 are given the same reference numerals and their description will be omitted.

[0160] In the second embodiment, instead of the moving object position estimation process on the camera image in the first embodiment (step S13 in FIG. 4), a moving object coordinate estimation process equivalent to the first embodiment is performed in a predetermined coordinate system in step S24. Note that part of the process shown in FIG. 10 may be omitted or changed, or other processes may be added to the process shown in FIG. 10.

[0161] <Step S23> Step S23 is performed using the moving object detection result information obtained by performing the moving object detection process on the image signal in step S12.

[0162] In step S23, the coordinate conversion unit 264 determines coordinate information of the moving object in a predetermined coordinate system based on position information on the image of the moving object detected from multiple regions of interest in the image represented by the image signal (in the second embodiment, the position information determined by the moving object detection unit 262 in step S12). Details of the coordinate conversion process in step S23 are the same as in the first embodiment (step S14 in FIG. 4). The coordinate conversion unit 264 adds the coordinate information determined by the coordinate conversion process to the moving object detection result information and stores it in the moving object detection result database.

[0163] <Step S24> In step S24, the moving object position estimation unit 263 performs moving object coordinate estimation processing using the coordinate information obtained in step S23 by the coordinate conversion unit 264. For each region of interest, the moving object position estimation unit 263 obtains estimated coordinates of the first moving object at the time of the other frame based on the coordinates of the first moving object detected from images of a predetermined number of frames (e.g., two or more frames) among the multiple frames of the image signal, and determines that the first moving object and the second moving object are the same when an error between the coordinates of the second moving object detected from the images of the other frames and the estimated coordinates is within a predetermined range, and stores information on the coordinates of the second moving object detected from the images of the other frames as coordinate information of the first moving object.

[0164] As an example, first, the moving object position estimation unit 263 reads moving object detection result information from the moving object detection result database in the cache memory 27, and when moving objects are detected from images of consecutive first and second frames among the multiple frames of the image signal with coordinates shifted within a predetermined distance, it determines that they are the same moving object (first moving object). The moving object position estimation unit 263 assigns a moving object ID (identification information) to the first moving object, associates the moving object ID and coordinate information of the first moving object with accompanying information, and stores the association result in the moving object tracking information database in the cache memory 27 as moving object tracking information.

[0165] From then on, the moving object position estimation unit 263 reads the moving object tracking information of the first moving object from the moving object tracking information database, and calculates the estimated coordinates of the first moving object at the time of the third frame following the second frame based on the coordinates of the first moving object detected from the images of the first and second frames. The moving object position estimation unit 263 stores the information of the estimated coordinates of the first moving object at the time of the third frame in the moving object tracking information database.

[0166] Next, a moving object coordinate comparison process is performed. The moving object position estimation unit 263 sequentially reads out moving object detection result information from the moving object detection result database, and if the error between the coordinates of the second moving object detected from the image of the third frame and the estimated coordinates is within a predetermined range, determines that the first moving object and the second moving object are the same, and stores the information of the coordinates of the second moving object detected from the image of the third frame in the moving object tracking information database as coordinate information of the first moving object.

[0167] As a result, the information on the estimated coordinates of the first moving object at the time of the third frame is rewritten with the coordinate information of the first moving object, and the moving object tracking information database is updated. Furthermore, the moving object position estimation unit 263 deletes the used moving object detection result information from the moving object detection result database.

[0168] On the other hand, if no moving object is detected in the image of the third frame whose coordinate error from the estimated coordinates is within a predetermined range, the moving object position estimation unit 263 will not update the moving object tracking information database, and will use the estimated coordinate information as the coordinate information of the first moving object at the time of the third frame.

[0169] Furthermore, if the error between the coordinates of the second moving object detected from the image of the third frame and the estimated coordinates exceeds a predetermined range, the moving object position estimation unit 263 determines that the first moving object is different from the second moving object. That is, the moving object position estimation unit 263 determines that a second moving object different from the first moving object has been discovered, assigns a new moving object ID to the second moving object, associates the moving object ID and coordinate information of the second moving object with the accompanying information, and stores the association result in the moving object tracking information database as moving object tracking information.

[0170] The above moving body coordinate comparison process is repeated the number of times equal to the number of pieces of moving body detection result information obtained by the moving body detection process and coordinate conversion process for the image signals obtained from the imaging device 10. Furthermore, when m moving bodies are detected from the images of the first group of frames of the image signals obtained from the imaging device 10 (m is a natural number), the moving body coordinate estimation process of step S24 is repeated for the m moving bodies.

[0171] If n new moving objects are discovered in the process (n is a natural number), the moving object coordinate estimation process of step S24 is repeated for the n new moving objects. Furthermore, the processes of steps S12 to S24 are repeated the number of times equal to the number of regions of interest set by the image processing unit 261.

[0172] Alternatively, the moving object position estimation unit 263 may assign the same moving object ID to the first moving object and the second moving object when the error between the coordinates of the second moving object detected from the image of the third frame and the estimated coordinates is within a predetermined range. If a moving object ID has already been assigned to the first moving object, the moving object position estimation unit 263 may assign the same moving object ID as the moving object ID of the first moving object to the second moving object. As a result, the first moving object and the second moving object that are determined to be the same are linked to each other.

[0173] As in the first embodiment, the above estimation can use an object position prediction method using a Kalman filter, or extrapolation or interpolation, which is a statistical method for estimating data in an unknown range.

[0174] <Step S25> In step S25, the moving object tracking unit 265 performs tracking of the moving object (moving object tracking process) by determining the correspondence relationship of the moving object detected from the multiple regions of interest based on the coordinate information of the moving object in a predetermined coordinate system. The detailed process in step S25 is the same as in the first embodiment (step S15 in FIG. 4).

[0175] <Step S26> In step S26, the tracking result processing unit 266 obtains information on the position, moving path, moving direction, moving speed, moving time, etc. of at least one moving object across multiple regions of interest based on the coordinate information in the predetermined coordinate system of the moving object determined to correspond by the moving object tracking unit 265 in step S25. The detailed processing in step S26 is the same as in the first embodiment (step S16 in FIG. 4).

[0176] <Effects of the embodiment> According to the first or second embodiment, a first region of interest A1 and a second region of interest A2 having overlapping portions are set in an image of each frame of an image signal obtained from at least one imaging device 10, a moving object is detected from these regions of interest, coordinate information of the moving object in a predetermined coordinate system is obtained based on position information of the moving object in the image, and a correspondence relationship between the moving object detected from the first region of interest A1 and the moving object detected from the second region of interest A2 is determined based on the coordinate information of the moving object, thereby tracking the moving object across multiple regions of interest (moving object tracking process).This makes it possible to narrow the target region for moving object detection from the entire image to a relatively narrow region of interest, reducing the number of false detections that may occur during moving object detection, and continuously tracking a moving object moving between near and far from the imaging device 10.

[0177] The present invention is not limited to the above-described embodiments, and many modifications within the technical spirit of the present invention are possible by those skilled in the art. For example, it is also possible to combine at least a part of the first embodiment with at least a part of the second embodiment. [Industrial Applicability]

[0178] The present invention can be used in a mobile object information processing device or the like that obtains information about the position or travel route of a mobile object such as an automobile based on an image signal. [Explanation of symbols]

[0179] 10...imaging device, 10a...recording device, 20...mobile object information processing device, 21...operation unit, 22...display unit, 23...audio input / output unit, 24...communication circuit, 25...interface, 26...CPU, 260...image signal acquisition unit, 261...image processing unit, 262...mobile object detection unit, 263...mobile object position estimation unit, 264...coordinate conversion unit, 265...mobile object tracking unit, 266...tracking result processing unit, 27...cache memory, 28...storage unit, 31...mobile object information receiving device, 32...self-position detection device, 33...mobile object information providing device, 34...driving control device, 35...accelerator / brake system, 36...steering system

Claims

1. A mobile object information processing device that obtains information about a mobile object based on an image signal obtained from at least one imaging device that captures an image of a spatial region and generates an image signal, comprising: an image processing unit that sets a first region of interest and a second region of interest having overlapping portions within an image of each frame of the image signal; a moving object detection unit that detects a moving object from the first and second regions of interest in a plurality of frame images of the image signal; a coordinate conversion unit that calculates coordinate information of the moving object in a predetermined coordinate system based on position information on the image of the moving object detected from the first and second regions of interest; a moving body position estimation unit that, for each region of interest, (1) calculates an estimated position of a first moving body at the time of another frame based on the position of a first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the estimated position and the position of a second moving body detected from the images of the other frames is within a predetermined range, determines that the first moving body and the second moving body are the same and saves information on the position of the second moving body as position information on the image of the first moving body; or (2) calculates estimated coordinates of the first moving body at the time of another frame based on the coordinates of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the estimated coordinates and the coordinates of the second moving body detected from the images of the other frames is within a predetermined range, determines that the first moving body and the second moving body are the same and saves information on the coordinates of the second moving body as coordinate information of the first moving body; a moving object tracking unit that determines a correspondence relationship between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest based on coordinate information of the moving object in the predetermined coordinate system; a tracking result processing unit that obtains information regarding the position, moving path, moving direction, moving speed, or moving time of at least one moving object across multiple regions of interest based on coordinate information in the predetermined coordinate system of the moving object that is determined to correspond by the moving object tracking unit; A mobile information processing device comprising:

2. the image processing unit sets, within an image of each frame of the image signal, a first region of interest corresponding to a first spatial region that is relatively close to the imaging device within a spatial region imaged by the imaging device, and a second region of interest corresponding to a second spatial region that is relatively far from the imaging device; A mobile object information processing device as described in claim 1, wherein the moving object detection unit resamples image data representing an image in at least the second region of interest in order to extract a predetermined number of pixels to be targeted for moving object detection processing, thereby reducing the rate of reduction in the number of pixels compared to resampling image data representing the entire image.

3. the image processing unit sets, within an image of each frame of the image signal, a first region of interest corresponding to a first spatial region that is relatively close to the imaging device within a spatial region imaged by the imaging device, and a second region of interest that corresponds to a second spatial region that is relatively far from the imaging device and has a size smaller than the first region of interest; 2. The mobile object information processing device of claim 1, wherein the moving object detection unit resamples image data representing an image in the first region of interest and image data representing an image in the second region of interest to extract a predetermined number of pixels to be subjected to the moving object detection process, thereby reducing the rate of reduction in the number of pixels in the second region of interest to be lower than the rate of reduction in the number of pixels in the first region of interest.

4. 2. The mobile object information processing device according to claim 1, wherein the moving object tracking unit determines that the two moving objects correspond to each other when the distance between the coordinates of the two moving objects obtained from the first and second regions of interest in the image of the same frame of the image signal is within a predetermined range.

5. 2. The mobile object information processing device of claim 1, wherein when a mobile object is not detected whose coordinate distance from the first and second regions of interest in an image of the same frame of the image signal is within a predetermined range, the mobile object tracking unit determines the correspondence relationship of the mobile object using information on estimated coordinates obtained based on the position or coordinates of the mobile object detected from the first or second regions of interest in an image of another predetermined number of frames of the image signal.

6. 2. The mobile object information processing device of claim 1, wherein the mobile object position estimation unit determines an estimated position of the first mobile object at the time of the other frame, and if no mobile object is detected in the image of the other frame whose position error from the estimated position is within a predetermined range, uses information on the estimated position as position information on the image of the first mobile object at the time of the other frame.

7. 2. The mobile object information processing device according to claim 1, wherein the mobile object position estimation unit determines estimated coordinates of the first mobile object at the time of the other frame, and if no mobile object is detected in the image of the other frame whose coordinate error from the estimated coordinates is within a predetermined range, uses information on the estimated coordinates as coordinate information of the first mobile object at the time of the other frame.

8. a mobile object information processing device according to any one of claims 1 to 7, further comprising a communication circuit for transmitting information determined by the tracking result processing unit to a vehicle; a mobile information receiving device mounted on the vehicle and configured to receive information transmitted from the communication circuit of the mobile information processing device; a self-position detection device mounted on the vehicle and detecting the position of the vehicle; a mobile object information providing device mounted on the vehicle, which provides information about other mobile objects in the form of electronic data, images, or audio based on the information received by the mobile object information receiving device and the detection results of the self-location detecting device; A mobile information processing system comprising:

9. 9. The mobile information processing system according to claim 8, further comprising a driving control device mounted on the vehicle and performing control operations related to driving of the vehicle using the information received by the mobile information receiving device and the detection results of the self-location detection device.

10. 1. A mobile object information processing program used in a mobile object information processing device that obtains information about a mobile object based on an image signal obtained from at least one imaging device that captures an image of a spatial region and generates an image signal, comprising: a step (a) of setting a first region of interest and a second region of interest having overlapping portions within an image of each frame of the image signal; (b) detecting a moving object from the first and second regions of interest in a plurality of frame images of the image signal; (c) calculating coordinate information of the moving object in a predetermined coordinate system based on position information on the image of the moving object detected from the first and second regions of interest; For each region of interest, (1) prior to step (c), an estimated position of the first moving body at the time of another frame is calculated based on the position of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the position of the second moving body detected from the images of the other frames and the estimated position is within a predetermined range, the first moving body and the second moving body are determined to be the same, and information on the position of the second moving body is saved as position information on the image of the first moving body; or (2) following step (c), a step (d) is performed, in which an estimated coordinate of the first moving body at the time of another frame is calculated based on the coordinate of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the coordinate of the second moving body detected from the images of the other frames and the estimated coordinate is within a predetermined range, the first moving body and the second moving body are determined to be the same, and information on the coordinate of the second moving body is saved as coordinate information of the first moving body; a step (e) of determining a correspondence relationship between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest based on coordinate information of the moving object in the predetermined coordinate system; a step (f) of determining information on the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple regions of interest based on coordinate information in the predetermined coordinate system of the moving object determined to correspond in the step (e); A mobile information processing program that causes a CPU to execute the above.

11. 1. A mobile object information processing method for obtaining information about a mobile object based on an image signal obtained from at least one imaging device that captures an image of a spatial region and generates an image signal, comprising: (a) setting a first region of interest and a second region of interest having overlapping portions within an image of each frame of the image signal; (b) detecting a moving object from the first and second regions of interest in the images of the plurality of frames of the image signal; (c) determining coordinate information of the moving object in a predetermined coordinate system based on position information on the image of the moving object detected from the first and second regions of interest; For each region of interest, (1) prior to step (c), an estimated position of the first moving body at the time of another frame is calculated based on the position of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the position of the second moving body detected from the images of the other frames and the estimated position is within a predetermined range, the first moving body and the second moving body are determined to be the same, and information on the position of the second moving body is saved as position information on the image of the first moving body; or (2) following step (c), a step (d) is performed, in which an estimated coordinate of the first moving body at the time of another frame is calculated based on the coordinate of the first moving body detected from images of a predetermined number of frames among the multiple frames of the image signal, and if an error between the coordinate of the second moving body detected from the images of the other frames and the estimated coordinate is within a predetermined range, the first moving body and the second moving body are determined to be the same, and information on the coordinate of the second moving body is saved as coordinate information of the first moving body; (e) determining a correspondence relationship between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest based on coordinate information of the moving object in the predetermined coordinate system; a step (f) of determining information regarding a position, a moving path, a moving direction, a moving speed, or a moving time of at least one moving object across a plurality of regions of interest based on coordinate information in the predetermined coordinate system of the moving object determined to correspond in the step (e); A mobile information processing method comprising:

Images