Mobile information processing device, system, program, and method

Abstract

The present invention provides a mobile object information processing device, etc., capable of continuously tracking a moving object that moves between the vicinity and distance of an imaging device. [Solution] This mobile object information processing device includes an image processing unit that sets a first region of interest and a second region of interest having overlapping portions within the image of each frame of an image signal; a mobile object detection unit that detects a mobile object from the first and second regions of interest; a coordinate transformation unit that obtains coordinate information of the mobile object in a predetermined coordinate system based on the image position information of the detected mobile object; a mobile object tracking unit that determines a correspondence between at least one mobile object detected from the first region of interest and at least one mobile object detected from the second region of interest based on the coordinate information of the mobile object; and a tracking result processing unit that obtains information regarding the position of the mobile object across multiple regions of interest based on the coordinate information of the mobile objects that have been determined to correspond.

Description

Technical Field

[0001] The present invention relates to a moving body information processing apparatus and a moving body information processing method for obtaining information regarding the position, moving route, moving direction, moving speed, or moving time of a moving body such as an automobile based on an image signal. Further, the present invention relates to a moving body information processing system including such a moving body information processing apparatus, a moving body information processing program used in the moving body information processing apparatus, and the like. In the present application, examples of the moving body include automobiles including automated guided vehicles and autonomous vehicles, vehicles including motorcycles and bicycles, forklifts, autonomous robots, or walking humans.

Background Art

[0002] As a technology for detecting the position and the like of a vehicle passing on a road, for example, a preventive safety system for preventing accidents in order to reduce the number of casualties due to traffic accidents has been developed. Such a preventive safety system operates in a situation where an accident is likely to occur, alerts a driver by means of an alarm when a collision in front of the host vehicle is likely to occur, or reduces the damage to passengers by means of an automatic brake when a collision cannot be avoided.

[0003] Patent Document 1 discloses a vehicle detection method that can be realized at a low cost by performing vehicle detection with a monocular camera installed in a vehicle and can detect various types of vehicles by using the edges at both ends of the vehicle. However, since the appearance of a vehicle varies depending on the distance, high detection accuracy cannot be achieved by simply applying the same process regardless of whether it is in the far vicinity or not. For example, since the resolution decreases in the distance, features with high discriminability cannot be captured and the detection accuracy decreases.

[0004] Patent Document 2 discloses an external environment recognition device that analyzes images captured by an in-vehicle camera to detect objects, and performs object detection suitably regardless of distance. This external environment recognition device comprises a processing area setting unit that sets a first region and a second region for object detection within the image, and first and second object detection units that perform object detection in the set first region and second region, respectively. When the first object detection unit performs object detection, it uses only the object pattern, and when the second object detection unit performs object detection, it uses both the object pattern and the background pattern of the object pattern. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2005-156199 (paragraphs 0002-0008) [Patent Document 2] Japanese Patent Publication No. 2013-061919 (paragraphs 0002-0009, 0014-0015, Figure 1A) [Disclosure of the Invention] [Problems that the invention aims to solve]

[0006] In Patent Document 2, object detection is performed in a processing area set within the image. By narrowing the target area for object detection from the entire image to a relatively small processing area, it is possible to reduce the number of false detections that may occur during object detection. However, referring to Figure 1A of Patent Document 2, the nearby processing area 8 and the distant processing area 9 are far apart. Therefore, if the object to be detected moves between the vicinity and the distance of the in-vehicle camera, it is difficult to continuously track the moving object.

[0007] Furthermore, the background pattern in Patent Document 2 refers to patterns other than the object pattern to be detected (the rear pattern of the vehicle in Figure 1A) in the processing area for object detection, and some patterns are unclear, such as the background pattern 14 of the distant object 12. Therefore, even if both the object pattern and the background pattern of the object pattern are used, it is not guaranteed that the detection accuracy of objects in the distant region will improve.

[0008] Furthermore, when using on-board cameras to detect moving objects in front of the vehicle, situations may arise where objects in the camera's blind spot cannot be detected, making a collision unavoidable. In addition, there is the problem of the burden placed on the user when adding or maintaining on-board cameras to their own vehicle.

[0009] Therefore, in view of the above, the first object of the present invention is to provide a mobile object information processing device, program, or method that can continuously track a mobile object moving between the vicinity and distance of an imaging device while reducing the number of false detections that may occur during mobile object detection. The second object of the present invention is to improve the detection accuracy of mobile objects in the distance region in such a mobile object information processing device or the like.

[0010] Furthermore, a third object of the present invention is to provide a mobile information processing system that can prevent traffic accidents using such a mobile information processing device. A fourth object of the present invention is to realize such a mobile information processing system using any imaging device other than an in-vehicle camera with a limited range of capture. [Means for solving the problem]

[0011] To solve at least some of the above problems, a mobile object information processing device according to a first aspect of the present invention is a mobile object information processing device that obtains information about a mobile object based on an image signal obtained from at least one imaging device that captures a spatial region and generates an image signal, and comprises: an image processing unit that sets a first region of interest and a second region of interest having overlapping portions within the image of each frame of the image signal; a mobile object detection unit that detects a mobile object from the first and second regions of interest in the images of multiple frames of the image signal; a coordinate transformation unit that obtains coordinate information of a mobile object in a predetermined coordinate system based on the position information of the mobile object on the image detected from the first and second regions of interest; a mobile object tracking unit that determines a correspondence between at least one mobile object detected from the first region of interest and at least one mobile object detected from the second region of interest based on the coordinate information of the mobile object in the predetermined coordinate system; and a tracking result processing unit that obtains information about the position, movement path, movement direction, movement speed, or movement time of at least one mobile object spanning multiple regions of interest based on the coordinate information of the mobile object determined to be corresponding by the mobile object tracking unit in the predetermined coordinate system.

[0012] In a mobile object information processing device according to a second aspect of the present invention, the image processing unit sets a first region of interest corresponding to a first spatial region relatively close to the imaging device within the spatial region captured by the imaging device, and a second region of interest corresponding to a second spatial region relatively far from the imaging device, within the image of each frame of the image signal, and the mobile object detection unit resamples image data representing the image in at least the second region of interest in order to extract a predetermined number of pixels that are the target of the mobile object detection process, thereby reducing the rate of decrease in the number of pixels compared to resampling image data representing the entire image.

[0013] Furthermore, a mobile object information processing program according to the first aspect of the present invention is a mobile object information processing program used in a mobile object information processing device that obtains information about a mobile object based on an image signal obtained from at least one imaging device that captures a spatial region and generates an image signal, and causes the CPU to execute the following steps: (a) setting a first region of interest and a second region of interest having overlapping portions within the image of each frame of the image signal; (b) detecting a mobile object from the first and second regions of interest in the images of multiple frames of the image signal; (c) obtaining coordinate information of the mobile object in a predetermined coordinate system based on the image position information of the mobile object detected from the first and second regions of interest; (d) determining a correspondence between at least one mobile object detected from the first region of interest and at least one mobile object detected from the second region of interest based on the coordinate information of the mobile object in the predetermined coordinate system; and (e) obtaining information about the position, movement path, movement direction, movement speed, or movement time of at least one mobile object spanning multiple regions of interest based on the coordinate information of the mobile object determined to correspond in step (d).

[0014] Furthermore, a method for processing information about a moving object according to a first aspect of the present invention is a method for processing information about a moving object based on an image signal obtained from at least one imaging device that captures a spatial region and generates an image signal, comprising: (a) setting a first region of interest and a second region of interest having overlapping portions within the image of each frame of the image signal; (b) detecting a moving object from the first and second regions of interest in the images of multiple frames of the image signal; (c) determining coordinate information of the moving object in a predetermined coordinate system based on the image position information of the moving object detected from the first and second regions of interest; (d) determining a correspondence between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest based on the coordinate information of the moving object in the predetermined coordinate system; and (e) determining information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object spanning multiple regions of interest based on the coordinate information of the moving object determined to correspond in step (d) in the predetermined coordinate system.

[0015] A mobile information processing system according to a third aspect of the present invention may further include, in addition to a mobile information processing device according to any aspect of the present invention, at least one imaging device that captures a spatial area and generates an image signal, a mobile information receiving device mounted on a vehicle, a self-position detection device, a mobile information providing device, or a driving control device. Here, the mobile information processing device further includes a communication circuit that transmits information obtained by the tracking result processing unit to the vehicle, the mobile information receiving device receives information transmitted from the communication circuit of the mobile information processing device, the self-position detection device detects the position of the vehicle, and the mobile information providing device provides information about other mobile objects in the form of telegram data, images, or audio, based on the information received by the mobile information receiving device and the detection result of the self-position detection device. [Effects of the Invention]

[0016] According to a first aspect of the present invention, a first region of interest and a second region of interest having overlapping portions are set within the image of each frame of an image signal obtained from at least one imaging device, a moving object is detected from these regions of interest, the coordinate information of the moving object in a predetermined coordinate system is obtained based on the position information of the moving object on the image, and the correspondence between the moving object detected from the first region of interest and the moving object detected from the second region of interest is determined based on the coordinate information of the moving object, thereby enabling tracking of the moving object across multiple regions of interest. This narrows the target area for moving object detection from the entire image to relatively narrow regions of interest, reducing the number of false detections that may occur during moving object detection, while continuously tracking a moving object moving between the vicinity and distance of the imaging device.

[0017] Furthermore, according to a second aspect of the present invention, the image processing unit sets a first region of interest corresponding to the vicinity of the imaging device and a second region of interest corresponding to the far region of the imaging device within the image of each frame of the image signal, and the moving object detection unit resamples the image data representing the image in at least the second region of interest in order to extract a predetermined number of pixels that are the target of the moving object detection process. This reduces the rate of decrease in the number of pixels compared to resampling the image data representing the entire image. As a result, the resolution of the image portion corresponding to the far region can be improved, thereby improving the accuracy of moving object detection.

[0018] Furthermore, according to a third aspect of the present invention, the vehicle receives information transmitted from the communication circuit of a mobile information processing device according to any aspect of the present invention, detects its own position, and provides information about other mobile objects in the form of telegram data, images, or audio based on the received information and the detection result of its own position. Therefore, by using at least one imaging device placed on the road or elsewhere to acquire images of other mobile objects that are in blind spots that cannot be captured by the vehicle's onboard camera, and transmitting information about the position of the other mobile objects to the vehicle, it is possible to provide a mobile information processing system that can prevent traffic accidents. [Brief explanation of the drawing]

[0019] [Figure 1] It is a schematic diagram showing a first configuration example of a mobile body information processing system according to an embodiment of the present invention. [Figure 2] It is a schematic diagram showing a second configuration example of a mobile body information processing system according to an embodiment of the present invention. [Figure 3] It is a block diagram showing a configuration example of the mobile body information processing device shown in FIG. 1 or FIG. 2. [Figure 4] It is a flowchart showing a mobile body information processing method according to the first embodiment of the present invention. [Figure 5] It is a schematic diagram showing examples of a plurality of regions of interest set in a camera image obtained from an imaging device. [[ID=十九]] [Figure 6] It is a schematic diagram exemplarily showing changes in the position of a mobile body over time. [Figure 7] It is a schematic diagram for explaining the relationship between a camera image obtained from an imaging device and a map image in a predetermined coordinate system. [Figure 8] It is a flowchart showing a specific example of mobile body tracking processing. [Figure 9] It is a schematic diagram showing an image in which a mobile body specified on a map image is identified in a camera image. [Figure 10] It is a flowchart showing a mobile body information processing method according to the second embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The same reference numerals are assigned to the same components, and duplicate descriptions will be omitted. <Mobile body information processing system 1> FIG. 1 is a schematic diagram showing a first configuration example of a mobile body information processing system according to an embodiment of the present invention.

[0021] As shown in Figure 1, this system may include at least one imaging device 10 that captures a spatial region and generates an image signal (video signal), and a mobile object information processing device 20 that obtains information about a moving object based on the image signal obtained from the imaging device 10. In Figure 1, automobile MA and automobile MB are shown as examples of moving objects, but this system is capable of obtaining information such as the position or movement path of at least one moving object.

[0022] For example, this system can obtain information on the location or movement paths of an unspecified number of vehicles passing between the vicinity and distance of the imaging device 10, and monitor the number of vehicles passing through a road under investigation in a traffic volume survey. Alternatively, this system can be widely applied to factories, airports, and stores to obtain information on the location or movement paths of moving vehicles, forklifts, and autonomous robots.

[0023] In this system, the imaging device 10 captures a spatial area A0 and generates an image signal. For example, the imaging device 10 may be fixedly positioned on a predetermined road to detect the position or path of a moving object traveling on the road. An existing security camera may also be used as the imaging device 10.

[0024] The imaging device 10 includes a camera unit that captures an image of a subject with an image sensor via an optical system including at least one lens and generates a digital or analog image signal, and a communication unit that communicates wirelessly or via a wired connection with the mobile information processing device 20 and transmits the image signal generated by the camera unit to the mobile information processing device 20.

[0025] Alternatively, a recording device 10a that records the image signal generated by the camera unit may be used. The recording device 10a may be mounted inside the imaging device 10, or it may be located in a building or the like outside the imaging device 10, as shown in Figure 1. The image signal recorded in the recording device 10a is supplied to the mobile information processing device 20 by wireless or wired communication, or using a portable recording medium.

[0026] The recording medium in the recording device 10a, or the portable recording medium, can be a hard disk, flexible disk, optical disk, magneto-optical disk, magnetic tape, SSD (solid state drive), or various types of memory including USB memory.

[0027] Communication between the imaging device 10 and the mobile information processing device 20, communication between the imaging device 10 and the recording device 10a, or communication between the recording device 10a and the mobile information processing device 20 may be carried out via various networks such as wired LAN, wireless LAN, intranet, internet, or mobile communication network.

[0028] The image signal generated by the imaging device 10, or the image signal recorded in the recording device 10a, is supplied to the mobile information processing device 20 as measurement data, along with supplementary information including identification information of the imaging device or timing information related to the time of imaging. At least a portion of the supplementary information may be included in the image signal, or included in the file name of the image signal or measurement data.

[0029] The mobile object information processing device 20 obtains information such as the position, path, direction of movement, speed of movement, or time of movement of a mobile object based on the image signal obtained from the imaging device 10, which captures a spatial area and generates an image signal. For example, the mobile object information processing device 20 consists of a server installed in a data center or a control center of a research company or manufacturing company, or a PC (personal computer) and a display, etc.

[0030] <Mobile Information Processing System 2> Figure 2 is a schematic diagram showing a second configuration example of a mobile information processing system according to one embodiment of the present invention. In order to prevent traffic accidents, this system obtains information about mobile objects such as automobiles and pedestrians traveling on the road (for example, automobile MB shown in Figure 1) using a mobile information processing device 20, and provides this information to the driver of automobile MA.

[0031] The imaging device 10 is positioned and positioned to capture, for example, the area in front of the vehicle MA or a blind spot area where moving object information is provided. A blind spot area refers to an area that cannot be captured by the vehicle MA's onboard camera due to obstacles such as buildings or oncoming vehicles when turning right, or an area that is obstructed from the driver's view of the vehicle MA. If there are multiple blind spots, multiple imaging devices 10 may be positioned.

[0032] In the second configuration example shown in Figure 2, immediacy is required for the mobile information processing device 20. In order to provide mobile information in real time, it is desirable to directly connect the imaging device 10 and the mobile information processing device 20 to reduce transmission delay. For example, an edge computer installed near the road where the imaging device 10 is located can be used as the mobile information processing device 20.

[0033] When connecting the imaging device 10 and the mobile information processing device 20, etc., with a wired connection, methods such as HDMI (High-Definition Multimedia Interface; registered trademark) or SDI (Serial Digital Interface) may be used.

[0034] The automobile MA is equipped with, for example, a mobile information receiving device 31, a self-position detection device 32, a mobile information providing device 33, and a driving control device 34. The mobile information receiving device 31 receives information transmitted from the communication circuit 24 of the mobile information processing device 20 and obtains information such as the position, travel path, travel direction, travel speed, or travel time of at least one mobile object. Communication between the mobile information processing device 20 and the mobile information receiving device 31 may be carried out via various networks such as a wireless LAN, intranet, internet, or mobile communication network.

[0035] The self-position detection device 32 detects the position of the vehicle MA. For example, the self-position detection device 32 includes a receiver compliant with a positioning method such as GNSS (Global Navigation Satellite System), RTK (Real Time Kinematic), CLAS (Centimeter Level Augmentation Service), or SLAS (Submeter Level Augmentation Service), and calculates the current position of the vehicle MA based on the radio waves received by the receiver.

[0036] Alternatively, the self-position detection device 32 may detect the position of the automobile MA using SLAM (Simultaneous Localization and Mapping) technology, which simultaneously determines its own position and grasps the structure of the surrounding environment. SLAM technology mainly includes LiDAR SLAM, which uses a laser sensor called LiDAR (Light Detection and Ranging) as the sensor, and Visual SLAM, which uses a camera as the sensor.

[0037] Furthermore, in order to make it easier to detect the location of the vehicle MA, AR (Augmented Reality) markers, AR tags, QR codes (registered trademarks), or checkerboards may be placed at key points along the road on which the vehicle MA travels.

[0038] The mobile information provider 33 provides information about other mobile objects traveling on the road in the form of telegram data, images, or audio, based on the information received by the mobile information receiver 31 and the detection results of the self-position detection device 32. Information about other mobile objects includes information indicating the presence of other mobile objects near the automobile MA, or information indicating the possibility of collision with other mobile objects.

[0039] To this end, the mobile information providing device 33 supplies image signals to a display unit (such as a screen) and audio signals to an audio output unit (such as a speaker). The display unit and audio output unit may be configured as part of the mobile information providing device 33, or they may be those already installed in the automobile MA.

[0040] The driving control device 34 uses the message data provided by the mobile information provider 33 to determine that there is a high probability that the vehicle MA will collide with another mobile object, and performs control operations related to the driving of the vehicle MA. In this case, the driving control device 34 may also control the accelerator / brake system 35 and the steering system 36 of the vehicle MA.

[0041] <Mobile Information Processing Device> Figure 3 is a block diagram showing an example configuration of the mobile information processing device shown in Figure 1 or Figure 2. As shown in Figure 3, the mobile information processing device 20 includes an operation unit 21, a display unit 22, an audio input / output unit 23, a communication circuit 24, an interface 25, a CPU (Central Processing Unit) 26, a cache memory 27, and a storage unit 28. The interface 25 to the storage unit 28 are connected to each other via a bus line. Note that some of the components shown in Figures 1 to 3 may be omitted or changed, or other components may be added to the components shown in Figures 1 to 3.

[0042] The operation unit 21 consists of, for example, a keyboard and mouse, and is used to input various commands and data to the mobile information processing device 20. The display unit 22 includes, for example, an LCD (liquid crystal display), and displays the operation screen, etc. The audio input / output unit 23 includes, for example, a microphone, amplifier, and speaker, and converts audio signals into electrical signals and electrical signals into audio signals.

[0043] In the second configuration example shown in Figure 2, when an edge computer is used as the mobile information processing device 20, an HMI (Human Machine Interface) or GUI (Graphical User Interface) may be used instead of the display unit 22 or the audio input / output unit 23.

[0044] The communication circuit 24 has the function of performing data communication with the imaging device 10, the recording device 10a, or the mobile information receiving device 31. The interface 25 is connected to the operation unit 21 to the communication circuit 24 and to external devices such as portable recording media, and transmits various commands and data between them and the CPU 26. In addition, data input to the CPU 26 via the interface 25 is written to the cache memory 27 or the storage unit 28.

[0045] The CPU 26 performs various calculations and data processing according to the various software (including mobile information processing programs) stored in the storage unit 28. The cache memory 27 and storage unit 28 store various data used to obtain information about mobile objects and tracking results information of mobile objects in multiple databases (DBs).

[0046] The cache memory 27 is composed of RAM (Random Access Memory) or the like. Various types of memory, including hard disks, flexible disks, optical disks, magneto-optical disks, magnetic tapes, SSDs, or ROM (Read-Only Memory), can be used as recording media (storage media) in the storage unit 28.

[0047] Here, the CPU 26 and the mobile object information processing program stored in the storage unit 28 configure the following as functional blocks: an image signal acquisition unit 260, an image processing unit 261, a mobile object detection unit 262, a mobile object position estimation unit 263, a coordinate transformation unit 264, a mobile object tracking unit 265, and a tracking result processing unit 266.

[0048] The image signal acquisition unit 260 acquires measurement data, including image signals and associated information, from the imaging device 10. For example, the image signal acquisition unit 260 may control the communication circuit 24 to receive measurement data from the imaging device 10, or to receive measurement data from the recording device 10a that records the image signals generated by the imaging device 10.

[0049] Alternatively, the image signal acquisition unit 260 may acquire measurement data from a portable recording medium on which measurement data is recorded when the recording medium is connected to the interface 25. For example, the image signal acquisition unit 260 stores the measurement data acquired from the imaging device 10 in the image database of the cache memory 27, and stores the measurement data acquired from the recording device 10a or the portable recording medium in the image database of the storage unit 28.

[0050] The moving object detection unit 262 to the tracking result processing unit 266 obtain information such as the position, path, direction, speed, or time of movement of at least one moving object based on the image signals included in the measurement data. For example, the position or path of the moving object on a predetermined map is obtained and displayed on the display unit 22.

[0051] <Example of operation 1> Next, a first example of operation of the mobile information processing device shown in Figure 3 will be described with reference to Figures 1 to 9. Figure 4 is a flowchart of the mobile information processing method according to the first embodiment of the present invention. Note that some of the processes shown in Figure 4 may be omitted or modified, or other processes may be added to the processes shown in Figure 4.

[0052] <Step S10> In step S10 of Figure 4, the image signal acquisition unit 260 acquires measurement data, including image signals and associated information, from the imaging device 10, recording device 10a, or portable recording medium shown in Figure 1. The image signal acquisition unit 260 stores the acquired measurement data in the cache memory 27 or the image database of the storage unit 28. As a result, multiple frames of image signals obtained from the imaging device 10 are stored in the image database.

[0053] <Step S11> In step S11, the image processing unit 261 reads the image signal collected by the image signal acquisition unit 260 in step S10 from the image database. The image processing unit 261 sets up multiple regions of interest within each frame of the image (camera image) of the image signal obtained from the imaging device 10, with overlapping portions, according to the operation of the operator using the operation unit 21, or according to region information stored in the storage unit 28 in advance, corresponding to the position and orientation of the imaging device 10.

[0054] For example, the image processing unit 261 extracts image data of the set region of interest from each frame of the image signal and supplies the extracted image data to the moving object detection unit 262. If two regions of interest are set within one frame of the camera image, image data of two regions of interest will be extracted from the image signal of one frame.

[0055] Figure 5 is a schematic diagram showing an example of multiple regions of interest set within a camera image obtained from an imaging device. Figure 5 shows camera images of frame FA and frame FB obtained by imaging device 10 of an automobile MB traveling on a road from a distance to the vicinity of the imaging device 10. The origin O(0,0) on the camera image is located in the upper left corner of the image.

[0056] For example, the image processing unit 261 sets a first region of interest A1 and a second region of interest A2 within the image (camera image) of each frame of the image signal obtained from the imaging device 10, with overlapping portions. Note that using rectangular regions of interest makes it easier to set the regions.

[0057] The first region of interest A1 corresponds to the first spatial region (neighborhood region) that is relatively close to the imaging device 10 within the spatial region A0 (Figure 1) captured by the imaging device 10. On the other hand, the second region of interest A2 corresponds to the second spatial region (far region) that is relatively far from the imaging device 10, and may have a smaller size (fewer pixels) than the first region of interest A1.

[0058] Here, if the imaging device 10 is positioned at a predetermined height and angled downwards, and the angle is set so that the imaging device 10 can capture images from the near-field to the far-field area of ​​the road surface, then in the camera image, the first region of interest A1, which corresponds to the near-field area, will be located relatively low, and the second region of interest A2, which corresponds to the far-field area, will be located relatively high. In other words, in the camera image, the center of gravity of the second region of interest A2 is located higher than the center of gravity of the first region of interest A1.

[0059] Frame FA, shown in Figure 5, was obtained by imaging the automobile MB with the imaging device 10 at a first time point, and the automobile MB is located within the second region of interest A2. Frame FB was obtained by imaging the automobile MB with the imaging device 10 at a second time point, after a predetermined time (a predetermined number of frames) has elapsed from the first time point, and the automobile MB is located within the first region of interest A1.

[0060] <Step S12> In step S12, the moving object detection unit 262 performs moving object detection processing on the image signals representing each region of interest set by the image processing unit 261 in step S11. For example, the moving object detection unit 262 performs moving object detection processing on the image data of each region of interest supplied by the image processing unit 261. As a result, the moving object detection unit 262 detects moving objects from multiple regions of interest in multiple frames of the image signal. Consequently, the position information of the moving object on the image is obtained for each frame of the camera image.

[0061] Generally, when imaging both the near-field and far-field areas of space with a single imaging device, objects appear smaller, especially in the image portion corresponding to the far-field area. Compared to using multiple imaging devices arranged along a road, this can lead to a loss of detail, reduced image resolution, and a decrease in the accuracy of detecting moving objects.

[0062] Furthermore, in AI (Artificial Intelligence) motion detection processing, for example, the size of the input pixel matrix (number of pixels) is fixed, so it is necessary to resample the input image before performing motion detection processing. In this case, downsampling is often performed, and since downsampling is performed uniformly on the input image, information about objects may be lost due to the reduction in the number of pixels, especially in images of distant regions composed of fewer pixels.

[0063] Therefore, as shown in Figure 5, by setting a second region of interest A2 that captures the image portion corresponding to a distant spatial region, the moving object detection unit 262 can reduce the rate of pixel reduction compared to resampling the image data representing at least the second region of interest A2 in order to extract a predetermined number of pixels that are the target of the moving object detection process. As a result, the resolution of the image portion corresponding to the distant region can be improved, and the accuracy of moving object detection can be improved. A similar effect can also be obtained by resampling the image data representing the image in the first region of interest A1 that captures the image portion corresponding to a nearby spatial region.

[0064] In particular, when the second region of interest A2 has a smaller size (fewer pixels) than the first region of interest A1, the moving object detection unit 262 can reduce the rate of decrease in the number of pixels in the second region of interest A2 to a lower rate than the rate of decrease in the number of pixels in the first region of interest A1 by resampling the image data representing the image in the first region of interest A1 and the image data representing the image in the second region of interest A2 in order to extract a predetermined number of pixels that are the target of the moving object detection process.

[0065] Alternatively, the moving object detection unit 262 may upsample (oversample) the image data representing the image in at least the second region of interest A2 in order to extract a predetermined number of pixels that are the target of the moving object detection process, thereby increasing the number of pixels in the second region of interest A2. This improves the resolution of the image in the second region of interest A2 that captures the image portion corresponding to the distant region, and thus improves the accuracy of moving object detection.

[0066] Here, let (M0, N0) be the total number of pixels in the image (horizontal and vertical directions), (M1, N1) be the number of pixels in the first region of interest A1 (horizontal and vertical directions), and (M2, N2) be the number of pixels in the second region of interest A2 (horizontal and vertical directions). Also, let (Mr0, Nr0), (Mr1, Nr1), and (Mr2, Nr2) be the number of pixels after resampling, respectively.

[0067] The percentage decrease in the number of pixels P0 when resampling image data representing the entire image is given by P0 = (M0 × N0 - Mr0 × Nr0) / (M0 × N0). Similarly, the percentage decrease in the number of pixels P1 when resampling image data representing the image in the first region of interest A1 is given by P1 = (M1 × N1 - Mr1 × Nr1) / (M1 × N1), and the percentage decrease in the number of pixels P2 when resampling image data representing the image in the second region of interest A2 is given by P2 = (M2 × N2 - Mr2 × Nr2) / (M2 × N2).

[0068] In that case, it is desirable to resample the image data representing the image in the first region of interest A1 and the second region of interest A2, rather than resampling the image data representing the entire image, so that P0 > P1 > P2. Note that when the number of pixels is increased by upsampling, P1 < 0 or P2 < 0 will occur in the above equation. In this application, "suppressing the rate of decrease in the number of pixels" means making the value of the rate of decrease smaller, and when the number of pixels is increased, the rate of decrease will take a negative value.

[0069] The first method of image resampling in this embodiment is to interpolate new pixels between the original pixels of the input image when upsampling. For example, the value of a new pixel may be the average of the values ​​of K (where K is an integer of 2 or more) surrounding original pixels. Even if the resolution of the original camera image is limited, by applying appropriate interpolation processing, the resolution of the image in the region of interest can be improved, thereby improving the accuracy of detecting moving objects.

[0070] The second method involves resampling the input image to a ratio different from its aspect ratio. That is, the horizontal and vertical directions of the input image are resampled at different ratios. This allows for motion detection processing to be applied to images with a nearly constant number of pixels (horizontal pixels × vertical pixels), even when a rectangle with an arbitrary aspect ratio is set as the region of interest. Furthermore, the third method involves copying nearby pixel values ​​in uniform regions of the input image and resampling only the complex regions. This reduces computation time.

[0071] In this embodiment, the target area for moving object detection can be narrowed down from the entire camera image to a relatively narrow region of interest, and the moving object detection process can be performed on the region of interest. This reduces the number of false detections that may occur during moving object detection, while enabling continuous tracking of moving objects moving between the vicinity and distance of the imaging device 10. This moving object detection process can utilize AI-based object detection processing, image recognition processing, and the like.

[0072] AI-based object detection processing, for example, divides images in each region of interest into numerous small segments, labels each segment indicating what is depicted, and associates it with a category. If objects with the same label or category are detected in two consecutive frames, but at different positions within a predetermined distance, it is recognized that the same object has moved. This makes it possible to detect moving objects whose positions change across multiple frames of an image signal.

[0073] Alternatively, the moving object detection unit 262 may detect a moving object from the regions of interest of multiple frames of images by applying predetermined image recognition processing to the image signals of multiple frames representing each region of interest. In this image recognition processing, for example, an active appearance model can be used, which separates the image of the target object into shape and texture, and reduces the dimensionality of each by principal component analysis, thereby enabling the representation of changes in the shape and texture of the target object with fewer parameters.

[0074] In an active appearance model, the shape vector, which represents all feature points, is expressed using the mean shape vector obtained from the training data and the eigenvector matrix obtained by principal component analysis of the deviations from the mean shape vector. In other words, the shape vector is expressed as the mean shape vector plus the product of the eigenvector matrix and the shape parameters.

[0075] Similarly, the appearance vector, which is a sequence of normalized texture luminance values, is represented using the average appearance vector obtained from the training data and the eigenvector matrix obtained by principal component analysis of the deviations from the average appearance vector. In other words, the appearance vector is expressed as the average appearance vector plus the product of the eigenvector matrix and the appearance parameter.

[0076] Shape parameters and appearance parameters are parameters that represent the change from the mean, and by changing them, the shape and appearance can be changed. Furthermore, since there is a correlation between shape and appearance, by further performing principal component analysis on the shape parameters and appearance parameters, it is possible to represent the shape vector and appearance vector using low-dimensional parameter vectors (local parameters) that control both shape and appearance.

[0077] Next, parameters related to the global changes in where, what size, and in what orientation the target object exists within the image (movement parameters) are considered. In the active appearance model, model exploration involves changing the above parameters to locally and globally change the model, generating an image of the target object, comparing the generated image with the input image, and finding the parameter values ​​that minimize the error.

[0078] If the error falls below the threshold, it may be determined that a group of feature points matching the model being searched exists in the input image, and the error minimization process may be terminated. On the other hand, if the error does not fall below the threshold, it may be determined that a group of feature points matching the model being searched does not exist in the input image.

[0079] In this embodiment, since the target object is limited to a moving object, the difference image signal between multiple frames is obtained, the target object is estimated based on the difference image signal, and the data of its feature points is used together with the training data, or used in place of the training data, to obtain the average shape vector and average appearance vector.

[0080] When the moving object detection unit 262 detects a target object in the region of interest of an image in one frame, it changes the above-mentioned parameters to locally and globally modify the model of the target object and generate images of the target object in other frames. It then compares the generated images with the images in other frames, and if the error is minimized or falls below a threshold, it extracts multiple feature points corresponding to the image of the target object in the images of other frames. This makes it possible to detect moving objects from regions of interest of images in multiple frames.

[0081] Once the object detection process is complete, the object detection unit 262 may assign a bounding box to each detected object. A bounding box is a sub-region surrounding an object in an image and is generally used to determine the position of a particular object within an image and to classify that object.

[0082] Classification is a method of classifying objects into different types. For example, an image showing a moving object, such as a car, would be classified as a "passenger car," "truck," or "bus." It is also possible to calculate the probability that the moving object is a "passenger car," "truck," or "bus."

[0083] Bounding boxes play a crucial role in a wide range of fields, such as autonomous driving systems and security camera systems, because they allow us to pinpoint the location of specific objects within an image. Furthermore, using rectangular bounding boxes makes image annotation a simpler process than segmentation (labeling at the pixel level).

[0084] When several moving objects are detected by applying the moving object detection process described above to multiple frames of image signals representing the region of interest, the moving object detection unit 262 associates the positional information (which may include bounding boxes) of the detected moving objects on the image with the associated information and stores it as moving object detection result information in the moving object detection result database of the cache memory 27.

[0085] <Step S13> Step S13 is performed using the moving object detection result information obtained in step S12 by applying moving object detection processing to the image signal.

[0086] In step S13, the moving object position estimation unit 263 determines the estimated position of the first moving object at the time of the other frames based on the position of the first moving object detected from the images of a predetermined number of frames (for example, two or more frames) out of multiple frames of the image signal for each region of interest. If the error between the position of the second moving object detected from the images of the other frames and the estimated position is within a predetermined range, the unit determines that the first moving object and the second moving object are the same and saves the position information of the second moving object detected from the images of the other frames as the position information of the first moving object on the image.

[0087] As an example, first, the mobile object position estimation unit 263 reads mobile object detection result information from the mobile object detection result database in the cache memory 27. If a mobile object is detected in consecutive first and second frames of multiple frames of an image signal, shifting its position within a predetermined distance, the unit determines that they are the same mobile object (first mobile object). The mobile object position estimation unit 263 assigns a mobile object ID (identification information) to the first mobile object, associates the mobile object ID and position information of the first mobile object with the associated information, and stores it as mobile object tracking information in the mobile object tracking information database in the cache memory 27.

[0088] Next, the mobile object position estimation unit 263 reads the mobile object tracking information for the first mobile object from the mobile object tracking information database and, based on the position of the first mobile object detected from the images of the first and second frames, determines the estimated position of the first mobile object at the time of the third frame following the second frame. The mobile object position estimation unit 263 stores the information of the estimated position of the first mobile object at the time of the third frame in the mobile object tracking information database.

[0089] Next, a mobile object position comparison process is performed. The mobile object position estimation unit 263 sequentially reads mobile object detection result information from the mobile object detection result database, and if the error between the position of the second mobile object detected from the image of the third frame and the estimated position is within a predetermined range, it determines that the first mobile object and the second mobile object are the same, and saves the position information of the second mobile object detected from the image of the third frame as the position information of the first mobile object in the mobile object tracking information database.

[0090] As a result, the information of the estimated position of the first mobile object at the time of the third frame is overwritten with the position information of the first mobile object, and the mobile object tracking information database is updated. Furthermore, the mobile object position estimation unit 263 deletes the used mobile object detection result information from the mobile object detection result database.

[0091] On the other hand, if no moving object is detected in the third frame image whose positional error with the estimated position is within a predetermined range, the moving object position estimation unit 263 does not update the moving object tracking information database, and therefore the estimated position information is used as the positional information of the first moving object on the image at the time of the third frame.

[0092] Furthermore, if the error between the position of the second moving object detected from the image of the third frame and the estimated position exceeds a predetermined range, the moving object position estimation unit 263 determines that the first moving object and the second moving object are different. That is, the moving object position estimation unit 263 determines that a second moving object different from the first moving object has been discovered, assigns a new moving object ID to the second moving object, associates the moving object ID and position information of the second moving object with the associated information, and stores it in the moving object tracking information database as moving object tracking information.

[0093] The above moving object position comparison process is repeated for each moving object detection result obtained by the moving object detection process on the image signal obtained from the imaging device 10. Furthermore, if m moving objects are detected from the first group of frames of the image signal obtained from the imaging device 10 (where m is a natural number), the moving object position estimation process in step S13 is repeated for each of the m moving objects.

[0094] If n new moving objects are discovered during this process (where n is a natural number), the moving object position estimation process in step S13 is repeated for each of the n additional moving objects. Furthermore, the processes in steps S12 to S13 are repeated a number of times equal to the number of regions of interest set by the image processing unit 261.

[0095] Alternatively, the mobile object position estimation unit 263 may assign the same mobile object ID to the first mobile object and the second mobile object if the error between the position of the second mobile object detected from the image of the third frame and the estimated position is within a predetermined range. If the first mobile object has already been assigned a mobile object ID, the mobile object position estimation unit 263 may assign the same mobile object ID to the second mobile object as to the first mobile object. In this way, the first mobile object and the second mobile object, which have been determined to be the same, are linked to each other.

[0096] The above estimation can utilize methods such as the Kalman filter for predicting object positions, or extrapolation or interpolation, which are statistical techniques for estimating data in unknown ranges.

[0097] A Kalman filter is a type of infinite impulse response filter used to estimate or control the state of a dynamic system using error-laden observations. It is used to estimate constantly changing quantities (for example, the position and velocity of a moving object) from discrete, error-laden observations.

[0098] For example, the Kalman filter calculates a prior estimate of the object's position at time (t+1) by combining the prior estimate at time (t+1), which is calculated from the posterior estimate at time t, with the observed value at time (t+1). The prior estimate at time (t+1) can be obtained by adding the object's movement vector to the posterior estimate at time t.

[0099] Here, the motion vector of the moving object can be determined as the difference vector between the position of the moving object at time t and the position of the moving object at time (t-1). Furthermore, the position of the moving object determined by the moving object detection unit 262 can be used as the observed value. Note that the observed values ​​at each time point can be used as the two prior estimates immediately after the start of object position prediction.

[0100] A weighted average, for example, is used to combine the pre-estimated values ​​and observed values. In this combination, the weight given to each of the pre-estimated and observed values ​​is controlled by a parameter called the Kalman gain. Since it is not always possible to detect the same moving object in one frame as in the previous frame, the Kalman gain may be varied from frame to frame.

[0101] Figure 6 is a schematic diagram illustrating the change in the position of a moving object over time. While the position of the automobile MB shown in Figure 1, etc., is best displayed on a two-dimensional map, here, for the sake of simplicity, the position of the automobile MB is shown in one-dimensional coordinates. In Figure 6, the vertical axis represents the one-dimensional coordinate x, and the horizontal axis represents time t. In addition, frame numbers F1, F2, F3, ... are shown along the time axis as timing information related to the time of imaging.

[0102] Figure 6 shows the coordinates of the automobile MB detected from the second region of interest A2 and the first region of interest A1 in the image obtained by the imaging device 10 shown in Figure 1 capturing spatial region A0 (Figure 1) over multiple frames, indicated by circles and squares, respectively. The lines connecting them represent the movement path (trajectory) of the automobile MB.

[0103] However, from the perspective of the imaging device 10, the car MB may be hidden behind an obstacle or change direction, so the car MB is not detected in the second region of interest A2 of the image in frame F3 and in the first region of interest A1 of the image in frame F9 (*).

[0104] In such cases, the moving object position estimation unit 263 may, with respect to each region of interest, make the weighting coefficient of the observed value equal to or greater than the weighting coefficient of the prior estimate for frames in which the moving object detection unit 262 was able to detect the automobile MB, and make the weighting coefficient of the observed value smaller than the weighting coefficient of the prior estimate (for example, to zero) for frames in which the moving object detection unit 262 was unable to detect the automobile MB.

[0105] In this way, by changing the weighting of the prior estimate and the observed value for each frame, it is possible to obtain a post-mortem estimate of the position of the car MB at the time of frame F3, even if the car MB is not detected in the second region of interest A2 of the image in frame F3. Furthermore, even if the car MB is not detected in the first region of interest A1 of the image in frame F9, it is possible to obtain a post-mortem estimate of the position of the car MB at the time of frame F9.

[0106] In addition to the object position prediction method using the Kalman filter described above, the moving object position estimation unit 263 may also extrapolate the position of the same moving object at the time of imaging of one frame based on the positions of the same moving object detected from the same region of interest in images of multiple frames preceding or following a frame in which the same moving object was not detected by the moving object detection unit 262.

[0107] Furthermore, the moving object position estimation unit 263 may also determine the position of the same moving object at the time of imaging of the one frame by interpolation, based on the position of the same moving object detected from the same region of interest in the images of multiple frames that surround a frame in which the same moving object was not detected by the moving object detection unit 262.

[0108] <Step S14> In step S14, the coordinate transformation unit 264 determines the coordinate information of the moving object in a predetermined coordinate system based on the position information of the moving object on the image detected from multiple regions of interest in the image represented by the image signal (in the first embodiment, the position information obtained by the moving object position estimation unit 263 in step S13).

[0109] The specified coordinate system is, for example, a one- or two-dimensional coordinate system that is substantially orthogonal to the vertical direction, a three-dimensional coordinate system with two axes substantially orthogonal to the vertical direction, a one- or two-dimensional coordinate system substantially parallel to the ground surface at the location where any imaging device is placed, or a three-dimensional coordinate system with two axes substantially parallel to that ground surface. Alternatively, an absolute coordinate system obtained using a positioning method such as GNSS, RTK, CLAS, or SLAS may be used.

[0110] If the coordinates of three or more points defining a predetermined surface region including a moving object in an image represented by an image signal are known, the position of the moving object in the image can be transformed into the coordinates of the moving object in a predetermined coordinate system using a transformation matrix such as a perspective projection transformation matrix. The coordinate transformation definitions used for coordinate transformation processing by the coordinate transformation unit 264 are stored in the coordinate transformation information database of the storage unit 28.

[0111] The above coordinate transformation process is repeated for each mobile object tracking information stored in the mobile object tracking information database of the cache memory 27. The coordinate transformation unit 264 adds the coordinate information obtained by the coordinate transformation process to the mobile object tracking information and stores it in the mobile object tracking information database.

[0112] Figure 7 is a schematic diagram illustrating the relationship between camera images obtained from an imaging device and map images in a predetermined coordinate system. In Figure 7, (a) shows the camera image of frame FA, and (b) shows the camera image of frame FB. (c) shows the map image onto which the camera images of frames FA and FB are projected.

[0113] For example, the camera image has 980 x 490 pixels, with the origin O(0,0) located in the upper left corner of the image. The map image has 420 x 1390 pixels. In the camera image, multiple reference points are set on the road surface or on a reference plane within a predetermined distance from the road surface.

[0114] The positions of the reference points can be determined arbitrarily, but in the example shown in Figure 7, eight reference points A to H are set so as to overlap with the positions of the four vertices of the first region of interest A1 and the four vertices of the second region of interest A2 in the camera image. A coordinate transformation definition is calculated and stored in the coordinate transformation information database in order to project these reference points onto the eight reference points A' to H' on the map image.

[0115] Alternatively, to improve the accuracy of coordinate transformations, a coordinate transformation definition may be calculated for each region of interest. In that case, for example, for the first region of interest A1, a first coordinate transformation definition is calculated to project four reference points A to D on the camera image onto four reference points A' to D' on the map image, and for the second region of interest A2, a second coordinate transformation definition is calculated to project four reference points E to H on the camera image onto four reference points E' to H' on the map image.

[0116] The coordinate transformation unit 264 reads a coordinate transformation definition from the coordinate transformation information database. The coordinate transformation unit 264 also sequentially reads mobile object tracking information from the mobile object tracking information database and uses the coordinate transformation definition identified by the mobile object tracking information to calculate the coordinates of the mobile object on the map image based on the position of the mobile object on the camera image.

[0117] As a result, the position P1(581,80) of a vehicle traveling on the road in the camera image of frame FA is converted to coordinates P1'(256,269) on the map image. Similarly, the position P2(544,329) of a vehicle traveling on the road in the camera image of frame FB is converted to coordinates P2'(231,897) on the map image.

[0118] <Step S15> In step S15, the moving object tracking unit 265 performs moving object tracking (moving object tracking process) by determining the correspondence between moving objects detected from multiple regions of interest based on the coordinate information of the moving objects in a predetermined coordinate system. This makes it possible to easily track the same moving object on the map image even when many moving objects are captured in one region of interest and it is difficult to track the same moving object across multiple regions of interest.

[0119] For example, the moving object tracking unit 265 determines that two moving objects correspond to each other if the distance (error) between the coordinates of two moving objects obtained from two regions of interest in the same frame of the image signal is within a predetermined range.

[0120] Alternatively, if the coordinates of multiple moving objects are obtained from one region of interest in the same frame of the image signal, the moving object tracking unit 265 may determine that the moving object whose coordinates have the smallest distance (error) between it and the coordinates of the moving object obtained from the other region of interest, within a predetermined range, corresponds to the moving object whose coordinates were obtained from the other region of interest.

[0121] Furthermore, if no moving object is detected in the same frame of the image signal from two regions of interest within a predetermined distance, the moving object tracking unit 265 may use information on estimated coordinates (coordinates at the time of the same frame) obtained based on the position or coordinates of the moving object detected in at least one of the regions of interest in images of a predetermined number of other frames (e.g., two or more frames) of the image signal to determine the correspondence between the moving objects.

[0122] In the example shown in Figure 6, the moving object tracking unit 265 determines the correspondence between at least one moving object detected from the first region of interest A1 and at least one moving object detected from the second region of interest A2, based on the coordinate information of the moving object in a predetermined coordinate system.

[0123] For example, in the image of frame F6, if the distance between the coordinates of a moving object obtained from the first region of interest A1 (marked with a square) and the coordinates of a moving object obtained from the second region of interest A2 (marked with a circle) is within a predetermined range, then it is determined that these moving objects correspond to each other. The coordinates of the moving objects in frame F6 may be the coordinates obtained from the first region of interest A1 (marked with a square), the coordinates obtained from the second region of interest A2 (marked with a circle), or their average value, etc.

[0124] On the other hand, in the image of frame 6, if no moving object with coordinates within a predetermined range is detected in the second region of interest A2, for example, the coordinates of the moving object in frame F6 can be estimated based on the position or coordinates of the moving object detected in the second region of interest A2 in the images of frames F4 and F5.

[0125] In such cases, if the distance between the coordinates of the moving object obtained from the first region of interest A1 (marked with a square) and the estimated coordinates obtained from the second spatial region A2 is within a predetermined range, then it is determined that the moving objects correspond to each other. The coordinates of the moving object in frame F6 may be those obtained from the first region of interest A1 (marked with a square).

[0126] Figure 8 is a flowchart illustrating a specific example of the mobile object tracking process. As a premise, the mobile object tracking information database in the cache memory 27 stores N mobile object tracking information (where N is an integer greater than or equal to 2), and the mobile object tracking unit 265 references the N mobile object tracking information using variable (i) or variable (j) (i or j = 1, 2, ..., N).

[0127] In step S151, the mobile object tracking unit 265 starts a loop of variable (i) by setting i=0. In step S152, the mobile object tracking unit 265 increments variable (i) by "1" and reads the mobile object tracking information M(i) from the mobile object tracking information database in the cache memory 27.

[0128] In step S153, the mobile object tracking unit 265 starts a loop of the variable (j) by setting j=0. In step S154, the mobile object tracking unit 265 increments the variable (j) by "1" and reads the mobile object tracking information M(j) from the mobile object tracking information database in the cache memory 27.

[0129] In step S155, the mobile object tracking unit 265 determines whether the mobile object tracking information M(i) and the mobile object tracking information M(j) relate to different regions of interest in the same frame of the image. If the mobile object tracking information M(i) and the mobile object tracking information M(j) relate to different regions of interest in the same frame of the image (YES), the process proceeds to step S156; otherwise (NO), the process proceeds to step S158.

[0130] In step S156, the mobile object tracking unit 265 determines whether or not map coordinates (coordinate information in a predetermined coordinate system) exist in the mobile object tracking information M(i) and mobile object tracking information M(j). Here, the existence of map coordinates means that the map coordinates stored in the mobile object tracking information database of the cache memory 27 are not empty (for example, the coordinate data has not been erased).

[0131] If map coordinates exist in the mobile object tracking information M(i) and mobile object tracking information M(j), the mobile object tracking unit 265 determines whether the distance between the two map coordinates is within a predetermined range. This determines the correspondence between the mobile object identified by the mobile object tracking information M(i) and the mobile object identified by the mobile object tracking information M(j).

[0132] If it is determined in step S156 that the distance between the map coordinates of the moving object tracking information M(i) and the moving object tracking information M(j) is within a predetermined range (YES), the process proceeds to step S157; otherwise (NO), the process proceeds to step S158.

[0133] In step S157, the mobile object tracking unit 265 determines that the mobile object identified by the mobile object tracking information M(i) corresponds to (is the same as) the mobile object identified by the mobile object tracking information M(j), and adds at least a portion of the map coordinates of the mobile object tracking information M(j) as part of the map coordinates of the mobile object tracking information M(i). Subsequently, the mobile object tracking unit 265 clears the map coordinates of the used mobile object tracking information M(j) and saves it (for example, by deleting the coordinate data of the mobile object tracking information M(j)).

[0134] In step S158, the mobile object tracking unit 265 determines whether the loop for variable (j) has finished. If the loop for variable (j) has not finished, the process returns to step S154. If the loop for variable (j) has finished, the process proceeds to step S159.

[0135] In step S159, the mobile object tracking unit 265 determines whether the loop for variable (i) has finished. If the loop for variable (i) has not finished, the process returns to step S152. Once the loop for variable (i) has finished, the mobile object tracking process ends.

[0136] <Step S16> In step S16, the tracking result processing unit 266 obtains information such as the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple regions of interest, based on the coordinate information in a predetermined coordinate system of the moving object that the moving object tracking unit 265 determined to be corresponding to in step S15.

[0137] For example, the tracking result processing unit 266 sequentially reads mobile tracking information from the mobile tracking information database in the cache memory 27, selects mobile tracking information that includes non-empty map coordinates, and saves it to the tracking result information database in the storage unit 28. Furthermore, the tracking result processing unit 266 may use this to store the tracking result information obtained by the following tracking result processing, associated with supplementary information including the mobile ID, in the tracking result information database.

[0138] The tracking result processing unit 266 may read map data representing a two-dimensional or three-dimensional map from the map database in the storage unit 28, generate an image signal to display the position of at least one moving object on the map, and display a map image on the display unit 22 that identifies the position, path, direction, speed, or time of movement of at least one moving object. Alternatively, the tracking result processing unit 266 may control the communication circuit 24 to transmit information about the detected moving object to an external device via a network.

[0139] Referring to Figure 6, the tracking result processing unit 266 obtains information regarding the position, movement path, or direction of movement of the vehicle MB based on the coordinate information of the vehicle MB detected in the first region of interest A1 and the second region of interest A2. The tracking result processing unit 266 can also obtain information regarding the vehicle MB's movement speed or movement time, or average movement speed or average movement time, based on the difference in the coordinate information of the vehicle MB between two points.

[0140] Furthermore, the tracking result processing unit 266 may measure the movement routes of multiple moving objects, such as an unspecified number of automobiles, and based on that, count the number of automobiles passing through a specific area. This makes it possible to monitor the number of automobiles passing through the roads being surveyed in traffic volume surveys.

[0141] Furthermore, the tracking result processing unit 266 may calculate the inverse matrix of the transformation matrix used by the coordinate transformation unit 264 to obtain the inverse coordinate transformation definition, and when a moving object is specified on the map image, it may use the inverse coordinate transformation definition to display an image on the display unit 22 that represents the state in which the specified moving object has been identified in the corresponding region of interest.

[0142] In other words, the tracking result processing unit 266 may reverse-transform the coordinates of the moving object specified on the map image to a position on the camera image, select a frame in which the reverse-transformed position of the moving object is located within the image of the moving object in the region of interest, generate an image signal representing an image that includes a marker identifying the moving object at the corresponding position in the camera image of the selected frame, and display that image on the display unit 22.

[0143] For example, the tracking result processing unit 266 displays an image on the display unit 22 that includes the mobile object ID, the type of mobile object, or the location information in the camera image, corresponding to the identified mobile object in the image. In this case, the bounding box obtained by the mobile object detection unit 262 may be used.

[0144] Figure 9 is a schematic diagram showing an image in which a moving object specified on a map image has been identified in a camera image. For example, when a moving object with coordinates P1'(256,269) is specified on the map image shown in Figure 7(c), the tracking result processing unit 266 performs an inverse transformation on the coordinates P1'(256,269) of the specified moving object to calculate the position P1(581,80) in the camera image of frame FA. Furthermore, the tracking result processing unit 266 identifies the moving object at that position and displays an image on the display unit 22 that includes the ID "01" of the identified moving object, the type of moving object "car" (passenger car), or the position "P1(581,80)" in the camera image.

[0145] Furthermore, when a moving object with coordinates P2'(231,897) is specified on the map image shown in Figure 7(c), the tracking result processing unit 266 performs an inverse transformation on the specified moving object's coordinates P2'(231,897) to calculate the position P2(544,329) in the camera image of frame FB. In addition, the tracking result processing unit 266 identifies the moving object at that position and displays an image on the display unit 22 that includes the identified moving object's ID "01", the type of moving object "car" (passenger car), or the position "P2(544,329)" in the camera image.

[0146] As another example, as shown in Figure 2, the mobile information processing device 20 can communicate data with a mobile information receiving device 31 mounted on a vehicle MA. The vehicle MA may be an autonomous bus, passenger car, or truck. When the tracking result processing device 266 obtains information (mobile information) such as the position, travel path, travel direction, travel speed, or travel time of a mobile object traveling on a road (for example, the vehicle MB shown in Figure 1), it controls the communication circuit 24 to transmit that mobile object information to the vehicle MA. The communication circuit 24 transmits the mobile object information obtained by the tracking result processing device 266 to the vehicle MA.

[0147] In the vehicle MA, the mobile information receiving device 31 receives mobile information transmitted from the communication circuit 24 of the mobile information processing device 20, and the self-position detection device 32 detects the position of the vehicle MA and obtains the coordinate information of the vehicle MA in a predetermined coordinate system. Based on the mobile information received by the mobile information receiving device 31 and the coordinate information of the vehicle MA obtained by the self-position detection device 32, the mobile information providing device 33 may provide the driver with information about other mobile objects that may potentially collide with the vehicle MA, either as images or audio.

[0148] For example, the mobile information providing device 33 compares the coordinates of the automobile MA obtained from the self-position detection device 32 with the coordinates of other mobile objects obtained from the mobile information receiving device 31 in the same coordinate system to determine the distance between the automobile MA and the other mobile object, and to determine whether or not the other mobile object is located within a predetermined distance from the automobile MA.

[0149] The mobile information providing device 33 may control the display unit or audio output unit of the vehicle MA to display or announce information indicating that another mobile object is present near the vehicle MA, or that there is a possibility of collision with the other mobile object, if it determines that another mobile object is present within a predetermined distance from the vehicle MA.

[0150] Furthermore, the mobile object information provider 33 may provide the driving control device 34 with information about other mobile objects that may potentially collide with the vehicle MA, as message data, based on the mobile object information received by the mobile object information receiver 31 and the coordinate information of the vehicle MA obtained by the self-position detection device 32. The driving control device 34 uses the message data provided by the mobile object information provider 33 to perform control operations related to the driving of the vehicle MA. For example, the driving control device 34 calculates the possibility of the vehicle MA colliding with another vehicle or pedestrian, or the predicted time until a collision.

[0151] The driving control device 34 may control the display unit or audio output unit of the vehicle MA to display or announce information regarding the possibility of collision or the predicted time until collision, based on the calculation results. Alternatively, the driving control device 34 may compare the degree of risk represented by the calculation results with at least one threshold and perform various control operations according to the degree of risk.

[0152] For example, the driving control device 34 may determine that the level of danger is moderate when it exceeds a first threshold and control the audio output unit of the vehicle MA to generate a warning sound to alert the driver. Alternatively, the driving control device 34 may determine that the level of danger is high when it exceeds a second threshold and control the accelerator / brake system 35 to decelerate or stop the vehicle MA, or control the steering system 36 to divert the direction of travel of the vehicle MA away from other vehicles or pedestrians.

[0153] Furthermore, in the example shown in Figure 1, if the automobile MA continues to move forward, the automobile MA may enter the imaging range of the imaging device 10. In such cases, it is desirable to associate the coordinate information of the moving object obtained from the moving object information receiving device 31 with the coordinate information of the automobile MA obtained from the self-position detection device 32 so that the automobile MA captured by the imaging device 10 is not recognized as another moving object.

[0154] For example, the mobile information providing device 33 may compare the coordinates of the mobile object MC obtained from the mobile information receiving device 31 and the coordinates of the automobile MA obtained from the self-position detection device 32 in the same coordinate system, and determine that the mobile object MC and the automobile MA correspond to each other (are the same mobile object) if the distance between them is within a predetermined range.

[0155] Alternatively, the mobile information provider 33 may collect the coordinates of the mobile object MC obtained from the mobile information receiver 31 and the coordinates of the automobile MA obtained from the self-position detection device 32 over a predetermined period, and compare the travel distance and direction over the predetermined period to determine whether they belong to the same mobile object. Even if the coordinate information obtained from the self-position detection device 32 contains errors, the effect of the errors can be reduced by comparing the difference in travel distance and direction.

[0156] The mobile object information provider 33 can avoid providing false information that another mobile object is present near the vehicle MA when the mobile object MC captured by the imaging device 10 corresponds to the vehicle MA. In addition, the determination result of the mobile object information provider 33 is supplied to the driving control device 34 as part of the message data, so that the driving control device 34 can avoid malfunctions in the driving control of the vehicle MA.

[0157] Furthermore, if the imaging device 10 determines that the moving object MC corresponds to the vehicle MA, the mobile object information providing device 33 or the driving control device 34 may use the coordinate information of the mobile object MC obtained from the mobile object information receiving device 31 as the coordinate information of the vehicle MA. This allows for accurate determination of the relative positional relationship between the vehicle MA and other moving objects, even if the coordinate information obtained from the self-position detection device 32 contains errors.

[0158] According to the mobile information processing system shown in Figure 2, by using at least one imaging device placed on the road or elsewhere to acquire images of other mobile objects that are in blind spots that cannot be captured by the vehicle MA's onboard camera, and by transmitting information such as the location of the other mobile objects to the vehicle MA, it is possible to provide a mobile information processing system that can prevent traffic accidents.

[0159] <Second example of operation> Next, a second example of operation of the mobile information processing device shown in Figure 3 will be described with reference to Figures 1 to 10. Figure 10 is a flowchart of the mobile information processing method according to the second embodiment of the present invention. In Figure 10, the same reference numerals are used for the same steps as in the first embodiment shown in Figure 4, and their explanation is omitted.

[0160] In the second embodiment, instead of the process of estimating the position of a moving object on the camera image in the first embodiment (step S13 in Figure 4), step S24 is performed, which is equivalent to estimating the coordinates of a moving object in a predetermined coordinate system. Note that some of the processes shown in Figure 10 may be omitted or modified, or other processes may be added to the processes shown in Figure 10.

[0161] <Step S23> Step S23 is performed using the moving object detection result information obtained in step S12 by applying moving object detection processing to the image signal.

[0162] In step S23, the coordinate transformation unit 264 determines the coordinate information of a moving object in a predetermined coordinate system based on the position information of the moving object on the image (in the second embodiment, the position information obtained by the moving object detection unit 262 in step S12) detected from multiple regions of interest in the image represented by the image signal. The details of the coordinate transformation process in step S23 are the same as in the first embodiment (step S14 in Figure 4). The coordinate transformation unit 264 adds the coordinate information obtained by the coordinate transformation process to the moving object detection result information and stores it in the moving object detection result database.

[0163] <Step S24> In step S24, the mobile object position estimation unit 263 performs mobile object coordinate estimation processing using the coordinate information obtained by the coordinate transformation unit 264 in step S23. For each region of interest, the mobile object position estimation unit 263 determines the estimated coordinates of the first mobile object at the time of other frames based on the coordinates of the first mobile object detected from images of a predetermined number of frames (for example, two or more frames) among multiple frames of the image signal. If the error between the estimated coordinates and the coordinates of the second mobile object detected from the images of the other frames is within a predetermined range, the unit determines that the first mobile object and the second mobile object are the same and saves the coordinate information of the second mobile object detected from the images of the other frames as the coordinate information of the first mobile object.

[0164] As an example, first, the mobile object position estimation unit 263 reads mobile object detection result information from the mobile object detection result database in the cache memory 27. If a mobile object is detected in consecutive first and second frames of multiple frames of an image signal with coordinates shifted within a predetermined distance, the unit determines that they are the same mobile object (first mobile object). The mobile object position estimation unit 263 assigns a mobile object ID (identification information) to the first mobile object, associates the mobile object ID and coordinate information of the first mobile object with the associated information, and stores it as mobile object tracking information in the mobile object tracking information database in the cache memory 27.

[0165] Next, the mobile object position estimation unit 263 reads the mobile object tracking information for the first mobile object from the mobile object tracking information database and, based on the coordinates of the first mobile object detected from the images of the first and second frames, determines the estimated coordinates of the first mobile object at the time of the third frame following the second frame. The mobile object position estimation unit 263 stores the information of the estimated coordinates of the first mobile object at the time of the third frame in the mobile object tracking information database.

[0166] Next, a mobile object coordinate comparison process is performed. The mobile object position estimation unit 263 sequentially reads mobile object detection result information from the mobile object detection result database, and if the error between the coordinates of the second mobile object detected from the image of the third frame and the estimated coordinates is within a predetermined range, it determines that the first mobile object and the second mobile object are the same, and saves the coordinate information of the second mobile object detected from the image of the third frame as the coordinate information of the first mobile object in the mobile object tracking information database.

[0167] As a result, the estimated coordinate information of the first moving object at the time of the third frame is overwritten with the coordinate information of the first moving object, and the moving object tracking information database is updated. Furthermore, the moving object position estimation unit 263 deletes the used moving object detection result information from the moving object detection result database.

[0168] On the other hand, if no moving object is detected in the third frame image whose coordinate error with the estimated coordinates is within a predetermined range, the moving object position estimation unit 263 does not update the moving object tracking information database, and therefore uses the estimated coordinate information as the coordinate information of the first moving object at the time of the third frame.

[0169] Furthermore, if the error between the coordinates of the second moving object detected from the image of the third frame and the estimated coordinates exceeds a predetermined range, the moving object position estimation unit 263 determines that the first moving object and the second moving object are different. That is, the moving object position estimation unit 263 determines that a second moving object different from the first moving object has been discovered, assigns a new moving object ID to the second moving object, associates the moving object ID and coordinate information of the second moving object with the associated information, and stores it in the moving object tracking information database as moving object tracking information.

[0170] The above moving object coordinate comparison process is repeated for each moving object detection result information obtained by the moving object detection process and coordinate transformation process on the image signal obtained from the imaging device 10. Furthermore, if m moving objects are detected from the first group of frames of the image signal obtained from the imaging device 10 (where m is a natural number), the moving object coordinate estimation process in step S24 is repeated for each of the m moving objects.

[0171] If n new moving objects are discovered during this process (where n is a natural number), the moving object coordinate estimation process in step S24 is repeated for each of the n additional moving objects. Furthermore, the processes in steps S12 to S24 are repeated a number of times equal to the number of regions of interest set by the image processing unit 261.

[0172] Alternatively, the mobile object position estimation unit 263 may assign the same mobile object ID to the first mobile object and the second mobile object if the error between the coordinates of the second mobile object detected from the image of the third frame and the estimated coordinates is within a predetermined range. If the first mobile object has already been assigned a mobile object ID, the mobile object position estimation unit 263 may assign the same mobile object ID to the second mobile object as to the first mobile object. In this way, the first mobile object and the second mobile object, which have been determined to be the same, are linked to each other.

[0173] Similar to the first embodiment, the above estimation can be performed using methods such as object position prediction using a Kalman filter, or statistical methods for estimating data in an unknown range, such as extrapolation or interpolation.

[0174] <Step S25> In step S25, the moving object tracking unit 265 performs moving object tracking (moving object tracking process) by determining the correspondence between moving objects detected from multiple regions of interest based on the coordinate information of the moving object in a predetermined coordinate system. The detailed processing in step S25 is the same as in the first embodiment (step S15 in Figure 4).

[0175] <Step S26> In step S26, the tracking result processing unit 266 obtains information such as the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple regions of interest, based on the coordinate information in a predetermined coordinate system of the moving object that the moving object tracking unit 265 determined to be corresponding to in step S25. The detailed processing in step S26 is the same as in the first embodiment (step S16 in Figure 4).

[0176] <Effects of the Embodiment> According to the first or second embodiment, a first region of interest A1 and a second region of interest A2 having overlapping portions are set within the image of each frame of the image signal obtained from at least one imaging device 10, a moving object is detected from these regions of interest, the coordinate information of the moving object in a predetermined coordinate system is obtained based on the position information of the moving object on the image, and the correspondence between the moving object detected from the first region of interest A1 and the moving object detected from the second region of interest A2 is determined based on the coordinate information of the moving object, thereby performing tracking of the moving object (moving object tracking process) across multiple regions of interest. This makes it possible to narrow the target area for moving object detection from the entire image to a relatively narrow region of interest, thereby reducing the number of false detections that may occur during moving object detection, while continuously tracking a moving object moving between the vicinity and the distance of the imaging device 10.

[0177] The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the invention by those who have ordinary skill in the art. For example, it is possible to implement at least a part of the first embodiment and at least a part of the second embodiment in combination. [Industrial applicability]

[0178] The present invention can be used in a mobile information processing device that obtains information such as the position or travel path of a moving object, such as an automobile, based on an image signal. [Explanation of symbols]

[0179] 10...Imaging device, 10a...Recording device, 20...Mobile object information processing device, 21...Operation unit, 22...Display unit, 23...Audio input / output unit, 24...Communication circuit, 25...Interface, 26...CPU, 260...Image signal acquisition unit, 261...Image processing unit, 262...Mobile object detection unit, 263...Mobile object position estimation unit, 264...Coordinate transformation unit, 265...Mobile object tracking unit, 266...Tracking result processing unit, 27...Cache memory, 28...Storage unit, 31...Mobile object information receiving device, 32...Self-position detection device, 33...Mobile object information providing device, 34...Driving control device, 35...Accelerator / brake system, 36...Steering wheel system

Claims

1. A mobile object information processing device that obtains information about a moving object based on an image signal obtained from at least one imaging device that captures a spatial region and generates an image signal, An image processing unit sets a first region of interest and a second region of interest having overlapping portions within the image of each frame of the aforementioned image signal, A moving object detection unit that detects a moving object from the first and second regions of interest in the images of multiple frames of the aforementioned image signal, A coordinate transformation unit that obtains coordinate information of a moving object in a predetermined coordinate system based on the position information of the moving object on the image detected from the first and second regions of interest, A moving object tracking unit determines the correspondence between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest, based on the coordinate information of the moving object in the predetermined coordinate system. A tracking result processing unit obtains information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple regions of interest, based on the coordinate information of the moving object in the predetermined coordinate system determined to correspond to the moving object by the moving object tracking unit. A mobile information processing device equipped with the following features.

2. The image processing unit sets a first region of interest within the image of each frame of the image signal, corresponding to a first spatial region relatively close to the imaging device within the spatial region captured by the imaging device, and a second region of interest corresponding to a second spatial region relatively far from the imaging device. The mobile object information processing apparatus according to claim 1, wherein the mobile object detection unit resamples image data representing an image in at least the second region of interest in order to extract a predetermined number of pixels that are the target of the mobile object detection process, thereby reducing the rate of decrease in the number of pixels compared to resampling image data representing the entire image.

3. The image processing unit sets, within the image of each frame of the image signal, a first region of interest corresponding to a first spatial region relatively close to the imaging device within the spatial region captured by the imaging device, and a second region of interest corresponding to a second spatial region relatively far from the imaging device and having a smaller size than the first region of interest. The mobile object information processing apparatus according to claim 1, wherein the mobile object detection unit resamples image data representing an image in the first region of interest and image data representing an image in the second region of interest in order to extract a predetermined number of pixels that are the target of the mobile object detection process, thereby suppressing the rate of decrease in the number of pixels in the second region of interest to a rate of decrease in the number of pixels in the first region of interest.

4. The mobile object information processing apparatus according to claim 1, wherein the mobile object tracking unit determines that the two mobile objects correspond to each other when the distance between the coordinates of the two mobile objects, respectively, obtained from the first and second regions of interest in the same frame of the image signal, is within a predetermined range.

5. The mobile object information processing device according to claim 1, wherein, if the mobile object tracking unit does not detect a mobile object whose coordinate distance from the first and second regions of interest in the same frame of the image signal is within a predetermined range, it determines the correspondence of mobile objects using information of estimated coordinates obtained based on the position or coordinates of mobile objects detected from the first or second regions of interest in images of a predetermined number of other frames of the image signal.

6. The mobile object information processing device according to claim 1, further comprising a mobile object position estimation unit that, with respect to each region of interest, determines the estimated position of the first mobile object at the time of another frame based on the position of the first mobile object detected from a predetermined number of frames of the image signal, determines that the first mobile object and the second mobile object are the same if the error between the position of the second mobile object detected from the other frames and the estimated position is within a predetermined range, and stores the position information of the second mobile object detected from the other frames as position information of the first mobile object on the image.

7. The mobile object information processing device according to claim 6, wherein if the mobile object position estimation unit does not detect a mobile object in the image of the other frame whose positional error with the estimated position is within a predetermined range, it uses the information of the estimated position as the positional information of the first mobile object on the image at the time of the other frame.

8. The mobile object information processing device according to claim 1, further comprising a mobile object position estimation unit that, with respect to each region of interest, determines the estimated coordinates of the first mobile object at the time of another frame based on the coordinates of the first mobile object detected from a predetermined number of frames of the image signal, determines that the first mobile object and the second mobile object are the same if the error between the coordinates of the second mobile object detected from the images of the other frames and the estimated coordinates is within a predetermined range, and stores the coordinate information of the second mobile object detected from the images of the other frames as coordinate information of the first mobile object.

9. The mobile object information processing device according to claim 8, wherein if the mobile object position estimation unit does not detect a mobile object in the image of the other frame whose coordinate error with the estimated coordinates is within a predetermined range, it uses the estimated coordinate information as the coordinate information of the first mobile object at the time of the other frame.

10. A mobile information processing device according to any one of claims 1 to 9, further comprising a communication circuit for transmitting information obtained by the tracking result processing unit to an automobile, A mobile information receiving device mounted on the aforementioned automobile, which receives information transmitted from the communication circuit of the mobile information processing device, A self-position detection device mounted on the aforementioned automobile for detecting the position of the aforementioned automobile, A mobile information providing device mounted on the aforementioned automobile provides information about other mobile objects in the form of telegram data, images, or audio, based on information received by the mobile information receiving device and the detection results of the self-position detection device. A mobile information processing system equipped with the following features.

11. The mobile information processing system according to claim 10, further comprising a driving control device mounted on the vehicle, which performs control operations related to the driving of the vehicle using information received by the mobile information receiving device and the detection results of the self-position detection device.

12. A mobile object information processing program used in a mobile object information processing device that obtains information about a moving object based on an image signal obtained from at least one imaging device that captures a spatial region and generates an image signal, A procedure (a) for setting a first region of interest and a second region of interest having overlapping portions within the image of each frame of the aforementioned image signal, (b) A procedure for detecting a moving object from the first and second regions of interest in images of multiple frames of the aforementioned image signal, A procedure (c) for determining the coordinate information of a moving object in a predetermined coordinate system based on the position information of the moving object on the image detected from the first and second regions of interest, A procedure (d) for determining the correspondence between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest, based on the coordinate information of the moving object in the predetermined coordinate system, Step (d) is a procedure to obtain information regarding the position, path, direction of movement, speed, or time of movement of at least one moving object across multiple regions of interest, based on the coordinate information of the moving object in the predetermined coordinate system determined to correspond in step (d). A mobile information processing program that instructs the CPU to execute.

13. A method for processing information about a moving object, which obtains information about a moving object based on an image signal obtained from at least one imaging device that captures a spatial region and generates an image signal, Step (a) of setting a first region of interest and a second region of interest having overlapping portions within the image of each frame of the aforementioned image signal, (b) step of detecting a moving object from the first and second regions of interest in the images of multiple frames of the image signal, (c) A step of determining the coordinate information of a moving object in a predetermined coordinate system based on the position information of the moving object on the image detected from the first and second regions of interest, (d) A step of determining a correspondence between at least one moving object detected from the first region of interest and at least one moving object detected from the second region of interest, based on the coordinate information of the moving object in the predetermined coordinate system. Step (e) involves obtaining information regarding the position, movement path, movement direction, movement speed, or movement time of at least one moving object across multiple regions of interest, based on the coordinate information of the moving object in the predetermined coordinate system determined to correspond in step (d), A mobile information processing method comprising the following:

Images