Information processing device, information processing method, and computer program

Abstract

To provide an information processor capable of reducing influence of a detected position error caused by a position estimation error factor on a road surface.SOLUTION: An information processor acquires an image, detects, on the basis of the image, an object position on the image, and converts the object position on the image to an object position in world coordinates; in moving object tracking (prediction unit) 34, a reliability calculation unit 710 calculates reliability of the object position in the world coordinates; an error factor existence determination unit 711 determines whether the detected object position on the image is affected by a position estimation error factor on a road surface; a reliability change unit 712 changes the reliability of the object position according to a determination result of the error factor existence determination unit 711; and a speed estimation unit 713 estimates speed of the object using a past history of the object position in the world coordinates and a past history of the changed reliability.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, a computer program, and the like that detects the position of an object. [Background technology]

[0002] There are conventional techniques for detecting the position of an object (such as a vehicle) on an image, converting the detected position on the image into world coordinates, and estimating the speed using the converted object's position information history. In addition, a representative technique for estimating the speed of an object from the object's position information history and position reliability information is the Kalman filter.

[0003] Patent Document 1 discloses a technique for changing the observation noise, which is the reliability of the detected position of an object, depending on the type and position / posture of the object, in order to improve the accuracy of estimating the speed of the object using a Kalman filter. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-191728 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when road markings or the like are present around an object (such as a vehicle), the detected position of the object in the image may become unstable, resulting in a decrease in the accuracy of speed estimation.

[0006] Patent Document 1 describes a technique for changing the observation noise, which is the reliability of the position, depending on the type and position and orientation of the object. However, there is no description of application processing depending on the background, such as road markings. Therefore, in order to deal with position estimation error factors on the road surface, such as road markings, it is necessary to obtain reliability information of the detected position of the object in the actual use state after the installation of the imaging unit.

[0007] Therefore, one object of the present invention is to provide an information processing device capable of reducing the influence of detected position errors caused by position estimation error factors on the road surface. [Means for solving the problem]

[0008] An information processing device according to one aspect of the present invention includes: image acquisition means for acquiring an image; an object position detection means for detecting an object position on the image based on the image; a coordinate conversion means for converting the object position on the image into an object position in world coordinates; a reliability calculation means for calculating a reliability of the object position in the world coordinates; an error factor presence determination means for determining whether or not the object position detected by the object position detection means is affected by a position estimation error factor on a road surface; a reliability changing means for changing the reliability of the object position in accordance with a result of the determination by the error factor existence determining means; and a velocity estimation means for estimating the velocity of the object using a past history of the object position in the world coordinates and a past history of the changed reliability. Effect of the Invention

[0009] According to the present invention, it is possible to provide an information processing device capable of reducing the influence of detected position errors caused by position estimation error factors on the road surface. [Brief description of the drawings]

[0010] [Figure 1] 1 is an overhead view of a system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of observation devices 100, 101 and an integrating device 110. [Diagram 3] FIG. 2 is a functional block diagram illustrating functions executed by a CPU 201. [Figure 4] 11 is a flowchart showing an example of processing performed by a CPU 201. [Diagram 5] FIG. 4 is a diagram for explaining coordinate transformation in the first embodiment. [Figure 6] 11 is a flowchart illustrating processing performed by a CPU 221. [Figure 7] FIG. 2 is a functional block diagram for explaining a detailed configuration of a moving object tracking (prediction) unit 34. [Figure 8] 5 is a flowchart for explaining detailed processing in step S430 in FIG. 4. [Figure 9] 13(A) to 13(F) are diagrams for explaining an example in which an error occurs in the detection position of an observed vehicle due to the influence of road markings. [Figure 10] 6A to 6I are graphs illustrating the effect of the present invention on the influence of detection position errors of an observed vehicle caused by road markings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified. EXAMPLES

[0012] 1 is an overhead view of a driving assistance system according to a first embodiment of the present invention. Observation devices 100 and 101 are placed near an intersection, and the observation devices 100 and 101 are capable of communicating with an integrating device 110. The observation devices 100 and 101 observe the movements of observed vehicles 130 and 131, etc., and transmit data to the integrating device 110.

[0013] When the notification target vehicle 120 makes a right turn, the integrating device 110 predicts the time when the observed vehicles 130, 131 will arrive at the intersection, and notifies the notification target vehicle 120 of notification information such as whether or not the vehicle can turn right. If the notification target vehicle 120 is, for example, an autonomous vehicle, it can decide to accelerate, decelerate, stop, etc. based on the notification information.

[0014] When receiving notification information that a right turn is not permitted, the notification target vehicle 120 preferably stops before the collision warning area 140. Even if the notification target vehicle 120 is not an autonomous vehicle, the driver of the notification target vehicle 120 can use the notification information to determine acceleration, deceleration, stopping, etc. Incidentally, the observation devices 100 and 101 function as moving object tracking devices serving as information processing devices in this embodiment.

[0015] 2 is a block diagram showing an example of the configuration of the observation devices 100, 101, and the integrated device 110. Since the observation devices 100 and 101 have the same configuration, only the observation device 100 will be described. The observation device 100 includes a PC 200, an image capture unit 204, a communication unit 205, and the like.

[0016] The PC 200 is composed of a CPU 201 as a computer, a RAM 202 as a memory, a ROM 203, etc., and can communicate with each other via a bus. The image capturing unit 204 is composed of a network camera, etc. The communication unit 205 can be a mobile router, etc. The PC 200, the image capturing unit 204, and the communication unit 205 are connected to a LAN via a switching hub and can communicate with each other.

[0017] The integrated device 110 comprises a PC 220, a notification unit 224, and a communication unit 225. The PC 220 comprises a CPU 221 as a computer, a RAM 222 as a memory, a ROM 223, etc., which can communicate with each other via a bus.

[0018] The notification unit 224 only needs to have a means for notifying the target vehicle 120, and may be the same as the communication unit 225 if the notification is by wireless communication. Alternatively, the notification unit 224 may be a device such as a display or a speaker. A mobile router or the like may be used for the communication unit 225. The PC 220, the notification unit 224, and the communication unit 225 are connected to a LAN via a switching hub and are capable of communicating with each other.

[0019] The configuration shown in Fig. 2 is merely one embodiment, and is not limited thereto. For example, any number of observation devices and imaging units may be used. Furthermore, the multiple PCs, CPUs, RAMs, ROMs, and communication units may be shared, or may be further divided. The communication means may be either wired or wireless.

[0020] Next, Fig. 3 is a functional block diagram explaining the functions performed by the CPU 201. The functions performed by the CPU 211 are similar, so the explanation will be omitted. Some of the functional blocks shown in Fig. 3 are realized by making the CPU 201, which serves as a computer included in the observation device 100, execute a computer program stored in a memory, which serves as a storage medium.

[0021] However, a part or all of these may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0022] In addition, the functional blocks shown in Fig. 3 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 3 also applies to Fig. 7.

[0023] 3 includes an image acquisition unit 30, an object position detection unit 31, an erroneous detection removal unit 32, a coordinate conversion unit 33, an object tracking (prediction) unit 34, and a data transmission unit 35. The image acquisition unit 30 (image acquisition means) executes an image acquisition step of acquiring a photographed image from the photographing unit 204. The object position detection unit 31 (object position detection means) performs image recognition based on the photographed image to detect the position of an object on the image.

[0024] The erroneous detection removal unit 32 removes erroneous detections made by the object position detection unit 31. The coordinate conversion unit 33 (coordinate conversion means) converts the object position on the image (image coordinates) into the object position in world coordinates. The object tracking (prediction) unit 34 estimates the speed, traveling direction, etc. of the object. The data transmission unit 1005 transmits the estimation result to the integration device 110.

[0025] Next, Fig. 4 is a flowchart showing an example of processing performed by the CPU 201. Note that the CPU 201 as a computer executes a computer program stored in memory to perform the operations of each step in the flowchart in Fig. 4. Note that the processing performed by the CPU 211 is similar, so a description thereof will be omitted.

[0026] The correspondence between the functions in the block diagram of Fig. 3 and the processes in Fig. 4 is as follows: The object position detection unit 31 performs the process of step S400. The false detection removal unit 32 performs the process of step S410. The coordinate conversion unit 33 performs the process of step S420. The object tracking (prediction) unit 34 performs the process of step S430. The data transmission unit 35 performs the process of step S440.

[0027] The flow in Fig. 4 starts when a captured image is acquired from the image capturing unit 204. The image capturing unit 204 periodically acquires captured images at, for example, 60 frames per second, and the flow chart in Fig. 4 is executed for each frame. The CPU 201 reads a computer program stored in advance in the ROM 203, expands it in the RAM 202, and performs processing.

[0028] In step S400 (object position detection step), the CPU 201 detects an object such as a vehicle from the captured image acquired from the image capture unit 204 by image recognition. That is, in step S400, the object position on the image is detected based on the image. A model trained by deep learning can be used as a means for detecting the object. A bounding box is assigned to the detected object.

[0029] In step S410, the CPU 201 removes erroneous detections from the objects detected in step S400. In order to accurately detect objects such as vehicles and motorcycles to be detected, the classifier is changed in advance or a threshold value for the classification result is adjusted to suppress the detection of unintended objects.

[0030] On the other hand, when a process of adding a bounding box to a detected object in a captured image is performed, there may be cases where a plurality of bounding boxes exist and there are areas where the bounding boxes overlap each other.

[0031] In that case, processing may be performed so that all objects are positioned at the position of the bounding box that is determined to be the closest to the image capture unit. This allows, for example, when a person is riding a motorcycle, processing to determine that the motorcycle and the person are present at the motorcycle's position, improving the accuracy of the determination of whether or not the notification target vehicle 120 is allowed to make a right turn.

[0032] In step S420 (coordinate conversion step), CPU 201 performs coordinate conversion (projective conversion) of the position of the object in the captured image from the camera coordinates (plane) to the world coordinates (plane). That is, in step S420, conversion is performed from the object position on the image to the object position in the world coordinates.

[0033] Fig. 5 is a diagram for explaining coordinate conversion in the first embodiment. Fig. 5(B) is a captured image, and a position on the captured image can be regarded as a camera coordinate (plane). Fig. 5(A) corresponds to a world coordinate (plane). The world coordinate (plane) is the same as the road surface (plane) (it does not match widely, but it is assumed that it is the same locally).

[0034] The projection transformation matrix for coordinate conversion may be generated from the position of the observation device and 3D information of the object to be photographed. A 3D scanner or the like may be used to acquire the 3D information. Alternatively, as another method, a projection transformation matrix for coordinate conversion may be created by linking an orthoimage including map information such as latitude and longitude with corresponding points of an image photographed by the photographing device of the observation device 100.

[0035] For example, the feature points of the road marking 520 in Fig. 5(A) may be linked to the feature points of the road marking 520 in Fig. 5(B). By storing a projective transformation matrix in advance in the ROM 203, the CPU 201 may use the projective transformation matrix for coordinate transformation. By the coordinate transformation, the CPU 201 can obtain depth information from the image capture device.

[0036] 5B is obtained by detecting the observed vehicle 130 in step S400. The position of the observed vehicle 130 can be represented by position 512, which is the midpoint of the bottom side of the bounding box 511.

[0037] By the coordinate conversion in step S420, the position of the observed vehicle 130 in the world coordinates (plane) can be represented by a single point, position 512 in Fig. 5(A). Note that the above method of determining the position of the observed vehicle 130 is merely one example, and is not limited to this method.

[0038] In step S430, the CPU 201 performs moving object tracking (prediction) of the position of the observed vehicle 130. The image capturing unit 204 periodically captures images, and by tracking the position of the observed vehicle 510 in multiple frames, it becomes possible to predict the future position.

[0039] In step S440, the CPU 201 transmits information such as the predicted position of the observed vehicle 130 to the CPU 221 of the PC 220 of the integrating device 110 via the communication units 205 and 225.

[0040] Next, Fig. 6 is a flowchart explaining the processing performed by the CPU 221. The CPU 221 as a computer executes a computer program stored in memory to perform the operations of each step in the flowchart in Fig. 6. The CPU 221 reads out the computer program stored in advance in the ROM 223, expands it in the RAM 222, and performs processing.

[0041] In step S600, the CPU 221 receives information such as the predicted position of the observed vehicle from each of the observation devices 100 and 101. The number of observed vehicles is not limited to one, but may be multiple. The CPU 221 performs synchronization processing based on the time information when each of the observation devices 100 and 101 acquired an image.

[0042] In step S610, the CPU 221 predicts the time or arrival time of the observed vehicle to reach the collision-warning region 140 at the intersection. If there are multiple observed vehicles, the CPU 221 predicts the earliest arrival time of the observed vehicle to reach the collision-warning region 140.

[0043] When there are multiple lanes, the CPU 221 predicts the leading vehicle in each lane. Also, the CPU 221 can determine the likelihood of a collision between the observed vehicle and the notification target vehicle 120 based on the characteristics of the notification target vehicle 120, and can determine and notify whether the notification target vehicle 120 can make a right turn.

[0044] The characteristics of the notification target vehicle 120 include at least one of the acceleration force, mass, size, and vehicle type (large vehicle, small / medium-sized vehicle, motorcycle, etc.) of the notification target vehicle 120. The CPU 221 can also determine whether the notification target vehicle 120 can turn right depending on environmental conditions such as road structure (incline, curve, etc.) and weather conditions.

[0045] In step S620, the CPU 221 generates notification information such as whether or not a right turn is possible to be notified to the notification target vehicle 120. Information other than whether or not a right turn is possible may include information such as the arrival time of the observed target vehicle 130 and the vehicle type. In step S630, the CPU 221 notifies the notification target vehicle 120 of the notification information. The notification may be made by wireless communication via the communication unit 225.

[0046] 7 is a functional block diagram for explaining a detailed configuration of the moving object tracking (prediction) unit 34. The object tracking (prediction) unit 34 has a reliability calculation unit 710, an error factor existence determination unit 711, a reliability change unit 712, a speed estimation unit 713, a basic observation noise table 720, and a corrected observation noise table 721. The observation noise represents the uncertainty of the observation value, and is represented by an error covariance matrix.

[0047] The reliability calculation unit 710 (reliability calculation means) executes a reliability calculation step to calculate the reliability of the object position in world coordinates. The reliability here is that of observation noise that is not related to position estimation error factors on the road surface. In this embodiment, the observation noise calculated here is called basic observation noise. Note that typical examples of position estimation error factors include road markings (crosswalks, stop lines, maximum speed signs, etc.).

[0048] An error factor existence determination unit 711 (error factor existence determination means) determines whether or not the object position detected by the object position detection unit is affected by a position estimation error factor on the road surface (road markings, etc.). A reliability change unit 712 (reliability change means) changes the reliability (basic observation noise) of the object position according to the determination result of the error factor existence determination unit 711. In this embodiment, the corrected amount of observation noise calculated by the reliability change unit 712 is called corrected observation noise, and the corrected observation noise calculated here is called applied observation noise.

[0049] A table for calculating the corrected observation noise is held in the corrected observation noise table 721. A speed estimation unit 713 (speed estimation means) estimates the speed of the object using the past history of the object position in the world coordinates and the past history of the reliability after the change.

[0050] Fig. 8 is a flowchart for explaining the detailed processing in step S430 in Fig. 4. The operation of each step in the flowchart in Fig. 8 is performed by CPU 201 as a computer executing a computer program stored in memory. The processing flow of the moving object tracking (prediction) unit in Fig. 8 estimates the actual position and speed of the observed vehicle from history information of the detected position of the observed vehicle, which includes errors.

[0051] The correspondence between the functional blocks in Fig. 7 and the processes in Fig. 8 is as follows: The reliability calculation unit 710 performs the processes of steps S800 and S801. The error factor existence determination unit 711 performs the process of step S801. The reliability change unit 712 performs the processes of steps S803 and S804. The speed estimation unit 713 performs the process of step S805.

[0052] First, in step S800, the Euclidean distance between the position of the image capture unit 204 in the world coordinates and the detected position of the observed vehicle 130 is calculated. In this embodiment, the Euclidean distance calculated here is called the detection distance. The position of the image capture unit 204 in the world coordinates is set in advance. The detected position of the observed vehicle 130 in the world coordinates is the one calculated in step S420. The difference between these two positions is taken, and the detection distance is calculated as the Euclidean distance.

[0053] In the next step S801 (reliability calculation step), basic observation noise, which is the observation noise when there are no position estimation error factors (road markings, etc.) on the road surface, is calculated as the reliability. That is, in step S801, the reliability for the object position in the world coordinates is calculated.

[0054] The basic observation noise is changed according to the detection distance calculated in step S800. An observation noise table for the detection distance is prepared and stored in advance, and the basic observation noise is calculated from the detection distance calculated in step S800 and the observation noise table.

[0055] The observation noise table at this time may be changed according to the recognition type, the installation height or the installation angle of the imaging unit, or a fixed value regardless of the detection distance.

[0056] The observation noise table is created from statistical data of the detection position of the observed vehicle when there are no position estimation error factors (road markings, etc.) on the road surface measured in advance and from correct position data of the observed vehicle that is separately prepared. The correct data may be generated by installing the observed vehicle and measuring it.

[0057] In addition, sensors such as a Global Positioning System (GPS), a Light Detection and Ranging (LiDAR), and a camera may be installed separately and used as correct data.

[0058] As a method of statistical processing, the variance is calculated for the difference between the detection position of the observed vehicle and the correct answer data. At this time, the variance may be calculated for the difference between the average value of the detection position of the observed vehicle without using the correct answer data. In addition, the observation noise table may be calculated by performing statistical processing for each recognition type, each installation height of the image capture unit, and each installation angle of the image capture unit.

[0059] The observation noise table may be calculated from statistical data of the detection positions of the observed vehicle in the camera coordinates. In this case, the variance value in the camera coordinates may be converted to the variance value in the world coordinates using coordinate transformation to create the table data.

[0060] In the next step S802, it is determined whether the detected position of the observed vehicle 130 is within a preset judgment area. In this embodiment, the area for determining whether or not the object position is affected by a position estimation error factor on the road surface (road markings, etc.) is called a judgment area, and step S802 also functions as a judgment area determination unit that determines the judgment area. Step S802 also functions as an error factor presence determination step that determines whether or not the object position detected in step S400 as an object position detection step is affected by a position estimation error factor on the road surface.

[0061] In this embodiment, the judgment area can be set by the user using a setting value. It may also be set according to the type of road marking or the size of the road marking in camera coordinates. The judgment area may also be specified as an area in camera coordinates or as an area in world coordinates.

[0062] An example of how to determine the setting value of the judgment area is shown below. Statistical data on the detection position of the observed vehicle when there is a road marking and separately prepared correct position data of the observed vehicle are collected in advance. If the difference between the detection position of the observed vehicle on the camera coordinates and the correct position is equal to or greater than a threshold, it is determined to be erroneous detection data influenced by position estimation error factors on the road surface.

[0063] At this time, the detection position of the observed vehicle of the erroneous detection data is somewhere near the road marking. A road marking coordinate system is set based on a specific point of the road marking, and the detection position of the observed vehicle in the camera coordinates is converted to the road marking coordinate system. At this time, statistical data of the erroneous detection data of the observed vehicle in the road marking coordinate system is obtained.

[0064] An area including the positions of erroneous detection occurring in this statistical data is set as a judgment area. The criterion for the included area may be an area that includes 90% of the positions of erroneous detection. When acquiring this statistical data, statistics may be taken for each road marking type, road marking size in camera coordinates, road marking inclination, or camera coordinate size of the observed vehicle.

[0065] In addition, by collecting this statistical data before installing the imaging unit to be used, it is possible to select an appropriate judgment area from the statistical data when setting the judgment area. However, if there is no appropriate statistical data, the statistical data may be collected after installing the imaging unit to be used, and the judgment area may be set.

[0066] The method of determining the judgment area in the judgment area determination unit may be configured to automatically set the judgment area using the road marking detection unit. In this case, the road markings are learned in advance by deep learning or the like, and the type and position of the road markings are made recognizable. Also, statistical data on false detection data is collected for each type of road marking recognized in advance, and for each size and posture in the camera coordinates.

[0067] A table is created for setting judgment areas according to the recognized road marking type, size and orientation of the road marking in the camera coordinates using this statistical data. After the photographing unit is installed, the road marking detection unit recognizes the road markings, and the judgment area determination unit automatically sets the judgment area based on the recognized road marking type, size and orientation in the camera coordinates. In other words, the judgment area is calculated in advance according to the detection results by the road marking detection unit.

[0068] Also, the determination as to whether or not the object position is affected by a position estimation error factor on the road surface (road markings, etc.) may be made using the amount of change in the bounding box size. That is, the object position detection unit 31 may generate a bounding box, and the error factor existence determination unit 711 may determine whether or not the object position is affected by a position estimation error factor on the road surface based on the amount of change in the size of the bounding box. In that case, the determination may be made by referring to the history of the bounding box information.

[0069] If the information used is the aspect ratio of the bounding box, it is determined that the road surface is affected by a position estimation error factor when the change in the aspect ratio is equal to or greater than a threshold. It may also be configured to make the determination based on a change in at least one of the aspect ratio, vertical size, horizontal size, and bottom edge position of the bounding box.

[0070] Also, the determination of whether or not the road surface is affected by position estimation error factors may be made using the amount of change in the ratio of white areas within the bounding box. An embodiment will be described below. The inside of the bounding box is binarized by the signal level to calculate the white area.

[0071] Next, the ratio of the white area calculated by binarization to the number of pixels in the bounding box is calculated as the white area ratio. If the change in the white area ratio is equal to or greater than a threshold, it is determined that the road surface is affected by position estimation error factors.

[0072] Note that, instead of the white area ratio, the determination may be made using, for example, a change in the ratio of a yellow area, etc. In other words, it may be possible to determine whether or not the object position is affected by position estimation error factors on the road surface based on the amount of change in a predetermined color area within a bounding box in an image.

[0073] In the next step S803, a correction observation noise for reducing the influence of the position estimation error factors on the road surface is calculated. This process is carried out when it is determined that the position estimation is influenced by the position estimation error factors on the road surface.

[0074] Here, a corrected observation noise table stored in advance is used to calculate the corrected observation noise according to the detection distance. This corrected observation noise table may be changed according to the recognition type. Also, the table data may be changed according to the installation height and installation angle of the image capture unit. Also, it may be changed according to the size of the road marking in the camera coordinates. Also, it may be a fixed value regardless of the detection distance.

[0075] In addition, the corrected observation noise table is created from statistical data of the detection position of the observed vehicle when there is a road marking measured in advance and statistical data of the correct data. When the difference between the detection position of the observed vehicle on the camera coordinates and the correct position is equal to or greater than a threshold, it is determined to be erroneous detection data that is influenced by position estimation error factors on the road surface. Statistical processing is performed using this erroneous detection data and the correct data.

[0076] As a statistical processing method, the variance is calculated for the difference between the false detection position of the observed vehicle in the world coordinate and the correct answer data. At this time, the variance may be calculated for the difference between the average value of the false detection position of the observed vehicle in the world coordinate without using the correct answer data.

[0077] The corrected observation noise table may also calculate statistical data for each type of road marking, each size of the camera coordinates of the road marking, each type of object recognition, each installation height of the camera unit, and each installation angle of the camera unit. The correct answer data may be generated by installing an observation target vehicle and measuring it. Also, a separate sensor such as a GPS, LiDAR, or camera may be installed and used as the correct answer data.

[0078] The table may also be calculated from statistical data of vehicle detection positions in camera coordinates. In this case, the variance value in the camera coordinates may be converted to a variance value in the world coordinates using coordinate conversion to create the table data. Furthermore, the correction observation noise table may hold the difference value of the values ​​calculated in the basic observation noise table.

[0079] In the next step S804, the basic observation noise is corrected using the correction observation noise calculated in step S803 to calculate the applied observation noise. The correction process is performed by overwriting the correction observation noise with the basic observation noise. Alternatively, the basic observation noise and the correction observation noise may be added together.

[0080] In this way, steps S803 and S804 function as a reliability changing step for changing the reliability (observation noise) of the object position depending on the result of the error factor existence determining step.

[0081] Next, in step S805, the velocity of the object is estimated by Kalman filter processing using the past history of the position information in world coordinates and the past history of the reliability of the position information. Here, step S805 functions as a velocity estimation step that estimates the velocity of the object using the past history of the object position in world coordinates and the past history of the reliability after the change.

[0082] The inputs for the Kalman filter processing are the detected position of the observed vehicle in world coordinates, the applied observation noise, and the detection time. This Kalman filter processing unit may use conventional technology as it is.

[0083] The Kalman filter is divided into a prediction section and an observation update section. The Kalman filter also holds internal state estimates, which are the three-dimensional position and three-dimensional velocity of an object in world coordinates.

[0084] The covariance matrix of the error, which is the reliability of the state estimate, is also stored. This covariance matrix of the error, which is the reliability of the state estimate, is called the estimate noise. The inputs to the Kalman filter are the detection position of the observed vehicle in world coordinates, which is the observation value, the observation time, and the applied observation noise.

[0085] The moving object tracking (prediction) unit predicts the speed and position at the observation time using the past speed and position (state estimates) that it has stored. At this time, the estimate noise is also updated. The observation update unit updates the state estimate and state noise using the detected position of the observed vehicle in world coordinates and the applied observation noise.

[0086] The Kalman filter outputs the state estimates of the observation update unit (such as the three-dimensional position and three-dimensional speed of the observed vehicle). Incidentally, when the applied observation noise is greater than the estimated value noise, the output of the state estimates of the observation update unit is weighted more heavily by the state estimates of the prediction unit. The Kalman filter process may also be configured to estimate the positions and speeds of multiple observed vehicles. It may also be configured to be capable of detecting multiple object positions.

[0087] As an example of this case, a multi-tracker Kalman filter will be explained. Here, a combination of state estimates and state noise is called a tracker, and a configuration that holds and processes multiple trackers is called a multi-tracker. In a multi-tracker Kalman filter, multiple combinations of the state estimates and state noise of the above Kalman filter are held.

[0088] In a multi-tracker Kalman filter, there are multiple trackers and the detected positions of multiple observed vehicles. Therefore, it is necessary to determine which tracker corresponds to the detected observed vehicle. At this time, the position of the detected observed vehicle and the state estimate (position) of the tracker that are close to each other are determined to be in correspondence.

[0089] Alternatively, to determine correspondence, the positions of the world coordinates may be converted back to the coordinate system on the image, and those with a high overlap rate of the bounding boxes may be determined to be corresponding. After determining correspondence, Kalman filter processing is performed on the corresponding objects. The processing at this time is the same as that of the Kalman filter for a single tracker.

[0090] In the above embodiments, examples have been described in which road markings are included as factors of position estimation error on the road surface. However, the factors of position estimation error on the road surface in this embodiment are not limited to road markings. For example, factors of position estimation error on the road surface may be structures on the road surface such as manholes, or objects temporarily present on the road surface such as snow, puddles, and fallen objects (fallen leaves).

[0091] In the above embodiment, only vehicles are described as objects to be observed. However, the objects in the present embodiment are not limited to vehicles. Any moving object, such as a person or an animal, may be used.

[0092] In the above embodiment, only an example has been described in which a Kalman filter is used as the velocity estimation unit 713 that estimates the velocity of an object by filtering using the past history of position information in world coordinates and the past history of the reliability of the position information. However, the velocity estimation unit 713 in this embodiment is not limited to the Kalman filter.

[0093] The speed estimation unit 713 may be a Bayes filter (such as a particle filter) other than the Kalman filter, or may be an Infinite Impulse Response Filter (IIR filter) or a Finite Impulse Response Filter (FIR filter).

[0094] In the case of an IIR filter, the feedback gain may be changed instead of changing the observation noise, and in the case of an FIR filter, the weighting may be changed instead of changing the observation noise.

[0095] In addition, the above description has been given of a configuration in which previously acquired statistical data such as the basic observation noise table 720 and the corrected observation noise table 721 are stored as tables and used. However, the storage format of the statistical data in this embodiment is not limited to tables. The statistical data may be modeled and expressed as a function formula.

[0096] By carrying out the above-described processing, it is possible to suppress a decrease in the accuracy of speed estimation caused by detected position errors that arise due to factors that cause position estimation errors on the road surface, such as road markings.

[0097] Next, an example of the effect of this embodiment will be described with reference to Fig. 9 and Fig. 10. First, Fig. 9(A) to (F) are diagrams for explaining an example in which an error occurs in the detection position of an observed vehicle due to the influence of road markings.

[0098] Figures 9(A), (B), and (C) show the behavior of the observed vehicle in world coordinates, and Figures 9(A), (B), and (C) show the position of the observed vehicle at each time. The observed vehicle 130 moves in the positive direction of position X. Figures 9(D), (E), and (F) show the behavior of the observed vehicle in camera coordinates.

[0099] Fig. 9(D) shows the behavior of the observed vehicle at the same time as Fig. 9(A), Fig. 8(E) shows the same time as Fig. 9(B), and Fig. 9(F) shows the behavior of the observed vehicle at the same time as Fig. 9(C). Bounding boxes 511 in Fig. 9(D), (E), and (F) are the detection results of the observed vehicle, and position 512 is the position of the observed vehicle in camera coordinates. In Fig. 9(D), position 512 of the observed vehicle is near the underside of observed vehicle 130.

[0100] On the other hand, when road markings are present around the observed vehicle, a bounding box 511 including the road markings (crosswalk 801) and the observed vehicle may be detected as shown in Fig. 9(E). In this case, the detection position 512 in the camera coordinates is significantly shifted from the underside of the observed vehicle.

[0101] When the observed vehicle in the camera coordinate system is subjected to coordinate conversion, the detected position 512 of the observed vehicle in world coordinates different from the actual object position is calculated as shown in Fig. 9(B). In this way, the detected position of the observed vehicle may deviate from the desired position due to the influence of road markings.

[0102] Next, Fig. 10(A)-(I) are graphs for explaining the effect of this embodiment on the influence of detection position error of the observed vehicle due to road markings. First, the actual position X, speed V, intersection entry grace time Ty of the observed vehicle 130, and the possibility of the notification target vehicle 120 turning right will be explained using Fig. 10(A), (B), and (C).

[0103] Fig. 10(A) is a graph showing the correct position of the observed vehicle 130 in world coordinates, with the horizontal axis representing time and the vertical axis representing the correct speed of the observed vehicle 130 in world coordinates. As shown in Fig. 10(B), the observed vehicle 130 moves at a constant speed.

[0104] 10C is a graph in which the horizontal axis indicates time and the vertical axis indicates the grace period until the observed vehicle enters the intersection. If the target vehicle 120 can turn right with the grace period Ty1, the target vehicle 120 can turn right up to time T4, but cannot turn right after time T4.

[0105] Next, an example will be described using Figures 10(D), (E), and (F) where detection position errors caused by road surface position estimation error factors such as road markings are not taken into consideration. Figure 10(D) is a graph in which the horizontal axis is time and the vertical axis is the detection position of the observed vehicle in world coordinates. In this example, an error in the detection position of the observed vehicle occurs due to the influence of road markings, and the detection position of the observed vehicle in world coordinates is almost constant from time T1 to T3.

[0106] Fig. 10(E) is a graph showing the estimated speed of the observed vehicle (before measures were taken: when the factors of position estimation error on the road surface were not considered) with the horizontal axis being time and the vertical axis being world coordinates. The estimated speed fluctuates significantly from time T1 to T3 due to the influence of road markings.

[0107] Fig. 10(F) is a graph showing the time on the horizontal axis and the estimated grace period on the vertical axis, in which no measures are taken against the road surface position estimation error factors. The estimated grace period also fluctuates greatly due to the large fluctuations in the estimated speed as shown in Fig. 10(E).

[0108] As a result, in this example, the grace period Ty1 falls short at time T2, and it is determined that a right turn is not possible after T2. As a result, the notification target vehicle 120 is unable to make a right turn even when the timing is actually right.

[0109] Next, an example will be described with reference to Figures 10(G), (H), and (I) in which the detection position error caused by road markings and other factors that cause position estimation errors on the road surface is taken into consideration, as in this embodiment. Figure 10(G) is a graph in which the horizontal axis is time and the vertical axis is the detection position of the observed vehicle in world coordinates, similar to Figure 10(D). This detection position is the same in this embodiment as in Figure 10(D).

[0110] Fig. 10(H) is a graph of the estimated speed of the observed vehicle 130 in this embodiment, with the horizontal axis representing time and the vertical axis representing world coordinates. It can be seen that the influence of road markings is reduced from time T1 to T3 compared to Fig. 10(E). This is because, between times T1 and T3, it was determined that there was an influence of position estimation error factors, and as a result, the observation noise was increased, resulting in the speed estimate value before T1 being adopted at a large value.

[0111] Fig. 10(I) is a graph in this embodiment, with the horizontal axis being time and the vertical axis being the estimated grace period. As shown in Fig. 10(H), the fluctuation in the estimated speed is suppressed compared to Fig. 10(E). As a result, as shown in Fig. 10(i), the fluctuation in the estimated grace period is also suppressed. As a result, in this example, it is determined that a right turn is permitted up until time T4.

[0112] In this way, in this embodiment, it is possible to suppress a decrease in the speed estimation accuracy caused by a detected position error that occurs due to a position estimation error factor on the road surface such as a road marking, etc. This also reduces an error in the entry grace period of the observed vehicle into the intersection, and reduces cases in which the notification target vehicle 120 determines that it cannot turn right even when it is actually able to do so, and vice versa.

[0113] In this embodiment, this can be realized by using the basic observation noise table 720 and the corrected observation noise table 721. This table data is acquired in advance under various conditions, and a table is selected according to the situation after the imaging unit is installed. Therefore, it is desirable to prepare the table data before installing the imaging unit.

[0114] By doing so, it is possible to suppress the decline in speed estimation accuracy caused by detection position errors arising from position estimation error factors on the road surface, such as road markings, without measuring the reliability of the detection position of the observed vehicle after installing the imaging unit.

[0115] In this way, according to this embodiment, it is possible to provide a moving object tracking device that makes it possible to suppress a decrease in speed estimation accuracy due to detection position errors that occur due to position estimation error factors on the road surface, such as road markings, without having to measure the reliability of the object's detection position after installing the imaging unit.

[0116] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The above embodiments include the following combinations.

[0117] (Configuration 1) An information processing device comprising: an image acquisition means for acquiring an image; an object position detection means for detecting an object position on the image based on the image; a coordinate conversion means for converting the object position on the image into an object position in world coordinates; a reliability calculation means for calculating a reliability of the object position in the world coordinates; an error factor existence determination means for determining whether the object position detected by the object position detection means is affected by a position estimation error factor on a road surface; a reliability change means for changing the reliability of the object position depending on a determination result of the error factor existence determination means; and a speed estimation means for estimating the speed of an object using a past history of the object position in the world coordinates and a past history of the reliability after the change.

[0118] (Configuration 2) The information processing device according to configuration 1, wherein the position estimation error factors include road markings.

[0119] (Configuration 3) An information processing device as described in configuration 1 or 2, further comprising a judgment area determination unit which determines in advance a judgment area within which it is determined whether or not there is an influence of the position estimation error factor on a road surface, and the error factor existence determination means makes the determination based on whether or not the object position detected by the object position detection means is within the range of the judgment area.

[0120] (Configuration 4) The information processing device according to configuration 3, wherein the judgment area can be set by a user through a setting value.

[0121] (Configuration 5) The information processing device according to configuration 3 or 4, further comprising a road marking detection unit that detects road markings from the image, and calculates the judgment area in advance in accordance with a detection result by the road marking detection unit.

[0122] (Configuration 6) The information processing device according to any one of configurations 1 to 5, wherein the object position detection means generates a bounding box, and the error factor existence determination means makes the determination based on an amount of change in size of the bounding box.

[0123] (Configuration 7) An information processing device according to any one of configurations 1 to 6, characterized in that the object position detection means generates a bounding box, and the error factor existence determination means makes the determination based on an amount of change in a predetermined color area within the bounding box in the image.

[0124] (Method) An information processing method comprising: an image acquisition step of acquiring an image; an object position detection step of detecting an object position on the image based on the image; a coordinate transformation step of transforming the object position on the image into an object position in world coordinates; a reliability calculation step of calculating a reliability for the object position in the world coordinates; an error factor existence determination step of determining whether the object position detected in the object position detection step is affected by a position estimation error factor on the road surface; a reliability changing step of changing the reliability of the object position in accordance with the determination result of the error factor existence determination step; and a speed estimation step of estimating the speed of an object using a past history of the object position in the world coordinates and a past history of the reliability after the change.

[0125] (Program) A computer program for controlling each means of the information processing device according to any one of configurations 1 to 7 by a computer.

[0126] In order to realize a part or all of the control in this embodiment, a computer program that realizes the functions of the above-mentioned embodiment may be supplied to an information processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the information processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0127] 100, 101: Observation equipment 110: Integrated equipment 120: Abandoned vehicle 130, 131: Vehicle to be observed 140: Collision caution area

Claims

1. image acquisition means for acquiring an image; an object position detection means for detecting an object position on the image based on the image; a coordinate conversion means for converting the object position on the image into an object position in world coordinates; a reliability calculation means for calculating a reliability of the object position in the world coordinates; an error factor presence determination means for determining whether or not the object position detected by the object position detection means is affected by a position estimation error factor on a road surface; a reliability changing means for changing the reliability of the object position in accordance with a result of the determination by the error factor existence determining means; and a velocity estimation means for estimating a velocity of the object using a past history of the object position in the world coordinates and a past history of the changed reliability.

2. 2. The information processing apparatus according to claim 1, wherein the position estimation error factors include road markings.

3. a judgment area determining means for determining in advance a judgment area for determining whether or not the vehicle is affected by the position estimation error factors on a road surface; 2. The information processing apparatus according to claim 1, wherein said error factor presence determining means makes said determination based on whether or not said object position detected by said object position detecting means is within said determination area.

4. 4. The information processing apparatus according to claim 3, wherein the determination area can be set by a user using a setting value.

5. 4. The information processing apparatus according to claim 3, further comprising road marking detection means for detecting road markings from the image, and calculating the judgment area in advance in accordance with a result of detection by the road marking detection means.

6. 2. The information processing apparatus according to claim 1, wherein the object position detection means generates a bounding box, and the error factor presence determination means makes the determination based on an amount of change in size of the bounding box.

7. 2. The information processing apparatus according to claim 1, wherein the object position detection means generates a bounding box, and the error factor presence determination means makes the determination based on an amount of change in a predetermined color area within the bounding box in the image.

8. An information processing method executed by one or more processors, comprising: an image acquisition step of acquiring an image; an object position detection step of detecting an object position on the image based on the image; a coordinate transformation step of transforming the object position on the image into an object position in world coordinates; a reliability calculation step of calculating a reliability of the object position in the world coordinates; an error factor presence determination step of determining whether or not the object position detected in the object position detection step is affected by a position estimation error factor on a road surface; a reliability changing step of changing the reliability of the object position in accordance with a result of the error factor existence determining step; a velocity estimating step of estimating the velocity of the object using a past history of the object position in the world coordinates and a past history of the changed reliability.

9. A computer program for controlling each unit of the information processing apparatus according to any one of claims 1 to 7 by a computer.

Images