MOBILE BODY DETECTION DEVICE, MOBILE BODY DETECTION METHOD, AND PROGRAM

Abstract

A moving body detection device 10 comprises: a feature point extraction unit 11 for extracting a feature point from image data from an imaging device of a vehicle; an optical flow specifying unit 12 for specifying an optical flow for each feature point; a projection processing unit 13 for projecting the optical flow onto a coordinate system of a real space; a background flow specifying unit 14 for specifying a background flow from the optical flow after projection; a correction processing unit 15 for correcting the position of a start point of the optical flow after projection on the basis of a vector of the background flow, and removing an optical flow having a length equal to or shorter than a predetermined range; a clustering unit 16 for performing clustering on an optical flow, which has not been removed, on the basis of a position of the optical flow; and a moving body identification unit 17 for identifying a moving body in the image data from the clustering result.

Description

[Technical Field]

[0001] The present disclosure relates to a moving object detection device and a moving object detection method for detecting a moving object from an image captured by an imaging device, and further relates to a program for realizing these. Mu Regarding. [Background technology]

[0002] In recent years, in order to improve the safety of a vehicle and its surroundings, technologies have been proposed that detect moving objects around the vehicle based on image data from a camera attached to the vehicle and notify the driver of the approach of the moving object (see, for example, Patent Documents 1 and 2).

[0003] Specifically, both Patent Documents 1 and 2 disclose a moving object detection device mounted on a vehicle. The moving object detection devices disclosed in Patent Documents 1 and 2 first extract feature points from each image data output from a camera on a frame-by-frame basis, compare the extracted feature points between frames, and generate an optical flow connecting the feature points.

[0004] Next, the moving object detection devices disclosed in Patent Documents 1 and 2 select an optical flow that satisfies set conditions. There are two set conditions at this time: one is that an extension of the optical flow intersects with a vanishing point within a predetermined error range; the other is that, in the extension of the optical flow, one end point of the optical flow is set as an external division point, and the external division ratio of the line segment connecting the other end point of the optical flow to the vanishing point is equal within a predetermined error range. Then, the moving object detection device disclosed in Patent Document 1 detects a moving object from the selected optical flow.

[0005] The moving object detection devices disclosed in Patent Documents 1 and 2 can detect moving objects around the vehicle, so for example, if the moving object is detected based on images from a rear camera that captures the area behind the vehicle, the driver will be able to recognize vehicles located behind the vehicle. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-97126 [Patent Document 2] Japanese Patent Application Laid-Open No. 2011-43922 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the moving object detection devices disclosed in Patent Documents 1 and 2 detect moving objects by selecting optical flows that intersect at a single vanishing point. Therefore, the moving object detection devices disclosed in Patent Documents 1 and 2 have a problem in that it is difficult to detect moving objects that are running parallel to the host vehicle. In addition, the moving object detection devices disclosed in Patent Documents 1 and 2 also have a problem in that they erroneously detect objects that are part of the background, such as road signs, parked vehicles, and billboards, because they move relative to the host vehicle.

[0008] An example of an objective of the present disclosure is to solve the above problems and improve the detection accuracy of moving objects. [Means for solving the problem]

[0009] In order to achieve the above object, a moving body detection device according to one aspect of the present disclosure includes: a feature point extraction unit that extracts feature points from image data output at set intervals by an imaging device attached to the vehicle; an optical flow determination unit that compares the latest image data with image data output earlier than the latest image data and determines, for each of the extracted feature points, an optical flow of the feature point; a projection processing unit that projects the identified optical flow onto a coordinate system in real space with the imaging device as the origin; a background flow identification unit that identifies, from the projected optical flows, an optical flow of an object that constitutes a background as a background flow; a correction processing unit that corrects the positions of the start points of all the projected optical flows based on the vectors of the background flow, and removes the optical flows whose lengths become less than a predetermined range as a result of the correction; a clustering unit that performs clustering on the optical flows that have not been removed based on their positions; a moving object identifying unit that identifies a moving object in the image data using a result of the clustering; The present invention is characterized by the following features.

[0010] In order to achieve the above object, a moving object detection method according to one aspect of the present disclosure includes: a feature point extraction step of extracting feature points from image data output at set intervals by an imaging device attached to the vehicle; an optical flow determination step of comparing the latest image data with image data output earlier than the latest image data, and determining an optical flow of each of the extracted feature points; a projection processing step of projecting the identified optical flow onto a coordinate system in real space with the imaging device as the origin; a background flow identification step of identifying an optical flow of an object constituting a background from the projected optical flows as a background flow; a correction processing step of correcting the positions of the start points of all the projected optical flows based on the vectors of the background flow, and removing the optical flows whose lengths fall within a predetermined range as a result of the correction; a clustering step of clustering the optical flows that have not been removed based on their positions; a moving object identifying step of identifying a moving object in the image data using a result of the clustering; The present invention is characterized by having the following.

[0011] Furthermore, in order to achieve the above object, in one aspect of the present disclosure, program teeth, On the computer, a feature point extraction step of extracting feature points from image data output at set intervals by an imaging device attached to the vehicle; an optical flow determination step of comparing the latest image data with image data output earlier than the latest image data, and determining an optical flow of each of the extracted feature points; a projection processing step of projecting the identified optical flow onto a coordinate system in real space with the imaging device as the origin; a background flow identification step of identifying an optical flow of an object constituting a background from the projected optical flows as a background flow; a correction processing step of correcting the positions of the start points of all the projected optical flows based on the vectors of the background flow, and removing the optical flows whose lengths fall within a predetermined range as a result of the correction; a clustering step of clustering the optical flows that have not been removed based on their positions; a moving object identifying step of identifying a moving object in the image data using a result of the clustering; Run Ru, It is characterized by: [Effects of the Invention]

[0012] As described above, according to the present disclosure, it is possible to improve the detection accuracy of a moving object. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a moving body detection device according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing a specific configuration of a moving body detection device according to an embodiment of the present invention. [Figure 3]FIG. 3 is a diagram for explaining the process of identifying the optical flow of a feature point according to the embodiment. [Figure 4] FIG. 4 is a diagram for explaining the optical flow projection process according to the embodiment. [Figure 5] FIG. 5 is a diagram for explaining the process of identifying a background flow according to the embodiment. [Figure 6] FIG. 6 is a diagram for explaining the correction process according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating the optical flow clustering process according to the embodiment. [Figure 8] FIG. 8 is a diagram for explaining the calculation process of the relative velocity of the moving object in the embodiment. [Figure 9] FIG. 9 is a flowchart showing the operation of the moving body detection device according to the embodiment. [Figure 10] FIG. 10 is a block diagram illustrating an example of a computer that realizes a moving body detection device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] (Embodiment) Hereinafter, a moving body detection device according to an embodiment will be described with reference to FIGS.

[0015] [Device configuration] First, the form of implementation In a positive manner The schematic configuration of the moving object detection device in the embodiment will be described with reference to FIG. In a positive manner 1 is a diagram showing a schematic configuration of a moving body detection device in FIG.

[0016] 1, a moving object detection device 10 is a device for detecting moving objects from image data output by an imaging device attached to a vehicle. As shown in FIG. 1, the moving object detection device 10 includes a feature point extraction unit 11, an optical flow identification unit 12, a projection processing unit 13, a background flow identification unit 14, a correction processing unit 15, a clustering unit 16, and a moving object identification unit 17.

[0017] The feature point extraction unit 11 extracts feature points from image data output at set intervals by an imaging device attached to a vehicle. The optical flow determination unit 12 compares the latest image data with image data output earlier than that, and determines the optical flow of each extracted feature point.

[0018] The projection processing unit 13 projects the identified optical flows onto a coordinate system in real space with the image capture device as the origin. The background flow identification unit 14 identifies, from the projected optical flows, the optical flows of objects that constitute the background as background flows.

[0019] The correction processing unit 15 corrects the positions of the starting points of all projected optical flows based on the vectors of the background flow, and removes optical flows whose lengths fall below a predetermined range as a result of the correction.

[0020] The clustering unit 16 performs clustering based on the positions of the optical flows that have not been removed by the correction processing unit 15. The moving object identifying unit 17 uses the results of the clustering to identify moving objects in the image data.

[0021] In this manner, in the embodiment, the position of the starting point of the optical flow of the feature point is corrected based on the background flow vector, thereby removing the background flow. Then, based on the clustering results of the optical flow that has not been removed, the moving object is identified. Therefore, according to the embodiment, even a moving object traveling parallel to a vehicle can be detected, thereby improving the accuracy of moving object detection.

[0022] Next, the configuration of the moving body detection device in this embodiment will be described in more detail with reference to Fig. 2. Fig. 2 is a block diagram showing the specific configuration of the moving body detection device in this embodiment of the present invention.

[0023] 2, the moving object detection device 10 in the embodiment is connected to the imaging device 30 and is mounted on a vehicle together with the imaging device 30. The imaging device 30 is a camera equipped with a solid-state imaging element, and when capturing an image, outputs image data to the moving object detection device 10. In the embodiment, the imaging device 30 outputs image data continuously at a set time interval (frame rate).

[0024] Also, as shown in FIG. 2, in this embodiment, the moving body detection device 10 includes an image acquisition unit 18, a tracking unit 19, and a moving body speed calculation unit 20 in addition to the feature point extraction unit 11, optical flow identification unit 12, projection processing unit 13, background flow identification unit 14, correction processing unit 15, clustering unit 16, and moving body identification unit 17 described above.

[0025] The image acquisition unit 18 acquires image data to be processed from the imaging device 30. Specifically, since the imaging device 30 outputs image data of a color image, the image acquisition unit 18 first performs image conversion on the output image data to convert the color image into a grayscale image. Next, the image acquisition unit 18 performs distortion correction on the image data as necessary, and then inputs the corrected image data to the feature point extraction unit 11 and the optical flow determination unit 12. Note that distortion correction is performed, for example, when the photographic lens attached to the imaging device 30 is a fisheye lens or the like.

[0026] In this embodiment, the feature point extraction unit 11 first calculates an unbiased variance of luminance change using image data for a preset number of frames (e.g., 10 frames), and compares the calculated value with a threshold to determine whether each pixel corresponds to a moving pixel or a still pixel.The feature point extraction unit 11 then determines an area containing pixels determined to be moving pixels as a moving area where a moving object exists.Next, the feature point extraction unit 11 extracts feature points in the moving area of the image data of the current frame using, for example, a general FAST algorithm.

[0027] In the embodiment, for example, the optical flow identification unit 12 first compares the feature points of the previous frame immediately before the current frame with the feature points of the current frame, and links the corresponding feature points together.

[0028] 3 is a diagram illustrating the process of identifying the optical flow of feature points in the embodiment. In the example of FIG. 3, the optical flow identification unit 12 links feature point 1 of the immediately previous frame with feature point 3 of the current frame, and links feature point 2 of the immediately previous frame with feature point 4 of the current frame. The optical flow identification unit 12 then connects the linked feature points together to identify the optical flow.

[0029] Specifically, the optical flow determination unit 12 calculates the similarity between feature points and associates feature points whose calculated similarity is equal to or greater than a threshold. The similarity in this case may be a Hamming distance calculated from a binary code generated by the Rotated BRIEF method.

[0030] Furthermore, in order to shorten processing time, the optical flow determination unit 12 can divide the image, group each area obtained by the division and its surrounding areas, and then link feature points and determine the optical flow for each group.

[0031] In this embodiment, the projection processing unit 13 projects the identified optical flow onto a world coordinate system in real space with the imaging device as the origin, using camera parameters (e.g., the installation position and installation angle of the imaging device 30, lens distortion coefficient, etc.).

[0032] 4 is a diagram for explaining the projection processing of the optical flow in the embodiment. Specifically, as shown in FIG. 4, the projection processing unit 13 projects the coordinates (u, v) of the optical flow in the image coordinate system onto the world coordinate system (XYZ), which is the coordinate system of real space. However, since both the vehicle (hereinafter also referred to as "host vehicle") on which the moving object detection device 10 is mounted and the moving object exist on the ground, Y is always set to 0 (zero) in the world coordinate system (Y=0). Therefore, the coordinates after projection become (X, 0, Z).

[0033] The projection process will now be described in detail. First, various parameters are defined as follows: (u,v): Coordinates in the image coordinate system to be converted to the world coordinate system x0: x-axis coordinate before distortion correction in the camera coordinate system y0: y-axis coordinate before distortion correction in the camera coordinate system x: the x-axis coordinate after distortion correction in the camera coordinate system y: y-axis coordinate after distortion correction in the camera coordinate system o x :X-axis coordinate of the optical center in the image coordinate system o y : Y-axis coordinate of the optical center in the image coordinate system f w :Focal length in pixels (horizontal) f w = (number of horizontal light receiving elements / horizontal length of solid-state image sensor) x focal length [mm] f h :Focal length in pixels (vertical direction) f h = (number of vertical light receiving elements / vertical length of solid-state image sensor) x focal length [mm] k1, k2, k3, k4, k5, k6, p1, p2: camera distortion coefficients ('u,'v): Corrected coordinates in the image coordinate system H: Bird's-eye view transformation matrix 'X,'Z: Normalized coordinates in the reciprocal of 'Y' in the world coordinate system 'Y:'X,'Z denormalization coefficient X: X-axis coordinate in the world coordinate system Y: Y-axis coordinate in the world coordinate system Z: Z-axis coordinate in the world coordinate system

[0034] Of the above parameters, x0 and y0 are expressed by the following equation 1. Then, the projection processing unit 13 executes the calculation using the following equation 2 multiple times (for example, five times) to update x and y.

[0035]

number

[0036]

number

[0037] Next, the projection processing unit 13 applies the calculated x and y to the following equation 3 to calculate ('u,'v). Then, the projection processing unit 13 applies the calculated ('u,'v) to the following equations 4 and 5 to calculate X and Z. In this way, the coordinates on the image are projected onto the world coordinate system.

[0038]

number

[0039]

number

[0040]

number

[0041] Moreover, the bird's-eye view transformation matrix H shown in the above equation 4 is calculated by the following equation 6. In the following equation 6, M is an internal parameter matrix of the image capture device 30, and is calculated by the following equation 7. Moreover, E is an external parameter matrix of the image capture device 30, and is calculated by the following equation 8.

[0042]

number

[0043]

number

[0044]

number

[0045] In the above formula 8, r 11 , r 13 , r 21 , r 23 , r 31 , and r 33 are the elements of the first and third columns of the rotation matrix R shown in the following equation 9. Also, in the above equation 8, t x , t y , t z indicates an element of the translation vector t shown in the following equation 10.

[0046]

number

[0047] In the above equation 9, roll indicates the rotation angle [rad] due to the roll of the imaging device 30, pitch indicates the rotation angle [rad] due to the pitching of the imaging device 30, and yaw indicates the rotation angle [rad] due to the yawing of the imaging device 30.

[0048]

number

[0049] In this embodiment, the background flow identification unit 14 identifies, as a background flow, the optical flow of an object that constitutes the background from among the optical flows projected onto the world coordinate system, based on the angle and length.

[0050] 5 is a diagram for explaining the process of identifying a background flow in an embodiment. As shown in FIG. 5, the direction of the background flow changes when the moving direction of the host vehicle changes. In the example of FIG. 5, the background flow when the imaging device 30 is installed facing forward of the host vehicle is indicated by an arrow.

[0051] The top diagram in Figure 5 shows the background flow when the host vehicle is traveling straight. The middle diagram in Figure 5 shows the background flow when the host vehicle is turning right at the maximum turning angle. The bottom diagram in Figure 5 shows the background flow when the host vehicle is turning left at the maximum turning angle. Because there is a limit to the turning angle of the vehicle's tires, the range in which the direction of the background flow changes is within a certain range. Also, as shown in Figure 5, in the world coordinate system, the background flow faces in approximately the same direction and has approximately the same length.

[0052] For this reason, the background flow identification unit 14 generates a histogram with the length of the optical flow as the class on the horizontal axis, and identifies the background flow based on the generated histogram. Specifically, the background flow identification unit 14 first extracts optical flows whose directions are within a certain range from all optical flows, and generates a histogram using the extracted optical flows. Then, the background flow identification unit 14 calculates a vector average using, for example, the optical flows belonging to the bin with the highest frequency and its adjacent bins, and identifies the calculated vector average as the background flow.

[0053] The background flow is a vector, and its direction is opposite to the direction of movement of the vehicle, but its amount of movement is the same as the amount of movement of the vehicle. Therefore, in this embodiment, the correction processing unit 15 calculates the direction and amount of movement of the vehicle based on the vector of the background flow. Then, the correction processing unit 15 performs correction by moving the positions of the starting points of all projected optical flows in the direction opposite to the direction of movement of the vehicle by the amount of movement of the vehicle. Specifically, since the background flow is a vector, the correction processing unit 15 adds the vector of the background flow to each optical flow. Figure 6 is a diagram for explaining the correction processing in this embodiment.

[0054] In Fig. 6, 31 indicates a vehicle other than the subject vehicle, and 32 indicates a roadside tree (an object constituting the background). Also in Fig. 6, the solid line indicates the image in the current frame, and the dashed line indicates the frame immediately before the current frame. Furthermore, in Fig. 6, the arrows indicate the optical flow projected onto the world coordinate system. The upper diagram in Fig. 6 shows the image before correction, and the lower diagram shows the image after correction.

[0055] As shown in Fig. 6, since roadside trees 32 are objects that make up the background, the background flow coincides with the optical flow of roadside trees 32. Therefore, when correction processing unit 15 executes correction processing, the amount of movement of roadside trees 32 on the image becomes zero, as shown in the lower diagram of Fig. 6, and the length of their optical flow also becomes zero.

[0056] For this reason, the correction processing unit 15 removes from the optical flow group any optical flow whose length has become zero due to the correction. Note that the optical flows to be removed may include optical flows whose length is not strictly zero but is substantially zero. The range in which the length becomes substantially zero is set appropriately in advance.

[0057] In this embodiment, the clustering unit 16 performs clustering using, for example, an x-mean algorithm on the optical flows that have not been removed by the correction processing unit 15. Fig. 7 is a diagram for explaining the clustering process of the optical flows in this embodiment.

[0058] Specifically, as shown in the upper diagram of Fig. 7, the clustering unit 16 first identifies the position of each optical flow that has not been removed by the correction processing unit 15. Then, as shown in the middle diagram of Fig. 7, the clustering unit 16 calculates the distance between the optical flows based on the identified positions, and performs clustering based on the calculated distance.

[0059] Next, for each cluster generated by clustering, the clustering unit 16 identifies optical flows whose length, position (coordinates on the screen), and orientation are significantly different from the average values of the optical flows in that cluster. Since the identified optical flows are unlikely to originate from the same moving object, the clustering unit 16 removes the identified optical flows from the original cluster and generates a new cluster, as shown in the lower diagram of Figure 7.

[0060] Furthermore, after clustering, if the positional relationship between multiple clusters and the lengths and directions of their optical flows satisfy set conditions, the clustering unit 16 can combine these clusters into one cluster, since the optical flows of these clusters are considered to originate from the same moving object.

[0061] Specifically, the clustering unit 16 first identifies two clusters that are candidates for merging under the conditions that the clusters overlap with each other and that the average length and direction of the optical flow within the clusters are similar between the clusters. Next, the clustering unit 16 compares the direction of the optical flow in the central area (e.g., 20 pixels x 20 pixels) when the centers of the two clusters that are candidates for merging are connected, and if the directions of the optical flows match between the clusters, it merges these candidates for merging.

[0062] The tracking unit 19 determines, for each cluster obtained by the clustering unit 16, whether or not the position of that cluster overlaps with a position predicted from image data other than the latest image data.

[0063] Specifically, the tracking unit 19 compares the cluster to be determined in the current frame with the cluster to be determined in multiple past frames (for example, the past six frames), and determines that they overlap if they match in a certain number of past frames or more.The tracking unit 19 then notifies the moving object identifying unit 17 of the clusters that it has determined to overlap.

[0064] In addition, the tracking unit 19 determines that a cluster in the current frame and a cluster in the past frame match not only when they completely match, but also when they overlap at a certain rate or more. Furthermore, to prevent clusters corresponding to different moving objects from matching, the tracking unit 19 can reduce the area of the cluster to be determined (for example, to 1 / 4) before determining whether they match.

[0065] Furthermore, the tracking unit 19 can calculate the average coordinates of the cluster to be judged between the coordinates of the cluster in the current frame and the coordinates of the cluster in the past frame that match the coordinates of the cluster to be judged, and use the calculated average coordinates as the coordinates of the cluster to be judged.

[0066] Furthermore, if the number of matching past frames is less than a threshold value as a result of comparing the cluster of the current frame with the clusters of multiple past frames, the tracking unit 19 determines that there is a high possibility of erroneous detection, and therefore deletes the cluster to be judged. However, if the deleted cluster was judged to overlap in the previous judgment, the tracking unit 19 can also restore the deleted cluster.

[0067] In this embodiment, the moving object identifying unit 17 identifies a moving object in an image based on clusters determined by the tracking unit 19 to overlap. Specifically, since the optical flows of objects constituting the background have already been removed by the correction process performed by the correction processing unit 15, all clusters determined by the tracking unit 19 to overlap correspond to moving objects. Therefore, for each cluster determined to overlap, the moving object identifying unit 17 identifies a set of feature points that are the basis of the optical flows included in that cluster, and identifies the identified set of feature points as a moving object. Furthermore, since the optical flow of a moving object traveling parallel to the host vehicle does not become zero after correction, the identified moving objects also include vehicles traveling parallel to the host vehicle.

[0068] The moving object identification unit 17 also notifies the vehicle of information about the identified moving object. The notified information includes the moving object's position on the image, moving direction, speed, etc. The speed is calculated by the moving object speed calculation unit 20, which will be described later.

[0069] The moving object speed calculation unit 20 calculates the speed of the moving object using the movement amount of the host vehicle obtained from the background flow vector by the correction processing unit 15. Specifically, the moving object speed calculation unit 20 first calculates the speed of the host vehicle from the movement amount of the host vehicle and the frame rate of the imaging device 30. Next, the moving object speed calculation unit 20 calculates the relative speed of the identified moving object with respect to the host vehicle using the direction and length of the optical flow of the moving object identified by the moving object identification unit 17. Then, the moving object speed calculation unit 20 calculates the speed of the moving object from the calculated relative speed and the speed of the host vehicle.

[0070] FIG. 8 is a diagram illustrating a calculation process of the relative velocity of a moving object in an embodiment. More specifically, as shown in the upper diagram of FIG. 8, the moving object velocity calculation unit 20 first identifies the coordinates of the point of the identified moving object closest to the host vehicle (the bottom right or bottom left point on the screen of the moving object) in each of the current frame and the immediately previous frame. The coordinates identified in the immediately previous frame are set as start coordinates, and the coordinates identified in the current frame are set as end coordinates. The moving object velocity calculation unit 20 can also estimate the start coordinates from the average length and angle of the optical flow of the identified moving object.

[0071] Next, as shown in the middle diagram of Fig. 8, the moving object velocity calculation unit 20 converts the start coordinates and end coordinates into coordinates in the world coordinate system in real space using the camera parameters of the imaging device 30. The coordinate conversion is performed by a process similar to the projection process by the projection processing unit 13. Then, the moving object velocity calculation unit 20 calculates the distance between the start coordinates and end coordinates in real space from each of the converted coordinates, and sets this as the absolute movement distance of the moving object in real space.

[0072] The moving object speed calculation unit 20 then calculates the absolute speed of the moving object from the time from the immediately preceding frame to the current frame and the calculated absolute movement distance, and further calculates the relative speed of the moving object with respect to the host vehicle from the calculated absolute speed of the moving object and the host vehicle speed. The time from the immediately preceding frame to the current frame can be determined from the frame rate. For example, if the frame rate is 10 fps, the time is 0.1 seconds.

[0073] [Device operation] Next, the operation of the mobile body detection device 10 in the embodiment will be described with reference to FIG. 9. FIG. 9 is a flow diagram showing the operation of the mobile body detection device in the embodiment. In the following description, reference will be made to FIGS. 1 to 8 as appropriate. In the embodiment, a mobile body detection method is implemented by operating the mobile body detection device. Therefore, the description of the mobile body detection method in the embodiment will be replaced by the description of the operation of the mobile body detection device below.

[0074] figure 9 As shown in FIG. 1, first, the image acquisition unit 18 acquires image data output from the imaging device 30 for each frame (step A1).

[0075] Specifically, in step A1, the image acquisition unit 18 first converts the acquired image data into a grayscale image, corrects distortion, etc. Then, when the number of frames of the acquired image data reaches a set number, the image acquisition unit 18 passes the acquired image data to the feature point extraction unit 11.

[0076] Next, the feature point extraction unit 11 extracts feature points moving within the image from the image data acquired in step A1 (step A2).

[0077] Specifically, in step A2, the feature point extraction unit 11 calculates an unbiased variance of luminance change using image data of a preset number of frames (e.g., 10 frames) and determines whether each pixel corresponds to a moving pixel or a still pixel. Then, the feature point extraction unit 11 extracts feature points from a moving region where pixels determined to be moving pixels exist.

[0078] Next, the optical flow determination unit 12 compares the latest image data with image data output earlier than that, and determines the optical flow of each extracted feature point (step A3).

[0079] Specifically, in step A3, the optical flow identification unit 12 compares the feature points of the previous frame immediately before the current frame with the feature points of the current frame, links corresponding feature points together, and connects the linked feature points together to identify the optical flow.

[0080] Next, the projection processing unit 13 projects the optical flow identified in step A3 onto a world coordinate system in real space with the image capturing device as the origin, using the camera parameters (step A4).

[0081] Specifically, in step A4, for each identified optical flow, the projection processing unit 13 projects the coordinates (u, v) of the optical flow in the image coordinate system onto the world coordinate system (XYZ), which is the coordinate system of real space, using the above equations 1 to 10.

[0082] Next, the background flow identification unit 14 identifies, from the optical flows projected in step A4, the optical flows of objects that make up the background as background flows (step A5).

[0083] Specifically, in step A5, the background flow identification unit 14 extracts optical flows whose directions are within a certain range from all optical flows, generates a histogram using the extracted optical flows, and calculates a vector average using, for example, the optical flows belonging to the bin with the highest frequency and its two adjacent bins, and identifies the calculated vector average as the background flow.

[0084] Next, the correction processing unit 15 corrects the positions of the starting points of all optical flows projected in step A4 based on the vector of the background flow, and removes optical flows whose lengths fall below a predetermined range as a result of the correction (step A6).

[0085] Specifically, in step A6, the correction processor 15 calculates the direction and amount of movement of the vehicle based on the vector of the background flow. Then, the correction processor 15 corrects the positions of the start points of all projected optical flows by the amount of movement of the vehicle in the direction opposite to the direction of movement of the vehicle. As a result of this correction, the length of the background flow becomes approximately zero.

[0086] Next, the clustering unit 16 performs clustering on the optical flows that were not removed in step A6 based on their positions (step A7).

[0087] Specifically, the clustering unit 16 first calculates the number of clusters in the upper row of FIG. and middle sectionAs shown in the figure, the distances of the optical flows that were not removed by the correction processing unit 15 are calculated, and clustering is performed based on the calculated distances. Next, for each cluster, the clustering unit 16 identifies optical flows whose length, position (coordinates on the screen), and orientation deviate significantly from the average values of the optical flows in that cluster, removes them from the original cluster, and generates a new cluster.

[0088] Next, for each cluster obtained by step A7, the tracking unit 19 determines whether the position of that cluster overlaps with the position predicted from image data other than the latest image data, and notifies the moving body identification unit 17 of the clusters that are determined to overlap (step A8).

[0089] Specifically, in step A8, the tracking unit 19 compares the cluster to be judged in the current frame with the cluster to be judged in multiple past frames (e.g., the past six frames), and determines that they overlap if they match in a certain number of past frames or more.

[0090] Next, the moving object identifying unit 17 identifies a moving object in the image based on the clusters determined to overlap in step A8 (step A9).

[0091] Specifically, in step A9, the moving body identification unit 17 identifies, for each cluster determined to overlap, the set of feature points that are the basis of the optical flow contained in that cluster, and identifies the identified set of feature points as a moving body.

[0092] Next, the moving object speed calculation unit 20 calculates the speed of the moving object using the amount of movement of the host vehicle obtained from the vector of the background flow (step A10).

[0093] Specifically, in step A10, the moving object speed calculation unit 20 first calculates the speed of the host vehicle from the movement amount of the host vehicle calculated in step A6 and the frame rate of the imaging device 30. Next, the moving object speed calculation unit 20 calculates the relative speed of the identified moving object with respect to the host vehicle, using the direction and length of the optical flow of the moving object identified in step A9. Then, the moving object speed calculation unit 20 calculates the speed of the moving object from the calculated relative speed and the speed of the host vehicle.

[0094] Thereafter, the moving body identifying unit 17 notifies the vehicle of the position, moving direction, and speed of the moving body on the image as information about the moving body identified in step A9 (step A11).

[0095] [Effects of the embodiment] As described above, in this embodiment, the position of the starting point of the optical flow of the feature point is corrected based on the background flow vector, thereby removing the background flow. Moving objects are then identified from the remaining optical flows. However, since the optical flows of moving objects running parallel to the vehicle are not removed by the correction, the identified moving objects also include vehicles running parallel to the vehicle. Therefore, according to this embodiment, even moving objects running parallel to the vehicle can be detected, thereby improving the accuracy of moving object detection.

[0096] [program] The program in the embodiment may be a program that causes a computer to execute steps A1 to A11 shown in Fig. 9. By installing and executing this program in a computer, the moving object detection device 10 and the moving object detection method in the embodiment can be realized. In this case, the processor of the computer functions as a feature point extraction unit 11, an optical flow identification unit 12, a projection processing unit 13, a background flow identification unit 14, a correction processing unit 15, a clustering unit 16, a moving object identification unit 17, an image acquisition unit 18, a tracking unit 19, and a moving object speed calculation unit 20, and performs processing.

[0097] Examples of the computer include an in-vehicle computer, a general-purpose PC (Personal Computer), a smartphone, and a tablet terminal device. The program in the embodiment may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of a feature point extraction unit 11, an optical flow identification unit 12, a projection processing unit 13, a background flow identification unit 14, a correction processing unit 15, a clustering unit 16, a moving object identification unit 17, an image acquisition unit 18, a tracking unit 19, and a moving object speed calculation unit 20.

[0098] [Physical configuration] Here, a computer that realizes the moving body detection device 10 by executing the program in the embodiment will be described with reference to Fig. 10. Fig. 10 is a block diagram showing an example of a computer that realizes the moving body detection device in the embodiment.

[0099] 10, a computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0100] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0101] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0102] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0103] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0104] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0105] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0106] Note that the mobile object detection device 10 in the embodiments can be realized not by a computer with a program installed, but by hardware corresponding to each part, such as an electronic circuit. Furthermore, the mobile object detection device 10 may be partially realized by a program and the remaining part by hardware. In the embodiments, the computer is not limited to the computer shown in FIG. 10.

[0107] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 12) described below, but are not limited to the following descriptions.

[0108] (Appendix 1) a feature point extraction unit that extracts feature points from image data output at set intervals by an imaging device attached to the vehicle; an optical flow determination unit that compares the latest image data with image data output earlier than the latest image data and determines, for each of the extracted feature points, an optical flow of the feature point; a projection processing unit that projects the identified optical flow onto a coordinate system in real space with the imaging device as the origin; a background flow identification unit that identifies, from the projected optical flows, an optical flow of an object that constitutes a background as a background flow; a correction processing unit that corrects the positions of the start points of all the projected optical flows based on the vectors of the background flow, and removes the optical flows whose lengths become less than a predetermined range as a result of the correction; a clustering unit that performs clustering on the optical flows that have not been removed based on their positions; a moving object identifying unit that identifies a moving object in the image data using a result of the clustering; A moving object detection device comprising:

[0109] (Appendix 2) 10. The moving object detection device according to claim 1, the correction processing unit calculates the direction and amount of movement of the vehicle based on the vector of the background flow, and corrects the positions of the start points of all the projected optical flows by moving the start points by the amount of movement in a direction opposite to the direction of movement. A moving object detection device characterized by:

[0110] (Appendix 3) 3. The moving object detection device according to claim 2, The system further includes a moving object speed calculation unit that calculates a speed of the moving object using the amount of movement of the vehicle calculated based on the vector of the background flow. A moving object detection device characterized by:

[0111] (Appendix 4) A moving body detection device according to any one of Supplementary Notes 1 to 3, a tracking unit that determines, for each cluster obtained by the clustering, whether or not a position of the cluster overlaps with a position predicted from image data other than the latest image data; The moving object specifying unit, based on the clusters determined to overlap, The aforementioned image data Identifying moving objects within A moving object detection device characterized by:

[0112] (Appendix 5) a feature point extraction step of extracting feature points from image data output at set intervals by an imaging device attached to the vehicle; an optical flow determination step of comparing the latest image data with image data output earlier than the latest image data, and determining an optical flow of each of the extracted feature points; a projection processing step of projecting the identified optical flow onto a coordinate system in real space with the imaging device as the origin; a background flow identification step of identifying an optical flow of an object constituting a background from the projected optical flows as a background flow; a correction processing step of correcting the positions of the start points of all the projected optical flows based on the vectors of the background flow, and removing the optical flows whose lengths fall within a predetermined range as a result of the correction; a clustering step of clustering the optical flows that have not been removed based on their positions; a moving object identifying step of identifying a moving object in the image data using a result of the clustering; A moving body detection method comprising:

[0113] (Appendix 6) 6. The moving object detection method according to claim 5, In the correction processing step, a moving direction and a moving amount of the vehicle are calculated based on the vector of the background flow, and the positions of the start points of all the projected optical flows are moved in a direction opposite to the moving direction by the moving amount, thereby performing correction. A moving object detection method comprising:

[0114] (Appendix 7) 7. The moving object detection method according to claim 6, The method further includes a moving object speed calculation step of calculating a speed of the moving object using the movement amount of the vehicle calculated based on the vector of the background flow. A moving object detection method comprising:

[0115] (Appendix 8) A moving body detection method according to any one of Supplementary Notes 5 to 7, The method further includes a tracking step of determining, for each cluster obtained by the clustering, whether or not a position of the cluster overlaps with a position predicted from image data other than the latest image data, Based on the clusters determined to overlap in the moving object identifying step, The aforementioned image data Identifying moving objects within A moving object detection method comprising:

[0116] (Appendix 9) On the computer, a feature point extraction step of extracting feature points from image data output at set intervals by an imaging device attached to the vehicle; an optical flow determination step of comparing the latest image data with image data output earlier than the latest image data, and determining an optical flow of each of the extracted feature points; a projection processing step of projecting the identified optical flow onto a coordinate system in real space with the imaging device as the origin; a background flow identification step of identifying an optical flow of an object constituting a background from the projected optical flows as a background flow; a correction processing step of correcting the positions of the start points of all the projected optical flows based on the vectors of the background flow, and removing the optical flows whose lengths fall within a predetermined range as a result of the correction; a clustering step of clustering the optical flows that have not been removed based on their positions; a moving object identifying step of identifying a moving object in the image data using a result of the clustering; Run Ru, Program Hmm.

[0117] (Appendix 10) As described in Appendix 9 program And, In the correction processing step, a moving direction and a moving amount of the vehicle are calculated based on the vector of the background flow, and the positions of the start points of all the projected optical flows are moved in a direction opposite to the moving direction by the moving amount, thereby performing correction. Characterized by program .

[0118] (Appendix 11) As described in Appendix 10 program And, The program causes the computer to The method further includes an instruction to execute a moving object speed calculation step of calculating a speed of the moving object using the movement amount of the vehicle calculated based on the vector of the background flow. Characterized by program .

[0119] (Appendix 12) Any of Supplementary Notes 9 to 11 program And, The program causes the computer to The method further includes an instruction to execute a tracking step of determining, for each cluster obtained by the clustering, whether or not a position of the cluster overlaps with a position predicted from image data other than the latest image data; Based on the clusters determined to overlap in the moving object identifying step, The aforementioned image data Identifying moving objects within Characterized by program .

[0120] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0121] This application claims priority based on Japanese Patent Application No. 2022-041142, filed on March 16, 2022, the disclosure of which is incorporated herein in its entirety. [Industrial Applicability]

[0122] As described above, according to the present disclosure, it is possible to improve the accuracy of detecting a moving object. The present disclosure is useful for an information processing device of a vehicle that is required to detect a moving object. [Explanation of symbols]

[0123] 10. Moving object detection device 11 Feature point extraction unit 12 Optical flow determination unit 13 Projection processing unit 14 Background flow identification part 15 Correction processing section 16 Clustering Department 17 Mobile Identification Unit 18 Image acquisition unit 19 Tracking Section 20 Moving object speed calculation section 30 Imaging device 31 vehicles 32 Street trees 110 Computer 111 CPU 112 main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display Device 120 Recording Media 121 Bus

Claims

1. a feature point extraction unit that extracts feature points from image data output at set intervals by an imaging device attached to the vehicle; an optical flow determination unit that compares the latest image data with image data output earlier than the latest image data and determines, for each of the extracted feature points, an optical flow of the feature point; a projection processing unit that projects the identified optical flow onto a coordinate system in real space with the imaging device as the origin; a background flow identification unit that identifies, from the projected optical flows, an optical flow of an object that constitutes a background as a background flow; a correction processing unit that corrects the positions of the start points of all the projected optical flows based on the vectors of the background flow, and removes the optical flows whose lengths become less than a predetermined range as a result of the correction; a clustering unit that performs clustering on the optical flows that have not been removed based on their positions; a moving object identifying unit that identifies a moving object in the image data using a result of the clustering; A moving object detection device comprising:

2. The moving body detection device according to claim 1, the correction processing unit calculates the direction and amount of movement of the vehicle based on the vector of the background flow, and corrects the positions of the start points of all the projected optical flows by moving the start points in the direction opposite to the direction of movement by the amount of movement. A moving object detection device characterized by:

3. The moving body detection device according to claim 2, The system further includes a moving object speed calculation unit that calculates a speed of the moving object using the amount of movement of the vehicle calculated based on the vector of the background flow. A moving object detection device characterized by:

4. The moving body detection device according to claim 1, a tracking unit that determines, for each cluster obtained by the clustering, whether or not a position of the cluster overlaps with a position predicted from image data other than the latest image data; the moving object identifying unit identifies a moving object in the image data based on the clusters determined to overlap; A moving object detection device characterized by:

5. Extracting feature points from image data output at set intervals by an imaging device attached to the vehicle, comparing the latest image data with the image data output earlier than the latest image data, and identifying an optical flow of each of the extracted feature points; projecting the identified optical flow onto a coordinate system in real space with the imaging device as the origin; Among the projected optical flows, an optical flow of an object constituting a background is identified as a background flow; correcting the positions of the start points of all the projected optical flows based on the vectors of the background flow, and removing the optical flows whose lengths are within a predetermined range as a result of the correction; Clustering the optical flows that have not been removed based on their positions; using the clustering results to identify moving objects in the image data; A moving object detection method comprising:

6. The moving body detection method according to claim 5, In the correction, a moving direction and a moving amount of the vehicle are calculated based on the vector of the background flow, and the positions of the starting points of all the projected optical flows are moved in a direction opposite to the moving direction by the moving amount, thereby performing the correction. A moving object detection method comprising:

7. 7. A moving body detection method according to claim 6, further calculating a speed of the moving object using the amount of movement of the vehicle calculated based on the vector of the background flow; A moving object detection method comprising:

8. The moving body detection method according to claim 5, Furthermore, for each cluster obtained by the clustering, it is determined whether or not the position of the cluster overlaps with a position predicted from image data other than the latest image data; identifying the moving object in the image data based on the clusters determined to overlap in the identification of the moving object; A moving object detection method comprising:

9. On the computer, Extracting feature points from image data output at set intervals by an imaging device attached to the vehicle, comparing the latest image data with the image data output earlier than the latest image data, and identifying the optical flow of each of the extracted feature points; projecting the identified optical flow onto a coordinate system in real space with the imaging device as the origin; Among the projected optical flows, an optical flow of an object constituting a background is identified as a background flow; correcting the positions of the start points of all the projected optical flows based on the vectors of the background flow, and removing the optical flows whose lengths are within a predetermined range as a result of the correction; Clustering the optical flows that have not been removed based on their positions; Identifying a moving object in the image data using the clustering result. program.

10. 10. The program according to claim 9, In the correction, a moving direction and a moving amount of the vehicle are calculated based on the vector of the background flow, and the positions of the starting points of all the projected optical flows are moved in a direction opposite to the moving direction by the moving amount, thereby performing the correction. A program characterized by:

11. The program according to claim 10, The program causes the computer to The method further includes an instruction to calculate a speed of the moving object using the amount of movement of the vehicle calculated based on the vector of the background flow. A program characterized by:

12. 10. The program according to claim 9, The program causes the computer to The method further includes an instruction to determine, for each cluster obtained by the clustering, whether or not the position of the cluster overlaps with a position predicted from image data other than the latest image data, identifying the moving object in the image data based on the clusters determined to overlap in the identification of the moving object; A program characterized by:

Images