Object recognition method and object recognition device
The method uses time-series ambient images to track horizontal position changes of feature points in row images, addressing the challenge of hidden objects in object detection systems and enhancing the recognition of partially obscured moving objects.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- NISSAN MOTOR CO LTD
- Filing Date
- 2023-05-26
- Publication Date
- 2026-06-30
Smart Images
Figure 0007882429000001 
Figure 0007882429000002 
Figure 0007882429000003
Abstract
Description
Technical Field
[0001] The present invention relates to a moving object recognition method and a moving object recognition device.
Background Art
[0002] In the object detection device of Patent Document 1, within the captured image, an optical flow regarding the time-series moving direction of each pixel is calculated, the calculation results are grouped, and from the grouped calculation results, regions of stationary three-dimensional objects and regions of moving objects are estimated. The object detection device detects the boundary between the estimated regions of stationary three-dimensional objects and moving objects, and detects a protruding object within the image of the moving object region.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] Near the boundary of two objects whose parts overlap front and back in the captured image, the textures of both objects are continuous without a gap. When the textures of both objects are continuous, it becomes difficult to correctly track the optical flow of each object from the captured image. The object of the present invention is to preferably recognize a moving object on the back side partially hidden by the object in front.
Means for Solving the Problems
[0005] A moving object recognition method and apparatus according to one aspect of the present invention, which solves the above-mentioned problems, acquires multiple time-series ambient images taken multiple times in a time-series manner, and recognizes moving objects in the surroundings from the multiple time-series ambient images using a computer. In this method and apparatus, row images at predetermined vertical positions are extracted from each of the multiple time-series ambient images, and a time-series image is generated by arranging the extracted row images vertically in chronological order. Furthermore, the moving object is recognized based on the difference in the horizontal position of feature points where the feature quantities within the row images change between the row images of the time-series image. [Effects of the Invention]
[0006] According to the present invention, a moving object in the background that is partially hidden by an object in the foreground can be effectively recognized. [Brief explanation of the drawing]
[0007] [Figure 1] Figure 1 shows a vehicle equipped with a moving object recognition device according to the first embodiment. [Figure 2] Figure 2 shows an example of the time-series image generation process. [Figure 3] Figure 3 shows an example of the feature image generation process. [Figure 4] Figure 4 shows the approximate linear components of the set of feature points. [Figure 5] Figure 5 shows the distribution of the slope and perpendicular length of the linear component of the feature points. [Figure 6] Figure 6 shows the assignment of moving feature points to objects in the surrounding image. [Figure 7] Figure 7 is a flowchart of an example of a procedure for recognizing a moving object. [Figure 8] Figure 8 is a flowchart of an example of the procedure for generating time-series images. [Figure 9] Figure 9 is a flowchart of an example procedure for recognizing a moving object. [Figure 10] Figure 10 shows a vehicle equipped with a moving object recognition device according to the second embodiment. [Figure 11]Figure 11 is a flowchart of an example of a procedure for recognizing a moving object. [Modes for carrying out the invention]
[0008] Embodiments of the present invention will be described below with reference to the drawings. In the drawings, identical parts are denoted by the same reference numerals and their descriptions are omitted. [Configuration of a vehicle equipped with a moving object recognition device] As shown in Figure 1, the vehicle 100 is equipped with a moving object recognition device 1, an object detection unit 3, and an imaging unit 5. When the vehicle 100 starts moving, the moving object recognition device 1 uses the distance from the distance measurement point detected by the object detection unit 3 to the object in the vicinity of the vehicle 100 and the image captured by the imaging unit 5 to determine the presence of an object moving around the vehicle 100.
[0009] The object detection unit 3 measures the relative position of objects present around the vehicle 100. The relative position measured by the object detection unit 3 may be, for example, a relative position based on the image acquisition position by the imaging unit 5, which will be described later. The relative position measured by the object detection unit 3 includes information about the distance from the vehicle 100 to the object. When the relative position measured by the object detection unit 3 is based on the image acquisition position, the distance information can be treated as information about the distance from the image acquisition position to the object. The detected relative position is output to the moving object recognition device 1 and stored in memory (not shown). The object detection unit 3 can be any sensor capable of detecting the distance to objects present around the vehicle 100. For example, a stereo camera can be used for the object detection unit 3. LiDAR (Light Detection and Ranging), laser radar, and ToF (Time of Flight) cameras can be used for the object detection unit 3. The imaging unit 5 can be, for example, a camera that takes images. The camera has an image sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 5 is mounted on the vehicle 100 and takes pictures of the surroundings. The image of the vehicle 100's surroundings captured by the imaging unit 5 is output to the moving object recognition device 1 and stored in a memory (not shown).
[0010] The moving object recognition device 1 is mounted on the vehicle 100 and recognizes objects in its surroundings. Specifically, the moving object recognition device 1 extracts row images at predetermined vertical positions from each of multiple time-series surrounding images captured multiple times by the imaging unit 5, and generates a time-series image by arranging the extracted row images vertically in chronological order. The moving object recognition device 1 can determine the presence of a moving object moving around the vehicle 100 based on the difference in the horizontal position of feature points whose feature quantities change within the row images of the generated time-series image. The moving object recognition device 1 has, for example, a general-purpose microcontroller as a computer. The microcontroller includes a CPU (Central Processing Unit) and memory. The memory includes ROM (Read Only Memory) and RAM (Random Access Memory). The microcontroller can virtually construct multiple information processing circuits by having the CPU execute a program stored in the memory.
[0011] [First Embodiment] The moving object recognition device 1 of the first embodiment shown in Figure 1 can implement the moving object recognition method according to the first embodiment. In the moving object recognition device 1 of this embodiment, multiple information processing circuits of the microcontroller can constitute the determination image generation unit 11, the linear component extraction unit 17, the movement determination unit 19, and the moving object determination unit 21 of the moving object recognition device 1. In this embodiment, an example is shown in which multiple information processing circuits are implemented by software. Of course, it is also possible to configure the information processing circuits by preparing dedicated hardware to perform each of the information processing operations of each unit 11, 17 to 21 as shown below. Alternatively, multiple information processing circuits may be configured by individual hardware. Dedicated hardware includes devices such as application-specific integrated circuits (ASICs) arranged to perform the functions of each unit 11, 17 to 21, and conventional circuit components.
[0012] The judgment image generation unit 11 receives ambient images captured multiple times by the imaging unit 5 as input. The judgment image generation unit 11 generates a time-series feature image from the input ambient images. The judgment image generation unit 11 has a time-series image generation unit 23 and a feature image generation unit 25. The time-series image generation unit 23 generates a time-series image from multiple ambient images in chronological order that are input. The feature image generation unit 25 converts the input ambient images into feature images. In the judgment image generation unit 11, for multiple ambient images in a time series, the time-series image generation unit 23 sequentially generates a time-series image and the feature image generation unit 25 sequentially converts it into a feature image to generate a time-series feature image. The order in which the time-series image generation unit 23 generates the time-series image and the feature image generation unit 25 converts it into a feature image to generate a time-series feature image. If the conversion to a feature image is performed first, the feature image generation unit 25 generates a time-series feature image, and the time-series image generation unit 23 generates a time-series image using the multiple time-series feature images converted from multiple time-series surrounding images, thereby generating a time-series feature image. In the following explanation, the generation of the time-series image by the time-series image generation unit 23 is performed first, and the conversion to a feature image by the feature image generation unit 25 is performed afterward.
[0013] Multiple time-series images are input to the time-series image generation unit 23. The time-series image generation unit 23 generates a time-series image that shows the time changes of the multiple images input to the time-series image generation unit 23, row by row. Figure 2 shows a case where the multiple time-series images input to the time-series image generation unit 23 include ambient images 27t1 to 27t5 that have been captured multiple times by the imaging unit 5. The time-series image generation unit 23 may receive time-series image data one image at a time as it progresses, or it may receive data each time a predetermined number of images have been accumulated. The period for the time-series image generation unit 23 to generate a time-series image shown in Figure 1 may be longer than the imaging period for ambient images by the imaging unit 5. When the time-series image generation unit 23 generates a time-series image using ambient images from the imaging unit 5, it may use the same number of ambient images as the imaging unit 5 captures during one period of time-series image generation, or it may use a larger or smaller number of ambient images. For example, if the imaging unit 5 captures 10 ambient images during one cycle for generating a time-series image, the time-series image generation unit 23 may generate time-series images for each cycle using ambient images 0-20, 10-30, and 20-40 in the 1st to 3rd cycles. Alternatively, the time-series image generation unit 23 may generate time-series images using ambient images 0-5 in the 1st cycle, ambient images 10-15 in the 2nd cycle, and ambient images 20-25 in the 3rd cycle.
[0014] The images used by the time-series image generation unit 23 to generate time-series images may partially overlap with those used in preceding and succeeding processes, and some images may not be used to generate the time-series images. The number of images used to generate the time-series images should be as large as possible within a short unit of time, such that the lateral time-series movement of feature points in the time-series images can be approximated by a straight line. The more images used to generate the time-series images, the higher the resolution with which the lateral movement of feature points in the time-series images can be determined. When generating time-series images using the surrounding images from the imaging unit 5, if the imaging unit 5 captures the surrounding images over a short period, the acceleration of the movement of objects in the surrounding images can be largely ignored. Ignoring the acceleration of the movement of objects makes it easier to perform the process of approximating the amount of movement of objects in the surrounding images with a straight line from the trajectory of the object's position at each period. The time-series image generation unit 23 extracts row images at a predetermined position in the vertical direction from each of the images input to the time-series image generation unit 23, and generates a time-series image in which the extracted plurality of row images are arranged vertically in time-series order. In the example shown in FIG. 2, the row image 29t5 extracted from a predetermined position in the vertical direction of the surrounding image 27t5 and the row images 29t3, 29t4, 29t6, etc. extracted from predetermined positions in the vertical direction of other surrounding images are arranged vertically in time-series order to show the case where the time-series image 29 is generated. Information on the lateral movement of the object remains in the row image. In the detection of a moving object performed in the vehicle 100, an object moving in the lateral direction may, for example, enter from outside the path of the vehicle 100 into the path, and may become an important detection target. In the time-series image 29, information on the lateral movement of the object shown in the row image at a predetermined position in the vertical direction of each surrounding image is arranged in time-series in the vertical direction of the time-series image 29.
[0015] As in the example shown in FIG. 2, the row image 29t5 used to generate the time-series image 29 may be a part rather than the entire lateral direction. When a part of the row image 29t5 is used to generate the time-series image 29, the part of the row image used to generate the time-series image 29 may, by default, be, for example, a part of the same range in the lateral direction of each surrounding image. The predetermined position in the vertical direction of each surrounding image means a position where the content shown in the row image at that position is common among the surrounding images. The predetermined position in the vertical direction of each surrounding image may, by default, be, for example, the position of the same row number among a plurality of rows arranged in the vertical direction of each surrounding image. The predetermined position of each surrounding image may be shifted to positions of different row numbers among the surrounding images, for example, corresponding to changes in the imaging range of each surrounding image due to the movement of the vehicle 100. For example, when an area such as sky or road surface that is clearly different from an object moving around the vehicle 100 is shown in the surrounding image, the row image in that area may be excluded from the target for generating the time-series image 29. The determination resolution of the moving object improves as the number of rows for generating the time-series image 29 from the row images extracted from each surrounding image increases, but the rows for generating the time-series image 29 may be thinned out in consideration of the balance between the processing cost and the required accuracy.
[0016] The feature image generation unit 25 shown in FIG. 1 converts the image input to the feature image generation unit 25 into a feature image. The feature image represents feature points where the feature amount changes in the horizontal direction of the image. The feature amount can use, for example, luminance, hue, etc. The more the image input to the feature image generation unit 25 is converted into a feature image for many types of feature amounts, the higher the accuracy of extracting the linear component in the linear component extraction unit 17 described later. For example, when the feature amount is luminance, the feature point can be, for example, a point where the differential value of luminance, which is the amount of change in the feature amount between adjacent pixels in the horizontal direction, becomes a negative value. The feature point may be a point where the differential value of luminance becomes a positive value, or both a point where the differential value becomes a negative value and a point where the differential value becomes a positive value may be used as the feature point. The feature point may be a pixel with the maximum luminance or the minimum luminance in each row. For example, when the feature amount is hue, the feature point may be the differential value of each color of RGB (Red, Green, Blue) or the differential value of hue (HUE) between adjacent pixels in the horizontal direction. The feature point may be, for example, an edge portion in the image input to the feature image generation unit 25. The edge portion can be detected using a known algorithm. For known algorithms, for example, an edge detection method using a Sobel filter, an edge detection method using a Laplacian filter, a Canny method, etc. may be used. By obtaining the time-series change of the detected edge portion, the movement of the feature points in the surrounding image can be detected. When converting the image input to the feature image generation unit 25 into a feature image, a process of removing noise from the input image may be performed. For noise removal, for example, a normalization process such as a median filter, binarization, etc. can be used. The conversion of the image into a feature image may be performed by a process that also serves as noise removal.
[0017] Figure 3 shows a time-series feature image 31 obtained by converting the time-series image 29 from Figure 2 into a feature image, where the feature quantity is luminance. In the time-series feature image 31 of Figure 3, multiple row feature images, including row feature images 31t3 to 31t6 obtained by converting each row image 29t3 to 29t6 from Figure 2, are arranged vertically in chronological order. The symbol 33 in Figure 3 indicates a feature point. In Figure 3, a feature point 33 is defined as a pixel where the derivative of luminance between adjacent pixels is a negative value, and there is a gradient from light to dark. A feature point 33 may also be defined as a pixel where the derivative of luminance between adjacent pixels is a positive value, and there is a gradient from dark to light. A feature point 33 may also be defined as the absolute value of the derivative of luminance between adjacent pixels. When a feature point 33 is defined as the derivative of luminance between adjacent pixels, the time-series feature image 31 is the content extracted as feature points 33 from the edge portions in the image input to the feature image generation unit 25.
[0018] In the judgment image generation unit 11 of this embodiment shown in Figure 1, the time-series image generation unit 23 generates a time-series image first, and the feature image generation unit 25 converts the time-series image into a feature image afterward. In the generation of the time-series feature image 31 by the judgment image generation unit 11, noise in the surrounding image can be efficiently removed when the feature image generation unit 25 converts the time-series image into a feature image.
[0019] The linear component extraction unit 17 extracts multiple linear components from the time-series feature image 31 generated by the judgment image generation unit 11. row imageThe linear component of feature points 33 that are connected vertically over time is extracted. In the time-series feature image 31, feature points 33 connected in the t-axis direction of the time-series feature image 31 represent feature points 33 that exist continuously in the time series in the row of the surrounding image corresponding to the time-series feature image 31. Since the X-axis of the time-series feature image 31 is the horizontal axis of the surrounding image, if feature points 33 connected in the t-axis direction move in the X-axis direction of the time-series feature image 31 over time, then feature points 33 that exist continuously in time are moving horizontally in the surrounding image. In the time-series feature image 31, by estimating a straight line that approximates the set of feature points 33 connected in the t-axis direction and evaluating the slope of the approximate straight line with respect to the t-axis, it is possible to determine feature points 33 that move horizontally in the row of the surrounding image corresponding to the time-series feature image 31. In the time-series feature image 31, feature points 33 that are connected vertically are a set of points, not a line. The linear component extraction unit 17 estimates a straight line that approximates the set of feature points 33 and extracts the estimated straight line as the linear component of the feature points 33. The estimation of a straight line that approximates the set of feature points 33 can be performed using a known method. The following describes the case in which the approximate straight line of the set of feature points 33 is estimated using the Hough transform. When an approximate line of the set of feature points 33 is estimated using the Hough transform, the estimated approximate line 34 is represented by a slope θ and a length x in the space of the Xt plane coordinate system in which the time-series feature image 31 exists, as shown in Figure 4. The slope θ is the angle between the X axis of the time-series feature image 31 and the perpendicular 35 drawn from the approximate line 34 to the origin (X, t) = (0, 0) of the Xt plane coordinate system, and the length x is the length of the perpendicular 35. The slope θ is considered positive (θ>0) when the perpendicular 35 is tilted clockwise with respect to the X axis of the time-series feature image 31, and negative (θ<0) when the perpendicular 35 is tilted counterclockwise with respect to the X axis. A positive slope θ means that the set of feature points 33 approximated by the estimated line are feature points 33 that move leftward in time series within the row image corresponding to the time-series feature image 31 of the surrounding image. A negative slope θ means that the set of feature points 33 that approximate the estimated straight line are feature points 33 that move to the right in time series within the row image corresponding to the time-series feature image 31 of the surrounding image. The magnitude of the slope θ is, row image It is proportional to the movement speed of feature point 33 within the structure. The linear component extraction unit 17 outputs to the movement determination unit 19 a set of the slope θ of an approximate line 34 estimated by the Hough transform from one time-series feature image 31 and the length x of the perpendicular 35, along with the score of that set as a linear component, as the linear component of the feature point 33 extracted from the time-series feature image 31. The score as a linear component may be such that, for example, the longer the set of feature points 33 in the time-series feature image 31 is connected in the t-axis direction of the time-series feature image 31, the higher the score. The linear component of the feature point 33 extracted by the linear component extraction unit 17 includes those extracted from the time-series feature image 31 of rows that are not signed in the time-series image 29 of Figure 2, and those extracted from the time-series feature image 31 of rows that are not shown in the time-series image 29 of Figure 2. If there are multiple feature points 33 that move horizontally at different positions in the row of the surrounding image corresponding to the time-series feature image 31, multiple lines approximating the set of feature points 33 are estimated from one time-series feature image 31. Even if the set of feature points 33 in the time-series feature image 31 corresponding to a single feature point 33 moving horizontally does not lie on a straight line, and the set of feature points 33 can be approximated by multiple straight lines, multiple straight lines approximating the set of feature points 33 are estimated from the single time-series feature image 31. When multiple approximating straight lines are estimated from the single time-series feature image 31, the linear component extraction unit 17 outputs the slope θ, the length x of the perpendicular, and the score of each approximating straight line to the movement determination unit 19 as multiple linear components of the feature point 33 extracted from the single time-series feature image 31.
[0020] The movement determination unit 19 determines the movement feature points for each row of the surrounding image in Figure 2 based on the linear components of the feature points 33 in the time-series images for each row in the surrounding image in Figure 2, which are input from the linear component extraction unit 17. A movement feature point is a feature point 33 that moves horizontally in a row. If there are multiple linear components for a feature point 33 in each row, the movement determination unit 19 groups each linear component into linear components for feature points 33 that move horizontally at different positions. Linear components of feature points 33 that do not match in content but are considered to correspond to the same feature point 33 are sorted into the same group. Linear components of feature points 33 that move horizontally at different positions are considered to have different slopes θ of their approximate lines 34 or lengths x of their perpendiculars 35. Linear components of feature points 33 that do not match in content but are considered to correspond to the same feature point 33 are considered to have matching or close slopes θ of their approximate lines 34 or lengths x of their perpendiculars 35, and therefore different scores. The slope θ of the approximate line 34 and the length x of the perpendicular line 35 are parameters corresponding to the direction of movement and linear component of the moving feature point, and can be used to group each linear component when there are multiple linear components for the feature point 33 in each row. The movement determination unit 19 can sort the linear components of feature points 33 that are moving horizontally at different positions using the slope θ of the approximate line 34 and the length x of the perpendicular line 35. The movement determination unit 19 plots points on the x-θ plane coordinate system that correspond to the slope θ and the length x of the perpendicular 35 of each linear component of the feature point 33 extracted from the time-series feature image 31, with respect to the approximate line 34. The linear components of the feature point 33 that are targeted for plotting on the x-θ plane coordinate system are limited to linear components that have a score above a predetermined threshold. If the difference between the slope θ and the length x of the perpendicular 34 of two linear components of the feature point 33 is within a predetermined range, the movement determination unit 19 may classify the linear components of two feature points 33 into the same group, assuming that the parameter difference corresponding to the linear components of the feature point 33 is within a predetermined range. Figure 5 shows the distribution of the slope θ and perpendicular length x of the approximate line 34 for the linear component of the feature point 33 extracted from the time-series feature image 31 in Figure 3. In the x-θ plane coordinate system of Figure 5, there are regions 37 and 39 where points corresponding to the slope θ and perpendicular length x of the linear component are concentrated, in the quadrant where the slope θ is negative and the perpendicular length x is positive, and in the quadrant where the slope θ is positive and the perpendicular length x is positive. The points concentrated in region 37 in the quadrant with a negative slope θ correspond to the set of feature points 33 that are continuous from the upper left to the lower right, located on the right side of the time-series feature image 31 in Figure 3. The points concentrated in region 39 in the quadrant with a positive slope θ correspond to the set of feature points 33 that are continuous from the upper right to the lower left, located on the left side of the time-series feature image 31 in Figure 3. The slope θ or perpendicular length x of the approximate line indicated by each point in regions 37 and 39 in Figure 5 are either identical or close to each other, respectively, within regions 37 and 39. The linear components of feature points 33 corresponding to points concentrated in each region 37 and 39 can be considered as a set of linear components corresponding to a single feature point 33 moving horizontally. The movement determination unit 19 determines that multiple linear components of feature points 33 corresponding to points concentrated in region 37 and multiple linear components of feature points 33 corresponding to points concentrated in region 39 are each a single moving feature point. The movement determination unit 19 then performs an approximation regarding the feature points 33 determined to be moving feature points. straight line The inclination θ of 34 and the row information of the time-series feature image 31 corresponding to its feature point 33 are output to the moving object detection unit 21 as information of moving feature points in the row image. Approximation of feature point 33 determined to be a moving feature point. straight line The inclination θ of 34 can be used as information about the lateral movement direction and movement speed of the moving feature point. The moving feature point information is used by the moving object determination unit 21 to approximate the feature point 33 in the time-series feature image 31. straight line This can be used to determine the presence of an object moving around the vehicle 100, based on the lateral movement speed of the feature point 33 represented by the inclination θ of 34.
[0021] The moving object detection unit 21 determines the presence of an object moving around the vehicle 100. Specifically, the moving object detection unit 21 determines an object moving laterally within the surrounding image based on the moving feature points determined for each row of the time-series feature image 31, which is information on moving feature points input from the moving detection unit 19. The moving object detection unit 21 clusters the moving feature points for each row determined by the moving detection unit 19 according to the direction and speed of movement in the horizontal direction of the row image. The moving object detection unit 21 clusters the moving feature points and groups moving feature points with similar direction and speed into the same cluster. The set of moving feature points in each grouped cluster is determined to be the moving feature points of a single object. Existing methods such as the k-means method, a non-hierarchical clustering algorithm, and hierarchical clustering can be used for clustering moving feature points. In Figure 6, the feature points 33 corresponding to the moving feature points, which have been divided into two groups, are distributed and assigned to two people 41 and 43 moving on the surrounding image 27t5 in Figure 2. In Figure 6, the moving object detection unit 21 divides the feature points 33 corresponding to the moving feature points into two groups based on similar movement direction and speed. Each group is a set of feature points 33 moving leftward and rightward within the surrounding image, respectively, and is shown by different symbols (□, ■) in Figure 6. The moving object detection unit 21 uses the set of feature points 33 with a leftward movement direction 45 to determine that person 41 is moving leftward on the surrounding image 27t5 in Figure 2, and uses the set of feature points 33 with a rightward movement direction 47 to determine that person 41 is moving rightward on the surrounding image 27t5.
[0022] The flowchart in Figure 7 shows the processing steps performed by the microcontroller to recognize a moving object. As shown in Figure 7, the microcontroller acquires multiple time-series ambient images captured multiple times by the imaging unit 5 (step S101). The time-series image generation unit 23 of the microcontroller extracts row images at predetermined vertical positions from each of the acquired time-series ambient images (step S103). The microcontroller generates a time-series image by arranging the extracted row images in chronological order (step S105). The microcontroller recognizes a moving object based on the difference in the horizontal position of feature points where the feature quantities within the row images change between the row images of the generated time-series image 29 (step S107). The microcontroller recognizes a moving object based on a time-series feature image 31 generated using the time-series image 29 generated by the time-series image generation unit 23. The processing in step S107 can be performed, for example, by the feature image generation unit 25, the linear component extraction unit 17, the movement determination unit 19, and the moving object determination unit 21.
[0023] The flowchart in Figure 8 shows an overview of the processing procedure performed by the microcontroller's time-series image generation unit 23 for generating time-series images 29. The processing procedure shown in Figure 8 can be used, for example, as a detailed processing procedure when the time-series image generation unit 23 performs the processing in steps S101 to S105 of Figure 7. As shown in Figure 8, the time-series image generation unit 23 acquires images captured by the imaging unit 5 as ambient images, representing the scene around the vehicle 100 (step S201). The time-series image generation unit 23 stores the acquired ambient images in the microcontroller's RAM memory (step S203). The time-series image generation unit 23 checks whether the number of ambient images in RAM has reached the number required to generate time-series images 29 (step S205). If the number of ambient images in RAM is insufficient (NO in step S205), the unit returns to step S201. If the RAM contains the required number of surrounding images (YES in step S205), the time-series image generation unit 23 generates a time-series image 29 using row images at predetermined vertical positions extracted from multiple time-series surrounding images stored in RAM (step S207).
[0024] The flowchart in Figure 9 shows an overview of the processing procedure performed by the microcontroller when extracting multiple linear components of feature points 33 from a time-series feature image 31 generated using the time-series image 29. The processing procedure shown in Figure 9 can be a detailed processing procedure when, for example, the feature image generation unit 25 and the linear component extraction unit 17 perform part of the processing in step S107 of Figure 7. As shown in Figure 9, the feature image generation unit 25 removes noise from the time-series image 29 generated by the time-series image generation unit 23 (step S301). For noise removal, for example, normalization processing such as median filtering or binarization can be used. The feature image generation unit 25 then performs conversions of the noise-removed time-series image 29 into feature image A and feature image B, respectively, using kernel A and kernel B of the Sobel filter (steps S303 and S305). By converting to feature image A, a time-series feature image 31 is obtained in which the lateral edge portions of the time-series image 29 are extracted. By converting to feature image B, a time-series feature image 31 is obtained by extracting the vertical edge portions of the time-series image 29. In steps S303 and S305, pixels with a gradient from light to dark, whose pixel values after applying the Sobel filter are negative, are extracted. In steps S303 and S305, pixels with a gradient from dark to light, whose pixel values after applying the Sobel filter are positive, may also be extracted. In steps S303 and S305, both pixels with positive and negative pixel values after applying the Sobel filter may be extracted. The processing in steps S303 and S305 may be performed either one first or both in parallel. Either the processing in steps S303 or S305 may be omitted. The linear component extraction unit 17 performs a Hough transform on each time-series feature image 31 obtained by the feature image generation unit 25 converting the time-series image 29 to feature image A and feature image B (steps S307 and S309). The linear component extraction unit 17 can estimate an approximate linear line 34 of the set of feature points 33 of each time-series feature image 31 by applying a Hough transform to each time-series feature image 31.If either step S303 or step S305 is omitted, the corresponding step S307 or step S309 may be omitted. The linear component extraction unit 17 stores the slope θ of the approximate line 34 of the feature points 33 extracted by performing a Hough transform on the time-series feature image 31, the length x of the perpendicular line 35, and the score in the RAM of the microcontroller's memory (step S311).
[0025] In this embodiment, a time-series image 29 is generated by arranging row images of predetermined vertical positions extracted from multiple time-series surrounding images in a time-series manner. This removes information about the vertical movement of objects in the surrounding images and arranges information about the horizontal movement of objects in a time-series manner. A time-series feature image 31 is generated by converting the time-series image 29 into a feature image. This extracts feature points 33 whose feature quantities change in the horizontal direction for each row image of the time-series image 29. The slope of the linear component of the feature point 33 in the time-series feature image 31 represents the horizontal movement speed of the point corresponding to the feature point 33 that is captured at a predetermined vertical position in the surrounding image. Therefore, a moving feature point, which is a feature point 33 that is moving on the surrounding image, can be determined from this slope. The determined moving feature points can be used as material for recognizing moving objects around the vehicle 100 based on the horizontal movement speed of the feature point represented by the slope of the linear component of the feature point 33 in the time-series feature image 31. In this embodiment, a moving object in the background that is partially hidden by an object in the foreground can be suitably recognized. In this embodiment, the time-series image generation unit 23 extracts row images at predetermined positions from the input image to generate a time-series image 29, while the feature image generation unit 25 does not extract row images at predetermined positions from the input image, but converts the input image directly into a feature image. It is efficient to perform the noise removal process on the entire image. In the judgment image generation unit 11, by performing the processing of the time-series image generation unit 23 first, when the feature image generation unit 25 processes after the processing of the time-series image generation unit 23, which extracts row images from the input image, the noise removal process for the time-series feature image 31 can be performed efficiently. In this embodiment, the feature image generation unit 25 generates an image representing the edge portion in the surrounding image as a feature image, and by the time-series change of the edge portion, it is possible to track moving objects present in the surrounding image. In order to track moving objects present in the surrounding image using optical flow, pixel information over a wider area than edge detection is required in order to recognize the moving object as a surface. By detecting the movement of objects in the surrounding image by determining the time-series changes in the edge regions, it is possible to detect the movement of small objects more effectively than by tracking object movement using optical flow.
[0026] Since the time-series feature image 31 has rows of images arranged vertically in chronological order, if the linear component of the feature point 33 extracted from the time-series feature image 31 is not tilted and extends vertically, then the feature point 33 on the surrounding image has not moved horizontally over time. The direction of the tilt of the linear component of the feature point 33 indicates whether the feature point 33 on the surrounding image moved horizontally to the left or right over time. Based on whether the feature point 33 on the surrounding image moved horizontally to the left or right over time, the moving feature points can be grouped according to the object moving on the surrounding image. The angle of the tilt of the linear component of the feature point 33 contains information about the amount of horizontal movement of the feature point 33 in the surrounding image and time information in the time series, and represents the speed of movement of the feature point 33 that moved horizontally on the surrounding image. Based on the speed of movement of the feature point 33 on the surrounding image, the moving feature points can be grouped according to the object moving on the surrounding image. Multiple moving feature points, determined based on the slope of the linear component of each feature point 33 extracted from each row of the time-series feature image 31, may include moving feature points corresponding to different objects in the surrounding image. When multiple moving feature points are grouped based on the direction of movement of the moving feature points in each row of the time-series feature image 31 and the parameters corresponding to the linear component of the feature point 33, moving feature points with the same direction of movement and close positions are sorted into the same group. Moving feature points with similar direction of movement and positions in the time-series feature image 31 may correspond to a common object in the surrounding image. If the score of the grouped moving feature points is above a predetermined threshold, it can be determined that there is an object moving in the surroundings that corresponds to the grouped moving feature points. If the difference between the slope θ of the approximate line 34, which is a parameter corresponding to the linear component of the feature point 33 on the time-series feature image 31, and the length x of the perpendicular line 35 is within a predetermined range, the positions of the two moving feature points are considered to be close. Since the difference in the slope θ of the approximate line 34 is within a predetermined range, the difference in movement speed between the two moving feature points in the surrounding image is also considered to be small. In this case, the two moving feature points may correspond to a common object in the surrounding image. If their positions in the surrounding image are close and the difference in their movement speed is within a predetermined value, the two moving feature points can be grouped into the same group and determined to correspond to a common object in the surrounding image.
[0027] [Second Embodiment] The moving object recognition device 1 of the second embodiment shown in Figure 10 can implement the moving object recognition method according to the second embodiment. The moving object recognition device 1 of this embodiment can recognize objects that move outward from the boundary of a structure that constitutes occlusion in the surrounding image of the vehicle 100. In the moving object recognition device 1 of this embodiment, multiple information processing circuits of the microcontroller can constitute the various parts 11, 15 to 21 of the moving object recognition device 1 of the first embodiment, as well as the structure estimation unit 13 and the structure movement calculation unit 15. Of course, dedicated hardware may be prepared to perform the information processing of each part 13, 15 as described below, and the information processing circuit may be configured accordingly.
[0028] The structure estimation unit 13 estimates occlusion in the surrounding image. Occlusion is what hides moving objects in the surrounding image from view, and includes structures. Structures that constitute occlusion are, for example, stationary three-dimensional objects. Stationary three-dimensional objects may include, for example, permanently existing three-dimensional objects such as buildings, temporarily occurring three-dimensional objects such as construction sites and parked vehicles, and newly occurring permanent structures. The structure estimation unit 13 can, for example, estimate permanently existing structures such as buildings in advance using previously acquired information. For estimating permanently existing three-dimensional objects, for example, information from high-precision 3D map data (HD map, High Definition MAP) and position information from the vehicle 100's GNSS (Global Navigation Satellite System) sensor can be used. Temporary or newly occurring three-dimensional objects can be estimated, for example, from the parallax of images from the stereo camera of the object detection unit 3. Permanently existing three-dimensional objects may also be estimated from the parallax of images captured by the stereo camera of the object detection unit 3. The structure estimation unit 13 can estimate, for example, an object determined to have a volume greater than a predetermined amount from the image captured by the stereo camera of the object detection unit 3 as an occlusion structure. Occlusion is not limited to structures. Occlusion may also include points where pedestrians may suddenly appear. Points where pedestrians may suddenly appear include, for example, intersections with terrain that has poor visibility. An occluded structure may move within the surrounding image due to the movement of, for example, the vehicle 100. The structure movement calculation unit 15 calculates the direction and speed of movement of the occluded structure within the surrounding image. The direction and speed of movement of the occluded structure within the surrounding image can be calculated by tracking the movement of the structure estimated by the structure estimation unit 13 within the surrounding image. The movement of the structure within the surrounding image can be tracked by, for example, optical flow, template matching, etc. The movement of the structure within the surrounding image may also be estimated using, for example, the distance from the imaging position detected by the object detection unit 3 to the structure and the result of self-position detection of the vehicle 100 by odometry. When an occluded structure moves within the surrounding image, the time-series image generation unit 23 may shift the position of the row images used to generate the time-series image 29 or time-series feature image 31 in the surrounding image in at least one of the vertical and horizontal directions of the surrounding image. In this embodiment, the time-series image generation unit 23 generates the time-series image 29 before the feature image generation unit 25 converts it to a feature image, so it shifts the position of the row images used to generate the time-series image 29. The time-series image generation unit 23 may shift the position of the row images used to generate the time-series image 29 based on the movement direction and movement speed of the structure that becomes occlusion in the surrounding image calculated by the structure movement calculation unit 15. The time-series image generation unit 23 may shift a predetermined position of the row images used to generate the time-series image 29 in the vertical direction of the surrounding image, or it may shift the portion of the row images used to generate the time-series image 29 in the horizontal direction of the surrounding image.
[0029] In this embodiment, the moving object detection unit 21 may determine objects moving laterally within the surrounding image by limiting the determination to the area surrounding the boundary of an occluded structure. To this end, the movement detection unit 19 may determine movement feature points by limiting them to the aforementioned surrounding area, and the linear component extraction unit 17 may extract the linear component of the feature point 33 by limiting it to the aforementioned surrounding area. To this end, the determination image generation unit 11 may generate the time-series feature image 31 by limiting it to the aforementioned surrounding area, and the generation of the time-series image 29 by the time-series image generation unit 23 and the conversion of the image to a feature image by the feature image generation unit 25 may also be limited to the aforementioned surrounding area.
[0030] The flowchart in Figure 11 outlines the procedure for determining objects moving outside the boundary of an occluded structure. The microcontroller extracts objects with height present around the vehicle 100 from the image captured by the stereo camera of the object detection unit 3 (step S401). The microcontroller estimates that objects among the extracted objects that are determined to have a volume greater than a predetermined amount are occluded structures (step S403). The microcontroller extracts feature points of the occluded structures from the image captured by the object detection unit 3, tracks the extracted feature points in a time series, and determines the movement speed of the occluded structures in the surrounding image of the imaging unit 5 (step S405). Optical flow, for example, can be used to track the feature points. The movement speed to be determined includes information on the direction of movement of the occluded structures in the surrounding image. The microcontroller extracts the boundary of the occluded structures. The boundary is the contour of the occluded structures. The extracted boundaries include, at a minimum, the portion of the occluded structure that constitutes the boundary of the occluded structure in the lateral direction of the surrounding image. The time-series image generation unit 23 of the microcontroller generates a time-series image 29 for each extracted boundary (step S407). When the occluded structure moves within the surrounding image, the time-series image generation unit 23 generates the time-series image 29 while shifting the position of the row image in at least one of the vertical and horizontal directions of the surrounding image in accordance with the speed and direction of the movement (step S409). The processing in step S409 can be performed as a process to correct the time-series image 29 generated in the processing of step S407 as needed. The microcontroller determines moving objects that are moving from the boundary of an occluded structure to the outside of the occluded structure based on a time-series feature image 31 generated using a time-series image 29 generated by a time-series image generation unit 23 (step S411). The processing in step S411 can be performed by the feature image generation unit 25, the linear component extraction unit 17, the movement determination unit 19, and the moving object determination unit 21. The microcontroller determines the absolute speed of the moving objects in the space surrounding the vehicle 100 (step S413). The absolute speed of the object's movement can be determined, for example, using the distance between the object and the imaging position detected by the object detection unit 3 and the movement of the imaging position based on the self-position detection result of the vehicle 100 by odometry. The microcontroller determines the action plan of the vehicle 100 from the determined absolute speed (step S415). The action plan of the vehicle 100 may be determined by the controller of the vehicle 100's driving control system instead of the microcontroller. The controller for the vehicle 100's driving control system may be, for example, an automatic driving ECU (Electronic Control Unit).
[0031] In the surrounding area of a structure's boundary in the surrounding image, objects protruding from behind the structure may appear. In this embodiment, by focusing on determining moving feature points, particularly in the area surrounding the structure's boundary within the surrounding image, the presence of moving objects in the surrounding area can be efficiently determined. If the structure is fixed, when the structure moves on the surrounding image, the imaging position of the surrounding image moves relative to the structure. When the imaging position of the surrounding image moves relative to the structure, the content of the row images when converting the surrounding image into a time-series image is offset in at least one of the vertical and horizontal directions in accordance with the movement of the imaging position of the surrounding image. By offsetting the row images, which are arranged in chronological order in the time-series image, in at least one of the vertical and horizontal directions based on the movement speed of the structure on the surrounding image, the offset of the row images caused by the movement of the imaging position is canceled out. This cancellation ensures that the content of the row images captured during the movement of the imaging position and the content of the row images captured before or after the movement are aligned in at least one of the vertical and horizontal directions on the time-series image 29. The movement speed of a moving feature point in the surrounding image is the relative movement speed of the moving feature point with respect to the surrounding image. This relative movement of the moving feature point with respect to the surrounding image includes the relative movement of the feature point due to the movement of the image acquisition position in the surrounding image. By detecting the movement of the image acquisition position in the surrounding image and detecting the distance from the acquisition position to the object corresponding to the moving feature point in the surrounding image, the relative movement speed of the moving feature point in the surrounding image can be converted into an absolute movement speed based on these factors. The converted absolute movement speed allows for accurate recognition of moving objects in the surroundings.
[0032] In each of the above embodiments, the moving object recognition device 1 is mounted on the vehicle 100, so that moving objects around the vehicle 100 can be suitably recognized on the surrounding image of the vehicle 100 captured from the vehicle 100, even if they are partially hidden by objects in the foreground. It should be noted that the moving object recognition method may also be performed using the moving object recognition device 1 without mounting it on the vehicle 100. The above embodiments are examples of the present invention. Therefore, the present invention is not limited to the above embodiments, and of course, various modifications can be made to forms other than these embodiments, as long as they do not depart from the technical spirit of the present invention, depending on the design and so on. [Explanation of symbols]
[0033] 1 Moving object recognition device, 17 Linear component extraction unit, 19 Movement determination unit, 21 Moving object determination unit, 23 Time series image generation unit, 25 Feature image generation unit, 27t1~27t5 Surrounding images, 29 Time series images, 29t3~29t5 row image 31 Time-series feature images, 33 Feature points, 41, 43 People (moving objects), 100 Vehicles
Claims
1. A computer method for recognizing moving objects, comprising acquiring multiple time-series images of the surroundings by capturing images of the surroundings multiple times in a time-series manner, and recognizing moving objects in the surroundings from the multiple time-series images of the surroundings, Row images at predetermined vertical positions are extracted from each of the multiple surrounding images in the aforementioned time series, and a time series image is generated by arranging the extracted row images vertically in chronological order. A plurality of time-series images are generated in which the predetermined positions of the row images are different from each other. The feature points in the multiple time-series images that move between the row images and whose feature quantities within the row images change are grouped based on the direction of movement of the lateral position of the feature point and a parameter corresponding to a linear component that approximates the time-series change of the lateral position. If the difference in the movement speed of the lateral position of two feature points that move between the row images is within a predetermined value, and the difference in the parameters is within a predetermined range, the two feature points are clustered into the same group. Based on the difference in the lateral position of the feature points between the row images of the time-series images clustered into the same group, the moving object is recognized. A method for recognizing moving objects, including the following.
2. The method for recognizing a moving object according to claim 1, which involves extracting feature points from each of the multiple row images of the time-series image and detecting the difference in the lateral position of the extracted feature points between the row images.
3. The method for recognizing a moving object according to claim 1 or 2, wherein the edge portion in the row image is extracted as the feature point.
4. A method for recognizing a moving object according to claim 1 or 2, wherein the feature points in the row image are extracted based on the amount of negative change in the horizontal direction of the feature quantity.
5. A method for recognizing a moving object according to claim 1 or 2, wherein the moving object is recognized based on the movement speed of the feature point whose lateral position moves between the row images of the time series.
6. The moving object is recognized when the score of the grouped feature points, which increases as the predetermined positions are connected in a long vertical line, is equal to or greater than a predetermined judgment threshold. The method for recognizing a moving object according to claim 1.
7. The moving object is recognized in the area surrounding the boundary of a structure present in the aforementioned surrounding image. A method for recognizing a moving object according to claim 1 or 2.
8. The method for recognizing a moving object according to claim 7, wherein the row images at positions offset vertically from the predetermined position are arranged in the time-series image based on the movement speed of the structure on the surrounding image.
9. A moving object recognition device that acquires multiple time-series images of the surroundings by capturing images of the surroundings multiple times in a time series, and recognizes a moving object in the surroundings from the acquired multiple time-series images of the surroundings, A time-series image generation unit extracts row images at predetermined vertical positions from each of the multiple surrounding images in the aforementioned time series, and generates a time-series image by arranging the extracted row images vertically in chronological order. A moving object determination unit determines the presence of the moving object based on the difference in the horizontal position of feature points where the feature quantity within the row image changes between the row images of the time-series image, Movement determination unit, Equipped with, The time-series image generation unit generates a plurality of time-series images in which the predetermined positions of the row images are different from each other. The movement determination unit groups the feature points that move between the row images of the plurality of time-series images based on the direction of movement of the lateral position of the feature point and a parameter corresponding to a linear component that approximates the time-series change of the lateral position. The moving object determination unit is, If the difference in the movement speed of the lateral position of two feature points that move between the row images is within a predetermined value, and the difference in the parameters is within a predetermined range, the two feature points are clustered into the same group. Based on the difference in the lateral position of the feature points between the row images of the time-series images clustered into the same group, the moving object is recognized. Moving object recognition device.