Object recognition method and object recognition device

Abstract

This moving object recognition method involves acquiring a plurality of time series surroundings images (27t1 to 27t5), which have been obtained by imaging a surroundings a plurality of times in time series, and recognizing, by means of a computer (1), a moving object moving in the surroundings from the plurality of time series surroundings images (27t1 to 27t5). The computer (1) extracts row images (29t3 to 29t5) at a prescribed position in the vertical direction from each of the plurality of time series surroundings images (27t1 to 27t5) and extracts the plurality of extracted row images (29t3 to 29t5). The computer (1) generates a time series image (29) in which the plurality of extracted row images (29t3 to 29t5) are arranged in the vertical direction in chronological order. The computer (1) recognizes the moving object (41, 43) on the basis of a feature point (33) of which a feature value in the row images (29t3 to 29t5) changes between the row images (29t3 to 29t5) of the time series image (29).

Description

Object recognition method and object recognition device

[0001] The present invention relates to a moving object recognition method and a moving object recognition device.

[0002] The object detection device in Patent Document 1 calculates the optical flow of each pixel in the time-series movement direction within a captured image, groups the calculation results, and estimates the areas of stationary three-dimensional objects and moving objects from the grouped calculation results.The object detection device detects the boundary between the estimated areas of stationary three-dimensional objects and the areas of moving objects, and detects any protruding objects within the image of the moving object area.

[0003] Japanese Patent Application Laid-Open No. 2016-24602

[0004] In a captured image, near the boundary between two objects that are partially overlapping, the textures of both objects are continuous with no gap between them. When the textures of both objects are continuous, it becomes difficult to accurately track the optical flow of each object from the captured image. The present invention aims to effectively recognize a moving object in the background that is partially hidden by a moving object in the foreground.

[0005] In one aspect of the present invention, which solves the above-described problems, a moving object recognition method and a moving object recognition device acquire a plurality of time-series surrounding images obtained by capturing images of the surroundings multiple times in a time series, and recognize moving objects moving in the surroundings from the plurality of time-series surrounding images by a computer. The method and device extract row images at predetermined vertical positions from each of the plurality of time-series surrounding images, and generate a time-series image in which the extracted row images are arranged vertically in chronological order. Furthermore, the moving object is recognized based on the difference in the horizontal positions of feature points at which feature amounts change within the row images between the row images in the time-series images.

[0006] According to the present invention, it is possible to preferably recognize a moving object at the back side that is partially hidden by an object at the front side.

[0007] FIG. 1 is a diagram showing a vehicle equipped with a moving object recognition device according to a first embodiment. FIG. 2 is a diagram showing an example of a process for generating time-series images. FIG. 3 is a diagram showing an example of a process for generating feature images. FIG. 4 is a diagram showing approximate linear components of a set of feature points. FIG. 5 is a distribution diagram of the slopes and perpendicular lengths of linear components of feature points. FIG. 6 is a diagram showing moving feature points assigned to objects in a surrounding image. FIG. 7 is a flowchart of an example of a procedure for recognizing a moving object. FIG. 8 is a flowchart of an example of a procedure for generating time-series images. FIG. 9 is a flowchart of an example of a procedure for recognizing a moving object. FIG. 10 is a diagram showing a vehicle equipped with a moving object recognition device according to a second embodiment. FIG. 11 is a flowchart of an example of a procedure for recognizing a moving object.

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same parts are denoted by the same reference numerals and the description thereof will be omitted. [Configuration of a Vehicle Equipped with a Moving Object Recognition Device] As shown in Fig. 1, a vehicle 100 is equipped with a moving object recognition device 1, an object detection unit 3, and an image capture unit 5. When the vehicle 100 starts traveling, the moving object recognition device 1 determines the presence of an object moving around the vehicle 100 using the distance from the ranging point detected by the object detection unit 3 to an object around the vehicle 100 and the image captured by the image capture unit 5.

[0009] The object detection unit 3 measures the relative position from the vehicle 100 of an object present around the vehicle 100. The relative position measured by the object detection unit 3 may be, for example, a relative position based on the image capture position of an image captured by the imaging unit 5 (described later). The relative position measured by the object detection unit 3 includes information on the distance from the vehicle 100 to the object. When the relative position measured by the object detection unit 3 is based on the image capture position, the distance information can be treated as information on the distance from the image capture position to the object. The detected relative position is output to the moving object recognition device 1 and stored in a memory (not shown). The object detection unit 3 may be any sensor that can detect the distance to an object present around the vehicle 100. For example, a stereo camera can be used for the object detection unit 3. For example, a LiDAR (Light Detection and Ranging), a laser radar, or a ToF (Time of Flight) camera can be used for the object detection unit 3. For example, the imaging unit 5 can be configured with a camera that captures images. The camera has an imaging element such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 5 is mounted on the vehicle 100 and captures images of the surroundings. The images of the surroundings of the vehicle 100 captured by the imaging unit 5 are output to the moving object recognition device 1 and stored in a memory (not shown).

[0010] The moving object recognition device 1 is mounted on a vehicle 100 and recognizes surrounding objects. In particular, the moving object recognition device 1 extracts row images at predetermined vertical positions from each of a plurality of time-series surrounding images captured multiple times by an imaging unit 5, and generates a time-series image in which the extracted row images are arranged vertically in chronological order. The moving object recognition device 1 can determine the presence of a moving object moving around the vehicle 100 based on the difference in the horizontal positions of feature points at which feature amounts change between the row images in the generated time-series images. The moving object recognition device 1 includes, for example, a general-purpose microcontroller as a computer. The microcontroller includes a central processing unit (CPU) and a memory. The memory includes a read-only memory (ROM) and a random access memory (RAM). The microcontroller can virtually construct multiple information processing circuits by having the CPU execute a program stored in the memory.

[0011] [First Embodiment] A moving object recognition device 1 according to the first embodiment, shown in FIG. 1, can implement a moving object recognition method according to the first embodiment. In the moving object recognition device 1 according to this embodiment, multiple information processing circuits of a microcontroller can configure the determination image generation unit 11, the line component extraction unit 17, the movement determination unit 19, and the moving object determination unit 21 of the moving object recognition device 1. This embodiment illustrates an example in which the multiple information processing circuits are implemented by software. Of course, the information processing circuit may be configured by providing dedicated hardware for executing the information processing of each of the units 11, 17-21 described below. Alternatively, the multiple information processing circuits may be configured by individual hardware. The dedicated hardware may include devices such as application-specific integrated circuits (ASICs) and conventional circuit components arranged to perform the functions of each of the units 11, 17-21.

[0012] Surrounding images captured multiple times by the imaging unit 5 are input to the image-for-determination generation unit 11. The image-for-determination generation unit 11 generates time-series feature images from the input surrounding images. The image-for-determination generation unit 11 has a time-series image generation unit 23 and a feature image generation unit 25. The time-series image generation unit 23 generates time-series images from the input surrounding images in chronological order. The feature image generation unit 25 converts the input surrounding images into feature images. The image-for-determination generation unit 11 generates time-series images for the multiple time-series surrounding images by the time-series image generation unit 23 and then converts the time-series images into feature images by the feature image generation unit 25, in order, to generate time-series feature images. Either the generation of time-series images by the time-series image generation unit 23 or the conversion to feature images by the feature image generation unit 25 may be performed first. If the generation of time-series images is first, the feature image generation unit 25 converts the time-series images generated by the time-series image generation unit 23 into feature images to generate time-series feature images. If the conversion to feature images is performed first, the time-series image generation unit 23 generates time-series images using a plurality of time-series feature images converted from a plurality of time-series surrounding images by the feature image generation unit 25, thereby generating time-series images. In the following description, it is assumed that the generation of time-series images by the time-series image generation unit 23 is performed first, and the conversion to feature images by the feature image generation unit 25 is performed later.

[0013] A plurality of images in a time series are input to the time series image generation unit 23. The time series image generation unit 23 generates a time series image that represents, row by row, the changes over time of the plurality of images input to the time series image generation unit 23. FIG. 2 illustrates a case in which the plurality of images in a time series input to the time series image generation unit 23 include surrounding images 27t1 to 27t5 captured multiple times by the imaging unit 5. The time series image generation unit 23 may receive data of the time series images one by one as needed, or may receive data of the images each time a predetermined number of images are accumulated. The cycle in which the time series image generation unit 23 illustrated in FIG. 1 generates the time series images may be longer than the cycle in which the imaging unit 5 captures the surrounding images. When generating the time series images using the surrounding images captured by the imaging unit 5, the time series image generation unit 23 may use the number of surrounding images captured by the imaging unit 5 during one cycle of generating the time series images, or may use a greater or lesser number of surrounding images. For example, if the imaging unit 5 captures 10 surrounding images during one cycle of generating time-series images, the time-series image generating unit 23 may generate time-series images for each cycle from the 0th to 20th surrounding images, the 10th to 30th surrounding images, and the 20th to 40th surrounding images in the 1st to 3rd cycles. Alternatively, the time-series image generating unit 23 may generate time-series images from the 0th to 5th surrounding images in the 1st cycle, the 10th to 15th surrounding images in the 2nd cycle, and the 20th to 25th surrounding images in the 3rd cycle.

[0014] The images used by the time-series image generation unit 23 to generate the time-series images may partially overlap between previous and subsequent processes, or some images may not be used to generate the time-series images. The number of images used to generate the time-series images should be as large as possible within a unit of time short enough that the horizontal movement of feature points in the time-series images can be approximated by a straight line. The more images used to generate the time-series images, the higher the resolution of the horizontal movement of feature points in the time-series images. When generating time-series images using surrounding images captured by the imaging unit 5, if the capturing period of the surrounding images by the imaging unit 5 is short, the acceleration of the movement of objects in the surrounding images can be ignored to a great extent. Ignoring the acceleration of the object's movement makes it easier to perform a process of approximating the amount of movement of an object in the surrounding images to a straight line based on the trajectory of the object's position per period. The time-series image generation unit 23 extracts row images at predetermined vertical positions from each image input to the time-series image generation unit 23 and generates a time-series image in which the extracted row images are arranged vertically in chronological order. In the example shown in FIG. 2 , a row image 29t5 extracted from a predetermined vertical position of the surrounding image 27t5 and row images 29t3, 29t4, 29t6, etc. extracted from predetermined vertical positions of other surrounding images are arranged vertically in chronological order to generate a time-series image 29. The row images retain information about the lateral movement of an object. In detecting moving objects in the vehicle 100, an object moving laterally may, for example, move from outside the path of the vehicle 100 into the path, and may be an important target for detection. In the time-series image 29, information about the lateral movement of an object captured in a row image at a predetermined vertical position of each surrounding image is arranged in chronological order vertically in the time-series image 29.

[0015] As shown in the example of FIG. 2 , the row image 29t5 used to generate the time-series image 29 may be a portion rather than the entire horizontal area. When a portion of the row image 29t5 is used to generate the time-series image 29, the portion of the row image used to generate the time-series image 29 may, by default, be, for example, a portion of the same horizontal range of each surrounding image. The predetermined vertical position of each surrounding image means a position where the content captured in the row image at that position is common to each surrounding image. By default, the predetermined vertical position of each surrounding image may, for example, be the same row number among multiple rows aligned vertically in each surrounding image. The predetermined position of each surrounding image may be shifted to a different row number among each surrounding image, for example, in response to changes in the imaging range of each surrounding image as the vehicle 100 moves. For example, if a surrounding image contains an area of ​​sky, road surface, or the like that is clearly different from objects moving around the vehicle 100, the row image in that area may be excluded from the target for generating the time-series image 29. The resolution for determining moving objects improves as the number of rows extracted from each surrounding image to generate the time series image 29 increases, but the number of rows to generate the time series image 29 may be thinned out, taking into account the balance between processing cost and the required accuracy.

[0016] The feature image generation unit 25 shown in FIG. 1 converts an image input thereto into a feature image. The feature image represents a feature point where a feature quantity changes in the horizontal direction of the image. The feature quantity can be, for example, luminance, hue, etc. The more feature quantities that are converted from the image input to the feature image generation unit 25 into feature images for a greater number of feature quantities, the more accurate the linear component extraction by the linear component extraction unit 17 (described later) becomes. For example, if the feature quantity is luminance, the feature point can be a point where the differential value of luminance, which is the amount of change in the feature quantity between horizontally adjacent pixels, is negative. The feature point may be a point where the differential value of luminance is positive, or both points where the differential value is negative and positive may be used as the feature point. The feature point may be the pixel with the maximum or minimum luminance in each row. For example, if the feature quantity is hue, the feature point may be a differential value of each RGB (red, green, blue) color or a differential value of hue (hue) between horizontally adjacent pixels. The feature points may be, for example, edge portions in the image input to the feature image generation unit 25. The edge portions can be detected using a known algorithm. Examples of known algorithms that may be used include edge detection using a Sobel filter, edge detection using a Laplacian filter, and the Canny algorithm. By determining the time series changes in the detected edge portions, the movement of feature points in the surrounding image can be detected. When converting the image input to the feature image generation unit 25 into a feature image, a noise removal process may be performed on the input image. For example, normalization processes such as a median filter or binarization may be used to remove noise. The conversion of an image into a feature image may be performed using a process that also removes noise.

[0017] FIG. 3 shows a time-series feature image 31 obtained by converting the time-series image 29 in FIG. 2 into a feature image when the feature amount is luminance. In the time-series feature image 31 in FIG. 3, a plurality of row feature images, including row feature images 31t3 to 31t6 obtained by converting the row images 29t3 to 29t6 in FIG. 2, are arranged vertically in chronological order. Reference numeral 33 in FIG. 3 denotes a feature point. In FIG. 3, the feature point 33 is defined as a pixel where the differential value of luminance between adjacent pixels is a negative value and exhibits a gradient from light to dark. The feature point 33 may also be defined as a pixel where the differential value of luminance between adjacent pixels is a positive value and exhibits a gradient from dark to light. The feature point 33 may also be defined as the absolute value of the differential value of luminance between adjacent pixels. When the feature point 33 is defined as the differential value of luminance between adjacent pixels, the time-series feature image 31 is obtained by extracting edge portions from the image input to the feature image generation unit 25 as the feature points 33.

[0018] 1 , the determination image generating unit 11 of this embodiment first generates time-series images using the time-series image generating unit 23, and then converts the time-series images into feature images using the feature image generating unit 25. When the determination image generating unit 11 generates the time-series feature images 31, noise in the surrounding images can be efficiently removed when the feature image generating unit 25 converts the time-series images into feature images.

[0019] The straight-line component extraction unit 17 extracts straight-line components of feature points 33 that are successively arranged vertically across multiple segmented images from the time-series feature image 31 generated by the determination image generation unit 11. In the time-series feature image 31, feature points 33 that are successively arranged in the t-axis direction of the time-series feature image 31 refer to feature points 33 that are continuously present in the time series in a row of the surrounding image corresponding to the time-series feature image 31. Because the X-axis of the time-series feature image 31 is the horizontal axis of the surrounding image, if feature points 33 that are successively arranged in the t-axis direction move in the X-axis direction of the time-series feature image 31 in the time series, the feature points 33 that are continuously present in the time series move in the horizontal direction of the surrounding image. By estimating a straight line that approximates the set of feature points 33 that are successively arranged in the t-axis direction in the time-series feature image 31 and evaluating the slope of the approximated straight line with respect to the t-axis, it is possible to determine feature points 33 that move horizontally in a row of the surrounding image corresponding to the time-series feature image 31. The feature points 33 arranged vertically in the time-series feature image 31 are a set of points, not a line. The line component extraction unit 17 estimates a line approximating the set of feature points 33 and extracts the estimated line as a line component of the feature points 33. The line approximating the set of feature points 33 can be estimated using a known method. Below, a case where an approximation line of the set of feature points 33 is estimated by a Hough transform will be described. When an approximation line of the set of feature points 33 is estimated by a Hough transform, as shown in FIG. 4 , the estimated approximation line 34 is represented by a slope θ and a length x in the space of the X-t plane coordinate system in which the time-series feature image 31 exists. The slope θ is the angle between the X-axis of the time-series feature image 31 and a perpendicular line 35 drawn from the approximation line 34 to the origin (X, t) = (0, 0) of the X-t plane coordinate system, and the length x is the length of the perpendicular line 35. The slope θ is positive (θ>0) when the perpendicular line 35 is inclined clockwise with respect to the X-axis of the time-series feature image 31, and negative (θ<0) when the perpendicular line 35 is inclined counterclockwise with respect to the X-axis. A positive slope θ means that the set of feature points 33 approximated to the estimated straight line are feature points 33 that move leftward in time series within the row image corresponding to the time-series feature image 31 of the surrounding image. A negative slope θ means that the set of feature points 33 approximated to the estimated straight line are feature points 33 that move rightward in time series within the row image corresponding to the time-series feature image 31 of the surrounding image.The magnitude of the slope θ is proportional to the movement speed of the feature points 33 within the segmented images. The straight line component extraction unit 17 outputs a set of the slope θ of the approximate line 34 and the length x of the perpendicular line 35 estimated from one time-series feature image 31 by the Hough transform, and a score for the set as a straight line component, to the movement determination unit 19 as the straight line components of the feature points 33 extracted from the time-series feature image 31. For example, the score for the straight line component may be higher the longer the set of feature points 33 in the time-series feature image 31 is connected in the t-axis direction of the time-series feature image 31. The straight line components of the feature points 33 extracted by the straight line component extraction unit 17 include those extracted from time-series feature images 31 in rows not assigned with symbols in the time-series image 29 in FIG. 2 and those extracted from time-series feature images 31 in rows not shown in the time-series image 29 in FIG. 2. When a row of the surrounding image corresponding to the time-series feature image 31 contains multiple feature points 33 moving laterally at different positions, multiple straight lines approximating the set of feature points 33 are estimated from the single time-series feature image 31. When a set of feature points 33 in the time-series feature image 31 corresponding to one feature point 33 moving laterally does not exist on a straight line and the set of feature points 33 can be approximated by multiple straight lines, multiple straight lines approximating the set of feature points 33 are also estimated from the single time-series feature image 31. When multiple approximate straight lines are estimated from one time-series feature image 31, the straight line component extraction unit 17 outputs the slope θ, the length x of the perpendicular line, and the score of each approximate straight line to the movement determination unit 19 as multiple straight line components of the feature points 33 extracted from one time-series feature image 31.

[0020] The movement determination unit 19 determines moving feature points for each row of the surrounding image in FIG. 2 based on the straight line components of the feature points 33 in the time-series images for each row in the surrounding image in FIG. 2 input from the straight line component extraction unit 17. Moving feature points are feature points 33 that move horizontally within the row among the feature points 33. When there are multiple straight line components for the feature points 33 in each row, the movement determination unit 19 groups the straight line components into straight line components for each feature point 33 that move horizontally at different positions. Line components of feature points 33 that do not match in content but are considered to correspond to the same feature point 33 are sorted into the same group. Line components of feature points 33 that move horizontally at different positions are considered to differ in the slope θ of their approximation lines 34 or the length x of their perpendicular lines 35. Line components of feature points 33 that do not match in content but are considered to correspond to the same feature point 33 are considered to differ in the slope θ of their approximation lines 34 or the length x of their perpendicular lines 35, and therefore have different scores. The slope θ of the approximate line 34 and the length x of the perpendicular line 35 are parameters corresponding to the movement direction and linear component of the moving feature point, and can be used to group each linear component when there are multiple linear components of feature points 33 in each row. The movement determination unit 19 can sort the linear components of feature points 33 that move laterally at different positions based on the slope θ of the approximate line 34 and the length x of the perpendicular line 35. For each row of time-series feature images 31, the movement determination unit 19 plots points corresponding to the slope θ of the approximate line 34 of each linear component of feature points 33 extracted from the time-series feature images 31 and the length x of the perpendicular line 35 on the x-θ plane coordinates. The linear components of feature points 33 targeted for plotting points corresponding to the slope θ and the length x of the perpendicular line 35 on the x-θ plane coordinates may be limited to linear components having a score equal to or greater than a predetermined threshold. When the difference between the slope θ and the length x of the perpendicular to the approximation line 34 of the two line components of the feature point 33 is within a predetermined range, the movement determination unit 19 may classify the line components of the two feature points 33 into the same group, assuming that the parameter difference corresponding to the line components of the feature point 33 is within the predetermined range. Fig. 5 shows the distribution of the slope θ and the length x of the perpendicular to the approximation line 34 of the line components of the feature point 33 extracted from the time-series feature images 31 of Fig. 3 .In the quadrant on the x-θ plane coordinate system in Figure 5 where the slope θ is negative and the length x of the perpendicular is positive, and in the quadrant where the slope θ is positive and the length x of the perpendicular is positive, there are regions 37 and 39 where points corresponding to the slope θ of the line component and the length x of the perpendicular are concentrated. The points concentrated in region 37 in the quadrant with a negative slope θ correspond to a set of consecutive feature points 33 located on the right side of the time-series feature image 31 in Figure 3 and extending from the upper left to the lower right. The points concentrated in region 39 in the quadrant with a positive slope θ in Figure 5 correspond to a set of consecutive feature points 33 located on the left side of the time-series feature image 31 in Figure 3 and extending from the upper right to the lower left. The slope θ of the approximate line or the length x of the perpendicular indicated by each point in regions 37 and 39 in Figure 5 is the same or close to each other for each region 37 and 39. The line components of feature points 33 corresponding to the points concentrated in each region 37 and 39 are considered to be a set of line components corresponding to one feature point 33 moving in the horizontal direction. The movement determination unit 19 determines that the multiple line components of the feature points 33 corresponding to the points concentrated in region 37 and the multiple line components of the feature points 33 corresponding to the points concentrated in region 39 are each one moving feature point. The movement determination unit 19 outputs the slope θ of the approximation curve 34 for the feature points 33 determined to be moving feature points and information on the rows in the time-series feature images 31 corresponding to those feature points 33 to the moving object determination unit 21 as information on the moving feature points in the row images. The slope θ of the approximation curve 34 for the feature points 33 determined to be moving feature points can be used as information on the lateral movement direction and movement speed of the moving feature points. The moving object determination unit 21 can use the information on the moving feature points when determining the presence of an object moving around the vehicle 100, based on the lateral movement speed of the feature points 33 represented by the slope θ of the approximation curve 34 for the feature points 33 in the time-series feature images 31.

[0021] The moving object determination unit 21 determines the presence of an object moving around the vehicle 100. In particular, the moving object determination unit 21 determines an object moving laterally within the surrounding image based on the moving feature points determined for the time-series feature image 31 of each row, which is information on the moving feature points input from the movement determination unit 19. The moving object determination unit 21 clusters the moving feature points of each row determined by the movement determination unit 19 based on the direction and speed of movement in the lateral direction of the row image. The moving object determination unit 21 clusters the moving feature points and groups moving feature points with similar moving directions and speeds into the same cluster. The set of moving feature points in each grouped cluster is determined to be the moving feature points of a single object. For clustering of moving feature points, existing methods such as the k-means method, which is a non-hierarchical clustering algorithm, and hierarchical clustering can be used. In Fig. 6, feature points 33 corresponding to the two groups of moving feature points are assigned to two people 41 and 43 moving on the surrounding image 27t5 of Fig. 2. In Fig. 6, the moving object determination unit 21 groups the feature points 33 corresponding to the moving feature points into two groups based on similar moving directions and moving speeds. Each group is a collection of feature points 33 moving leftward and rightward within the surrounding image, respectively, and is indicated by different symbols (□, ■) in Fig. 6. The moving object determination unit 21 determines the person 41 moving to the left on the surrounding image 27t5 of Fig. 2 from the collection 45 of feature points 33 moving leftward, and determines the person 41 moving to the right on the surrounding image 27t5 from the collection 47 of feature points 33 moving rightward.

[0022] The flowchart in FIG. 7 illustrates a processing procedure performed by the microcontroller to recognize a moving object. As shown in FIG. 7 , the microcontroller acquires multiple time-series surrounding images captured multiple times by the imaging unit 5 (step S101). The time-series image generation unit 23 of the microcontroller extracts row images at predetermined vertical positions from each of the acquired time-series surrounding images (step S103). The microcontroller generates time-series images by arranging the extracted row images in chronological order (step S105). The microcontroller recognizes a moving object based on the difference in horizontal positions of feature points, at which feature amounts change, between the row images in the generated time-series images 29 (step S107). The microcontroller recognizes a moving object based on time-series feature images 31 generated using the time-series images 29 generated by the time-series image generation unit 23. The processing of step S107 can be performed by, for example, the feature image generation unit 25, the linear component extraction unit 17, the movement determination unit 19, and the moving object determination unit 21.

[0023] The flowchart in FIG. 8 shows an overview of the processing steps performed by the time-series image generating unit 23 of the microcontroller to generate the time-series images 29. The processing steps shown in FIG. 8 may be, for example, detailed processing steps performed by the time-series image generating unit 23 when performing the processing of steps S101 to S105 in FIG. 7 . As shown in FIG. 8 , the time-series image generating unit 23 acquires images captured by the imaging unit 5 as surrounding images, representing scenes around the vehicle 100 (step S201). The time-series image generating unit 23 stores the acquired surrounding images in the RAM of the microcontroller (step S203). The time-series image generating unit 23 checks whether the number of surrounding images stored in the RAM has reached the number required to generate the time-series images 29 (step S205). If the number of surrounding images stored in the RAM is insufficient for the required number (NO in step S205), the process returns to step S201. When the required number of surrounding images in the RAM is reached (YES in step S205), the time series image generation unit 23 generates a time series image 29 using row images at a predetermined vertical position extracted from the multiple time series surrounding images stored in the RAM (step S207).

[0024] The flowchart in FIG. 9 outlines the processing steps performed by the microcontroller when extracting multiple line components of feature points 33 from a time-series feature image 31 generated using the time-series images 29. The processing steps shown in FIG. 9 may be, for example, detailed processing steps performed by the feature image generation unit 25 and the line component extraction unit 17 when performing part of the processing of step S107 in FIG. 7 . As shown in FIG. 9 , the feature image generation unit 25 removes noise from the time-series images 29 generated by the time-series image generation unit 23 (step S301). For example, normalization processing such as a median filter or binarization can be used to remove noise. The feature image generation unit 25 converts the noise-removed time-series images 29 into feature images A and B using kernels A and B of a Sobel filter, respectively (steps S303 and S305). The conversion to feature image A results in a time-series feature image 31 in which horizontal edge portions of the time-series images 29 are extracted. By converting the time-series image 29 into feature image B, a time-series image 31 is obtained by extracting vertical edge portions of the time-series image 29. In steps S303 and S305, pixels having a gradient from light to dark, whose pixel values ​​become negative as a result of applying the Sobel filter, are extracted. In steps S303 and S305, pixels having a gradient from dark to light, whose pixel values ​​become positive as a result of applying the Sobel filter, may also be extracted. In steps S303 and S305, both pixels having positive and negative pixel values ​​as a result of applying the Sobel filter may also be extracted. Either step S303 or step S305 may be performed first, or both may be performed in parallel. Either step S303 or step S305 may be omitted. The straight line component extraction unit 17 performs a Hough transform on each of the time-series feature images 31 obtained by the feature image generation unit 25 converting the time-series image 29 into feature image A and feature image B (steps S307 and S309). The straight line component extraction unit 17 can estimate an approximate straight line 34 of a set of feature points 33 of each time-series feature image 31 by performing a Hough transform on each time-series feature image 31 .If either one of the processes of step S303 and step S305 is omitted, either one of the processes of step S307 and step S309 may be omitted. The straight line component extraction unit 17 stores the slope θ of the approximation line 34 of the feature points 33 extracted by performing a Hough transform on the time-series feature image 31, the length x of the perpendicular line 35, and the score in the RAM of the memory of the microcontroller (step S311).

[0025] In this embodiment, row images at predetermined vertical positions extracted from multiple time-series surrounding images are arranged in time series to generate a time-series image 29, thereby removing information about the vertical movement of objects in the surrounding images and arranging information about the horizontal movement of the objects in time series. The time-series image 29 is converted into feature images to generate a time-series feature image 31, and feature points 33 whose feature amounts change in the horizontal direction are extracted for each row image of the time-series image 29. The slope of the linear component of the feature point 33 in the time-series feature image 31 represents the horizontal movement speed of the point corresponding to the feature point 33 at a predetermined vertical position in the surrounding images. Therefore, this slope can be used to determine the moving feature point, which is the moving feature point 33 in the surrounding images. The determined moving feature point can be used to recognize moving objects around the vehicle 100 based on the horizontal movement speed of the feature point represented by the slope of the linear component of the feature point 33 in the time-series feature image 31. In this embodiment, a moving object in the background that is partially hidden by a foreground object can be effectively recognized. In this embodiment, the time-series image generation unit 23 extracts line images at predetermined positions from the input image to generate a time-series image 29, and the feature image generation unit 25 converts the input image directly into a feature image without extracting line images at predetermined positions from the input image. It is efficient to perform image noise removal processing on the entire image. By performing the processing of the time-series image generation unit 23 first in the judgment image generation unit 11, noise removal processing on the time-series feature image 31 can be efficiently performed when the processing of the feature image generation unit 25 is performed after the processing of the time-series image generation unit 23, which extracts line images from the input image. In this embodiment, the feature image generation unit 25 generates images representing edge portions in the surrounding image as feature images, and enables tracking of moving objects in the surrounding image based on time-series changes in the edge portions. Tracking moving objects in the surrounding image using optical flow requires pixel information from a wider area than edge detection in order to recognize the moving object as a surface. Detecting the movement of an object in the surrounding image by determining the time-series changes in the edge portion makes it possible to detect the movement of a small object more easily than tracking the movement of an object using optical flow.

[0026] Since the time-series feature image 31 has row images arranged vertically in chronological order, if the linear component of the feature point 33 extracted from the time-series feature image 31 is not tilted and extends vertically, the feature point 33 on the surrounding image has not moved horizontally in time series. The tilt direction of the linear component of the feature point 33 indicates whether the feature point 33 on the surrounding image has moved horizontally in time series, left or right. Based on whether the feature point 33 on the surrounding image has moved horizontally in time series, the moving feature points can be grouped for each object moving in the surrounding image. The tilt angle of the linear component of the feature point 33 includes information on the amount of horizontal movement of the feature point 33 in the surrounding image and information on the time in the time series, and indicates the movement speed of the feature point 33 that moved horizontally in the surrounding image. Based on the horizontal movement speed of the feature point 33 on the surrounding image, the moving feature points can be grouped for each object moving in the surrounding image. The multiple moving feature points determined based on the gradient of the linear components of the feature points 33 extracted from each row of the time-series feature image 31 may include moving feature points corresponding to different objects in the surrounding image. When the multiple moving feature points are grouped based on parameters corresponding to the moving direction of the moving feature points in each row of the time-series feature image 31 and the linear components of the feature points 33, moving feature points that have the same moving direction and are close to each other are sorted into the same group. Moving feature points that have close moving directions and positions in the time-series feature image 31 may be moving feature points corresponding to a common object in the surrounding image. If the score of the grouped moving feature points is equal to or greater than a predetermined determination threshold, it can be determined that there is an object moving in the surroundings that corresponds to the grouped moving feature points. If the difference between the gradient θ of the approximated line 34 and the length x of the perpendicular line 35, which are parameters corresponding to the linear components of the feature points 33 in the time-series feature image 31, is within a predetermined range, the positions of the two moving feature points are considered to be close. Since the difference in the gradient θ of the approximated line 34 is within a predetermined range, the difference in the moving speed of the two moving feature points in the surrounding image is also considered to be small. In this case, the two moving feature points may correspond to a common object on the surrounding image. If the two moving feature points are close to each other on the surrounding image and the difference in their moving speeds is within a predetermined value, the two moving feature points can be grouped together and determined to correspond to a common object on the surrounding image.

[0027] [Second Embodiment] A moving object recognition device 1 of a second embodiment shown in Figure 10 can implement a moving object recognition method according to the second embodiment. The moving object recognition device 1 of this embodiment can recognize an object moving outward from the boundary of a structure that is an occlusion in an image of the surroundings of the vehicle 100. In the moving object recognition device 1 of this embodiment, multiple information processing circuits of a microcontroller can configure the units 11, 15 to 21 of the moving object recognition device 1 of the first embodiment, as well as the structure estimation unit 13 and structure movement calculation unit 15. Of course, the information processing circuit may also be configured by providing dedicated hardware for executing the information processing of the units 13 and 15 described below.

[0028] The structure estimation unit 13 estimates occlusions in the surrounding image. Occlusions are objects that hide moving objects in the surrounding image from view, including structures. Occlusions are, for example, stationary three-dimensional objects. Stationary three-dimensional objects may include, for example, permanently existing three-dimensional objects such as buildings, temporary three-dimensional objects such as construction sites and parked vehicles, and newly occurring permanent structures. The structure estimation unit 13 can, for example, estimate permanently existing structures such as buildings in advance using information acquired in advance. Permanently existing three-dimensional objects can be estimated using, for example, information from high-precision three-dimensional map data (HD map, High Definition MAP) and position information from a GNSS (Global Navigation Satellite System) sensor on the vehicle 100. Temporarily existing or newly occurring three-dimensional objects can be estimated, for example, from the parallax of images captured by the stereo camera of the object detection unit 3. Permanently existing three-dimensional objects may also be estimated from the parallax of images captured by the stereo camera of the object detection unit 3. The structure estimation unit 13 can estimate, for example, an object determined to have a volume equal to or greater than a predetermined value from images captured by the stereo camera of the object detection unit 3 as an occluding structure. Occlusions are not limited to structures. Occlusions may also include pedestrian jumping-out caution points. Pedestrian jumping-out caution points include, for example, intersections with poor visibility. Occlusions may move within the surrounding image due to the movement of the vehicle 100, for example. The structure movement calculation unit 15 calculates the movement direction and movement speed of the occluding structure within the surrounding image. The movement direction and movement speed of the occluding structure within the surrounding image can be calculated by tracking the movement of the structure estimated by the structure estimation unit 13 within the surrounding image. The movement of the structure within the surrounding image can be tracked using, for example, optical flow, template matching, etc. The movement of a structure in the surrounding image may be estimated using, for example, the distance from the imaging position to the structure detected by the object detection unit 3 and the result of detecting the self-position of the vehicle 100 by odometry.When an occluding structure moves within the surrounding image, the time-series image generation unit 23 may shift the position of the line image in the surrounding image used to generate the time-series image 29 or the time-series feature image 31 in at least one of the vertical and horizontal directions of the surrounding image. In this embodiment, the time-series image generation unit 23 generates the time-series image 29 before the feature image generation unit 25 converts the time-series image 29 into a feature image, so the time-series image generation unit 23 shifts the position of the line image used to generate the time-series image 29. The time-series image generation unit 23 may shift the position of the line image used to generate the time-series image 29 based on the movement direction and movement speed of the occluding structure in the surrounding image calculated by the structure movement calculation unit 15. The time-series image generation unit 23 may shift a predetermined position of the line image used to generate the time-series image 29 in the vertical direction of the surrounding image, or may shift a portion of the line image used to generate the time-series image 29 in the horizontal direction of the surrounding image.

[0029] The moving object determination unit 21 of this embodiment may determine an object moving laterally within the surrounding image by limiting the determination to a peripheral region around the boundary of the occluding structure. To this end, the movement determination unit 19 may determine moving feature points by limiting the determination to the peripheral region, and the straight line component extraction unit 17 may extract straight line components of feature points 33 by limiting the extraction to the peripheral region. To this end, the determination image generation unit 11 may generate time-series feature images 31 by limiting the generation to the peripheral region, and the generation of time-series images 29 by the time-series image generation unit 23 and the conversion of images into feature images by the feature image generation unit 25 may be performed by limiting the generation to the peripheral region.

[0030] The flowchart in FIG. 11 outlines the procedure for determining an object moving outside the boundary of an occluding structure. The microcontroller extracts tall objects present around the vehicle 100 from images captured by the stereo camera of the object detection unit 3 (step S401). The microcontroller estimates any extracted object determined to have a volume equal to or greater than a predetermined value as an occluding structure (step S403). The microcontroller extracts feature points of the occluding structure from the images captured by the object detection unit 3, tracks the extracted feature points over time, and calculates the moving speed of the occluding structure in the surrounding image captured by the image capture unit 5 (step S405). Optical flow, for example, can be used to track the feature points. The calculated moving speed includes information about the moving direction of the occluding structure in the surrounding image. The microcontroller extracts the boundary of the occluding structure. The boundary is the outline of the occluding structure. The extracted boundary includes at least a portion of the occluding structure that is a boundary of the occluding structure in the horizontal direction of the surrounding image. The time-series image generation unit 23 of the microcontroller generates a time-series image 29 for each extracted boundary (step S407). When the occluding structure moves within the surrounding image, the time-series image generation unit 23 generates the time-series image 29 while shifting the position of the row image in at least one of the vertical and horizontal directions of the surrounding image in accordance with the speed and direction of the movement (step S409). The processing of step S409 can be performed as a processing of correcting the time-series image 29 generated in the processing of step S407 as necessary. The microcontroller determines a moving object moving from the boundary of the occluding structure to the outside of the occluding structure based on the time-series feature image 31 generated using the time-series image 29 generated by the time-series image generation unit 23 (step S411). The process of step S411 can be performed by the feature image generating unit 25, the straight line component extracting unit 17, the movement determining unit 19, and the moving object determining unit 21. The microcontroller obtains the absolute movement speed of the moving object in the space around the vehicle 100 (step S413).The absolute speed of the object's movement can be calculated, for example, using the distance between the object and the image capture position detected by the object detection unit 3 and the movement of the image capture position based on the result of self-position detection of the vehicle 100 by odometry. The microcontroller determines a behavior plan for the vehicle 100 from the calculated absolute speed (step S415). The behavior plan for the vehicle 100 may be determined by a controller of the driving control system of the vehicle 100 instead of the microcontroller. The controller of the driving control system of the vehicle 100 may be, for example, an autonomous driving ECU (Electronic Control Unit).

[0031] In the area surrounding the boundary of a structure in the surrounding image, there is a possibility that an object may appear protruding from behind the structure. In this embodiment, by focusing on determining moving feature points in the surrounding image, particularly in the area surrounding the boundary of the structure, the presence of a moving object in the surroundings can be efficiently determined. If the structure is fixed, when the structure moves in the surrounding image, the imaging position of the surrounding image moves relative to the structure. When the imaging position of the surrounding image moves relative to the structure, the content of the row images when converting the surrounding image into a time-series image is offset in at least one of the vertical and horizontal directions corresponding to the movement of the imaging position of the surrounding image. By offsetting the row images arranged in chronological order in the time-series image in at least one of the vertical and horizontal directions based on the movement speed of the structure in the surrounding image, the offset of the row images caused by the movement of the imaging position is canceled. This cancellation ensures that the content of the row images captured during the movement of the imaging position and the content of the row images captured before or after the movement are consistent in at least one of the vertical and horizontal directions in the time-series image 29. The movement speed of a moving feature point in a surrounding image is the speed of the relative movement of the moving feature point with respect to the surrounding image. The relative movement of a moving feature point with respect to the surrounding image also includes the relative movement of the moving feature point due to movement of the imaging position of the surrounding image. By detecting the movement of the imaging position of the surrounding image and detecting the distance from the imaging position to the object corresponding to the moving feature point in the surrounding image, the relative movement speed of the moving feature point in the surrounding image can be converted into an absolute movement speed based on these. The converted absolute movement speed allows for accurate recognition of surrounding moving objects.

[0032] In each of the above embodiments, the moving object recognition device 1 is mounted on the vehicle 100, so that a moving object around the vehicle 100 can be suitably recognized on an ambient image capturing the periphery of the vehicle 100 captured from the vehicle 100, even if the moving object is partially hidden by an object in the foreground. Note that the moving object recognition method may also be executed using the moving object recognition device 1 without mounting the device on the vehicle 100. The above-described embodiment is an example of the present invention. Therefore, the present invention is not limited to the above-described embodiment, and various modifications can be made depending on the design, etc., even in embodiments other than the above-described embodiment, as long as they do not deviate from the technical concept of the present invention.

[0033] 1 Moving object recognition device, 17 Line component extraction unit, 19 Movement determination unit, 21 Moving object determination unit, 23 Time series image generation unit, 25 Feature image generation unit, 27t1 to 27t5 Surrounding images, 29 Time series images, 29t3 to 29t5 Segmented images, 31 Time series feature images, 33 Feature points, 41, 43 Person (moving object), 100 Vehicle

Claims

1. A moving object recognition method for a computer, which acquires a plurality of time-series surrounding images by capturing images of surroundings a plurality of times in a time series, and recognizes a moving object moving in the surroundings from the plurality of time-series surrounding images, extracting a row image at a predetermined position in the vertical direction from each of the plurality of surrounding images in the time series, and generating a time series image in which the extracted plurality of row images are arranged vertically in chronological order; generating a plurality of the time-series images in which the predetermined positions of the row images are different from each other; grouping feature points of the plurality of time-series images, which move between the row images and whose feature amounts change within the row images, based on a direction of movement of the lateral positions of the feature points and a parameter corresponding to a linear component that approximates the time-series change of the lateral positions; clustering the two feature points into the same group when a difference in the movement speed of the lateral positions of the two feature points that move between the row images is within a predetermined value and a difference in the parameters is within a predetermined range; recognizing the moving object based on a difference in lateral positions of the feature points between the row images of the time-series images clustered into the same group; A moving object recognition method comprising:

2. 2. The moving object recognition method according to claim 1, wherein the feature points are extracted from each of the plurality of row images of the time-series images, and the difference in the lateral positions of the extracted feature points between the row images is detected.

3. The moving object recognition method according to claim 1 or 2, wherein edge portions in the row image are extracted as the feature points.

4. 4. The moving object recognition method according to claim 1, wherein the feature points in the row image are extracted based on a negative change amount in the horizontal direction of the feature amount.

5. The moving object recognition method according to any one of claims 1 to 4, wherein the moving object is recognized based on the movement speed of the feature point whose lateral position moves between the row images of the time-series images.

6. The moving object is recognized when a score of the grouped feature points, which increases as the predetermined positions of the grouped feature points are connected in a longer line in the vertical direction, is equal to or greater than a predetermined determination threshold. The moving object recognition method according to claim 1 .

7. The moving object is recognized in a peripheral area of ​​a boundary of a structure present in the surrounding image. The moving object recognition method according to any one of claims 1 to 6.

8. The moving object recognition method according to claim 7 , wherein the row images are arranged in the time-series images at positions vertically offset from the predetermined position based on the moving speed of the structure on the surrounding image.

9. A moving object recognition device that acquires a plurality of time-series surrounding images obtained by capturing images of surroundings multiple times in a time series, and recognizes a moving object moving in the surroundings from the acquired plurality of time-series surrounding images, a time-series image generating unit that extracts row images at predetermined positions in a vertical direction from each of the plurality of surrounding images in the time series, and generates a time-series image in which the extracted plurality of row images are arranged vertically in chronological order; a moving object determination unit that determines the presence of the moving object based on a difference in lateral position of a feature point where a feature amount in the row image changes between the row images of the time-series images; Equipped with the time-series image generation unit generates a plurality of the time-series images in which the predetermined positions of the row images are different from one another; a movement determination unit that classifies feature points of the plurality of time-series images, which move between the row images and whose feature amounts change within the row images, into groups based on a movement direction of the lateral position of the feature points and a parameter corresponding to a linear component that approximates the time-series change of the lateral position; a moving object determination unit that clusters the two feature points into the same group when a difference in the movement speed of the lateral positions of the two feature points that move between the row images is within a predetermined value and a difference in the parameters is within a predetermined range; Furthermore, the moving object determination unit recognizes the moving object based on a difference in lateral positions of the feature points between the row images of the time-series images clustered into the same group. Moving object recognition device.