Information processing device, information processing method, and program

Abstract

To appropriately estimate a vehicle's lane of travel even when road markings cannot be detected from a road surface image.SOLUTION: An information processing device includes: a detection unit configured to detect, from image data captured around a moving object, one or more other moving objects; a calculation unit configured to calculate a relative position of the other moving object in a lateral direction of the moving object and a relative movement amount of the other moving object in a longitudinal direction of the moving object; a clustering unit configured to cluster the other moving object based on the relative position of the other moving object in the lateral direction and the relative movement amount of the other moving object in the longitudinal direction; an identification unit configured to identify a lane area based on clustering results; and an estimation unit configured to estimate a center line of the lane on which the moving object is traveling based on the lane area.SELECTED DRAWING: Figure 1

Description

Technical Field

[0006] , , ,

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, a technique for detecting a lane in which a vehicle travels has been known. For example, in Patent Document 1, candidate lines that are candidates for lane lines are extracted from a road surface image, the line type, line color, and presence or absence of the influence of backlight of the extracted candidate lines are determined, and it is determined whether the extracted candidate lines constitute a multiple line. Using these determination results, a candidate line that becomes a lane line is selected, the selected candidate line is recognized, and a technique for estimating the shape of a lane is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technique described in Patent Document 1 is premised on extracting candidate lines that are candidates for lane lines from a road surface image. However, for example, when a plurality of other vehicles are traveling around the host vehicle or when the road lane lines are not clear in the first place, there are cases where the traveling lane of the host vehicle cannot be appropriately estimated.

[0005] The present invention has been made in consideration of such circumstances, and an object thereof is to provide an information processing apparatus, an information processing method, and a program that can appropriately estimate the traveling lane of the host vehicle even when the road lane lines cannot be detected from a road surface image.

Means for Solving the Problems

[0006] The information processing apparatus, information processing method, and program according to this invention employ the following configuration. (1) An information processing device according to one aspect of the present invention includes: a detection unit that detects one or more other moving objects from image data captured around a moving object; a calculation unit that calculates the relative position of the other moving objects with respect to the moving object in the lateral direction and the relative amount of movement of the other moving objects with respect to the moving object in the vertical direction; a clustering unit that clusters the other moving objects based on the relative position of the other moving objects in the lateral direction and the relative amount of movement of the other moving objects in the vertical direction; a specification unit that identifies a lane area based on the results of the clustering; and an estimation unit that estimates the center line of the lane in which the moving object is traveling based on the lane area.

[0007] (2): In the embodiment of (1) above, the identification unit identifies, based on the clustering results, an on-lane region representing the area of ​​the on-lane in which the moving body travels, and an on-lane region representing the area opposite the moving body, and the estimation unit estimates the center line as the line between the on-lane region and the on-lane region.

[0008] (3) In the embodiment of (2) above, the identifying unit identifies a group of mobile bodies approaching the mobile body from among a plurality of mobile body groups belonging to each of the plurality of clusters obtained by the clustering, and identifies the region including the cluster to which the identified mobile body group belongs as the opposing lane region.

[0009] (4): In the embodiment of (1) above, the estimation unit estimates a first vanishing point from the edge of the lane area and estimates the other moving body furthest from the moving body as a second vanishing point, and identifies the first vanishing point or the second vanishing point as a vanishing point if the distance between the first vanishing point and the second vanishing point is within a threshold.

[0010] (5) In the embodiment of (1) above, if the estimation unit determines that the moving body is traveling on a curved road, it estimates a first vanishing point from the edge of the lane area and estimates the other moving body furthest from the moving body as a second vanishing point, and identifies the first vanishing point or the second vanishing point as a vanishing point if the distance between the first vanishing point and the second vanishing point is within a threshold.

[0011] (6) In the embodiment of (5) above, if the estimation unit determines that a segment exists on the center line, it determines that the moving body is traveling on a curved road.

[0012] (7) In the embodiment of (1) above, the clustering unit clusters one or more other moving objects detected from the image data of multiple frames captured over time.

[0013] (8) In the embodiment of (1) above, the clustering unit clusters the other moving bodies based on the relative position of the other moving bodies in the lateral direction and the moving average of the relative movement of the other moving bodies in the vertical direction.

[0014] (9): In the embodiment of (1) above, the clustering unit clusters only the other mobile bodies which are four-wheeled vehicles.

[0015] (10): In the embodiment of (1) above, the detection unit detects the other moving body located behind the moving body, and the information processing device further comprises a movement direction estimation unit that estimates the movement direction of the other moving body based on the relative position of the other moving body in the lateral direction and the relative amount of movement of the other moving body in the longitudinal direction, and a determination unit that determines whether or not the other moving body will overtake based on the estimated movement direction of the other moving body and the lane area.

[0016] (11): In the embodiment of (10) above, the determination unit further comprises a notification unit that notifies the occupants of the moving body of the overtaking when it is determined that the other moving body is overtaking the moving body.

[0017] (12): An information processing method according to another aspect of the present invention involves a computer detecting one or more other moving objects from image data captured around a moving object, calculating the relative position of the other moving objects with respect to the moving object in the lateral direction and the relative amount of movement of the other moving objects with respect to the moving object in the vertical direction, clustering the other moving objects based on the relative position of the other moving objects in the lateral direction and the relative amount of movement of the other moving objects in the vertical direction, identifying a lane region based on the clustering results, and estimating the center line of the lane in which the moving object is traveling based on the lane region.

[0018] (13): A program according to another aspect of the present invention causes a computer to detect one or more other moving objects from image data captured around a moving object, calculate the relative position of the other moving objects with respect to the moving object in the lateral direction and the relative amount of movement of the other moving objects with respect to the moving object in the vertical direction, cluster the other moving objects based on the relative position of the other moving objects in the lateral direction and the relative amount of movement of the other moving objects in the vertical direction, identify a lane area based on the clustering results, and estimate the center line of the lane in which the moving object is traveling based on the lane area. [Effects of the Invention]

[0019] According to the embodiments of (1) to (13), even when road markings cannot be detected from the road surface image, the vehicle's lane can be appropriately estimated. [Brief explanation of the drawing]

[0020] [Figure 1] This figure shows an example of the operating environment for the terminal device 100 installed in the vehicle M. [Figure 2]It is a diagram showing an example of the configuration of the terminal device 100. [Figure 3] It is a diagram showing an example of one or more other vehicles detected by the detection unit 110 from the captured image IM. [Figure 4] It is a diagram showing an example of a scene where the calculation unit 120 calculates the relative position between the host vehicle M and the other vehicle Mk. [Figure 5] It is a diagram for explaining a method by which the calculation unit 120 calculates the vertical position between the host vehicle M and the other vehicle. [Figure 6] It is a diagram for explaining a method by which the calculation unit 120 calculates the horizontal position between the host vehicle M and the other vehicle. [Figure 7] It is a diagram showing an example of a bird's-eye view generated by the calculation unit 120. [Figure 8] It is a diagram for explaining an outline of clustering executed by the clustering unit 130. [Figure 9] It is a diagram for explaining a method by which the specifying unit 140 specifies a lane area including the host vehicle M. [Figure 10] It is a diagram showing an example of the center line and vanishing point estimated by the estimation unit 150. [Figure 11] It is a diagram for explaining the processing of the estimation unit 150 when the host vehicle M travels on a curved road. [Figure 12] It is a flowchart showing an example of the processing flow executed by the terminal device 100. [Figure 13] It is a diagram showing an example of the usage environment of the terminal device 200 mounted on the host vehicle M according to the second embodiment. [Figure 14] It is a diagram showing an example of the configuration of the terminal device 200 according to the second embodiment. [Figure 15] It is a diagram for explaining details of overtaking determination and notification processing. [Figure 16] It is a flowchart showing an example of the processing flow executed by the terminal device 200.

Embodiments for Carrying Out the Invention

[0021] Hereinafter, embodiments of the information processing apparatus, information processing method, and program of the present invention will be described with reference to the drawings.

[0022] [composition] Figure 1 shows an example of the operating environment of the terminal device 100 mounted on the vehicle M. The vehicle M is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its power source is an internal combustion engine such as a diesel engine or gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a generator connected to the internal combustion engine, or power discharged from a secondary battery or fuel cell.

[0023] As shown in Figure 1, the terminal device 100 is installed on the vehicle M so as to be able to image the area in front of the vehicle M in the direction of travel. The terminal device 100 is, for example, a computer device such as a smartphone or a tablet terminal. The terminal device 100 is held by, for example, an in-vehicle holder (not shown) attached to the dashboard of the vehicle M, and images the area in front of the vehicle M.

[0024] Figure 2 shows an example of the configuration of the terminal device 100. As shown in Figure 2, the terminal device 100 includes, for example, a camera 10, a display unit 20, a detection unit 110, a calculation unit 120, a clustering unit 130, a identification unit 140, and an estimation unit 150. The detection unit 110, the calculation unit 120, the clustering unit 130, the identification unit 140, and the estimation unit 150 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or by the cooperation of software and hardware. The program may be stored in advance on a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device equipped with a non-transient storage medium), or it may be stored on a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is inserted into a drive device. In the following description, the functions of the detection unit 110, the calculation unit 120, the clustering unit 130, the identification unit 140, and the estimation unit 150 may be collectively referred to as the "information processing application." The information processing application is installed on the terminal device 100 and is activated, for example, when the user of the terminal device 100 starts driving their vehicle M. Furthermore, the terminal device 100 that installs the information processing application is an example of an "information processing device."

[0025] Camera 10 is a digital camera that uses a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Display unit 20 is a display device such as a touch panel or liquid crystal display. Display unit 20 displays the estimation results from the estimation unit 150, which will be described later. The user of terminal device 100 positions the terminal device 100 to a predetermined height (initial height of vanishing point V, which will be described later) according to the guide lines displayed on the display unit 20. Figure 1 shows the terminal device 100 positioned, capturing an image of the area near the upper end of another vehicle M1 of the user's vehicle M, in a substantially horizontal direction relative to the road surface.

[0026] [Detection unit] The detection unit 110 recognizes objects captured within the image IM taken by the camera 10. More specifically, for example, the detection unit 110 detects objects using a trained model that has been trained to output information such as the presence, location, and type of an object when an image taken by the camera 10 is input. Using this trained model, the detection unit 110 detects one or more other vehicles in the captured image IM while distinguishing between types such as motorcycles and automobiles.

[0027] Figure 3 shows an example of one or more other vehicles detected by the detection unit 110 from the captured image IM. In Figure 3, the symbols M1 to M3 represent four-wheeled vehicles detected by the detection unit 110, and the symbols B1 to B3 represent two-wheeled vehicles detected by the detection unit 110. The terminal device 100 displays the other vehicles detected by the detection unit 110 on the display unit 20, for example, by enclosing them with a bounding box. In Figure 3, the detected four-wheeled and two-wheeled vehicles are displayed with the same bounding box, but the two-wheeled and four-wheeled vehicles may be displayed on the display unit 20 in different display modes.

[0028] [Calculation Section] When the detection unit 110 detects one or more other vehicles, the calculation unit 120 further calculates the vertical and horizontal positions of one or more other vehicles relative to the vehicle M (hereinafter, the combination of vertical and horizontal positions may be referred to as "relative position"). Figure 4 is a diagram showing an example of the calculation unit 120 calculating the relative position between the vehicle M and the other vehicle Mk. In Figure 4, the symbol hA indicates the height from the road position corresponding to the lower end of the display unit 20 to the vanishing point V of the image, and the symbol hB indicates the height from the road position corresponding to the lower end of the detected other vehicle M1 to the vanishing point V of the image.

[0029] Figure 5 is a diagram illustrating how the calculation unit 120 calculates the longitudinal position between its own vehicle M and another vehicle. In Figure 5, the symbol IS indicates the image sensor included in the camera 10, the symbol D indicates the display included in the camera 10 (the end of the camera 10), the symbol O indicates the central part of the image sensor, the symbol A indicates the position on the display D corresponding to the road position F projected onto the bottom of the display D, the symbol B indicates the position on the display D corresponding to the road position G of the rear end of the other vehicle M1, the symbol C indicates the intersection of the imaging direction by the image sensor and the display D, the symbol H indicates the height of the camera 10 relative to the road surface, the symbol DA indicates the distance from the position of the image sensor IS to the road position F projected onto the bottom of the display D, and the symbol DB indicates the distance from the position of the image sensor IS to the road position G of the rear end of the other vehicle M1.

[0030] In Figure 5, triangles OAC and OEF are similar, and triangles OBC and OEG are similar. That is, with respect to distance, L:hA=DA:H and L:hB=DB:H hold true, so by rearranging, we obtain DA=L×H / hA and DB=L×H / hB. Therefore, the calculation unit 120 can calculate the distance to the road position G of the rear end of the other vehicle M1 using the calculation formula DB=DA×hA / hB. Here, the heights hA and hB to the vanishing point are calculated in advance based on the image captured by the camera 10, and the distance DA, which does not depend on the position of the other vehicle M1, can be calculated in advance according to the installation position of the terminal device 100. Note that the above calculation can be performed using only the height information of the vanishing point, without requiring all of the coordinate information of the vanishing point.

[0031] The calculation unit 120 further calculates the lateral position between its own vehicle M and the other vehicle. Figure 6 is a diagram illustrating how the calculation unit 120 calculates the lateral position between its own vehicle M and the other vehicle. In Figure 6, the symbol V indicates the vanishing point of the image, the symbol Wb indicates the number of pixels in the lateral direction relative to the vanishing point V of the other vehicle M1, and the symbol Wa indicates the number of pixels when the number of pixels Wb is moved to the bottom edge of the display D.

[0032] In Figure 6, triangle VT'T and triangle VS'S are similar. That is, with respect to distance, Wa:hA == Wb:hB holds, so by rearranging, we obtain Wa = Wb × hA / hB. Here, assuming that the total number of pixels Wsc at the bottom edge of the display D and the road width Wrd on which the vehicle M is traveling are known, the calculation unit 120 can calculate the actual lateral distance W corresponding to the number of pixels Wa using the formula W = Wrd × Wa / Wsc. In this way, the calculation unit 120 calculates the relative position between the vehicle M and the other vehicle M1.

[0033] The calculation unit 120 calculates the relative positions of one or more other vehicles relative to its own vehicle M, and then maps these one or more other vehicles to a bird's-eye view relative to its own vehicle M. Figure 7 shows an example of a bird's-eye view generated by the calculation unit 120. The left side of Figure 7 shows a screen where, based on the situation shown in Figure 3, the calculation unit 120 calculates the relative positions of one or more other vehicles relative to its own vehicle M and maps them to a bird's-eye view.

[0034] The calculation unit 120 further calculates the relative position of other vehicles for T frames (T is a positive integer) of images captured in time series by the camera 10, and calculates the relative movement of other vehicles relative to the own vehicle M based on the difference in these time-series relative positions. More specifically, the calculation unit 120 calculates the relative position of other vehicles at time t(k) (k is an integer between 1 and T) and the relative movement of other vehicles over the period t(k)-t(k-1). The arrows shown on the right side of Figure 7 represent the direction and magnitude of the relative movement of each other vehicle.

[0035] [Clustering section] The clustering unit 130 clusters one or more other vehicles based on the relative position of other vehicles in the lateral direction of its own vehicle M and the relative movement amount of other vehicles in the vertical direction of its own vehicle M, which are calculated by the calculation unit 120.

[0036] Figure 8 is a diagram illustrating the overview of clustering performed by the clustering unit 130. In this embodiment, the clustering unit 130 represents the relative vertical movement and lateral position calculated for each other vehicle on a two-dimensional graph, and clusters the other vehicles by applying the k-means method to these values. Figure 8 shows the clustering result with the lateral position as the X coordinate, the relative movement as the Y coordinate, and k=2. More generally, to determine the value of k, the clustering unit 130 may calculate the distance between the mean value of each cluster and each data value when clustering is performed for each k value (e.g., 2 to 5), take the sum of these distances, and adopt the k value that minimizes the sum of distances.

[0037] In this embodiment, as shown in Figure 8, the clustering unit 130 performs clustering only on four-wheeled vehicles among the other detected vehicles. This is because, generally, four-wheeled vehicles have smoother trajectories and a stronger tendency to travel along the lane being estimated compared to two-wheeled vehicles. By limiting the target of clustering to four-wheeled vehicles, the accuracy of the identified vanishing points can be improved compared to when two-wheeled vehicles are included. Alternatively, the clustering unit 130 may perform clustering including two-wheeled vehicles as well as four-wheeled vehicles. When clustering is performed including two-wheeled vehicles, the clustering unit 130 may determine whether there are more than a predetermined number of four-wheeled vehicles in the captured image, and only if it is determined that there are fewer than a predetermined number of four-wheeled vehicles, it may perform clustering including two-wheeled vehicles.

[0038] Furthermore, the present invention is not limited to the k-means method, and clustering may be performed using other unsupervised algorithms (e.g., Gaussian mixture models, hypervolume methods, etc.). Also, Figure 8 shows the case where the lateral position of other vehicles at time t(k) and the relative movement of other vehicles over the period t(k)-t(k-1) are clustered. In order to stabilize the clustering results, clustering may be performed on the moving average value of the lateral position and / or relative movement starting from time t(1) for each other vehicle. This will stabilize the clustering results. Alternatively, instead of the moving average value, for example, a Bayesian estimator may be derived from time-series observations of relative movement and lateral position, and clustering may be performed on the derived Bayesian estimator.

[0039] [Specific part] Figure 9 is a diagram illustrating how the identification unit 140 identifies the lane area including the vehicle M. When the clustering unit 130 obtains each cluster, the identification unit 140 determines the width of the cluster, identifies the left and right ends of the determined width as the left road lane line and the right road lane line, respectively (hereinafter, the combination of the left road lane line and the right road lane line may be referred to as an "edge"), and identifies the area enclosed by the left road lane line and the right road lane line as the lane area. In the case of Figure 9, the identification unit 140 identifies the left road lane line LL and the right road lane line CL for cluster C1, and identifies the area enclosed by these left road lane line LL and right road lane line CL as the lane area LD. Furthermore, the identification unit 140 identifies the left road lane line CL and the right road lane line RL for cluster C2, and identifies the area enclosed by these left road lane line CL and right road lane line RL as the lane area RL.

[0040] When the identification unit 140 identifies one or more lane regions, it identifies the region representing the lane in which the vehicle M is traveling as the vehicle's own lane region, and identifies the lane regions other than the vehicle's own lane region as other lane regions. The identification unit 140 further determines whether the other lane region is an oncoming lane region representing a region facing the vehicle M, based on the relative movement of the other vehicles constituting each cluster. More specifically, for example, the identification unit 140 can determine that a cluster is an oncoming lane region if it determines that the other vehicle is approaching the vehicle M (i.e., if it determines that the relative movement is negative), based on the relative movement of the other vehicles constituting each cluster. If a cluster includes multiple other vehicles, the identification unit 140 may determine that the cluster is an oncoming lane region if, for example, the sum of the relative movement amounts is negative, or it may determine whether the cluster is an oncoming lane region by a majority vote based on the number of other vehicles with negative relative movement amounts.

[0041] [Estimation part] The estimation unit 150 estimates the center line of the lane in which the vehicle M is traveling, based on the lane region identified by the identification unit 140. More specifically, for example, if there are multiple lane regions identified by the identification unit 140, the estimation unit 150 estimates the line between the identified lane regions as the center line. For example, in the case of Figure 9, the estimation unit 150 estimates the line CL passing between lane region LD and lane region RD as the center line. In particular, if the identification unit 140 identifies the vehicle's own lane region and the opposing lane region, the estimation unit 150 can estimate the line CL passing between the vehicle's own lane region and the opposing lane region as the center line.

[0042] Figure 10 shows an example of a center line and vanishing point estimated by the estimation unit 150. When the estimation unit 150 estimates the center line CL, the terminal device 100 displays the estimated center line CL on the display unit 20. At this time, as shown in Figure 10, the terminal device 100 also displays the left road lane marking CL and the right road lane marking RL in addition to the estimated center line CL. The driver of the vehicle M can determine the lane in which the vehicle M is traveling by referring to these center line CL, left road lane marking CL, and right road lane marking RL. In other words, according to this embodiment, even if road lane markings cannot be detected from the road surface image, the vehicle's lane can be appropriately estimated.

[0043] When the left road lane marking CL and the right road lane marking RL are identified by the identification unit 140, the estimation unit 150 identifies the intersection of these identified left road lane marking CL and right road lane marking RL (i.e., edges) as the vanishing point V. Alternatively, the estimation unit 150 may identify the intersection of either the left road lane marking CL or the right road lane marking RL with the center line CL as the vanishing point V. Once the vanishing point V is identified in this way, the detection unit 110 and the calculation unit 120 use the identified vanishing point V to perform the process described through Figures 3 to 7 again and regenerate the bird's-eye view. By repeating this process, the accuracy of the bird's-eye view can be improved.

[0044] [Presence of segmental points] The process described above is applicable regardless of whether the road on which the vehicle M is traveling is a straight road or a curved road. However, it has been found that when the vehicle M is traveling on a curved road, the lane estimation based on the clustering described above is unstable. Therefore, the estimation unit 150 determines whether or not there is a point on the road on which the vehicle M is traveling that marks the transition from a straight road to a curved road. More specifically, for example, the estimation unit 150 may calculate the second derivative of the center line CL identified by the identification unit 140 and estimate the point where the sign of the calculated derivative changes as the point on the lane DP. Alternatively, for example, the estimation unit 150 may calculate the curvature of the center line CL identified by the identification unit 140 and estimate the point where the calculated curvature is greater than or equal to a threshold as the point on the lane DP.

[0045] If the estimation unit 150 determines that a segmentation point exists, it considers the vanishing point V1 as the intersection of the left road lane marking CL and the right road lane marking RL mentioned above, as well as the other vehicle furthest from its own vehicle M as vanishing point V2, and verifies the reliability of the lane estimation based on clustering by determining whether or not there is a discrepancy between these vanishing points V1 and V2.

[0046] Figure 11 is a diagram illustrating the processing of the estimation unit 150 when the vehicle M is traveling on a curved road. As an example, Figure 11 shows the case where clusters C1 to C4 are detected as a result of clustering.

[0047] If the estimation unit 150 determines that a segmentation point exists, it considers the other vehicle M1 furthest from its own vehicle M as vanishing point V2. Next, the estimation unit 150 determines whether the distance between vanishing point V1, which is the intersection of the left road lane marking CL and the right road lane marking RL, and vanishing point V2, which is the other vehicle M1, is within a threshold. If the estimation unit 150 determines that the distance between vanishing point V1 and vanishing point V2 is within a threshold, it identifies either vanishing point V1 or vanishing point V2 as vanishing point V. Once vanishing point V is identified in this way, the detection unit 110 and the calculation unit 120 use the identified vanishing point V to re-execute the process described in Figures 3 to 7 and regenerate the bird's-eye view. By repeating this process, the accuracy of the bird's-eye view can be improved.

[0048] In this embodiment, as an example, the terminal device 100 is installed to image the area in front of the vehicle M and to identify the vanishing point in the area in front of the vehicle M. However, the present invention is not limited to such a configuration, and the terminal device 100 may be installed to image the area behind the vehicle M and to identify the vanishing point in the area behind the vehicle M. In that case, the generated bird's-eye view will show the area behind the vehicle M, and the occupants of the vehicle M can check the situation behind the vehicle M by checking the bird's-eye view. Alternatively, for example, two or more terminal devices 100 may be installed to image the area in front of and the area behind the vehicle M, and the bird's-eye views of the front and rear areas generated by these two or more terminal devices 100 may be integrated and displayed on either terminal device 100 or the navigation device of the vehicle M.

[0049] [Process Flow] Next, with reference to Figure 12, the processing flow performed by the terminal device 100 will be described. Figure 12 is a flowchart showing an example of the processing flow performed by the terminal device 100. The processing shown in the flowchart in Figure 12 is repeatedly performed, for example, while the vehicle M is in motion.

[0050] First, the detection unit 110 detects multiple other vehicles in the T-frame images captured in time series by the camera 10 (step S100). Next, the calculation unit 120 calculates the relative position and relative movement of the detected other vehicles (step S102). Next, the clustering unit 130 clusters the other vehicles based on the calculated relative position and relative movement (step S104). Next, the identification unit 140 identifies the lane area based on the clustering results (step S106). Next, the estimation unit 150 estimates the center line based on the identified lane area (step S108).

[0051] Next, the estimation unit 150 determines whether or not a segmentation point exists on the estimated center line (step S110). If it is determined that no segmentation point exists on the estimated center line, the estimation unit 150 identifies a vanishing point V as the intersection of the edges of the identified lane area (step S112). On the other hand, if it is determined that a segmentation point exists on the estimated center line, the estimation unit 150 estimates a first vanishing point V1 as the intersection of the edges of the lane area, and also estimates a distant vehicle as a second vanishing point V2 (step S114).

[0052] Next, the estimation unit 150 determines whether the distance between the first vanishing point V1 and the second vanishing point V2 is within a threshold (step S116). If it is determined that the distance between the first vanishing point V1 and the second vanishing point V2 is not within a threshold, the terminal device 100 returns to step S100. On the other hand, if it is determined that the distance between the first vanishing point V1 and the second vanishing point V2 is within a threshold, the estimation unit 150 identifies either the first vanishing point V1 or the second vanishing point V2 as vanishing point V (step S118). This completes the processing in this flowchart.

[0053] As described above, according to this embodiment, other moving objects are clustered based on their relative position and relative movement, lane areas are identified based on the clustering results, and the lane center line is estimated based on the lane area. This makes it possible to appropriately estimate the vehicle's lane even when road markings cannot be detected from the road surface image.

[0054] [Second Embodiment] Figure 13 shows an example of the usage environment of the terminal device 200 mounted on the vehicle M according to the second embodiment. In the above embodiment, the terminal device 100 was installed on the vehicle M so as to be able to image the area in front of the vehicle M in the direction of travel. On the other hand, in the second embodiment, as shown in Figure 13, the terminal device 200 is installed on the vehicle M so as to be able to image the area behind the vehicle M in the opposite direction of travel. As will be explained below, in the second embodiment, the terminal device 200 not only identifies the lane area in the opposite direction of travel of the vehicle M, but also determines whether or not another vehicle traveling in the area behind the vehicle M will overtake the vehicle M based on the relative position and relative movement of the other vehicle, and if it is determined that the other vehicle will overtake the vehicle M, it notifies the occupants of the vehicle M.

[0055] Figure 14 shows an example of the configuration of a terminal device 200 according to the second embodiment. In addition to the functions of the terminal device 100 according to the first embodiment, the terminal device 200 includes a second estimation unit 160, a determination unit 170, and a notification unit 180. The second estimation unit 160, the determination unit 170, and the notification unit 180 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or by the cooperation of software and hardware. The program may be stored in advance in a storage device such as an HDD or flash memory (a storage device equipped with a non-transient storage medium), or it may be stored in a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is mounted on a drive device.

[0056] The functions of the detection unit 110, calculation unit 120, clustering unit 130, identification unit 140, and estimation unit 150 are the same as in the first embodiment. Specifically, the detection unit 110 detects multiple other vehicles in T frames of images captured in time series by the camera 10, which show the area behind the vehicle M. The calculation unit 120 calculates the lateral relative position and longitudinal relative movement of the other vehicles detected in the area behind the vehicle M. The clustering unit 130 clusters the other vehicles based on the calculated relative position and relative movement. The identification unit 140 identifies the lane area based on the clustering results. The estimation unit 150 estimates the center line based on the identified lane area.

[0057] The second estimation unit 160 estimates the direction of movement of the other vehicle based on the relative position in the lateral direction and the relative movement in the longitudinal direction calculated by the calculation unit 120. More specifically, for example, the second estimation unit 160 calculates the relative position in the lateral direction of the other vehicle in a time series, and estimates the direction of movement of the other vehicle in the lateral direction based on whether the difference in the calculated relative position in the time series is a positive value (rightward direction) or a negative value (leftward direction). The second estimation unit 160 can further estimate the direction of movement of the other vehicle in the longitudinal direction based on whether the relative movement in the longitudinal direction is a positive value (forward direction) or a negative value (backward direction). By combining these estimation results for the lateral and longitudinal directions, the second estimation unit 160 can estimate that the direction of travel of the other vehicle is "rightward forward direction" if, for example, the difference in the calculated relative position in the time series is a positive value (rightward direction) and the relative movement in the longitudinal direction is a positive value (forward direction).

[0058] Furthermore, the second estimation unit 160 may estimate the direction of movement using a threshold, taking into account slight differences and errors in relative position and relative movement. For example, the second estimation unit 160 may estimate whether the difference is positive (rightward) or negative (leftward) only if the absolute value of the difference in the relative position of the calculated time series is greater than or equal to the threshold. In addition, the second estimation unit 160 may estimate whether the relative movement is positive (forward) or negative (backward) only if the absolute value of the relative movement in the vertical direction is greater than or equal to the threshold.

[0059] Figure 15 is a diagram illustrating the details of the overtaking determination and notification process. In Figure 15, the symbols LL and RL represent the left road lane marking LL and the right road lane marking CL, respectively, identified by the identification unit 140. The identification unit 140 identifies the area enclosed by these left road lane markings LL and CL as the lane area LD. Figure 15 also shows, as an example, the case where the second estimation unit 160 estimates that the direction of travel of the other vehicle M1 is "right and forward". The determination unit 170 determines whether the other vehicle M1 will overtake the vehicle M based on the estimated direction of movement estimated by the second estimation unit 160 and the lane area LD identified by the identification unit 140. More specifically, for example, the determination unit 170 determines that if the estimated direction of movement estimated by the second estimation unit 160 is "forward right" or "forward left", and the position of the other vehicle M1 is within a predetermined distance from the left road lane marking LL or the right road lane marking CL, then the other vehicle M1 will overtake the vehicle M. At this time, the determination unit 170 may also consider whether the relative lengthwise movement of the other vehicle M1 is greater than or equal to a threshold (i.e., whether the other vehicle M1 is traveling at a higher speed than the vehicle M). In another embodiment, the determination unit 170 may also determine that if the estimated direction of movement estimated by the second estimation unit 160 is "right" or "left" (i.e., even if the relative lengthwise movement is not a positive value), then the position of the other vehicle M1 is within a predetermined distance from the left road lane marking LL or the right road lane marking CL, then the other vehicle M1 will overtake the vehicle M. More generally, the determination unit 170 can perform an overtaking determination using at least one of the estimated direction of movement estimated by the second estimation unit 160 and the lane area LD identified by the identification unit 140.

[0060] If the determination unit 170 determines that another vehicle M1 is about to overtake the vehicle M, the notification unit 180 notifies the occupants of the vehicle M of the overtaking. More specifically, for example, as shown on the right side of Figure 15, the notification unit 180 displays alert information on the display unit 20 indicating that another vehicle M1 is about to overtake the vehicle M. In this case, if the notification unit 180 determines that the other vehicle M1 is about to overtake the vehicle M from the right, it may display the alert information on the right side of the display unit 20, or if it determines that the other vehicle M1 is about to overtake the vehicle M from the left, it may display the alert information on the left side of the display unit 20. Alternatively, the notification unit 180 may use the voice output function of the terminal device 100 to notify the occupants of the vehicle M that another vehicle M1 is about to overtake the vehicle M by voice.

[0061] Figure 16 is a flowchart showing an example of the processing flow performed by the terminal device 200. The processing shown in the flowchart in Figure 16 is performed on the premise that the lane area of ​​the vehicle M has been identified by the identification unit 140 while the vehicle M is in motion.

[0062] First, the second estimation unit 160 estimates the direction of movement of the other vehicle based on the relative lateral position and relative vertical movement of the other vehicle calculated by the calculation unit 120 (step S200). Next, the determination unit 170 determines whether the other vehicle will overtake the self-vehicle based on the estimated direction of movement estimated by the second estimation unit 160 and the lane area identified by the identification unit 140 (step S202). If it is determined that the other vehicle will not overtake the self-vehicle, the terminal device 200 terminates processing. On the other hand, if it is determined that the other vehicle will overtake the self-vehicle, the notification unit 180 notifies the display unit 20 of the lane change (step S204). This completes the processing of this flowchart.

[0063] According to the second embodiment described above, even when road markings cannot be detected from the road surface image, the vehicle's lane can be appropriately estimated, and the estimated lane can be used to suitably assist the vehicle's occupants in driving.

[0064] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, The processor executes the computer-readable instructions to: One or more other moving objects are detected from image data captured around a moving object. The relative position of the other moving body in the lateral direction of the moving body and the relative amount of movement of the other moving body in the vertical direction of the moving body are calculated, Based on the relative position of the other moving objects in the lateral direction and the relative amount of movement of the other moving objects in the vertical direction, the other moving objects are clustered. Based on the results of the aforementioned clustering, the lane region is identified. Based on the lane area, the center line of the lane in which the moving object is traveling is estimated. Information processing device.

[0065] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0066] 10 Cameras 20 Display section 100 terminal devices 110 Detection unit 120 Calculation Unit 130 Clustering section 140 Specific section 150 Estimation part

Claims

1. A detection unit that detects multiple other moving objects from image data captured around a moving object, A calculation unit for each of the plurality of other moving bodies calculates the relative position of the other moving body in the lateral direction relative to the moving body and the relative amount of movement of the other moving body in the vertical direction relative to the moving body, A clustering unit that clusters the plurality of other moving objects based on the relative position of the other moving objects in the lateral direction and the relative amount of movement of the other moving objects in the vertical direction, Based on the results of the clustering, a specific unit identifies the lane area as the width of each cluster obtained by the clustering, The system includes an estimation unit that estimates the center line of the lane in which the moving object is traveling, based on the lane area. Information processing device.

2. Based on the results of the clustering, the identifying unit identifies a self-lane region representing the area of ​​the self-lane in which the moving object is traveling, and an opposing lane region representing the area facing the moving object. The estimation unit estimates the center line as the line between the in-lane area and the oncoming lane area. The information processing apparatus according to claim 1.

3. The identifying unit identifies a group of mobile bodies approaching the mobile body from among the group of mobile body groups belonging to each of the group of clusters obtained by the clustering, and identifies the region including the cluster to which the identified group of mobile body groups belongs as the opposing lane region. The information processing apparatus according to claim 2.

4. The estimation unit estimates a first vanishing point from the edge of the lane region and estimates the other moving object furthest from the moving object as a second vanishing point. If the distance between the first vanishing point and the second vanishing point is within a threshold, the estimation unit identifies either the first vanishing point or the second vanishing point as a vanishing point. The information processing apparatus according to claim 1.

5. If the estimation unit determines that the moving object is traveling on a curved road, it estimates a first vanishing point from the edge of the lane area and estimates the other moving object furthest from the moving object as a second vanishing point. If the distance between the first vanishing point and the second vanishing point is within a threshold, it identifies either the first vanishing point or the second vanishing point as a vanishing point. The information processing apparatus according to claim 1.

6. If the estimation unit determines that a segmentation point exists on the center line, it determines that the moving object is traveling on a curved road. The information processing apparatus according to claim 5.

7. The clustering unit clusters one or more other moving objects detected from the image data of multiple frames captured over time. The information processing apparatus according to claim 1.

8. The clustering unit clusters the other moving objects based on the moving average of the relative position of the other moving objects in the lateral direction and the relative movement amount of the other moving objects in the vertical direction. The information processing apparatus according to claim 1.

9. The clustering unit clusters only the other mobile bodies that are four-wheeled vehicles. The information processing apparatus according to claim 1.

10. The detection unit detects the other moving body located behind the moving body, The aforementioned information processing device is A movement direction estimation unit estimates the movement direction of the other moving body based on the relative position of the other moving body in the lateral direction and the relative amount of movement of the other moving body in the vertical direction. The system includes a determination unit that determines whether or not the other moving body will overtake based on the estimated direction of movement of the other moving body and the lane area. The information processing apparatus according to claim 1.

11. The information processing device further includes a notification unit that notifies the occupants of the moving vehicle of the overtaking when it is determined that the other moving vehicle is overtaking the moving vehicle. The information processing apparatus according to claim 10.

12. Computers Multiple other moving objects are detected from image data captured around a moving object. For each of the aforementioned multiple other moving bodies, the relative position of the other moving body in the lateral direction of the moving body and the relative amount of movement of the other moving body in the vertical direction of the moving body are calculated, Based on the relative position of the other moving bodies in the lateral direction and the relative amount of movement of the other moving bodies in the vertical direction, the plurality of other moving bodies are clustered. Based on the results of the clustering, the lane area is identified as the width of each cluster obtained by the clustering, Based on the lane area, the center line of the lane in which the moving object is traveling is estimated. Information processing methods.

13. On the computer, By detecting multiple other moving objects from image data captured around a moving object, For each of the aforementioned multiple other moving bodies, the relative position of the other moving body in the lateral direction of the moving body and the relative amount of movement of the other moving body in the vertical direction of the moving body are calculated, Based on the relative position of the other moving bodies in the lateral direction and the relative amount of movement of the other moving bodies in the vertical direction, the plurality of other moving bodies are clustered. Based on the results of the clustering, the lane area is identified as the width of each cluster obtained by the clustering. Based on the lane area, the center line of the lane in which the moving object is traveling is estimated. program.

Images