Information processing device, information processing method, and program

Abstract

To appropriately estimate a vehicle's lane of travel even when road markings cannot be detected from a road surface image.SOLUTION: An information processing device includes: a detection unit configured to detect, from image data captured around a moving object, one or more other moving objects; a calculation unit configured to calculate a relative position of the other moving object in a lateral direction of the moving object and a relative movement amount of the other moving object in a longitudinal direction of the moving object; a clustering unit configured to cluster the other moving object based on the relative position of the other moving object in the lateral direction and the relative movement amount of the other moving object in the longitudinal direction; an identification unit configured to identify a lane area based on clustering results; and an estimation unit configured to estimate a center line of the lane on which the moving object is traveling based on the lane area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, there are known techniques for detecting lanes on which a vehicle is traveling. For example, Patent Document 1 discloses a technique for extracting candidate lines that are candidates for marking lines from a road surface image, determining the line type, line color, and presence or absence of backlighting effects of the extracted candidate lines, determining whether the extracted candidate lines constitute multiple lines, selecting candidate lines that will become marking lines using the results of these determinations, recognizing the selected candidate lines, and estimating the shape of the lanes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-106259 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 is based on the premise of extracting candidate lines that are candidates for lane markings from a road surface image. However, there are cases where it is not possible to properly estimate the lane in which the vehicle is traveling, for example, when multiple other vehicles are traveling around the vehicle or when the road markings are not clear to begin with.

[0005] The present invention has been made in consideration of these circumstances, and one of its objectives is to provide an information processing device, an information processing method, and a program that can appropriately estimate the lane in which the vehicle is traveling even when road dividing lines cannot be detected from a road surface image. [Means for solving the problem]

[0006] The information processing device, the information processing method, and the program according to the present invention employ the following configuration. (1): An information processing device according to one embodiment of the present invention includes a detection unit that detects one or more other moving bodies from image data captured around a moving body; a calculation unit that calculates the relative position of the other moving bodies in a lateral direction of the moving body and the relative movement amount of the other moving bodies in a longitudinal direction of the moving body, using the moving body as a reference; a clustering unit that clusters the other moving bodies based on the relative position of the other moving bodies in the lateral direction and the relative movement amount of the other moving bodies in the longitudinal direction; an identification unit that identifies lane areas based on the results of the clustering; and an estimation unit that estimates the center line of the lane in which the moving body is traveling based on the lane area.

[0007] (2): In the above aspect (1), the identification unit identifies a current lane area representing the area of ​​the current lane in which the moving body is traveling and an oncoming lane area representing the area opposite the moving body based on the results of the clustering, and the estimation unit estimates the center line as the line between the current lane area and the oncoming lane area.

[0008] (3): In the aspect (2) above, the identification unit identifies a group of moving objects approaching the moving object from among a plurality of groups of moving objects each belonging to a plurality of clusters obtained by the clustering, and identifies an area including the cluster to which the identified group of moving objects belongs as the oncoming lane area.

[0009] (4): In the above aspect (1), the estimation unit estimates a first vanishing point from the edge of the lane area, and estimates another moving body that is farthest from the moving body as a second vanishing point, and if the distance between the first vanishing point and the second vanishing point is within a threshold, identifies the first vanishing point or the second vanishing point as the vanishing point.

[0010] (5): In the above aspect (1), when the estimation unit determines that the moving body is traveling on a curved road, it estimates a first vanishing point from the edge of the lane area and estimates another moving body that is farthest from the moving body as a second vanishing point, and when the distance between the first vanishing point and the second vanishing point is within a threshold value, it identifies the first vanishing point or the second vanishing point as the vanishing point.

[0011] (6): In the above aspect (5), when the estimation unit determines that a segmentation point exists on the center line, it determines that the moving object is traveling on a curved road.

[0012] (7) In the above aspect (1), the clustering unit clusters one or more other moving objects detected from the image data of a plurality of frames captured over a time series.

[0013] (8): In the aspect (1) above, the clustering unit clusters the other moving bodies based on the relative position of the other moving bodies in the horizontal direction and the moving average of the relative movement amount of the other moving bodies in the vertical direction.

[0014] (9) In the aspect (1) above, the clustering unit clusters only the other moving bodies that are four-wheeled vehicles.

[0015] (10): In the above aspect (1), the detection unit detects the other moving body located behind the moving body, and the information processing device further includes a movement direction estimation unit that estimates the movement direction of the other moving body based on the relative position of the other moving body in the lateral direction and the relative movement amount of the other moving body in the longitudinal direction, and a determination unit that determines whether the other moving body will overtake based on the estimated movement direction of the other moving body and the lane area.

[0016] (11): In the aspect (10) above, the determination unit further includes an alarm unit that, when it is determined that the other moving body will overtake the moving body, notifies an occupant of the moving body of the overtaking.

[0017] (12): Another aspect of the information processing method of the present invention involves a computer detecting one or more other moving bodies from image data of the area around a moving body, calculating the relative position of the other moving bodies in a lateral direction of the moving body and the relative movement amount of the other moving bodies in a longitudinal direction of the moving body, clustering the other moving bodies based on the relative position of the other moving bodies in the lateral direction and the relative movement amount of the other moving bodies in the longitudinal direction, identifying lane areas based on the results of the clustering, and estimating the center line of the lane in which the moving body is traveling based on the lane areas.

[0018] (13): Another aspect of the present invention provides a program that causes a computer to detect one or more other moving bodies from image data captured around a moving body, calculate the relative position of the other moving bodies in a lateral direction of the moving body and the relative movement amount of the other moving bodies in a longitudinal direction of the moving body, cluster the other moving bodies based on the relative position of the other moving bodies in the lateral direction and the relative movement amount of the other moving bodies in the longitudinal direction, identify lane areas based on the results of the clustering, and estimate the center line of the lane in which the moving body is traveling based on the lane areas. [Effects of the Invention]

[0019] According to aspects (1) to (13), even if road dividing lines cannot be detected from the road surface image, the lane in which the host vehicle is traveling can be appropriately estimated. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a diagram showing an example of a usage environment of a terminal device 100 mounted on a vehicle M. FIG. [Figure 2]1 is a diagram illustrating an example of the configuration of a terminal device 100. FIG. [Figure 3] 10 is a diagram showing an example of one or more other vehicles detected by a detection unit 110 from a captured image IM. FIG. [Figure 4] 10 is a diagram showing an example of a situation in which the calculation unit 120 calculates the relative position between the host vehicle M and another vehicle Mk. FIG. [Figure 5] 10 is a diagram for explaining a method by which the calculation unit 120 calculates the longitudinal position between the vehicle M and another vehicle. FIG. [Figure 6] 10 is a diagram for explaining a method by which the calculation unit 120 calculates the lateral position between the host vehicle M and another vehicle. FIG. [Figure 7] 10 is a diagram showing an example of a bird's-eye view generated by a calculation unit 120. FIG. [Figure 8] FIG. 2 is a diagram for explaining an outline of clustering performed by clustering section 130. [Figure 9] 10 is a diagram for explaining a method for identifying a lane area including the host vehicle M by the identification unit 140. FIG. [Figure 10] 10A and 10B are diagrams illustrating an example of a center line and a vanishing point estimated by an estimation unit 150. FIG. [Figure 11] 10 is a diagram for explaining the processing of the estimation unit 150 when the host vehicle M is traveling on a curved road. FIG. [Figure 12] 10 is a flowchart showing an example of the flow of processing executed by the terminal device 100. [Figure 13] FIG. 10 is a diagram showing an example of a usage environment of a terminal device 200 mounted on a vehicle M according to a second embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of the configuration of a terminal device 200 according to a second embodiment. [Figure 15] 10A and 10B are diagrams for explaining details of overtaking determination and notification processing. [Figure 16] 10 is a flowchart showing an example of the flow of processing executed by the terminal device 200. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an information processing device, an information processing method, and a program according to an embodiment of the present invention will be described with reference to the drawings.

[0022] [composition] 1 is a diagram showing an example of a usage environment of a terminal device 100 mounted on a host vehicle M. The host vehicle M may be, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharged power from a secondary battery or a fuel cell.

[0023] 1, the terminal device 100 is installed on the host vehicle M so as to be able to capture an image of the area ahead in the traveling direction of the host vehicle M. The terminal device 100 is, for example, a computer device such as a smartphone or a tablet terminal. The terminal device 100 is held, for example, by an in-vehicle holder (not shown) attached to the dashboard of the host vehicle M, and captures an image of the area ahead of the host vehicle M.

[0024] 2 is a diagram illustrating an example of the configuration of the terminal device 100. As illustrated in FIG. 2, the terminal device 100 includes, for example, a camera 10, a display unit 20, a detection unit 110, a calculation unit 120, a clustering unit 130, an identification unit 140, and an estimation unit 150. The detection unit 110, the calculation unit 120, the clustering unit 130, the identification unit 140, and the estimation unit 150 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as a hard disk drive (HDD) or flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device. In the following description, the functions of the detection unit 110, the calculation unit 120, the clustering unit 130, the identification unit 140, and the estimation unit 150 may be collectively referred to as an "information processing application." The information processing application is installed in the terminal device 100, and is activated, for example, when the user of the terminal device 100 starts driving the vehicle M. Furthermore, the terminal device 100 equipped with the information processing application is an example of an "information processing device."

[0025] The camera 10 is a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The display unit 20 is a display device such as a touch panel or a liquid crystal display. The display unit 20 displays the estimation result by the estimation unit 150, which will be described later. The user of the terminal device 100 aligns the terminal device 100 to a predetermined height (the initial height of a vanishing point V, which will be described later) by following the guide lines displayed on the display unit 20. FIG. 1 shows how the terminal device 100, after aligning, captures an image of the vicinity of the upper end of another vehicle M1 of the host vehicle M in a substantially horizontal direction with respect to the road surface.

[0026] [Detection section] The detection unit 110 recognizes objects captured in the image IM captured by the camera 10. More specifically, for example, the detection unit 110 detects objects using a trained model that has been trained to output information such as the presence, position, and type of an object when an image captured by the camera 10 is input. The detection unit 110 uses this trained model to detect one or more other vehicles in the captured image IM while distinguishing between types such as two-wheeled vehicles and four-wheeled vehicles.

[0027] Fig. 3 is a diagram showing an example of one or more other vehicles detected by the detection unit 110 from the captured image IM. In Fig. 3, symbols M1 to M3 represent four-wheeled vehicles detected by the detection unit 110, and symbols B1 to B3 represent two-wheeled vehicles detected by the detection unit 110. The terminal device 100 displays the other vehicles detected by the detection unit 110 on the display unit 20, for example, by enclosing them in a bounding box. In Fig. 3, the detected four-wheeled vehicle and two-wheeled vehicle are displayed using the same bounding box, but the two-wheeled vehicle and the four-wheeled vehicle may be displayed on the display unit 20 in different display modes.

[0028] [Calculation section] When the detection unit 110 detects one or more other vehicles, the calculation unit 120 further calculates the longitudinal position and lateral position of the one or more other vehicles relative to the host vehicle M (hereinafter, the combination of the longitudinal position and the lateral position may be referred to as the "relative position"). Fig. 4 is a diagram showing an example of a scene in which the calculation unit 120 calculates the relative position between the host vehicle M and other vehicles Mk. In Fig. 4, symbol hA indicates the height from the road position corresponding to the bottom edge of the display unit 20 to the vanishing point V of the image, and symbol hB indicates the height from the road position corresponding to the bottom edge of the detected other vehicle M1 to the vanishing point V of the image.

[0029] Fig. 5 is a diagram illustrating a method by which calculation unit 120 calculates the longitudinal position between host vehicle M and another vehicle. In Fig. 5, symbol IS indicates an image sensor included in camera 10, symbol D indicates a display (an end of camera 10) included in camera 10, symbol O indicates a center of the image sensor, symbol A indicates a position on display D corresponding to road position F shown on the bottom of display D, symbol B indicates a position on display D corresponding to road position G of the rear end of another vehicle M1, symbol C indicates an intersection between the imaging direction of the image sensor and display D, symbol H indicates the height of camera 10 relative to the road surface, symbol DA indicates the distance from the position of image sensor IS to road position F shown on the bottom of display D, and symbol DB indicates the distance from the position of image sensor IS to road position G of the rear end of another vehicle M1.

[0030] In FIG. 5, triangles OAC and OEF are similar to each other, and triangles OBC and OEG are similar to each other. That is, since L:hA = DA:H and L:hB = DB:H hold true for distance, by transformation, DA = L×H / hA and DB = L×H / hB are obtained. Therefore, the calculation unit 120 can calculate the distance to the road position G of the rear end of the other vehicle M1 using the calculation formula DB = DA×hA / hB. Here, the heights hA and hB to the vanishing point are calculated in advance based on the image captured by the camera 10, and the distance DA, which is independent of the position of the other vehicle M1, can be calculated in advance according to the installation position of the terminal device 100. The above calculation can be performed using only the height information of the vanishing point, without requiring all coordinate information of the vanishing point.

[0031] The calculation unit 120 further calculates the lateral position between the host vehicle M and the other vehicle. Fig. 6 is a diagram for explaining a method by which the calculation unit 120 calculates the lateral position between the host vehicle M and the other vehicle. In Fig. 6, the symbol V indicates the vanishing point of the image, the symbol Wb indicates the number of pixels in the lateral direction based on the vanishing point V of the other vehicle M1, and the symbol Wa indicates the number of pixels when the number of pixels Wb is moved to the bottom edge of the display D.

[0032] In FIG. 6, triangle VT'T and triangle VS'S are similar to each other. That is, since Wa:hA == Wb:hB holds true for distance, transformation yields Wa = Wb × hA / hB. Here, assuming that the total number of pixels Wsc at the bottom edge of display D and the width Wrd of the road on which host vehicle M is traveling are known, calculation unit 120 can calculate the actual lateral distance W corresponding to the number of pixels Wa using the formula W = Wrd × Wa / Wsc. In this way, calculation unit 120 calculates the relative position between host vehicle M and other vehicle M1.

[0033] When the calculation unit 120 calculates the relative positions of one or more other vehicles with respect to the host vehicle M as a reference, the calculation unit 120 maps these one or more other vehicles onto a bird's-eye view map with respect to the host vehicle M as a reference. Fig. 7 is a diagram showing an example of a bird's-eye view map generated by the calculation unit 120. The left part of Fig. 7 shows a screen on which the calculation unit 120 calculates the relative positions of one or more other vehicles with respect to the host vehicle M as a reference based on the situation shown in Fig. 3 and maps these onto the bird's-eye view map.

[0034] Calculation unit 120 further calculates the relative positions of other vehicles for T frames (T is a positive integer) of images captured in time series by camera 10, and calculates the amount of relative movement of other vehicles relative to host vehicle M based on the difference between these time-series relative positions. More specifically, calculation unit 120 calculates both the relative position of other vehicles at time point t(k) (k is an integer between 1 and T) and the amount of relative movement of other vehicles over the period t(k)-t(k-1). The arrows shown on the right side of FIG. 7 indicate the direction and magnitude of the amount of relative movement of each other vehicle.

[0035] [Clustering Department] The clustering unit 130 clusters one or more other vehicles based on the relative position of the other vehicle in the lateral direction of the host vehicle M and the relative movement amount of the other vehicle in the longitudinal direction of the host vehicle M calculated by the calculation unit 120.

[0036] FIG. 8 is a diagram illustrating an overview of clustering performed by the clustering unit 130. In this embodiment, the clustering unit 130 represents the relative movement amount in the longitudinal direction and the lateral position calculated for each other vehicle on a two-dimensional graph, and performs the k-means method on these values ​​to cluster the other vehicles. FIG. 8 shows the results of clustering where the lateral position is the X coordinate, the relative movement amount is the Y coordinate, and k=2. More generally, to determine the value of k, the clustering unit 130 may calculate the distance between the average value of each cluster and each data value when clustering is performed for each value of k (e.g., 2 to 5), take the sum of the distances, and adopt the value of k that minimizes the sum of the distances.

[0037] In this embodiment, as shown in FIG. 8 , the clustering unit 130 performs clustering only on four-wheeled vehicles among the detected other vehicles. This is because, compared to two-wheeled vehicles, four-wheeled vehicles generally have smoother travel trajectories and tend to travel along the lane that is the target of estimation. By limiting the targets of clustering to four-wheeled vehicles, the accuracy of the identified vanishing points can be improved compared to when two-wheeled vehicles are included. Alternatively, the clustering unit 130 may perform clustering not only on four-wheeled vehicles but also on two-wheeled vehicles. When performing clustering including two-wheeled vehicles, the clustering unit 130 may determine whether a predetermined number or more of four-wheeled vehicles are present in the captured image, and perform clustering including two-wheeled vehicles only if it is determined that fewer than the predetermined number of four-wheeled vehicles are present.

[0038] Note that the present invention is not limited to the k-means algorithm, and clustering may be performed using other unsupervised algorithms (e.g., a Gaussian mixture model, a hypervolume algorithm, etc.). While FIG. 8 illustrates a case in which the lateral positions of other vehicles at time t(k) and the relative movement amounts of other vehicles over the period t(k)-t(k-1) are clustered, to stabilize the clustering results, clustering may be performed on moving average values ​​of the lateral positions and / or relative movement amounts of each other vehicle, starting from time t(1). This stabilizes the clustering results. Alternatively, instead of using moving average values, a Bayesian estimator may be derived from time-series observation values ​​of the relative movement amounts and lateral positions, and the derived Bayesian estimator may be clustered.

[0039] [Specific part] FIG. 9 is a diagram illustrating a method for the identification unit 140 to identify a lane area including the host vehicle M. When each cluster is obtained by the clustering unit 130, the identification unit 140 calculates the width of the cluster, identifies the left and right ends of the calculated width as the left road dividing line and the right road dividing line, respectively (hereinafter, the combination of the left road dividing line and the right road dividing line may be referred to as an "edge"). The identification unit 140 then identifies the area surrounded by the left road dividing line and the right road dividing line as the lane area. In the example shown in FIG. 9, the identification unit 140 identifies the left road dividing line LL and the right road dividing line CL for cluster C1 and identifies the area surrounded by these left road dividing line LL and right road dividing line CL as the lane area LD. Furthermore, the identification unit 140 identifies the left road dividing line CL and the right road dividing line RL for cluster C2 and identifies the area surrounded by these left road dividing line CL and right road dividing line RL as the lane area RL.

[0040] When the identification unit 140 identifies one or more lane areas, it identifies an area representing the lane in which the host vehicle M is traveling as the host lane area, and identifies lane areas other than the host lane area as other lane areas. The identification unit 140 further identifies whether the other lane area is an oncoming lane area representing an area facing the host vehicle M, based on the relative movement amounts of the other vehicles constituting each cluster. More specifically, for example, when the identification unit 140 determines that the other vehicles constituting each cluster are approaching the host vehicle M based on the relative movement amounts of the other vehicles, the identification unit 140 can identify the cluster as an oncoming lane area. When a cluster includes multiple other vehicles, the identification unit 140 may identify the cluster as an oncoming lane area if, for example, the sum of the relative movement amounts is negative, or may identify whether the cluster is an oncoming lane area by a majority vote based on the number of other vehicles with negative relative movement amounts.

[0041] [Estimation part] The estimation unit 150 estimates the center line of the lane area in which the host vehicle M is traveling, based on the lane area identified by the identification unit 140. More specifically, for example, when there are multiple lane areas identified by the identification unit 140, the estimation unit 150 estimates the line between the identified multiple lane areas as the center line. For example, in the case of FIG. 9 , the estimation unit 150 estimates the line CL passing between the lane area LD and the lane area RD as the center line. In particular, when the identification unit 140 identifies the host lane area and an oncoming lane area, the estimation unit 150 can estimate the line CL passing between the host lane area and the oncoming lane area as the center line.

[0042] FIG. 10 is a diagram showing an example of a center line and vanishing points estimated by the estimation unit 150. When the estimation unit 150 estimates the center line CL, the terminal device 100 displays the estimated center line CL on the display unit 20. At this time, as shown in FIG. 10, the terminal device 100 also displays a left road dividing line CL and a right road dividing line RL in addition to the estimated center line CL. The driver of the host vehicle M can determine the lane in which the host vehicle M is traveling by referring to the center line CL, left road dividing line CL, and right road dividing line RL. In other words, according to this embodiment, even if road dividing lines cannot be detected from the road surface image, the traveling lane of the host vehicle can be appropriately estimated.

[0043] When the left road dividing line CL and the right road dividing line RL are identified by the identification unit 140, the estimation unit 150 identifies the intersection of these identified left road dividing line CL and right road dividing line RL (i.e., edges) as the vanishing point V. Alternatively, the estimation unit 150 may identify the intersection of either the left road dividing line CL or the right road dividing line RL with the center line CL as the vanishing point V. Once the vanishing point V has been identified in this way, the detection unit 110 and the calculation unit 120 use the identified vanishing point V to again perform the processing described with reference to FIGS. 3 to 7 and regenerate the bird's-eye view. Repeating this processing can improve the accuracy of the bird's-eye view.

[0044] [Existence of segment points] The above-described process is applicable regardless of whether the road on which the host vehicle M is traveling is a straight road or a curved road. However, it has been found that the above-described clustering-based lane estimation lacks stability when the host vehicle M is traveling on a curved road. Therefore, the estimation unit 150 determines whether or not a segmentation point from a straight road to a curved road exists on the road on which the host vehicle M is traveling. More specifically, for example, the estimation unit 150 may calculate a second-order derivative of the center line CL identified by the identification unit 140 and estimate the point where the sign of the calculated derivative changes as the segmentation point DP. Furthermore, for example, the estimation unit 150 may calculate the curvature of the center line CL identified by the identification unit 140 and estimate the point where the calculated curvature is equal to or greater than a threshold as the segmentation point DP.

[0045] When the estimation unit 150 determines that a segmentation point exists, in addition to the vanishing point V1 as the intersection of the left road dividing line CL and the right road dividing line RL described above, it considers the other vehicle that is farthest from the vehicle M to be the vanishing point V2, and verifies the reliability of the lane estimation based on clustering by determining whether there is a deviation between these vanishing points V1 and V2.

[0046] Fig. 11 is a diagram for explaining the processing of the estimation unit 150 when the host vehicle M is traveling on a curved road. Fig. 11 shows, as an example, a case where clusters C1 to C4 are detected as a result of clustering.

[0047] If the estimation unit 150 determines that a segmentation point exists, it regards the other vehicle M1 located farthest from the host vehicle M as the vanishing point V2. Next, the estimation unit 150 determines whether the distance between the vanishing point V1, which is the intersection of the left road dividing line CL and the right road dividing line RL, and the vanishing point V2, which is the other vehicle M1, is within a threshold. If it determines that the distance between the vanishing point V1 and the vanishing point V2 is within the threshold, the estimation unit 150 identifies either the vanishing point V1 or the vanishing point V2 as the vanishing point V. Once the vanishing point V has been identified in this way, the detection unit 110 and the calculation unit 120 use the identified vanishing point V to re-execute the processing described with reference to FIGS. 3 to 7 and regenerate the bird's-eye view. Repeating this processing can improve the accuracy of the bird's-eye view.

[0048] In the present embodiment, as an example, a case has been described in which the terminal device 100 is installed to capture an image of a region ahead of the host vehicle M and a vanishing point in the region ahead of the host vehicle M is identified. However, the present invention is not limited to such a configuration, and the terminal device 100 may be installed to capture an image of a region behind the host vehicle M and a vanishing point in the region behind the host vehicle M may be identified. In this case, the generated bird's-eye view image will depict the region behind the host vehicle M, and the occupants of the host vehicle M can confirm the situation behind the host vehicle M by checking the bird's-eye view image. Furthermore, for example, two or more terminal devices 100 may be installed to capture images of the region ahead and the region behind the host vehicle M, respectively, and the bird's-eye view image of the region ahead and the bird's-eye view image of the region behind the host vehicle M generated by these two or more terminal devices 100 may be integrated and displayed on one of the terminal devices 100 or the navigation device of the host vehicle M.

[0049] [Processing flow] Next, the flow of processing executed by the terminal device 100 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the flow of processing executed by the terminal device 100. The processing of the flowchart shown in Fig. 12 is executed repeatedly while the host vehicle M is traveling, for example.

[0050] First, the detection unit 110 detects multiple other vehicles from T frames of images captured in time series by the camera 10 (step S100). Next, the calculation unit 120 calculates the relative positions and relative movement amounts of the detected other vehicles (step S102). Next, the clustering unit 130 clusters the other vehicles based on the calculated relative positions and relative movement amounts of the other vehicles (step S104). Next, the identification unit 140 identifies lane areas based on the clustering results (step S106). Next, the estimation unit 150 estimates a center line based on the identified lane areas (step S108).

[0051] Next, the estimation unit 150 determines whether or not a segmentation point exists on the estimated center line (step S110). If it is determined that a segmentation point does not exist on the estimated center line, the estimation unit 150 identifies a vanishing point V as an intersection of the edges of the identified lane area (step S112). On the other hand, if it is determined that a segmentation point exists on the estimated center line, the estimation unit 150 estimates a first vanishing point V1 as an intersection of the edges of the lane area, and estimates a distant vehicle as a second vanishing point V2 (step S114).

[0052] Next, the estimation unit 150 determines whether the distance between the first vanishing point V1 and the second vanishing point V2 is within a threshold value (step S116). If it is determined that the distance between the first vanishing point V1 and the second vanishing point V2 is not within the threshold value, the terminal device 100 returns the process to step S100. On the other hand, if it is determined that the distance between the first vanishing point V1 and the second vanishing point V2 is within the threshold value, the estimation unit 150 identifies the first vanishing point V1 or the second vanishing point V2 as the vanishing point V (step S118). This ends the process of this flowchart.

[0053] According to the present embodiment described above, other moving objects are clustered based on the relative positions and relative movement amounts of the detected other moving objects, lane areas are identified based on the clustering results, and the center lines of the lanes are estimated based on the lane areas. This makes it possible to appropriately estimate the lane in which the host vehicle is traveling, even when road dividing lines cannot be detected from the road surface image.

[0054] [Second embodiment] FIG. 13 is a diagram showing an example of a usage environment of the terminal device 200 mounted on the host vehicle M according to the second embodiment. In the above embodiment, the terminal device 100 was installed on the host vehicle M so as to be able to capture an image of the area ahead in the traveling direction of the host vehicle M. On the other hand, in the second embodiment, as shown in FIG. 13, the terminal device 200 is installed on the host vehicle M so as to be able to capture an image of the area behind the host vehicle M in the opposite traveling direction. As will be described below, in the second embodiment, the terminal device 200 not only identifies the lane area in the opposite traveling direction of the host vehicle M, but also determines whether another vehicle traveling in the area behind the host vehicle M will overtake the host vehicle M based on the relative position and relative movement amount of the other vehicle, and if it is determined that the other vehicle will overtake the host vehicle M, notifies the occupants of the host vehicle M.

[0055] FIG. 14 is a diagram illustrating an example of the configuration of a terminal device 200 according to the second embodiment. In addition to the functions of the terminal device 100 according to the first embodiment, the terminal device 200 includes a second estimation unit 160, a determination unit 170, and a notification unit 180. The second estimation unit 160, the determination unit 170, and the notification unit 180 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device such as an HDD or flash memory (a storage device having a non-transitory storage medium), or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device.

[0056] The functions of the detection unit 110, calculation unit 120, clustering unit 130, identification unit 140, and estimation unit 150 are the same as those in the first embodiment. That is, the detection unit 110 detects multiple other vehicles from T frames of images captured in time series by the camera 10, depicting the area behind the host vehicle M. The calculation unit 120 calculates the relative lateral positions and relative longitudinal movement amounts of the other vehicles detected in the area behind the host vehicle M. The clustering unit 130 clusters the other vehicles based on the calculated relative positions and relative movement amounts of the other vehicles. The identification unit 140 identifies lane areas based on the clustering results. The estimation unit 150 estimates the center line based on the identified lane areas.

[0057] The second estimation unit 160 estimates the direction of movement of the other vehicle based on the relative position in the lateral direction and the relative movement amount in the longitudinal direction of the other vehicle calculated by the calculation unit 120. More specifically, for example, the second estimation unit 160 calculates the relative position in the lateral direction of the other vehicle in time series, and can estimate the direction of movement of the other vehicle in the lateral direction based on whether the difference in the calculated relative positions in the time series is positive (to the right) or negative (to the left). The second estimation unit 160 can further estimate the direction of movement of the other vehicle in the longitudinal direction based on whether the relative movement amount in the longitudinal direction is positive (forward) or negative (backward). The second estimation unit 160 combines these estimation results in the lateral and longitudinal directions, and can estimate that the traveling direction of the other vehicle is "right-forward," for example, if the difference in the calculated relative positions in the time series is positive (to the right) and the relative movement amount in the longitudinal direction is positive (forward).

[0058] The second estimation unit 160 may estimate the movement direction using a threshold value, taking into account slight differences and errors in the relative positions and the relative movement amounts. For example, the second estimation unit 160 may estimate whether the difference between the calculated time-series relative positions will be a positive value (rightward) or a negative value (leftward) only when the absolute value of the difference is equal to or greater than a threshold value. Furthermore, the second estimation unit 160 may estimate whether the relative movement amount in the vertical direction will be a positive value (forward) or a negative value (backward) only when the absolute value of the relative movement amount in the vertical direction is equal to or greater than a threshold value.

[0059] FIG. 15 is a diagram illustrating details of the overtaking determination and notification process. In FIG. 15, the symbols LL and RL represent the left road dividing line LL and the right road dividing line CL identified by the identification unit 140. The identification unit 140 identifies the area surrounded by the left road dividing line LL and the right road dividing line CL as the lane area LD. Also, FIG. 15 illustrates, as an example, a case in which the second estimation unit 160 estimates that the traveling direction of the other vehicle M1 is "to the right front." The determination unit 170 determines whether the other vehicle M1 will overtake the host vehicle M based on the estimated movement direction estimated by the second estimation unit 160 and the lane area LD identified by the identification unit 140. More specifically, for example, the determination unit 170 determines that the other vehicle M1 will overtake the host vehicle M when the estimated movement direction estimated by the second estimation unit 160 is a "right front direction" or a "left front direction" and the other vehicle M1 is located within a predetermined distance from the left road dividing line LL or the right road dividing line CL. At this time, the determination unit 170 may consider whether the relative movement amount of the other vehicle M1 in the longitudinal direction is equal to or greater than a threshold (i.e., whether the other vehicle M1 is traveling at a higher speed than the host vehicle M). In another aspect, the determination unit 170 may determine that the other vehicle M1 will overtake the host vehicle M when the other vehicle M1 is located within a predetermined distance from the left road dividing line LL or the right road dividing line CL even when the estimated movement direction estimated by the second estimation unit 160 is a "right direction" or a "left direction" (i.e., even when the relative movement amount in the longitudinal direction is not a positive value). More generally, the determination unit 170 may use at least one of the estimated movement direction estimated by the second estimation unit 160 and the lane area LD identified by the identification unit 140 to perform the overtaking determination.

[0060] When the determination unit 170 determines that the other vehicle M1 will overtake the host vehicle M, the notification unit 180 notifies the occupants of the host vehicle M of the overtaking. More specifically, for example, as shown in the right part of FIG. 15 , the notification unit 180 causes the display unit 20 to display alert information indicating that the other vehicle M1 is about to overtake the host vehicle M. At this time, when the notification unit 180 determines that the other vehicle M1 will overtake the host vehicle M from the right, the notification unit 180 may display the alert information on the right side of the display unit 20, or when the notification unit 180 determines that the other vehicle M1 will overtake the host vehicle M from the left, the notification unit 180 may display the alert information on the left side of the display unit 20. Alternatively, the notification unit 180 may use an audio output function of the terminal device 100 to notify the occupants of the other vehicle M1 that the other vehicle M1 is about to overtake the host vehicle M by audio.

[0061] Fig. 16 is a flowchart showing an example of the flow of processing executed by the terminal device 200. The processing of the flowchart shown in Fig. 16 is executed on the premise that the lane area of ​​the host vehicle M has been identified by the identification unit 140 while the host vehicle M is traveling.

[0062] First, the second estimation unit 160 estimates the moving direction of the other vehicle based on the relative lateral position and the relative longitudinal movement amount of the other vehicle calculated by the calculation unit 120 (step S200). Next, the determination unit 170 determines whether the other vehicle will overtake the host vehicle based on the estimated moving direction estimated by the second estimation unit 160 and the lane area identified by the identification unit 140 (step S202). If it is determined that the other vehicle will not overtake the host vehicle, the terminal device 200 ends the processing. On the other hand, if it is determined that the other vehicle will overtake the host vehicle, the notification unit 180 notifies the display unit 20 of a lane change (step S204). This ends the processing of this flowchart.

[0063] According to the second embodiment described above, even if road dividing lines cannot be detected from the road surface image, the lane in which the vehicle is traveling can be appropriately estimated, and the estimated lane can be used to provide appropriate driving assistance to the occupants of the vehicle.

[0064] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: Detecting one or more other moving objects from image data captured around the moving object; Calculating a relative position of the other moving body in a lateral direction of the moving body and a relative movement amount of the other moving body in a longitudinal direction of the moving body, with the moving body as a reference; clustering the other moving objects based on the relative positions of the other moving objects in the horizontal direction and the relative movement amounts of the other moving objects in the vertical direction; Identifying lane regions based on the results of the clustering; estimating a center line of the lane in which the moving object is traveling based on the lane area; Information processing device.

[0065] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0066] 10 Camera 20 Display section 100 Terminal Device 110 Detector 120 Calculation Unit 130 Clustering Department 140 Specific section 150 Estimation part

Claims

1. a detection unit that detects one or more other moving objects from image data captured around the moving object; a calculation unit that calculates a relative position of the other moving body in a horizontal direction of the moving body and a relative movement amount of the other moving body in a vertical direction of the moving body, with the moving body as a reference; a clustering unit that clusters the other moving objects based on the relative positions of the other moving objects in the horizontal direction and the relative movement amounts of the other moving objects in the vertical direction; an identification unit that identifies a lane area based on a result of the clustering; an estimation unit that estimates a center line of the lane in which the moving object is traveling based on the lane area, Information processing device.

2. the identification unit identifies an own lane area representing an area of ​​an own lane in which the moving object is traveling and an opposite lane area representing an area opposite to the moving object, based on a result of the clustering; the estimation unit estimates the center line as a line between the current lane area and the oncoming lane area; The information processing device according to claim 1 .

3. the identification unit identifies a group of moving objects approaching the moving object from among a plurality of groups of moving objects each belonging to a plurality of clusters obtained by the clustering, and identifies an area including the cluster to which the identified group of moving objects belongs as the oncoming traffic lane area. The information processing device according to claim 2 .

4. the estimation unit estimates a first vanishing point from an edge of the lane area, and estimates another moving object that is farthest from the moving object as a second vanishing point, and identifies the first vanishing point or the second vanishing point as the vanishing point when the distance between the first vanishing point and the second vanishing point is within a threshold value. The information processing device according to claim 1 .

5. When it is determined that the moving body is traveling on a curved road, the estimation unit estimates a first vanishing point from an edge of the lane area, and estimates another moving body that is farthest from the moving body as a second vanishing point, and when a distance between the first vanishing point and the second vanishing point is within a threshold, identifies the first vanishing point or the second vanishing point as the vanishing point. The information processing device according to claim 1 .

6. When the estimation unit determines that a segmentation point exists on the center line, the estimation unit determines that the moving object is traveling on a curved road. The information processing device according to claim 5 .

7. the clustering unit clusters one or more other moving objects detected from the image data of a plurality of frames captured over a time series; The information processing device according to claim 1 .

8. the clustering unit clusters the other moving objects based on a moving average of the relative positions of the other moving objects in the horizontal direction and the relative movement amounts of the other moving objects in the vertical direction; The information processing device according to claim 1 .

9. the clustering unit clusters only the other moving objects that are four-wheeled vehicles; The information processing device according to claim 1 .

10. the detection unit detects the other moving object located behind the moving object, The information processing device includes: a movement direction estimation unit that estimates a movement direction of the other moving object based on a relative position of the other moving object in the horizontal direction and a relative movement amount of the other moving object in the vertical direction; a determination unit that determines whether the other moving object will overtake based on the estimated moving direction of the other moving object and the lane area, The information processing device according to claim 1 .

11. the information processing device further includes a notification unit that, when it is determined that the other moving body will overtake the moving body, notifies an occupant of the moving body of the overtaking; The information processing device according to claim 10.

12. The computer Detecting one or more other moving objects from image data captured around the moving object; Calculating a relative position of the other moving body in a lateral direction of the moving body and a relative movement amount of the other moving body in a longitudinal direction of the moving body, with the moving body as a reference; clustering the other moving objects based on the relative positions of the other moving objects in the horizontal direction and the relative movement amounts of the other moving objects in the vertical direction; Identifying lane regions based on the results of the clustering; estimating a center line of the lane in which the moving object is traveling based on the lane area; Information processing methods.

13. On the computer, Detecting one or more other moving objects from image data captured around the moving object; calculating a relative position of the other moving body in a horizontal direction of the moving body and a relative movement amount of the other moving body in a vertical direction of the moving body, with the moving body as a reference; clustering the other moving objects based on the relative positions of the other moving objects in the horizontal direction and the relative movement amounts of the other moving objects in the vertical direction; Identifying lane regions based on the results of the clustering; estimating a center line of the lane in which the moving object is traveling based on the lane area; program.

Images