Information processing apparatus, information processing method, and storage medium

Abstract

The invention provides an information processing device, an information processing method, and a storage medium capable of appropriately estimating a driving lane of a vehicle even if a road division line cannot be detected from a road surface image. This information processing device is provided with: a detection unit that detects one or more other moving bodies from image data obtained by imaging the periphery of a moving body; a calculation unit that calculates the relative position of the other moving body in the lateral direction of the moving body and the relative movement amount of the other moving body in the longitudinal direction of the moving body, based on the moving body; a clustering unit that clusters the other moving bodies on the basis of the relative position of the other moving bodies in the lateral direction and the relative movement amount of the other moving bodies in the longitudinal direction; a determination unit that determines a lane region on the basis of a result of the clustering; and an estimation unit that estimates, on the basis of the lane region, a center line of a lane on which the moving body travels.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing method, and storage medium. Background Technology

[0002] Previously, technologies for detecting the lanes in which vehicles are traveling were known. For example, Japanese Patent Application Publication No. 2018-106259 disclosed a technology that extracts candidate lines as dividing lines from a road surface image, determines the type, color, and presence of backlighting of the extracted candidate lines, and determines whether the extracted candidate lines constitute multiple lines. Using these determination results, candidate lines are selected as dividing lines, and the selected candidate lines are identified to infer the shape of the lane.

[0003] The technology described in Japanese Patent Application Publication No. 2018-106259 is based on extracting candidate lines as alternative lane markings from road surface images. However, for example, in situations where multiple other vehicles are traveling around the vehicle, or where road markings are inherently unclear, it may be impossible to properly presume the vehicle's lane. Summary of the Invention

[0004] The present invention was made in consideration of such circumstances, and one of its objectives is to provide an information processing apparatus, information processing method, and storage medium that can properly estimate the driving lane of a vehicle even when road markings cannot be detected from a road surface image.

[0005] The information processing apparatus, information processing method, and storage medium of the present invention adopt the following structure.

[0006] (1): An information processing apparatus according to a solution of the present invention includes: a detection unit that detects one or more other moving objects from image data obtained by capturing the periphery of a moving object; a calculation unit that calculates the relative positions of the other moving objects in the lateral direction of the moving object and the relative movement amount of the other moving objects in the longitudinal direction of the moving object, with the moving object as a reference; a clustering unit that clusters the other moving objects based on the relative positions of the other moving objects in the lateral direction and the relative movement amount of the other moving objects in the longitudinal direction; a determination unit that determines a lane area based on the clustering result; and an estimation unit that estimates the center line of the lane in which the moving object travels based on the lane area.

[0007] (2): Based on the above (1) scheme, the determining unit determines the lane area representing the area of ​​the lane in which the moving body travels and the opposing lane area representing the area opposite to the moving body based on the clustering result, and the estimating unit estimates the center line as the line between the lane area and the opposing lane area.

[0008] (3): Based on the above (2) scheme, the determining unit determines the mobile body group that is approaching the mobile body from among the multiple mobile body groups that belong to the multiple clusters obtained by the clustering, and determines the area including the cluster to which the determined mobile body group belongs as the opposite lane area.

[0009] (4): Based on the above (1) scheme, the estimation unit estimates the first vanishing point according to the edge of the lane area, and estimates other moving bodies that are furthest from the moving body as the second vanishing point. If the distance between the first vanishing point and the second vanishing point is within a threshold, the first vanishing point or the second vanishing point is identified as the vanishing point.

[0010] (5): Based on the above (1) scheme, when the estimation unit determines that the moving body is traveling on a curved road, it estimates the first vanishing point based on the edge of the lane area, and estimates other moving bodies that are farthest from the moving body as the second vanishing point. When the distance between the first vanishing point and the second vanishing point is within a threshold, it identifies the first vanishing point or the second vanishing point as the vanishing point.

[0011] (6): Based on the above (5) scheme, the estimation unit determines that the moving body is traveling on a curved road when it determines that there is a branch node on the central line.

[0012] (7): Based on the above (1) scheme, the clustering unit will cluster one or more other moving bodies detected from the image data of multiple frames captured in the time series.

[0013] (8): Based on the above (1) scheme, the clustering unit clusters the other moving bodies based on the moving average of the relative positions of the other moving bodies in the horizontal direction and the relative movement of the other moving bodies in the vertical direction.

[0014] (9): Based on the above (1) scheme, the clustering department will only cluster the other mobile bodies that are four-wheeled vehicles.

[0015] (10): Based on the above (1) scheme, the detection unit detects other moving bodies located behind the moving body, and the information processing device further includes: a movement direction estimation unit, which estimates the movement direction of the other moving body based on the relative position of the other moving body in the lateral direction and the relative movement amount of the other moving body in the longitudinal direction; and a determination unit, which determines whether the other moving body is overtaking based on the estimated movement direction of the other moving body and the lane area.

[0016] (11): Based on the above (10) scheme, the information processing device further includes a notification unit that notifies the occupants of the mobile body of the overtaking when the determination unit determines that the other mobile body has overtaken the mobile body.

[0017] (12): The information processing method of another embodiment of the present invention causes a computer to perform the following processing: detect one or more other moving objects from image data obtained by photographing the periphery of the moving object; calculate the relative positions of the other moving objects in the lateral direction of the moving object and the relative movement of the other moving objects in the longitudinal direction of the moving object, with the moving object as a reference; cluster the other moving objects based on the relative positions of the other moving objects in the lateral direction and the relative movement of the other moving objects in the longitudinal direction; determine a lane area based on the clustering result; and estimate the center line of the lane in which the moving object travels based on the lane area.

[0018] (13): Storage medium storing program of another embodiment of the present invention, the program causing a computer to perform the following processing: detecting one or more other moving objects from image data obtained by capturing the periphery of a moving object; calculating the relative positions of the other moving objects in the lateral direction of the moving object and the relative movement of the other moving objects in the longitudinal direction of the moving object, with the moving object as a reference; clustering the other moving objects based on the relative positions of the other moving objects in the lateral direction and the relative movement of the other moving objects in the longitudinal direction; determining a lane area based on the clustering result; and estimating the center line of the lane in which the moving object travels based on the lane area.

[0019] According to the schemes (1) to (13), even if the road markings cannot be detected from the road surface image, the driving lane of the vehicle can be properly estimated. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating an example of the usage environment of the terminal device 100 mounted on the vehicle M.

[0021] Figure 2 This is a diagram illustrating an example of the structure of a terminal device.

[0022] Figure 3 This is an example of one or more other vehicles detected by the detection unit from a captured image.

[0023] Figure 4 This diagram illustrates an example of a scenario where the calculation unit calculates the relative position of the vehicle to other vehicles.

[0024] Figure 5This is a diagram illustrating the method by which the calculation unit calculates the longitudinal position of the vehicle relative to other vehicles.

[0025] Figure 6 This is a diagram illustrating the method by which the calculation unit calculates the lateral position of the vehicle relative to other vehicles.

[0026] Figure 7 This is a diagram showing an example of a bird's-eye view generated by the calculation unit.

[0027] Figure 8 It is a diagram used to illustrate the outline of the clustering performed by the clustering department.

[0028] Figure 9 This is a diagram illustrating a method for determining the lane area containing the vehicle M by a determining unit.

[0029] Figure 10 This is a diagram showing an example of the central line and vanishing point estimated by the presumption section.

[0030] Figure 11 This diagram illustrates the presupposition process when the vehicle is traveling on a curved road.

[0031] Figure 12 This is a flowchart illustrating an example of a process performed by a terminal device.

[0032] Figure 13 This is a diagram illustrating an example of the usage environment of the terminal device mounted on the vehicle M according to the second embodiment.

[0033] Figure 14 This is a diagram illustrating an example of the structure of the terminal device according to the second embodiment.

[0034] Figure 15 This diagram illustrates the details of overtaking determination and notification processing.

[0035] Figure 16 This is a flowchart illustrating an example of a process performed by a terminal device. Detailed Implementation

[0036] Hereinafter, embodiments of the information processing apparatus, information processing method, and storage medium of the present invention will be described with reference to the accompanying drawings.

[0037] [structure]

[0038] Figure 1This diagram illustrates an example of the usage environment of the terminal device 100 mounted on the vehicle M. The vehicle M is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electricity generated by a generator connected to the internal combustion engine, or discharge electricity from a secondary battery or fuel cell.

[0039] like Figure 1 As shown, the terminal device 100 is installed in the vehicle M in a manner capable of capturing images of the area in front of the vehicle M in its direction of travel. The terminal device 100 is, for example, a computer device such as a smartphone or tablet. The terminal device 100 is held, for example, by an in-vehicle retainer (not shown) mounted on the dashboard of the vehicle M, and captures images of the area in front of the vehicle M.

[0040] Figure 2 This is a diagram illustrating an example of the structure of the terminal device 100. (As shown...) Figure 2 As shown, the terminal device 100 includes, for example, a camera 10, a display unit 20, a detection unit 110, a calculation unit 120, a clustering unit 130, a determination unit 140, and an estimation unit 150. The detection unit 110, calculation unit 120, clustering unit 130, determination unit 140, and estimation unit 150 are implemented, for example, by executing a program (software) using a hardware processor such as a CPU (Central Processing Unit). Some or all of these components can be implemented using hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or through the coordinated use of software and hardware. The program can be pre-stored on a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device with a non-transitory storage medium), or stored on a removable storage medium such as a DVD or CD-ROM (a non-transitory storage medium), and installed by mounting the storage medium onto a drive device. In the following description, the functions of the detection unit 110, calculation unit 120, clustering unit 130, determination unit 140, and estimation unit 150 may be collectively referred to as "information processing application". The information processing application is mounted on the terminal device 100, for example, when the user of the terminal device 100 begins driving the vehicle M. Furthermore, the terminal device 100 equipped with the information processing application is an example of an "information processing device".

[0041] Camera 10 is, for example, a digital camera that utilizes a solid-state imaging element such as CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor). Display unit 20 is, for example, a display device such as a touch panel or liquid crystal display. Display unit 20 displays the estimation result estimated by estimation unit 150, which will be described later. The user of terminal device 100 aligns terminal device 100 at a predetermined height (the initial height of the vanishing point V, which will be described later) according to the guide lines displayed on display unit 20. Figure 1 The result of the alignment terminal device 100 indicates the situation near the top of other vehicles M1 of this vehicle M, which is photographed along the road surface in a generally horizontal direction.

[0042] [Testing Department]

[0043] The detection unit 110 identifies objects reflected in the image IM captured by the camera 10. More specifically, for example, the detection unit 110 uses a learned model that has been trained to output information such as the presence, position, and category of objects when an image captured by the camera 10 is input, to detect objects. Using this learned model, the detection unit 110 distinguishes between categories such as two-wheeled vehicles and four-wheeled vehicles, while also detecting one or more other vehicles in the captured image IM.

[0044] Figure 3 This diagram illustrates an example of more than one other vehicle detected by the detection unit 110 from the captured image IM. Figure 3 In the accompanying drawings, reference numerals M1 to M3 indicate four-wheeled vehicles detected by the detection unit 110, and reference numerals B1 to B3 indicate two-wheeled vehicles detected by the detection unit 110. The terminal device 100 displays other vehicles detected by the detection unit 110 on the display unit 20, for example, by means of a bounding box. Figure 3 In the display unit 20, the same bounding box is used to display the detected four-wheeled vehicles and two-wheeled vehicles, but two-wheeled vehicles and four-wheeled vehicles can also be displayed in different display schemes.

[0045] [Calculation Department]

[0046] When the detection unit 110 detects one or more other vehicles, the calculation unit 120 calculates the longitudinal and lateral positions of one or more other vehicles relative to the vehicle M (hereinafter, there is a case where the combination of longitudinal and lateral positions is referred to as "relative position"). Figure 4 This diagram illustrates an example of a scenario where the calculation unit 120 calculates the relative position between the vehicle M and other vehicles Mk. Figure 4 In the attached figure, the symbol h is used. A The reference numeral h indicates the height from the road position corresponding to the lower end of the display unit 20 to the vanishing point V of the image. BThis represents the height from the road position corresponding to the lower end of the detected other vehicle M1 to the vanishing point V of the image.

[0047] Figure 5 This diagram illustrates the method by which the calculation unit 120 calculates the longitudinal position of the vehicle M relative to other vehicles. Figure 5 In the attached figures, reference numeral IS indicates the image sensor included in camera 10, reference numeral D indicates the display included in camera 10 (the end of camera 10), reference numeral O indicates the central part of the image sensor, reference numeral A indicates the position on display D corresponding to the road position F displayed at the bottom of display D, reference numeral B indicates the position on display D corresponding to the road position G of the rear end of other vehicle M1, reference numeral C indicates the intersection of the shooting direction taken by the image sensor and display D, reference numeral H indicates the height of camera 10 relative to the road surface, and reference numeral D... A The figure shows the distance from the position of the image sensor IS to the road position F displayed at the bottom of the display D, with the reference numeral D. B This represents the distance from the position of the image sensor IS to the road position G at the rear end of another vehicle M1.

[0048] exist Figure 5 In the diagram, triangles OAC and OEF are similar, and triangles OBC and OEG are similar. That is, regarding distance, L:h A =D A ∶H and L∶h B =D B :H holds true, therefore, by transformation, we obtain D. A =L×H / h A and D B =L×H / h B Therefore, the calculation unit 120 can use D. B =D A ×h A / h B The formula is used to calculate the distance G to the road position G of the rear end of other vehicles M1. Here, the height h to the vanishing point is... A h B The distance D, calculated in advance based on the image captured by camera 10, is independent of the position of other vehicles M1. A It can be calculated in advance based on the setting position of the terminal device 100. It should be noted that the above calculation does not require all the coordinate information of the vanishing point; it can be performed using only the height information of the vanishing point.

[0049] The calculation unit 120 also calculates the lateral position of vehicle M relative to other vehicles. Figure 6This diagram illustrates the method by which the calculation unit 120 calculates the lateral position of vehicle M relative to other vehicles. Figure 6 In the figure, reference numeral V indicates the vanishing point of the image, reference numeral Wb indicates the number of pixels of other vehicle M1 in the horizontal direction relative to the vanishing point V, and reference numeral Wa indicates the number of pixels when the number of pixels Wb is moved to the bottom of the display D.

[0050] exist Figure 6 In the triangle VT'T, triangle VS'S is similar to triangle VS'S. That is, with respect to distance, Wa∶h A ==Wb∶h B Therefore, by transformation, we obtain Wa = Wb × h. A / h B Here, assuming that the total number of pixels Wsc at the bottom of the display D and the road width Wrd of the vehicle M are known, the calculation unit 120 can use the formula W=Wrd×Wa / Wsc to calculate the actual lateral distance W corresponding to the number of pixels Wa. As described above, the calculation unit 120 calculates the relative position of the vehicle M and other vehicles M1.

[0051] When the calculation unit 120 calculates the relative position of one or more other vehicles based on the vehicle M, it draws the one or more other vehicles on a bird's-eye view based on the vehicle M. Figure 7 This is a diagram showing an example of a bird's-eye view generated by the calculation unit 120. Figure 7 The left part represents the calculation part 120 based on Figure 3 The situation shown is used to calculate the relative positions of one or more other vehicles with this vehicle M as the reference and to plot them on the bird's-eye view.

[0052] The calculation unit 120 also calculates the relative positions of other vehicles from the T frames (T being a positive integer) of images captured by the camera 10 in a time sequence, and calculates the relative movement of other vehicles relative to the vehicle M based on the difference in the relative positions of these time sequences. More specifically, the calculation unit 120 also calculates the relative positions of other vehicles at time point t(k) (k being an integer greater than or equal to 1 and less than or equal to T) and the relative movement of other vehicles during the period t(k)-t(k-1). Figure 7 The arrows shown on the right indicate the direction and magnitude of the relative movement of each other vehicle.

[0053] [Clustering Department]

[0054] The clustering unit 130 clusters one or more other vehicles based on the relative positions of other vehicles in the lateral direction of the vehicle M and the relative movement of other vehicles in the longitudinal direction of the vehicle M, calculated by the calculation unit 120.

[0055] Figure 8 This is a diagram illustrating the outline of the clustering performed by the clustering unit 130. In this embodiment, the clustering unit 130 displays the longitudinal relative movement and lateral position calculated for each of the other vehicles on a two-dimensional graph, and applies the k-means method to these values ​​to cluster the other vehicles. Figure 8 This indicates that the horizontal position is set as the X coordinate, the relative movement is set as the Y coordinate, and k=2 is used for clustering. More generally, in order to determine the value of k, the clustering unit 130 can also calculate the distance between the average value of each cluster and each data value when clustering is performed with each value of k (e.g., 2 to 5), and take the sum of the distances, using the k value with the smallest sum of distances.

[0056] In this embodiment, such as Figure 8 As shown, the clustering unit 130 only performs clustering on four-wheeled vehicles among the detected other vehicles. This is because, generally speaking, four-wheeled vehicles have smoother driving trajectories compared to two-wheeled vehicles, and they also have a stronger tendency to travel along the lanes that are presumed to be the target. By limiting the clustering objects to four-wheeled vehicles, the accuracy of the identified vanishing point can be improved compared to including two-wheeled vehicles. Alternatively, the clustering unit 130 may also perform clustering not only on four-wheeled vehicles but also on two-wheeled vehicles. When clustering includes two-wheeled vehicles, the clustering unit 130 may also determine whether there are more than a certain number of four-wheeled vehicles in the captured image, and only include two-wheeled vehicles for clustering if it is determined that there are fewer than the certain number of four-wheeled vehicles.

[0057] It should be noted that this invention is not limited to the k-means method; for example, other untaught algorithms (e.g., Gaussian mixture model, hypervolume method, etc.) can also be used for clustering. Furthermore, Figure 8 This represents the case where the lateral positions of other vehicles at time point t(k) are clustered with the relative movements of other vehicles during the period t(k)-t(k-1). However, to stabilize the clustering results, clustering can also be performed on the moving averages of the lateral positions and / or relative movements of each other vehicle starting from time point t(1). This stabilizes the clustering results. Alternatively, instead of the moving averages, Bayesian inferences can be derived, for example, from the observations of the time series related to the relative movements and lateral positions, and clustering can be performed on the derived Bayesian inferences.

[0058] [Determination Department]

[0059] Figure 9This diagram illustrates the method by which the determining unit 140 determines the lane area including the vehicle M. When the clustering unit 130 obtains each cluster, the determining unit 140 calculates the lateral width of that cluster, and determines the left and right ends of the calculated lateral width as the left and right road dividing lines, respectively (hereinafter, there is a case where the combination of the left and right road dividing lines is referred to as an "edge"), and determines the area enclosed by the left and right road dividing lines as the lane area. Figure 9 In the case of cluster C1, the determining unit 140 determines the left road dividing line LL and the right road dividing line CL, and defines the area enclosed by these left road dividing lines LL and right road dividing lines CL as the lane area LD. Furthermore, the determining unit 140 determines the left road dividing line CL and the right road dividing line RL, and defines the area enclosed by these left road dividing lines CL and right road dividing lines RL as the lane area RL.

[0060] When determining more than one lane area, the determining unit 140 determines the area representing the lane in which the vehicle M is traveling as the "this lane area," and determines lane areas other than the "this lane area" as "other lane areas." The determining unit 140 also determines whether an other lane area is an opposing lane area representing an area opposite to the vehicle M based on the relative movement of other vehicles constituting each cluster. More specifically, for example, if the determining unit 140 determines that an other vehicle is approaching the vehicle M based on the relative movement of other vehicles constituting each cluster (i.e., if the relative movement is negative), it can determine that the cluster is an opposing lane area. If a cluster includes multiple other vehicles, the determining unit 140 can, for example, determine that the cluster is an opposing lane area if the sum of the relative movements is negative, or it can determine whether the cluster is an opposing lane area by a majority vote based on the number of other vehicles with negative relative movements.

[0061] [Presumption Section]

[0062] The estimation unit 150 estimates the center line of the lane area in which the vehicle M travels based on the lane area determined by the determination unit 140. More specifically, for example, if there are multiple lane areas determined by the determination unit 140, the estimation unit 150 estimates the line between the multiple determined lane areas as the center line. For example, in Figure 9 In this case, the estimation unit 150 estimates the line CL that passes between lane area LD and lane area RD as the center line. In particular, when the determination unit 140 has determined the lane area and the opposite lane area, the estimation unit 150 can estimate the line CL that passes between the lane area and the opposite lane area as the center line.

[0063] Figure 10This diagram illustrates an example of the center line and vanishing point estimated by the estimation unit 150. When the estimation unit 150 estimates the center line CL, the terminal device 100 displays the estimated center line CL on the display unit 20. At this time, as shown... Figure 10 As shown, in addition to the estimated center line CL, the terminal device 100 also displays the left road dividing line CL and the right road dividing line RL. The driver of the vehicle M can determine the lane in which the vehicle M is traveling by referring to these center line CL, left road dividing line CL, and right road dividing line RL. That is, according to this embodiment, even when road dividing lines cannot be detected from the road surface image, the vehicle's driving lane can be properly estimated.

[0064] When the determination unit 140 determines the left road dividing line CL and the right road dividing line RL, the estimation unit 150 identifies the intersection of these determined left road dividing lines CL and right road dividing lines RL (i.e., edges) as the vanishing point V. Alternatively, the estimation unit 150 may identify the intersection of either the left road dividing line CL or the right road dividing line RL with the center line CL as the vanishing point V. When the vanishing point V is identified in this way, the detection unit 110 and the calculation unit 120 use the identified vanishing point V to perform the calculation again. Figures 3 to 7 The described processing steps are then used to generate the bird's-eye view. By repeating this process, the accuracy of the bird's-eye view can be improved.

[0065] [The existence of sub-nodes]

[0066] The processing described above can be applied regardless of whether the road on which the vehicle M travels is a straight road or a curved road. However, it is known that when the vehicle M travels on a curved road, the lane estimation based on the above clustering lacks stability. Therefore, the estimation unit 150 determines whether there is a branch node from a straight road to a curved road on the road on which the vehicle M travels. More specifically, for example, the estimation unit 150 can also calculate the second derivative of the centerline CL determined by the determination unit 140, and estimate the point where the sign of the calculated derivative changes as the branch node DP. In addition, for example, the estimation unit 150 can also calculate the curvature of the centerline CL determined by the determination unit 140, and estimate the point where the calculated curvature is above a threshold as the branch node DP.

[0067] When the estimation unit 150 determines that there are sub-nodes, in addition to the vanishing point V1, which is the intersection of the left road dividing line CL and the right road dividing line RL, other vehicles that are furthest away from the current vehicle M are also regarded as vanishing points V2, and it is determined whether there is any deviation at these vanishing points V1 and vanishing points V2, thereby verifying the reliability of the lane estimation based on clustering.

[0068] Figure 11This diagram illustrates the processing of the estimation unit 150 when the vehicle M is traveling on a curved road. Figure 11 As an example, the clustering results show the detection of clusters C1 to C4.

[0069] When the estimation unit 150 determines that a vanishing point exists, it considers the other vehicle M1, which is furthest from the current vehicle M, as the vanishing point V2. Next, the estimation unit 150 determines whether the distance between the vanishing point V1 (the intersection of the left road dividing line CL and the right road dividing line RL) and the vanishing point V2 (the vanishing point of the other vehicle M1) is within a threshold. If the distance between vanishing points V1 and V2 is determined to be within the threshold, the estimation unit 150 identifies either vanishing point V1 or vanishing point V2 as vanishing point V. When vanishing point V is identified in this way, the detection unit 110 and the calculation unit 120 use the identified vanishing point V to perform the calculation again. Figures 3 to 7 The described processing steps are then used to generate the bird's-eye view. By repeating this process, the accuracy of the bird's-eye view can be improved.

[0070] It should be noted that, in this embodiment, as an example, the terminal device 100 is configured to capture the area in front of the vehicle M and determine the vanishing point in that area. However, the present invention is not limited to that structure; the terminal device 100 may also be configured to capture the area behind the vehicle M and determine the vanishing point in that area. In this case, the generated bird's-eye view becomes a bird's-eye view reflecting the area behind the vehicle M, and the occupants of the vehicle M can confirm the situation behind the vehicle M by checking this bird's-eye view. Alternatively, for example, two or more terminal devices 100 may be configured to capture both the area in front of and the area behind the vehicle M, and the bird's-eye views of the front and rear areas generated by the two or more terminal devices 100 may be combined and displayed on either the terminal device 100 or the navigation device of the vehicle M.

[0071] [Processing flow]

[0072] Next, refer to Figure 12 The process flow performed by the terminal device 100 is described. Figure 12 This is a flowchart illustrating an example of the process executed by the terminal device 100. Figure 12 The process shown in the flowchart is executed repeatedly, for example, while the vehicle M is in motion.

[0073] First, the detection unit 110 detects multiple other vehicles in a T-frame image sequence captured by the camera 10 over time (step S100). Next, the calculation unit 120 calculates the relative positions and relative movements of the detected other vehicles (step S102). Next, the clustering unit 130 clusters the other vehicles based on the calculated relative positions and relative movements (step S104). Next, the determination unit 140 determines lane regions based on the clustering results (step S106). Next, the estimation unit 150 estimates the center line based on the determined lane regions (step S108).

[0074] Next, the estimation unit 150 determines whether a sub-node exists on the estimated centerline (step S110). If it is determined that no sub-node exists on the estimated centerline, the estimation unit 150 identifies the vanishing point V as the intersection of the edges of the determined lane area (step S112). On the other hand, if it is determined that a sub-node exists on the estimated centerline, the estimation unit 150 estimates the first vanishing point V1 as the intersection of the edges of the lane area and estimates distant vehicles as the second vanishing point V2 (step S114).

[0075] Next, the estimation unit 150 determines whether the distance between the first vanishing point V1 and the second vanishing point V2 is within a threshold (step S116). If it is determined that the distance between the first vanishing point V1 and the second vanishing point V2 is not within the threshold, the terminal device 100 returns the process to step S100. On the other hand, if it is determined that the distance between the first vanishing point V1 and the second vanishing point V2 is within the threshold, the estimation unit 150 identifies either the first vanishing point V1 or the second vanishing point V2 as vanishing point V (step S118). Thus, the processing of this flowchart ends.

[0076] According to this embodiment as described above, other moving objects are clustered based on their relative positions and relative movement amounts. Lane regions are determined based on the clustering results, and the center line of the lane is estimated based on the lane regions. Thus, even when road markings cannot be detected from the road surface image, the vehicle's driving lane can be appropriately estimated.

[0077] [Second Implementation]

[0078] Figure 13 This diagram illustrates an example of the usage environment of the terminal device 200 mounted on the vehicle M according to the second embodiment. In the above embodiment, the terminal device 100 is installed on the vehicle M in such a way that it can capture images of the area in front of the vehicle M in the direction of travel. On the other hand, in the second embodiment, as... Figure 13As shown, the terminal device 200 is installed on the vehicle M in a manner capable of capturing images of the area behind the vehicle M in the direction of travel. As explained below, in the second embodiment, the terminal device 200 not only determines the lane area in the direction of travel of the vehicle M, but also determines whether other vehicles traveling in the area behind the vehicle M have overtaken the vehicle M based on their relative positions and relative movement. If it is determined that other vehicles have overtaken the vehicle M, the occupants of the vehicle M are notified.

[0079] Figure 14 This diagram illustrates an example of the structure of the terminal device 200 according to the second embodiment. In addition to the functions of the terminal device 100 of the first embodiment, the terminal device 200 also includes a second estimation unit 160, a determination unit 170, and a notification unit 180. The second estimation unit 160, the determination unit 170, and the notification unit 180 are implemented, for example, by executing a program (software) using a hardware processor such as a CPU. Some or all of these components can be implemented using hardware (including the circuitry) such as LSI, ASIC, FPGA, and GPU, or through the coordinated operation of software and hardware. The program can be pre-saved in a storage device such as an HDD or flash memory (a storage device with a non-transitory storage medium), or it can be saved in a removable storage medium such as a DVD or CD-ROM (a non-transitory storage medium) and installed via a drive device.

[0080] The detection unit 110, calculation unit 120, clustering unit 130, determination unit 140, and estimation unit 150 function the same as in the first embodiment. Specifically, the detection unit 110 detects multiple other vehicles in an image of T frames captured by the camera 10 in a time sequence, which reflects the rear region of the vehicle M. The calculation unit 120 calculates the lateral relative positions and longitudinal relative movements of the other vehicles detected in the rear region of the vehicle M. The clustering unit 130 clusters the other vehicles based on the calculated relative positions and relative movements. The determination unit 140 determines lane regions based on the clustering results. The estimation unit 150 estimates the centerline based on the determined lane regions.

[0081] The second estimation unit 160 estimates the direction of movement of other vehicles based on the lateral relative positions and longitudinal relative movements of other vehicles calculated by the calculation unit 120. More specifically, for example, the second estimation unit 160 calculates the lateral relative positions of other vehicles in a time series and can estimate the lateral direction of movement of other vehicles based on whether the difference in the calculated relative positions in the time series is positive (right direction) or negative (left direction). The second estimation unit 160 can also estimate the longitudinal direction of movement of other vehicles based on whether the longitudinal relative movement is positive (forward direction) or negative (rearward direction). The second estimation unit 160 combines these lateral and longitudinal estimation results; for example, if the difference in the calculated relative positions in the time series is positive (right direction) and the longitudinal relative movement is positive (forward direction), the second estimation unit 160 can estimate that the direction of movement of the other vehicle is "right-forward direction".

[0082] It should be noted that the second estimation unit 160 may also consider minor differences and errors related to relative position and relative movement, and use a threshold to estimate the direction of movement. For example, the second estimation unit 160 may estimate whether the difference in relative position of the calculated time series is positive (rightward direction) or negative (leftward direction) only if the absolute value of the difference is above the threshold. Furthermore, the second estimation unit 160 may estimate whether the relative movement is positive (forward direction) or negative (backward direction) only if the absolute value of the relative movement in the vertical direction is above the threshold.

[0083] Figure 15 This diagram illustrates the details of overtaking determination and notification processing. Figure 15 In the attached diagram, reference numerals LL and RL denote the left road dividing line LL and the right road dividing line CL as determined by the determining unit 140. The determining unit 140 defines the area enclosed by these left road dividing lines LL and right road dividing lines CL as the lane area LD. Furthermore, in Figure 15As an example, this illustrates a case where the second estimation unit 160 estimates that the direction of travel of another vehicle M1 is "right-forward". The determination unit 170 determines whether the other vehicle M1 has overtaken the vehicle M based on the estimated direction of movement estimated by the second estimation unit 160 and the lane area LD determined by the determination unit 140. More specifically, for example, if the estimated direction of movement estimated by the second estimation unit 160 is "right-forward" or "left-forward" and the position of the other vehicle M1 is within a predetermined distance of the left lane dividing line LL or the right lane dividing line CL, the determination unit 170 determines that the other vehicle M1 has overtaken the vehicle M. In this case, the determination unit 170 may also consider whether the longitudinal relative movement of the other vehicle M1 is above a threshold (i.e., whether the other vehicle M1 is traveling at a higher speed than the vehicle M). Alternatively, as another option, the determination unit 170 may determine that another vehicle M1 is overtaking vehicle M1 even when the estimated direction of movement estimated by the second estimation unit 160 is "right" or "left" (i.e., even when the relative movement in the longitudinal direction is not a positive value), if the position of the other vehicle M1 is within a predetermined distance of the left lane dividing line LL or the right lane dividing line CL. More generally, the determination unit 170 may use at least one of the estimated direction of movement estimated by the second estimation unit 160 and the lane area LD determined by the determination unit 140 to make the overtaking determination.

[0084] If the determination unit 170 determines that another vehicle M1 has overtaken vehicle M, the notification unit 180 notifies the occupants of vehicle M of the overtaking. More specifically, for example, if... Figure 15 As shown on the right side, the notification unit 180 causes the display unit 20 to display an alarm message indicating that another vehicle M1 intends to overtake the vehicle M. At this time, the notification unit 180 may display the alarm message on the right side of the display unit 20 if it determines that another vehicle M1 is overtaking the vehicle M from the right, or on the left side of the display unit 20 if it determines that another vehicle M1 is overtaking the vehicle M from the left. Alternatively, the notification unit 180 may utilize the sound output function of the terminal device 100 to notify the other vehicle M1 of its intention to overtake the vehicle M by sound.

[0085] Figure 16 This is a flowchart illustrating an example of the process executed by the terminal device 200. Figure 16 The process shown in the flowchart is performed under the premise that the lane area of ​​the vehicle M is determined by the determination unit 140 while the vehicle M is in motion.

[0086] First, the second estimation unit 160 estimates the direction of movement of other vehicles based on the lateral relative positions and longitudinal relative movements of other vehicles calculated by the calculation unit 120 (step S200). Next, the determination unit 170 determines whether other vehicles have overtaken this vehicle based on the estimated direction of movement estimated by the second estimation unit 160 and the lane area determined by the determination unit 140 (step S202). If it is determined that other vehicles have not overtaken this vehicle, the terminal device 200 ends the processing. On the other hand, if it is determined that other vehicles have overtaken this vehicle, the notification unit 180 notifies the display unit 20 of the overtaking (step S204). Thus, the processing of this flowchart ends.

[0087] According to the second embodiment described above, even when road markings cannot be detected from the road surface image, the vehicle's driving lane can be properly estimated, and the estimated driving lane can be utilized to provide appropriate driving support for the vehicle's occupants.

[0088] The implementation methods described above can be performed as follows.

[0089] An information processing device, wherein,

[0090] The information processing device includes:

[0091] Storage medium, which stores computer-readable instructions; and

[0092] The processor, which is connected to the storage medium,

[0093] The processor performs the following processing by executing computer-readable instructions:

[0094] Detect one or more other moving objects from image data obtained from the surrounding area of ​​a moving object;

[0095] Calculate the relative positions of the other moving bodies in the lateral direction of the moving body and the relative displacement of the other moving bodies in the longitudinal direction of the moving body, with the moving body as the reference.

[0096] The other moving bodies are clustered based on their relative positions in the horizontal direction and their relative movement in the vertical direction.

[0097] Based on the clustering results, lane regions are determined; and

[0098] Based on the lane area, the center line of the lane in which the moving body is traveling is estimated.

[0099] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. An information processing device, wherein, The information processing device includes: The detection unit detects one or more other moving objects from image data obtained from the surrounding area of ​​the moving object. The calculation unit calculates the relative positions of the other moving bodies in the lateral direction of the moving body and the relative movement of the other moving bodies in the longitudinal direction of the moving body, with the moving body as the reference. The clustering unit clusters the other moving bodies based on their relative positions in the horizontal direction and their relative movement in the vertical direction. The determination unit determines the lane region based on the clustering results; as well as The estimation unit estimates the center line of the lane in which the moving body travels, based on the lane area.

2. The information processing apparatus according to claim 1, wherein, Based on the clustering results, the determining unit determines the current lane region, which represents the area in the lane where the moving vehicle is traveling, and the opposite lane region, which represents the area opposite to the moving vehicle. The estimation unit estimates the center line as the line between the lane area and the opposite lane area.

3. The information processing apparatus according to claim 2, wherein, The determining unit identifies the moving body groups that are approaching the moving body from among the multiple moving body groups that belong to the multiple clusters obtained by the clustering, and defines the area including the cluster to which the determined moving body group belongs as the oncoming lane area.

4. The information processing apparatus according to claim 1, wherein, The estimation unit estimates a first vanishing point based on the edge of the lane area, and estimates other moving objects that are furthest from the moving object as second vanishing points. If the distance between the first vanishing point and the second vanishing point is within a threshold, the first vanishing point or the second vanishing point is identified as a vanishing point.

5. The information processing apparatus according to claim 1, wherein, When the estimation unit determines that the moving body is traveling on a curved road, it estimates a first vanishing point based on the edge of the lane area, and estimates other moving bodies that are furthest from the moving body as second vanishing points. If the distance between the first vanishing point and the second vanishing point is within a threshold, it identifies the first vanishing point or the second vanishing point as a vanishing point.

6. The information processing apparatus according to claim 5, wherein, If the estimation unit determines that there is a branch node on the center line, it determines that the moving body is traveling on a curved road.

7. The information processing apparatus according to claim 1, wherein, The clustering unit will cluster one or more other moving bodies detected from the image data of multiple frames captured in a time series.

8. The information processing apparatus according to claim 1, wherein, The clustering unit clusters the other moving bodies based on the moving average of their relative positions in the horizontal direction and their relative movement in the vertical direction.

9. The information processing apparatus according to claim 1, wherein, The clustering department will only cluster the other mobile bodies that are four-wheeled vehicles.

10. The information processing apparatus according to claim 1, wherein, The detection unit detects other moving bodies located behind the moving body. The information processing device includes: The movement direction estimation unit estimates the movement direction of the other moving bodies based on their relative positions in the lateral direction and the relative movement amount of the other moving bodies in the longitudinal direction; as well as The determination unit determines whether the other moving body should overtake the other moving body based on the estimated direction of movement of the other moving body and the lane area.

11. The information processing apparatus according to claim 10, wherein, The information processing device further includes a notification unit that notifies the occupants of the other mobile vehicle of the overtaking if it is determined that the other mobile vehicle has overtaken the mobile vehicle.

12. An information processing method, wherein, The information processing method causes the computer to perform the following processing: Detect one or more other moving objects from image data obtained from the surrounding area of ​​a moving object; Calculate the relative positions of the other moving bodies in the lateral direction of the moving body and the relative displacement of the other moving bodies in the longitudinal direction of the moving body, with the moving body as the reference. The other moving bodies are clustered based on their relative positions in the horizontal direction and their relative movement in the vertical direction. The lane area is determined based on the clustering results; as well as Based on the lane area, the center line of the lane in which the moving body is traveling is estimated.

13. A storage medium having a stored program, wherein, The program causes the computer to perform the following processing: Detect one or more other moving objects from image data obtained from the surrounding area of ​​a moving object; Calculate the relative positions of the other moving bodies in the lateral direction of the moving body and the relative displacement of the other moving bodies in the longitudinal direction of the moving body, with the moving body as the reference. The other moving bodies are clustered based on their relative positions in the horizontal direction and their relative movement in the vertical direction. The lane area is determined based on the clustering results; as well as Based on the lane area, the center line of the lane in which the moving body is traveling is estimated.

Images