Image processing device

Abstract

To provide an image processing device capable of enhancing the recognition accuracy of a road.SOLUTION: According to one embodiment of the present disclosure, an image processing device includes an estimation circuit which can estimate the position of a partition line on the surface of a road on the basis of picked-up image data including an image of a partition line partitioning the road, and a correction circuit which can correct the position of the partition line on the surface of the road, estimated by the estimation circuit on the basis of a height position based on the surface of the road of an imaging device which generates the picked-up image data.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an image processing device that recognizes a road based on a captured image. [Background technology]

[0002] In a vehicle, the road on which the vehicle is traveling is often recognized based on a captured image. For example, Patent Document 1 discloses a technology for detecting marking lines that divide a road based on a captured image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 09-319872 Summary of the Invention [Problem to be solved by the invention]

[0004] In such image processing devices mounted on vehicles, it is desirable to accurately recognize the roadway, and high accuracy in recognizing the roadway is expected.

[0005] It is desirable to provide an image processing device that can improve the recognition accuracy of a road. [Means for solving the problem]

[0006] An image processing device according to an embodiment of the present disclosure includes an estimation circuit and a correction circuit. The estimation circuit is capable of estimating the position of the marking line on the road surface of the road based on captured image data including an image of the marking line dividing the road. The correction circuit is capable of correcting the position of the marking line on the road surface of the road estimated by the estimation circuit based on the height position of the imaging device that generated the captured image data with respect to the road surface of the road. Effect of the Invention

[0007] According to an image processing device according to an embodiment of the present disclosure, it is possible to improve the recognition accuracy of a road. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of a configuration of a road recognition device according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the roadway recognition device shown in FIG. [Figure 3A] FIG. 3A is an explanatory diagram illustrating an example of a captured image. [Figure 3B] FIG. 3B is an explanatory diagram illustrating another example of the captured image. [Figure 4] FIG. 4 is a block diagram illustrating an example of the configuration of the lane marking detection unit illustrated in FIG. [Diagram 5] FIG. 5 is a flowchart illustrating an example of an operation of the lane marking detection unit illustrated in FIG. [Figure 6] FIG. 6 is an explanatory diagram illustrating an example of an operation of the height position detection unit shown in FIG. [Figure 7] FIG. 7 is another explanatory diagram illustrating an operation example of the height position detection unit shown in FIG. [Figure 8] FIG. 8 is an explanatory diagram illustrating an example of an operation of the lane line inference unit illustrated in FIG. [Figure 9] FIG. 9 is an explanatory diagram illustrating an example of an operation of the lane marking correction unit illustrated in FIG. [Figure 10] FIG. 10 is a block diagram illustrating an example of the configuration of a lane marking detection unit according to a reference example. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of a road recognition device according to a modified example. [Figure 12] FIG. 12 is a block diagram illustrating an example of the configuration of a lane line detection unit according to another modified example. [Figure 13] FIG. 13 is a flowchart illustrating an example of an operation of the lane marking detection unit illustrated in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0010] <Embodiment Mode> [Configuration example] 1 and 2 show an example of the configuration of a road recognition device 10 equipped with an image processing device according to an embodiment. In this example, the road recognition device 10 is mounted on a vehicle 1 such as an automobile, and is configured to recognize the road on which the vehicle 1 is traveling. The road recognition device 10 includes a stereo camera 11 and an image processing device 20.

[0011] The stereo camera 11 is configured to capture an image in front of the vehicle 1, thereby generating a set of image data (left image data PL and right image data PR) having a parallax between them. The stereo camera 11 has a left camera 11L and a right camera 11R. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor. In this example, the left camera 11L and the right camera 11R are disposed in the interior of the vehicle 1, near the upper part of the windshield of the vehicle 1, and spaced apart by a predetermined distance in the width direction of the vehicle 1. The left camera 11L generates left image data PL, and the right camera 11R generates right image data PR. The left image data PL and the right image data PR constitute stereo image data PIC. The stereo camera 11 performs an imaging operation at a predetermined frame rate (for example, 60 [fps]), thereby generating a series of stereo image data PIC, and supplies the generated stereo image data PIC to the image processing device 20.

[0012] The image processing device 20 is configured to recognize the road on which the vehicle 1 is traveling based on the stereo image data PIC supplied from the stereo camera 11. For example, the vehicle 1 can perform driving control of the vehicle 1 or display information about the recognized road on a console monitor based on information about the road recognized by the image processing device 20. The image processing device 20 is configured, for example, with a CPU (Central Processing Unit) that executes a program, a RAM (Random Access Memory) that temporarily stores processing data, and a ROM (Read Only Memory) that stores the program. The image processing device 20 has a distance image generating unit 21, a lane marking detection unit 22, and a road recognition unit 23.

[0013] The distance image generating unit 21 is configured to generate distance image data PD by performing predetermined image processing including stereo matching processing based on the left image data PL and the right image data PR. The pixel values ​​of the distance image data PD indicate the distance from the stereo camera 11 to the subject in three-dimensional real space. By performing stereo matching processing, the distance image generating unit 21 is configured to obtain a parallax by detecting corresponding points including image points in the left image and image points in the right image that correspond to each other, and to calculate the distance to the subject based on the parallax.

[0014] The lane marking detection unit 22 is configured to detect lane marks that divide the road on which the vehicle 1 is traveling based on the left image data PL and right image data PR, and the distance image data PD, and to generate lane marking data DL indicating the position of the lane markings on the road surface of the road.

[0015] In this example, the lane marking detection unit 22 infers the position of the lane marking using a machine learning technique based on the captured image obtained by the stereo camera 11. Then, the lane marking detection unit 22 generates the lane marking data DL by correcting the inferred position of the lane marking based on the height position of the stereo camera 11 with respect to the road surface of the road. That is, the height from the road surface of the road to the stereo camera 11 may vary depending on, for example, the type of vehicle 1, the number of passengers, the amount of luggage carried, etc. Also, the height to the stereo camera 11 may change, for example, when the vehicle height is reduced due to aging deterioration of the suspension. As shown below, the image of the lane marking in the captured image may vary depending on the height from the road surface of the road to the stereo camera 11, so the lane marking detection unit 22 corrects the inferred position of the lane marking based on the height position of the stereo camera 11.

[0016] 3A and 3B show an example of an image captured by the stereo camera 11, where FIG. 3A shows a case where the height position of the stereo camera 11 is low, and FIG. 3B shows a case where the height position of the stereo camera 11 is higher than that of FIG. 3A. For example, as shown in FIGS. 3A and 3B, when the height position of the stereo camera 11 is high (FIG. 3B), the interval between the image portion of the left demarcation line 101L and the image portion of the right demarcation line 101R in the captured image is narrower than when the height position of the stereo camera 11 is low (FIG. 3A). In this way, for example, even if the interval between the two actual demarcation lines is the same, the interval between the image portion of the left demarcation line 101L and the image portion of the right demarcation line 101R in the captured image may differ depending on the height position of the stereo camera 11. Therefore, in this case, the detection accuracy of the position of the demarcation line may be reduced.

[0017] On the other hand, the lane marking detection unit 22 infers the positions of the lane marks on the road surface of the road based on the captured image, and corrects the inferred positions of the lane marks based on the height position of the stereo camera 11. This allows the lane marking detection unit 22 to improve the detection accuracy of the lane marking positions.

[0018] 4 shows an example of the configuration of the lane marking detection unit 22. The lane marking detection unit 22 has a height position detection unit 31, a lane marking inference unit 32, a correction coefficient calculation unit 33, and a lane marking correction unit .

[0019] The height position detection unit 31 is configured to detect the height position of the stereo camera 11 with respect to the road surface of the road, based on the left image data PL, the right image data PR, and the distance image data PD.

[0020] The lane line inference unit 32 is configured to infer the position of the lane line on the road surface of the road using a machine learning model M stored in the storage unit 35 based on the left image data PL and the right image data PR. The machine learning model M is, for example, a machine learning model of a deep neural network. The machine learning model M is configured to receive image data and output the position of the lane line on the road surface of the road. This machine learning model M is generated by a machine learning device configured using, for example, a personal computer performing machine learning processing, and is stored in advance in the storage unit 35. In addition to the machine learning model M, the storage unit 35 also stores reference data REF indicating the height position of the stereo camera that generated the image data used to generate the machine learning model M.

[0021] The correction coefficient calculation unit 33 is configured to generate a correction coefficient based on the height position of the stereo camera 11 detected by the height position detection unit 31 and the height position indicated by the reference data REF stored in the storage unit 35.

[0022] The lane marking correction unit 34 is configured to generate lane marking data DL indicating the position of the lane marking on the road surface of the roadway by correcting the position of the lane marking inferred by the lane marking inference unit 32 using the correction coefficient generated by the correction coefficient calculation unit 33.

[0023] The storage unit 35 is configured using, for example, a non-volatile memory, and is configured to store the machine learning model M and the reference data REF.

[0024] With this configuration, the lane marking detection unit 22 (Figure 2) infers the position of the lane marking based on the captured image, and generates lane marking data DL by correcting the inferred position of the lane marking based on the height position of the stereo camera 11.

[0025] The roadway recognition unit 23 is configured to generate a recognition result RES by recognizing the roadway on which the vehicle 1 is traveling based on the lane marking data DL generated by the lane marking detection unit 22. Specifically, the roadway recognition unit 23 recognizes the shape and curvature of the roadway ahead of the vehicle 1, and outputs the result as the recognition result RES.

[0026] Here, the lane markings 101L, 101R correspond to a specific example of a "lane marking" in an embodiment of the present disclosure. The left image data PL and the right image data PR correspond to a specific example of "captured image data" in an embodiment of the present disclosure. The lane marking inference unit 32 corresponds to a specific example of an "estimation circuit" in an embodiment of the present disclosure. The stereo camera 11 corresponds to a specific example of an "imaging device" in an embodiment of the present disclosure. The lane marking correction unit 34 corresponds to a specific example of a "correction circuit" in an embodiment of the present disclosure. The correction coefficient calculation unit 33 corresponds to a specific example of a "correction coefficient calculation circuit" in an embodiment of the present disclosure.

[0027] [Actions and Actions] Next, the operation and function of the roadway recognition device 10 of the present embodiment will be described.

[0028] (Overview of overall operation) First, an overview of the overall operation of the roadway recognition device 10 will be described with reference to FIGS.

[0029] The stereo camera 11 captures an image ahead of the vehicle 1 to generate a set of image data (left image data PL and right image data PR) having a mutual parallax. The distance image generation unit 21 of the image processing device 20 performs predetermined image processing including stereo matching processing based on the left image data PL and right image data PR to generate distance image data PD. The lane line detection unit 22 detects lane lines dividing the road on which the vehicle 1 is traveling based on the left image data PL and right image data PR and the distance image data PD to generate lane line data DL indicating the positions of the lane lines on the road surface of the road. The lane recognition unit 23 recognizes the road on which the vehicle 1 is traveling based on the lane line data DL generated by the lane line detection unit 22 to generate a recognition result RES.

[0030] (Details of operation) FIG. 5 shows an example of the operation of the lane marking detection unit 22. As shown in FIG.

[0031] First, the height position detection unit 31 of the lane marking detection unit 22 detects the height position of the stereo camera 11 based on the road surface of the road, based on the left image data PL, the right image data PR, and the distance image data PD (step S101). In this example, the height position detection unit 31 detects the height position of the stereo camera 11 based on one of the left image data PL and the right image data PR, and the distance image data PD.

[0032] Fig. 6 shows an example of image data input to the height position detection unit 31. This image data is one of the left image data PL and the right image data PR. The image of this image data includes an image portion of the left lane marking 101L and an image portion of the right lane marking 101R. As shown in Fig. 6, the distance image data PD includes, for example, distance values ​​at multiple characteristic points (five characteristic points W in this example) on the road surface.

[0033] FIG. 7 shows an example of the operation of the height position detection unit 31. In FIG. 7, the Z direction indicates the vehicle length direction of the vehicle 1, and the Y direction indicates the height direction of the vehicle 1. The height position detection unit 31 plots the coordinates of a plurality of characteristic points on the road surface in the YZ plane, and calculates an approximation curve 102 using, for example, a quadratic function. The approximation curve 102 is a curve that indicates the road surface ahead of the vehicle 1. In the example of FIG. 7, the road is slightly uphill. The height position detection unit 31 then calculates the height position of the stereo camera 11 by extrapolating the approximation curve 102 to just below the stereo camera 11. The height position detection unit 31 calculates the height position of the stereo camera 11 based on, for example, each of the image data of a plurality of images captured at different times, and adopts the average value of the height positions as the height position Hs of the stereo camera 11. That is, for example, when the vehicle 1 is traveling, the calculated value of the height position may change due to bumps in the road surface, so the height position detection unit 31 calculates the height position Hs of the stereo camera 11 by calculating the average value of multiple height positions.

[0034] The height position Hg shown in this figure is the height position indicated by the reference data REF. That is, this height position Hg is the height position of the stereo camera in the vehicle that generated the image data used in generating the machine learning model M. In this example, in the image data used in generating the machine learning model M, the road is horizontal. In this example, the height position Hs is higher than the height position Hg in the vehicle that generated the image data used in generating the machine learning model M. Specifically, for example, the vehicle height of the vehicle 1 is higher than the vehicle height of the vehicle that generated the image data used in generating the machine learning model M.

[0035] Next, the lane line inference unit 32 of the lane line detection unit 22 infers the position of the lane line on the road surface of the road based on the left image data PL and the right image data PR using the machine learning model M (step S102). In this example, the lane line inference unit 32 infers the position of the lane line using the machine learning model M based on one of the left image data PL and the right image data PR.

[0036] FIG. 8 shows an example of the operation of the lane line inference unit 32. The lane line inference unit 32 infers the positions of the lane lines as shown in FIG. 8, for example, based on the image data shown in FIG. 6. The X direction indicates the vehicle width direction of the vehicle 1, and the Z direction indicates the vehicle length direction of the vehicle 1. In FIG. 8, multiple points indicate the positions of the lane lines. In this example, five points on the left side indicate the position of the lane line 101L, and five points on the right side indicate the position of the lane line 101R. In this example, the lane line inference unit 32 uses multiple points to represent the positions of the lane lines, but is not limited to this, and the positions of the lane lines may be represented by lines.

[0037] Next, the correction coefficient calculation unit 33 of the lane marking detection unit 22 generates a correction coefficient based on the height position of the stereo camera 11 detected by the height position detection unit 31 and the height position indicated by the reference data REF (step S103). In this example, the correction coefficient is Hs / Hg. Here, Hs is the height position detected in step S101, and Hg is the height position indicated by the reference data REF.

[0038] Next, the lane line correction unit 34 of the lane line detection unit 22 corrects the position of the lane line inferred in step S102 based on the correction coefficient obtained in step S103 (step S104).

[0039] 9 shows an example of an operation of the lane marking correction unit 34, where (A) shows an example when the height position of the stereo camera 11 is low, and (B) shows an example of an operation when the height position of the stereo camera 11 is high. In this example, the height position Hs of the stereo camera 11 in the example of (A) is the same as the height position Hg of the stereo camera in the vehicle that generated the image data used in generating the machine learning model M.

[0040] As shown in FIG. 9, the lane line inference unit 32 infers the position of the lane line on the road surface of the road using the machine learning model M based on, for example, one of the left image data PL and the right image data PR (step S102). In FIG. 9(A), the height position of the stereo camera 11 is low, and in FIG. 9(B), the height position of the stereo camera 11 is high, so that the processing result of the lane line inference unit 32 has a difference in the positions of the lane lines 101L and 101R. Specifically, when the height position of the stereo camera 11 is high (FIG. 9B), the intervals between the five points indicating the lane line 101L are narrower, the intervals between the five points indicating the lane line 101R are narrower, and the intervals between the five points indicating the lane line 101L and the five points indicating the lane line 101R are narrower than when the height position of the stereo camera 11 is low (FIG. 9A).

[0041] Then, the lane marking correction unit 34 corrects the position of the lane marking inferred by the lane marking inference unit 32 based on the correction coefficient (Hs / Hg) (step S104).

[0042] 9A, the height position Hs of the stereo camera 11 detected by the height position detection unit 31 is equal to the height position Hg indicated by the reference data REF (Hs=Hg). Therefore, as shown in FIG. 9A, the processing result of the lane marking correction unit 34 is the same as the processing result of the lane marking inference unit 32.

[0043] On the other hand, in the example of Fig. 9(B), the height position Hs of the stereo camera 11 detected by the height position detection unit 31 is higher than the height position Hg indicated by the reference data REF (Hs>Hg). Therefore, as shown in Fig. 9(B), in the processing result of the demarcation line correction unit 34, the intervals between the five points indicating the demarcation line 101L are wider, the intervals between the five points indicating the demarcation line 101R are wider, and the intervals between the five points indicating the demarcation line 101L and the five points indicating the demarcation line 101R are wider.

[0044] 9(B), the height position Hs of the stereo camera 11 detected by the height position detection unit 31 is different from the height position Hg indicated by the reference data REF, and the position of the lane line is corrected in accordance with this difference in height position. As a result, as shown in FIGS. 9(A) and 9(B), the processing result of the lane line correction unit 34 when the height position Hs of the stereo camera 11 is low (FIG. 9(A)) is substantially the same as the processing result of the lane line correction unit 34 when the height position Hs of the stereo camera 11 is high (FIG. 9(B)). In other words, the lane line detection unit 22 can suppress the influence of the height position of the stereo camera 11 on the position of the lane line.

[0045] This is the end of this process.

[0046] In this manner, in the lane line detection unit 22, the lane line correction unit 34 corrects the position of the lane line inferred by the lane line inference unit 32 based on the height position of the stereo camera 11. This makes it easier for the lane line detection unit 22 to perform machine learning processing compared to the case of the reference example described below.

[0047] (Reference example) Next, the lane marking detection unit 22R according to a reference example will be described. In this reference example, the data input to the machine learning model is different from that of the present embodiment. That is, in the present embodiment, image data is input to the machine learning model, but instead, in this reference example, image data and data on the height position of the stereo camera 11 are input to the machine learning model. Other configurations are the same as those of the present embodiment.

[0048] 10 shows an example of the configuration of the lane marking detection unit 22R. The lane marking detection unit 22R has a height position detection unit 31, a lane marking inference unit 32R, and a storage unit 35R.

[0049] The lane line inference unit 32R is configured to infer the position of the lane line on the road surface of the road using a machine learning model MR stored in the memory unit 35R based on the left image data PL and the right image data PR and the height position of the stereo camera 11 detected by the height position detection unit 31. This machine learning model MR is configured to receive the image data and the height position data of the stereo camera 11 and output the position of the lane line on the road surface of the road. This machine learning model MR is generated by a machine learning device configured using, for example, a personal computer performing machine learning processing, and is stored in advance in the memory unit 35R.

[0050] The storage unit 35R is configured using, for example, a non-volatile memory, and is configured to store the machine learning model MR.

[0051] In this reference example, image data and data on the height position of the stereo camera 11 are input to the machine learning model MR. Therefore, the machine learning process for generating this machine learning model MR is performed using a data set including the image data and data on the height position of the stereo camera 11. That is, it is necessary to prepare a plurality of data sets in which the height positions of the stereo camera 11 are different from each other, and perform the machine learning process using the plurality of data sets. Therefore, many data sets are required to perform the machine learning process, and the machine learning process may take a long time.

[0052] On the other hand, in the lane marking detection unit 22 according to the present embodiment, image data is input to the machine learning model M. Therefore, the machine learning process for generating this machine learning model M is performed using a data set including image data associated with one height position. That is, there is no need to prepare a plurality of data sets having different height positions. In addition, since a plurality of data sets having different height positions are not used, the time required for the machine learning process can be shortened. Therefore, the machine learning process can be made easier.

[0053] In this way, the image processing device 20 is provided with a demarcation line inference unit 32 capable of estimating the position of the demarcation line on the road surface of the road based on the captured image data (left image data PL and right image data PR) including images of the demarcation lines dividing the road, and a demarcation line correction unit 34 capable of correcting the position of the demarcation line on the road surface of the road estimated by the demarcation line inference unit 32 based on the height position of the stereo camera 11 that generated the captured image data with respect to the road surface of the road. This makes it possible to more accurately estimate the position of the demarcation line in various vehicle models, for example, and to more accurately estimate the position of the demarcation line even if the number of passengers or the amount of luggage loaded on the vehicle 1 changes. In addition, for example, the position of the demarcation line can be more accurately estimated even if the vehicle height is lowered due to aging deterioration of the suspension. As a result, the image processing device 20 can improve the recognition accuracy of the road.

[0054] That is, for example, the vehicle height varies depending on the model, so the height position of the stereo camera 11 may vary depending on the model. Also, for example, if the load on the vehicle 1 changes, the state of the suspension changes, so the height position of the stereo camera 11 may change. For example, when the height position of the stereo camera 11 is low, the captured image becomes an image as shown in FIG. 3A, and it may be determined that the interval between the marking lines is wide. Also, for example, when the height position of the stereo camera 11 is high, the captured image becomes an image as shown in FIG. 3B, and it may be determined that the interval between the marking lines is narrow. This results in a decrease in the recognition accuracy of the road.

[0055] Meanwhile, in the image processing device 20, the position of the lane line estimated by the lane line inference unit 32 is corrected based on the height position of the stereo camera 11. This makes it possible to suppress the influence of the height position of the stereo camera 11 on the position of the lane line, as shown in FIG. 9. Specifically, for example, in the image processing device 20, the position of the lane line is corrected based on the height position of the stereo camera 11 in various vehicle models. Also, for example, in the image processing device 20, the position of the lane line is corrected based on the height position of the stereo camera 11 according to the number of passengers, the amount of luggage, etc. As a result, the image processing device 20 can improve the recognition accuracy of the road.

[0056] The image processing device 20 is also provided with a height position detection unit 31 capable of detecting the height position of the stereo camera 11 based on the captured image data (left image data PL and right image data PR). The lane line correction unit 34 is capable of correcting the position of the lane line based on the height position of the stereo camera 11 detected by the height position detection unit 31. This makes it possible to correct the position of the lane line, for example, when the load on the vehicle 1 changes, based on the height position of the stereo camera 11 at that time. Therefore, the image processing device 20 can improve the recognition accuracy of the road.

[0057] The image processing device 20 further includes a correction coefficient calculation unit 33 that calculates a correction coefficient. The lane line inference unit 32 can estimate the position of the lane line using a machine learning model based on the captured image data (left image data PL and right image data PR), and the correction coefficient calculation unit 33 can generate a correction coefficient based on the height position of the stereo camera 11 and the height position of the reference. The correction coefficient calculation unit 33 can correct the position of the lane line based on the correction coefficient. This makes it easier to perform machine learning processing, as described in comparison with the reference example.

[0058] [effect] As described above, this embodiment is equipped with a lane marking inference unit that can estimate the position of the lane markings on the road surface of the road based on captured image data including images of the lane markings that divide the road, and a lane marking correction unit that can correct the position of the lane markings on the road surface of the road estimated by the lane marking inference unit based on the height position of the stereo camera that generated the captured image data with respect to the road surface of the road, thereby improving the accuracy of lane recognition.

[0059] In this embodiment, a height position detection unit is provided that can detect the height position of the stereo camera based on the captured image data. The lane marking correction unit is capable of correcting the position of the lane marking based on the height position of the stereo camera detected by the height position detection unit. This can improve the recognition accuracy of the road.

[0060] In this embodiment, a correction coefficient calculation unit that calculates a correction coefficient is further provided. The lane line inference unit is capable of estimating the position of the lane line using a machine learning model based on the captured image data, and the correction coefficient calculation unit is capable of generating a correction coefficient based on the height position of the stereo camera and the height position of the reference. The correction coefficient calculation unit is capable of correcting the position of the lane line based on the correction coefficient. This makes it easier to perform the machine learning process.

[0061] [Variation 1] In the above embodiment, the stereo camera 11 is provided, but the present invention is not limited to this, and instead, for example, a monocular camera may be provided. A detailed description will be given below of a road recognition device 10A according to this modified example.

[0062] 11 shows an example of the configuration of a road recognition device 10A. The road recognition device 10A includes an imaging device 11A, a distance sensor 12A, and an image processing device 20A.

[0063] The imaging device 11A is a monocular camera and is configured to generate image data P by capturing an image of the area ahead of the vehicle 1. The distance sensor 12A is, for example, a Lidar (Light Detection And Ranging) sensor and is configured to generate distance image data PD by detecting the distance to a subject.

[0064] The image processing device 20A is configured to recognize the road on which the vehicle 1 is traveling, based on the image data P supplied from the imaging device 11A and the distance image data PD supplied from the distance sensor 12A. The image processing device 20A has a lane marking detection unit 22A and a road recognition unit 23.

[0065] The lane marking detection unit 22A is configured to detect lane marks dividing the road on which the vehicle 1 is traveling, based on the image data P and the distance image data PD, and generate lane marking data DL indicating the positions of the lane marks on the road surface of the road. The lane marking detection unit 22A has a similar configuration to the lane marking detection unit 22 (FIG. 4) according to the above embodiment.

[0066] Here, the image data P corresponds to a specific example of "captured image data" in an embodiment of the present disclosure. The imaging device 11A corresponds to a specific example of "imaging device" in an embodiment of the present disclosure. The distance sensor 12A corresponds to a specific example of "distance sensor" in an embodiment of the present disclosure.

[0067] [Variation 2] In the above embodiment, the height position detection unit 31 is provided, but this is not limited thereto, and instead, for example, the height position detection unit 31 may not be provided. The following describes in detail the demarcation line detection unit 22B according to this modified example.

[0068] 12 shows an example of the configuration of the lane marking detection unit 22B. The lane marking detection unit 22B has a lane marking inference unit 32, a correction coefficient calculation unit 33B, a lane marking correction unit 34, and a storage unit 35B.

[0069] The correction coefficient calculation unit 33B is configured to generate a correction coefficient based on data on the height position Hs of the stereo camera 11 stored in the storage unit 35 and reference data REF stored in the storage unit 35.

[0070] The storage unit 35B is configured to store the machine learning model M, data on the height position Hs of the stereo camera 11, and reference data REF. The height position Hs is set to a value according to the type of vehicle 1, for example. Here, the storage unit 35B corresponds to a specific example of "storage unit" in an embodiment of the present disclosure.

[0071] FIG. 13 shows an example of the operation of the lane marking detection unit 22B.

[0072] First, the lane marking inference unit 32 of the lane marking detection unit 22B infers the positions of the lane marks on the road surface of the road based on the left image data PL and the right image data PR using the machine learning model M (step S111). This operation is the same as the operation of step S102 in the above embodiment.

[0073] Next, the correction coefficient calculation unit 33B of the lane marking detection unit 22B generates a correction coefficient based on the height position Hs of the stereo camera 11 stored in the storage unit 35 (step S112). The correction coefficient is Hs / Hg, as in the above embodiment.

[0074] Next, the lane marking correction unit 34 of the lane marking detection unit 22B corrects the position of the lane marking inferred in step S111 based on the correction coefficient obtained in step S112 (step S113). This operation is similar to the operation of step S104 in the above embodiment.

[0075] This is the end of this process.

[0076] As a result, the lane line correction unit 34 of the lane line detection unit 22B corrects the position of the lane line estimated by the lane line inference unit 32 based on the height position of the stereo camera 11. In this example, the height position of the stereo camera 11 is a value according to the vehicle type and is stored in the storage unit 35B. Therefore, the lane line detection unit 22B can perform correction according to the vehicle type of the vehicle 1.

[0077] In this example, the correction coefficient calculation unit 33B is provided, and the correction coefficient calculation unit 33B calculates the correction coefficient, but the present invention is not limited to this. Alternatively, for example, the correction coefficient calculation unit 33B may not be provided. In this case, the memory unit 35B stores the correction coefficient, and the lane line correction unit 34 can correct the position of the inferred lane line based on the correction coefficient stored in the memory unit 35B.

[0078] [Other variations] Also, two or more of these variations may be combined.

[0079] Although the present technology has been described above by giving embodiments and some modified examples, the present technology is not limited to these embodiments and the like, and various modifications are possible.

[0080] For example, in the above embodiment, the height position detection unit 31 calculates the height position Hs of the stereo camera 11 using the approximation curve 102, but this is not limited to this, and other methods may be used to calculate the height position Hs of the stereo camera 11. Specifically, the height position detection unit 31 may detect the height position of the stereo camera 11 using a machine learning technique based on the left image data PL, the right image data PR, and the distance image data PD, for example.

[0081] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0082] Furthermore, the present disclosure may take the following aspects.

[0083] (1) an estimation circuit capable of estimating a position of a marking line on a road surface of a road based on captured image data including an image of the marking line that divides the road; a correction circuit capable of correcting the position of the lane marking on the road surface of the road, which is estimated by the estimation circuit, based on a height position of an imaging device that generated the captured image data with respect to the road surface of the road; An image processing device comprising: (2) a height position detection circuit capable of detecting the height position of the imaging device based on the captured image data, The correction circuit is capable of correcting the position of the demarcation line based on the height position of the imaging device detected by the height position detection circuit. The image processing device according to (1) above. (3) A correction coefficient calculation circuit is further provided to calculate a correction coefficient, The estimation circuit is capable of estimating a position of the lane marking by using a machine learning model based on the captured image data, the correction coefficient calculation circuit is capable of generating the correction coefficient based on the height position of the imaging device and a reference height position; The correction circuit is capable of correcting the position of the demarcation line based on the correction coefficient. The image processing device according to (2) above. (4) The captured image data includes left image data and right image data, The height position detection circuit is capable of estimating the height position of the imaging device based on the captured image data and distance image data corresponding to the captured image data. The image processing device according to (2) or (3). (5) The height position detection circuit is further capable of estimating the height position of the imaging device based on a detection result of a distance sensor capable of detecting a distance to a subject of the imaging device. The image processing device according to (2) or (3). (6) a memory circuit capable of storing the height position of the imaging device; The correction circuit is capable of correcting a position of the demarcation line based on the height position of the imaging device stored in the memory circuit. The image processing device according to (1) above. [Explanation of symbols]

[0084] 1...vehicle, 10,10A...roadway recognition device, 11...stereo camera, 11A...imaging device, 11L...left camera, 11R...right camera, 12A...distance sensor, 20,20A...image processing device, 21...distance image generation unit, 22,22A,22B...land line detection unit, 23...roadway recognition unit, 31...height position detection unit, 32...land line inference unit, 33,33B...correction coefficient calculation unit, 34...land line correction unit, 35,35B...memory unit, DL...land line data, Hg,Hs...height position, M...machine learning model, P...image data, PD...distance image data, PIC...stereo image data, PL...left image data, PR...right image data, REF...reference data, RES...recognition result.

Claims

1. an estimation circuit capable of estimating a position of a marking line on a road surface of a road based on captured image data including an image of the marking line that divides the road; a correction circuit capable of correcting the position of the lane marking on the road surface of the road, which is estimated by the estimation circuit, based on a height position of an imaging device that generated the captured image data with respect to the road surface of the road; An image processing device comprising:

2. a height position detection circuit capable of detecting the height position of the imaging device based on the captured image data, The correction circuit is capable of correcting the position of the demarcation line based on the height position of the imaging device detected by the height position detection circuit. The image processing device according to claim 1 .

3. A correction coefficient calculation circuit is further provided to calculate a correction coefficient, The estimation circuit is capable of estimating a position of the lane marking by using a machine learning model based on the captured image data, the correction coefficient calculation circuit is capable of generating the correction coefficient based on the height position of the imaging device and a reference height position; The correction circuit is capable of correcting the position of the demarcation line based on the correction coefficient. The image processing device according to claim 2 .

4. The captured image data includes left image data and right image data, The height position detection circuit is capable of estimating the height position of the imaging device based on the captured image data and distance image data corresponding to the captured image data. The image processing device according to claim 2 .

5. The height position detection circuit is further capable of estimating the height position of the imaging device based on a detection result of a distance sensor capable of detecting a distance to a subject of the imaging device. The image processing device according to claim 2 .

6. a memory circuit capable of storing the height position of the imaging device; The correction circuit is capable of correcting a position of the demarcation line based on the height position of the imaging device stored in the memory circuit. The image processing device according to claim 1 .

Images