Object recognition method and object recognition device

Abstract

To provide an object recognition method and an object recognition device which can accurately recognize a road surface sign.SOLUTION: An object recognition device 1 comprises: a camera 11 which images a prescribed range in the periphery of a vehicle including a road surface; a sensor group 10 which emits light to a range at least overlapping the camera 11 and detects a position of a range finding point being a reflection point on the basis of reflection light; and a controller 20 which processes data acquired from the camera 11 and the sensor group 10. The controller 20 extracts a reference reflection point group being a reference of brightness on a photographed image in a pavement region, calculates a reference pixel value indicating a reference of the reference reflection point group, estimates the relative illuminance of each pixel on the photographed image by using a reference pixel value, corrects a luminance value of each pixel of the photographed image by using the relative illuminance, and recognizes a road surface sign from the correction image subjected to correction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an object recognition method and an object recognition device. [Background technology]

[0002] Conventionally, there has been known an invention that identifies shaded and non-shaded areas on an image, calculates color information of a light source based on hue information and luminance information of the shaded and non-shaded areas, and detects and removes the shaded areas (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-237976 Summary of the Invention [Problem to be solved by the invention]

[0004] However, since the invention described in Patent Document 1 determines whether an object is the same or not based on hue, there is a risk that an adjacent asphalt area and a white line area may be erroneously determined as a shaded area and a non-shaded area of ​​the same object.

[0005] The present invention has been made in view of the above problems, and an object of the present invention is to provide an object recognition method and an object recognition device that are capable of recognizing road markings with high accuracy. [Means for solving the problem]

[0006] An object recognition method according to one aspect of the present invention extracts road surface reflection points, which are reflection points from the road surface, from among a plurality of distance measurement points based on the positions of the distance measurement points detected by a position detection means; extracts a road marking area indicating the area where road markings exist in an image captured by an imaging means, and a pavement area indicating the area of ​​the road surface excluding the road marking area, based on the reflection intensity of the road surface reflection points; extracts a reference reflection point group that serves as a standard for brightness in the captured image in the pavement area; calculates a reference pixel value that indicates the standard for the reference reflection point group; estimates the relative illuminance of each pixel in the captured image using the reference pixel value; corrects the luminance value of each pixel in the captured image using the relative illuminance; and recognizes road surface markings from the corrected image. [Effects of the Invention]

[0007] According to the present invention, it is possible to recognize road markings with high accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of an object recognition device 1 according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of the reference reflection point group. [Figure 3] FIG. 3 is a diagram illustrating an example of relative illuminance. [Figure 4] FIG. 4 is a diagram illustrating an example of an image correction method. [Figure 5] FIG. 5 is a flowchart illustrating an example of the operation of the object recognition device 1. [Figure 6] FIG. 6 is a flowchart illustrating an example of the operation of the object recognition device 1. [Figure 7] FIG. 7 is a flowchart illustrating an example of the operation of the object recognition device 1. [Figure 8] FIG. 8 is a diagram illustrating an example of object recognition. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0010] An example of the configuration of an object recognition device 1 will be described with reference to Fig. 1. As shown in Fig. 1, the object recognition device 1 includes a sensor group 10, a camera 11, and a controller 20.

[0011] The object recognition device 1 may be mounted on a vehicle with an automatic driving function, or on a vehicle without an automatic driving function. The object recognition device 1 may also be mounted on a vehicle that is capable of switching between automatic driving and manual driving. The automatic driving function may also be a driving assistance function that automatically controls only some of the vehicle control functions, such as steering control, braking force control, and driving force control, to assist the driver in driving. In this embodiment, the object recognition device 1 will be described as being mounted on a vehicle with an automatic driving function.

[0012] 1, the object recognition device 1 may control various actuators such as a steering actuator, an accelerator pedal actuator, and a brake actuator based on the recognition results (position, shape, posture, etc. of the object) by the image recognition unit 24. This may enable highly accurate autonomous driving.

[0013] The sensor group 10 mainly includes sensors that measure the distance and direction to objects around the vehicle. One example of such a sensor is a LIDAR (Laser Imaging Detection and Ranging). A LIDAR emits light (laser light) at objects around the vehicle and measures the time it takes for the reflected light to hit the object and bounce back, thereby measuring the distance and direction to the object and recognizing the shape of the object. Furthermore, a LIDAR can also obtain the positional relationship of objects in three dimensions. The principles and mechanisms of a LIDAR are well known, so a detailed description will be omitted. Mapping is also possible using the reflected intensity of laser light. The sensor group 10 may also include a GPS receiver or GNSS receiver that detects the position of the vehicle. The sensor group 10 may also include a speed sensor, acceleration sensor, steering angle sensor, gyro sensor, brake oil pressure sensor, accelerator position sensor, etc. that detect the state of the vehicle. Information obtained by the sensor group 10 is output to the controller 20. Unless otherwise specified, the sensor group 10 will be described as a representative of the sensor group 10, and information output to the controller 20 will be described as information obtained by a LIDAR.

[0014] The camera 11 has an imaging element such as a CCD (charge-coupled device) or a CMOS (complementary metal oxide semiconductor). The installation location of the camera 11 is not particularly limited, but as an example, the camera 11 is installed in front, on the side, or behind the vehicle. The camera 11 continuously captures images of the surroundings of the vehicle at a predetermined cycle. The camera 11 detects objects present around the vehicle (pedestrians, bicycles, motorbikes, other vehicles, etc.) and information ahead of the vehicle (demarcation lines, traffic lights, signs, crosswalks, intersections, etc.). The images captured by the camera 11 are output to the controller 20. The images captured by the camera 11 are stored in a storage device (not shown), and the controller 20 may refer to the images stored in the storage device.

[0015] The area measured (detected) by the lidar and the area imaged (detected) by the camera 11 overlap in whole or in part.

[0016] The controller 20 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), memory, and input / output units. A computer program for functioning as the object recognition device 1 is installed in the microcomputer. By executing the computer program, the microcomputer functions as multiple information processing circuits included in the object recognition device 1. Note that while an example is shown here in which the multiple information processing circuits included in the object recognition device 1 are realized by software, it is of course possible to configure the information processing circuits by providing dedicated hardware for executing each of the information processes described below. The multiple information processing circuits may also be configured by individual hardware. The controller 20 includes, as examples of multiple information processing circuits (information processing functions), a point cloud acquisition unit 21, a road surface reflection point estimation unit 22, an image acquisition unit 23, an image recognition unit 24, a reference reflection point extraction unit 25, a reference pixel value calculation unit 26, a relative illuminance estimation unit 27, and an image illuminance correction unit 28. Note that the controller 20 may also be referred to as an ECU (Electronic Control Unit).

[0017] The point cloud acquisition unit 21 acquires a range measurement point cloud (a set of multiple range measurement points in three-dimensional coordinates) from the lidar. The point cloud acquisition unit 21 outputs the acquired range measurement point cloud to the road surface reflection point estimation unit 22.

[0018] The road surface reflection point estimation unit 22 estimates road surface reflection points based on the three-dimensional positions and shapes of the range-finding point cloud. In this embodiment, "road surface reflection points" refer to reflection points from the road surface among the multiple range-finding points. Reflection points other than those on the road surface while driving include, for example, reflection points from buildings. The road surface reflection point estimation unit 22 outputs the estimated road surface reflection points to the reference reflection point extraction unit 25.

[0019] The image acquisition unit 23 acquires an image captured by the camera 11. The image acquisition unit 23 outputs the acquired image to the image recognition unit 24. Note that if an area where an object to be recognized is likely to exist is known, the image acquisition unit 23 may extract and output only that area.

[0020] The image recognition unit 24 performs road marking recognition processing on the image acquired from the image acquisition unit 23. Road marking recognition processing is an example of image processing, and is a process of detecting and identifying road markings around the vehicle (mainly ahead of the vehicle) and associating their attribute information with each pixel. This type of image processing is well known, and for example, semantic segmentation, which estimates the likelihood that each pixel is a road marking, can be used. Also, attributes may be distinguished based on color, such as white lines and yellow lines. In this embodiment, "road markings" refer to markings on the road surface with special paint to provide guidance, guidance, warning, regulation, instructions, etc. necessary for road traffic, and examples include dividing lines (broken white lines), pedestrian crossings, stop lines, and direction arrows. The image recognition unit 24 outputs the processing results to the reference reflection point extraction unit 25.

[0021] Based on the reflection intensity of the laser light from the road surface reflection points and the image recognition results, the reference reflection point extraction unit 25 extracts reflection point groups (reference reflection point groups) that serve as a reference for the brightness of pixels on the image for each of the paved area and road marking area on the road surface. In this embodiment, the "paved area" refers to the asphalt portion of the road surface excluding areas where road markings exist. The "road marking area" refers to the area where road markings exist. The "reference reflection point group of the paved area" refers to a collection of reflection points from the paved area. Furthermore, the "reference reflection point group of the road marking area" refers to a collection of reflection points from the road marking area. The reference reflection point extraction unit 25 outputs the extracted reference reflection point groups to the reference pixel value calculation unit 26. Note that for the road marking area, clustering may be performed based on the color (hue, saturation) of the pixel values ​​of the points projected onto the image plane, and each cluster may be treated as a separate reference reflection point group. Furthermore, since the reflectivity of light (laser light) from paved road surfaces such as asphalt and concrete is higher than that of light (laser light) from white lines and other road markings, the lidar can determine whether a road surface reflection point is a paved road surface such as asphalt or concrete, or a road marking such as a white line, based on the reflectivity (i.e., the intensity of the reflected light). In other words, it is possible to determine that an area of ​​the road surface with a relatively high reflection intensity is a road marking area, and that an area of ​​the road surface with a relatively low reflection intensity is a paved area other than a road marking area.

[0022] The reference pixel value calculation unit 26 projects the reference reflection point cloud acquired from the reference reflection point extraction unit 25 onto the image plane, and calculates reference pixel values ​​that serve as references for each of the pavement area and the road marking area from the pixel values ​​of the projected point cloud. There are no particular restrictions on the method for calculating the reference pixel value, but as an example, it is calculated using the median and average values. If there are multiple reference reflection point clouds depending on the color of the road marking, a reference pixel value is calculated for each of them. The reference pixel value calculation unit 26 outputs the calculated reference pixel values ​​to the relative illuminance estimation unit 27.

[0023] The relative illuminance estimation unit 27 estimates the relative illuminance for each pixel of the image based on the reference pixel value calculated by the reference pixel value calculation unit 26, and generates a relative illuminance image. The relative illuminance estimation unit 27 outputs the generated relative illuminance image to the image illuminance correction unit 28. The relative illuminance is calculated by dividing the pixel value of each pixel by the reference pixel value. In addition, since the LIDAR ranging points are generally sparser than the image, areas on the image where there are no corresponding ranging points are calculated by interpolation.

[0024] The image illuminance correction unit 28 corrects the brightness of the image captured by the camera 11 using the relative illuminance image generated by the relative illuminance estimation unit 27. The correction is performed by dividing the image by the relative illuminance image. The corrected image is output to the image recognition unit 24, where it is subjected to object recognition processing again and output as the final recognition result. As described above, the object recognition device 1 may control various actuators such as a steering actuator, accelerator pedal actuator, and brake actuator based on the final recognition result (object position, shape, attitude, etc.) by the image recognition unit 24. This can realize highly accurate autonomous driving.

[0025] Next, each function of the controller 20 will be described in detail with reference to FIGS.

[0026] The scene (image 30) shown in FIG. 2 is a view ahead of the vehicle traveling on a road. Image 30 was captured by a camera 11 installed in front of the vehicle. As shown in FIG. 2, the vehicle is traveling in the left lane of a two-lane road. In FIG. 2, there is a space on the left side where vehicles can stop, which is a bus stop. Reference numerals 40 and 41 in FIG. 2 indicate asphalt areas (driving lane areas) on the road surface. More specifically, reference numeral 40 indicates the asphalt area on the left side, and reference numeral 41 indicates the asphalt area on the right side. Reference numerals 42 and 43 indicate dividing lines (white lines). Reference numeral 50 indicates an area illuminated by the sun (sunlight). Because the light in the area indicated by reference numeral 50 is strong, it may be difficult for the camera 11 to recognize an object (e.g., a white line). Therefore, in this embodiment, the area indicated by reference numeral 50 is corrected (corrected on the image) using information acquired by a lidar and an image 30 captured by the camera 11, thereby accurately recognizing an object (e.g., a white line) in the image. The region indicated by the reference numeral 50 may be expressed as a region where the spatial frequency is equal to or greater than a predetermined value.

[0027] The road surface reflection point estimation unit 22 estimates road surface reflection points based on the three-dimensional positions and shapes of the range-finding point cloud. In Fig. 2, road surface reflection points refer to reflection points from the areas indicated by the reference numerals 40 to 43, and 50. Note that in Fig. 2, the road surface includes areas other than those indicated by the reference numerals 40 to 43, and 50, so the road surface also includes reflection points from areas other than those indicated by the reference numerals 40 to 43, and 50. However, for convenience of explanation, the road surface reflection points will be explained here as reflection points from the areas indicated by the reference numerals 40 to 43, and 50.

[0028] Based on the reflection intensity of the road surface reflection points and the image recognition results, the reference reflection point extraction unit 25 extracts reflection point groups (reference reflection point groups) that serve as a reference for the brightness of pixels on the image for each of the pavement area and road marking area on the road surface. In FIG. 2, the "reference reflection point group for the pavement area" refers to a collection of reflection points from the areas indicated by reference numerals 40 and 41. The "reference reflection point group for the road marking area" refers to a collection of reflection points from the areas indicated by reference numerals 42 and 44. In this embodiment, the detection area of ​​the lidar and the imaging area of ​​the camera 11 overlap. By comparing the detection results of both, the reference reflection point group is extracted with high accuracy. For the area indicated by reference numeral 50, the "reference reflection point group for the pavement area" and the "reference reflection point group for the road marking area" are extracted based on the detection results of the lidar (the reflection intensity of the road surface reflection points).

[0029] The reference pixel value calculation unit 26 projects the reference reflection point cloud onto the image plane and calculates reference pixel values ​​for each of the pavement area and the road marking area from the pixel values ​​of the projected point cloud. The relative illuminance estimation unit 27 calculates the relative illuminance for each pixel of the image based on the reference pixel values ​​calculated by the reference pixel value calculation unit 26, generating a relative illuminance image. Specifically, in the area indicated by reference numeral 50, in the area (area on the image) detected by the lidar as a white line (road marking area), the pixel value of the white line is divided by the "reference pixel value of the road marking area." Similarly, in the area indicated by reference numeral 50, in the area (area on the image) detected by the lidar as asphalt (pavement area), the pixel value of the asphalt is divided by the "reference pixel value of the pavement area." This division corrects the brightness value of each pixel in the image 30. This division calculates the relative illuminance indicated by reference numeral 60 in Figure 3. By projecting this relative illuminance onto a two-dimensional image, a relative illuminance image is generated, as shown at 61 in FIG.

[0030] Next, the image illuminance correction unit 28 corrects the brightness of the image captured by the camera 11 using the relative illuminance image 61 generated by the relative illuminance estimation unit 27. Specifically, the image 30 in FIG. 2 is used as an input image, and the image illuminance correction unit 28 divides this input image by the relative illuminance image 61. This generates the illuminance-corrected image 70 shown in FIG. 4. As can be seen from the illuminance-corrected image 70, the brightness of the area 50 is corrected. Road markings such as white lines can be recognized with high accuracy on the corrected image.

[0031] Next, an example of the operation of the object recognition device 1 will be described with reference to the flowchart in Fig. 5. In step S101, the camera 11 captures an image of the area ahead of the vehicle. The process proceeds to step S103, where the image recognition unit 24 performs road marking recognition processing on the image acquired in step S101. This recognizes the white lines, asphalt, etc. shown in Fig. 2. In step S105, the point cloud acquisition unit 21 acquires a ranging point cloud (a collection of multiple ranging points in three-dimensional coordinates) from the lidar. The process proceeds to step S107, where the road surface reflection point estimation unit 22 estimates road surface reflection points based on the three-dimensional positions and shapes of the ranging point cloud acquired in step S105.

[0032] The process proceeds to step S109, where the reference reflection point extraction unit 25 extracts reflection point groups (reference reflection point groups) that serve as standards for the brightness of pixels on the image for each of the paved area and road marking area on the road surface, based on the reflection intensities of the road surface reflection points estimated in step S107 and the image recognition results. The process proceeds to step S111, where the reference pixel value calculation unit 26 projects the reference reflection point groups extracted in step S109 onto the image plane and calculates reference pixel values ​​that serve as standards for each of the paved area and road marking area from the pixel values ​​of the projected point groups. The process proceeds to step S113, where the relative illuminance estimation unit 27 estimates relative illuminance using the reference pixel values ​​calculated in step S111, and generates a relative illuminance image 61 shown in FIG. 3.

[0033] The process proceeds to step S115, where the image illuminance correction unit 28 corrects the brightness of the image captured by the camera 11 using the relative illuminance image 61 generated in step S113 (see FIG. 4). The process proceeds to step S117, where the image recognition unit 24 recognizes road markings such as white lines using the image corrected in step S115. This makes it possible to recognize road markings on the image even when the camera 11 cannot recognize them due to strong sunlight.

[0034] An example of a method for extracting the reference reflection point cloud for a road marking area and the reference reflection point cloud for a pavement area will be described with reference to Figures 6 and 7. First, an example of a method for extracting the reference reflection point cloud for a road marking area will be described with reference to Figure 6. In step S201, the road surface reflection point estimation unit 22 determines whether each ranging point has been reflected from the road surface or something else (a three-dimensional object) based on the three-dimensional position and shape of the ranging point cloud. From step S201 onwards, the description will be given assuming that each ranging point has been determined to have reflected from the road surface. If there are unprocessed road surface reflection points in step S203 (YES in step S203), the process proceeds to step S205, where one unprocessed road surface reflection point is selected. If NO in step S203, the process ends.

[0035] If the reflection intensity of the ranging point is equal to or greater than the threshold in step S207 (YES in step S207), the process proceeds to step S209, where the ranging point is projected onto the image plane. If the projected ranging point in step S201 is recognized as a road marking on the image (YES in step S211), the process proceeds to step S213, where the ranging point is determined to be a reference reflection point in the road marking area. The threshold may be changed depending on the distance to the ranging point and the normal direction. Specifically, the threshold may be made smaller as the distance to the ranging point increases. The threshold may be made smaller as the angle between the laser incident direction and the normal direction of the ranging point increases. If NO in step S207, NO in step S211, or after the process of step S213, the process returns to step S203.

[0036] Next, an example of a method for extracting a reference reflection point group for a pavement area will be described with reference to Figure 7. However, the processing of steps S301 to S305 and 309 is the same as the processing of steps S201 to S205 and 209 shown in Figure 6, and therefore description thereof will be omitted. In step S307, if the reflection intensity of the ranging point is equal to or less than the threshold (YES in step S307), it is determined to be a reflection point for a pavement area. In step S311, if the projected ranging point is not recognized as a road marking on the image (YES in step S311), the processing proceeds to step S315, where it is determined whether or not a reference reflection point for the road marking area extracted in the processing of step S213 in Figure 6 exists near the ranging point. If a reference reflection point for the road marking area exists near the ranging point (YES in step S315), the processing proceeds to step S317, where it is determined to be a reference reflection point for a pavement area. If the result of step S307 is NO, if the result of step S311 is NO, if the result of step S315 is NO, or after the processing of step S317, the processing returns to step S303. Note that the threshold value may be changed depending on the distance to the ranging point and the normal direction, as in FIG. 6.

[0037] FIG. 8 shows an example of an image after correction in step S115 shown in FIG. 8. Reference numeral 81 in FIG. 8 indicates a crosswalk, reference numeral 82 indicates an arrow line, and reference numeral 83 indicates a dividing line (white line). Reference numerals 81 to 83 indicate areas where strong sunlight hits and objects cannot be recognized by the camera 11. Image correction according to this embodiment makes it possible to recognize road markings on the image even in cases where strong sunlight makes it impossible for the camera 11 to recognize them. Note that the scene shown in FIG. 8 is different from the scene shown in FIG. 2.

[0038] (Action and effect) As described above, the object recognition device 1 according to this embodiment provides the following advantageous effects.

[0039] The object recognition device 1 includes an imaging means for capturing an image of a predetermined range around the vehicle, including the road surface; a position detection means for emitting light at least within a range overlapping the imaging means and detecting the positions of ranging points, which are reflection points, based on the reflected light; and a controller 20 for processing data acquired from the imaging means and the position detection means. An example of the "imaging means" is a camera 11. An example of the "position detection means" is a lidar. An example of "light" is laser light. Based on the positions of the ranging points detected by the position detection means, the controller 20 extracts road surface reflection points, which are reflection points from the road surface, from among the multiple ranging points. Based on the reflection intensity of the road surface reflection points, the controller 20 extracts a road marking area, which indicates the area where road markings exist in the captured image captured by the imaging means, and a pavement area, which indicates the area of ​​the road surface excluding the road marking area. The controller 20 extracts a reference reflection point group that serves as a reference for the brightness of the captured image in the pavement area. The controller 20 calculates a reference pixel value that indicates the reference for the reference reflection point group. The controller 20 estimates the relative illuminance of each pixel in the captured image using the reference pixel value. The controller 20 corrects the brightness value of each pixel in the captured image using the relative illuminance. The controller 20 recognizes road markings from the corrected image. According to this embodiment, it is possible to correct the brightness on the road surface due to the shadows of structures and objects around the road, making it possible to recognize road markings with high accuracy. Furthermore, because the reflection intensity of the laser light emitted from the lidar is used, it is possible to distinguish between objects with high reflection intensity (road markings) and objects with low reflection intensity (asphalt) regardless of the hue of the target and the adjacent relationship of the areas. In the above embodiment, an example was described in which reference pixel values ​​of both the "reference reflection point cloud of the pavement area" and the "reference reflection point cloud of the road marking area" are used, but this is not limited to this. Only the reference pixel value of the "reference reflection point cloud of the pavement area" may be used.

[0040] The reference reflection point group may be defined as a first reference reflection point group, and the reference pixel value may be defined as a first reference pixel value. The controller 20 may extract a second reference reflection point group that serves as a reference for brightness on the captured image in the road marking area, calculate a second reference pixel value that indicates the reference for the second reference reflection point group, estimate the relative illuminance of each pixel on the captured image using the second reference pixel value, and correct the luminance value of each pixel in the captured image using the relative illuminance. This enables road markings to be recognized with high accuracy.

[0041] The controller 20 may cluster the second reference reflection point group according to the hue and saturation of the corresponding pixels and estimate the relative illuminance of each cluster. By performing clustering processing for each road marking color (white, yellow, etc.), it is possible to set an appropriate reference pixel value, so that it is possible to calculate an appropriate relative illuminance and correct shading even when road markings of multiple colors exist.

[0042] The greater the distance from the position detection means to the road surface reflection point, the smaller the threshold value of the reflection intensity used to determine whether the road surface reflection point is a road marking or not. The greater the distance to the road surface reflection point (distance measurement point), the weaker the reflection intensity becomes, but by setting the threshold value in this way, it is possible to suppress this effect, making it possible to accurately distinguish between paved areas (asphalt) and road markings.

[0043] The controller 20 may estimate the normal direction for each road surface reflection point based on its shape. The larger the angle between the incident direction of light (laser light) and the normal direction, the smaller the reflection intensity threshold used to determine whether the road surface reflection point is a road marking. The larger the incident angle of the light emitted by the lidar to the target surface (the closer the angle between the normal to the target surface and the incident direction of the emitted light is to a right angle), the weaker the reflection intensity will be. However, by setting the threshold in this way, the effect of this can be suppressed, making it possible to accurately distinguish between paved areas (asphalt) and road markings.

[0044] The controller 20 may perform a smoothing process when interpolating the relative illuminance on the captured image to generate a relative illuminance image. Because pavement areas and road markings have a variety of colors microscopically, small errors will occur if the relative illuminance is calculated based on a single point. However, in outdoor environments, illuminance generally changes smoothly except at the boundaries between shaded and non-shaded areas. Therefore, smoothing the relative illuminance image makes it possible to suppress the effects of small color differences on the road surface.

[0045] The controller 20 may recognize road markings from the corrected image only in areas on the captured image where the relative illuminance is outside a predetermined range. An example of an "area where the relative illuminance is outside a predetermined range" is the area indicated by reference numeral 50 in FIG. 2. By performing image recognition on only a portion of the image, the computational load can be reduced. Of course, as described above, image recognition may also be performed on the entire image. By performing image recognition on the entire image, it becomes possible to recognize road markings with high accuracy using the entire image as a clue.

[0046] Each function described in the above embodiments may be implemented by one or more processing circuits. A processing circuit includes a programmed processing device, such as a processor including electrical circuitry. A processing circuit also includes devices, such as application specific integrated circuits (ASICs) or circuit components, arranged to perform the described functions.

[0047] Although the embodiments of the present invention have been described above, the descriptions and drawings that form part of this disclosure should not be understood to limit the present invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure. [Explanation of symbols]

[0048] 1 object recognition device, 10 sensor group, 11 camera, 20 controller, 21 point cloud acquisition unit, 22 road surface reflection point estimation unit, 23 image acquisition unit, 24 image recognition unit, 25 reference reflection point extraction unit, 26 reference pixel value calculation unit, 27 relative illuminance estimation unit, 28 image illuminance correction unit,

Claims

1. An object recognition method for an object recognition device comprising: an imaging means for imaging a predetermined range around a vehicle including a road surface; a position detection means for emitting light into at least a range overlapping with the imaging means and detecting the position of a distance measurement point, which is a reflection point, based on the reflected light; and a controller for processing data acquired from the imaging means and the position detection means, The controller extracting road surface reflection points, which are reflection points from the road surface, from among the plurality of distance measurement points based on the positions of the distance measurement points detected by the position detection means; extracting a road marking area indicating an area where road markings exist on the image captured by the imaging means and a pavement area indicating an area of ​​the road surface excluding the road marking area based on the reflection intensities of the road surface reflection points; extracting a first reference reflection point group that serves as a reference for brightness on the captured image of the paved area; extracting a second reference reflection point group that serves as a reference for brightness on the captured image in the road marking area; calculating a first reference pixel value indicating a reference for the first reference reflection point group from a pixel value of each pixel of the captured image; calculating a second reference pixel value indicating a reference for the second reference reflection point group from the pixel value of each pixel of the captured image; estimating relative illuminance of each pixel on the captured image using the first reference pixel value and the second reference pixel value; correcting the luminance value of each pixel of the captured image using the relative illuminance; The road marking is recognized from the corrected image.

1. An object recognition method comprising:

2. The controller clusters the second reference reflection points according to the hue and saturation of the corresponding pixels and estimates the relative illuminance of each cluster.

2. The object recognition method according to claim 1.

3. An object recognition method for an object recognition device comprising: an imaging means for imaging a predetermined range around a vehicle including a road surface; a position detection means for emitting light into at least a range overlapping with said imaging means and detecting the position of a ranging point which is a reflection point based on the reflected light; and a controller for processing data acquired from said imaging means and said position detection means, The controller extracting road surface reflection points, which are reflection points from the road surface, from among the plurality of distance measurement points based on the positions of the distance measurement points detected by the position detection means; extracting a road marking area indicating an area where road markings exist on the image captured by the imaging means and a pavement area indicating an area of ​​the road surface excluding the road marking area based on the reflection intensities of the road surface reflection points; extracting a reference reflection point group that serves as a reference for brightness on the captured image of the pavement area; calculating a reference pixel value indicating a reference for the reference reflection point group from the pixel value of each pixel of the captured image; estimating the relative illuminance of each pixel on the captured image using the reference pixel value; correcting the luminance value of each pixel of the captured image using the relative illuminance; Recognizing the road marking from the corrected image; The greater the distance from the position detection means to the road surface reflection point, the smaller the threshold value of the reflection intensity used to determine whether the road surface reflection point is the road surface marking.

1. An object recognition method comprising:

4. An object recognition method for an object recognition device comprising: an imaging means for imaging a predetermined range around a vehicle including a road surface; a position detection means for emitting light into at least a range overlapping with said imaging means and detecting the position of a ranging point which is a reflection point based on the reflected light; and a controller for processing data acquired from said imaging means and said position detection means, The controller extracting road surface reflection points, which are reflection points from the road surface, from among the plurality of distance measurement points based on the positions of the distance measurement points detected by the position detection means; extracting a road marking area indicating an area where road markings exist on the image captured by the imaging means and a pavement area indicating an area of ​​the road surface excluding the road marking area based on the reflection intensities of the road surface reflection points; extracting a reference reflection point group that serves as a reference for brightness on the captured image of the pavement area; calculating a reference pixel value indicating a reference for the reference reflection point group from the pixel value of each pixel of the captured image; estimating the relative illuminance of each pixel on the captured image using the reference pixel value; correcting the luminance value of each pixel of the captured image using the relative illuminance; Recognizing the road marking from the corrected image; The controller estimates a normal direction for each of the road surface reflection points based on the shape of the road surface reflection points; The larger the angle between the incident direction of the light and the normal direction, the smaller the threshold value of the reflection intensity used to determine whether the road surface reflection point is the road surface marking.

1. An object recognition method comprising:

5. An object recognition method for an object recognition device comprising: an imaging means for imaging a predetermined range around a vehicle including a road surface; a position detection means for emitting light into at least a range overlapping with said imaging means and detecting the position of a ranging point which is a reflection point based on the reflected light; and a controller for processing data acquired from said imaging means and said position detection means, The controller extracting road surface reflection points, which are reflection points from the road surface, from among the plurality of distance measurement points based on the positions of the distance measurement points detected by the position detection means; extracting a road marking area indicating an area where road markings exist on the image captured by the imaging means and a pavement area indicating an area of ​​the road surface excluding the road marking area based on the reflection intensities of the road surface reflection points; extracting a reference reflection point group that serves as a reference for brightness on the captured image of the pavement area; calculating a reference pixel value indicating a reference for the reference reflection point group from the pixel value of each pixel of the captured image; estimating the relative illuminance of each pixel on the captured image using the reference pixel value; correcting the luminance value of each pixel of the captured image using the relative illuminance; Recognizing the road marking from the corrected image; The controller performs a smoothing process when generating a relative illuminance image by interpolating the relative illuminance on the captured image.

1. An object recognition method comprising:

6. An object recognition method for an object recognition device comprising: an imaging means for imaging a predetermined range around a vehicle including a road surface; a position detection means for emitting light into at least a range overlapping with said imaging means and detecting the position of a ranging point which is a reflection point based on the reflected light; and a controller for processing data acquired from said imaging means and said position detection means, The controller extracting road surface reflection points, which are reflection points from the road surface, from among the plurality of distance measurement points based on the positions of the distance measurement points detected by the position detection means; extracting a road marking area indicating an area where road markings exist on the image captured by the imaging means and a pavement area indicating an area of ​​the road surface excluding the road marking area based on the reflection intensities of the road surface reflection points; extracting a reference reflection point group that serves as a reference for brightness on the captured image of the pavement area; calculating a reference pixel value indicating a reference for the reference reflection point group from the pixel value of each pixel of the captured image; estimating the relative illuminance of each pixel on the captured image using the reference pixel value; correcting the luminance value of each pixel of the captured image using the relative illuminance; Recognizing the road marking from the corrected image; The controller recognizes the road marking from the corrected image only in an area on the captured image where the relative illuminance is outside a predetermined range.

1. An object recognition method comprising:

7. an imaging means for imaging a predetermined range around the vehicle including the road surface; a position detection means for emitting light to at least an area overlapping with the imaging means and detecting the position of a distance measurement point, which is a reflection point, based on the reflected light; a controller that processes data acquired from the imaging means and the position detection means, The controller extracting road surface reflection points, which are reflection points from the road surface, from among the plurality of distance measurement points based on the positions of the distance measurement points detected by the position detection means; extracting a road marking area indicating an area where road markings exist on the image captured by the imaging means and a pavement area indicating an area of ​​the road surface excluding the road marking area based on the reflection intensities of the road surface reflection points; extracting a reference reflection point group that serves as a reference for brightness on the captured image of the pavement area; calculating a reference pixel value indicating a reference for the reference reflection point group from the pixel value of each pixel of the captured image; estimating the relative illuminance of each pixel on the captured image using the reference pixel value; correcting the luminance value of each pixel of the captured image using the relative illuminance; Recognizing the road marking from the corrected image; The controller recognizes the road marking from the corrected image only in an area on the captured image where the relative illuminance is outside a predetermined range. An object recognition device characterized by:

8. An imaging means for imaging a predetermined range around a vehicle including a road surface; a position detection means for emitting light to at least an area overlapping with the imaging means and detecting the position of a distance measurement point, which is a reflection point, based on the reflected light; a controller that processes data acquired from the imaging means and the position detection means, The controller extracting road surface reflection points, which are reflection points from the road surface, from among the plurality of distance measurement points based on the positions of the distance measurement points detected by the position detection means; extracting a road marking area indicating an area where road markings exist on the image captured by the imaging means and a pavement area indicating an area of ​​the road surface excluding the road marking area based on the reflection intensities of the road surface reflection points; extracting a reference reflection point group that serves as a reference for brightness on the captured image of the pavement area; calculating a reference pixel value indicating a reference for the reference reflection point group from the pixel value of each pixel of the captured image; estimating the relative illuminance of each pixel on the captured image using the reference pixel value; correcting the luminance value of each pixel of the captured image using the relative illuminance; Recognizing the road marking from the corrected image; The controller performs a smoothing process when generating a relative illuminance image by interpolating the relative illuminance on the captured image. An object recognition device characterized by:

Images