Image Recognition Device

Abstract

To provide an image recognition device which can appropriately recognize a road marking for a self vehicle.SOLUTION: An image recognition device comprises: a marking object detection unit 21 which can detect a marking object on the basis of an image obtained by taking an image in front of a self vehicle; a demarcation line recognition unit 22 which can recognize a pair of demarcation lines demarcating a lane on the basis of the image; a corresponding point specification unit 23 which specifies a pair of corresponding points on the pair of demarcation lines corresponding to a position of the marking object in a depth direction based on the self vehicle on the basis of the image; and an estimation unit 24 which estimates that the lane demarcated by the pair of demarcation lines to which the pair of corresponding points belong is the lane targeted by the marking object when the marking object is located between the pair of corresponding points in a width direction of the lane.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image recognition device. [Background technology]

[0002] There are techniques for detecting road markings, road signs, or traffic signals based on images captured by an on-board camera. For example, Patent Document 1 describes a technique for detecting road signs lined up above a road. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-156389 A Summary of the Invention [Problem to be solved by the invention]

[0004] Information on the detected road signs, etc. is used for driving assistance to the host vehicle, for example, for informing the driver, driving control, etc. As a result of studies by the present inventors, a problem was found in that there is a possibility that driving assistance to the host vehicle may not be appropriately performed due to the recognition of road signs, etc. that are not intended for the host vehicle, for example, road signs, etc. that are intended for a lane other than the host vehicle's own lane.

[0005] One aspect of the present disclosure provides an image recognition device capable of appropriately recognizing road markings, road signs, or traffic signals targeted at a host vehicle. [Means for solving the problem]

[0006] One aspect of the present disclosure is an image recognition device including a marking object detection unit (21), a demarcation line recognition unit (22), a corresponding point identification unit (23), and an estimation unit (24). The marking object detection unit is configured to detect a marking object based on an image captured in front of the vehicle. The demarcation line recognition unit is configured to recognize a pair of demarcation lines that define a lane based on the image. The corresponding point identification unit is configured to identify a pair of corresponding points on the pair of demarcation lines that correspond to the position of the marking object in the depth direction based on the vehicle based on the image. The estimation unit is configured to estimate that when the marking object is between the pair of corresponding points in the width direction of the lane, the lane defined by the pair of demarcation lines to which the pair of corresponding points belong is the lane targeted by the marking object.

[0007] According to this configuration, an image recognition device capable of appropriately recognizing road markings targeting the host vehicle is provided. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an image recognition device according to a first embodiment. [Diagram 2] 13 is a flowchart showing a target lane estimation process. [Diagram 3] FIG. 1 is a diagram showing the results of road marking detection and lane recognition in a certain scene. [Figure 4] FIG. 13 is a diagram illustrating a method for comparing the coordinates of a road marking with the coordinates of a demarcation line. [Diagram 5] FIG. 11 is a block diagram showing the configuration of an image recognition device according to a second embodiment. [Figure 6] 13 is a flowchart showing a traffic light identification process. [Figure 7] FIG. 13 is a diagram showing a scene in which two traffic signals are detected within the own lane area. [Figure 8] FIG. 8 is a diagram showing the identification result of a traffic signal in the scene shown in FIG. 7. [Figure 9] FIG. 13 is a diagram showing a situation in which the vehicle's own lane area cannot be recognized. [Figure 10]FIG. 10 is a diagram showing the identification result of a traffic signal in the scene shown in FIG. [Figure 11] FIG. 13 is a block diagram showing the configuration of an image recognition device according to a third embodiment. [Figure 12] 13 is a flowchart showing a road sign determination process. [Figure 13] FIG. 13 is a diagram showing a situation in which a road sign for a lane branching off from the own lane is detected. [Figure 14] 14 is a diagram showing the depth from the vehicle to a road sign in the scene shown in FIG. 13. FIG. [Figure 15] FIG. 13 is a diagram showing a situation in which a structure installed at the boundary between a main road and a side road is detected. [Figure 16] FIG. 16 is a diagram showing the distance from a structure installed on the boundary between a main road and a side road to a road sign in the scene shown in FIG. 15. [Figure 17] FIG. 13 is a diagram showing a scene in which a road sign targeting the host vehicle is detected. [Figure 18] FIG. 18 is a diagram showing the distance from a guardrail installed at the boundary between the sidewalk and the roadway to a road sign in the scene shown in FIG. 17. [Figure 19] 13 is a diagram showing the distance between a road sign and the lane closest to the road sign when some lanes on a road with many lanes on one side cannot be detected; FIG. [Figure 20] 13 is a diagram showing the distance between a lane closest to a road sign and a road sign when some lanes and three-dimensional objects on a road with many lanes on one side cannot be detected; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. [1. First embodiment] [1-1. Configuration] 1 is a device that is mounted on a vehicle such as an automobile, detects a marking based on an image input from an imaging device 11 mounted on the vehicle, and estimates the lane that the detected marking corresponds to. Hereinafter, the vehicle on which the image recognition device 101 is mounted is referred to as the subject vehicle.

[0010] The imaging device 11 is a camera system mounted, for example, on the central front side of the vehicle and captures images of the area in front of the vehicle at predetermined time intervals. A monocular camera is used for the imaging device 11. The imaging device 11 outputs the captured image (hereinafter, referred to as the "captured image") to the image recognition device 101.

[0011] The image recognition device 101 is mainly configured with a microcomputer including a CPU, semiconductor memories such as RAM and ROM, I / O, bus lines connecting these components, etc. As shown in Fig. 1, the image recognition device 101 includes a marker detection unit 21, a demarcation line recognition unit 22, a corresponding point identification unit 23, and an estimation unit 24 as functional blocks realized by the CPU executing a program stored in the semiconductor memory.

[0012] The sign detection unit 21 is configured to be able to detect a sign based on a captured image. The "sign" here refers to a sign that is placed on the road surface or around the road and indicates information related to traffic. In this embodiment, the sign detection unit 21 detects, as a sign, a road marking drawn on the road surface that indicates information related to a specific lane. The sign detection unit 21 detects the road marking based on a model constructed by machine learning. When a sign is detected, the sign detection unit 21 sets a rectangular detection frame that surrounds a portion of the captured image where the sign is present.

[0013] The sign detection unit 21 also obtains the coordinates of the detected sign in a world coordinate system with three axes: the width direction of the vehicle, the height direction of the vehicle, and the depth direction based on the vehicle. The coordinates in the world coordinate system are estimated using a moving stereo method using a monocular camera. This method estimates the three-dimensional position of the target object based on the principle of triangulation, by associating feature points between two-dimensional captured images at different positions obtained by moving the vehicle, and the amount of movement of the vehicle.

[0014] The demarcation line recognition unit 22 is configured to recognize a pair of demarcation lines that define lanes on a road based on a captured image. The demarcation line recognition unit 22 recognizes the pair of demarcation lines by detecting lines drawn on the road based on a model constructed by machine learning. Similarly to the sign detection unit 21, the demarcation line recognition unit 22 determines the coordinates in the world coordinate system for the recognized pair of demarcation lines.

[0015] The corresponding point specifying unit 23 is configured to specify a pair of corresponding points on a pair of demarcation lines that correspond to the position of the sign in the depth direction, based on the detection result by the sign detection unit 21 and the recognition result by the demarcation line recognition unit 22. A method for specifying a pair of corresponding points will be described in detail later.

[0016] The estimation unit 24 is configured to estimate a lane that is the target of a sign detected by the sign detection unit 21. A method for estimating the target lane will also be described in detail later. [1-2. Processing] Next, the target lane estimation process executed by the image recognition device 101 will be described with reference to the flowchart of Fig. 2. The target lane estimation process is executed every time a captured image is input from the imaging device 11.

[0017] First, in S101, the image recognition device 101 performs the marking detection process in the captured image as described above. For example, in the captured image shown in Fig. 3, a road marking 51 on the own lane and a road marking 52 on the adjacent lane to the right of the own lane are detected, and detection frames 53 and 54 surrounding the road markings 51 and 52, respectively, are set.

[0018] Next, in S102, the image recognition device 101 determines whether or not a sign has been detected in the process of S101. If it is determined in S102 that no sign has been detected, the image recognition device 101 ends the target lane estimation process in FIG.

[0019] On the other hand, when the image recognition device 101 determines that a sign has been detected in S102, the process proceeds to S103, where the image recognition device 101 performs a recognition process of a pair of demarcating lines that define a lane as described above. Specifically, the image recognition device 101 recognizes a pair of demarcating lines that define the own lane, and, if there are adjacent lanes on the left and right sides of the own lane, a pair of demarcating lines that define each of the adjacent lanes on the left and right sides. For example, in the captured image shown in FIG. 3, demarcating lines 55a and 55b are recognized as a pair of demarcating lines that define the own lane, and demarcating lines 55b and 55c are recognized as a pair of demarcating lines that define the adjacent lane to the right of the own lane. Note that, when only one of the pair of demarcating lines can be detected due to reasons such as the line on the road being thin, the image recognition device 101 recognizes the pair of demarcating lines by assuming the lane width and estimating the other demarcating line.

[0020] Next, in S104, the image recognition device 101 determines whether or not a pair of defining lines defining the own lane has been recognized. If the image recognition device 101 determines in S104 that the pair of defining lines defining the own lane cannot be recognized, the image recognition device 101 proceeds to S105 and determines that there is no estimation result.

[0021] On the other hand, when the image recognition device 101 determines in S104 that the pair of demarcation lines defining the own lane have been recognized, the process proceeds to S106, where the image recognition device 101 identifies a pair of corresponding points on the pair of demarcation lines corresponding to the position of the detected marking in the depth direction. Specifically, as shown in Fig. 4, among the points constituting each of the pair of demarcation lines 55a, 55b defining the own lane, any point within the detection frame 53 of the road marking 51 in the depth direction, for example, a point closest to the center of gravity 57 of the detection frame 53, is identified as a pair of corresponding points 58a, 58b.

[0022] Next, in S107, the image recognition device 101 compares the coordinates of the marking with the coordinates of the pair of corresponding points to estimate the lane targeted by the marking. Specifically, when the marking is between the pair of corresponding points in the width direction of the vehicle, the image recognition device 101 estimates that the lane defined by the pair of demarcation lines to which the pair of corresponding points belong is the lane targeted by the marking. For example, in the captured image shown in FIG. 4, the coordinates of the center of gravity 57 of the road marking 51 are between the pair of corresponding points 58a, 58b on the pair of demarcation lines 55a, 55b that define the own lane in the width direction of the vehicle, so that the lane targeted by the road marking 51 is estimated to be the own lane.

[0023] After that, the image recognition device 101 ends the target lane estimation process in FIG. [1-3. Effects] According to the first embodiment described above in detail, the following effects can be obtained.

[0024] (1a) The image recognition device 101 identifies a pair of corresponding points on a pair of demarcation lines that correspond to the position of the marking object in the depth direction based on the vehicle, based on the captured image. Then, when the marking object is between the pair of corresponding points in the width direction of the vehicle, the image recognition device 101 estimates that the lane defined by the pair of demarcation lines to which the pair of corresponding points belongs is the lane targeted by the marking object.

[0025] That is, the image recognition device 101 does not compare the coordinates of the marking and the pair of demarcation lines on the captured image, which is a two-dimensional coordinate system, but compares the coordinates of the marking and the pair of demarcation lines in the world coordinate system, which is a three-dimensional coordinate system, and can therefore accurately estimate the lane that the marking targets.

[0026] (1b) The image recognition device 101 identifies, among the points constituting the pair of demarcation lines, points closest to the marking in the depth direction as a pair of corresponding points. With this configuration, points on the pair of demarcation lines that are almost directly to the side of the marking are identified as a pair of corresponding points, so that the lane targeted by the marking can be accurately estimated.

[0027] In the first embodiment, S101 corresponds to processing performed by the marker detection unit 21, S103 corresponds to processing performed by the demarcation line recognition unit 22, S106 corresponds to processing performed by the corresponding point identification unit 23, and S107 corresponds to processing performed by the estimation unit 24.

[0028] [2. Second embodiment] [2-1. Configuration] The second embodiment has a basic configuration similar to that of the first embodiment, and therefore differences will be described below. Note that the same reference numerals as those in the first embodiment indicate the same configuration, and the preceding description will be referred to.

[0029] The image recognition device 102 shown in FIG. 5 is mounted on a vehicle and detects traffic signals based on captured images input from an imaging device 11 mounted on the vehicle, and identifies traffic signals targeted at the vehicle itself (hereinafter also referred to as "traffic signals for the vehicle itself").

[0030] The image recognition device 102 includes a traffic light detection unit 31, an area recognition unit 32, a calculation unit 33, and a traffic light identification unit 34 as functional blocks realized by a CPU executing a program stored in a semiconductor memory.

[0031] The traffic light detection unit 31 is configured to detect a traffic light based on the captured image and identify the lighting color of the traffic light. Specifically, the traffic light detection unit 31 first detects a traffic light based on a model constructed by machine learning. When a traffic light is detected, the traffic light detection unit 31 sets a rectangular detection frame that surrounds the part of the captured image where the traffic light is present. Then, the traffic light detection unit 31 detects the part of the traffic light that is lit (hereinafter referred to as the "lit part"). The lit part is detected by recognizing a certain area within the detection frame where the hue, saturation, and brightness are equal to or higher than a threshold value previously set for each lighting color such as red, blue, and yellow.

[0032] Additionally, the traffic light detection unit 31 determines the coordinates in the above-mentioned world coordinate system for the detected traffic lights and their lighting units. The area recognition unit 32 is configured to recognize a lane area, which is an area corresponding to the own lane, based on the captured image. Specifically, the area recognition unit 32 is configured to recognize a pair of demarcation lines that define the own lane in the captured image based on a model constructed by machine learning, and to obtain the space on the own lane in the world coordinate system as the own lane area. The method of obtaining the own lane area will be described later in detail.

[0033] The calculation unit 33 is configured to calculate the degree to which the front of the traffic signal detected by the traffic signal detection unit 31 faces the vehicle (hereinafter referred to as the "vehicle orientation degree") based on the captured image. A method for calculating the vehicle orientation degree will be described in detail later.

[0034] The traffic light identification unit 34 is configured to identify a traffic light for the host vehicle based on the detection result of the traffic light detection unit 31, the recognition result of the area recognition unit 32, and the calculation result of the calculation unit 33. A method for identifying a traffic light for the host vehicle will also be described in detail later.

[0035] [2-2. Processing] Next, the traffic light identification process executed by the image recognition device 102 will be described with reference to the flowchart of Fig. 6. The traffic light identification process is executed every time a captured image is input from the imaging device 11.

[0036] First, in S201, the image recognition device 102 performs the traffic signal detection process in the captured image as described above. For example, in the captured image shown in Fig. 7, two traffic signals 61 and 62 are detected, and detection frames 63 and 64 surrounding the traffic signals 61 and 62, respectively, are set.

[0037] Next, in S202, the image recognition device 102 determines whether or not a traffic signal has been detected in the process of S201. If the image recognition device 102 determines in S202 that a traffic signal has not been detected, the image recognition device 102 ends the traffic signal identification process in FIG.

[0038] On the other hand, if it is determined in S202 that a traffic signal has been detected, the image recognition device 102 proceeds to S203, where it determines whether or not multiple traffic signals have been detected. If the image recognition device 102 determines that multiple traffic signals have not been detected, i.e., that only one traffic signal has been detected, the image recognition device 102 proceeds to S204 and determines that the traffic signal is for the host vehicle.

[0039] On the other hand, if the image recognition device 102 determines in S203 that a plurality of traffic signals have been detected, the process proceeds to S205, where it determines whether or not the lighting colors of the plurality of traffic signals are different. If the image recognition device 102 determines in S205 that the lighting colors of the multiple traffic signals are the same, the process proceeds to S206, where the image recognition device 102 determines that the traffic signal closest to the vehicle is intended for the vehicle. When multiple traffic signals with the same lighting color are detected, it is assumed that multiple traffic signals that perform the same operation are installed on a road with many lanes on one side, and therefore the image recognition device 102 determines that these traffic signals are intended for the vehicle. However, when intersections are consecutively located at a short distance in the traveling direction of the vehicle, there is a possibility that a traffic signal at an intersection in front and a traffic signal at an intersection in the back are detected. Therefore, the image recognition device 102 determines that the traffic signal closest to the vehicle in the depth direction in the world coordinate system is intended for the vehicle.

[0040] On the other hand, if the image recognition device 102 determines in S205 that the illumination colors of the multiple traffic signals are different, the process proceeds to S207, where the image recognition device 102 performs a recognition process for the current vehicle lane area. Specifically, as shown in Fig. 7, the image recognition device 102 recognizes a pair of demarcation lines 65a, 65b that define the current vehicle lane in the world coordinate system, and obtains a pair of extension lines 66a, 66b that extend the pair of demarcation lines 65a, 65b in the depth direction. The image recognition device 102 then recognizes the space on the planar area that is sandwiched between these lines as the current vehicle lane area.

[0041] Next, in S208, the image recognition device 102 determines whether or not the own lane area has been recognized. For example, in the captured image shown in Fig. 9, a pair of demarcation lines defining the own lane are outside the angle of view and are not detected, so the image recognition device 102 determines that the own lane area has not been recognized.

[0042] When the image recognition device 102 determines in S208 that the own lane area has been recognized, the process proceeds to S209, where it determines whether the number of traffic signals in the own lane area is zero or more.

[0043] The determination of whether each traffic signal is within the own lane area is performed by the same method as the processing of S106 and S107 in the first embodiment. Specifically, the image recognition device 102 identifies, among the points constituting each of the pair of extension lines described above, points closest in the depth direction to the center of gravity of the detection frame of the traffic signal as a pair of corresponding points. Then, the image recognition device 103 determines whether the traffic signal is within the own lane area based on whether the center of gravity of the detection frame of the traffic signal is between the coordinates of the pair of corresponding points in the width direction of the own vehicle.

[0044] In this manner, by determining whether each traffic signal is within the current lane area, the number of traffic signals within the current lane area is determined. When the image recognition device 102 determines in S209 that the number of traffic lights in the own lane area is one, the process proceeds to S210 and determines that the traffic light is for the own vehicle. On the other hand, when the image recognition device 102 determines in S209 that the number of traffic lights in the own lane area is zero or more, the process proceeds to S211.

[0045] Returning to the explanation of S208, if the image recognition device 102 determines that it has not been able to recognize the own lane area, it skips S209 and proceeds to S211. In S211, the image recognition device 102 calculates the vehicle orientation degree for each traffic signal in the vehicle lane area. Specifically, as shown in FIG. 8, a binarization process is performed to separate the area within the detection frame of each traffic signal into a lighted portion and a portion other than the lighted portion, thereby extracting the lighted portion. Then, a covariance matrix is ​​obtained from the coordinates of the pixels constituting the lighted portion, and two eigenvalues ​​are calculated. Of the two eigenvalues, the larger one is the first eigenvalue, and the smaller one is the second eigenvalue. Then, the ratio of the second eigenvalue to the first eigenvalue is calculated as the vehicle orientation degree. This ratio represents the ratio of the length of the minor axis to the length of the major axis when the lighted portion is approximated to an ellipse. When the front of the traffic signal faces directly in front of the vehicle, the lighted portion becomes a perfect circle, and the above ratio is 1. Then, it is estimated that the closer the above ratio is to 1, the closer the shape of the lighted portion is to a perfect circle, and the higher the degree to which the front of the traffic signal faces the vehicle.

[0046] Next, in S212, the image recognition device 102 determines that the traffic signal having the largest vehicle orientation degree among the multiple traffic signals is directed toward the vehicle. For example, in the captured image shown in FIG. 7, the traffic signal 61 on the right side is directed toward the vehicle, and the traffic signal 62 on the left side is directed toward a vehicle on a road hidden behind a building 67 on the left side of the vehicle. In this scene, as shown in FIG. 8, the vehicle orientation degree of the traffic signal 61 on the right side is greater than the vehicle orientation degree of the traffic signal 62 on the left side. Therefore, the traffic signal 61 on the right side is correctly determined to be directed toward the vehicle. Also, for example, in the captured image shown in FIG. 9, the traffic signal 68 on the left side is directed toward a vehicle on a road intersecting the vehicle lane, and the traffic signal 69 on the left side is directed toward the vehicle. In this scene, as shown in FIG. 10, the vehicle orientation degree of the traffic signal 69 on the right side is greater than the vehicle orientation degree of the traffic signal 68 on the left side. Therefore, the traffic signal 69 on the right side is correctly determined to be directed toward the vehicle.

[0047] After that, the image recognition device 102 ends the traffic light identification process of FIG. [2-3. Effects] According to the second embodiment described above in detail, the following effects can be obtained.

[0048] (2a) The image recognition device 102 is configured to calculate the vehicle orientation of the traffic signal based on the captured image, and to identify the traffic signal that targets the vehicle based on the vehicle orientation. Specifically, the image recognition device 102 identifies the traffic signal that targets the vehicle among multiple traffic signals and has the largest vehicle orientation as the traffic signal that targets the vehicle.

[0049] Traffic signals are usually installed to face the vehicle that the traffic signal is intended to indicate. Therefore, from the view of the vehicle, a traffic signal intended for the vehicle appears to be facing directly ahead, while a traffic signal not intended for the vehicle, for example, a traffic signal on another road that intersects with the road on which the vehicle is traveling, appears to be tilted. Therefore, by using the vehicle orientation degree, which is the degree to which the front of the traffic signal faces the vehicle, as in the above configuration, the traffic signal intended for the vehicle can be appropriately identified.

[0050] (2b) The image recognition device 102 identifies traffic signals targeting the own vehicle based on the own vehicle direction degree as described above, on the condition that there are zero or multiple traffic signals in the own lane area. If there is only one traffic signal, not zero or multiple, in the own lane area, there is a high possibility that the traffic signal is directed toward the own vehicle. Therefore, in this case, the own vehicle direction degree is not calculated, thereby speeding up the traffic signal identification process.

[0051] (2c) The image recognition device 102 identifies traffic signals targeted at the host vehicle based on the host vehicle orientation degree as described above, on the condition that the lighting colors of the multiple traffic signals are different. When the lighting colors of multiple traffic signals are the same, it is assumed that multiple traffic signals that perform the same operation are installed on a road with many lanes on one side. Therefore, it is highly likely that the multiple detected traffic signals are directed toward the host vehicle. Therefore, in this case, the traffic signal identification process can be speeded up by not calculating the host vehicle orientation degree.

[0052] In the second embodiment, S201 corresponds to processing as the traffic light detection unit 31, S207 corresponds to processing as the area recognition unit 32, S211 corresponds to processing as the calculation unit 33, and S204, S210, and S212 correspond to processing as the traffic light identification unit 34.

[0053] [3. Third embodiment] [3-1. Configuration] The third embodiment has a basic configuration similar to that of the first embodiment, and therefore differences will be described below. Note that the same reference numerals as those in the first embodiment indicate the same configurations, and the preceding description will be referred to.

[0054] The image recognition device 103 shown in FIG. 11 is a device mounted on a vehicle, detects road signs based on captured images input from an imaging device 11 mounted on the vehicle, and determines whether or not the detected road signs target the vehicle itself.

[0055] The image recognition device 103 includes functional blocks realized by a CPU executing a program stored in a semiconductor memory, such as a sign detection unit 41, a lane recognition unit 42, a lane identification unit 43, a merging / branching identification unit 44, a three-dimensional object detection unit 45, and a judgment unit 46.

[0056] The sign detection unit 41 is configured to be able to detect road signs based on the captured image. Specifically, the sign detection unit 41 detects road signs based on a model constructed by machine learning, and calculates the coordinates of the detected road signs in the above-mentioned world coordinate system. In addition, when a road sign is detected, the sign detection unit 41 sets a rectangular detection frame that surrounds the portion of the captured image where the road sign is present.

[0057] The lane recognition unit 42 is configured to be able to recognize a plurality of lanes including at least the own lane based on the captured image. Specifically, the lane recognition unit 42 detects lines drawn on the road to which the own lane belongs based on a model constructed by machine learning, and recognizes a plurality of lanes separated by the lines. In addition, the lane recognition unit 42 obtains coordinates in the above-mentioned world coordinate system for the recognized plurality of lanes.

[0058] The lane identification unit 43 is configured to identify, based on the captured image, the lane that is closest to the road sign detected by the sign detection unit 41, among the multiple lanes recognized by the lane recognition unit 42. A method for identifying the lane that is closest to the road sign will be described in detail later.

[0059] The merging / branching identification unit 44 is configured to identify whether the lane identified by the lane identification unit 43 is a lane that merges with the own lane or a lane that branches off from the own lane. A method of identifying lanes will also be described in detail later.

[0060] The three-dimensional object detection unit 45 is configured to be able to detect three-dimensional objects that represent the road edges of the roads to which the multiple lanes recognized by the lane recognition unit 42 belong. Examples of the three-dimensional objects referred to here include curbs, blocks, fences, soundproof walls, guardrails, and median strips. The three-dimensional object detection unit 45 detects these three-dimensional objects based on a model constructed by machine learning, and determines the coordinates in the above-mentioned world coordinate system. The details of the method for detecting three-dimensional objects will also be described later.

[0061] The determination unit 46 is configured to determine whether or not the road sign detected by the sign detection unit 41 targets the vehicle. The determination method will also be described in detail later. [3-2. Processing] Next, the road sign determination process executed by the image recognition device 103 will be described with reference to the flowchart of Fig. 12. The road sign determination process is executed every time a captured image is input from the imaging device 11.

[0062] First, in S301, the image recognition device 103 performs the above-described road sign detection process in the captured image. For example, in the captured image shown in Fig. 13, a road sign 71 indicating a maximum speed limit is detected, and a detection frame 72 surrounding the sign 71 is set.

[0063] Next, in S302, the image recognition device 103 determines whether or not a road sign has been detected in the process of S301. If it is determined in S302 that no road sign has been detected, the image recognition device 103 ends the road sign determination process in FIG.

[0064] On the other hand, when it is determined in S302 that a road sign has been detected, the image recognition device 103 proceeds to S303, where it determines whether or not a distance x in the depth direction from the vehicle to the road sign is equal to or greater than a predetermined threshold, as shown in Fig. 14. The distance x is calculated based on the coordinate in the depth direction of the center of gravity of the detection frame 72 of the sign 71.

[0065] If the image recognition device 103 determines that the distance x is less than a predetermined threshold, the process proceeds to S304 and the determination is impossible. This is because if the road sign is too close to the vehicle, the recognition accuracy of the surrounding environment of the road sign is low, for example, the road lines are hidden outside the angle of view of the captured image and cannot be detected.

[0066] On the other hand, when the image recognition device 103 determines in S303 that the distance x is equal to or greater than the threshold, the process proceeds to S305, where the image recognition device 103 performs recognition processing of a plurality of lanes including the own lane as described above. For example, three lanes 73 to 75 are recognized in the captured image shown in FIG.

[0067] In addition, in S305, the image recognition device 103 detects a solid object representing the road edge as described above. Specifically, the image recognition device 103 detects, among the solid objects installed so as to extend substantially parallel to the own lane, the object closest to the own lane on each of the left and right sides of the own lane as the road edge. For example, in the captured image shown in FIG. 13, two sound insulation walls 76a and 76b installed at both ends of the road are detected as solid objects. Note that the sound insulation wall 77 installed at the boundary between the lane 73 and the lane 74, which can be seen in the back, is located far from the own vehicle and is not detected as a solid object at this stage. Also, for example, in the captured image shown in FIG. 15, a structure 84 installed at the boundary between the road to which the own lane 82 belongs and a side road 83 running parallel to the road, and a block 85 installed at the boundary with the oncoming lane are detected as solid objects.

[0068] Next, in S306, the image recognition device 103 determines whether or not a distance y between the road sign and the three-dimensional object in the width direction of the vehicle is equal to or smaller than a predetermined threshold value. For example, in the captured image shown in Fig. 15, the distance y is the distance from the structure 84 to the sign 81 in the direction away from the vehicle, as shown in Fig. 16.

[0069] In calculating the distance y, the image recognition device 103 first identifies, among the points constituting the three-dimensional object, corresponding points that are closest in the depth direction to the center of gravity of the detection frame of the road sign as a pair of corresponding points by using a method similar to the processing of S106 in the first embodiment. Then, the image recognition device 103 calculates the distance y based on the coordinate of the center of gravity of the detection frame of the road sign in the width direction of the host vehicle and the coordinate of the pair of corresponding points in the width direction of the host vehicle.

[0070] When the image recognition device 103 determines in S306 that the distance y is greater than the predetermined threshold, the process proceeds to S307, where the detected road sign is determined not to be directed to the vehicle. If the road sign is quite far from the solid object in the width direction of the vehicle, in the direction away from the vehicle, the road sign is highly likely to be installed for another road other than the road to which the vehicle's lane belongs, for example, a side road running parallel to the road to which the vehicle's lane belongs. For example, in the captured image shown in FIG. 17, as shown in FIG. 18, a sign 91 installed at the road edge of the road to which the vehicle's lane belongs is located at approximately the same position as a guardrail 92, which is a solid object installed at the boundary between the roadway and the sidewalk, in the width direction of the vehicle. In contrast, in the captured image shown in FIG. 15, as shown in FIG. 16, a sign 81 installed on a side road 83 is located at a position much farther away than a structure 84, which is a solid object installed at the boundary between the road to which the vehicle's lane 82 belongs and the side road 83. Therefore, when the distance y is greater than a predetermined threshold and the detected road sign is quite far away from the three-dimensional object in the width direction of the vehicle, away from the vehicle, the image recognition device 103 determines that the road sign is not intended for the vehicle.

[0071] On the other hand, when the image recognition device 103 determines in S306 that the distance y is equal to or less than the predetermined threshold, the image recognition device 103 proceeds to S308, and identifies the type of lane for each of the multiple lanes recognized in S305. Specifically, for each lane, it is recognized whether it is the own lane, a lane merging into the own lane or a lane branching off from the own lane, or another lane other than these (for example, an adjacent lane adjacent to the own lane, etc.). The own lane is recognized by recognizing a pair of defining lines that define the own lane. A lane merging into the own lane or a lane branching off from the own lane is identified based on, for example, the presence or absence of a zebra stripe 78 as shown in FIG. 13, the road shape, etc. Examples of the other lanes include lanes that run parallel to the own lane and exist continuously from the own lane, such as an adjacent lane adjacent to the adjacent lane, etc. For example, in the captured image shown in FIG. 13, lane 73 is identified as a lane branching off from the own lane, lane 74 is the own lane, and lane 75 is identified as an adjacent lane.

[0072] Next, in S309, the image recognition device 103 identifies the lane closest to the road sign among the multiple lanes. Specifically, first, the image recognition device 103 identifies the corresponding point closest in the depth direction to the center of gravity of the detection frame of the road sign among the points constituting the center line of each lane by using a method similar to the processing of S106 in the first embodiment. Then, the image recognition device 103 identifies the lane closest to the road sign by comparing the coordinates of each corresponding point in the width direction of the host vehicle with the coordinates of the center of gravity of the detection frame of the road sign in the width direction of the host vehicle. For example, in the captured image shown in FIG. 13, the lane 73 is identified as the lane closest to the sign 71 as shown in FIG. 14.

[0073] Next, in S310, the image recognition device 103 determines whether the distance z between the lane closest to the road sign in the width direction of the vehicle and the road sign in the direction away from the vehicle is equal to or less than a predetermined threshold. The distance z is calculated based on the coordinates of the center of gravity of the detection frame of the road sign in the width direction of the vehicle and the center line of the lane closest to the road sign, similar to the calculation of the distance y.

[0074] When the image recognition device 103 determines in S310 that the distance z is greater than a predetermined threshold, the image recognition device 103 proceeds to S304 and determines that the determination is impossible. For example, although a road sign is detected, a part or whole of the multiple lanes or the three-dimensional object may be hidden outside the angle of view of the captured image (for example, the lower side or the left side of the captured image), and therefore the multiple lanes and the three-dimensional object may not be detected properly. In such a case, it is not possible to determine whether the sign 95 is installed at the end of a road with many lanes on one side and is intended for the own vehicle, as shown in FIG. 19, or whether the sign 95 is installed on a side road 96 running parallel to the road to which the own lane belongs and is not intended for the own vehicle, as shown in FIG. 20. Therefore, when a road sign is located at a position significantly farther away than the lane closest to the road sign, the image recognition device 103 does not determine whether the road sign is intended for the own vehicle.

[0075] On the other hand, if the image recognition device 103 determines in S310 that the distance z is equal to or less than a predetermined threshold, it proceeds to S311 and determines whether the lane closest to the road sign is a lane that merges into the own lane or a lane that branches off from the own lane.

[0076] When the image recognition device 103 determines in S311 that the lane closest to the road sign is a lane merging into the own lane or a lane branching off from the own lane, the image recognition device 103 proceeds to S307 and determines that the detected road sign is not for the own vehicle. For example, in the captured image shown in Fig. 13, the lane 73 closest to the sign 71 is a lane branching off from the own lane, so that the sign 71 is correctly determined not to be for the own vehicle.

[0077] On the other hand, if the image recognition device 103 determines in S311 that the lane closest to the road sign is not a lane merging into the own lane or a lane branching off from the own lane, that is, is another lane such as the own lane or an adjacent lane, it proceeds to S312 and determines that the road sign is for the own vehicle.

[0078] After that, the image recognition device 103 ends the road sign determination process of FIG. [3-3. Effects] (3a) The image recognition device 103 identifies the lane closest to the detected road sign among the multiple lanes based on the captured image, and identifies whether the lane closest to the road sign is a lane merging into the own lane or a lane branching off from the own lane. If the image recognition device 103 identifies the lane closest to the road sign as a lane merging into the own lane or a lane branching off from the own lane, it determines that the road sign does not target the own vehicle.

[0079] With this configuration, information about road signs installed for lanes merging into or branching off from the own lane can be appropriately excluded from the information used to provide driving assistance to the own vehicle.

[0080] (3b) When a detected road sign is located more than a predetermined distance away from a three-dimensional object representing the edge of the road in the width direction of the vehicle, away from the vehicle, the image recognition device 103 determines that the detected road sign does not target the vehicle, regardless of whether the lane closest to the road sign is identified as a lane merging into the vehicle's lane or a lane branching off from the vehicle's lane.

[0081] With this configuration, information about road signs installed on roads other than the road on which the vehicle is traveling, such as side roads, can be appropriately excluded from the information used to provide driving assistance to the vehicle.

[0082] (3c) When a detected road sign is located more than a predetermined distance away from the lane closest to the road sign in the width direction of the vehicle, the image recognition device 103 does not determine whether the road sign targets the vehicle.

[0083] When a detected road sign is located at a position farther away than the lane closest to the road sign, there is a possibility that multiple lanes or three-dimensional objects around the road sign may not be properly detected. Therefore, with this configuration, it is possible to suppress erroneous determination of whether or not the detected road sign targets the vehicle.

[0084] In the third embodiment, S301 corresponds to the processing performed by the sign detection unit 41, S305 corresponds to the processing performed by the lane recognition unit 42 and the three-dimensional object detection unit 45, S309 corresponds to the processing performed by the lane identification unit 43, S311 corresponds to the processing performed by the merging / branching identification unit 44, and S304, S307, and S312 correspond to the processing performed by the determination unit 46.

[0085] [4. Other embodiments] Although the embodiments of the present disclosure have been described above, it goes without saying that the present disclosure is not limited to the above-described embodiments and can take various forms.

[0086] (4a) In the above first embodiment, the image recognition device 101 detects road markings that are drawn on the road surface and indicate information about a specific lane as a marking, but the markings detected by the image recognition device 101 are not limited to this. For example, the markings may be road signs that are placed around the road, specifically, on the road, and indicate information about a specific lane.

[0087] (4b) In the second embodiment, the vehicle orientation degree is calculated based on the length of the minor axis relative to the length of the major axis when the extracted lighting portion is approximated to an ellipse, but the calculation method of the vehicle orientation degree is not limited to this. For example, the vehicle orientation degree may be calculated based on the circularity calculated from the perimeter and area of ​​the lighting portion, the ratio of the area occupied by the lighting portion to the circumscribing circle of the lighting portion, etc. In addition, the vehicle orientation degree may not be calculated based on the closeness of the shape of the lighting portion in the traffic signal to the shape of the lighting portion in a state in which the front of the traffic signal is facing the vehicle, as in these cases, but may be calculated based on the ratio of the area occupied by the lighting portion to the detection frame of the traffic signal, etc. In addition, the vehicle orientation degree may be calculated by comparing the shape of the lighting portion or the shape characteristics of the detection frame of the traffic signal with a case database, or may be calculated based on a model constructed by machine learning.

[0088] (4c) In the second embodiment, the image recognition device 102 identifies the traffic signal having the largest vehicle orientation degree among multiple traffic signals as the traffic signal intended for the vehicle. However, the image recognition device 102 may simply determine that a traffic signal is intended for the vehicle when the vehicle orientation degree of the detected traffic signal is equal to or greater than a predetermined threshold value.

[0089] (4d) In the above first to third embodiments, the image recognition devices 101 to 103 output a judgment result for each single captured image input from the imaging device 11. However, it is also possible to subject the judgment results for multiple captured images taken at a predetermined time interval to a majority vote process and output a final judgment result.

[0090] (4e) The function of one component in the above embodiments may be distributed among multiple components, or the functions of multiple components may be integrated into one component. Also, part of the configuration of the above embodiments may be omitted. Also, at least part of the configuration of the above embodiments may be added to or substituted for the configuration of another of the above embodiments.

[0091] (4f) The method of realizing the functions of each unit included in the image recognition devices 101 to 103 is not limited to software, and some or all of the functions may be realized using one or more pieces of hardware. For example, when the functions are realized by an electronic circuit, which is hardware, the electronic circuit may be realized by a digital circuit, an analog circuit, or a combination of these.

[0092] (4g) The present disclosure can be realized in various forms, such as the above-mentioned image recognition devices 101 to 103, a system including the image recognition devices 101 to 103 as components, a program for causing a computer to function as the image recognition devices 101 to 103, a non-transitive substantial recording medium on which this program is recorded, and an image recognition method. [Explanation of symbols]

[0093] 11...imaging device, 21...sign detection unit, 22...demarcation line recognition unit, 23...corresponding point identification unit, 24...estimation unit, 31...traffic light detection unit, 32...area recognition unit, 33...calculation unit, 34...traffic light identification unit, 41...sign detection unit, 42...lane recognition unit, 43...lane identification unit, 44...merging / branching identification unit, 45...three-dimensional object detection unit, 46...determination unit, 101-103...image recognition device.

Claims

1. A sign detection unit (21) configured to detect a sign based on an image captured in front of the vehicle and set a detection frame surrounding the sign; a demarcation line recognition unit (22) configured to be able to recognize a pair of demarcation lines that define a lane based on the image; a corresponding point specifying unit (23) configured to specify a pair of corresponding points on the pair of demarcation lines corresponding to positions of specific points within the detection frame in a depth direction based on the host vehicle, based on the image; an estimation unit (24) configured to estimate, when the marking object is between the pair of corresponding points in the width direction of the lane, that the lane defined by the pair of defining lines to which the pair of corresponding points belongs is the lane targeted by the marking object; An image recognition device comprising:

2. the correspondence identification unit identifies, based on the image, a pair of corresponding points on the pair of demarcation lines corresponding to the vehicle's own lane, the pair of corresponding points corresponding to a position of a specific point within the detection frame in a depth direction based on the vehicle's own lane; 2. The image recognition device according to claim 1, wherein when the marking is between the pair of corresponding points in the width direction of the lane, the estimation unit estimates that the lane defined by the pair of defining lines to which the pair of corresponding points belongs is the lane targeted by the marking.

3. 3. The image recognition device according to claim 1, wherein the sign indicates information related to traffic and is placed on a road surface or around a road.

4. The image recognition device according to claim 1, wherein the corresponding point identification unit identifies, among the points constituting the pair of demarcation lines, a point that is closest to a specific point within the detection frame in the depth direction as the pair of corresponding points.

Images