In-vehicle devices and programs

Abstract

This invention provides an in-vehicle device and program that can accurately detect road markings. [Solution] An in-vehicle device according to one embodiment includes a controller. The controller detects a candidate road marking as a road marking when the confidence level of the candidate road marking included in the image recognition processing result for the image captured by the in-vehicle camera is above a threshold. The controller also changes the threshold according to the detection status of surrounding objects included in the image.

Description

Technical Field

[0001] The disclosed embodiments relate to in-vehicle devices and programs.

Background Art

[0002] Conventionally, in-vehicle devices that analyze images of in-vehicle cameras to detect crosswalks and stop lines are known. However, at crosswalks and stop lines, the white lines may be disappearing due to aging deterioration, and there is a risk that crosswalks and the like cannot be detected. In addition, since stop lines and the like have simple shapes, there is a risk of misdetecting simple-shaped objects that are not stop lines and the like as stop lines and the like.

[0003] Therefore, in order to detect a stop line that is disappearing, a technique has been proposed in which a bounding box that surrounds a region including a road marking is generated in an image obtained from an in-vehicle camera, and the position of the bottom side of the bounding box is detected as the stop line (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, for example, when a road marking is hidden by another vehicle that has stopped, a tree, or the like, in the prior art, a bounding box cannot be generated at an accurate position, and a stop line or the like cannot be detected. Thus, there is a risk that the prior art cannot accurately detect road surface markings including crosswalks and stop lines. -

[0006] One aspect of the embodiment has been made in view of the above, and an object thereof is to provide an in-vehicle device and a program that can accurately detect road surface markings. [Means for solving the problem]

[0007] To solve the above problems and achieve the objective, an in-vehicle device according to one embodiment includes a controller. The controller detects a candidate road marking as a road marking when the confidence level of the candidate road marking included in the image recognition processing result for the image captured by the in-vehicle camera is above a threshold. The controller also changes the threshold according to the detection status of surrounding objects included in the image. [Effects of the Invention]

[0008] In an in-vehicle device according to one embodiment, the controller detects a candidate road marking as a road marking when its confidence level is above a threshold. The controller also changes the threshold according to the detection status of surrounding objects included in the image. As a result, the in-vehicle device can change the threshold used in the process of detecting road markings to a value that is in line with the detection status of surrounding objects, and as a result can detect road markings with high accuracy. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a diagram illustrating the general method for detecting road markings performed by the in-vehicle device according to the embodiment. [Figure 2] Figure 2 is a block diagram showing an example configuration of an in-vehicle device according to an embodiment. [Figure 3] Figure 3 shows an example of threshold information for pedestrian crossings. [Figure 4] Figure 4 shows an example of threshold information for stop lines. [Figure 5] Figure 5 shows an example of the results information. [Figure 6] Figure 6 is a flowchart showing an example of a process performed by the controller of the in-vehicle device according to this embodiment. [Figure 7] Figure 7 is a diagram illustrating the processes performed by the controller. [Figure 8] Figure 8 is a flowchart showing an example of the processing performed by the controller of the in-vehicle device in the modified example. [Figure 9] Figure 9 is a diagram illustrating the process performed by the controller in the modified example. [Modes for carrying out the invention]

[0010] The embodiments of the in-vehicle device and program disclosed herein will be described in detail below with reference to the attached drawings. However, the present invention is not limited to the embodiments described below.

[0011] Furthermore, the expressions "designated," "specific," and "certain" in the following explanation may be interpreted as "predetermined."

[0012] (Embodiment) First, an overview of the road marking detection method performed by the in-vehicle device according to the embodiment will be described below with reference to Figure 1. Figure 1 is a diagram illustrating the overview of the road marking detection method performed by the in-vehicle device according to the embodiment.

[0013] The in-vehicle device 10 according to this embodiment (see Figure 2) is mounted on a vehicle (not shown). The in-vehicle device 10 performs image recognition processing on an image A captured by an in-vehicle camera (hereinafter sometimes referred to as "camera") and detects road markings B included in the image A.

[0014] The road marking B includes a crosswalk and a stop line provided on the road surface on which the vehicle travels. Note that the road marking B may include either the crosswalk or the stop line. That is, the road marking B may include at least either the crosswalk or the stop line. Note that the road marking B is not limited to the crosswalk or the like, and may include various markings provided on the road surface, such as lane dividing lines, speed markings indicating the legal speed, and turning prohibition signs indicating a prohibited turn. As the in-vehicle device 10, for example, a drive recorder can be used, but it is not limited to this, and any other type of device may be used as long as it is a device mounted on the vehicle and detects the road marking B.

[0015] As shown in FIG. 1, the camera captures an image A of the surroundings of the vehicle, including the front of the vehicle. Here, it is assumed that the image A includes a crosswalk B1 as the road marking B. In FIG. 1, an example is shown in which the white line of the crosswalk B1 is starting to disappear due to aging or the like.

[0016] The in-vehicle device 10 executes image recognition processing on the image (step S1). The in-vehicle device 10 extracts the road marking B (here, the crosswalk B1) shown in the image A as a road marking candidate Bx by the image recognition processing (step S2). Further, the in-vehicle device 10 detects the surrounding object C shown in the image A by the image recognition processing (step S3). In FIG. 1, an example is shown in which the surrounding object C is a road sign C1 indicating that there is a crosswalk.

[0017] Specifically explaining the processing of steps S1 to S3, the in-vehicle device 10 executes image recognition processing using an image recognition model 31 (see FIG. 2). The image recognition model 31 is an AI (Artificial Intelligence) model for image recognition. The AI model is, for example, a DNN (Deep Neural Network) model learned using a machine learning algorithm.

[0018] The image recognition model 31 operates as an image recognition AI by being loaded as an AI model into the controller 20 (see Fig. 2) of the in-vehicle device 10. The image recognition model 31 is pre-trained to extract such a road marking B as a road marking candidate Bx when a road marking B (accurately, something that may be the road marking B) is detected in the image A. The image recognition model 31 is pre-trained to output, as image recognition result information, the confidence level, class, position, size, etc. of the extracted road marking candidate Bx. Further, the image recognition model 31 is pre-trained to detect surrounding objects C (here, road signs C1) included in the image A. This surrounding object C is, for example, an object related to the road marking B or the road marking candidate Bx, and in other words, an object indicating that the road marking B or the like is provided in the vicinity. Note that the surrounding object C is not limited to the road sign C1 and includes pedestrians, traffic lights, etc., which will be described later.

[0019] The confidence level of the above-described road marking candidate Bx is an index indicating the certainty and accuracy of the road marking candidate Bx, and is also referred to as the reliability. The confidence level increases numerically as the possibility that the road marking candidate Bx is the road marking B increases. Therefore, as shown in Fig. 1, the confidence level of the road marking candidate Bx including the road marking B (pedestrian crossing B1) where the white line is starting to disappear is lower than the confidence level of the road marking candidate Bx including the road marking B that has not started to disappear.

[0020] As will be described later, when the confidence level of the road marking candidate Bx is equal to or higher than a threshold value, the in-vehicle device 10 detects the road marking candidate Bx as the road marking B, or in other words, determines the road marking candidate Bx as the road marking B. Here, assume that the confidence level of the road marking candidate Bx including the road marking B that is starting to disappear becomes lower than a preset initial threshold value (initial value), and in this state, the road marking candidate Bx cannot be detected as the road marking B.

[0021] Therefore, the in-vehicle device 10 according to this embodiment changes the threshold according to the detection status of surrounding objects C included in image A (step S4). Specifically, the detection status of surrounding objects C is the detection of a road sign C1 indicating the presence of a pedestrian crossing. In other words, the detection status of surrounding objects C indicates that there is a pedestrian crossing around the vehicle. Therefore, the in-vehicle device 10 changes the threshold to be lowered according to the detection status of surrounding objects C (in this case, road sign C1), in other words, changes the threshold to a smaller value.

[0022] Next, the in-vehicle device 10 compares the confidence level of the road marking candidate Bx with the modified threshold, and detects the road marking candidate Bx as road marking B if the confidence level of the road marking candidate Bx is equal to or greater than the threshold (step S5). Specifically, in the processing of step S4, the threshold is changed to be lowered, so the confidence level of the road marking candidate Bx becomes equal to or greater than the threshold, and the in-vehicle device 10 detects the road marking candidate Bx as road marking B. In other words, by changing the threshold, the in-vehicle device 10 can detect road marking candidate Bx with a reduced confidence level, including fading road marking B, as road marking B.

[0023] Thus, the in-vehicle device 10 according to this embodiment can change the threshold value used in the process of detecting the road marking B to a value that is appropriate for the detection status of the surrounding objects C (a value suitable for the detection status of the surrounding objects C), and as a result can detect the road marking B with high accuracy.

[0024] Next, the in-vehicle device 10 according to this embodiment will be described in detail with reference to Figure 2. Figure 2 is a block diagram showing an example configuration of the in-vehicle device 10 according to this embodiment. Note that the block diagram in Figure 2 shows only the components necessary to explain the features of the embodiment, and descriptions of general components are omitted.

[0025] As shown in Figure 2, the in-vehicle device 10 includes a camera 11, an output unit 12, a controller (control unit) 20, and a storage unit 30.

[0026] Camera 11 is installed at an appropriate position on the vehicle and captures images of the area around the vehicle. Specifically, camera 11 captures images of the area around the vehicle, including the area in front of the vehicle. Camera 11 outputs the captured images (image data) to the controller 20. The image data may be moving images or still images. Camera 11 is not limited to capturing images of the area around the vehicle, but may also capture images of the interior of the vehicle. Furthermore, camera 11 is, for example, a camera equipped with a lens and an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), but is not limited to these.

[0027] The output unit 12 outputs various types of information, such as notifications to alert drivers and warnings related to driving. For example, the output unit 12 includes at least one of a display unit such as a screen and an audio output unit such as a speaker, and outputs various types of information to the vehicle driver.

[0028] In the example shown in Figure 2, the in-vehicle device 10 is configured to include a camera 11 and an output unit 12, but it is not limited to this configuration. That is, one or both of the camera 11 and the output unit 12 may be mounted on the vehicle separately from the in-vehicle device 10, and one or both of the camera 11 and the output unit 12 may be connected to the in-vehicle device 10 in a communication manner.

[0029] The memory unit 30 is implemented by a memory device such as ROM (Read Only Memory), RAM (Random Access Memory), or flash memory. The memory unit 30 stores the image recognition model 31, threshold information 32, result information 33, various data, and various programs.

[0030] The image recognition model 31 has been described above, so its explanation is omitted here. Threshold information 32 is information about thresholds used in the process of detecting road markings. Threshold information 32 includes threshold information for each type of road marking. Specifically, threshold information 32 includes threshold information 32a for pedestrian crossings and threshold information 32b for stop lines. In addition to the threshold information 32a for pedestrian crossings and threshold information 32b for stop lines, threshold information 32 may also include threshold information corresponding to other types of road markings.

[0031] Here, the pedestrian crossing threshold information 32a will be explained with reference to Figure 3. Figure 3 is a diagram showing an example of the pedestrian crossing threshold information 32a. The pedestrian crossing threshold information 32a is information about thresholds used in the process of detecting pedestrian crossings.

[0032] As shown in Figure 3, the pedestrian crossing threshold information 32a has items such as "detection status" and "threshold for pedestrian crossing." The "detection status" has items such as "pedestrian," "road sign," and "stop line." The data for each item is related to (linked) to each other.

[0033] "Detection Status" indicates the detection status of surrounding objects included in the image. "Pedestrians" indicates whether or not pedestrians were detected as surrounding objects. "Road Signs" indicates whether or not road signs indicating the presence of a crosswalk were detected as surrounding objects. "Stop Lines" indicates whether or not stop lines were detected as surrounding objects. In Figure 3 and Figure 4 (described later), a circle (〇) indicates that the corresponding surrounding object was detected, and a horizontal line (-) indicates that it was not detected.

[0034] The "crosswalk threshold" is information about the threshold value for crosswalks, which is set according to the detection status of surrounding objects. In other words, the "crosswalk threshold" is information that indicates how the crosswalk threshold should be changed according to the detection status of surrounding objects. Here, the initial value (normal value) of the crosswalk threshold is assumed to be 0.25.

[0035] In the example shown in Figure 3, the pedestrian crossing threshold information 32a indicates that the "pedestrian crossing threshold" should be "raised (0.35)" when none of the surrounding objects, such as "pedestrians," "road signs," and "stop lines," are detected. In other words, since none of the objects associated with pedestrian crossings—pedestrians, road signs, and stop lines—are detected in the surroundings, there is a relatively high probability that no pedestrian crossing is provided nearby, and therefore the pedestrian crossing threshold is changed to be raised. As a result, even if, for example, the image recognition process extracts an object with a simple shape similar to a pedestrian crossing as a candidate for a road marking, the confidence level of that road marking candidate is less likely to exceed the changed threshold. This prevents the in-vehicle device 10 from mistakenly detecting an object extracted as a road marking candidate, i.e., an object with a simple shape similar to a pedestrian crossing, as a pedestrian crossing.

[0036] Furthermore, in the example shown in Figure 3, the pedestrian crossing threshold information 32a indicates that the "threshold for pedestrian crossings" is "normal (0.25)" when only "pedestrians" are detected as surrounding objects. In other words, although pedestrians are objects related to pedestrian crossings, they often simply walk on the sidewalk and have a relatively low correlation with pedestrian crossings, so the threshold is not changed. For example, in image recognition processing, when a road marking candidate is extracted, the confidence level of the extracted road marking candidate is compared with the unchanged threshold (initial value, 0.25). The in-vehicle device 10 then detects the road marking candidate as a pedestrian crossing if the confidence level of the road marking candidate is equal to or greater than the threshold.

[0037] Furthermore, in the example shown in Figure 3, the pedestrian crossing threshold information 32a indicates that if either a "road sign" or a "stop line" is detected as an object in the surrounding area, the "pedestrian crossing threshold" is "lowered (0.15)". In other words, since either a road sign or a stop line, which are objects related to pedestrian crossings, is detected in the surrounding area, there is a relatively high probability that a pedestrian crossing is located nearby, so the pedestrian crossing threshold is changed to be lowered. Therefore, for example, even if a pedestrian crossing with a fading white line is extracted as a road marking candidate in the image recognition process, it becomes possible to detect that road marking candidate as a pedestrian crossing. That is, the confidence level of road marking candidates that include a fading pedestrian crossing will be lower, but even in such cases, since the threshold is changed to be lowered, the confidence level of the road marking candidate is more likely to be above the threshold. As a result, the in-vehicle device 10 can detect road marking candidates that include a fading pedestrian crossing and have a reduced confidence level as pedestrian crossings.

[0038] Furthermore, in the example shown in Figure 3, the pedestrian crossing threshold information 32a indicates that the "pedestrian crossing threshold" should be "lowered (0.10)" when two of the surrounding objects, "pedestrians," "road signs," and "stop lines," are detected. The pedestrian crossing threshold information 32a indicates that the "pedestrian crossing threshold" should be "lowered (0.05)" when all of the surrounding objects, "pedestrians," "road signs," and "stop lines," are detected. In this way, the pedestrian crossing threshold information 32a is set so that the threshold for pedestrian crossings becomes smaller as the number of detected objects related to pedestrian crossings increases. As a result, even when the in-vehicle device 10 extracts a road marking candidate that includes a fading pedestrian crossing and has a reduced confidence level, the confidence level of the road marking candidate is more likely to be above the threshold, and therefore the device can detect the road marking candidate as a pedestrian crossing.

[0039] The above provides specific numerical values ​​for the revised threshold for pedestrian crossings, but these are merely examples and not limiting. Furthermore, the revised threshold for pedestrian crossings may be set differently depending on the type of surrounding object detected.

[0040] In the above example, the "detection status" of the pedestrian crossing threshold information 32a is shown as consisting of three items: "pedestrian," "road sign," and "stop line." However, it is not limited to this, and may include at least one of "pedestrian," "road sign," and "stop line." Furthermore, the pedestrian crossing threshold information 32a shown in Figure 3 is threshold information intended for detecting pedestrian crossings located in places without traffic lights. Therefore, the "detection status" of the pedestrian crossing threshold information 32a does not include "traffic light," but it is not limited to this, and may include the "traffic light" item. In other words, the "detection status" of the pedestrian crossing threshold information 32a only needs to include at least one of "pedestrian," "road sign," "traffic light," and "stop line."

[0041] Next, the stop line threshold information 32b will be explained with reference to Figure 4. Figure 4 is a diagram showing an example of the stop line threshold information 32b. The stop line threshold information 32b is information about the threshold used in the process of detecting the stop line.

[0042] As shown in Figure 4, the stop line threshold information 32b includes items such as "detection status" and "stop line threshold." The "detection status" includes items such as "pedestrian," "road sign," "traffic light," and "crosswalk." The data for each item are related to (linked) each other.

[0043] The "Detection Status" and "Pedestrians" are the same as in Figure 3, so their explanation is omitted here. "Road Signs" is information indicating whether or not a road sign indicating the presence of a stop line was detected as an object in the surroundings. "Traffic Lights" is information indicating whether or not a traffic light was detected as an object in the surroundings. "Pedestrian Crossings" is information indicating whether or not a pedestrian crossing was detected as an object in the surroundings.

[0044] The "stop line threshold" is information about the threshold value for the stop line, which is set according to the detection status of surrounding objects. In other words, the "stop line threshold" is information that indicates how the stop line threshold should be changed according to the detection status of surrounding objects. Here, the initial value (normal value) of the stop line threshold is assumed to be 0.25.

[0045] In the example shown in Figure 4, the stop line threshold information 32b indicates that the "stop line threshold" should be "raised (0.35)" when none of the surrounding objects, such as "pedestrians," "road signs," "traffic lights," and "crosswalks," are detected. In other words, since none of the objects related to stop lines—pedestrians, road signs, traffic lights, and crosswalks—are detected in the surroundings, there is a relatively high probability that no stop line is provided nearby, and therefore the stop line threshold is changed to be raised. As a result, even if, for example, the image recognition process extracts an object with a simple shape similar to a stop line as a candidate for a road marking, the confidence level of that road marking candidate is less likely to exceed the changed threshold. This prevents the in-vehicle device 10 from mistakenly detecting an object extracted as a road marking candidate, i.e., an object with a simple shape similar to a stop line, as a stop line.

[0046] Furthermore, in the example shown in Figure 4, the stop line threshold information 32b indicates that the "stop line threshold" is "normal (0.25)" when the detection situation is such that only "pedestrians" are detected as surrounding objects. In other words, although pedestrians are objects related to stop lines, they are often simply walking on the sidewalk and have a relatively low correlation with stop lines, so the threshold is not changed. For example, in image recognition processing, when a road marking candidate is extracted, the confidence level of the extracted road marking candidate is compared with the unchanged threshold (initial value, 0.25). Then, the in-vehicle device 10 detects the road marking candidate as a stop line if the confidence level of the road marking candidate is equal to or greater than the threshold.

[0047] Furthermore, in the example shown in Figure 4, the stop line threshold information 32b indicates that if one of the surrounding objects—"road sign," "traffic light," and "pedestrian crossing"—is detected, the "stop line threshold" is "lowered (0.15)." In other words, because one of the objects related to stop lines—road sign, traffic light, and pedestrian crossing—is detected in the surroundings, there is a relatively high probability that a stop line is present in the vicinity, and therefore the stop line threshold is changed to be lowered. As a result, even if, for example, a stop line with a fading white line is extracted as a road marking candidate in image recognition processing, it becomes possible to detect that road marking candidate as a stop line. That is, the confidence level of road marking candidates that include a fading stop line will be lower, but even in such cases, because the threshold is changed to be lowered, the confidence level of the road marking candidate is more likely to be above the threshold. As a result, the in-vehicle device 10 can detect road marking candidates that include a fading stop line and have a reduced confidence level as stop lines.

[0048] Furthermore, in the example in Figure 4, the stop line threshold information 32b indicates that if two of the surrounding objects—"pedestrians," "road signs," "traffic lights," and "crosswalks"—are detected, the "stop line threshold" should be "lowered (0.10)." The stop line threshold information 32b indicates that if three of the surrounding objects—"pedestrians," "road signs," "traffic lights," and "crosswalks"—are detected, the "stop line threshold" should be "lowered (0.05)." The stop line threshold information 32b indicates that if all of the surrounding objects—"pedestrians," "road signs," "traffic lights," and "crosswalks"—are detected, the "stop line threshold" should be "lowered (0.03)." Thus, the stop line threshold information 32b is set so that the stop line threshold becomes smaller as the number of objects related to the detected stop line increases. As a result, even if the in-vehicle device 10 extracts a road marking candidate that includes a fading stop line and has a reduced confidence level, the confidence level of the road marking candidate becomes more likely to exceed the threshold, allowing the device to detect the road marking candidate as a stop line.

[0049] The above provides specific numerical values ​​for the threshold values ​​of the modified stop lines, but these are merely examples and not limiting. Furthermore, the threshold values ​​for the modified stop lines may be set differently depending on the type of surrounding object detected.

[0050] In the above example, the "detection status" of the stop line threshold information 32b is shown as four items: "pedestrian," "road sign," "traffic light," and "pedestrian crossing," but it is not limited to this. In other words, the "detection status" of the stop line threshold information 32b only needs to include at least one of "pedestrian," "road sign," "traffic light," and "pedestrian crossing."

[0051] Returning to the explanation of Figure 2, the result information 33 is information indicating the result of the image recognition processing on the image captured by the camera 11. Here, the result information 33 will be explained with reference to Figure 5. Figure 5 is a diagram showing an example of the result information 33.

[0052] As shown in Figure 5, the result information 33 includes items such as "Result ID," "Class ID," "Road Marking Candidate," "Confidence Level," "Coordinates," and "Size." The data for each item are related to (linked) each other.

[0053] The "Result ID" is information that identifies the result of the image recognition processing. The "Class ID" is information that identifies the class (type) of the extracted road marking candidate. The "Road Marking Candidate" is information that indicates the name of the class of the extracted road marking candidate. "Road Marking Candidate" includes "pedestrian crossing," "stop line," etc.

[0054] "Confidence level" is information indicating the confidence level of the extracted road marking candidates. In the example shown in Figure 5, for convenience, "confidence level" is abstractly represented as "F01," but "F01" will store specific information. Other information may also be described abstractly below.

[0055] "Coordinates" refers to information indicating the coordinates (position) of the extracted road marking candidate image. "Size" refers to information indicating the size (dimensions) of the extracted road marking candidate image.

[0056] In the example shown in Figure 5, the result information identified by result ID "D01" indicates that the class ID is "E01", the road marking candidate is "pedestrian crossing", the confidence level is "F01", the coordinates are "G01", and the size is "H01". Similarly, the result information identified by result ID "D11" indicates that the class ID is "E02", the road marking candidate is "stop line", the confidence level is "F11", the coordinates are "G11", and the size is "H11".

[0057] Returning to the explanation of Figure 2, the controller 20 is composed of an information processing device including a computer. Specifically, the controller 20 corresponds to a so-called processor. The controller 20 is implemented by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), etc. The controller 20 executes a program according to an embodiment not shown in the figure, which is stored in the memory unit 30, using RAM as the working area. Note that the controller 20 may be composed of hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), either partially or entirely.

[0058] The controller 20 performs various processes, such as detecting road markings included in the image captured by the camera 11, and notifying the driver of driving warnings in accordance with the detected road markings.

[0059] Here, the processes performed by the controller 20 will be explained in detail with reference to Figures 6 and 7. Figure 6 is a flowchart showing an example of the processes performed by the controller 20 of the in-vehicle device 10 according to this embodiment. Figure 7 is a diagram illustrating the processes performed by the controller 20. Note that the processes shown in Figure 6 are repeatedly performed at predetermined intervals, but are not limited to this.

[0060] As shown in Figure 6, the controller 20 acquires image A1 (see Figure 7) captured by the camera 11 (step S100). Next, the controller 20 performs image recognition processing on image A1 (step S101). Specifically, the controller 20 performs image recognition processing on image A1 using the image recognition model 31 and determines whether or not road marking B (or something that could be road marking B) is detected in image A1. If the controller 20 determines that road marking B (pedestrian crossing B1 in the example of Figure 7) is detected in image A1, it extracts such road marking B as road marking candidate Bx. The controller 20 also calculates the confidence level, class, position, and size of the extracted road marking candidate Bx and stores the various information of the obtained road marking candidate Bx in the storage unit 30 as result information 33.

[0061] Next, the controller 20 determines whether or not there are road marking candidate Bx in image A1 as a result of the image recognition process (step S102). If the controller 20 determines that there are no road marking candidate Bx in image A1 (step S102, No), it skips the subsequent processing.

[0062] On the other hand, if the controller 20 determines that there is a candidate road marking Bx in image A1 (step S102, Yes), it detects surrounding objects C included in image A1 by performing image recognition processing on image A1 (step S103). Figure 7 shows an example in which a road sign C1 indicating the presence of a pedestrian crossing, a pedestrian C2, and a stop line C3 are detected as surrounding objects C.

[0063] Next, the controller 20 changes the threshold according to the detection status of surrounding objects C (step S104). Specifically, the controller 20 changes the threshold corresponding to the type of road marking candidate Bx according to the detection status of surrounding objects C. For example, since the road marking candidate Bx of a pedestrian crossing B1 has been extracted here, the controller 20 reads the pedestrian crossing threshold information 32a (see Figures 2 and 3) from the storage unit 30. Then, since road sign C1, pedestrian C2, and stop line C3 have been detected as surrounding objects C, the controller 20 changes the threshold for pedestrian crossings to be lowered, or more precisely, changes the threshold for pedestrian crossings to be lowered to 0.05.

[0064] Here, we will explain the region in image A1 where the threshold is changed. As shown in Figure 7, the controller 20 according to this embodiment changes the threshold in the detection region K1, which is a part of the entire region J of image A1. Conversely, the controller 20 does not change the threshold in the region of the entire region J of image A1 other than the detection region K1.

[0065] The detection area K1 is an arbitrary area pre-set within the entire area J of image A1. That is, the position and size of the detection area K1 are set to arbitrary values ​​in advance. Specifically, the detection area K1 is set in the area of ​​image A1 in which the road marking B is captured. For example, the detection area K1 may be set in the lower half of the entire area J of image A1. Alternatively, the detection area K1 may be set in the area several meters or tens of meters in front of the vehicle. Furthermore, the detection area K1 may be set in the area of ​​image A1 that includes at least the candidate road marking Bx.

[0066] Thus, since the controller 20 is configured to change the threshold in the detection region K1, which is part of image A1, it is possible to suppress the interference with the detection process of road marking candidates extracted in areas other than the detection region K1. Furthermore, since the controller 20 is configured to change the threshold in the detection region K1, which is part of image A1, the processing load can be reduced compared to changing the threshold in the entire region J of image A1.

[0067] Next, the controller 20 determines whether the confidence level of the road marking candidate Bx is above a threshold (step S105). If the controller 20 determines that the confidence level of the road marking candidate Bx is not above a threshold (step S105, No), that is, if it determines that the confidence level of the road marking candidate Bx is below a threshold, it terminates the process. Specifically, the controller 20 terminates the process without detecting the road marking candidate Bx as a road marking.

[0068] On the other hand, if the controller 20 determines that the confidence level of the road marking candidate Bx is above a threshold (step S105, Yes), it detects the road marking candidate Bx as road marking B (step S106).

[0069] Next, the controller 20 determines whether the notification conditions for notifying the driver about the detected road marking B have been met (step S107). If the controller 20 determines that the notification conditions have been met (step S107, Yes), it executes the notification process to notify the driver (step S108). If the controller 20 determines that the notification conditions have not been met (step S107, No), it terminates the process.

[0070] The processing in steps S107 and S108 will be explained in detail. For example, if a notification condition is set that the detected road marking B is a pedestrian crossing and a pedestrian attempting to cross the pedestrian crossing has been detected, and this notification condition is met, the controller 20 will notify the driver via the output unit 12 of a warning urging the vehicle to stop. Also, if a notification condition is set that the vehicle was not stopped despite the warning urging the vehicle to stop, and this notification condition is met, the controller 20 will notify the driver via the output unit 12 of a warning that this is a violation of traffic rules. Note that the notification conditions and notification contents described above are merely examples and are not limiting.

[0071] Furthermore, if the controller 20 changes the threshold in step S104, it may perform a process to revert the threshold to its original value at any time. For example, after detecting road marking candidate Bx as road marking B, the controller 20 may revert the threshold used for such detection to its original value. Alternatively, the controller 20 may revert the changed threshold to its original value after a predetermined time has elapsed since the threshold was changed.

[0072] As described above, the in-vehicle device 10 according to the embodiment includes a controller 20. The controller 20 detects a road marking candidate as a road marking when the confidence level of the road marking candidate included in the image recognition processing result for the image captured by the camera (an example of an in-vehicle camera) 11 is above a threshold. The controller 20 also changes the threshold according to the detection status of surrounding objects included in the image.

[0073] Thus, in the in-vehicle device 10, the controller 20 detects a candidate road marking as a road marking when its confidence level is above a threshold. Furthermore, the controller 20 changes the threshold according to the detection status of surrounding objects included in the image. As a result, the in-vehicle device 10 can change the threshold used in the road marking detection process to a value that is in line with the detection status of surrounding objects, and as a result can detect road markings with high accuracy.

[0074] Furthermore, by accurately detecting road markings, the in-vehicle device 10 can accurately provide notifications such as driving warnings in accordance with the road markings. For example, if the threshold is not changed and a simple object similar in shape to a pedestrian crossing is mistakenly detected as a pedestrian crossing, the in-vehicle device may make a false notification, such as a warning prompting the vehicle to stop at the mistakenly detected pedestrian crossing. In contrast, the in-vehicle device 10 according to this embodiment changes the threshold according to the detection status of surrounding objects, making it less likely to mistakenly detect a simple object similar in shape to a pedestrian crossing as a pedestrian crossing, and thus suppressing the occurrence of the above-mentioned false notifications.

[0075] Furthermore, for example, if the threshold is not changed, the in-vehicle device cannot detect road marking candidates, including fading pedestrian crossings, as pedestrian crossings. Therefore, the in-vehicle device cannot issue a warning to encourage the vehicle to stop, even though a pedestrian crossing is present. In contrast, the in-vehicle device 10 according to this embodiment changes the threshold according to the detection status of surrounding objects, so it can detect road marking candidates, including fading pedestrian crossings, as pedestrian crossings. Therefore, the in-vehicle device 10 according to this embodiment can provide accurate notifications, such as issuing a warning to encourage the vehicle to stop for detected pedestrian crossings.

[0076] Furthermore, the controller 20 detects at least one of a pedestrian crossing and a stop line as road markings on the road surface where the vehicle is traveling. This allows the controller 20 to accurately detect at least one of a pedestrian crossing and a stop line.

[0077] Furthermore, when the controller 20 detects a pedestrian crossing provided on the road surface where a vehicle is traveling as a road marking, it changes the threshold according to the detection status of at least one of the surrounding objects detected: pedestrians, road signs, traffic lights, and stop lines. In this way, the controller 20 can change the threshold for pedestrian crossings to a value that is more in line with the detection status of surrounding objects by using the detection status of pedestrians, road signs, traffic lights, and stop lines, which are objects related to pedestrian crossings.

[0078] Furthermore, when the controller 20 detects a stop line provided on the road surface on which a vehicle is traveling as a road marking, it changes the threshold according to the detection status of at least one of the surrounding objects detected: pedestrians, road signs, traffic lights, and crosswalks. In this way, the controller 20 can change the threshold of the stop line to a value that is more in line with the detection status of surrounding objects by using the detection status of pedestrians, road signs, traffic lights, and crosswalks, which are objects related to the stop line.

[0079] (modified version) Next, a modified in-vehicle device 10 will be described. In the embodiment described above, an example was shown in which the position and size of the detection area K1 are set to arbitrary values ​​in advance, but it is not limited to this and may be set to variable values. Specifically, in the modified example, at least one of the position and size of the detection area is set according to the detection status of surrounding objects.

[0080] Here, with reference to Figures 8 and 9, the processes executed by the controller 20 of the modified vehicle-mounted device 10 will be described. Figure 8 is a flowchart showing an example of the processes executed by the controller 20 of the modified vehicle-mounted device 10. Figure 9 is a diagram illustrating the processes executed by the controller 20 of the modified vehicle-mounted device. Note that in Figure 8, processes identical to those in the embodiment may be given the same step numbers and their explanations may be omitted.

[0081] As shown in Figures 8 and 9, in the modified example, a road marking B (stop line B2 in the example of Figure 9) is detected in image A2. If the controller 20 in the modified example determines that there is a candidate road marking Bx in image A2 (step S102, Yes), it proceeds to step S103 and detects surrounding objects C included in image A2. Figure 9 shows an example in which a road sign C4 indicating the presence of a stop line and lane markings C5 that demarcate lanes are detected as surrounding objects C.

[0082] Next, the controller 20 sets the position and size of the detection area K2 according to the detection status of surrounding objects C (step S103a). For example, if the controller 20 detects a lane marking C5 as a surrounding object C, it sets the area defined by the lane marking C5 as the detection area K2. Specifically, the detection area K2 includes a candidate road marking Bx. Here, the candidate road marking Bx is a stop line B2, and a stop line B2 is often placed between two lane markings C5. In other words, lane marking C5 is an object related to stop line B2. Therefore, the controller 20 sets the area between the two lane markings C5, which are objects related to stop line B2 (in other words, the area defined by lane marking C5) as the detection area K2.

[0083] Thus, in this modified version, the position and size of the detection area K2 are set according to the detection status of surrounding objects C related to the road marking B or candidate road marking Bx, so that the detection area K2 can be set to a position and size that is appropriate for the road marking B, etc.

[0084] In the modified example, the controller 20 executes the processes from step S104 onward after the processing in step S103a.

[0085] In the modified example described above, both the position and size of the detection area K2 are set according to the detection status of surrounding objects. However, the system is not limited to this, and either the position or size of the detection area K2 may be set. In other words, it is sufficient for at least one of the position or size of the detection area K2 to be set according to the detection status of surrounding objects.

[0086] In the above description, the in-vehicle device 10 performs image recognition processing to detect road markings, but the system is not limited to this configuration. In other words, some or all of the image recognition processing performed by the in-vehicle device 10 may be performed by a central device (server device) that is communicatively connected to the in-vehicle device 10. Specifically, the controller of the central device may perform image recognition processing on images acquired from the in-vehicle camera, and if the confidence level of a candidate road marking included in the processing result is above a threshold, it may be configured to detect the candidate road marking as a road marking. Alternatively, the controller of the central device may be configured to change the threshold depending on the detection status of surrounding objects included in the image.

[0087] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of symbols]

[0088] 10 Onboard equipment 11 Cameras 20 controllers

Claims

1. The system includes a controller that detects a candidate road marking as a road marking when the confidence level of the candidate road marking included in the image recognition processing results for an image captured by an in-vehicle camera is above a threshold. The aforementioned controller, The threshold is changed according to the detection status of surrounding objects included in the aforementioned image. In-vehicle device.

2. The aforementioned controller, The system detects at least one of a pedestrian crossing and a stop line, which are provided on the road surface where a vehicle travels, as road markings. The in-vehicle device according to claim 1.

3. The aforementioned controller, When detecting a pedestrian crossing provided on the road surface where a vehicle is traveling as a road marking, the threshold is changed according to the detection status of at least one of the surrounding objects detected: pedestrians, road signs, traffic lights, and stop lines. The in-vehicle device according to claim 1.

4. The aforementioned controller, When detecting a stop line provided on the road surface on which a vehicle is traveling as a road marking, the threshold is changed according to the detection status of at least one of the surrounding objects detected, such as a pedestrian, road sign, traffic light, and crosswalk. The in-vehicle device according to claim 1.

5. The aforementioned controller, The threshold is changed in the detection region, which is a part of the entire region of the aforementioned image. The in-vehicle device according to claim 1.

6. The aforementioned controller, Depending on the detection status of the surrounding objects, at least one of the position and size of the detection area is set. The in-vehicle device according to claim 5.

7. If the confidence level of a candidate road marking included in the image recognition processing results for an image captured by an in-vehicle camera is above a threshold, the candidate road marking is detected as a road marking. The threshold is changed according to the detection status of surrounding objects included in the aforementioned image, A program that causes a computer to execute something.

Images