Recognition device, recognition method, and recognition program

Abstract

Provided is a recognition device for detecting the boundary of a roadway by using time-series images captured by a camera mounted in a vehicle, the recognition device comprising an acquisition unit that acquires the time-series images, and a detection unit that classifies regions of the acquired time-series images according to the magnitude of variation in the regions into a roadway part that is a region for which the variation is small and a non-roadway part that is a region for which the variation is large, and detects a boundary between the region for which the variation is small and the region for which the variation is large as the boundary of the roadway.

Description

[Technical field]

[0001] The disclosed technology relates to a recognition device, a recognition method, and a recognition program. [Background technology]

[0002] In order to realize functions related to automated driving or driver assistance, it is necessary to provide a function that properly recognizes the vehicle's surrounding environment and controls the vehicle based on the recognition results, or notifies or suggests things to the driver. Such functions may use images captured by cameras mounted on the vehicle to recognize the vehicle's surrounding environment.

[0003] When an image is used, it is possible to detect a particular pattern, for example, by binarizing a simplified image (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-96151 A Summary of the Invention [Problem to be solved by the invention]

[0005] The surrounding environment that an automobile must recognize includes boundaries between roadways or sidewalks, and there are various patterns for the boundaries, such as straight lines, dotted lines, and combinations of these. However, there are also cases where it is not possible to patternize them. For example, there are various lane markings whose colors or shapes cannot be patterned, cases where lane markings do not exist, the influence of shadows from objects, deterioration of lane markings over time, and the influence of other automobiles. Note that the boundaries of the roadway include the boundaries between driving lanes and the boundaries between the roadway and areas other than the roadway (sidewalks).

[0006] The disclosed technology has been made in consideration of the above points, and aims to provide a recognition device, a recognition method, and a recognition program that can appropriately recognize the surrounding environment of a vehicle while it is moving. [Means for solving the problem]

[0007] An aspect of the present disclosure is a recognition device that detects the boundary of a roadway using time-series images captured by a camera mounted on a vehicle, and has an acquisition unit that acquires the time-series images, and a detection unit that classifies areas in the acquired time-series images into roadway, which are areas with little change, and non-roadway, which are areas with a large amount of change, based on the magnitude of the amount of change, and detects the boundary between the area with little change and the area with a large amount of change as the boundary of the roadway. Effect of the Invention

[0008] According to the disclosed technology, it is possible to appropriately recognize the surrounding environment of a vehicle while it is moving. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating a schematic diagram of a means for recognizing the surrounding environment captured on a drive recorder of a vehicle to be observed. [Diagram 2] FIG. 13 is a diagram showing an example of a case where a boundary cannot be patterned due to color scheme or the like. [Diagram 3] FIG. 13 is a diagram showing an example of a case where a boundary cannot be patterned due to the effect of shadows and the absence of partition lines. [Figure 4] FIG. 13 is a diagram showing an example of a case where a boundary cannot be patterned even if a dividing line exists. [Diagram 5] FIG. 2 is a diagram showing an example of time-series images captured by an in-vehicle camera. [Figure 6] 2 is a block diagram showing a hardware configuration of a recognition device according to the present disclosure. FIG. [Figure 7] 1 is a block diagram showing a functional configuration of a recognition device according to the present disclosure. [Figure 8] This is a diagram in which information on the number of lanes in road information is applied to an image. [Figure 9] FIG. 1 is an image of the differential method. [Figure 10] FIG. 13 is a diagram illustrating an example of calculation of a difference image. [Figure 11] FIG. 13 is a diagram showing an example of a difference image calculated by emphasizing a boundary line; [Figure 12] This is a difference image obtained by superimposing the obtained frame difference images over a certain time range. [Figure 13] FIG. 13 is a diagram showing an example of correction for truncating pixels with low frequency of occurrence in consideration of the frequency of occurrence. [Figure 14] FIG. 13 is a diagram showing an example of correction for truncating pixels with low frequency of occurrence in consideration of the frequency of occurrence. [Figure 15] FIG. 13 is a diagram showing an example of correction for truncating pixels with low frequency of occurrence in consideration of the frequency of occurrence. [Figure 16] FIG. 13 is a diagram showing an example of correction for dividing a difference image area from a vanishing point. [Figure 17] 13 is a diagram showing a process for removing a region in which the host vehicle is reflected in a differential image; [Figure 18] FIG. 13 is a diagram showing how regions are identified and classified from a difference image after correction. [Figure 19] This is an example of correcting the boundary between the roadway and the sidewalk when it is difficult to distinguish the boundary. [Figure 20] This is an example of detection of corrections that change drastically in what is called a zebra zone on a road. [Figure 21] FIG. 13 is a diagram showing an example of correction according to the width of a region. [Figure 22] FIG. 13 is a diagram showing an example of region division based on a vanishing point. [Diagram 23] FIG. 13 is a diagram illustrating an example of region correction using a vanishing point. [Figure 24] 1 is a flowchart showing a flow of a recognition process performed by the recognition device of the present disclosure. [Diagram 25] FIG. 13 is a diagram showing an example in which the recognition of the surrounding environment of the present technology is utilized to notify an alert to a driver or an automated driving vehicle. [Figure 26]FIG. 13 is a diagram showing an example in which the surrounding environment recognition of the present technology is utilized for object detection. [Figure 27] 13 illustrates an example configuration of a recognition device according to a modified example of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0011] First, an overview of the present disclosure will be described. The technology of the present disclosure can be applied to a means for recognizing the surrounding environment captured on the drive recorder of a vehicle to be observed, as shown in Fig. 1. The surrounding environment can be recognized by classifying objects around the vehicle into objects that appear to move relatively and objects that do not appear to move relatively, and detecting lane markings and boundaries between roadways and sidewalks.

[0012] As described in the above problem, there are cases where the boundary cannot be patterned. Examples of the boundary lines that cannot be patterned include cases where the color of the boundary line is not limited to the general white or orange, and cases where the shape is other than a simple solid line or a broken line, such as a boundary line for slowing down. FIG. 2 is a diagram showing an example of a boundary line that cannot be patterned. L1 in FIG. 2 is a case where letters are drawn on the road and multiple boundary line patterns are included. L2 to L4 are cases where multiple colored boundary lines such as white lines, blue lines, and green lines are mixed. Furthermore, L4 is an area for bicycle lanes, and it is difficult to determine where the road ends. In these cases, it is difficult to recognize the boundary as a pattern.

[0013] L5 in FIG. 3 shows an example of a case where there are no dividing lines and shadows are present. Due to sunlight conditions and shadows, "white" dividing lines do not always appear to be the same color. In addition, if "white" dividing lines are set as a detection condition, guardrails also meet this condition, and may be detected as boundaries, leading to false detection. L6 shows a scene where there are no dividing lines at the boundary between the roadway and the sidewalk, and the boundary between the roadway and the sidewalk is a block and not a dividing line.

[0014] Furthermore, even if there are dividing lines, there are cases where they cannot be patterned. Figure 4 shows an example of a dividing line that cannot be patterned. L7 in Figure 4 is a broken dividing line, but the area between the broken line and the block is less than one lane, so it is difficult to identify what area it is. L8 is a dividing line that has become faded due to deterioration over time. In the case of two or more lanes, it is considered normal for the left and right sides to be broken lines, but L9 is a case where, even in the case of two or more lanes, for example, the line just before the intersection is solid.

[0015] In view of the above, it is desired to detect the lane markings without problems even when they are difficult to recognize, regardless of the presence or absence of color or shadow. In addition to the lane markings, it is also desired to detect the boundary between the roadway and the sidewalk. Therefore, the technology disclosed herein recognizes the boundary by capturing the changing surroundings. For example, in the case where the color or shape cannot be patterned, even if the pattern cannot be formed, the overall characteristics are similar, so it is assumed that the change when viewed in the time-series image is smaller than that of the background. In addition, when the appearance changes over time, such as due to the influence of shadows, the tendency of the time-series image as a whole will change, but the relationship between the background with large changes and the roadway with small changes will not change (however, the amount of change will decrease overall, so the threshold value needs to be lowered). As described above, the surrounding environment can be captured by assuming that the area with relatively large changes is the surroundings. With this method, the surrounding environment of the vehicle can be recognized regardless of the presence or absence of lane markings, and the lane of the vehicle, the surrounding lanes, and the boundary between the roadway and the sidewalk can also be estimated. Note that the following description will be given of the case where the method is applied to the roadway, but the method can be applied to any area in real space having a predetermined characteristic and any other area. A region in real space having a predetermined characteristic is a region in which a plurality of patterns or similar patterns are continuous, and the proportion of features that do not fit the patterns is small. Therefore, the present technology may be applied to railroad tracks, etc.

[0016] 5 is a diagram showing an example of time-series images captured by an on-vehicle camera. From the time-series images shown in FIG. 5, the following logic (1) to (6) can be explained. (1) The vehicle moves. (2) When using the vehicle's onboard camera, it appears that the "surroundings" are moving, rather than the vehicle itself. (3) Of the “surroundings,” roadways are made of the same material, such as concrete, are constructed continuously over a certain distance, and similar colors appear in succession, making it difficult for the color to change. (4) Curb blocks, sidewalks, and buildings in relation to the roadway are prone to change due to their various colors. (5) Therefore, the areas with relatively large changes are considered to be the periphery, not the roadway. (6) It is possible to recognize the surrounding environment from the magnitude of change over time in pixels of the same coordinates or in the same area on an image.

[0017] The configuration of this embodiment will be described below.

[0018] FIG. 6 is a block diagram showing a hardware configuration of the recognition device 100 of the present disclosure.

[0019] 6, the recognition device 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0020] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14. In this embodiment, the recognition program is stored in the ROM 12 or the storage 14.

[0021] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system, and various data.

[0022] The input unit 15 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.

[0023] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0024] The communication interface 17 is an interface for communicating with other devices such as terminals. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0025] Next, a description will be given of each functional configuration of the recognition device 100. Fig. 7 is a block diagram showing the functional configuration of the recognition device 100 of the present disclosure. Each functional configuration is realized by the CPU 11 reading out a recognition program stored in the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it.

[0026] As shown in FIG. 7, the recognition device 100 includes an acquisition unit 110, a detection unit 112, and a road database 114. The recognition device 100 acquires time-series images captured by a camera mounted on a vehicle using the acquisition unit 110. The detection unit 112 is a processing unit that detects the boundary of the surrounding environment of the vehicle to be observed (hereinafter simply referred to as the vehicle) and outputs the detection result, and is functionally divided into a difference calculation unit 120, a shaping unit 122, and an evaluation unit 124. The road database 114 stores road information including the number of lanes on the roadway. As shown in FIG. 8, the number of lanes information in the road information is applied to the image, and it is possible to identify which lane is the lane of the vehicle.

[0027] The detection unit 112 classifies the areas in the time-series images acquired by the acquisition unit 110 according to the magnitude of the amount of change, and detects the boundary of the roadway. By classifying the areas into roadway, which is an area with a small amount of change, and non-roadway, which is an area with a large amount of change, the boundary between the area with a small amount of change and the area with a large amount of change is detected as the boundary of the roadway. The processing of each part in the detection unit 112 will be described.

[0028] The difference calculation unit 120 calculates a difference image between time-series images. A frame difference method (hereinafter, difference method) is used to calculate the difference image. An image of the difference method is shown in FIG. 9. The difference method is a technique for detecting "changed pixels" by taking the logical product of the calculation results of the absolute difference between frames. The difference method can be applied to a method for detecting the surrounding environment using a camera fixedly installed on a moving object.

[0029] Fig. 10 is a diagram showing an example of calculating a difference image. Based on an image in time series t, an image t+1, and an image t+2, a difference image is binarized and calculated so as to emphasize the change at t+1. This method focuses on the fact that the color of a concrete road basically looks the same except for construction marks, and therefore the color of the road does not change.

[0030] Also, a difference image with emphasized boundaries may be calculated using an adaptive threshold method. FIG. 11 is a diagram showing an example of a difference image calculated with emphasized boundaries. The top of FIG. 11 is a difference image when the entire image is binarized with the same threshold value according to the normal frame difference method. In this case, darker colors such as colored markings disappear, and temporary dark areas such as "shades" are also detected. On the other hand, the bottom of FIG. 11 is a difference image calculated with emphasized boundaries. Because this method emphasizes boundaries, only outline information is required for shades, and colored markings can also be detected. If the markings are truly painted uniformly, only outline information is detected, but in reality, differences are detected within the markings due to depressions or protrusions on the road surface, or scratches in the paint.

[0031] FIG. 12 shows a difference image obtained by overlapping the obtained frame difference images over a certain time range.

[0032] The shaping unit 122 corrects the difference image calculated by the difference calculation unit 120. As a correction method, a method suitable for each difference image is used, as described below.

[0033] As an example of correction, correction for cutting out low-frequency pixels in consideration of the frequency of appearance will be described. FIG. 13 shows an example of correction for cutting out low-frequency pixels in consideration of the frequency of appearance. In the difference image in the upper left of FIG. 13, low-frequency pixels that exist locally on the roadway and sidewalk, such as manholes and shaded areas, are removed. Also included are objects on the roadway that appear infrequently and are detected as large changes. In this case, pixels corresponding to lane lines and buildings that frequently change due to the influence of protrusions, dents, and slight left and right running shake of the vehicle remain without being removed.

[0034] In the image shown on the left of Fig. 14, the boundary between the roadway and the sidewalk can be extracted from the color changes of the road shoulder and step blocks. In the difference image shown on the right of Fig. 14, information on momentary shade and passing vehicles has been reduced. Vehicles can be further reduced by applying processing to the rectangular area detected by object detection.

[0035] Furthermore, when the colors of individual features or elements between features, such as guardrail posts, poles, or surrounding environmental features such as vegetation, differ, the boundary between the roadway and the sidewalk is further emphasized. For example, a guardrail is a single color on its own, but since it is installed by posts, the background sidewalk and buildings are naturally reflected between the guardrail posts, and the colors change frequently. Also, when the sidewalk itself is colorful (for example, when it is made up of tiles of two or more colors and depicts a grid or pattern), pixel values ​​change frequently, emphasizing the differences, making it even easier to distinguish. In this way, areas where there is little change in pixel value and where the differences are unlikely to differ are extracted as black pixels, and areas where there is a lot of change in pixel value and where the differences are emphasized are extracted as white pixels.

[0036] In the case of the top image in Figure 15, the colors of the roadway and the sidewalk are close and, when the sidewalk color is constant, there is little change in pixel value in the "sidewalk" area, so it is extracted as black pixels. However, it is possible to determine the boundary line from the change in color of the boundary block between the roadway and the sidewalk. In the case of the bottom image in Figure 15, there are geographical objects in the surrounding environment, and there are frequent changes in pixel value, emphasizing the differences, so it is extracted as white pixels. This concludes the explanation of correction based on frequency of occurrence.

[0037] The shaping unit 122 may also perform correction to divide the difference image area from the vanishing point. The difference image is divided into areas expressed by triangles, for example, with the vanishing point as the center, and divided into areas that have changed and areas that have not changed based on the ratio of black pixels to white pixels in each area. As shown in Fig. 16, if the installation position of the camera and information on the angle of view (the information on the angle of view is fixed for each model and internal settings), and the width of each lane on the road on which the vehicle is traveling are known, the position of the lane markings reflected in the image can be roughly identified.

[0038] The shaping unit 122 may also perform correction to remove the area of ​​the differential image in which the vehicle is reflected. As shown in Fig. 17, the lower area of ​​the differential image is cut because the vehicle may be reflected in the lower area. In this way, the shaping unit 122 performs processing so that the area in which the vehicle itself is photographed is not detected by the detection unit 112.

[0039] By the above-mentioned processing by difference calculation section 120 and shaping section 122, in the difference image, areas with a large amount of change are extracted in white, and areas with a small amount of change are extracted in black.

[0040] The evaluation unit 124 classifies the regions and detects boundaries by identifying the region of the vehicle's driving lane and the regions on the left and right of the lane from the difference image corrected by the shaping unit 122. The evaluation unit 124 predetermines a threshold value for distinguishing between black and white in the region classification. The region of the difference image after correction is divided into a "black" region with a small amount of change and a "white" region with a large amount of change. The "white" region includes lines extracted as dividing lines. In the classification of regions by the evaluation unit 124, for example, road information corresponding to the number of lanes of the roadway on which the vehicle is traveling is obtained from the road database 114, and the region with a small amount of change is classified as the vehicle's driving lane and the surrounding driving lanes based on the obtained road information and the extracted dividing lines. The evaluation unit 124 then classifies the region with a small amount of change as the roadway and the region with a large amount of change as other than the roadway.

[0041] An example of detection in the evaluation unit 124 will be described below. FIG. 18 is a diagram showing how the area is specified and classified from the difference image after correction. First, in the area (1), it is specified that the black area in the center of the image is the vehicle's driving lane. From the adjacent white dividing lines, it is specified that the surrounding driving lanes are in the area (2). Furthermore, from the adjacent white dividing lines, it is specified that the area (3A) has a driving lane, and the area (3B) does not have a driving lane. Furthermore, it is specified that the area (4) to the left of (3A) does not have a driving lane. From the above specification, the area is classified into a roadway, which is an area with a small amount of change, and a non-roadway, which is an area with a large amount of change, and it is detected that the boundary (5) that separates the white and black areas is the boundary between the roadway and the sidewalk. In addition, by referring to the number of lanes of the driving lanes on the left and right and road information, it is possible to detect that the vehicle is driving in the third lane based on the information that this road is a "four-lane road on one side."

[0042] As examples of correction by the shaping unit 122, several patterns will be described.

[0043] Figure 19 shows an example of boundary correction when the boundary between the roadway and the sidewalk is difficult to distinguish. When only similar colors appear continuously for a certain section of the sidewalk, it may look like the roadway, but elements such as the shoulder and curbstones become "white" areas. Therefore, these elements are considered to be a certain range of "white" areas, and the "white" area of ​​a certain range can be detected as the boundary.

[0044] 20 shows an example of correction of a region of a road where there are drastic changes, which corresponds to a so-called zebra zone. A certain range of continuous zebra zones can be corrected by the shaping unit 122 as a region where there are large changes.

[0045] An example of correction according to the width of the area is shown below. In the case of the left side of Fig. 21, the narrow "black" area is regarded as "white" and is filled in by the correction of the shaping unit 122. In this way, when the width of the "black" area is narrower than a predetermined length, the shaping unit 122 corrects it to a "white" area.

[0046] FIG. 22 is an example of area division based on the vanishing point. The "black" area directly below the vanishing point is considered to be the vehicle's driving lane. If the "white" area to the left is narrow and there is a "black" area of ​​a certain width or more further to the left, it is determined that there is a driving lane adjacent. In FIG. 22 as well, the narrow "black" area is considered to be "white" and filled in by the correction made by the shaping unit 122.

[0047] Figure 23 shows area correction using vanishing points. If the dividing line between lanes cannot be recognized and the wide "black" area is the vehicle's driving lane, the ratio of the left and right "black" areas is calculated, and if there is an extreme difference, the area is divided into two areas with a "white" area inserted between them.

[0048] Next, the operation of the recognition device 100 will be described.

[0049] 24 is a flowchart showing the flow of the recognition process by the recognition device 100 of the present disclosure. The CPU 11 reads out the recognition program from the ROM 12 or the storage 14, loads it into the RAM 13, and executes it to perform the recognition process.

[0050] In step S100, the CPU 11, functioning as the acquisition unit 110, acquires time-series images taken by a camera mounted on a vehicle.

[0051] In step S102, the CPU 11 functions as the difference calculation section 120 to calculate a difference image between the time-series images.

[0052] In step S104, the CPU 11, functioning as the shaping unit 122, corrects the calculated difference image.

[0053] In step S106, the CPU 11, as the evaluation unit 124, classifies the areas with a small amount of change into the vehicle's driving lane and the roadway of the surrounding driving lanes, and classifies the areas with a large amount of change into non-roadway. The classification is performed by acquiring road information corresponding to the number of lanes of the roadway on which the vehicle is traveling from the road database 114, and based on the road information and the lane markings extracted from the corrected difference image.

[0054] In step S108, the CPU 11 as the evaluation unit 124 detects road boundaries representing boundary lines between lanes of driving lanes and boundary lines between roadways and non-roadways from the classification results, and outputs the detection results.

[0055] As described above, the recognition device 100 of this embodiment can appropriately recognize the surrounding environment of a traveling vehicle.

[0056] An example of the use of this technology will be described. FIG. 25 shows an example of the case where the recognition of the surrounding environment of this technology is used to notify an alert to a driver or an automated driving vehicle. For example, in the situation of FIG. 25, if an automated driving vehicle mistakenly recognizes that "both the left and right are broken lines" and determines that it is driving in the second lane, and changes lanes to the left, there is a risk that it will run onto the sidewalk. The left side is not a broken line but is actually a bicycle zone. If the recognition of the surrounding environment of this technology shows that the boundary of the road is immediately to the left of the vehicle's driving lane, such a problem will not occur. Also, if it is known that there is a bicycle zone between the dividing line and the road boundary line, an alert can be issued when the driver moves to park on the side of the road.

[0057] Figure 26 shows an example of the use of the surrounding environment recognition of this technology for object detection. For example, when detecting a vehicle, a vehicle on the roadway and a vehicle on the premises (parking lot, etc.) may be captured at the same time. Even if there is no dividing line at the edge of the road or the line has been rubbed off and cannot be seen, if the boundary between the roadway and the sidewalk can be detected, it is possible to determine whether or not the vehicle is on the roadway and classify it.

[0058] (Modification) As a modification of the above embodiment, as shown in Fig. 27, a speed acquisition unit 130 that acquires the speed of the vehicle may be provided, and the difference calculation unit 120 of the detection unit 112 may calculate the difference image by using only images captured when the speed is equal to or higher than a predetermined threshold. The reason for selecting images according to the speed in this manner is that the key aspect of this technology is to capture that the "background is moving" as the observed vehicle moves. For example, a process is performed in which the difference image is calculated using only images taken when the vehicle is moving at 20 km / h or more, using information on the vehicle speed.

[0059] We will also explain examples of applications other than roadways. For example, it can be applied to trains and railway tracks to detect rails. Rails always appear to be a constant color (i.e. black) excluding the effects of shadows, and between the rails sleepers and ballast (gravel) are installed alternately, emphasizing the changes, and similarly, the ballast on the outside of the rails is not a constant color, so the changes are also emphasized. In other words, both sleepers and ballast can be detected as "white" areas.

[0060] When applying this technology to a moving robot, we can imagine a robot moving along a sidewalk and control the robot so that it does not stray from the road, which has a constant color, to obtain a time-series image that can be used with this technology.

[0061] In the above embodiment, the recognition process executed by the CPU by reading the software (program) may be executed by various processors other than the CPU. In this case, examples of the processor include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. The recognition process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, and a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0062] In the above embodiment, the recognition program is pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may be downloaded from an external device via a network.

[0063] The following supplementary notes are further disclosed regarding the above embodiment.

[0064] (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, A recognition device that detects road boundaries using time-series images captured by a camera mounted on a vehicle, comprising: Acquiring the time series images; Classifying areas in the acquired time-series images into roadway areas where the amount of change is small and non-roadway areas where the amount of change is large according to the magnitude of the change, and detecting the boundary between the area where the amount of change is small and the area where the amount of change is large as the boundary of the roadway. The recognition device is configured as follows.

[0065] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to perform a recognition process for detecting road boundaries using time-series images captured by a camera mounted on a vehicle, Acquire the time series images; Classifying areas in the acquired time-series images into roadway areas where the amount of change is small and non-roadway areas where the amount of change is large according to the magnitude of the change, and detecting the boundary between the area where the amount of change is small and the area where the amount of change is large as the boundary of the roadway. Non-transitory storage medium. [Explanation of symbols]

[0066] 100 recognition device 110 Acquisition Department 112 Detection unit 114 Road Database 120 Difference calculation part 122 Plastic Surgery Department 124 Evaluation Section 130 Speed ​​acquisition section

Claims

1. A recognition device that detects road boundaries using time-series images captured by a camera mounted on a vehicle, comprising: An acquisition unit for acquiring the time-series images; a detection unit that uses the acquired time-series images to generate an image in which changes between the time-series images are emphasized, calculates a difference image by overlaying the images in which the changes are emphasized, calculates an amount of change in an area based on changes in pixels of the difference image, classifies the areas in the time-series images into a roadway where the amount of change is small and a non-roadway where the amount of change is large according to the magnitude of the calculated amount of change, and detects the boundary between the area with the small amount of change and the area with the large amount of change as the boundary of the roadway; A recognition device having the above configuration.

2. The recognition device according to claim 1 , wherein the detection unit further performs processing so that an area in which the vehicle itself is photographed from the time-series images is not included in the detection target of the detection unit.

3. A speed acquisition unit that acquires a speed of the vehicle, 3. The recognition device according to claim 1, wherein the detection unit uses only images captured when the speed is equal to or greater than a predetermined threshold value.

4. The recognition device according to claim 1 , wherein, in a region of the difference image that is continuous over a certain range and is related to a driving lane, if the width of the region in which the amount of change is small is narrower than a predetermined length, the region is corrected to a region in which the amount of change is large.

5. The recognition device according to any one of claims 1 to 4, wherein the detection unit acquires road information corresponding to the number of lanes of the road on which the vehicle is traveling, and identifies the vehicle's driving lane and surrounding driving lanes as areas with little change based on the road information and the dividing line extracted from the time-series images.

6. A recognition method for a computer that detects road boundaries using time-series images captured by a camera mounted on a vehicle, comprising: Acquiring the time series images; Using the acquired time-series images, an image is generated in which the changes between the time-series images are emphasized, a difference image is calculated by overlaying the images in which the changes are emphasized, an amount of change in an area is calculated based on changes in pixels of the difference image, the area in the time-series images is classified into a roadway where the amount of change is small and a non-roadway where the amount of change is large according to the magnitude of the calculated amount of change, and the boundary between the area where the amount of change is small and the area where the amount of change is large is detected as the boundary of the roadway. A recognition method that causes a computer to execute a process.

7. A recognition program in a computer that detects road boundaries using time-series images captured by a camera mounted on a vehicle, comprising: Acquiring the time series images; Using the acquired time-series images, an image is generated in which the changes between the time-series images are emphasized, a difference image is calculated by overlaying the images in which the changes are emphasized, an amount of change in an area is calculated based on changes in pixels of the difference image, the area in the time-series images is classified into a roadway where the amount of change is small and a non-roadway where the amount of change is large according to the magnitude of the calculated amount of change, and the boundary between the area where the amount of change is small and the area where the amount of change is large is detected as the boundary of the roadway. A recognition program that causes a computer to carry out processing.

Images