Image analysis device, image analysis method, and program

Abstract

An image analysis device (10) according to the present disclosure is provided with: a region division unit (11) that performs region division on an image captured of a road surface and that generates a region division result obtained by classifying the image into image regions of a plurality of categories; a depth estimation unit (12) that performs depth estimation on the image and generates depth estimation information; and a detection unit (13) that detects projections and recesses present on the road surface, on the basis of the region division result and the depth estimation information. The plurality of categories include a road category. The detection unit (13) detects projections and recesses present on the road surface in an image region classified into the road category.

Description

[Technical field]

[0001] The present disclosure relates to an image analysis device, an image analysis method, and a computer-readable medium having a program stored thereon. [Background technology]

[0002] Patent Document 1 describes a device that uses an on-board camera to monitor the surroundings of a vehicle. This vehicle surroundings monitoring device detects edges parallel to the road based on an image captured by a CCD camera built into the left door mirror of the vehicle, and extracts edges of three-dimensional obstacles by excluding edges corresponding to paint markings on the road from among the detected edges. If the distance between the extracted edge of the three-dimensional obstacle and the vehicle is equal to or less than a predetermined distance, a display or speaker is used to notify the driver of the possibility of contact between the three-dimensional obstacle and the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-112977 A Summary of the Invention [Problem to be solved by the invention]

[0004] Many people feel uneasy when driving on narrow roads with curbs and gutters on both sides. When driving close to the edge of a narrow road to pass an oncoming vehicle, careless driving can cause the wheels to come into contact with the curb or run off into the gutter.

[0005] An object of the present disclosure is to provide an image analysis device, an image analysis method, and a computer-readable medium storing a program capable of selectively detecting unevenness on a road surface. [Means for solving the problem]

[0006] An image analysis device in one embodiment includes an area division means that performs area division on an image of a road surface and generates an area division result in which the image is classified into image areas of multiple categories, a depth estimation means that performs depth estimation on the image and generates depth estimation information, and a detection means that detects unevenness present on the road surface based on the area division result and the depth estimation information.

[0007] An image analysis method according to one embodiment performs region division on an image of a road surface, generates a region division result that classifies the image into image regions of multiple categories, performs depth estimation on the image, generates depth estimation information, and detects unevenness on the road surface based on the region division result and the depth estimation information.

[0008] A computer-readable medium in one embodiment stores a program that causes a computer to execute the following processes: performing area division on an image of a road surface and generating area division results that classify the image into image areas of multiple categories; performing depth estimation on the image and generating depth estimation information; and detecting unevenness present on the road surface based on the area division result and the depth estimation information. Effect of the Invention

[0009] According to the above-described aspects, it is possible to provide an image analysis device, an image analysis method, and a computer-readable medium storing a program capable of selectively detecting unevenness on a road surface. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of an image analysis device according to an embodiment; [Diagram 2] FIG. 1 is a diagram showing an example of an image of a surrounding road surface; [Diagram 3] FIG. 3 is a diagram showing a result of region segmentation generated from the image in FIG. 2. [Figure 4] 3 is a diagram showing depth estimation information generated from the image of FIG. 2. [Diagram 5] FIG. 13 is a diagram showing a detection result in which the region segmentation result and depth estimation information are superimposed. [Figure 6] FIG. 2 is a flow diagram showing an image analysis method according to an embodiment. [Figure 7] FIG. 2 is a block diagram showing a hardware configuration of a computer that realizes the image analysis device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. Note that the following description and drawings are omitted and simplified as appropriate for clarity of explanation. In addition, in each of the following drawings, the same elements are given the same reference numerals, and duplicated explanations are omitted as necessary. In addition, the specific numerical values ​​shown below are merely examples for facilitating understanding of the present disclosure, and are not limited thereto.

[0012] The embodiment relates to a technology used, for example, when driving a vehicle, for detecting unevenness on a road surface around the vehicle. Fig. 1 is a block diagram showing a configuration of an image analyzing device 10 according to the embodiment. The image analyzing device 10 is mounted on a vehicle as one of the vehicle-mounted safety devices.

[0013] A vehicle is equipped with multiple cameras. For example, the cameras may be mounted on the left side of the vehicle body. This camera can capture an image of the area around the left front wheel of the vehicle, which is a blind spot for the driver. FIG. 2 is a diagram showing an example of an image of the road surface around the vehicle. In FIG. 2, the captured image shows a road 1 and a part of the left side of the vehicle body of a vehicle 2. In this example, the road 1 includes a roadway 1a on which the vehicle 2 runs and a road shoulder 1b, which are demarcated by a white line.

[0014] As shown in Fig. 1, an image analysis device 10 includes a region division unit 11, a depth estimation unit 12, and a detection unit 13. The region division unit 11 performs region division (also called region recognition or segmentation) on the image of the surrounding road surface described above, and generates a region division result in which the image is classified into image regions of a plurality of categories. As a region division technique, for example, Semantic Segmentation using a Fully Convolutional Network (FCN) can be used.

[0015] Semantic segmentation performs class classification (inference) on a pixel-by-pixel basis for an input image, labels each classified pixel with a category, and divides the image into image regions of multiple categories, outputting the region division results. The region division results are, for example, an image with each divided image region color-coded.

[0016] The on-board camera may capture images of the vehicle's driving conditions in real time at a predetermined frame rate, and input the captured images to the image analysis device 10. The image analysis device 10 can acquire images input at a predetermined frame rate, classify objects included in the images, and output the class-classified images as region segmentation results at a predetermined frame rate.

[0017] FIG. 3 is a diagram showing a result of region segmentation generated from the image of FIG. 2. For the purpose of explanation, in FIG. 3, the result of region segmentation is superimposed on the captured image. Also, in FIG. 3, dashed lines are added to surround each of the multiple image regions in order to clarify each image region. The multiple categories include a road category. In the example of FIG. 3, the input image is segmented into two image regions: a road region 1A (a checkered pattern shaded region) which is in the road category, and a vehicle region 2A (a diagonal shaded region).

[0018] The depth estimation unit 12 performs depth estimation on the input image and generates depth estimation information. The depth estimation unit 12 analyzes the captured image using, for example, a model based on a depth estimation algorithm using a convolutional neural network, and estimates the depth of each pixel. The depth estimation algorithm is, for example, CNN-Depth. The depth estimation information indicates the correspondence between the position coordinates and the depth for each pixel in the captured image.

[0019] FIG. 4 is a diagram showing depth estimation information generated from the image of FIG. 2. FIG. 4 is a depth map that visualizes the estimated depth information, in which areas with high (deep) depth are drawn in dark colors and areas with low (shallow) depth are drawn in light colors. For the sake of explanation, FIG. 4 In Fig. 4, depth estimation information is overlaid on the captured image. Also, dashed lines have been added to surround areas of equal depth. 4 As shown in Fig. 1, it can be seen that a deep depth area 1B exists within the road shoulder 1b. In this way, boundaries on the depth map are generated in places where gutters, curbs that define the boundaries between the road and the sidewalk, etc. exist.

[0020] The detection unit 13 detects unevenness on the road surface based on the region division result and the depth estimation information. FIG. 5 is a diagram showing a detection result in which the region division result and the depth estimation information are superimposed. 5 5, the region division result and the depth estimation information are superimposed on the captured image. The detection unit 13 detects a deep depth region 1B in a road region 1A shown in FIG.

[0021] In this way, by using the region division result and the depth estimation information, it becomes possible to detect unevenness for each image region divided into categories. As a result, for example, when driving on a narrow road with gutters on both sides, it is possible to recognize the unevenness on the road surface and prevent the vehicle from running off into the gutters or hitting the curb. By analyzing images from multiple cameras with the image analysis device 10, it becomes possible to recognize the unevenness on the road surface around the vehicle and operate the vehicle more safely.

[0022] It is preferable that the detection unit 13 selectively detects unevenness present on the road surface in the road area 1A from among the divided categories. This makes it possible to selectively detect unevenness on the road where the vehicle 2 is expected to enter, and to give an appropriate warning to the driver.

[0023] In the above example, the depth estimation unit 12 performs depth estimation for the entire captured image, but is not limited to this. The depth estimation unit 12 may perform depth estimation for at least one of the image regions divided by the region division unit 11 (for example, the road region 1A).

[0024] Also, depending on the height (depth) of a groove or step on the road surface, it may not cause a hindrance to vehicle passage. Therefore, the detection unit 13 can detect only the points where the estimated depth is outside a predetermined threshold range as unevenness existing on the road surface. In other words, it becomes possible to appropriately detect deep grooves into which the tires of a vehicle may get stuck and steps too high to overcome. Note that the width and length of the groove or the like may be calculated based on the depth estimation information, and the unevenness on the road surface may be detected using the calculated size.

[0025] Here, the process of detecting unevenness on a road surface by the image analyzing device 10 will be described with reference to Fig. 6. Fig. 6 is a flow diagram showing an image analyzing method according to an embodiment. As shown in Fig. 6, first, the image analyzing device 10 acquires an image (Fig. 2) of the surrounding road surface captured by a camera mounted on a vehicle (step S10).

[0026] Next, the area division unit 11 divides the image acquired in S10 into areas and generates an area division result (step S11). After that, the depth estimation unit 12 estimates the depth of the image acquired in S10 and generates depth estimation information (step S12). Note that the area division and the depth estimation processes may be performed in either order, or may be performed simultaneously.

[0027] Then, the detection unit 13 detects unevenness on the road surface based on the region division result and the depth estimation information (step S13), and outputs the detection result (step S14). The detection result can be displayed, for example, on a display device inside the vehicle. Also, for example, when the distance between the detected unevenness on the road surface and the vehicle is equal to or less than a predetermined value, a warning sound may be output to notify the driver of a danger.

[0028] It is also possible to generate a steering intervention command to intervene in the steering wheel operation so as to move the vehicle away from the detected unevenness of the road surface, thereby forcibly preventing the vehicle from running off into a road ditch.

[0029] Furthermore, the vehicle may be provided with a function of notifying other vehicles of the presence of unevenness on the road surface. When notifying other vehicles of the detection result, the notification may be made using vehicle-to-vehicle communication (also called V2V communication). Also, the detection result may be notified to other vehicles using road-to-vehicle communication (also called V2I communication) via road infrastructure such as communication devices installed on traffic lights.

[0030] The timing for starting the process shown in Fig. 6 may include the following examples. First, when the image analyzing device 10 is mounted on a vehicle, it can start the unevenness detection process when a predetermined condition related to the driving situation of the vehicle is satisfied. The predetermined condition related to the driving of the vehicle is the driver or a passenger pressing a process start button. In addition, the position information of the vehicle can be acquired using a GPS (Global Positioning System) mounted on the vehicle, and the above process can be started when the vehicle enters, for example, a narrow alley.

[0031] The predetermined conditions regarding the driving of the vehicle may include the lane in which the vehicle is traveling. That is, the process of detecting unevenness on the road surface can be started according to the lane in which the vehicle is traveling. The lane in which the vehicle is traveling can be determined based on map information and position information such as GPS and GNSS. It is also possible to detect road dividing lines (lines formed on the road surface in white or yellow, etc.) from the captured image and determine the lane in which the vehicle is traveling based on the detection result.

[0032] The method for detecting road markings from a captured image can be a known method based on edges of paint marks drawn on the road surface included in the image, a method using machine learning, etc. In addition, if the category of white lines on the road surface is included in the categories divided by the area division unit 11, the result may be used to detect road markings.

[0033] For example, when the lane the vehicle is traveling in is the lane at the edge of the road, the process of detecting unevenness in the road surface can be started. This allows appropriate detection of dangers such as curbs on the road shoulder and gutters when the vehicle moves to the edge lane due to a lane change, when the number of lanes is reduced and the traveling lane becomes the edge lane, when the vehicle moves to a one-lane road by turning right or left, etc.

[0034] Furthermore, if the vehicle is equipped with a distance measurement function such as a millimeter wave radar, laser radar, or stereo camera, the above processing can be started when the distance between the vehicle and the side wall on the side of the road becomes less than a predetermined value.

[0035] In addition, each functional block performing various processes shown in the drawings can be configured in hardware terms with a processor, memory, and other circuits. Also, the above-mentioned processes can be realized by having a processor execute a program. Therefore, these functional blocks can be realized in various forms by only hardware, only software, or a combination of both, and are not limited to any one of them.

[0036] Next, a hardware configuration of the image analyzing device 10 according to the embodiment will be described. Fig. 7 is a block diagram showing the hardware configuration of the computer 20 that realizes the image analyzing device 10. However, the configuration shown in Fig. 7 is not intended to limit the configuration of the image analyzing device 10.

[0037] As shown in FIG. 7, the computer 20 includes a processor 21, a memory 22, a storage 23, an input / output interface (input / output I / F) 24, and a communication interface (communication I / F) 25, which are interconnected by a bus 26.

[0038] The processor 21 is, for example, a central processing unit (CPU) or a graphics processing unit (GPU). The memory 22 is, for example, a random access memory (RAM) or a read only memory (ROM). The storage 23 is, for example, a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a memory card.

[0039] The storage 23 stores programs that realize the functions of the components of the image analyzing device 10. The processor 21 executes these programs to realize the functions of the components of the image analyzing device 10. Here, when executing the above programs, the processor 21 may read these programs onto the memory 22 and then execute them, or may execute them without reading them onto the memory 22. The memory 22 and the storage 23 also serve to store information and data held by the components of the image analyzing device 10.

[0040] The above-mentioned program can be stored and provided to a computer using various types of non-transitory computer readable media. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be provided to a computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium can provide the program to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0041] The input / output interface 24 is connected to a display device 30, an input device 31, a sound output device 32, etc. The display device 30 is an LCD (Liquid Crystal Display) or the like, and displays an image of the captured road surface, an area division result processed by the processor 21, and an image corresponding to the depth estimation information. For example, when the distance between the vehicle and unevenness of the road surface obtained based on the area division result and the depth estimation information is equal to or less than a predetermined value, the display device 30 can display a warning to notify the driver of a danger.

[0042] The input device 31 is a device that accepts an operation input from the driver, and is, for example, a touch sensor. The display device 30 and the input device 31 may be integrated and realized as a touch panel. The sound output device 32 outputs a sound corresponding to the acoustic data processed by the processor 21. For example, the sound output device 32 can output a warning sound when the distance between the vehicle and the unevenness of the road surface obtained based on the region division result and the depth estimation information is equal to or less than a predetermined value, thereby notifying the driver of a danger.

[0043] The communication interface 25 transmits and receives data to and from an external device via, for example, a wireless communication path. For example, road condition information such as the width of the road and the presence or absence of a gutter can be acquired from the outside via the communication I / F 25. The image analyzing device 10 may start a process of detecting unevenness on the road surface according to the road condition information.

[0044] In addition, the image analysis device 10 can also be realized as a server installed in a data center or the like in another form. Image data captured by a camera mounted on a vehicle is transmitted to the server, and a process of detecting unevenness on the road surface is executed. The detection result from the server is received by a communication I / F mounted on the vehicle, and a warning display, a warning sound, etc. can be output.

[0045] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0046] Furthermore, some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0047] (Appendix 1) an area division means for dividing an image of a road surface into areas and generating an area division result in which the image is classified into image areas of a plurality of categories; a depth estimation means for performing depth estimation on the image and generating depth estimation information; a detection means for detecting unevenness present on the road surface based on the region division result and the depth estimation information; Equipped with Image analysis device. (Appendix 2) the plurality of categories includes a category of roads; The detection means detects irregularities present on the road surface within an image area classified into a road category. 2. The image analysis apparatus of claim 1. (Appendix 3) The detection means detects a location where the estimated depth is outside a predetermined threshold range as an unevenness existing on the road surface. 3. The image analysis apparatus according to claim 1 or 2. (Appendix 4) The image analysis device is a device mounted on a vehicle, the detection means starts the unevenness detection process when a predetermined condition related to the driving of the vehicle is satisfied. 4. The image analysis device according to claim 1, (Appendix 5) the detection means starts the process of detecting unevenness when the lane on which the vehicle is traveling satisfies a predetermined condition. 5. The image analysis apparatus of claim 4. (Appendix 6) The detection means identifies a lane in which the vehicle is traveling based on a lane marking included in the image. 6. The image analysis apparatus of claim 5. (Appendix 7) performing region segmentation on an image of a road surface, and generating a region segmentation result in which the image is classified into image regions of a plurality of categories; performing depth estimation on the image to generate depth estimation information; detecting irregularities present on the road surface based on the region division result and the depth estimation information; Image analysis methods. (Appendix 8) the plurality of categories includes a category of roads; Detecting irregularities present on the road surface within image regions classified into a road category; The image analysis method described in Appendix 7. (Appendix 9) A location where the estimated depth is outside a predetermined threshold range is detected as an unevenness existing on the road surface. 9. The image analysis method according to claim 7 or 8. (Appendix 10) The image analysis method is used when driving a vehicle, starting the unevenness detection process when a predetermined condition related to the driving of the vehicle is satisfied; 10. The image analysis method according to any one of claims 7 to 9. (Appendix 11) The process of detecting unevenness is started when the lane on which the vehicle is traveling satisfies a predetermined condition. 11. The image analysis method according to claim 10. (Appendix 12) Identifying a lane in which the vehicle is traveling based on a lane marking included in the image. 12. The image analysis method according to claim 11. (Appendix 13) A process of performing area segmentation on an image of a road surface and generating an area segmentation result in which the image is classified into image areas of a plurality of categories; performing depth estimation on the image to generate depth estimation information; A process of detecting unevenness present on the road surface based on the region division result and the depth estimation information; A computer-readable medium having a program stored thereon for causing a computer to execute the above. (Appendix 14) the plurality of categories includes a category of roads; In the process of detecting unevenness, unevenness present on the road surface within an image area classified into a road category is detected. 14. The computer-readable medium of claim 13. (Appendix 15) In the process of detecting unevenness, a location where the estimated depth is outside a predetermined threshold range is detected as an unevenness existing on the road surface. 15. The computer-readable medium of claim 13 or 14. (Appendix 16) The program is executed while the vehicle is in operation; A process of detecting unevenness is started when a predetermined condition related to the driving of the vehicle is satisfied. 16. The computer-readable medium according to any one of appendices 13 to 15. (Appendix 17) The process of detecting unevenness is started when the lane on which the vehicle is traveling satisfies a predetermined condition. 17. The computer-readable medium of claim 16. (Appendix 18) Identifying a lane in which the vehicle is traveling based on a lane marking included in the image. 18. The computer-readable medium of claim 17. [Explanation of symbols]

[0048] 1 road 1a roadway 1b Road shoulder 1A Road area 1B High depth area 2 Vehicles 2A Vehicle Area 10. Image analysis device 11 Area division part 12 Depth estimation section 13 Detection section 20. Computers 21 Processors 22 Memory 23. Storage 24 Input / Output Interface 25 Communication I / F 26 Bus 30 Display device 31 Input Devices 32 Sound output device

Claims

1. an area division means for dividing an image of a road surface captured from a traveling vehicle into areas and generating an area division result in which the image is classified into image areas of a plurality of categories; a depth estimation means for performing depth estimation on the image and generating depth estimation information; a detection means for detecting unevenness present on the road surface based on the region division result and the depth estimation information; Equipped with the detection means starts a process of detecting unevenness in a road surface when the lane in which the vehicle is traveling is a lane at the edge of a road. Image analysis device.

2. the plurality of categories includes a category of roads; The detection means detects irregularities present on the road surface within an image area classified into a road category. The image analysis device according to claim 1 .

3. The detection means detects a location where the estimated depth is outside a predetermined threshold range as an unevenness existing on the road surface.

3. The image analysis device according to claim 1 or 2.

4. The detection means identifies a lane in which the vehicle is traveling based on a lane marking included in the image. The image analysis device according to any one of claims 1 to 3.

5. The computer performing region segmentation on an image of a road surface captured from a traveling vehicle, and generating a region segmentation result in which the image is classified into image regions of a plurality of categories; performing depth estimation on the image to generate depth estimation information; Executing a process of detecting irregularities present on the road surface based on the region division result and the depth estimation information; The process of detecting unevenness of the road surface is started when the lane in which the vehicle is traveling is a lane at the edge of the road. Image analysis methods.

6. the plurality of categories includes a category of roads; Detecting irregularities present on the road surface within image regions classified into a road category; The image analysis method according to claim 5.

7. A process of performing region segmentation on an image of a road surface captured from a traveling vehicle, and generating a region segmentation result in which the image is classified into image regions of a plurality of categories; performing depth estimation on the image to generate depth estimation information; A process of detecting unevenness present on the road surface based on the region division result and the depth estimation information; A program for causing a computer to execute the following: The process of detecting unevenness of the road surface is started when the lane in which the vehicle is traveling is a lane at the edge of the road. program.

8. the plurality of categories includes a category of roads; In the process of detecting unevenness, unevenness present on the road surface within an image area classified into a road category is detected. The program according to claim 7.

Images