Road marking peeling rate estimation device, road marking peeling rate estimation system, road marking peeling rate estimation program, and method for creating a trained model.

Abstract

To provide a road marking peeling rate estimation device, a road marking peeling rate estimation system, a road marking peeling rate estimation program, and a method for creating a trained model that can estimate the peeling rate of road markings in a simple and highly accurate manner. [Solution] In a road marking peeling rate estimation system 1 in which a road marking peeling rate estimation device 10 is connected to a camera unit 20 installed on a vehicle via a communication network, the road marking peeling rate estimation device comprises a conversion unit that converts an image of the road surface captured into a bird's-eye view image, and an estimation unit that estimates the peeling rate of road markings included in the bird's-eye view image using a trained model. By combining estimation using a bird's-eye view image and estimation using a trained model, the peeling rate can be estimated more simply and with higher accuracy compared to conventional methods of identifying road markings by template matching, and the peeling rate of road markings can be estimated with higher accuracy in captured images of various road surfaces.

Description

Technical Field

[0001] The present disclosure relates to a road marking peeling rate estimation device, a road marking peeling rate estimation system, a road marking peeling rate estimation program, and a method for creating a learned model.

Background Art

[0002] Road markings such as lane lines, regulatory signs, and guiding signs are paved on the road surface. Due to the nature of being paved on the road surface, road markings are subject to peeling over time. Therefore, it is necessary to regularly inspect and maintain the condition of road markings.

[0003] Techniques for estimating the peeling degree of road markings from a photographed image of the road surface have been studied. For example, Patent Document 1 discloses a technique for determining the presence, type, and degree of blur of road markings reflected in an image for each section based on the photographed image. Further, Patent Document 2 discloses an estimation method and an estimation device capable of detecting the presence or absence of a white line included in a road surface image using machine learning.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, a method for easily and accurately estimating the peeling rate of road markings has not yet been established. Therefore, an object of the present invention is to provide a road marking peeling rate estimation device and the like that can easily and accurately estimate the peeling rate of road markings.

Means for Solving the Problems

[0006] One aspect of this disclosure provides an apparatus for estimating the rate of peeling of road markings, comprising: a conversion unit that converts an image of the road surface into a bird's-eye view image; and an estimation unit that estimates the rate of peeling of road markings included in the bird's-eye view image using a trained model.

[0007] This road marking peeling rate estimation device estimates the peeling rate of road markings using aerial images of the road surface, thus enabling highly accurate estimation of the peeling rate regardless of the road surface shooting conditions. Furthermore, because it estimates the peeling rate of road markings using a trained model, it can estimate the peeling rate of road markings more simply and accurately than conventional methods that identify road markings using template matching, etc. In particular, by combining estimation using aerial images with estimation using a trained model, the peeling rate of road markings can be estimated with high accuracy in various road surface images.

[0008] In the above-described road marking peeling rate estimation device, the estimation unit may identify an estimated portion in the bird's-eye view image that is estimated to be a road marking, and within the estimated portion, identify the remaining portion where the road marking remains and the peeling portion where the road marking has peeled off. According to this embodiment, since the estimation unit performs both the estimation of the road marking and the identification of the peeling portion in the estimated portion, it is possible to create a more robust road marking peeling rate estimation device that can be applied to various road surface images.

[0009] In the above-described road marking peeling rate estimation device, the estimation unit may estimate the ratio of the area occupied by the peeled portion to the estimation portion within a predetermined range of the estimation portion as the peeling rate within that range. Furthermore, in the above-described road marking peeling rate estimation device, the estimation unit may estimate the area of ​​the peeled portion in real space within a predetermined range of the estimation portion.

[0010] In the above-described road marking peeling rate estimation device, the estimation unit may extract estimated portions, generate an image containing the extracted multiple estimated portions, and use the generated image to identify the peeled portions. According to this embodiment, when estimating the peeling rate of road markings with a large aspect ratio, such as lane markings, the peeling rate can be estimated without resizing the image of the estimated portion. This makes the estimation of the peeling rate even simpler and more accurate.

[0011] The above-described road marking peeling rate estimation device further includes an output unit that outputs the estimation results from the estimation unit, and the output unit may output a peeling rate graph showing the peeling rate estimated by the estimation unit for each predetermined area of ​​the road surface. According to this embodiment, for example, the peeling rate of road markings can be estimated along the road.

[0012] In the above-described road marking peeling rate estimation device, the trained model is a model created using training data that includes (1) an image of the road surface and (2) training information that includes a combination of training information indicating the estimated portion of the image that is estimated to be a road marking and the peeling portion in the estimated portion where the road marking has peeled off. The training data may include, as the road surface image, a bird's-eye view image of the road surface and a modified image obtained by altering the color information of at least a portion of the road marking in the bird's-eye view image of the captured image. According to this embodiment, the peeling rate of road markings can be estimated with even higher accuracy in captured images of various road surfaces.

[0013] Another aspect of this disclosure provides a system for estimating the rate of peeling of road markings, comprising: a camera installed on a vehicle; a conversion unit that converts an image of the road surface captured by the camera into a bird's-eye view image; and an estimation unit that estimates the rate of peeling of road markings included in the bird's-eye view image using a trained model.

[0014] Another aspect of this disclosure is a program for estimating the rate of peeling of road markings, which causes a computer to function as a conversion unit that converts an image of a road surface into a bird's-eye view image, and an estimation unit that estimates the rate of peeling of road markings included in the bird's-eye view image using a trained model.

[0015] Another aspect of the present disclosure is that a computer obtains a combination of a bird's-eye view image of a captured image of a road surface and teacher information showing the bird's-eye view image and an estimated part estimated to be a road marking and a peeled part where the road marking is peeled off in the image, modifies the color information of at least a part of the road marking in the bird's-eye view image of the captured image to generate a modified image, associates the teacher information corresponding to the bird's-eye view image of the captured image used for generating the modified image as the teacher information of the modified image, and uses teacher data including a combination of the modified image and the teacher information associated therewith to create a learned model for estimating the peeling rate of the road marking from the bird's-eye view image of the captured image of the road surface.

[0016] According to these systems and programs, the peeling rate of the road marking can be estimated simply and with high accuracy. Further, according to the above creation method, the peeling rate of the road marking can be estimated simply and with high accuracy while using a small number of captured images of the road surface prepared as teacher data, for example.

Effect of the Invention

[0017] The present invention provides a road marking peeling rate estimation device and the like that can estimate the peeling rate of a road marking simply and with high accuracy.

Brief Description of the Drawings

[0018] [Figure 1] It is a schematic diagram of a road marking peeling rate estimation system according to the present embodiment. [Figure 2] It is a functional block diagram of a road marking peeling rate estimation device according to the present embodiment. [Figure 3] It is a diagram showing the physical configuration of a road marking peeling rate estimation device according to the present embodiment. [Figure 4] It is a conceptual diagram of an estimated part, a peeled part, and a remaining part on a road surface. [Figure 5] It is a flowchart of processing in a road marking peeling rate estimation device according to the present embodiment. [Figure 6]This is a schematic diagram of the conversion in the conversion unit of the road surface marking peeling rate estimation device according to this embodiment. [Figure 7] This is a flowchart of the processing in the estimation unit of the road surface marking peeling rate estimation device according to this embodiment. [Figure 8] This is a schematic diagram of the estimation in the estimation unit of the road surface marking peeling rate estimation device according to this embodiment. [Figure 9] This is a schematic diagram showing one process of the estimation in the estimation unit of the road surface marking peeling rate estimation device according to this embodiment. [Figure 10] This is a schematic diagram showing an example of the estimation result output by the output unit of the road surface marking peeling rate estimation device according to this embodiment.

Embodiments for Carrying Out the Invention

[0019] Embodiments of the present invention will be described with reference to the accompanying drawings. In each figure, those with the same reference numerals have the same or similar configurations.

[0020] [Road Surface Marking Peeling Rate Estimation System] FIG. 1 is a diagram showing a schematic configuration of a road surface marking peeling rate estimation system 1 according to this embodiment. The road surface marking peeling rate estimation system 1 includes a vehicle 30, a photographing unit 20 installed in the vehicle 30, and a road surface marking peeling rate estimation device 10. In the road surface marking peeling rate estimation system 1, the photographing unit 20 on the vehicle 30 photographs a moving image or a photograph of the road surface. The road surface marking peeling rate estimation device 10 estimates the peeling rate of the road surface marking based on the moving image or photograph photographed by the photographing unit 20. The road surface marking peeling rate estimation device 10 converts an image of the road surface photographed into an aerial view image, and estimates the peeling rate of the road surface marking included in the aerial view image using a learned model.

[0021] That is, the road surface marking peeling rate estimation system 1 includes a photographing unit 20 installed in the vehicle 30, and a road surface marking peeling rate estimation device 10 that converts an image of the road surface photographed by the photographing unit 20 into an aerial view image and estimates the peeling rate of the road surface marking included in the aerial view image using a learned model.

[0022] In this specification, "road markings" means markings applied to the road surface, including road markings (including regulatory and directional markings), lane markings, and non-statutory markings. Road markings are markings applied to the road surface with paint or the like, and may peel off (including fading and dirt) over time due to ultraviolet rays and physical wear from vehicles and pedestrians. The road marking peeling rate estimation system 1 can be used to estimate the peeling rate, which indicates the degree of such peeling, and to facilitate the management of road markings.

[0023] In the road marking peeling rate estimation system 1, the road marking peeling rate estimation device 10 is connected to the imaging unit 20 via a communication network N. Here, the communication network N may be a wired or wireless communication network. The road marking peeling rate estimation device 10 does not necessarily have to be a device independent of the imaging unit 20, and may be configured as an integral part of the imaging unit 20. In that case, the information processing terminal including the imaging unit 20 may function as the road marking peeling rate estimation device 10 by executing a road marking peeling rate estimation program installed on the information processing terminal.

[0024] The imaging unit 20 is not particularly limited as long as it is configured to capture video or images. The imaging unit 20 may be, for example, a camera, or a general-purpose information processing terminal such as a smartphone or tablet.

[0025] The imaging unit 20 is fixedly installed on the vehicle 30. That is, it moves in response to the movement of the vehicle 30. In the estimation by the road marking peeling rate estimation device 10, it is not necessary for the imaging unit 20 to be installed on the vehicle, but in this specification, the case in which the imaging unit 20 is installed on the vehicle 30 and moves with the movement of the vehicle 30 will be described. If the imaging unit 20 is not installed on the vehicle, it is preferable that the imaging unit 20 be held by a holder that has a mechanism that allows the imaging unit 20 to be moved while maintaining a constant height from the road surface. This can improve the accuracy of the estimation by the road marking peeling rate estimation device 10.

[0026] The camera unit 20 may be installed on any part of the vehicle 30 as long as it is in a position to photograph the road surface. For example, the camera unit 20 may be set in front of or behind the vehicle 30 in the direction of travel, and may be installed on the dashboard, windshield, rearview mirror, etc. of the vehicle 30 to photograph the road surface in front of the vehicle. The camera unit 20 may acquire images in front of or behind the vehicle 30 in the direction of travel. The camera unit 20 may periodically photograph the road surface at regular time intervals or at regular distance intervals.

[0027] The imaging unit 20 photographs the road surface, including road markings, which are the target for estimating the peeling rate, and acquires a video or image of the road surface. However, the imaging unit 20 may acquire a video or image of the road surface that does not include road markings. The imaging unit 20 may also acquire multiple images whose shooting ranges overlap. In this specification, unless otherwise specified, the term "image" encompasses both frames in a video and still images.

[0028] The camera unit 20 is installed on the vehicle 30, and the camera unit 20 photographs the road surface while the vehicle 30 is in motion. This allows for the acquisition of multiple images of road markings placed at different locations. By adjusting the vehicle 30's speed and the image acquisition interval of the camera unit 20 (which is the reciprocal of the frame rate when acquiring video, and the reciprocal of the continuous shooting speed when acquiring still images), the interval between the positions of the captured images can be arbitrarily changed.

[0029] The interval between positions for acquiring consecutive images may be, for example, several meters, 0.5 to 10 meters, or 1 to 5 meters. The road marking peeling rate estimation system 1 may acquire positional information of the vehicle 30 and the imaging unit 20 from an information processing terminal installed on the vehicle 30 based on the Global Positioning System (GPS) or Global Navigation Satellite System (GNSS), and based on this positional information, select images acquired at appropriate intervals from a plurality of images acquired by the imaging unit 20. The function of selecting such images may be provided by the information processing terminal installed on the vehicle 30, or by the road marking peeling rate estimation device 10. In this case, the information processing terminal installed on the vehicle 30 may function as the imaging unit 20.

[0030] Vehicle 30 may be an automobile that travels on the road surface with four wheels. However, vehicle 30 may also be a three-wheeled or two-wheeled vehicle, or a vehicle with five or more wheels. Vehicle 30 may be an automobile of any size.

[0031] (Road marking peeling rate estimation device) Figure 2 is a functional block diagram of the road marking peeling rate estimation device 10. The road marking peeling rate estimation device 10 comprises an acquisition unit 11, a conversion unit 12, an estimation unit 13, and an output unit 14. In this embodiment, the road marking peeling rate estimation device 10 is described using an example where the acquisition unit 11 acquires an image from the shooting unit 20 and the conversion unit 12 converts the image. However, for example, an information processing terminal having the function of the shooting unit 20 may also have the function of the conversion unit 12. In this case, the acquisition unit 11 of the road marking peeling rate estimation device 10 acquires a bird's-eye view image acquired by an information processing terminal including the shooting unit 20 and further converted to a bird's-eye view. Furthermore, as described above, if an information processing terminal including the shooting unit 20 has the function of the road marking peeling rate estimation device 10, the information processing terminal only needs to include the shooting unit 20, the conversion unit 12, the estimation unit 13, and the output unit 14. Furthermore, the output unit 14 may be omitted, may be provided in an information processing terminal including the imaging unit 20, or may be provided in an output device connected to the road marking peeling rate estimation device 10.

[0032] The acquisition unit 11 acquires one or more images obtained by the shooting unit 20 by shooting the road surface. The acquisition unit 11 may be implemented by the communication unit 10d described later. The acquisition unit 11 may acquire all of the multiple images taken by the shooting unit 20, or it may acquire only some of the images.

[0033] The acquisition unit 11 may acquire multiple images of the road surface on which the vehicle 30 is traveling, taken at regular intervals. Consecutive images may include the same location on the road surface multiple times.

[0034] The conversion unit 12 converts the image acquired by the acquisition unit 11 into a bird's-eye view image. The conversion unit 12 may be implemented by an image conversion program stored in RAM 10b or ROM 10c (described later) and executed by CPU 10a. The conversion unit 12 may convert all of the multiple images acquired by the acquisition unit 11, or it may convert only some of the images.

[0035] The camera unit 20 is installed on the vehicle 30 and photographs the road surface. Therefore, the image captured by the camera unit 20 and acquired by the acquisition unit 11 is an image of the road surface taken from diagonally above. The conversion unit 12 converts the image taken from diagonally above the road surface into an image that looks as if the road surface were photographed from directly above. The conversion means is not particularly limited, but specific examples will be described later.

[0036] The conversion unit 12 not only converts the image acquired by the acquisition unit 11 into a bird's-eye view image, but may also perform conversions as appropriate to improve the accuracy of the estimation by the estimation unit 13. Examples of such conversions include cropping and resizing.

[0037] The estimation unit 13 estimates the rate of peeling of road markings included in the bird's-eye view image converted by the conversion unit 12 using a trained model. Since the estimation unit 13 uses the bird's-eye view image of the captured road surface to estimate the rate of peeling of road markings, it can estimate the rate of peeling of road markings with high accuracy regardless of the road surface shooting conditions. Furthermore, because it estimates the rate of peeling of road markings using a trained model, it can estimate the rate of peeling of road markings in a simpler and more accurate way compared to conventional methods that identify road signs by template matching, etc. In particular, by combining estimation using bird's-eye view images with estimation using a trained model, it is possible to estimate the rate of peeling of road markings with high accuracy in captured images of various road surfaces.

[0038] As shown in Figure 4, the estimation unit 13 preferably identifies the estimated portion in the bird's-eye view image that is estimated to be a road marking, and then identifies the remaining portion where the road marking remains and the peeling portion where the road marking has peeled off within the estimated portion. In this embodiment, since the estimation unit 13 performs both the estimation of the road marking and the identification of the peeling portion in the estimated portion, it is possible to perform a more robust estimation that can be applied to various road surface images. In this embodiment, the trained model for identifying the estimated portion and the trained model for identifying the remaining portion and the peeling portion may be the same model, but it is preferable to use different models. By using different models for identifying the road marking and identifying the remaining portion and the peeling portion in this way, the peeling rate can be estimated with greater robustness and accuracy. The trained models will be described later.

[0039] The estimation unit 13 estimates the rate of peeling of road markings. For example, the estimation unit 13 may estimate the peeling rate of the entire road marking included in one image, the peeling rate of road markings included across multiple images, or the peeling rate of a portion of the road markings included in one or more images. Alternatively, the estimation unit 13 may estimate the peeling rate of road markings for each predetermined area of ​​the road surface. For example, when a vehicle 30 travels on a road and the camera 20 takes images of the road surface at regular intervals, the peeling rate of road markings included in the road traveled by the vehicle 30 may be estimated for each section.

[0040] The estimation unit 13 may estimate the ratio of the area occupied by the peeled portion to the estimation portion within a predetermined range of the estimation portion as the peeling rate within that range. The estimation unit 13 may also estimate the area of ​​the peeled portion in real space within a predetermined range of the estimation portion.

[0041] The trained model is a machine learning model trained on training data that takes a bird's-eye view image as input, recognizes road markings contained in the image, and outputs the peeling rate of those road markings.

[0042] The machine learning models used are not particularly limited as long as they are capable of recognizing and / or taking images as input. Examples include CNN (Convolutional Neural Network) based models, transformer-based models, classifier-based models such as SVM (Support Vector Machine), and models that combine these.

[0043] In particular, to estimate the delamination rate with high robustness and accuracy, it is preferable to use a model that can generate boundaries in complex environments when identifying the estimated portion. Examples of such models include, but are not limited to, a model that uses the architecture of ResNet50 (He, K., et al.: Deep residual learning for image recognition, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.) as the encoder and DeepLabV3+ (Chen, LC, et al.: Encoder-decoder with atrous separable convolution for semantic image segmentation, The European Conference on Computer Vision (ECCV), pp. 801-818, 2018.) as the base decoder for semantic segmentation. Furthermore, when identifying the residual and delamination portions, it is preferable to use a model that has high classification accuracy in fine regions.Examples of such models include, but are not limited to, models that use the ResNet18 architecture (He, K., et al.: Deep residual learning for image recognition, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.) as the encoder and Unet++ (Zhou, Z., et al.: Unet++: A nested u-net architecture for medical image segmentation, Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, Vol. 11045, pp. 3-11, 2018.) as the base semantic segmentation model.

[0044] The model used by the estimation unit 13 is preferably a model trained on training data that includes images of the road surface, including road markings. The training data preferably includes images of various road surfaces, for example, multiple images in which the movement of the vehicle 30 at the time of shooting, the type of road, the positioning angle of the shooting unit 20, the weather conditions (rain or snow), and the state and type of road markings have been changed.

[0045] The road surface image in the training data is accompanied by training information regarding the estimated portion, the remaining portion, and / or the peeled portion. In this embodiment, the training data includes (1) an image of the road surface, and (2) a combination of training information indicating the estimated portion in the image that is estimated to be a road marking and the peeled portion in the estimated portion where the road marking has peeled off. For example, training information regarding the estimated portion may be added by enclosing a specific area in the image with a polygon.

[0046] For example, training information regarding remaining and peeling portions may be attached only to the areas identified as estimated portions. According to this embodiment, since remaining and peeling portions can be classified by focusing only on the estimated portions, the remaining and peeling portions can be identified with high accuracy. Training information regarding remaining and peeling portions may be attached by binarizing the estimated portions. If factors (shadows, obstacles, color differences, etc.) exist in the extracted estimated portions that hinder accurate binarization, the estimated portions may be divided into multiple areas to improve the accuracy of binarization, and then binarization may be performed on each area. According to this embodiment, training information regarding remaining and peeling portions can be attached with extremely high accuracy compared to simply binarizing the road markings. Note that the division of the estimated portions and the setting of thresholds for binarization are performed by the creator of the training data.

[0047] To increase the number of images with accurately assigned training information regarding estimated, remaining, and detached portions, data augmentation techniques may be applied to existing images with assigned training information. Examples of data augmentation techniques, though not limited to those mentioned above, include cropping (partial trimming), horizontal inversion, vertical inversion, and modification of color information for at least a portion of road markings. Since images that have undergone these transformations already have assigned training information, there is no need to assign new training information to them.

[0048] In particular, the training data preferably includes, as images of the road surface, a bird's-eye view image of the road surface and a modified image obtained by altering the color information of at least a portion of the road markings in the bird's-eye view image of the road surface. The modified image may have its color information altered so that the color of a predetermined area of ​​the road surface image that includes the road markings becomes darker (lower in brightness). According to this embodiment, the peeling rate can be estimated with even greater accuracy even for images of the road surface in which shadows or obstacles are reflected on the road markings.

[0049] In other words, a trained model may be created by a method that includes the following steps: the computer obtains a bird's-eye view image of a captured road surface and training information indicating estimated portions of the image that are estimated to be road markings and portions where the road markings have peeled off; modifies the color information of at least a portion of the road markings in the bird's-eye view image of the captured road surface to generate a modified image; associates the training information corresponding to the bird's-eye view image of the captured road surface used to generate the modified image with the training information of the modified image; and creates a trained model that estimates the rate of peeling of road markings from a bird's-eye view image of the captured road surface using training data that includes the combination of the modified image and the training information associated with it. The methods described above may be used for each processing step. A trained model can be created by performing machine learning using training data while referring to various known methods.

[0050] The data augmentation technique may be applied to the images included in the training data with random probability, and may be performed in combination of two or more modifications.

[0051] The estimation unit 13 may be implemented by a program stored in the RAM 10b or ROM 10c (described later) and executed by the CPU 10a.

[0052] The output unit 14 outputs the estimation result from the estimation unit 13 in an appropriate format. The output unit 14 may display the estimation result on the display unit 10f (described later), or it may transmit the estimation result to another terminal via the communication unit 10d. The estimation result may be a three-dimensional or two-dimensional graph, or it may be numerical data. The output unit 14 may output a peeling rate graph showing the peeling rate estimated by the estimation unit 13.

[0053] Figure 3 shows the physical configuration of the road marking peeling rate estimation device 10. The road marking peeling rate estimation device 10 includes a CPU (Central Processing Unit) 10a, which corresponds to a processor, a RAM (Random Access Memory) 10b and a ROM (Read-only Memory) 10c, which correspond to storage units, a communication unit 10d, an input unit 10e, and a display unit 10f. These components are connected to each other via a bus so that data can be sent and received from each other. In this example, the case in which the road marking peeling rate estimation device 10 is composed of a single computer is described, but the road marking peeling rate estimation device 10 may be realized by combining multiple computers. Also, the configuration shown in Figure 3 is just one example, and the road marking peeling rate estimation device 10 may have other configurations, or may not have some of these configurations.

[0054] The CPU 10a is a control unit that performs control, calculations, and processing of data related to the execution of programs stored in the RAM 10b or ROM 10c. The CPU 10a is a calculation unit that executes a program (road marking peeling rate estimation program) that estimates the rate of peeling of road markings based on images acquired by the imaging unit 20. The CPU 10a receives various data from the input unit 10e and the communication unit 10d, and displays the calculation results of the data on the display unit 10f or stores them in the RAM 10b or ROM 10c.

[0055] RAM10b is a data-rewritable memory unit and may be composed of, for example, semiconductor memory elements. RAM10b may store the road marking peeling rate estimation program executed by CPU10a. Note that these are examples, and RAM10b may store other data.

[0056] ROM10c is a data readable portion of the storage unit and may be composed of, for example, semiconductor memory elements. ROM10c may store, for example, an image editing program or data that is not rewritten.

[0057] The communication unit 10d is an interface for connecting the road marking peeling rate estimation device 10 to other devices. The communication unit 10d may be connected to a communication network N such as the Internet.

[0058] The input unit 10e accepts data input from the user and may include, for example, a keyboard and a touch panel.

[0059] The display unit 10f visually displays the calculation results from the CPU 10a and may be configured as, for example, an LCD (Liquid Crystal Display). The display unit 10f may display a graph showing the estimated rate of peeling of road markings.

[0060] The road marking peeling rate estimation program may be stored and provided on a computer-readable storage medium such as RAM 10b or ROM 10c, or it may be provided via a communication network connected by the communication unit 10d. In the road marking peeling rate estimation device 10, the CPU 10a executes the road marking peeling rate estimation program, thereby realizing the operation of the acquisition unit 11, conversion unit 12, estimation unit 13, and output unit 14 as described with reference to Figure 2. Note that these physical configurations are illustrative and do not necessarily have to be independent. For example, the road marking peeling rate estimation device 10 may include an LSI (Large-Scale Integration) in which the CPU 10a and RAM 10b and / or ROM 10c are integrated.

[0061] (Processing flow) Figure 5 is a flowchart of the processing in the road marking peeling rate estimation device 10 according to this embodiment. First, in the road marking peeling rate estimation device 10, the acquisition unit 11 acquires an image of the road surface taken from the shooting unit 20 (S10). Here, the acquisition unit 11 may acquire one or more images.

[0062] Next, the road marking peeling rate estimation device 10 converts the image into a bird's-eye view image in the conversion unit 12 (S11). The conversion unit 12 performs the bird's-eye view conversion according to, for example, the following formula. The following formula converts an image acquired by the shooting unit 20 from diagonally above the road surface into an image acquired from a virtual camera installed directly above the road surface.

[0063]

number

[0064] Here, x and y are the pixel positions in the image before conversion, and x' and y' are the pixel positions in the image after conversion. Also, f and f' are the pixel-equivalent focal lengths of the imaging unit 20 and the virtual camera, respectively, and θ is the distance between the imaging unit 20 and the road surface, H VC This is the height of the virtual camera from the road surface, and D VC H is the parallel distance between the shooting unit 20 and the virtual camera. C This is the height of the camera unit 20 from the road surface. By adjusting various parameters in the virtual camera, the bird's-eye view image can be calibrated. For more details, see S. Tanaka et al., International Journal of Vehicular Technology, Volume 2011, Article ID 279739 (2011).

[0065] In the conversion unit 12, the various parameters in the virtual camera may be set by the user of the road marking peeling rate estimation device 10, or by the manufacturer of the road marking peeling rate estimation device 10. Alternatively, the road marking peeling rate estimation device 10 may automatically set the parameters by an algorithm designed to improve the accuracy of road marking peeling rate estimation while adjusting the various parameters in the virtual camera. Alternatively, the various parameters may be adjusted manually or automatically so that known shapes such as lanes and manholes on the road surface become those known shapes in the converted bird's-eye view image. By adjusting the various parameters, the conversion unit 12 may make each pixel in the bird's-eye view image appear as a square with physically equal width and height. In this case, the shooting interval of the shooting unit 20 and the position information of the shooting unit 20 obtained by GPS or GNSS may be used to make the width and height of each pixel correspond to the physical length in real space.

[0066] The conversion unit 12 may further convert the converted bird's-eye view image. Such conversions include cropping to cut out a portion of the bird's-eye view image and resizing to change the size of the bird's-eye view image. In particular, due to the effects of the bird's-eye view conversion, the upper part of the converted photograph tends to become relatively unclear, and irrelevant black space is displayed at the bottom. Therefore, it is preferable to crop the top, left, and right edges of the bird's-eye view image. The left and right edges may be cropped by, for example, 1 / 4 of the width of the bird's-eye view image, and the top edge may be cropped by, for example, 1 / 3 of the height of the bird's-eye view image.

[0067] Figure 6 shows an example of an image acquired in step S10 and transformed in step S11. The top image in Figure 6 is an image of the road surface acquired in step S10, the middle image is a bird's-eye view image obtained by transforming the said image, and the bottom image is an image with the left, right, and top edges of the bird's-eye view image cropped. Below, we will describe an example of processing using the bird's-eye view transformed image with the left, right, and top edges cropped. However, cropping may be omitted.

[0068] Returning to Figure 5, following step S11, the estimation unit 13 of the road marking peeling rate estimation device 10 estimates the peeling rate of road markings included in the bird's-eye view image using a trained model (S12). The estimation by the estimation unit 13 may be carried out as described above, but below we will explain the case where the estimation is carried out using the flowchart shown in Figure 7.

[0069] Figure 7 is a flowchart of the estimation process in the estimation unit 13 of the road marking peeling rate estimation device 10 according to this embodiment. First, the estimation unit 13 identifies the estimated portion that is estimated to be a road marking in the bird's-eye view image (S121). The estimation unit 13 preferably uses a trained model that has been tuned for identifying the estimated portion as described above. The estimation unit 13 may estimate the estimated portion as a predetermined shape. The predetermined shape may be a circle, ellipse, polygon, or other shape, or it may be a character. The estimation unit 13 may select the shape of the estimated portion from a set of shapes or characters in advance. In step S121, an image may be generated in which the estimated portion in the bird's-eye view image is highlighted so that it can be identified, or an extracted image may be generated in which only the estimated portion is extracted.

[0070] Next, the estimation unit 13 identifies (classifies) the remaining portion of the road markings and the peeling portion of the road markings in the estimation area (S122). The estimation unit 13 preferably uses a trained model that has been tuned for identifying the remaining portion and the peeling portion as described above. In step S122, an image may be generated in which the remaining portion and the peeling portion in the estimation area of ​​the bird's-eye view image are highlighted so that they can be identified, and information indicating whether each pixel in the estimation area is a remaining portion or a peeling portion may be attached. The processing in steps 121 and 122 will be explained with reference to Figure 8.

[0071] Figure 8 shows an image (a) in which the estimated portion of the bird's-eye view image is highlighted for identification, and an image (b) in which the remaining and detached portions of the estimated portion are highlighted for identification. In Figure 8(a), the portion of the bird's-eye view image that is estimated to be a road marking is highlighted with a color not normally used for road markings (e.g., green) so that it can be identified. In Figure 8(b), the remaining and detached portions of the road marking are highlighted with a color different from the color of the road marking and the road surface (e.g., red) so that they can be identified. Note that Figure 8(b) is an image extracted from the central region of Figure 8(a), which is 1 / 3 of the height.

[0072] In steps 121 and 122, the estimation unit 13 may extract estimated portions, generate an image containing the extracted estimated portions, and use the generated image to identify the detached portions. This process will be explained with reference to Figure 9.

[0073] In this process, as shown in the left part of Figure 9, in step 121, the estimation unit 13 extracts only the estimated portion that is estimated to be a road marking in the bird's-eye view image. Then, in step 121, as shown in the right part of Figure 9, an image containing the extracted estimated portion is generated. In this embodiment, the remaining portion and the peeled portion are identified in step 122 using the image containing the multiple estimated portions generated in this process. After the identification of the remaining portion and the peeled portion is completed, the image is divided again into each estimated portion to obtain the estimated portion in which the remaining portion and the peeled portion have been identified. According to this embodiment, when estimating the peeling rate of road markings with a large aspect ratio, such as lane markings, the peeling rate can be estimated without resizing the image of the estimated portion. This makes the estimation of the peeling rate even simpler and more accurate.

[0074] Returning to Figure 7, following step S122, the estimation unit 13 calculates the peeling rate in the estimated portion from the identified remaining portion and peeled portion (S123). For example, the estimation unit 13 may estimate the peeling rate as the ratio of the area occupied by the peeled portion to the estimated portion within a predetermined range of the estimated portion. For example, if the area of ​​the estimated portion is X pixels and the area of ​​the peeled portion is Y pixels, the peeling rate may be calculated as Y / X × 100%.

[0075] Through the processing flow described above, the road marking peeling rate estimation device 10 according to this embodiment can estimate the peeling rate of road markings. However, the road marking peeling rate estimation device 10 may also estimate the area of ​​the peeled portion in real space using the shooting interval of the shooting unit 20 or the position information of the shooting unit 20 acquired by GPS or GNSS. Alternatively, the area of ​​the peeled portion in real space may be estimated using the size of an object with known dimensions, such as a manhole, as reference data.

[0076] In this embodiment, the road marking peeling rate estimation device 10 may output the estimation results from the estimation unit 13 via the output unit 14. An example of the output estimation results is shown in Figure 10. For example, by acquiring images of the road surface at regular intervals while a vehicle 30 is driving on the road, a peeling rate graph showing the peeling rate estimated by the estimation unit 13 for each predetermined area of ​​the road surface can be output, as shown in Figure 10. Using such a peeling rate graph makes it easy to determine at which location the road markings should be repaired. Alternatively, the output unit 14 may indicate the degree of peeling in several stages (for example, stages 1 to 5) based on the estimated peeling rate. By outputting the degree of peeling, it can be used as an indicator of whether or not the road markings need to be repaired.

[0077] The embodiments described above are provided to facilitate understanding of the present invention and are not intended to limit its interpretation. The elements, their arrangement, and conditions of the embodiments are not limited to those exemplified and can be modified as appropriate. Furthermore, it is possible to partially substitute or combine the configurations shown in different embodiments.

[0078] [Note] This disclosure includes the following embodiments. [1] A conversion unit that converts images of the road surface into bird's-eye view images, An estimation unit that estimates the rate of peeling of road markings included in the bird's-eye view image using a trained model, Equipped with, A device for estimating the rate of peeling of road markings. [2] The estimation unit, In the aforementioned bird's-eye view image, we identify the portion that is presumed to be a road marking, In the aforementioned estimated portion, the remaining portion where the road marking remains and the peeled portion where the road marking has peeled off are identified. The apparatus described in [1]. [3] The estimation unit estimates the ratio of the area occupied by the peeled portion to the estimated portion within a predetermined range of the estimation portion as the peeling rate within that range. The apparatus described in [2]. [4] The estimation unit estimates the area of ​​the peeled portion in real space within a predetermined range of the estimation portion. The apparatus described in [2] or [3]. [5] The estimation unit, Extracting the aforementioned estimated portion, An image is generated that includes the extracted multiple estimated portions. The generated image is used to identify the peeled portion. The device described in any one of [2] to [4]. [6] The system further includes an output unit that outputs the estimation results from the estimation unit, The output unit outputs a peeling rate graph showing the peeling rate estimated by the estimation unit for each predetermined area of ​​the road surface. The device described in any one of [1] to [5]. [7] The aforementioned trained model is a model created using training data that includes (1) an image of the road surface, and (2) training information that includes a combination of training information indicating estimated portions in the image that are estimated to be road markings and peeled portions in the estimated portions where the road markings have peeled off. The training data includes, as the image of the road surface, a bird's-eye view image of the road surface and a modified image obtained by altering the color information of at least a portion of the road surface markings in the bird's-eye view image of the road surface. The device described in any one of [1] to [6]. [8] The camera unit installed in the vehicle, A conversion unit that converts the road surface image captured by the aforementioned shooting unit into a bird's-eye view image, An estimation unit that estimates the rate of peeling of road markings included in the bird's-eye view image using a trained model, Equipped with, A system for estimating the rate of peeling of road markings. [9] Computers, A conversion unit that converts images of the road surface into bird's-eye view images, An estimation unit that estimates the rate of peeling of road markings included in the bird's-eye view image using a trained model, To make it function as A program to estimate the rate of peeling of road markings.

[10] Computers The process involves obtaining a bird's-eye view image of the road surface, and a combination of training information that shows the estimated portion of the image that is presumed to be a road marking and the peeled portion of the estimated portion where the road marking has peeled off. The process involves modifying the color information of at least a portion of the road markings in the bird's-eye view image of the aforementioned captured image to generate a modified image, The training information corresponding to the bird's-eye view image of the captured image used to generate the modified image is linked as the training information for the modified image, Using training data that includes the combination of the modified image and the training information linked thereto, a trained model is created to estimate the rate of peeling of road markings from a bird's-eye view image of the road surface. A method for creating a pre-trained model, including the execution of [this process]. [Explanation of Symbols]

[0079] 1...Road marking peeling rate estimation system, 10...Road marking peeling rate estimation device, 10a...CPU, 10b...RAM, 10c...ROM, 10d...Communication unit, 10e...Input unit, 10f...Display unit, 11...Acquisition unit, 12...Conversion unit, 13...Estimation unit, 14...Output unit, 20...Photography unit, 30...Vehicle, N...Communication network.

Claims

1. A conversion unit that converts images of the road surface into bird's-eye view images, An estimation unit that estimates the rate of peeling of road markings included in the bird's-eye view image using a trained model, Equipped with, A device for estimating the rate of peeling of road markings.

2. The estimation unit, In the aforementioned bird's-eye view image, we identify the portion that is presumed to be a road marking, In the aforementioned estimated portion, the remaining portion where the road marking remains and the peeled portion where the road marking has peeled off are identified. The apparatus according to claim 1.

3. The estimation unit estimates the ratio of the area occupied by the peeled portion to the estimated portion within a predetermined range of the estimation portion as the peeling rate within that range. The apparatus according to claim 2.

4. The estimation unit estimates the area of ​​the peeled portion in real space within a predetermined range of the estimation portion. The apparatus according to claim 2.

5. The estimation unit, Extracting the aforementioned estimated portion, An image is generated that includes the extracted multiple estimated portions. The generated image is used to identify the peeled portion. The apparatus according to claim 2.

6. The system further includes an output unit that outputs the estimation results from the estimation unit, The output unit outputs a peeling rate graph showing the peeling rate estimated by the estimation unit for each predetermined area of ​​the road surface. The apparatus according to any one of claims 1 to 5.

7. The aforementioned trained model is a model created using training data that includes (1) an image of the road surface, and (2) training information that includes a combination of training information indicating estimated portions in the image that are estimated to be road markings and peeled portions in the estimated portions where the road markings have peeled off. The training data includes, as the image of the road surface, a bird's-eye view image of the road surface and a modified image obtained by altering the color information of at least a portion of the road surface markings in the bird's-eye view image of the road surface. The apparatus according to any one of claims 1 to 5.

8. The camera unit installed in the vehicle, A conversion unit that converts the road surface image captured by the aforementioned shooting unit into a bird's-eye view image, An estimation unit that estimates the rate of peeling of road markings included in the bird's-eye view image using a trained model, Equipped with, A system for estimating the rate of peeling of road markings.

9. Computers, A conversion unit that converts images of the road surface into bird's-eye view images, An estimation unit that estimates the rate of peeling of road markings included in the bird's-eye view image using a trained model, To make it function as A program to estimate the rate of peeling of road markings.

10. Computers The process involves obtaining a bird's-eye view image of the road surface, and a combination of training information that shows the estimated portion of the image that is presumed to be a road marking and the peeled portion of the estimated portion where the road marking has peeled off. The process involves modifying the color information of at least a portion of the road markings in the bird's-eye view image of the aforementioned captured image to generate a modified image, The training information corresponding to the bird's-eye view image of the captured image used to generate the modified image is linked as the training information for the modified image, Using training data that includes the combination of the modified image and the training information linked thereto, a trained model is created to estimate the rate of peeling of road markings from a bird's-eye view image of the road surface. A method for creating a pre-trained model, including the execution of [this process].

Images