Arithmetic unit, arithmetic method and program

Abstract

To appropriately estimate a state of a road surface.SOLUTION: An arithmetic unit 14 includes: an image acquisition section 30 for acquiring a picked-up image of a road picked up by a camera provided in a vehicle traveling on the road; an unevenness information acquisition section 32 for acquiring unevenness information, which is detected by a sensor provided in the vehicle, indicating an uneven state of the road; a position calculation section 34 for calculating position information of a region of interest, which is shot in the picked-up image, on the road; an unevenness information extraction section 36 for extracting unevenness information in the region of interest from the unevenness information; and a processing section 38 which executes processing for making image data in the region of interest in the picked-up image correspondent to the unevenness information in the region of interest.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a computing device, a computing method, and a program. [Background technology]

[0002] For example, as disclosed in Patent Document 1, a technique is known in which the acceleration of a vehicle traveling on a road is detected and the acceleration data is input into a learning model to estimate the state of the road surface. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-86960 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, it is also conceivable to estimate the road surface condition from captured images, road surface unevenness information, etc. However, while captured images can capture and record a wide area of ​​the road surface within the angle of view, the information is limited to image data, so the road surface condition may not be calculated with high accuracy. On the other hand, unevenness information can calculate the road surface condition with high accuracy, but the measurement range may be narrow. Therefore, when estimating the road surface condition while a vehicle is running, it is necessary to appropriately estimate the road surface condition.

[0005] The present invention has been made in view of the above, and has an object to provide a calculation device, a calculation method, and a program that are capable of appropriately estimating the state of a road surface. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objectives, the computing device of the present disclosure includes an image acquisition unit that acquires an image of the road captured by a camera mounted on a vehicle traveling on the road, an unevenness information acquisition unit that acquires unevenness information indicating the unevenness state of the road detected by a sensor mounted on the vehicle, a position calculation unit that calculates position information of a region of interest on the road that appears in the captured image, an unevenness information extraction unit that extracts the unevenness information in the region of interest from the unevenness information based on the position information of the region of interest, and a processing unit that executes processing to associate image data of the region of interest in the captured image with the unevenness information in the region of interest.

[0007] In order to solve the above-mentioned problems and achieve the objectives, the calculation method of the present disclosure includes the steps of acquiring an image of a road captured by a camera mounted on a vehicle traveling on the road, acquiring unevenness information indicating the unevenness state of the road detected by a sensor mounted on the vehicle, calculating position information of a region of interest on the road shown in the image, extracting the unevenness information in the region of interest from the unevenness information based on the position information of the region of interest, and performing a process of matching image data of the region of interest in the image captured image with the unevenness information in the region of interest.

[0008] In order to solve the above-mentioned problems and achieve the objectives, the program of the present disclosure causes a computer to execute the following steps: acquiring an image of a road captured by a camera mounted on a vehicle traveling on the road; acquiring unevenness information indicating the unevenness state of the road detected by a sensor mounted on the vehicle; calculating position information of a region of interest on the road shown in the image; extracting the unevenness information in the region of interest from the unevenness information based on the position information of the region of interest; and performing a process to associate image data of the region of interest in the image captured with the unevenness information in the region of interest. [Effects of the Invention]

[0009] According to the present invention, the road surface condition can be appropriately estimated. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic block diagram of a detection system according to this embodiment. [Figure 2] FIG. 2 is a schematic diagram of a vehicle. [Figure 3] FIG. 3 is a schematic block diagram of the arithmetic unit. [Figure 4] FIG. 4 is a schematic diagram showing an example of a captured image. [Figure 5] FIG. 5 is a schematic diagram showing an example of concave-convex information. [Figure 6] FIG. 6 is a schematic diagram for explaining extraction of concave / convex information in a region of interest. [Figure 7] FIG. 7 is a flowchart illustrating the processing flow of the arithmetic device according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0012] (Detection System) FIG. 1 is a schematic block diagram of a detection system according to this embodiment. As shown in FIG. 1, the detection system 1 according to this embodiment includes a vehicle 10, a measurement data acquisition device 12, and a calculation device 14. The detection system 1 calculates the road surface condition based on the behavior information using the calculation device 14. In this embodiment, the road surface condition is an index indicating the degree of road surface roughness. More specifically, in this embodiment, the road surface condition is the International Roughness Index (IRI). However, the road surface condition is not limited to the IRI and may be any index indicating the road surface condition. For example, the road surface condition may be at least one of the IRI, road surface flatness, cracks, rutting, and the Maintenance Control Index (MCI).

[0013] In the detection system 1, the vehicle 10 detects unevenness information and position information while traveling on a road, captures images of the road, and transmits the detected unevenness information, position information, and captured images to the measurement data acquisition device 12. The unevenness information, position information, and captured images will be described later. The measurement data acquisition device 12 is, for example, a device (computer) managed by an entity that manages the road. The measurement data acquisition device 12 transmits the unevenness information, position information, and captured images transmitted from the vehicle 10 to the calculation device 14. In this way, the calculation device 14 acquires the unevenness information, position information, and captured images via the measurement data acquisition device 12, but this is not limited to this. For example, the detection system 1 may not be provided with the measurement data acquisition device 12, and the calculation device 14 may acquire the unevenness information, position information, and captured images from the vehicle 10.

[0014] (vehicle) FIG. 2 is a schematic diagram of a vehicle. As shown in FIG. 2, the vehicle 10 includes a position sensor 10A, a camera 10B, a sensor 10C, and a measuring device 10D. The position sensor 10A is a sensor that acquires position information of the vehicle 10. The position information of the vehicle 10 is information indicating the position of the vehicle 10 in a predetermined coordinate system that can define the positions of the vehicle 10 and the road. The predetermined coordinate system here may be any coordinate system that can define the position of the vehicle 10 and the road, and in this embodiment, for example, may be the Earth coordinate system. In this case, the position sensor 10A is a module for a Global Navigation Satellite System (GNSS). Note that the Z direction in FIG. 2 indicates the vertical upward direction, and FIG. 2 can be said to be a schematic diagram of the vehicle 10 viewed from vertically above.

[0015] Camera 10B is a camera that captures images of the surroundings of vehicle 10. More specifically, in this embodiment, camera 10B captures images of the road on which vehicle 10 is traveling. Hereinafter, an image of the road on which vehicle 10 is traveling, captured by camera 10B, will be referred to as a captured image, as appropriate.

[0016] The sensor 10C is a sensor that detects unevenness information. The unevenness information is data that indicates the unevenness of the road on which the vehicle 10 is traveling, but is different from the image captured by the camera 10B. Furthermore, the unevenness information is data from which the above-mentioned road surface conditions (IRI, etc.) can be calculated, or in other words, it can be said to be data that is the basis for calculating the road surface conditions.

[0017] In this embodiment, the unevenness information may refer to behavior information that indicates the behavior of the vehicle 10. Because the behavior of the vehicle 10 changes depending on the unevenness of the road, the behavior information can be data that indicates the unevenness. The behavior information may be any information that indicates the behavior of the vehicle 10 while traveling on a road. However, for example, it is preferable that the sensor 10C detects the acceleration of the vehicle 10 as the behavior information. In this case, the sensor 10C is an acceleration sensor that detects acceleration, and more preferably an acceleration sensor that detects acceleration in three axes. Furthermore, the behavior information detected by the sensor 10C is not limited to acceleration, and may be, for example, at least one of acceleration, speed of the vehicle 10, angular velocity of the vehicle 10, steering angle of the vehicle 10, braking amount of the vehicle 10, operation of the wipers of the vehicle 10, and operation amount of the suspension of the vehicle 10. The sensor 10C that detects the speed of the vehicle 10 is, for example, a speed sensor, the sensor 10C that detects the speed of the vehicle 10 is, for example, a three-axis gyro sensor, the sensor 10C that detects the steering angle of the vehicle 10 is, for example, a steering sensor, the sensor 10C that detects the amount of braking of the vehicle 10 is, for example, a brake sensor, the sensor 10C that detects the operation of the wipers of the vehicle 10 is, for example, a wiper sensor, and the sensor 10C that detects the amount of operation of the suspension of the vehicle 10 is, for example, a suspension sensor.

[0018] The unevenness information is not limited to behavior information, and may be, for example, distance information indicating the distance from the sensor 10C to each position on the road surface. In other words, the unevenness information may be at least one of behavior information and distance information. The distance from the sensor 10C to each position on the road surface changes depending on the unevenness of the road, so the distance information can be said to be data indicating the unevenness. In this case, the sensor 10C is a distance measuring sensor that measures distance, and may use, for example, a LiDAR (Light Detection and Ranging). In this case, the sensor 10C irradiates laser light to each position on the road on which the vehicle 10 is traveling, and detects the distance to each position on the road as distance information based on the light reflected from the laser light.

[0019] The number of sensors 10C and the locations at which they are provided may be arbitrary.

[0020] The measuring device 10D controls the position sensor 10A, the camera 10B, and the sensor 10C to capture images while detecting position information and unevenness information, and records the position information, unevenness information, and captured images. In other words, the measuring device 10D functions as a data logger that records the position information, unevenness information, and captured images. The measuring device 10D can also be considered a computer and includes a control unit 10D1, a storage unit 10D2, and a communication unit 10D3. The control unit 10D1 is a calculation device and includes a calculation circuit such as a central processing unit (CPU). The storage unit 10D2 is a memory that stores various information such as the calculation contents and programs of the control unit 10D1, and the position information and behavior information of the vehicle 10. For example, the storage unit 10D2 includes at least one of a main storage device such as a random access memory (RAM), a read-only memory (ROM), and a non-volatile storage device such as a flash memory or a hard disk drive (HDD). The program for control unit 10D1 saved in storage unit 10D2 may be stored in a recording medium readable by measurement device 10D. Communication unit 10D3 is a communication module that communicates with an external device, such as an antenna.

[0021] The control unit 10D1 reads a program stored in the memory unit 10D2 and controls the position sensor 10A, the camera 10B, and the sensor 10C. While the vehicle 10 is traveling on a road, the control unit 10D1 causes the camera 10B to capture an image of the road at predetermined intervals and acquires the captured image. Similarly, while the vehicle 10 is traveling on the road, the control unit 10D1 causes the sensor 10C to detect unevenness information at predetermined intervals and acquires the unevenness information. The control unit 10D1 causes the position sensor 10A to detect the position information of the vehicle 10 at predetermined intervals and acquires the position information. That is, each time the vehicle 10 travels for a predetermined period of time, the control unit 10D1 causes the camera 10B to capture an image, the sensor 10C to detect unevenness information, and the position sensor 10A to detect the position information. Here, the predetermined period is preferably a fixed period of time, such as one minute, but the predetermined period is not limited to a fixed period of time and may be any length. That is, the predetermined period of time may change each time.

[0022] The control unit 10C1 associates the acquired captured image with the position information and stores the associated data in the storage unit 10C2. Similarly, the control unit 10C1 associates the acquired unevenness information with the position information and stores the associated data in the storage unit 10C2. That is, captured images detected at the same timing are associated with the position information, and unevenness information detected at the same timing is associated with the position information. The storage unit 10C2 stores the associated data for each detection timing. Note that the associated data is detected at the same timing, but it does not have to be exactly the same timing and may be detected at different timings. In this case, for example, captured images and position information whose detection timing difference is equal to or less than a predetermined value are treated as having been detected at the same timing. Similarly, unevenness information and position information whose detection timing difference is equal to or less than a predetermined value are treated as having been detected at the same timing and are associated with each other. Note that the above description is based on the assumption that all sensors have the same sampling period, but if the sampling periods of the sensors differ, appropriate adjustments are made.

[0023] The control unit 10C1 transmits the associated captured image and position information, and the associated unevenness information and position information to the measurement data acquisition device 12 via the communication unit 10C3. The measurement data acquisition device 12 transmits this information (captured image, unevenness information, and position information) received from the vehicle 10 to the calculation device 14. Note that if the measurement data acquisition device 12 is not provided, the control unit 10C1 may transmit the captured image, unevenness information, and position information directly to the calculation device 14.

[0024] (computing device) 3 is a schematic block diagram of a computing device. As shown in FIG. 3, the computing device 14 is, for example, a computer, and includes a communication unit 20, a storage unit 22, and a control unit 24. The communication unit 20 is a communication module that communicates with an external device, such as an antenna. The storage unit 22 is a memory that stores the computational content and programs of the control unit 24, and includes, for example, at least one of a RAM, a main storage device such as a ROM, and a non-volatile storage device such as a flash memory or a HDD. The program for the control unit 24 saved in the storage unit 22 may be stored in a recording medium readable by the computing device 14.

[0025] The control unit 24 is a computing device and includes an arithmetic circuit such as a CPU (Central Processing Unit). The control unit 24 includes an image acquisition unit 30, a concave-convex information acquisition unit 32, a position calculation unit 34, a concave-convex information extraction unit 36, and a processing unit 38. The control unit 24 implements the image acquisition unit 30, the concave-convex information acquisition unit 32, the position calculation unit 34, the concave-convex information extraction unit 36, and the processing unit 38 by reading and executing a program (software) from the storage unit 22. The control unit 24 may implement these processes using a single CPU, or may be provided with multiple CPUs and execute the processes using the multiple CPUs. At least a portion of the image acquisition unit 30, the concave-convex information acquisition unit 32, the position calculation unit 34, the concave-convex information extraction unit 36, and the processing unit 38 may be implemented using hardware.

[0026] (Image acquisition section) FIG. 4 is a schematic diagram showing an example of a captured image. The image acquisition unit 30 acquires a captured image captured by the camera 10B. The image acquisition unit 30 acquires the captured image from the measurement data acquisition device 12 via the communication unit 20. The image acquisition unit 30 acquires each captured image captured at a predetermined time interval, and more specifically, the associated captured image and position information for each predetermined time interval. In the example of FIG. 4, the captured image C shows a road R ahead of the vehicle 10, and a manhole A on the road R. Note that FIG. 4 is an example of a captured image.

[0027] (Unevenness information acquisition unit) FIG. 5 is a schematic diagram showing an example of unevenness information. The unevenness information acquisition unit 32 acquires unevenness information detected by the sensor 10C. The unevenness information acquisition unit 32 acquires the unevenness information from the measurement data acquisition device 12 via the communication unit 20. The image acquisition unit 30 acquires each piece of unevenness information detected at predetermined time intervals, and more specifically, acquires associated unevenness information and position information at predetermined time intervals. FIG. 5 shows an example where the unevenness information is behavior information (e.g., acceleration) of the vehicle 10. As shown in FIG. 5, the unevenness information is detected at predetermined time intervals.

[0028] (Position calculation unit) The position calculation unit 34 calculates position information of a region of interest on the road shown in the captured image based on the captured image acquired by the image acquisition unit 30. The position information of the region of interest is the position of the region of interest in a predetermined coordinate system (e.g., a global coordinate system). The region of interest may be any region on the road shown in the captured image. For example, in the example of FIG. 4, a manhole A on road R in captured image C may be set as the region of interest. However, the region of interest is not limited to a manhole, and may be any region on the road where any object exists, such as a cracked area, a depression, or a place with a white line on the road.

[0029] The position calculation unit 34 may calculate the position information of the area of ​​interest using any method based on the captured image, but in this embodiment, the position information of the area of ​​interest is calculated using the following method. The position calculation unit 34 identifies the location of the area of ​​interest in the captured image, in other words, the position of the area of ​​interest in the coordinate system of the captured image. The position calculation unit 34 calculates the position of the area of ​​interest in a predetermined coordinate system (e.g., a global coordinate system) as position information of the area of ​​interest based on the position of the area of ​​interest in the coordinate system of the captured image. Specifically, the position calculation unit 34 calculates the position information of the area of ​​interest based on position information of the vehicle 10 at the time the captured image was captured (position information of the vehicle 10 associated with the captured image) and the position of the area of ​​interest in the coordinate system of the captured image. More specifically, the position calculation unit 34 calculates the distance and direction from the vehicle 10 to the area of ​​interest in the predetermined coordinate system based on the position of the area of ​​interest in the coordinate system of the captured image. Then, the position calculation unit 34 calculates, as position information of the attention area, a position that is the calculated distance away in the direction from the vehicle 10 to the attention area from the position of the vehicle 10 at the time the captured image was captured (the position in a predetermined coordinate system). Note that any method for calculating the distance from the vehicle 10 to the attention area may be used, and for example, the distance may be calculated based on the height and orientation at which the camera 10B is set (the installation state of the camera 10B), or a plurality of cameras 10B may be provided and triangulation may be performed from the captured images, or the distance may be calculated from the type of distortion based on the lens aberration of one camera 10B.

[0030] The position of the attention area in the captured image may be set by any method. For example, the position calculation unit 34 may acquire the attention area in the captured image set by the user. Alternatively, for example, the position calculation unit 34 may identify the position of the attention area using image recognition technology. In this case, for example, the position calculation unit 34 identifies, as the position of the attention area, an area within the entire captured image where the feature amount of image data of each pixel satisfies a predetermined condition (for example, an area that satisfies the feature amount indicating that the pixel is a manhole). Note that the image data here refers to, for example, gradation values ​​or brightness values.

[0031] The position calculation unit 34 may identify the position of the attention area for each captured image by performing the same process for each captured image captured at predetermined intervals. Alternatively, the position calculation unit 34 may set multiple attention areas for one captured image and calculate position information for the multiple attention areas. Note that the individual attention areas may be the same object or different objects.

[0032] (Unevenness information extraction part) The unevenness information extraction unit 36 ​​extracts unevenness information in the region of interest from the unevenness information acquired by the unevenness information acquisition unit 32 based on the position information of the region of interest. That is, the unevenness information extraction unit 36 ​​selects unevenness information in the region of interest from the unevenness information detected at each predetermined time. The unevenness information in the region of interest is unevenness information detected in the region of interest. That is, for example, if the unevenness information is behavior information of the vehicle 10, the unevenness information in the region of interest is behavior information of the vehicle 10 detected at the time when the vehicle 10 is located on the region of interest (the time when the vehicle 10 passes over the region of interest). Also, for example, if the unevenness information is distance information to the road, the unevenness information in the region of interest is distance information indicating the distance from the vehicle 10 to the region of interest.

[0033] The unevenness information extraction unit 36 ​​may extract unevenness information in the region of interest using any method based on the position information of the region of interest. In this embodiment, however, the unevenness information extraction unit 36 ​​may extract unevenness information whose detected position overlaps with the position of the region of interest as unevenness information in the region of interest. However, the positions are not limited to strict overlap, and the unevenness information extraction unit 36 ​​may extract unevenness information whose detected position is within a predetermined distance range from the position of the region of interest as unevenness information in the region of interest. For example, if the unevenness information is information about the distance to the road, the unevenness information extraction unit 36 ​​calculates the detection position of the unevenness information (the detection position of the unevenness information in a predetermined coordinate system) from the position of the vehicle 10 when the unevenness information is detected and the distance information. Then, the unevenness information extraction unit 36 ​​may extract unevenness information whose detection position is within a predetermined distance from the position of the region of interest as unevenness information in the region of interest.

[0034] FIG. 6 is a schematic diagram illustrating extraction of unevenness information in the attention area. As shown in FIG. 6, for example, when the unevenness information is behavior information of the vehicle 10, the timing t1 at which the attention area is imaged may be earlier than the timing t2 at which the unevenness information in the attention area is detected (the timing at which the vehicle 10 reaches the attention area). Therefore, the unevenness information extraction unit 36 ​​may calculate the timing at which the unevenness information in the attention area is detected, and use the unevenness information detected at that timing as the unevenness information in the attention area. Specifically, for example, when the unevenness information is behavior information of the vehicle 10, the unevenness information extraction unit 36 ​​may calculate the arrival time required for the vehicle 10 to reach the attention area from the position information of the vehicle 10 when the captured image was captured and the position information of the attention area. For example, the unevenness information extraction unit 36 ​​may calculate the arrival time required for the vehicle 10 to reach the attention area from the position of the vehicle 10 when the captured image was captured and the traveling direction and speed of the vehicle 10 at that time. Then, the unevenness information extraction unit 36 ​​determines the timing (timing t1 in the example of FIG. 6) delayed by the arrival time from the timing (timing t0 in the example of FIG. 6) when the captured image is captured as the timing when unevenness information in the region of interest is detected, and determines the unevenness information at that timing as the unevenness information in the region of interest. Furthermore, the unevenness information extraction unit 36 ​​may determine the unevenness information in the region of interest based on position information of the vehicle 10 at the timing when the unevenness information is detected (or may determine the unevenness information in the region of interest).

[0035] (Processing section) The processing unit 38 executes a process of associating image data of the region of interest in the captured image with unevenness information of the region of interest. The processing unit 38 generates information that associates the image data of the region of interest in the captured image with the unevenness information of the region of interest, and calculates the road surface condition of a road whose road surface condition is unknown based on the associated information.

[0036] While a captured image can capture and record a wide range of the road surface within the angle of view, the information is limited to image data, which may prevent the road surface condition from being calculated with high accuracy. On the other hand, unevenness information can calculate the road surface condition with high accuracy, but the measurement range may be narrow. In contrast, in this embodiment, by associating image data of a region of interest in the captured image with unevenness information of the region of interest, it is possible to associate image data and unevenness information at the same position. This makes it possible to calculate the road surface condition with high accuracy and over a wide range, thereby enabling the road surface condition to be calculated appropriately.

[0037] The processing unit 38 may perform any processing to associate the image data of the region of interest with the unevenness information of the region of interest, but in this embodiment, data based on the image data of the region of interest and data based on the unevenness information of the region of interest are used as training data to train a learning model. An example of the processing by the processing unit 38 will be described below.

[0038] (Example of processing) The processing unit 38 uses the image data of the region of interest in the captured image and the data based on the unevenness information in the region of interest as training data, and causes the learning model N to perform machine learning. Note that the data based on the unevenness information in the region of interest here may be the unevenness information in the region of interest itself, or may be the road surface condition in the region of interest (e.g., IRI) calculated from the unevenness information in the region of interest. In the following, the case where the road surface condition in the region of interest is used will be taken as an example. Note that the method of calculating the road surface condition from the unevenness information may be arbitrary, and for example, the correspondence between the unevenness information and the road surface condition may be calculated using a model that has been machine-learned.

[0039] The processing unit 38 inputs a data set, in which the image data of the region of interest is used as an input value and the road surface condition of the region of interest (data based on unevenness information in the region of interest) is used as an output value, into the learning model as training data. It is preferable that the processing unit 38 prepares a plurality of data sets each consisting of image data and road surface conditions, and inputs each of the plurality of data sets into the learning model. In this way, the learning model N performs machine learning to learn the correspondence between image data of the road and the road surface condition on that road.

[0040] The learning model N is a learning model learned by deep learning, and is composed of a model (neural network configuration information) that defines a neural network that constitutes a classifier learned by deep learning, and variables. The learning model N can determine the label of input data based on the data. In the example of this embodiment, the learning model N is a CNN (Conventional Neural Network) model, but is not limited to a CNN model and may be a learning model of any type.

[0041] The processing unit 38 calculates the road surface condition of a road whose road surface condition is unknown using the trained learning model N. Specifically, the image acquisition unit 30 acquires a captured image of a road captured by a vehicle traveling on a road whose road surface condition is unknown. The processing unit 38 inputs image data of the captured image into the trained learning model N. In the learning model N, the image data of the captured image is input as input data and a calculation is performed. As a result, the learning model N outputs the road surface condition of the road shown in the captured image as output data. In other words, the learning model N can be said to be a trained program that outputs the road surface condition when image data of a captured image of a road is input. The processing unit 38 calculates the road surface condition output as output data as the road surface condition of that road.

[0042] In the above explanation, an example has been given in which the road surface condition is calculated using the learning model N, but if the learning model N has been made to learn the correspondence between image data and unevenness by machine learning, the processing unit 38 will calculate the unevenness of the road by inputting the image data into the learning model N. In this case, the processing unit 38 may calculate the road surface condition from the unevenness information.

[0043] In this way, by associating image data with unevenness and having the learning model N perform machine learning, it becomes possible to calculate road surface conditions with high accuracy simply by acquiring images captured over a wide area.

[0044] It is also possible to detect the unevenness of a road while taking an image while traveling on a road with unknown road surface conditions, and calculate the road surface condition from the unevenness. In this case, for example, the road surface condition can be recalculated with high accuracy based on the road surface condition calculated from the taken image and the road surface condition calculated from the unevenness.

[0045] (Example of processing) In the above example, the learning model N is trained to use image data as input values, but conversely, the learning model N may be trained to use unevenness as input values. An example of this case will be described below.

[0046] The processing unit 38 sets a road surface pattern for the region of interest based on the image data of the region of interest. The road surface pattern is information indicating the characteristics of the road surface, such as whether the road surface is a manhole or a cracked area. In other words, road surface patterns are information that can be classified into a finite number of types, unlike numerical values ​​such as IRI. The processing unit 38 may arbitrarily set the road surface pattern for the region of interest. For example, the processing unit 38 may acquire a road surface pattern for the region of interest set by a user and set the acquired road surface pattern as the road surface pattern for the region of interest. Alternatively, the processing unit 38 may set the road surface pattern for the region of interest using image recognition technology. In this case, for example, if the degree of match between the feature amounts of the image data of the region of interest and the feature amounts of a predetermined road surface pattern (e.g., the feature amounts of an image of a manhole) is equal to or greater than a predetermined value, the position calculation unit sets the road surface pattern for the region of interest.

[0047] The processing unit 38 uses as training data the labels indicating the road surface pattern set based on the image data of the region of interest and the data based on the unevenness information in the region of interest, and causes the learning model N to learn. Note that the data based on the unevenness information in the region of interest here may be the unevenness information in the region of interest itself, or may be the road surface condition in the region of interest calculated from the unevenness information in the region of interest, but hereinafter, an example will be given in which the data is the unevenness information in the region of interest.

[0048] The processing unit 38 inputs a data set, in which the unevenness information in the region of interest is used as an input value and the road surface pattern in the region of interest is used as an output value, into the learning model as training data. The processing unit 38 preferably prepares multiple data sets each consisting of unevenness information and road surface patterns, and inputs each of the multiple data sets into the learning model. This allows the learning model N to machine-learn the correspondence between the unevenness information and the road surface pattern of the road.

[0049] The processing unit 38 estimates the road surface pattern of a road with unknown road surface conditions using the trained learning model N. Specifically, the unevenness information acquisition unit 32 acquires unevenness information detected by a vehicle traveling on a road with unknown road surface conditions. The processing unit 38 inputs the unevenness information to the trained learning model N. In the learning model N, the unevenness information is input as input data and calculations are performed. As a result, the learning model N outputs the road surface pattern of the road for which the unevenness information was detected as output data. In other words, the learning model N can be said to be a trained program that outputs the road surface pattern of a road when unevenness information is input. The processing unit 38 can be said to identify the road surface pattern output as output data as the road surface pattern of that road.

[0050] In this way, by associating the road surface pattern identified from image data with the unevenness state and having the learning model N perform machine learning, it becomes possible to identify road surface patterns that are difficult to recognize only from an image, for example, simply by detecting the unevenness state, and the road surface state can be appropriately estimated. As a result, even if the road surface state calculated from the unevenness state indicates an abnormality, if the road surface pattern is identified as a manhole, it will be clear that the reason for the abnormality was driving over the manhole, and this can, for example, eliminate the need for unnecessary road surface repairs.

[0051] It is also possible to capture an image of the road while driving on a road with unknown road surface conditions, and identify the road surface pattern from the image data of the captured image. In this case, for example, the road surface pattern can be identified with high accuracy based on the road surface pattern identified from the image data and the road surface pattern identified from the road surface unevenness.

[0052] Alternatively, for example, the two above-described processing examples may be combined. For example, image data may be input to a learning model to calculate road surface conditions, and the road surface pattern may be identified by inputting the unevenness state to the learning model.

[0053] (Processing flow) Next, the processing flow of the above-described arithmetic device will be described. FIG. 7 is a flowchart illustrating the processing flow of the arithmetic device according to this embodiment. As shown in FIG. 7, the arithmetic device 14 acquires, via the image acquisition unit 30, a captured image captured by the camera 10B of the vehicle 10, and acquires unevenness information detected by the sensor 10C of the vehicle 10 (step S10). The arithmetic device 14 calculates, via the position calculation unit 34, position information of the region of interest based on the captured image (step S12), and extracts, via the unevenness information extraction unit 36, unevenness information in the region of interest based on the position information of the region of interest (step S14). The arithmetic device 14 then executes a process of associating the unevenness information in the region of interest with image data in the region of interest (step S16).

[0054] (effect) As described above, the arithmetic device 14 according to this embodiment includes an image acquisition unit 30, an unevenness information acquisition unit 32, a position calculation unit 34, an unevenness information extraction unit 36, and a processing unit 38. The image acquisition unit 30 acquires a captured image of a road captured by a camera 10B mounted on a vehicle 10 traveling on the road. The unevenness information acquisition unit 32 acquires unevenness information indicating the unevenness state of the road detected by a sensor 10C mounted on the vehicle 10. The position calculation unit 34 calculates position information of a region of interest on the road shown in the captured image based on the captured image. The unevenness information extraction unit 36 ​​extracts unevenness information for the region of interest from the unevenness information based on the position information of the region of interest. The processing unit 38 executes a process of associating image data of the region of interest in the captured image with the unevenness information for the region of interest.

[0055] Here, while a captured image can capture and record a wide area of ​​the road surface that fits within the angle of view, the information is limited to image data, so it may not be possible to calculate the road surface condition with high accuracy. On the other hand, unevenness information can calculate the road surface condition with high accuracy, but the measurement range may be narrow. In response to this, the calculation device 14 according to this embodiment can associate image data of a region of interest in the captured image with unevenness information of the region of interest, thereby associating image data and unevenness information at the same position. This makes it possible to calculate, for example, the road surface condition with high accuracy and over a wide range, and appropriately estimate the road surface condition.

[0056] The sensor 10C may include at least one of a behavior sensor that detects the behavior of the vehicle 10 and a distance measurement sensor that detects the distance from the sensor 10C (vehicle 10) to the road. By detecting the behavior of the vehicle 10 and the distance to the road as unevenness information and associating it with image data, it becomes possible to calculate the road surface condition with high precision and over a wide range, and the road surface condition can be appropriately estimated.

[0057] The processing unit 38 uses the data based on the image data in the region of interest and the data based on the unevenness information in the region of interest as training data, and causes the learning model N to learn. In this way, by using the image data and the unevenness information as training data for machine learning, it is possible to appropriately calculate the road surface condition and road surface pattern.

[0058] The processing unit 38 trains the learning model N using the image data of the region of interest and data based on the unevenness information of the region of interest as training data. The processing unit 38 calculates the road surface condition by inputting the image data of the road into the trained learning model N. By associating the image data with the unevenness condition in this way and training the learning model N to learn the road surface condition with high accuracy, it becomes possible to calculate the road surface condition with high accuracy simply by acquiring images captured over a wide area.

[0059] The processing unit 38 trains the learning model N using as training data labels indicating the road surface pattern of the region of interest that are set based on the image data of the region of interest and data based on the unevenness information in the region of interest. The processing unit 38 inputs data based on the road unevenness information into the trained learning model N, thereby acquiring the road surface pattern at the position on the road where the unevenness information was acquired. In this way, by associating the road surface pattern identified from the image data with the unevenness state and having the learning model N train it by machine learning, it becomes possible to identify road surface patterns that are difficult to recognize only from an image, for example, by simply detecting the unevenness state.

[0060] Although the embodiments and examples of the present invention have been described above, the embodiments are not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments. [Explanation of symbols]

[0061] 1. Detection System 10 vehicles 10A Position Sensor 10B Camera 10C Sensor 14 Arithmetic unit 30 Image acquisition unit 32 Concave / convex information acquisition section 34 Position calculation section 36 Unevenness information extraction part 38 Processing section

Claims

1. an image acquisition unit that acquires an image of the road captured by a camera provided on a vehicle traveling on the road; an unevenness information acquisition unit that acquires unevenness information indicating an uneven state of the road detected by a sensor provided in the vehicle; a position calculation unit that calculates position information of the attention area on the road shown in the captured image based on a position of the vehicle when the captured image was acquired and a distance between the vehicle and the attention area calculated from the captured image; an unevenness information extraction unit that extracts unevenness information in the region of interest from the unevenness information based on position information of the region of interest; a processing unit that executes a process of associating image data in a region of interest of the captured image with unevenness information in the region of interest; Including, Computing device.

2. The computing device according to claim 1 , wherein the sensor includes at least one of a behavior sensor that detects a behavior of the vehicle, and a distance measuring sensor that detects a distance from the sensor to the road.

3. The arithmetic device according to claim 1 , wherein the processing unit causes a learning model to learn data based on image data in the region of interest and data based on unevenness information in the region of interest as teacher data.

4. The processing unit includes: The image data in the region of interest and data based on unevenness information in the region of interest are used as training data to train the learning model; The computing device according to claim 3 , wherein image data of a road is input to the learned learning model, and the computing device outputs the road surface condition of the road.

5. The processing unit includes: A label indicating a road surface pattern of the region of interest, which is set based on image data of the region of interest, and data based on unevenness information of the region of interest are used as training data to train the learning model; The computing device according to claim 3 , wherein the road surface pattern at the position on the road where the unevenness information was acquired is acquired by inputting data based on road unevenness information into the learned learning model.

6. acquiring an image of a road captured by a camera mounted on a vehicle traveling on the road; acquiring unevenness information indicating an uneven state of the road detected by a sensor provided on the vehicle; calculating position information of the attention area on the road shown in the captured image based on a position of the vehicle when the captured image was acquired and a distance between the vehicle and the attention area calculated from the captured image; extracting unevenness information in the region of interest from the unevenness information based on position information of the region of interest; executing a process of associating image data in a region of interest of the captured image with unevenness information in the region of interest; Including, Calculation method.

7. acquiring an image of a road captured by a camera mounted on a vehicle traveling on the road; acquiring unevenness information indicating an uneven state of the road detected by a sensor provided on the vehicle; calculating position information of the attention area on the road shown in the captured image based on a position of the vehicle when the captured image was acquired and a distance between the vehicle and the attention area calculated from the captured image; extracting unevenness information in the region of interest from the unevenness information based on position information of the region of interest; executing a process of associating image data in a region of interest of the captured image with unevenness information in the region of interest; to cause a computer to execute program.

Images