Road surface condition determination device, road surface condition determination system, vehicle, road surface condition determination method, and program

Abstract

To improve the accuracy of road surface state determination using images.SOLUTION: A road surface state determination device comprises a control portion that, on the basis of image data for each section obtained by capturing a plurality of sections of a road using an image sensor attached to a vehicle traveling on the road, acquires identification results of a road surface state of each section, when each of the plurality of sections is a first section and at least one section before, after, or before and after the first section is a second section, determines the road surface state of the first section by referring to a first identification result that is an identification result of the road surface state of the first section and a second identification result that is an identification result of the road surface state of the second section.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a road surface condition determination device, a road surface condition determination system, a vehicle, a road surface condition determination method, and a program. [Background technology]

[0002] Patent Document 1 discloses a device that is mounted on a vehicle and determines road surface conditions. This device detects the pavement type of the road surface based on the vehicle position and map information, determines a first road surface condition corresponding to the pavement type and the running sound, determines a second road surface condition corresponding to the pavement type and the road surface image, and outputs the first road surface condition or the second road surface condition to another device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-309832 A Summary of the Invention [Problem to be solved by the invention]

[0004] When judging road surface conditions from an image, an accurate judgment may not be possible depending on the image due to factors such as the amount of light or the presence of objects on the road.

[0005] An object of the present disclosure is to improve the accuracy of determining road surface conditions from images. [Means for solving the problem]

[0006] A road surface condition determination device as one aspect of the present disclosure is provided with a control unit that acquires identification results for road surface conditions for each section based on image data for each section obtained by capturing images of multiple sections of a road using an image sensor attached to a vehicle traveling on the road, and when each of the multiple sections is defined as a first section and at least one section before, after, or before and after the first section is defined as a second section, determines the road surface condition of the first section by referring to a first identification result that is an identification result of the road surface condition of the first section and a second identification result that is an identification result of the road surface condition of the second section. According to this aspect, the accuracy of determining the road surface condition from an image is improved.

[0007] In one embodiment of the present disclosure, the control unit inputs image data for each section into an identification model for identifying road surface conditions, and obtains an identification result of the road surface condition for each section from the identification model. According to this embodiment, the efficiency of judging the road surface condition from an image is improved.

[0008] In one embodiment of the present disclosure, the identification result of the road surface condition for each section includes a probability that the road surface condition for each section falls into each of a plurality of classes, and the control unit determines the class to which the road surface condition of the first section falls by comparing the probabilities contained in the first identification result and the second identification result. According to this embodiment, it becomes easier to prevent erroneous determination of the road surface condition of the first section.

[0009] In one embodiment of the present disclosure, the control unit determines a class having the highest probability in the greatest number of the first and second classification results as a class to which the road surface condition of the first section corresponds. According to this embodiment, the identification result of the road surface condition of the first section can be corrected.

[0010] As one embodiment of the present disclosure, the control unit determines a class having the highest average probability in the first classification result and the second classification result as a class to which the road surface condition of the first section corresponds. According to this embodiment, the identification result of the road surface condition of the first section can be corrected.

[0011] As one embodiment of the present disclosure, the control unit acquires weather data indicating the weather at the time the road was imaged, and weights the probability that each of the first classification result and the second classification result will be included for each class based on the acquired weather data. According to this embodiment, the accuracy of determining the road surface condition from an image is further improved.

[0012] In one embodiment of the present disclosure, the discriminative model includes a trained model that has been subjected to machine learning. According to this embodiment, the accuracy of determining road surface conditions from images can be improved by training a discrimination model using a large amount of data.

[0013] A road surface condition determination system according to one aspect of the present disclosure includes the road surface condition determination device and the image sensor. According to this aspect, the accuracy of determining the road surface condition from an image is improved.

[0014] A vehicle according to one aspect of the present disclosure includes the road surface condition determination system. According to this aspect, road surface conditions can be determined in a vehicle using images.

[0015] A road surface condition determination method according to one aspect of the present disclosure involves capturing images of multiple sections of a road using an image sensor attached to a vehicle traveling on the road, obtaining identification results for the road surface condition for each section by a computer based on image data for each section obtained as the captured image, and, when each of the multiple sections is designated a first section and at least one section before, after, or before and after the first section is designated a second section, the computer refers to a first identification result that is an identification result of the road surface condition of the first section and a second identification result that is an identification result of the road surface condition of the second section to determine the road surface condition of the first section. According to this aspect, the accuracy of determining the road surface condition from an image is improved.

[0016] A program as one aspect of the present disclosure causes a computer to perform a process of obtaining an identification result of the road surface condition for each section based on image data for each section obtained by capturing images of the multiple sections of a road using an image sensor attached to a vehicle traveling on the road, and a process of determining the road surface condition of the first section by referring to a first identification result that is an identification result of the road surface condition of the first section and a second identification result that is an identification result of the road surface condition of the second section, when each of the multiple sections is defined as a first section and at least one section before, after, or before and after the first section is defined as a second section. According to this aspect, the accuracy of determining the road surface condition from an image is improved. Effect of the Invention

[0017] According to the present disclosure, the accuracy of determining road surface conditions from images is improved. [Brief description of the drawings]

[0018] [Figure 1] 1 is a diagram illustrating a configuration of a road surface condition determination system according to an embodiment of the present disclosure. [Diagram 2] FIG. 4 is a diagram showing a procedure for determining a road surface condition according to an embodiment of the present disclosure. [Diagram 3] 11A to 11C are diagrams illustrating an example of image data for each section and an example of a result of identifying a road surface condition for each section according to an embodiment of the present disclosure. [Figure 4] 1 is a block diagram showing a configuration of a road surface condition determination device according to an embodiment of the present disclosure. [Diagram 5] 4 is a flowchart showing an operation of the road surface condition determination system according to the embodiment of the present disclosure. [Figure 6] 4 is a flowchart showing an example of an operation of the road surface condition determination device according to the embodiment of the present disclosure. [Figure 7] 10 is a flowchart showing another example of the operation of the road surface condition determination device according to the embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0020] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0021] The configuration of a road surface condition determination system 10 according to this embodiment will be described with reference to FIG.

[0022] The road surface condition determination system 10 includes a road surface condition determination device 20 and an image sensor 30.

[0023] The road surface condition determination device 20 is a computer. In this embodiment, the road surface condition determination device 20 is a PC, but it may be a mobile device such as a smartphone or a tablet, a server device belonging to a cloud computing system or other computing system, or a dedicated device. "PC" is an abbreviation for personal computer.

[0024] The image sensor 30 is attached to a vehicle 50 traveling on a road 11. Specifically, the image sensor 30 is attached on the dashboard or near the rearview mirror facing the front of the vehicle 50 so as to be able to capture an image of the area in front of the vehicle 50. The image sensor 30 is, for example, a CMOS sensor, a CCD sensor, or a camera incorporating either of these. "CMOS" is an abbreviation for complementary metal oxide semiconductor. "CCD" is an abbreviation for charge coupled device. The image sensor 30 may be able to capture an image of the rear.

[0025] The vehicle 50 is any type of automobile, such as a gasoline vehicle, a diesel vehicle, an HV, a PHV, an EV, or an FCV. "HV" is an abbreviation for hybrid vehicle. "PHV" is an abbreviation for plug-in hybrid vehicle. "EV" is an abbreviation for electric vehicle. "FCV" is an abbreviation for fuel cell vehicle.

[0026] The present embodiment will be outlined with reference to FIG. 1 and FIG.

[0027] The image sensor 30 captures images of multiple sections of the road 11. The road surface condition determination device 20 acquires an identification result of the road surface condition for each section based on image data 60 for each section obtained as a captured image. Specifically, the road surface condition determination device 20 inputs the image data 60 for each section to the identification model 70, thereby acquiring an identification result of the road surface condition for each section from the identification model 70. When each of the multiple sections is a first section, and at least one section before, after, or before and after the first section is a second section, the road surface condition determination device 20 judges the road surface condition of the first section by referring to a first identification result that is an identification result of the road surface condition of the first section and a second identification result that is an identification result of the road surface condition of the second section. Therefore, according to this embodiment, the accuracy of road surface condition determination by image is improved.

[0028] The identification model 70 may be any identification model for identifying road surface conditions, but in this embodiment, it includes a trained model that has undergone machine learning. For example, by linking actual road surface conditions as labels to image data previously obtained by the image sensor 30, it is possible to create training data for machine learning. Then, by using this training data to perform machine learning using a known machine learning algorithm such as a neural network or deep learning, it is possible to generate a trained model as the identification model 70. According to this embodiment, the accuracy of road surface condition determination from images can be improved by having the identification model 70 learn a large amount of data.

[0029] In this embodiment, the identification model 70 outputs a probability for each label as an identification result of the road surface condition for each section. The probability for each label is the probability that the road surface condition corresponds to a class corresponding to each label. That is, in this embodiment, the identification result of the road surface condition for each section includes the probability that the road surface condition for each section corresponds to each of multiple classes.

[0030] In this embodiment, seven classes are defined as multiple classes: "dry," "semi-humid," "wet and moist," "sherbet," "accumulated snow," "packed snow," and "frozen." However, instead of these seven classes, any two or more classes may be defined, such as five classes: "dry," "semi-humid," "wet and moist," "sherbet," and "snow."

[0031] FIG. 3 shows a first section image 61, a previous section image 62, and a subsequent section image 63 as examples of image data 60 for each section. The length of each section is set to a fixed distance, such as 10 meters. When a certain section is defined as a first section and two sections, one before and one after the first section, are defined as second sections, the first section image 61 is an example of an image obtained by imaging the first section. The previous section image 62 is an example of an image obtained by imaging the section immediately before the first section. The subsequent section image 63 is an example of an image obtained by imaging the section immediately after the first section.

[0032] FIG. 3 further shows, as an example of the classification result of the road surface condition for each section, the classification results output from the classification model 70 when the first section image 61, the previous section image 62, and the next section image 63 are input to the classification model 70. The probabilities of "moist" and "dry" included in the classification result corresponding to the first section image 61 are 30% and 50%, respectively. If the probabilities of other labels included in the classification result corresponding to the first section image 61 are all lower than 30%, it is considered that the road surface condition of the first section is highly likely to be dry. However, since the probability of "moist" included in the classification result corresponding to the previous section image 62 is 90%, it is highly likely that the road surface condition of the section immediately preceding the first section is moist. Since the probability of "moist" included in the classification result corresponding to the next section image 63 is also 90%, it is highly likely that the road surface condition of the section immediately following the first section is also moist. Therefore, it is highly likely that the road surface condition of the first section is also moist. In fact, the wipers are captured in the first section image 61 shown in FIG. 3, which is thought to have adversely affected the results of identifying the road surface condition of the first section.

[0033] In this embodiment, when determining the road surface condition of the first section, not only the classification result corresponding to the first section image 61 but also the classification result corresponding to the previous section image 62, the classification result corresponding to the next section image 63, or both of them are referenced. This makes it easier to prevent erroneous determination of the road surface condition of the first section.

[0034] The configuration of a road surface condition determining device 20 according to this embodiment will be described with reference to FIG.

[0035] The road surface condition determination device 20 includes a control unit 21 , a storage unit 22 , a communication unit 23 , an input unit 24 , and an output unit 25 .

[0036] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 executes processes related to the operation of the road surface condition determination device 20 while controlling each part of the road surface condition determination device 20.

[0037] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM or a ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The storage unit 22 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used in the operation of the road surface condition determination device 20 and data obtained by the operation of the road surface condition determination device 20.

[0038] The communication unit 23 includes at least one communication interface. The communication interface is, for example, a LAN interface, an interface compatible with a mobile communication standard such as LTE, the 4G standard, or the 5G standard, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "LAN" is an abbreviation for local area network. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 23 receives data used in the operation of the road surface condition determination device 20, and transmits data obtained by the operation of the road surface condition determination device 20.

[0039] The input unit 24 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen integral with a display, a camera, or a microphone. The input unit 24 accepts an operation to input data used for the operation of the road surface condition determination device 20. The input unit 24 may be connected to the road surface condition determination device 20 as an external input device instead of being provided in the road surface condition determination device 20. As the connection interface, for example, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used. "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface.

[0040] The output unit 25 includes at least one output interface. The output interface is, for example, a display or a speaker. The display is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescence. The output unit 25 outputs data obtained by the operation of the road surface condition determination device 20. The output unit 25 may be connected to the road surface condition determination device 20 as an external output device instead of being provided in the road surface condition determination device 20. As the connection interface, for example, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0041] The functions of the road surface condition determination device 20 are realized by executing a program according to this embodiment in a processor serving as a control unit 21. That is, the functions of the road surface condition determination device 20 are realized by software. The program causes a computer to execute the operations of the road surface condition determination device 20, thereby causing the computer to function as the road surface condition determination device 20. That is, the computer functions as the road surface condition determination device 20 by executing the operations of the road surface condition determination device 20 in accordance with the program.

[0042] The program may be stored in a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include a flash memory, a magnetic recording device, an optical disk, a magneto-optical recording medium, or a ROM. The program may be distributed, for example, by selling, transferring, or lending a portable medium such as an SD card, DVD, or CD-ROM on which the program is stored. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer. The program may be provided as a program product.

[0043] A computer temporarily stores in a main storage device, for example, a program stored in a portable medium or a program transferred from a server. Then, the computer reads the program stored in the main storage device with a processor, and executes processing according to the read program with the processor. The computer may read the program directly from a portable medium and execute processing according to the program. The computer may execute processing according to the received program each time a program is transferred from the server to the computer. Processing may be executed by a so-called ASP-type service that realizes functions only by execution instructions and result acquisition without transferring a program from the server to the computer. "ASP" is an abbreviation for application service provider. A program is information used for processing by a computer, and includes those equivalent to a program. For example, data that is not a direct command to a computer but has the property of defining computer processing falls under "those equivalent to a program."

[0044] A part or all of the functions of the road surface condition determination device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. In other words, a part or all of the functions of the road surface condition determination device 20 may be realized by hardware.

[0045] The operation of the road surface condition determination system 10 according to this embodiment will be described with reference to Fig. 5. This operation corresponds to the road surface condition determination method according to this embodiment.

[0046] The process of step S1 is performed for each section of a fixed distance, such as 10 meters, that the vehicle 50 travels. The processes of steps S2 to S4 are also performed for each section. In this embodiment, the processes of steps S2 to S4 are performed after the process of step S1 is performed for all of the multiple sections of the road 11, but after the process of step S1 is performed for one or more sections, they may be performed in parallel with the process of step S1 being performed for the remaining sections. As a variation of this embodiment, the process of step S1 may be performed at regular time intervals.

[0047] In step S1, the image sensor 30 captures an image of each section of the road 11. Image data 60 obtained as the captured image is stored in a medium such as a flash memory or a magnetic recording device mounted on the vehicle 50, or is transmitted to the outside via a network such as the Internet by a communication device mounted on the vehicle 50. The control unit 21 of the road surface condition determination device 20 acquires the image data 60 stored in the medium mounted on the vehicle 50 via the communication unit 23 or another interface corresponding to a standard such as USB. Alternatively, the control unit 21 acquires the image data 60 by receiving the image data 60 transmitted by the communication device mounted on the vehicle 50 via the communication unit 23.

[0048] Each of the multiple sections is a first section, and at least one section before, after, or before and after the first section is a second section. Which section is the second section and how many sections are the second section may be determined arbitrarily, but in this embodiment, if the section imaged first is the first section, the section immediately following the first section is the second section. If the section imaged last is the first section, the section immediately preceding the first section is the second section. In other cases, the section immediately preceding the first section and the section immediately following the first section are each determined as the second section.

[0049] In step S2, the control unit 21 of the road surface condition determination device 20 obtains a first identification result based on the image data 60 of the first section obtained in step S1. Specifically, the control unit 21 extracts a road surface image of the first section from the image data 60 of the first section obtained in step S1. Any method may be used to extract the road surface image, such as a method of cutting out an area of ​​a fixed number of pixels as a road surface image, a method of cutting out a road surface area recognized by performing image analysis as a road surface image, or a method of cutting out a road surface area identified using machine learning as a road surface image. The road surface image may be cut out for each lane. The control unit 21 inputs the extracted road surface image of the first section to a discrimination model 70 for discriminating the road surface condition, thereby obtaining the discrimination result of the road surface condition of the first section from the discrimination model 70 as the first discrimination result. In this embodiment, the first discrimination result includes the probability that the road surface condition of the first section corresponds to each of the seven classes of "dry", "semi-humid", "wet and moist", "sherbet", "snowfall", "compacted snow", and "frozen".

[0050] In step S3, the control unit 21 of the road surface condition determination device 20 obtains a second identification result based on the image data 60 of the second section obtained in step S1. Specifically, the same process as that performed in step S2 on the image data 60 of the first section is performed on the image data 60 of the second section. That is, a road surface image of the second section is extracted from the image data 60 of the second section obtained in step S1. The control unit 21 inputs the extracted road surface image of the second section to the identification model 70, thereby obtaining an identification result of the road surface condition of the second section from the identification model 70 as a second identification result. In this embodiment, the second identification result includes a probability that the road surface condition of the second section corresponds to each of the seven classes of "dry", "semi-humid", "wet and moist", "sherbet", "snowfall", "packed snow", and "frozen".

[0051] In this embodiment, except when the section imaged first is the first section, the identification result of the road surface condition of the first section has already been acquired as the second identification result in the process of step S3 executed when the section immediately preceding the first section was set as the first section. Therefore, in step S2, the control unit 21 of the road surface condition determination device 20 may simply acquire the identification result as the first identification result.

[0052] In this embodiment, when the section imaged second is the first section, the identification result of the road surface condition of the section immediately preceding the first section has already been acquired as the first identification result in the process of step S2 executed when the section immediately preceding the first section was the first section. Therefore, in step S3, the control unit 21 of the road surface condition determination device 20 may simply acquire the identification result as the second identification result. Except when the section imaged first is the first section and when the section imaged second is the first section, the identification result of the road surface condition of the section immediately preceding the first section has already been acquired as the second identification result in the process of step S3 executed when the section two sections prior to the first section was the first section. Therefore, in step S3, the control unit 21 of the road surface condition determination device 20 may simply acquire the identification result as the second identification result.

[0053] In step S4, the control unit 21 of the road surface condition determination device 20 determines the road surface condition by referring to the first and second identification results acquired in steps S2 and S3, respectively. Specifically, the control unit 21 determines the class to which the road surface condition of the first section belongs by comparing the probability of being included in the first and second identification results.

[0054] As an example of the process of step S4, the control unit 21 of the road surface condition determination device 20 determines the class with the highest probability in the largest number of the first and second identification results as the class to which the road surface condition of the first section corresponds. This process may be performed in any order, but is, for example, performed in the order shown in FIG. 6.

[0055] 6, the control unit 21 sets a counter i to 1. In step S402, the control unit 21 sets Vi, which records the number of votes for the label Li, to 0. In step S403, the control unit 21 sets a counter j to 1.

[0056] In step S404 of Fig. 6, the control unit 21 judges whether the probability of the label Li is the highest in the classification result of the j-th imaged section among the first classification result and the second classification result. In this embodiment, the label L1 is "dry", the label L2 is "semi-humid", the label L3 is "moist", the label L4 is "sherbet", the label L5 is "accumulated snow", the label L6 is "compacted snow", and the label L7 is "frozen". If the probability of the label Li is the highest in the classification result of the j-th imaged section, the control unit 21 increments Vi by 1 in step S405. If the probability of a label other than the label Li is the highest in the classification result of the j-th imaged section, the process of step S405 is skipped.

[0057] In step S406 of Fig. 6, the control unit 21 determines whether the counter j is smaller than the number of identification results acquired in steps S2 and S3. In this embodiment, the number of identification results is 3. If the counter j is smaller than the number of identification results, the control unit 21 increments the counter j by 1 in step S407. Then, the process of step S404 is executed again. If the counter j is the same as the number of identification results, the process of step S408 is executed.

[0058] In step S408 of Fig. 6, the control unit 21 judges whether the counter i is smaller than the number of classes. In this embodiment, the number of classes is 7. If the counter i is smaller than the number of classes, the control unit 21 increments the counter i by 1 in step S409. Then, the process of step S402 is executed again. If the counter i is the same as the number of classes, the process of step S410 is executed.

[0059] In step S410 of FIG. 6, the control unit 21 determines the class corresponding to the label with the largest number of votes as the class to which the road surface condition corresponds. Suppose that the probabilities of "moist" and "dry" included in the first classification result are 30% and 50%, respectively, and the probabilities of all other labels are lower than 30%. Suppose that the probabilities of "moist" and "dry" included in the second classification result for the section immediately preceding the first section are 90% and 5%, respectively. Suppose that the probabilities of "moist" and "dry" included in the second classification result for the section immediately following the first section are also 90% and 5%, respectively. In this case, V1, which records the number of votes for the label L1 "dry", is 1, and V3, which records the number of votes for the label L3 "moist" is 2. Since the number of votes for "moist" is the largest, the control unit 21 determines that the road surface condition is moist.

[0060] As another example of the process of step S4, the control unit 21 of the road surface condition determination device 20 may determine that the class having the highest average probability in the first identification result and the second identification result is the class to which the road surface condition of the first section corresponds. This process may be performed in any order, but is, for example, performed in the order shown in FIG.

[0061] 7, the control unit 21 sets a counter i to 1. In step S412, the control unit 21 calculates an average Ai of the probabilities of the labels Li included in the classification results acquired in steps S2 and S3.

[0062] In step S413 of Fig. 7, the control unit 21 judges whether the counter i is smaller than the number of classes. In this embodiment, the number of classes is 7. If the counter i is smaller than the number of classes, the control unit 21 increments the counter i by 1 in step S414. Then, the process of step S412 is executed again. If the counter i is the same as the number of classes, the process of step S415 is executed.

[0063] In step S415 of FIG. 7, the control unit 21 determines the class corresponding to the label with the highest average probability as the class to which the road surface condition corresponds. Suppose that the probabilities of "moist" and "dry" included in the first classification result are 30% and 50%, respectively, and the probabilities of all other labels are lower than 30%. Suppose that the probabilities of "moist" and "dry" included in the second classification result for the section immediately preceding the first section are 90% and 5%, respectively. Suppose that the probabilities of "moist" and "dry" included in the second classification result for the section immediately following the first section are also 90% and 5%, respectively. In this case, the average A1 of the probability of the label L1 "dry" is 20%, and the average A3 of the probability of the label L3 "moist" is 70%. Since the average probability of "moist" is the highest, the control unit 21 determines that the road surface condition is moist.

[0064] In step S410 in Fig. 6, the road surface condition is determined to correspond to the class corresponding to the label with the most votes, and in step S415 in Fig. 7, the road surface condition is determined to correspond to the class corresponding to the label with the highest probability, but as another example, it is also possible to differentiate the priority of the determination depending on the class. For example, for a class corresponding to a dangerous road surface such as a packed snow road surface or an icy road surface, or a class corresponding to a semi-wet road surface that appears locally, it is easier to realize road surface management that ensures safety by giving priority to the determination result even if the class has a small number of votes or a low probability.

[0065] As described above, in this embodiment, the control unit 21 of the road surface condition determination device 20 acquires an identification result of the road surface condition for each section based on image data 60 for each section obtained by capturing images of the multiple sections of the road 11 by the image sensor 30 attached to the vehicle 50 traveling on the road 11. When each of the multiple sections is defined as a first section and at least one section before, after, or before and after the first section is defined as a second section, the control unit 21 determines the road surface condition of the first section by referring to a first identification result that is an identification result of the road surface condition of the first section and a second identification result that is an identification result of the road surface condition of the second section. Therefore, according to this embodiment, the accuracy of road surface condition determination using images is improved.

[0066] In this embodiment, the control unit 21 of the road surface condition determination device 20 inputs image data 60 for each section to a discrimination model 70 for discriminating road surface conditions, and obtains a discrimination result of the road surface condition for each section from the discrimination model 70. Therefore, according to this embodiment, the efficiency of determining road surface conditions from images is improved.

[0067] In this embodiment, the control unit 21 of the road surface condition determination device 20, when performing road surface condition determination based on images obtained by capturing images of the road 11 at regular intervals from a traveling vehicle 50, can correct the determination result at a certain time by using the determination results before, after, or both before and after the time in the time direction. Specifically, the control unit 21 can correct the determination result by executing a weighted voting process in the time direction as shown in Fig. 6. This can reduce erroneous determination of the road surface condition.

[0068] As a modification of this embodiment, meteorological information such as a weather forecast may be reflected. In such a modification, the control unit 21 of the road surface condition determination device 20 acquires meteorological data indicating the weather when the road 11 was imaged. Based on the acquired meteorological data, the control unit 21 weights the probability of each of the first and second classification results being included for each class. This modification further improves the accuracy of road surface condition determination from an image.

[0069] The present disclosure is not limited to the above-mentioned embodiment. For example, two or more blocks shown in the block diagram may be integrated, or one block may be divided. Instead of executing two or more steps shown in the flowchart in chronological order as described, they may be executed in parallel or in a different order depending on the processing capacity of the device executing each step, or as necessary. Other modifications are possible without departing from the spirit of the present disclosure.

[0070] For example, the road surface condition determination device 20 may be an in-vehicle device. That is, a vehicle 50 may be equipped with the road surface condition determination system 10. According to this example, the road surface condition determination based on an image can be performed by the vehicle 50.

[0071] For example, the probability of being included in each of the first and second classification results may be weighted so that the road surface condition is more likely to be determined to be wet before and after a section where the road surface condition is determined to be slush, snowy, or packed snow. [Industrial Applicability]

[0072] The present disclosure relates to a road surface condition determination device, a road surface condition determination system, a vehicle, a road surface condition determination method, and a program. [Explanation of symbols]

[0073] 10: Road surface condition determination system, 11: Road, 20: Road surface condition determination device, 21: Control unit, 22: Storage unit, 23: Communication unit, 24: Input unit, 25: Output unit, 30: Image sensor, 50: Vehicle, 60: Image data, 61: First section image, 62: Previous section image, 63: Next section image, 70: Recognition model

Claims

1. A road surface condition determination device comprising a control unit that obtains a road surface condition identification result for each section, including a probability that the road surface condition for each section falls into each of a plurality of classes, based on image data for each section obtained by capturing images of multiple sections of a road with an image sensor attached to a vehicle traveling on the road, and determines the class to which the road surface condition of the first section falls by comparing the probability included in a first identification result, which is the identification result of the road surface condition of the first section, with a second identification result, which is the identification result of the road surface condition of the second section, when each of the multiple sections is a first section and at least one section before, after, or before and after the first section is a second section.

2. The road surface condition determination device according to claim 1 , wherein the control unit inputs image data for each section into an identification model for identifying road surface conditions, and obtains an identification result of the road surface condition for each section from the identification model.

3. The road surface condition determination device according to claim 2 , wherein the identification model includes a trained model that has been subjected to machine learning.

4. 4. The road surface condition determination device according to claim 1, wherein the control unit determines a class that has the highest probability in a majority of the first and second identification results as a class to which the road surface condition of the first section corresponds.

5. 4. The road surface condition determination device according to claim 1, wherein the control unit determines a class having the highest average probability in the first identification result and the second identification result as a class to which the road surface condition of the first section corresponds.

6. 6. The road surface condition determination device according to claim 1, wherein the control unit acquires weather data indicating the weather at the time the road was imaged, and weights the probability of each of the first classification result and the second classification result being included for each class based on the acquired weather data.

7. The road surface condition determination device according to any one of claims 1 to 6, The image sensor; A road surface condition determination system comprising:

8. A vehicle comprising the road surface condition determination system according to claim 7.

9. capturing images of a plurality of sections of a road using an image sensor attached to a vehicle traveling on the road; acquiring, by a computer, a classification result of the road surface condition for each section, including a probability that the road surface condition for each section corresponds to each of a plurality of classes, based on image data for each section obtained as a captured image; A road surface condition determination method in which, when each of the multiple sections is defined as a first section, and at least one section before, after, or before and after the first section is defined as a second section, the computer compares the probability that the first identification result, which is an identification result of the road surface condition of the first section, will be included in a first identification result, which is an identification result of the road surface condition of the second section, to determine the class to which the road surface condition of the first section falls.

10. A process of acquiring a classification result of a road surface condition for each section, including a probability that the road surface condition for each section falls into each of a plurality of classes, based on image data for each section obtained by capturing images of the plurality of sections of the road using an image sensor attached to a vehicle traveling on the road; a process of determining a class to which the road surface condition of the first section corresponds by comparing a probability that the first identification result, which is an identification result of the road surface condition of the first section, is included in a first identification result, which is an identification result of the road surface condition of the second section, when each of the plurality of sections is a first section, and at least one section before, after, or before and after the first section is a second section; A program that causes a computer to execute the following.

Images