Road surface condition determination device, road surface condition determination system, vehicle, road surface condition determination method, and program

Abstract

To improve the accuracy of road surface state determination using sounds.SOLUTION: A road surface state determination device comprises a control portion that analyzes time-series data obtained by measuring sound produced when a vehicle travels on a road using a sound sensor attached to the vehicle, generates analysis data indicating change over time of signal intensity of each frequency component, inputs the generated analysis data into an identification model for identifying a road surface state of the road to acquire an identification result of the road surface state from the identification model, and determines the road surface state by referring to the acquired identification result.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a road surface condition determination device, a road surface condition determination system, a vehicle, a road surface condition determination method, and a program. [Background technology]

[0002] Patent Document 1 discloses a device that is mounted on a vehicle and determines road surface conditions. This device detects the pavement type of the road surface based on the vehicle position and map information, determines a first road surface condition corresponding to the pavement type and the running sound, determines a second road surface condition corresponding to the pavement type and the road surface image, and outputs the first road surface condition or the second road surface condition to another device.

[0003] Patent Document 2 discloses a device for evaluating sounds emitted from a moving vehicle. This device acquires time waveform data based on the sound, and generates an evaluation image showing a time axis, a frequency axis, and signal strength based on a part of the time waveform data. This device then refers to a reference database that stores associations between past evaluation images and reference data linked to the past evaluation images, and derives an evaluation result for the generated evaluation image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-309832 A [Patent Document 2] JP 2020-101447 A Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional devices are unable to judge road surface conditions from sounds with a high degree of accuracy.

[0006] An object of the present disclosure is to improve the accuracy of determining road surface conditions from sound. [Means for solving the problem]

[0007] A road surface condition determination device according to one aspect of the present disclosure includes a control unit that analyzes time series data obtained by measuring sounds generated when a vehicle is traveling on a road using a sound sensor attached to the vehicle, generates analysis data indicating changes in signal strength over time for each frequency component, inputs the generated analysis data into an identification model for identifying the road surface condition of the road, obtains an identification result of the road surface condition from the identification model, and determines the road surface condition by referring to the obtained identification result. According to this aspect, the accuracy of judging the road surface condition from sound is improved.

[0008] As one embodiment of the present disclosure, the control unit acquires a second identification result, which is an identification result of the road surface condition, based on image data obtained by capturing an image of the road by an image sensor attached to the vehicle, in addition to a first identification result, which is an identification result obtained from the identification model, and determines the road surface condition by referring to the acquired first identification result and second identification result. According to this embodiment, the accuracy of determining the road surface condition is further improved.

[0009] In one embodiment of the present disclosure, the image data includes road surface images from a first point to a second point on the road, and the control unit extracts data from the time series data from a first time point when the vehicle passes the first point to a second time point when the vehicle passes the second point, and analyzes the extracted data to generate the analysis data. According to this embodiment, the analysis data can be synchronized with the image data.

[0010] In one embodiment of the present disclosure, the first identification result and the second identification result each include a probability that the road surface condition falls into each of a plurality of classes, and the control unit determines the class to which the road surface condition falls by comparing the probabilities included in the first identification result and the second identification result. According to this embodiment, it is possible to adopt the result that is more likely to be reliable, between the result of classification based on sound and the result of classification based on image.

[0011] As one embodiment of the present disclosure, the control unit acquires meteorological data indicating the weather at the time the sound was measured, and weights the probability that the sound is included in each of the first classification result and the second classification result for each class based on the acquired meteorological data. According to this embodiment, the accuracy of determining the road surface condition is further improved.

[0012] As one embodiment of the present disclosure, the control unit weights each of the first identification result and the second identification result depending on a time period during which the sound is measured. According to this embodiment, it is possible to adopt the result that is more suitable for the time period, between the result of classification by sound and the result of classification by image.

[0013] As one embodiment of the present disclosure, the control unit weights each of the first identification result and the second identification result depending on an outside air temperature when the sound is measured. According to this embodiment, it is possible to adopt the result that is more suitable for the outside air temperature, between the result of classification by sound and the result of classification by image.

[0014] As one embodiment of the present disclosure, the control unit weights the first classification result and the second classification result in accordance with information caused by an external environment. According to this embodiment, the accuracy of determining the road surface condition is further improved.

[0015] As one embodiment of the present disclosure, the control unit extracts data of a certain range of frequencies from the time-series data, and analyzes the extracted data to generate the analysis data. According to this embodiment, the accuracy of judging the road surface condition from sound is further improved.

[0016] As one embodiment of the present disclosure, the control unit extracts data on a certain range of signal strength from the time-series data, and analyzes the extracted data to generate the analysis data. According to this embodiment, the accuracy of judging the road surface condition from sound is further improved.

[0017] In one embodiment of the present disclosure, the discriminative model includes a trained model that has been subjected to machine learning. According to this embodiment, the accuracy of judging the road surface condition from sound can be improved by having the discrimination model learn a large amount of data.

[0018] In one embodiment of the present disclosure, the analysis data includes a spectrogram or a scalogram. According to this embodiment, the efficiency of judging the road surface condition from sound is improved.

[0019] A road surface condition determination system according to one aspect of the present disclosure includes the road surface condition determination device and the sound sensor. According to this aspect, the accuracy of judging the road surface condition from sound is improved.

[0020] A vehicle according to one aspect of the present disclosure includes the road surface condition determination system. According to this aspect, road surface conditions can be determined from sounds in a vehicle.

[0021] A road surface condition determination method according to one aspect of the present disclosure includes measuring sounds generated when a vehicle is traveling on a road using a sound sensor attached to the vehicle, analyzing time-series data obtained as the measurement results using a computer to generate analysis data indicating changes in signal strength over time for each frequency component, inputting the analysis data into an identification model for identifying the road surface condition of the road by the computer, obtaining an identification result of the road surface condition from the identification model, and referring to the identification result by the computer to determine the road surface condition. According to this aspect, the accuracy of judging the road surface condition from sound is improved.

[0022] A program as one aspect of the present disclosure causes a computer to perform the following processes: analyzing time series data obtained by measuring sounds generated when a vehicle is traveling on a road using a sound sensor attached to the vehicle, and generating analysis data indicating changes in signal strength over time for each frequency component; inputting the analysis data into an identification model for identifying the road surface condition of the road, thereby obtaining an identification result of the road surface condition from the identification model; and determining the road surface condition by referring to the identification result. According to this aspect, the accuracy of judging the road surface condition from sound is improved. Effect of the Invention

[0023] According to the present disclosure, the accuracy of determining road surface conditions from sound is improved. [Brief description of the drawings]

[0024] [Figure 1] 1 is a diagram illustrating a configuration of a road surface condition determination system according to an embodiment of the present disclosure. [Diagram 2] FIG. 4 is a diagram showing a procedure for determining a road surface condition according to an embodiment of the present disclosure. [Diagram 3] FIG. 13 is a diagram illustrating an example of analysis data according to an embodiment of the present disclosure. [Figure 4] 1 is a block diagram showing a configuration of a road surface condition determination device according to an embodiment of the present disclosure. [Diagram 5] 4 is a flowchart showing an operation of the road surface condition determination system according to the embodiment of the present disclosure. [Figure 6] 4 is a flowchart showing an operation of the road surface condition determination device according to the embodiment of the present disclosure. [Figure 7] 4 is a flowchart showing an operation of the road surface condition determination device according to the embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0026] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0027] The configuration of a road surface condition determination system 10 according to this embodiment will be described with reference to FIG.

[0028] The road surface condition determination system 10 includes a road surface condition determination device 20, an image sensor 30, and a sound sensor 40.

[0029] The road surface condition determination device 20 is a computer. In this embodiment, the road surface condition determination device 20 is a PC, but it may be a mobile device such as a smartphone or a tablet, a server device belonging to a cloud computing system or other computing system, or a dedicated device. "PC" is an abbreviation for personal computer.

[0030] The image sensor 30 is attached to a vehicle 50 traveling on a road 11. Specifically, the image sensor 30 is attached on the dashboard or near the rearview mirror facing the front of the vehicle 50 so as to be able to capture an image of the area in front of the vehicle 50. The image sensor 30 is, for example, a CMOS sensor, a CCD sensor, or a camera incorporating either of these. "CMOS" is an abbreviation for complementary metal oxide semiconductor. "CCD" is an abbreviation for charge coupled device. The image sensor 30 may be able to capture an image of the rear.

[0031] The sound sensor 40 is also attached to the vehicle 50. Specifically, the sound sensor 40 is attached near the tires so as to measure the sound generated from the tires of the vehicle 50. In this embodiment, the sound sensor 40 is attached near the front wheels, but may be attached near the rear wheels. The sound sensor 40 is, for example, a microphone.

[0032] The vehicle 50 is any type of automobile, such as a gasoline vehicle, a diesel vehicle, an HV, a PHV, an EV, or an FCV. "HV" is an abbreviation for hybrid vehicle. "PHV" is an abbreviation for plug-in hybrid vehicle. "EV" is an abbreviation for electric vehicle. "FCV" is an abbreviation for fuel cell vehicle.

[0033] The present embodiment will be outlined with reference to FIGS. 1, 2, and 3. FIG.

[0034] The sound sensor 40 measures the sound generated when the vehicle 50 is traveling on the road 11. The road surface condition determination device 20 analyzes the time series data 61 obtained as the measurement result to generate analysis data 63 as shown in FIG. 3. The analysis data 63 is data showing the change over time in the signal strength for each frequency component. The road surface condition determination device 20 acquires a first identification result, which is an identification result of the road surface condition of the road 11, based on the generated analysis data 63. Specifically, the road surface condition determination device 20 inputs the analysis data 63 to the first identification model 71 to acquire the first identification result from the first identification model 71. The road surface condition determination device 20 judges the road surface condition by referring to the acquired first identification result. Therefore, according to this embodiment, the accuracy of road surface condition determination by sound is improved. The analysis data 63 in FIG. 3 changes depending on the traveling speed even for the same road surface, so that when acquiring the identification result, it is preferable to unify the traveling speed within a certain range or to perform correction depending on the traveling speed.

[0035] The first identification model 71 may be any identification model for identifying road surface conditions, but in this embodiment, it includes a trained model that has undergone machine learning. For example, by analyzing time-series data previously obtained by the sound sensor 40 to generate past analysis data, and linking this analysis data with the actual road surface conditions as labels, it is possible to create training data for machine learning. Then, by using this training data, it is possible to perform machine learning using a known machine learning algorithm, such as a neural network or deep learning, to generate a trained model as the first identification model 71. According to this embodiment, it is possible to improve the accuracy of road surface condition determination by sound by having the first identification model 71 learn a large amount of data.

[0036] In this embodiment, the first identification model 71 outputs a probability for each label as the first identification result. The probability for each label is the probability that the road surface condition corresponds to the class corresponding to each label. That is, in this embodiment, the first identification result includes the probability that the road surface condition corresponds to each of a plurality of classes.

[0037] In this embodiment, seven classes are defined as multiple classes: "dry," "semi-humid," "wet and moist," "sherbet," "accumulated snow," "packed snow," and "frozen." However, instead of these seven classes, any two or more classes may be defined, such as five classes: "dry," "semi-humid," "wet and moist," "sherbet," and "snow."

[0038] In this embodiment, the analysis data 63 includes a spectrogram, but may include an image generated using another frequency analysis method, such as a scalogram, or data in a format other than an image generated using any frequency analysis method. FIG. 3 shows a first spectrogram 64, a second spectrogram 65, and a third spectrogram 66 as examples of the analysis data 63. In each spectrogram, the horizontal axis represents the time axis, the vertical axis represents the frequency axis, and the color shading represents the signal intensity. The first spectrogram 64 is an example of a spectrogram of a sound generated when the vehicle 50 is running on a semi-humid road surface. The second spectrogram 65 is an example of a spectrogram of a sound generated when the vehicle 50 is running on a wet road surface. The third spectrogram 66 is an example of a spectrogram of a sound generated when the vehicle 50 is running on a slush road surface.

[0039] The image sensor 30 captures an image of the road 11. The road surface condition determination device 20 acquires a second identification result, which is an identification result of the road surface condition of the road 11, based on image data 62 obtained as a captured image, separately from the first identification result. Specifically, the road surface condition determination device 20 inputs the image data 62 to the second identification model 72, thereby acquiring the second identification result from the second identification model 72. The road surface condition determination device 20 judges the road surface condition by referring to not only the first identification result but also the acquired second identification result. Therefore, according to this embodiment, the accuracy of road surface condition determination is further improved.

[0040] The second identification model 72 may be any identification model for identifying road surface conditions, but in this embodiment, it includes a trained model that has undergone machine learning. For example, by linking actual road surface conditions as labels to image data previously obtained by the image sensor 30, training data for machine learning can be created. Then, by using this training data, machine learning can be performed using a known machine learning algorithm, such as a neural network or deep learning, to generate a trained model as the second identification model 72. According to this embodiment, the accuracy of road surface condition determination from images can be improved by having the second identification model 72 learn a large amount of data.

[0041] In this embodiment, the second classification model 72 outputs a probability for each label as the second classification result. That is, in this embodiment, the second classification result includes the probability that the road surface condition falls into each of a plurality of classes, just like the first classification result.

[0042] The configuration of a road surface condition determining device 20 according to this embodiment will be described with reference to FIG.

[0043] The road surface condition determination device 20 includes a control unit 21 , a storage unit 22 , a communication unit 23 , an input unit 24 , and an output unit 25 .

[0044] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 executes processes related to the operation of the road surface condition determination device 20 while controlling each part of the road surface condition determination device 20.

[0045] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM or a ROM. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The storage unit 22 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used in the operation of the road surface condition determination device 20 and data obtained by the operation of the road surface condition determination device 20.

[0046] The communication unit 23 includes at least one communication interface. The communication interface is, for example, a LAN interface, an interface compatible with a mobile communication standard such as LTE, the 4G standard, or the 5G standard, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "LAN" is an abbreviation for local area network. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 23 receives data used in the operation of the road surface condition determination device 20, and transmits data obtained by the operation of the road surface condition determination device 20.

[0047] The input unit 24 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen integral with a display, a camera, or a microphone. The input unit 24 accepts an operation to input data used for the operation of the road surface condition determination device 20. The input unit 24 may be connected to the road surface condition determination device 20 as an external input device instead of being provided in the road surface condition determination device 20. As the connection interface, for example, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used. "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface.

[0048] The output unit 25 includes at least one output interface. The output interface is, for example, a display or a speaker. The display is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescence. The output unit 25 outputs data obtained by the operation of the road surface condition determination device 20. The output unit 25 may be connected to the road surface condition determination device 20 as an external output device instead of being provided in the road surface condition determination device 20. As the connection interface, for example, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0049] The functions of the road surface condition determination device 20 are realized by executing a program according to this embodiment in a processor serving as a control unit 21. That is, the functions of the road surface condition determination device 20 are realized by software. The program causes a computer to execute the operations of the road surface condition determination device 20, thereby causing the computer to function as the road surface condition determination device 20. That is, the computer functions as the road surface condition determination device 20 by executing the operations of the road surface condition determination device 20 in accordance with the program.

[0050] The program may be stored in a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include a flash memory, a magnetic recording device, an optical disk, a magneto-optical recording medium, or a ROM. The program may be distributed, for example, by selling, transferring, or lending a portable medium such as an SD card, DVD, or CD-ROM on which the program is stored. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer. The program may be provided as a program product.

[0051] A computer temporarily stores in a main storage device, for example, a program stored in a portable medium or a program transferred from a server. Then, the computer reads the program stored in the main storage device with a processor, and executes processing according to the read program with the processor. The computer may read the program directly from a portable medium and execute processing according to the program. The computer may execute processing according to the received program each time a program is transferred from the server to the computer. Processing may be executed by a so-called ASP-type service that realizes functions only by execution instructions and result acquisition without transferring a program from the server to the computer. "ASP" is an abbreviation for application service provider. A program is information used for processing by a computer, and includes those equivalent to a program. For example, data that is not a direct command to a computer but has the property of defining computer processing falls under "those equivalent to a program."

[0052] A part or all of the functions of the road surface condition determination device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. In other words, a part or all of the functions of the road surface condition determination device 20 may be realized by hardware.

[0053] The operation of the road surface condition determination system 10 according to this embodiment will be described with reference to Fig. 5. This operation corresponds to the road surface condition determination method according to this embodiment.

[0054] In step S1, the image sensor 30 captures an image of the road 11. Image data 62 obtained as a captured image is stored in a medium such as a flash memory or a magnetic recording device mounted on the vehicle 50, or is transmitted to the outside via a network such as the Internet by a communication device mounted on the vehicle 50. At substantially the same time as the image of the road 11 is captured, the sound sensor 40 measures the sound generated when the vehicle 50 is traveling on the road 11. Time series data 61 obtained as a measurement result is stored in a medium mounted on the vehicle 50, like the image data 62, or is transmitted to the outside via a network by a communication device mounted on the vehicle 50.

[0055] In step S2, the control unit 21 of the road surface condition determination device 20 analyzes the time-series data 61 obtained in step S1 to generate analysis data 63 indicating the change over time in the signal strength for each frequency component. This process may be performed in any order, but in this embodiment, it is performed in the order shown in FIG.

[0056] 6, the control unit 21 acquires the time series data 61 stored in a medium mounted on the vehicle 50 in step S1 via the communication unit 23 or another interface conforming to a standard such as USB. Alternatively, the control unit 21 acquires the time series data 61 by receiving the time series data 61 transmitted by a communication device mounted on the vehicle 50 in step S1 via the communication unit 23.

[0057] 6, the control unit 21 extracts data of a certain range of frequencies from the time series data 61 acquired in step S201, and also extracts data of a certain range of signal strength from the time series data 61. That is, the control unit 21 extracts data of a certain range of frequencies and a certain range of signal strength from the time series data 61. The frequency range is preferably 1 kHz or more and 20 kHz or less. The signal strength range is preferably 40 dB or more and 60 dB or less. That is, the control unit 21 preferably extracts data of a frequency of 1 kHz or more and 20 kHz or less and a signal strength of 40 dB or more and 60 dB or less from the time series data 61.

[0058] 6, the control unit 21 analyzes the data extracted in step S202 to generate analysis data 63. Specifically, the control unit 21 performs a frequency analysis of the data extracted in step S202 to generate a spectrogram.

[0059] In step S3, the control unit 21 of the road surface condition determination device 20 obtains a first identification result based on the analysis data 63 generated in step S2. Specifically, the control unit 21 inputs the analysis data 63 to a first identification model 71 for identifying the road surface condition of the road 11, and obtains the identification result of the road surface condition as the first identification result from the first identification model 71. In this embodiment, the first identification result includes the probability that the road surface condition falls into each of seven classes, namely, "dry", "semi-humid", "wet and moist", "sherbet", "snow-covered", "packed snow", and "frozen".

[0060] In step S4, the control unit 21 of the road surface condition determination device 20 acquires a second identification result based on the image data 62 acquired in step S1. Specifically, the control unit 21 acquires the image data 62 stored in a medium mounted on the vehicle 50 in step S1 via the communication unit 23 or another interface corresponding to a standard such as USB. Alternatively, the control unit 21 acquires the image data 62 by receiving the image data 62 transmitted by a communication device mounted on the vehicle 50 in step S1 via the communication unit 23. The control unit 21 extracts a road surface image from the acquired image data 62. As a method for extracting the road surface image, any method may be used, such as a method of cutting out an area of ​​a fixed number of pixels as a road surface image, a method of cutting out a road surface area recognized by performing image analysis as a road surface image, or a method of cutting out a road surface area identified by using machine learning as a road surface image. The road surface image may be cut out for each lane. The control unit 21 inputs the extracted road surface image to a second classification model 72 for classifying the road surface condition of the road 11, and thereby acquires the classification result of the road surface condition as a second classification result from the second classification model 72. In this embodiment, like the first classification result, the second classification result includes the probability that the road surface condition falls into each of the seven classes, namely, "dry", "semi-humid", "wet and moist", "sherbet", "snow-covered", "packed snow", and "frozen".

[0061] In step S5, the control unit 21 of the road surface condition determination device 20 determines the road surface condition by referring to the first identification result and the second identification result acquired in step S3 and step S4, respectively. This process may be performed in any order, but in this embodiment, it is performed in the order shown in FIG.

[0062] 7, the control unit 21 weights each of the first and second identification results according to the time period in which the sound was measured in step S1. For example, if the time period is nighttime, when the accuracy of road surface condition determination using images is likely to decrease, the control unit 21 weights the probability that the sound will be included in the first identification result more than the probability that the sound will be included in the second identification result. If the time period is daytime, when the accuracy of road surface condition determination using images is likely to increase, the control unit 21 weights the probability that the sound will be included in the second identification result the same as the weighting of the probability that the sound will be included in the first identification result or weights the probability that the sound will be included in the first identification result more than the weighting of the probability that the sound will be included in the first identification result.

[0063] In step S502 of Fig. 7, the control unit 21 weights each of the first and second classification results according to the outside air temperature when the sound was measured in step S1. For example, if the outside air temperature is low enough to cause black ice, the control unit 21 weights the probability of the first classification result being included more heavily than the probability of the second classification result being included. If the outside air temperature is high enough not to cause black ice, the control unit 21 weights the probability of the second classification result being included the same as the weight of the probability of the first classification result being included or weights the probability of the first classification result being included more heavily than the weight of the probability of the first classification result being included.

[0064] In step S503 of FIG. 7, the control unit 21 compares the probabilities included in the first and second identification results weighted in steps S501 and S502, respectively, to determine the class to which the road surface condition corresponds. For example, the control unit 21 determines the class with the highest overall probability in the first and second identification results as the class to which the road surface condition corresponds. Suppose that the probabilities of "dry", "semi-humid", and "moist and wet" included in the first identification result are 10%, 80%, and 10%, respectively, and the probabilities of "dry", "semi-humid", and "moist and wet" included in the second identification result are 50%, 30%, and 20%, respectively. In this case, since the probability of "semi-humid" included in the first identification result is the highest overall, the control unit 21 determines that the road surface condition is a semi-humid state. Alternatively, the control unit 21 may determine the class with the highest average probability in the first and second identification results as the class to which the road surface condition corresponds. Suppose that the probabilities of "dry", "semi-humid", and "moist" included in the first classification result are 30%, 40%, and 30%, respectively, and the probabilities of "dry", "semi-humid", and "moist" included in the second classification result are 50%, 30%, and 20%, respectively. In this case, since the average probability of "dry" is the highest, the control unit 21 determines that the road surface condition is dry.

[0065] As described above, in this embodiment, the control unit 21 of the road surface condition determination device 20 analyzes the time series data 61 obtained by measuring the sound generated when the vehicle 50 is traveling on the road 11 using the sound sensor 40 attached to the vehicle 50, and generates analysis data 63 indicating the change over time in the signal strength for each frequency component. The control unit 21 inputs the generated analysis data 63 to a first identification model 71 for identifying the road surface condition of the road 11, thereby acquiring an identification result of the road surface condition from the first identification model 71. The control unit 21 judges the road surface condition by referring to the acquired identification result. Therefore, according to this embodiment, the accuracy of road surface condition judgment based on sound is improved.

[0066] In this embodiment, the control unit 21 of the road surface condition determination device 20 obtains a second identification result, which is an identification result of the road surface condition, based on image data 62 obtained by capturing an image of the road 11 by the image sensor 30 attached to the vehicle 50, in addition to a first identification result, which is an identification result obtained from the first identification model 71. The control unit 21 determines the road surface condition by referring to the obtained first identification result and second identification result. Therefore, according to this embodiment, the accuracy of the road surface condition determination is further improved.

[0067] As a modification of this embodiment, the analysis data 63 may be synchronized with the image data 62. In such a modification, the image data 62 includes a road surface image from a first point to a second point on the road 11. In step S202 of FIG. 6, the control unit 21 of the road surface condition determination device 20 extracts data from the time-series data 61, the data being from a first time point when the vehicle 50 passes the first point to a second time point when the vehicle 50 passes the second point. According to this modification, the width of the analysis data 63, which is data having a width before and after a certain point in the time direction, can be matched with the length of the section of the road 11 included in the road surface image. That is, synchronization on the time axis can be achieved.

[0068] As a modification of this embodiment, meteorological information such as a weather forecast may be reflected. In such a modification, the control unit 21 of the road surface condition determination device 20 acquires meteorological data indicating the weather at the time when the sound was measured. The control unit 21 weights the probability of each of the first and second classification results being included for each class based on the acquired meteorological data. This modification further improves the accuracy of road surface condition determination.

[0069] Weather information is an example of information caused by the external environment. The time zone information referred to in step S501 of Fig. 7 and the outside temperature information referred to in step S502 of Fig. 7 are also examples of information caused by the external environment. Information other than the weather information, the time zone information, and the outside temperature information may be reflected as information caused by the external environment. That is, the control unit 21 of the road surface condition determination device 20 may perform weighting according to information caused by the external environment other than the weather information, the time zone information, and the outside temperature information.

[0070] The present disclosure is not limited to the above-mentioned embodiment. For example, two or more blocks shown in the block diagram may be integrated, or one block may be divided. Instead of executing two or more steps shown in the flowchart in chronological order as described, they may be executed in parallel or in a different order depending on the processing capacity of the device executing each step, or as necessary. Other modifications are possible without departing from the spirit of the present disclosure.

[0071] For example, the road surface condition determination device 20 may be an in-vehicle device. That is, a vehicle 50 may be equipped with the road surface condition determination system 10. According to this example, the road surface condition determination based on sound can be performed by the vehicle 50.

[0072] For example, the image sensor 30 may be omitted. In such an example, the control unit 21 of the road surface condition determination device 20 determines the road surface condition without referring to the second identification result. Alternatively, instead of the image sensor 30, an acceleration sensor or an infrared sensor may be combined with the sound sensor 40 to determine the road surface condition.

[0073] For example, the series of processes from step S1 to step S5 may be executed for each section of a fixed distance, such as 100 meters, traveled by the vehicle 50. Alternatively, the process of step S1 may be executed for each section, and then the series of processes from step S2 to step S5 may be executed for the data obtained in each section. Alternatively, the series of processes from step S1 to step S5 may be executed at fixed time intervals.

[0074] For example, the processes in step S501, step S502, or both of these steps in FIG. 7 may be omitted. [Industrial Applicability]

[0075] The present disclosure relates to a road surface condition determination device, a road surface condition determination system, a vehicle, a road surface condition determination method, and a program. [Explanation of symbols]

[0076] 10: Road surface condition determination system, 11: Road, 20: Road surface condition determination device, 21: Control unit, 22: Memory unit, 23: Communication unit, 24: Input unit, 25: Output unit, 30: Image sensor, 40: Sound sensor, 50: Vehicle, 61: Time series data, 62: Image data, 63: Analysis data, 64: First spectrogram, 65: Second spectrogram, 66: Third spectrogram, 71: First discrimination model, 72: Second discrimination model

Claims

1. a control unit that analyzes time-series data obtained by measuring sounds generated when a vehicle is traveling on a road with a sound sensor attached to the vehicle, generates analysis data indicating changes in signal strength for each frequency component over time, and inputs the generated analysis data into an identification model for identifying a road surface condition of the road, thereby obtaining an identification result of the road surface condition from the identification model, and obtains, separately from a first identification result which is the identification result obtained from the identification model, a second identification result which is the identification result of the road surface condition based on image data obtained by capturing an image of the road with an image sensor attached to the vehicle, and judges the road surface condition by referring to the obtained first identification result and second identification result, the first classification result and the second classification result each include a probability that the road surface condition corresponds to each of a plurality of classes, the control unit determines a class to which the road surface condition corresponds by comparing a probability included in the first classification result and a probability included in the second classification result, The control unit acquires meteorological data indicating the weather at the time the sound was measured, and weights the probability that the first classification result and the second classification result are included for each class based on the acquired meteorological data.

2. The image data includes a road surface image from a first point to a second point of the road, 2. The road surface condition determination device according to claim 1, wherein the control unit extracts data from the time series data from a first time point when the vehicle passes the first point to a second time point when the vehicle passes the second point, and analyzes the extracted data to generate the analysis data.

3. a control unit that analyzes time-series data obtained by measuring sounds generated when a vehicle is traveling on a road with a sound sensor attached to the vehicle, generates analysis data indicating changes in signal strength for each frequency component over time, and inputs the generated analysis data into an identification model for identifying a road surface condition of the road, thereby obtaining an identification result of the road surface condition from the identification model, and obtains, separately from a first identification result which is the identification result obtained from the identification model, a second identification result which is the identification result of the road surface condition based on image data obtained by capturing an image of the road with an image sensor attached to the vehicle, and judges the road surface condition by referring to the obtained first identification result and second identification result, The control unit weights each of the first identification result and the second identification result depending on the time period during which the sound was measured.

4. a control unit that analyzes time-series data obtained by measuring sounds generated when a vehicle is traveling on a road with a sound sensor attached to the vehicle, generates analysis data indicating changes in signal strength for each frequency component over time, and inputs the generated analysis data into an identification model for identifying a road surface condition of the road, thereby obtaining an identification result of the road surface condition from the identification model, and obtains, separately from a first identification result which is the identification result obtained from the identification model, a second identification result which is the identification result of the road surface condition based on image data obtained by capturing an image of the road with an image sensor attached to the vehicle, and judges the road surface condition by referring to the obtained first identification result and second identification result, The control unit weights each of the first identification result and the second identification result in accordance with an outside air temperature when the sound is measured.

5. a control unit that analyzes time-series data obtained by measuring sounds generated when a vehicle is traveling on a road with a sound sensor attached to the vehicle, generates analysis data indicating changes in signal strength for each frequency component over time, and inputs the generated analysis data into an identification model for identifying a road surface condition of the road, thereby obtaining an identification result of the road surface condition from the identification model, and obtains, separately from a first identification result which is the identification result obtained from the identification model, a second identification result which is the identification result of the road surface condition based on image data obtained by capturing an image of the road with an image sensor attached to the vehicle, and judges the road surface condition by referring to the obtained first identification result and second identification result, The control unit weights the first identification result and the second identification result in accordance with information caused by an external environment.

6. The road surface condition determination device according to claim 1 , wherein the control unit extracts data of a certain range of frequencies from the time series data, and analyzes the extracted data to generate the analysis data.

7. 7. The road surface condition determination device according to claim 1, wherein the control unit extracts data on a certain range of signal strength from the time series data, and analyzes the extracted data to generate the analysis data.

8. The road surface condition determination device according to claim 1 , wherein the identification model includes a trained model that has been subjected to machine learning.

9. The road surface condition determination device according to claim 1 , wherein the analysis data includes a spectrogram or a scalogram.

10. The road surface condition determination device according to any one of claims 1 to 9, The sound sensor A road surface condition determination system comprising:

11. A vehicle comprising the road surface condition determination system according to claim 10.

12. A sound generated when a vehicle is traveling on a road is measured by a sound sensor attached to the vehicle; The time series data obtained as the measurement result is analyzed by a computer to generate analysis data showing the change over time in the signal strength for each frequency component. inputting the analysis data into an identification model for identifying a road surface condition of the road by the computer, thereby obtaining an identification result of the road surface condition from the identification model; a second classification result, which is a classification result of the road surface condition, is obtained by the computer based on image data obtained by capturing an image of the road by an image sensor attached to the vehicle, separately from a first classification result, which is a classification result obtained from the classification model; A road surface condition determination method for determining the road surface condition by referring to the first identification result and the second identification result by the computer, the first classification result and the second classification result each include a probability that the road surface condition corresponds to each of a plurality of classes, the computer determines a class to which the road surface condition belongs by comparing a probability included in the first classification result and a probability included in the second classification result; The computer acquires meteorological data indicating the weather at the time the sound was measured, and weights the probability that the first classification result and the second classification result are included for each class based on the acquired meteorological data.

13. A sound generated when a vehicle is traveling on a road is measured by a sound sensor attached to the vehicle; The time series data obtained as the measurement result is analyzed by a computer to generate analysis data showing the change over time in the signal strength for each frequency component. inputting the analysis data into an identification model for identifying a road surface condition of the road by the computer, thereby obtaining an identification result of the road surface condition from the identification model; a second classification result, which is a classification result of the road surface condition, is obtained by the computer based on image data obtained by capturing an image of the road by an image sensor attached to the vehicle, separately from a first classification result, which is a classification result obtained from the classification model; A road surface condition determination method for determining the road surface condition by referring to the first identification result and the second identification result by the computer, The method of determining road surface conditions, wherein the computer weights each of the first identification result and the second identification result depending on the time period during which the sound was measured.

14. A sound generated when a vehicle is traveling on a road is measured by a sound sensor attached to the vehicle; The time series data obtained as the measurement result is analyzed by a computer to generate analysis data showing the change over time in signal strength for each frequency component. inputting the analysis data into an identification model for identifying a road surface condition of the road by the computer, thereby obtaining an identification result of the road surface condition from the identification model; a second classification result, which is a classification result of the road surface condition, is obtained by the computer based on image data obtained by capturing an image of the road by an image sensor attached to the vehicle, separately from a first classification result, which is a classification result obtained from the classification model; A road surface condition determination method for determining the road surface condition by referring to the first identification result and the second identification result by the computer, The computer weights each of the first identification result and the second identification result in accordance with an outside air temperature when the sound is measured.

15. A sound generated when a vehicle is traveling on a road is measured by a sound sensor attached to the vehicle; The time series data obtained as the measurement result is analyzed by a computer to generate analysis data showing the change over time in the signal strength for each frequency component. inputting the analysis data into an identification model for identifying a road surface condition of the road by the computer, thereby obtaining an identification result of the road surface condition from the identification model; a second classification result, which is a classification result of the road surface condition, is obtained by the computer based on image data obtained by capturing an image of the road by an image sensor attached to the vehicle, separately from a first classification result, which is a classification result obtained from the classification model; A road surface condition determination method for determining the road surface condition by referring to the first identification result and the second identification result by the computer, The method of determining road surface conditions, wherein the computer weights the first identification result and the second identification result in accordance with information caused by an external environment.

16. A program that causes a computer to execute the operation of a road surface condition determination device described in any one of claims 1 to 9.

Images