Service providing apparatus and method for analyzing road surface condition based on driving noise, and non-transitory computer readable medium having computer program recorded thereon

Abstract

The present invention relates to a service providing device and method for analyzing road surface conditions based on driving noise, and a recording medium on which a computer program is recorded. More specifically, the invention relates to a service providing device and method for analyzing road surface conditions based on driving noise, and a recording medium on which a computer program is recorded, which supports rapid response to the occurrence of hazardous elements on the road by determining road surface conditions in real time, such as black ice, in road sections without cameras or in situations where cameras are malfunctioning, by training a deep learning model with acoustic images of driving noise and driving images of a vehicle, and applying acoustic images of driving noise acquired through an acoustic collection device to the deep learning model after training to generate and provide a predicted driving image of the road surface condition. The present invention can accurately provide visual information regarding the road environment and road surface conditions by providing predicted driving images and predicted result information regarding road surface conditions, even in situations where cameras are unavailable or malfunctioning, or under conditions where visibility is difficult to secure, such as at night or inclement weather. Furthermore, by supporting managers to proactively detect risk factors such as black ice through verification of the predicted result information and to respond quickly to dangerous situations on the road, it can prevent major traffic accidents and significantly improve road safety.

Description

Technology Field

[0001] The present invention relates to a service providing device and method for analyzing road surface conditions based on driving noise, and a recording medium on which a computer program is recorded. More specifically, the invention relates to a service providing device and method for analyzing road surface conditions based on driving noise, and a recording medium on which a computer program is recorded, which supports rapid response to the occurrence of hazardous elements on the road by determining road surface conditions in real time, such as black ice, in road sections without cameras or in situations where cameras are malfunctioning, by training a deep learning model with acoustic images of driving noise and driving images of a vehicle, and applying acoustic images of driving noise acquired through an acoustic collection device to the deep learning model after training to generate and provide a predicted driving image of the road surface condition. Background Technology

[0002] Black ice currently forming on roads is one of the factors seriously threatening winter road safety. It is a major cause of large-scale traffic accidents because snow or rain freezes thinly on the road surface, making it difficult to identify with the naked eye.

[0003] In particular, the risk of accidents increases further at night or in weather conditions where visibility is poor, as it is difficult for drivers to recognize black ice on the road in advance.

[0004] To address these problems, various technologies regarding road hazard detection solutions are being developed.

[0005] Existing road hazard detection systems have primarily been implemented by monitoring road surface conditions through image analysis of footage received from cameras; however, there were limitations in accurately assessing road surface conditions in sections without cameras, when cameras were malfunctioning, or during nighttime or adverse weather conditions.

[0006] In addition, existing technologies have a problem in that they lack the ability to identify and visualize road surface conditions in real time in environments with limited visibility when detecting and analyzing hazards on the road.

[0007] In particular, there are limitations in that it is difficult to provide visual information about road conditions at night, in sections where cameras are not installed, or in situations where cameras are malfunctioning, making it difficult to respond appropriately in a timely manner.

[0008] Therefore, there is a need for a technical solution capable of identifying road surface conditions and vehicle driving status in real time and visualizing and providing this information, even in environments where visibility is limited.

[0009] In particular, there is a need for an innovative approach capable of rapidly detecting and responding to risk factors such as black ice not only during the day but also at night. Prior art literature

[0010] Korean Published Patent No. 10-2019-0073144 [Title of Invention: Road Surface Condition Management System] The problem to be solved

[0011] The present invention aims to support a rapid response to road hazards by determining road surface conditions, such as black ice, in real time in road sections without cameras or in situations where cameras are malfunctioning, by training a deep learning model with acoustic images and driving images of vehicle driving noise, applying the acoustic images of currently measured driving noise to the deep learning model after the training of the deep learning model is completed to generate a driving image predicted for the road surface condition, and analyzing and providing the road surface condition based on this. means of solving the problem

[0012] A service providing device for road surface condition analysis based on driving noise according to an embodiment of the present invention may include: an acoustic processing unit that generates an acoustic image visualizing the waveform of an audio signal based on driving noise measurement information including an audio signal for a specific road section received from an acoustic collection device that measures driving noise and generates an audio signal; an image prediction unit that applies a current acoustic image generated through the acoustic processing unit based on the driving noise measurement information to an image prediction model that learns the correlation between an acoustic image and a driving image based on a plurality of different learning data including an acoustic image and a driving image generated at the time of capturing the driving image, each corresponding to a plurality of different driving images captured for the specific road section, and generates a predicted driving image corresponding to the current acoustic image through the image prediction model; a road surface condition prediction unit that labels road surface condition information for each of the plurality of driving images and trains the road surface condition prediction model, and then applies the predicted driving image to the road surface condition prediction model after training to generate road surface condition prediction information; and a result providing unit that generates and provides prediction result information including the predicted driving image and the road surface condition prediction information.

[0013] As an example related to the present invention, the road surface condition prediction information may be characterized by including a type of road surface condition for at least one of a normal road surface condition, a freezing condition, a black ice condition, a wet road surface condition, a crack condition, and a pothole condition.

[0014] As an example related to the present invention, the image prediction model is trained using training data including driving images taken during the day, excluding training data including driving images taken at night, and the image prediction unit may be characterized by generating and outputting a predicted driving image for the day corresponding to the current acoustic image through the image prediction model for the current acoustic image generated based on an audio signal measured at night.

[0015] As an example related to the present invention, the driving image is composed of an infrared driving image, which is an infrared image captured through an infrared camera; the image prediction model is learned based on a plurality of infrared driving images captured during the day and night, a plurality of acoustic images corresponding to each, and the plurality of infrared driving images; the image prediction unit generates a predicted driving image in the form of an infrared image corresponding to the current acoustic image through the image prediction model; the road surface condition prediction unit generates road surface condition prediction information corresponding to the predicted driving image through the road surface condition prediction model; and the result providing unit may be characterized by analyzing temperature-based color information for the predicted driving image and, if a danger area is identified, including marking information marked in the danger area in the predicted driving image.

[0016] As an example related to the present invention, the service providing device may further include a warning processing unit that generates warning information when the type of road surface condition according to the road surface condition prediction information corresponds to a preset type of dangerous road surface condition, and transmits the generated warning information to a preset external server or user terminal.

[0017] As an example related to the present invention, the result providing unit may be characterized by, when an administrator who has verified the prediction result information modifies the road surface condition type according to the road surface condition prediction information because it does not match the type of road surface condition actually observed, generating feedback learning data including the actual road surface condition type and a predicted driving image according to the prediction result information, and then improving the prediction accuracy of the road surface condition prediction model by further training the road surface condition prediction model using the feedback learning data through linkage with the road surface condition prediction unit.

[0018] As an example related to the present invention, the service providing device, in conjunction with the image prediction unit, analyzes the predicted driving image to identify one or more vehicle objects included in the predicted driving image, identifies a waveform section in the acoustic image that exceeds a predetermined amplitude threshold as a vehicle approach event, calculates a slope over a preset time period according to the amplitude change in the identified vehicle approach event section, applies the calculated slope value to a speed prediction model in which the correlation between slope and speed is defined or learned to calculate the driving speed of a specific vehicle corresponding to the vehicle approach event, analyzes the frequency characteristics and sound intensity corresponding to the vehicle approach event in the acoustic image to calculate the actual distance from the acoustic collection device to the vehicle, and then converts the calculated actual distance into a pixel distance on the predicted driving image through a preset distance conversion model in which the conversion relationship between the actual distance and the pixel distance on the predicted driving image is defined or learned, determines a specific vehicle object corresponding to the specific vehicle that is closest to the converted pixel distance from a reference position of the acoustic collection device preset within the predicted driving image among the identified one or more vehicle objects, and provides distance information regarding the actual distance mapped to the specific vehicle object on the predicted driving image. It may further include a vehicle monitoring unit that generates monitoring result information including at least one of the driving speed information of a specific vehicle calculated above.

[0019] As an example related to the present invention, the service providing device may further include a traffic control unit that calculates the time at which the specific vehicle is expected to reach a preset danger zone based on the calculated driving speed of the specific vehicle and the actual distance from the sound collection device to the vehicle, and if the expected time of arrival is less than or equal to a preset threshold, generates an approach notification information and transmits it to a preset output device so that the approach notification information is output through the output device.

[0020] A service providing method for road surface condition analysis based on driving noise of a service providing device according to an embodiment of the present invention may include: a step of generating an acoustic image that visualizes the waveform of an audio signal based on driving noise measurement information including an audio signal for a specific road section received from an acoustic acquisition device that measures driving noise and generates an audio signal; a step of applying a current acoustic image generated based on the driving noise measurement information to an image prediction model that learns the correlation between an acoustic image and a driving image based on a plurality of different learning data including an acoustic image and a driving image generated at the time of capturing the driving image, each corresponding to a plurality of different driving images captured for the specific road section, thereby generating a predicted driving image corresponding to the current acoustic image through the image prediction model; a step of labeling road surface condition information for each of the plurality of driving images and training the road surface condition prediction model, and then applying the predicted driving image to the road surface condition prediction model that has completed training to generate road surface condition prediction information; and a step of generating and providing prediction result information including the predicted driving image and the road surface condition prediction information.

[0021] A computer-readable non-volatile recording medium according to an embodiment of the present invention may have a computer program recorded thereon for enabling a computer to perform the above-described method for providing a service for road surface condition analysis based on driving noise. Effects of the invention

[0022] A service providing device according to an embodiment of the present invention generates an acoustic image based on an audio signal received from an acoustic collection device, applies it to a deep learning-based image prediction model to generate a predicted driving image that predicts a driving image of the road, and then analyzes the road surface condition based on the predicted driving image through a road surface condition prediction model. This allows for the accurate provision of visual information regarding the road environment and road surface condition information by providing the predicted driving image and predicted result information regarding the road surface condition, even in situations where there is no camera or the camera is malfunctioning, or under conditions where visibility is difficult to secure, such as at night or in bad weather. Furthermore, by supporting a manager to proactively detect risk factors such as black ice through verification of the predicted result information and to respond quickly to dangerous situations on the road, it is possible to prevent major traffic accidents and significantly improve road safety.

[0023] In addition, the present invention visualizes hazardous elements such as freezing sections or black ice using temperature-based color information through the generation of predictive driving images in the form of infrared images, and automatically identifies and marks low-temperature areas within the road, thereby supporting road managers in quickly recognizing and responding to hazardous road conditions regardless of the time of day or weather.

[0024] In addition, the present invention provides a function to extract the location and speed of a vehicle through audio waveform analysis, thereby enabling the detection of a vehicle's approach based solely on driving noise and the generation of an approach alert signal by calculating the estimated time of arrival at a danger zone, which can overcome the limitations of camera-based monitoring and significantly improve the efficiency and reliability of traffic safety management. Brief explanation of the drawing

[0025] FIG. 1 is a configuration diagram of a service providing system for road surface condition analysis based on driving noise according to an embodiment of the present invention. FIG. 2 is a configuration diagram of a service providing device for road surface condition analysis based on driving noise according to an embodiment of the present invention. FIG. 3 is an example diagram of the operation of a learning and prediction process for predicting a driving image of a service providing device for road surface condition analysis based on driving noise according to an embodiment of the present invention. FIG. 4 is an example diagram of a predicted driving image generated using an acoustic image after training is completed and training data trained on an image prediction model for predicting a driving image of a service providing device for road surface condition analysis based on driving noise according to an embodiment of the present invention. FIG. 5 is an example diagram of the operation of a learning and prediction process for predicting a road surface condition of a service providing device for road surface condition analysis based on driving noise according to an embodiment of the present invention. FIG. 6 is an example diagram of the process of generating training data to be trained on a road surface condition prediction model for predicting the road surface condition of a service providing device for road surface condition analysis based on driving noise according to an embodiment of the present invention. FIG. 7 is a flowchart of the operation of a service provision method for road surface condition analysis based on driving noise of a service provision device according to an embodiment of the present invention. Specific details for implementing the invention

[0026] It should be noted that the technical terms used in this invention are used merely to describe specific embodiments and are not intended to limit the invention. Furthermore, unless specifically defined otherwise in this invention, the technical terms used in this invention should be interpreted in the sense generally understood by those skilled in the art to which this invention pertains, and should not be interpreted in an overly broad or overly narrow sense. Additionally, if a technical term used in this invention is an incorrect technical term that fails to accurately express the concept of the invention, it should be replaced with a technical term that can be correctly understood by those skilled in the art. Moreover, general terms used in this invention should be interpreted according to their prior definitions or the context, and should not be interpreted in an overly narrow sense.

[0027] Furthermore, singular expressions used in the present invention include plural expressions unless the context clearly indicates otherwise. Terms such as "composed of" or "comprising" in the present invention should not be interpreted as necessarily including all of the various components or steps described in the invention, and should be interpreted as meaning that some of the components or steps may not be included, or that additional components or steps may be included.

[0028] Additionally, terms including ordinal numbers, such as first, second, etc., used in the present invention may be used to describe components, but the components should not be limited by the terms. The terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0029] Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. Identical or similar components are given the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted.

[0030] Furthermore, in describing the present invention, detailed descriptions of related prior art are omitted if it is determined that such descriptions could obscure the essence of the invention. Additionally, it should be noted that the attached drawings are intended only to facilitate an understanding of the concept of the present invention and should not be interpreted as limiting the concept of the present invention.

[0031] Detailed embodiments of the present invention will be described below with reference to the drawings.

[0032] FIG. 1 is a configuration diagram of a service providing system for road surface condition analysis based on driving noise (hereinafter, service providing system) according to an embodiment of the present invention.

[0033] As described above, a service providing system according to an embodiment of the present invention may be configured to include a sound collecting device (10) installed in a specific road section subject to checking the road surface condition and measuring (or capturing or collecting) noise generated on the road surface as a vehicle travels, a user terminal (20) of a user (manager) who manages the specific road section, and a service providing device (100) that communicates with the sound collecting device (10), the user terminal (20), etc., through a communication network.

[0034] Not all components of the service provision system illustrated in FIG. 1 are essential components, and the service provision system may be implemented with more components than those illustrated in FIG. 1, or with fewer components.

[0035] In addition, the user terminal (20) described in the present invention may include various terminals such as a smartphone equipped with communication functions, a portable terminal, a mobile terminal, a personal digital assistant (PDA), a portable multimedia player (PMP) terminal, a telematics terminal, a navigation terminal, a personal computer, a laptop computer, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., including a smartwatch, a smart glass, a head-mounted display (HMD), etc.), a Wibro terminal, an Internet Protocol Television (IPTV) terminal, a smart TV, a digital broadcasting terminal, an AVN (Audio Video Navigation) terminal, an A / V (Audio / Video) system, a flexible terminal, etc.

[0036] In addition, the communication network described in the present invention may include wired / wireless communication networks, and examples of such wireless communication networks include Wireless LAN (WLAN), DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), IEEE 802.16, Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced), Wireless Mobile Broadband Service (WMBS), and 5G mobile communication service. Bluetooth, LoRa (Long Range), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, etc. may be included.In addition, wired communication networks may include wired LAN (Local Area Network), wired WAN (Wide Area Network), Power Line Communication (PLC), USB communication, Ethernet, serial communication, optical / coaxial cables, etc.

[0037] Referring to FIG. 1, an acoustic collection device (or driving noise measuring device or acoustic measuring device) (10) can be installed around or on a road to collect noise generated as a vehicle drives. The acoustic collection device (10) may include a microphone that converts an acoustic signal received from the outside into an electrical signal, a control circuit that digitizes and processes the converted electrical signal to generate an audio signal and generates driving noise measuring information including the audio signal, a memory that stores the audio signal, and a communication module that transmits the driving noise measuring information to a service providing device (100) under the control of the control circuit.

[0038] At this time, the sound collection device (10) may be installed in multiple units at predetermined intervals on the road, and a unique identifier (ID) corresponding to the road section to which the sound collection device (10) is the target of sound collection is pre-set for each device, so that the service providing device (100) can accurately identify the source of the driving noise measurement information collected from the sound collection device (10) or a specific road section corresponding to the sound collection device (10) based on the unique identifier.

[0039] Additionally, the sound collection device (10) can generate the driving noise measurement information in real time or at a preset interval and transmit it to the service providing device (100).

[0040] Additionally, the user terminal (20) is a device used by a road manager or operator, and the user terminal (20) according to an embodiment of the present invention may be configured to include a terminal communication unit, a terminal storage unit, a terminal display unit, a user input unit, a terminal control unit, etc.

[0041] At this time, the terminal communication unit can communicate with the service providing device (100) through a communication network, the terminal storage unit can store various data necessary for the operation of the user terminal (20), the terminal display unit can display various information, the user input unit can receive user input by the administrator (user), and the terminal control unit can include RAM, ROM, CPU, GPU, bus, etc. and can perform overall control functions of the user terminal (20).

[0042] In addition, the terminal storage unit may be configured to include application-related data for transmitting and receiving various information with the service providing device (100) and processing and displaying the transmitted and received information.

[0043] Accordingly, the user terminal (20) can visually display the predicted result information regarding the road surface condition provided by the service providing device (100) through a web-based interface or a dedicated application, and if necessary, can provide feedback to the service providing device (100) by modifying the information included in the predicted result information.

[0044] Additionally, the service providing device (100) may be configured as a central processing unit that analyzes audio signals received from the sound collection device (10) via a communication network to generate sound images, predicts driving images based thereon, and analyzes road surface conditions, and may be configured in the form of a server, for example.

[0045] At this time, it goes without saying that some or all of the service providing device (100) may be composed of components included in the sound collecting device (10) or user terminal (20).

[0046] According to the above configuration, the service providing device (100) according to the embodiment of the present invention can generate a predicted driving image for the road section based solely on the driving noise of a vehicle according to the driving noise measurement information received from the sound collection device (10), and predict the road surface condition based thereon and provide related data information, thereby supporting effective road management and safety monitoring even in sections where cameras are not installed, situations where cameras are malfunctioning, and conditions where visibility is difficult to secure, such as night or bad weather. This will be explained in detail below with reference to the drawings.

[0047] FIG. 2 is a detailed configuration diagram of the service providing device (100) for road surface condition analysis based on driving noise according to an embodiment of the present invention.

[0048] As described above, a service providing device (100) according to an embodiment of the present invention may be configured to include a communication unit (110), a storage unit (120), a control unit (130), etc.

[0049] First, the communication unit (110) can communicate with one or more sound collection devices (10), user terminals (20), various external devices, etc. through the communication network.

[0050] At this time, the communication unit (110) may communicate with a camera unit (camera device) positioned in a road section where the sound collection device (10) is installed, as needed, and may receive a driving image captured by the camera unit that is necessary for learning and prediction related to the analysis of the road surface condition of the road section from the camera unit.

[0051] Here, the camera unit may generate the driving image by photographing an area corresponding to the road section, and may be a device temporarily installed for a limited period in the road section.

[0052] In addition, the storage unit (120) can store various types of information, and the storage unit (120) can be configured in various forms such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc., and can be configured as a DB or include one or more DBs.

[0053] For example, the storage unit (120) may be configured to include an image generation learning DB (or DB for image generation learning), a road surface condition learning DB (or DB for road surface condition learning), an image prediction model DB, a road surface condition prediction model DB, a prediction result DB, etc.

[0054] At this time, the image generation learning DB is a DB for storing learning data for generating a predicted driving image, and learning data consisting of pairs of acoustic images and corresponding actual driving images (or actual driving images, hereinafter referred to as driving images) may be stored.

[0055] In addition, the road surface condition learning DB is a DB for storing training data for generating road surface condition prediction information, and may store training data including driving images and road surface condition information labeled for the corresponding driving images.

[0056] In addition, the image prediction model DB may store model information, model parameters, etc. regarding the image prediction model, the road surface condition prediction model DB may store model information, model parameters, etc. regarding the road surface condition prediction model, and the prediction result DB may be configured as a DB for storing prediction result information.

[0057] In addition, the image generation learning DB, road surface condition learning DB, image prediction model DB, road surface condition prediction model DB, prediction result DB, etc., may each be configured as individual servers and configured to communicate with and interact with the service providing device (100).

[0058] Additionally, the control unit (130) performs overall control functions of the service providing device (100), and the control unit (130) may include RAM, ROM, CPU, GPU, and a bus, and the RAM, ROM, CPU, GPU, etc. may be connected to each other through a bus.

[0059] Additionally, the communication unit (110) and storage unit (120) may be configured to be included in the control unit (130).

[0060] Additionally, the control unit (130) can perform overall control functions of the service providing device (100) and may be configured to include a sound processing unit (131), an image prediction unit (132), a road surface condition prediction unit (133), a result providing unit (134), a warning processing unit (135), a vehicle monitoring unit (136), a traffic control unit (137), a learning data collection unit (138), etc.

[0061] At this time, at least one of the control unit (130) and the various components constituting the control unit (130) can communicate with the sound collection device (10), user terminal (20), various external devices, etc. through the communication unit (110), and the communication configuration through the communication unit (110) is omitted below.

[0062] A plurality of components constituting the control unit (130) may be implemented by a processor capable of processing data, and each may be implemented by a different processor separately, or may be functionally separated within a single processor.

[0063] In addition, although the following description explains that a plurality of components constituting the control unit (130) are included in the service providing device (100), it is obvious that at least one of the plurality of components constituting the control unit (130) may be configured as an individual server separated from the service providing device (100) and configured to interact with the service providing device (100) through communication with the service providing device (100).

[0064] Hereinafter, the detailed operation configuration of the service providing device (100) according to an embodiment of the present invention will be described, focusing on the configuration of the control unit (130) that performs the overall control function of the service providing device (100).

[0065] Before explaining, the service providing device (100) can manage a plurality of image prediction models corresponding to a plurality of different road sections and a plurality of road surface condition prediction models corresponding to a plurality of different road sections.

[0066] To this end, image prediction models trained for each road section can be stored in the image prediction model DB, and road surface condition prediction models trained for each road section can be stored in the road surface condition prediction model DB.

[0067] Through this, the service providing device (100) can perform an accurate prediction that reflects the characteristics of a specific road section.

[0068] For example, the image prediction model DB may store model information including a road section identifier (ID), road section location information (GPS coordinate range, road name, etc.), model version, model parameters, training completion date and time, model accuracy indicator, etc. for an image prediction model corresponding to a road section for each of multiple different road sections, and through this information, the service providing device (100) can select and utilize an image prediction model optimized for a specific road section.

[0069] In addition, the road surface condition prediction model DB may store model information for a road surface condition prediction model corresponding to a road section for each of multiple different road sections, including a road section identifier (ID), road section location information, model information (model version), model parameters, training completion date and time, model accuracy indicator, etc.

[0070] Through this, it is possible to predict road surface conditions tailored to the characteristics of each road section.

[0071] Accordingly, among the plurality of sound collection devices (10) arranged for each road section and corresponding to each road section, the sound collection device (10) corresponding to a specific road section can transmit the road section identifier (ID) of the specific road section corresponding to the sound collection device (10) and driving noise measurement information including the audio signal to the service providing device (100) whenever an audio signal is collected.

[0072] Accordingly, the service providing device (100) can identify a specific road section in which the sound collecting device (10) is placed based on the road section identifier of the received sound collecting device (10), and can select and load an image prediction model and a road surface condition prediction model corresponding to the identified road section from the image prediction model DB and the road surface condition prediction model DB, respectively, for use.

[0073] At this time, the road section identifier corresponding to the specific road section may be the device identifier of the sound collection device (10) corresponding to the specific road section.

[0074] Alternatively, the storage unit (120) may further include a device DB in which device information corresponding to each of the multiple sound collection devices (10) is stored, and the service providing device (100) may obtain information about a specific road section where the corresponding sound collection device (10) is installed by referring to the device DB based on a road section identifier included in the driving noise measurement information, and by selecting and applying an image prediction model and a road surface condition prediction model corresponding to the specific road section, it may generate an accurate prediction result that reflects the characteristics of the specific road section.

[0075] At this time, the device information may include various information such as a road section identifier corresponding to the sound collection device, device identification information (or device identifier), road section location information corresponding to the road section where the sound collection device is deployed (installed), device specification information, etc.

[0076] Accordingly, when the service providing device (100) receives an audio signal along with a unique identifier from an audio collection device (10) placed in a specific road section, it identifies the road section based on the unique identifier, selects an optimal image prediction model and a road surface condition prediction model suitable for the identified road section to generate an acoustic image from the audio signal, and thereby generates prediction result information including the predicted driving image and road surface condition prediction information described below.

[0077] In the following, the operation between the sound collecting device (10) and the service providing device (100) placed in a specific road section is described as an example, but it is obvious that the operation configuration of the service providing device (100) described below can also be applied to the sound collecting device (10) placed in other road sections.

[0078] Referring to the drawings in relation to the detailed operation configuration of the service providing device (100) above, FIG. 3 is an example diagram of the operation of the learning and prediction process for predicting a driving image of the service providing device (100) according to an embodiment of the present invention, FIG. 4 is an example diagram of the predicted driving image generated using the learning data learned by the image prediction model and the acoustic image after the completion of learning for the prediction of a driving image of the service providing device (100) according to an embodiment of the present invention, FIG. 5 is an example diagram of the operation of the learning and prediction process for predicting a road surface condition of the service providing device (100) according to an embodiment of the present invention, and FIG. 6 is an example diagram of the process of generating the learning data learned by the road surface condition prediction model for predicting a road surface condition of the service providing device (100) according to an embodiment of the present invention.

[0079] First, as illustrated in FIG. 3, the sound processing unit (131) receives driving noise measurement information including an audio signal (or sound signal) for a specific road section from a sound collection device (10) that measures or collects sound regarding driving noise (or road surface noise) generated on the road surface due to driving of a vehicle and generates an audio signal, and extracts the audio signal included in the driving noise measurement information to generate a sound image that visualizes the waveform of the audio signal.

[0080] At this time, the audio signal included in the driving noise measurement information may consist of a digitized audio signal or digital audio data.

[0081] For example, the sound processing unit (131) can clearly visualize the characteristics of road surface noise generated by vehicle driving by performing processes such as frequency spectrum analysis of the audio signal, time-frequency conversion, and noise filtering.

[0082] More specifically, the sound processing unit (131) can apply a Fourier Transform algorithm to the received (extracted) audio signal to convert it into the frequency domain, and then analyze the frequency change over time through a Short-Time Fourier Transform (STFT).

[0083] In this process, the sound processing unit (131) can effectively extract only the characteristics of vehicle driving noise (road surface noise due to vehicle driving) from the audio signal by applying a preset adaptive filter to remove environmental noise, wind sound, other interference signals, etc.

[0084] Additionally, the sound processing unit (131) can convert the characteristics of the extracted driving noise into a spectrogram form to generate an acoustic image that visually represents the temporal change of the frequency components.

[0085] At this time, the acoustic image may be configured such that the X-axis represents time and the Y-axis represents frequency, and may also be configured as a two-dimensional image in which the intensity of color or brightness represents the intensity of the signal.

[0086] These acoustic images can be input into the image prediction unit (132) and used as basic data for generating driving images.

[0087] In addition, the sound processing unit (131) can receive different driving noise measurement information in real time or periodically from the sound collection device (10) through the communication unit (110), and can perform the operation process described above whenever driving noise measurement information is received.

[0088] In addition, the components configured in the control unit (130) can also perform the operation process described below whenever the driving noise measurement information is received.

[0089] Meanwhile, the image prediction unit (132) can generate a predicted driving image corresponding to the current sound image (or driving noise measurement information) through the image prediction model by applying the current sound image generated through the sound processing unit (131) based on the driving noise measurement information to an image prediction model that learns the correlation between the sound image and the driving image based on multiple different learning data, each corresponding to a plurality of different driving images taken for the specific road section and including a plurality of sound images and driving images generated at the time of taking the driving images.

[0090] That is, the image prediction unit (132) can input the sound image (current sound image) provided by the sound processing unit (131) into a deep learning-based image prediction model that has completed learning about the correlation between the sound image and the driving image in conjunction with the sound processing unit (131), and can generate (output) a predicted driving image, which is a driving image predicted from the input sound image (current sound image) corresponding to the sound image (current sound image) input into the image prediction model, through the image prediction model.

[0091] The above image prediction model can be implemented based on image transformation architectures such as GAN (Generative Adversarial Network), U-Net, Pix2Pix, CycleGAN, etc., and in particular, the image prediction unit (132) can generate a realistic driving image according to the conditions of the input acoustic image by utilizing a conditional GAN ​​structure.

[0092] More specifically, the image prediction model may be configured with a structure including one or more encoders and one or more decoders, wherein the encoder extracts features of the input acoustic image and the decoder can generate a predicted driving image based on the extracted features.

[0093] During the training process, model parameters can be adjusted to minimize the difference between actual driving images and generated driving images, and a Discriminator Loss Function can be used for this purpose.

[0094] Based on the configuration described above, with reference to Fig. 3, the training and prediction process of the image prediction model can be configured as follows.

[0095] First, the image prediction unit (132) can train an image prediction model corresponding to a specific road section using multiple different training data consisting of pairs of acoustic images and actual driving images in various road environments corresponding to a specific road section, which are stored in advance in an image generation training DB.

[0096] In the image generation training DB, acoustic images and driving images collected at the same time on the same specific road section may be stored as pairs, and metadata such as weather conditions (clear, rain, snow, etc.) and time of day (day, night) may be recorded together with the acoustic images or driving images.

[0097] More specifically, the image prediction unit (132) can adjust the parameters of the model to receive an acoustic image as input during the learning process and generate a driving image corresponding to the acoustic image as output.

[0098] At this time, the image prediction model may be configured with a structure including one or more encoders and one or more decoders, and the acoustic image is transmitted as input data to the encoder of the image prediction model, the encoder can extract latent features of the image, and the extracted features can be transmitted to the decoder of the image prediction model to be converted into a predicted driving image.

[0099] In addition, the image prediction model can be trained in a direction that minimizes the difference between the generated predicted driving image and the actual driving image.

[0100] In addition, the image generation training DB can be configured to include not only data from daytime on clear days but also driving images on rainy, snowy, and nighttime days, thereby enabling the conversion capability for predicted driving images in various environments.

[0101] In particular, by utilizing acoustic image-driving image pairs collected for the same specific road section at different times and weather conditions, the model can be trained to be robust to environmental changes.

[0102] For example, on an icy road surface, the friction sound between the tires and the road surface exhibits characteristics different from those in a normal state depending on the time of day, and the correlation between these sound patterns and the visual characteristics of the road surface condition can be learned by an image prediction model.

[0103] Additionally, the image prediction unit (132) can evaluate the performance of the image prediction model through a verification dataset after sufficient training of the image prediction model is completed, and store the image prediction model showing optimal performance in the image prediction model DB for use in actual prediction.

[0104] Referring to Fig. 4, examples of data trained on an image prediction model and predicted driving images generated after training are visible.

[0105] At the top of Fig. 4, pairs of acoustic images and driving images used for training the image prediction model are displayed.

[0106] When a vehicle passes in an actual road environment, the driving noise generated from the road surface collected by the sound collection device (10) is converted into an acoustic image and can be used as training data by pairing it with a driving image captured by a camera at the same time.

[0107] This training data includes patterns corresponding to various environmental conditions and vehicle types, enabling the image prediction model to learn the diversity of the actual driving environment.

[0108] The images shown in the middle and bottom parts of Fig. 4 represent predicted driving images generated based on acoustic images of driving noise measured during the day and at night, respectively.

[0109] In the acoustic image of the day, the characteristics of the vehicle driving noise are clearly revealed, and based on this, the image prediction unit (132) can input the current acoustic image into an image prediction model and generate a predicted driving image in which the lanes of the road, surrounding terrain, road surface conditions, etc. corresponding to the current acoustic image are clearly expressed through the image prediction model.

[0110] In particular, in the predicted driving images generated during the day, it can be confirmed that vehicle position and speed information are visualized by accurately matching them with acoustic characteristics.

[0111] Acoustic images at night exhibit different patterns in specific frequency bands compared to daytime images, and the predicted driving images generated based on this reflect the characteristics of the night environment, allowing for the representation of dark backgrounds, the influence of vehicle headlights, and limited visibility.

[0112] Accordingly, the image prediction unit (132) inputs a current acoustic image corresponding to nighttime into the image prediction model, recognizes a nighttime-specific acoustic pattern from the current acoustic image corresponding to nighttime through the image prediction model, and based on this, generates a predicted driving image through the image prediction model that corresponds to the current acoustic image and expresses important elements such as lanes, road boundaries, and road surface textures even under nighttime conditions.

[0113] In addition, the above image prediction model can be trained based on training data consisting of pairs of acoustic images corresponding to audio signals measured in adverse weather conditions, such as heavy rain or fog, and driving images captured in said adverse weather conditions, and accordingly, the correlation between the acoustic images and driving images in adverse weather conditions can be learned by the image prediction model.

[0114] More specifically, in heavy rain conditions, the sound of raindrops colliding with the road surface or hitting the vehicle surface forms a strong pattern in a specific frequency band, and in foggy conditions, changes in acoustic propagation characteristics due to humidity changes are reflected in the acoustic image.

[0115] Accordingly, the image prediction unit (132) can train the image prediction model with sound patterns unique to such bad weather and related driving images, and after training is completed, input the current sound image related to the weather conditions of bad weather into the image prediction model, thereby generating a predicted driving image that clearly expresses the road surface condition and road environment even in weather conditions that are difficult to visually identify through the image prediction model.

[0116] In addition, the image prediction model learns acoustic characteristics associated with changes in road surface conditions (puddles, slippery surfaces, etc.) under adverse weather conditions and reflects them in the predicted driving image; by doing so, it is possible to provide a predicted driving image that closely approximates the actual driving image while reflecting the current weather conditions solely through the acoustic image.

[0117] However, in the above-described configuration, driving images captured at night may consist of images with extremely limited visibility where it is difficult to determine road conditions or vehicle positions in specific road sections without light sources such as streetlights, and as a result, the usability of the images may be reduced.

[0118] To address this, the image prediction model can be trained using training data that includes driving images taken during the day, excluding training data that includes driving images taken at night. In this case, by generating and outputting a predicted driving image for the day (corresponding to the day) even for the current acoustic image generated based on audio signals measured at night, it is possible to support clear identification of road conditions even under conditions where visibility is poor.

[0119] At this time, to distinguish between day and night, a time period corresponding to day and a time period corresponding to night can be pre-set in the image prediction unit (132).

[0120] For example, the image prediction unit (132) can train an image prediction model using only training data consisting of pairs of driving images taken during the day and corresponding sound images, excluding driving images taken at night.

[0121] Subsequently, when driving noise measurement information including an audio signal measured at night is received, the acoustic processing unit (131) generates a current acoustic image corresponding to night based on the audio signal, and the image prediction unit (132) inputs this into an image prediction model trained with a driving image taken during the day to generate a predicted driving image corresponding to the daytime time period.

[0122] That is, the image prediction unit (132) can support the conversion of the current road condition (or road surface condition) at night, which generated driving noise measured at night, into a road condition (road surface condition) at daytime that has a road condition (road surface condition) similar to the current road condition (road surface condition) at night, by generating and providing a predicted driving image at daytime corresponding to the current sound image through the image prediction model, based on an audio signal measured during daytime that has a sound pattern similar to the sound pattern appearing in the current sound image corresponding to nighttime.

[0123] Accordingly, through the predicted driving image provided by the image prediction unit (132) which converts nighttime road conditions into daytime road conditions, the user can check the road conditions as bright and clear as daytime even in nighttime or dark environments, and can more clearly identify risk factors such as black ice or freezing conditions. In addition, this approach supports efficient road safety management regardless of the time of day by providing road managers with a road monitoring environment of consistent quality 24 hours a day.

[0124] In this way, the image prediction model is trained with data collected from various road conditions, weather conditions, and time of day (day / night), and can be equipped with the ability to generate predicted driving images in various environments.

[0125] Additionally, the image prediction unit (132) can generate a predicted driving image that accurately reflects the actual topographical characteristics of a specific road section by utilizing fixed location information of a specific road section as reference data.

[0126] Through the above configuration, the service providing device (100) can provide a realistic predicted driving image that is close to the actual driving image captured by the camera (camera unit) for the specific road section, even if the camera (camera unit) is not configured in the specific road section or the camera (camera unit) placed in the specific road section is fixed.

[0127] Meanwhile, the road surface condition prediction unit (133) can generate road surface condition prediction information by labeling road surface condition information for each of the plurality of driving images, training the road surface condition prediction model, and then applying the predicted driving images to the trained road surface condition prediction model.

[0128] Referring to FIG. 5, the road surface condition prediction unit (133) can train a road surface condition prediction model by utilizing driving images stored in a road surface condition learning DB and corresponding road surface condition information.

[0129] At this time, the driving image learned by the above road surface condition prediction model may be a driving image captured through the above camera unit.

[0130] For example, the road surface condition prediction unit (133) can train the road surface condition prediction model with a plurality of different training data including a driving image of a road surface condition (normal, frozen, black ice, wet road surface, etc.) and corresponding road surface condition information (labeled road surface condition information).

[0131] Figure 6 is a diagram showing an example of the process of generating training data stored in a road surface condition learning DB that is trained on a road surface condition prediction model.

[0132] Driving images are captured from actual road conditions, and each image may be labeled with road surface condition information regarding the type of road surface condition (N / A, Dry, Wet, Icy, etc.).

[0133] In addition, the road surface condition prediction unit (133) can analyze the road surface condition from the predicted driving image by utilizing a CNN (Convolutional Neural Network) based image classification model.

[0134] At this time, the above road surface condition prediction model can be configured based on the latest CNN architectures such as ResNet, DenseNet, and EfficientNet, and can classify road surface conditions with high accuracy through training data on various road surface conditions.

[0135] More specifically, the road surface condition prediction model takes a predicted driving image as input, extracts image features through a convolutional layer and a pooling layer, and classifies the final road surface condition through a fully connected layer.

[0136] The features extracted at this time include road surface texture, reflectivity, color change, and crack patterns, and the road surface condition can be determined by comprehensively analyzing these features.

[0137] Such road surface condition prediction information may include types of road surface conditions for at least one of normal road surface conditions, freezing conditions, black ice conditions, wet road surface conditions, crack conditions, and pothole conditions, and is not limited thereto, but may include various types of road surface conditions in the road surface condition prediction information.

[0138] According to the above configuration, the road surface condition prediction unit (133) can be linked with the image prediction unit (132) and, when the image prediction unit (132) generates a predicted driving image corresponding to the driving noise measurement information received by the sound processing unit (131), the predicted driving image can be applied as input to the road surface condition prediction model that has completed learning, and road surface condition prediction information corresponding to the predicted driving image can be generated and output through the road surface condition prediction model.

[0139] At this time, the road surface condition prediction information is configured to include probability values ​​for each type of road surface condition, thereby supporting the user in evaluating the reliability of the prediction result.

[0140] Alternatively, the road surface condition prediction unit (133) may include only one or more types of road surface conditions that have the highest probability value or are greater than or equal to a preset threshold in the road surface condition prediction information.

[0141] Additionally, the road surface condition prediction unit (133) or road surface condition prediction model may be configured to utilize an attention mechanism capable of detecting minute changes within a driving image to identify risk factors that are difficult to detect visually, such as black ice.

[0142] Through this, risk factors that are difficult to identify with ordinary cameras can be effectively detected and notified to the user.

[0143] Next, the result providing unit (134) may generate and provide prediction result information including the predicted driving image and road surface condition prediction information corresponding to the driving noise measurement information received by the sound processing unit (131) by linking with the sound processing unit (131), the image prediction unit (132), and the road surface condition prediction unit (133).

[0144] To this end, the result providing unit (134) can combine the information generated by the image prediction unit (132) and the road surface condition prediction unit (133), process it into a form that is easy for the user to understand, and deliver it.

[0145] For example, the result providing unit (134) can visualize the prediction result information through various platforms such as a web interface, a mobile application, and a dedicated dashboard and transmit it to the user terminal (20).

[0146] Alternatively, the result providing unit (134) may display the prediction result information through a separate display device connected to the service providing device (100), or transmit the prediction result information to a preset external server or external device.

[0147] In addition, the result providing unit (134) can display road surface condition prediction information in an overlay form on the predicted driving image so that the user can intuitively understand the road conditions and risk factors.

[0148] At this time, the result providing unit (134) may, when the type of road surface condition according to the road surface condition prediction information corresponds to a preset type of dangerous road surface condition, set the color of the road surface condition prediction information to a preset color for danger indication purposes to generate a prediction result information of the condition displayed on the prediction driving image, or set the road surface condition prediction information to a blinking effect or a special icon and then generate a prediction result information of the condition displayed on the prediction driving image.

[0149] Additionally, the result providing unit (134) can set the measurement time in the prediction result information based on the information regarding the measurement time of the audio signal according to the driving noise measurement information, and then accumulate and store it in the prediction result DB.

[0150] Additionally, the result providing unit (134) may generate road surface analysis result information including graph information on the trend of change in road surface conditions by utilizing time series data according to measurement time included in each of the plurality of prediction result information accumulated and stored in the prediction result DB, and may transmit the road surface analysis result information to the user terminal (20).

[0151] Through this, users can identify how the road surface conditions of a specific road section change over time and analyze the correlation with weather data such as temperature changes or precipitation.

[0152] Meanwhile, the warning processing unit (135) can generate warning information in conjunction with the result providing unit (134) or the road surface condition prediction unit (133) when the warning information corresponds to a preset type of dangerous road surface condition based on the road surface condition prediction information, and can transmit the generated warning information to a preset external server or user terminal (20).

[0153] For example, the warning processing unit (135) monitors the road surface condition prediction information in real time in conjunction with the result providing unit (134) or the road surface condition prediction unit (133), and can immediately generate warning information when a type of road surface condition is detected that is a predefined dangerous road surface condition, such as black ice, severe freezing, or dangerous potholes, based on the road surface condition prediction information.

[0154] At this time, the warning information may include information such as the type of road surface condition, road section identifier (ID), road section location information, and measurement time according to the prediction result information or road surface condition prediction information corresponding to the driving noise measurement information.

[0155] More specifically, the warning processing unit (135) can operate a multi-level warning system according to the severity of the dangerous situation.

[0156] For example, the warning processing unit (135) can transmit notification information only to a user terminal (20) corresponding to an administrator when the type of road surface condition according to the prediction result information corresponds to a first dangerous road surface condition type which is a pre-set minor risk, and can simultaneously propagate (transmit) warning information to various external servers or external devices such as a road traffic management system, an emergency response service, and a nearby vehicle navigation system when it corresponds to a second dangerous road surface condition type which is a pre-set serious risk.

[0157] At this time, one or more road surface condition types may be set for each of the first dangerous road surface condition type and the second dangerous road surface condition type.

[0158] For example, a wet road surface condition may be set as the first type of dangerous road surface condition, and a plurality of road surface condition types such as freezing, black ice occurrence, and pothole occurrence may be set as the second type of dangerous road surface condition.

[0159] In addition, the warning processing unit (135) may have setting information for one or more types of dangerous road surface conditions pre-set.

[0160] In this way, the warning information generated by the warning processing unit (135) can be automatically transmitted to an external server of a public institution such as a road management agency, a local government, or a police agency.

[0161] These external servers of public institutions can take rapid response measures for specific road sections based on received warning information. For example, they can generate automated work instructions to dispatch snowplows, spray brine, or spread calcium chloride on road sections where black ice or freezing conditions are detected, or transmit road condition information to police patrol cars to implement traffic control in those areas. This preemptive response system can prevent major traffic accidents and effectively protect the safety of road users by responding quickly before dangerous situations escalate.

[0162] In addition, the warning processing unit (135) periodically transmits updated information when the dangerous situation persists, and transmits a release notification when the dangerous situation is resolved so that the relevant system can return to a normal operating state.

[0163] Through this, the safety of road users can be effectively protected, and the possibility of accidents can be minimized.

[0164] Meanwhile, the service providing device (100) may further include a learning data collection unit (138), and the learning data collection unit (138) may transmit a user interface to the user terminal (20) for collecting data to be stored in the image generation learning DB or the road surface condition learning DB.

[0165] At this time, the user interface may be configured in the form of a web page or may be configured as execution data executed by an application installed on the user terminal (20).

[0166] Accordingly, the learning data collection unit (138) receives learning data to be stored in the image generation learning DB or the road surface condition learning DB generated by the user terminal (20) through the user interface, and can store the data in the image generation learning DB or the road surface condition learning DB, and can use the data stored in the image generation learning DB or the road surface condition learning DB for the learning of the image prediction model or the road surface condition prediction model.

[0167] Alternatively, the learning data collection unit (138) may receive and collect audio signals and driving images at different times from the sound collection device (10) and the camera unit temporarily installed in the specific road section to collect learning data for generating a predicted driving image for the specific road section and analyzing the road surface condition, and generate learning data that pairs audio signals and driving images at the same time, and store them in the image generation learning DB.

[0168] Meanwhile, FIG. 7 is a flowchart of the operation of a service provision method for road surface condition analysis based on driving noise of a service provision device (100) according to an embodiment of the present invention according to the configuration described above.

[0169] As described above, the service provision method according to an embodiment of the present invention may include an audio signal reception step (S11), an acoustic image generation step (S12), a predicted driving image generation step (S13), a road surface condition prediction step (S14), a prediction result information generation and provision step (S15), a risk state determination step (S16, S17), a warning information generation and transmission step (S18), etc.

[0170] Looking specifically at each step, the service providing device (100) can receive driving noise measurement information including an audio signal of a specific road section from the sound collecting device (10) in the audio signal receiving step (S11).

[0171] Additionally, the service providing device (100) can process the received audio signal in the acoustic image generation step (S12) to generate an acoustic image in the time-frequency domain.

[0172] In addition, the service providing device (100) can generate a predicted driving image from an acoustic image through an image prediction model in the predicted driving image generation step (S13).

[0173] In addition, the service providing device (100) can generate road surface condition prediction information from a predicted driving image through a road surface condition prediction model in the road surface condition prediction step (S14).

[0174] In addition, the service providing device (100) can generate prediction result information by combining the prediction driving image and road surface condition prediction information in the prediction result information generation and provision step (S15) and provide it to the user terminal (20).

[0175] In addition, the service providing device (100) can determine whether it corresponds to a dangerous road surface condition type by analyzing road surface condition prediction information in the danger state determination step (S16).

[0176] Additionally, the service providing device (100) can generate warning information and transmit it to a relevant external server in the warning information generation and transmission step (S18) when it is determined to be a type of dangerous road surface condition (S17).

[0177] As described above, the service providing device (100) according to an embodiment of the present invention can provide a comprehensive road safety system capable of road monitoring and detection of dangerous situations even in situations where there is no camera or the camera is malfunctioning, and in conditions where visibility is difficult to secure, such as at night or in bad weather, by generating an acoustic image based on an audio signal received from an acoustic collection device (10), applying it to an image prediction model to generate a realistic predicted driving image, and then applying the generated predicted driving image to a road surface condition prediction model to analyze the road surface condition.

[0178] Meanwhile, the result providing unit (134) can provide a feedback function to continuously improve the accuracy of the road surface condition prediction model, and this feedback function can be configured as a cyclic improvement structure that corrects the prediction result through user verification and reflects it in learning.

[0179] Specifically, the result providing unit (134) can transmit the prediction result information to the administrator's user terminal (20).

[0180] At this time, the result providing unit (134) can generate the prediction result information including a road section identifier corresponding to the specific road section and transmit it to the user terminal (20).

[0181] Accordingly, if an administrator who has checked the prediction result information through the user terminal (20) finds that the type of road surface condition according to the road surface condition prediction information is inconsistent with the type of road surface condition actually observed and modifies it to the actual type of road surface condition, the administrator can generate corrected road surface condition information for the actual type of road surface condition through the user terminal (20).

[0182] For example, the administrator reviews the road surface condition prediction information based on the prediction result information provided through the user terminal (20), and if the predicted road surface condition is displayed as 'normal' but the administrator determines that it is a 'wet road surface' after checking the predicted driving image based on the prediction result information or the driving image received from the camera installed on the specific road section, the administrator may modify the road surface condition prediction information included in the prediction result information to 'wet road surface'. The user terminal (20) may generate corrected road surface condition information based on the modification of the road surface condition prediction information, and generate correction request information including the corrected road surface condition information and the predicted driving image included in the prediction result information, and transmit it to the service providing device (100).

[0183] At this time, the correction request information may include a predicted driving image based on the prediction result information, corrected road surface condition information regarding the type of actual road surface condition, and a road section identifier included in the prediction result information.

[0184] Additionally, the result providing unit (134) can receive the correction request information from the user terminal (20), and after generating feedback learning data based on the predicted driving image according to the correction request information and the corrected road surface condition information (or the actual road surface condition type according to the corrected road surface condition information), the prediction accuracy of the road surface condition prediction model can be improved by additionally training a road surface condition prediction model corresponding to the road section identifier according to the correction request information using the feedback learning data through linkage with the road surface condition prediction unit (133).

[0185] For example, the result providing unit (134) can store the feedback learning data in a road surface condition learning DB, and the feedback learning data stored in this way can be included in the learning of the road surface condition prediction model for the next cycle, thereby supporting the model to learn errors and make more accurate predictions in similar situations.

[0186] More specifically, the result providing unit (134) transmits a signal to the road surface condition prediction unit (133) whenever there is a preset learning update cycle (e.g., one week, one month, etc.) or whenever the feedback learning data reaches a preset threshold (e.g., 100, 500, etc.), and the road surface condition prediction unit (133) receives this signal and retrains the road surface condition prediction model by using the existing learning data stored in the road surface condition learning DB together with the newly added feedback learning data.

[0187] In this feedback learning process, the road surface condition prediction unit (133) may apply a weighted learning method that assigns higher weights to recently collected feedback learning data. This is to enable the model to better reflect the road environment and seasonal factors that change over time.

[0188] Meanwhile, according to another embodiment of the present invention, the driving image may be composed of an infrared image captured through an infrared camera (infrared camera unit).

[0189] Infrared images can provide clearer images based on the thermal properties of objects even under conditions with limited visibility, such as night or fog, compared to standard visible light cameras, making them effective for 24-hour road monitoring.

[0190] In this configuration, the image prediction model can be trained based on a plurality of infrared images (infrared driving images) captured during the day and at night, a plurality of acoustic images corresponding to each, and the plurality of infrared images.

[0191] At this time, the plurality of infrared images (infrared driving images) used for training the image prediction model may refer to a plurality of infrared images taken during the day and a plurality of infrared images taken at night.

[0192] To this end, an image generation training DB may store an infrared driving image captured by an infrared camera at the same time as an acoustic image in pairs, and the image prediction unit (132) may train the image prediction model using training data including the paired acoustic image and infrared driving image stored in the image generation training DB.

[0193] More specifically, infrared images have the characteristic of displaying different brightness levels depending on the surface temperature of an object, making them useful for identifying icy sections or wet road surfaces through temperature differences. Generally, icy road surfaces have lower temperatures than surrounding roads and appear relatively dark in infrared images; this characteristic can be learned by image prediction models and road surface condition prediction models.

[0194] Additionally, the image prediction unit (132) normalizes each pixel value to reflect temperature information when training an image prediction model using infrared images, thereby enabling the model to effectively learn temperature change patterns.

[0195] In addition, infrared images have the advantage that the difference in visibility between day and night is smaller than that of visible light images, allowing the same model to make predictions for both day and night.

[0196] Accordingly, the image prediction unit (132) can generate a predicted driving image in the form of an infrared image corresponding to the current acoustic image generated through the acoustic processing unit (131) through the image prediction model. The generated predicted driving image in the form of an infrared image can be applied to a road surface condition prediction model, just like a general visible light image, and used to analyze the road surface condition.

[0197] This infrared image-based road surface condition analysis method can provide more effective results, particularly at night or in adverse weather conditions. For instance, while visibility is limited with conventional cameras in dense fog, infrared cameras can penetrate the fog to capture road conditions. Accordingly, by generating infrared predictive driving images from acoustic images, it is possible to maintain a level of road monitoring capability similar to that provided by infrared cameras, even in situations where cameras fail or are unavailable.

[0198] In addition, the road surface condition prediction unit (133) stores and utilizes training data in the road surface condition learning DB in which infrared driving images and road surface condition information of the infrared driving images are labeled for the infrared driving images, in order to train a road surface condition prediction model specialized for infrared images.

[0199] Since infrared images provide information based on temperature differences, road surface condition prediction models can effectively learn the correlation between temperature patterns and road surface conditions.

[0200] Additionally, the road surface condition prediction unit (133) inputs the infrared-shaped predicted driving image generated by the image prediction unit (132) into a road surface condition prediction model specialized for infrared images, and can predict the road surface condition by analyzing the temperature pattern, texture, pattern change, etc. of the infrared image through the road surface condition prediction model.

[0201] At this time, the road surface condition prediction model can generate road surface condition prediction information by determining and predicting the road surface condition from a prediction driving image in the form of an infrared image, by learning characteristics such as the fact that the freezing section generally has a lower temperature than the surroundings and that the wet road surface exhibits different heat emission characteristics from the dry road surface.

[0202] Meanwhile, the result providing unit (134) can generate prediction result information including an infrared-shaped predicted driving image generated through the image prediction unit (132) based on the driving noise measurement information and road surface condition prediction information based thereon.

[0203] The predicted driving images included in these prediction result information include temperature-based visualizations unique to infrared images, and can be provided in a form with color mapping applied so that users can intuitively understand the temperature difference.

[0204] For example, in the predicted driving image, low-temperature areas (sections with a high probability of freezing) may be displayed in blue or purple, and relatively high-temperature areas in red or yellow.

[0205] Additionally, the result providing unit (134) can specify and highlight a dangerous section (or dangerous area) in the predicted driving image that is determined to have ice or black ice among the areas corresponding to the road, taking into account the characteristics of the infrared image.

[0206] For example, predictive driving images in the form of infrared images can have the characteristic of being displayed in different colors depending on the surface temperature of an object; generally, low temperatures are displayed in shades of blue and high temperatures in shades of red, allowing the temperature distribution to be intuitively understood.

[0207] By utilizing these characteristics, the result providing unit (134) can perform the function of automatically identifying and displaying dangerous sections by analyzing the temperature distribution of the road surface in the predicted driving image.

[0208] To this end, the result providing unit (134) has a pre-set image analysis algorithm for identifying road areas, and this algorithm can perform the function of distinguishing road and non-road areas in the predicted driving image.

[0209] The image analysis algorithm can be composed of a deep learning model for semantic segmentation such as U-Net or DeepLab, and can identify road areas at the pixel level by distinguishing road areas and non-road areas based on the characteristic shape and pattern of the road.

[0210] Accordingly, the result providing unit (134) can identify and detect a road area in the predicted driving image through the image analysis algorithm, and when the road area is identified, it can analyze color information according to temperature within the road area.

[0211] To this end, the result providing unit (134) may have a low-temperature identification criterion for identifying a low-temperature area set in advance, and according to the low-temperature identification criterion, a pixel area having a color corresponding to a pre-set temperature threshold (e.g., 0°C or lower or a section that is 2°C or lower than the surrounding road surface) may be identified as a low-temperature area.

[0212] For example, the result providing unit (134) may apply an HSV (Hue, Saturation, Value) color space to the predicted driving image according to the low temperature identification criteria (by converting the predicted driving image into an HSV color space) to extract pixels corresponding to a specific color range (e.g., a range of H values ​​corresponding to blue and purple) from the predicted driving image, analyze the connectivity of these pixels to identify a low temperature area larger than a preset size, and determine the identified low temperature area as a freezing area or a dangerous area with a high probability of black ice occurrence.

[0213] In this case, the low-temperature identification criteria may be configured as an algorithm.

[0214] In addition, the result providing unit (134) can perform various types of marking on the risk areas identified in the predicted driving image.

[0215] For example, the result providing unit (134) may include marking information on the danger area on the predicted driving image, such as highlighting the danger area with a red border, displaying a translucent red overlay on the danger area, placing a flashing warning icon around the danger area, or displaying pre-set danger information matching the identified danger area in a popup form.

[0216] In this way, based on the temperature color shown in the infrared image, freezing sections or sections with a high probability of black ice formation can be highlighted with a blinking effect or special indicator to draw the attention of the manager (user).

[0217] In addition, the result providing unit (134) can generate prediction result information including a prediction driving image containing the marking information and road surface condition prediction information and transmit it to a user terminal (20) or various external servers.

[0218] The method for predicting road surface conditions and providing results based on infrared images by the service providing device (100) according to the embodiment of the present invention is particularly effective in nighttime or environments with limited visibility, and can specify and display dangerous sections or dangerous areas, such as freezing sections or black ice sections, based on temperature differences, thereby providing an advantage in early detection of risk factors.

[0219] Meanwhile, the service providing device (100) according to an embodiment of the present invention can perform the function of detecting the vehicle location and speed based on the predicted driving image and performing traffic safety management based thereon.

[0220] To this end, the vehicle monitoring unit (136) can identify one or more vehicle objects included in the predicted driving image by analyzing the predicted driving image through a preset object detection algorithm in conjunction with the image prediction unit (132).

[0221] Additionally, the vehicle monitoring unit (136) identifies a waveform section exceeding a predetermined amplitude threshold in an acoustic image as a vehicle approach event, and can calculate a slope over a preset time based on the amplitude change in the identified vehicle approach event section.

[0222] Next, the vehicle monitoring unit (136) can calculate the driving speed of a specific vehicle corresponding to the vehicle approach event by applying the calculated slope value to a speed prediction model in which the correlation between slope and speed is defined or learned.

[0223] Such speed prediction models may be composed of deep learning-based models.

[0224] For example, based on the correlation that the steeper the slope of the amplitude change, the faster the vehicle approaches, if the slope of the amplitude change in a specific section is measured as 0.8, this can be applied to a speed prediction model to calculate a driving speed of approximately 60 km / h.

[0225] Additionally, the vehicle monitoring unit (136) can calculate the actual distance from the sound collection device (10) to the vehicle by analyzing the frequency characteristics and sound intensity corresponding to the vehicle approach event in the acoustic image.

[0226] For example, the vehicle monitoring unit (136) can calculate the actual distance using a distance calculation algorithm based on a widely known acoustic attenuation model and the Doppler effect.

[0227] Afterwards, the vehicle monitoring unit (136) can convert the calculated actual distance into a pixel distance on a predicted driving image through a pre-set distance conversion model in which a conversion relationship between the actual distance and the pixel distance on the predicted driving image is defined or learned.

[0228] Additionally, the vehicle monitoring unit (136) can determine a specific vehicle object corresponding to the specific vehicle located closest to the converted pixel distance from the reference position of the sound collection device (10) pre-set within the predicted driving image among the one or more identified vehicle objects, and generate monitoring result information including at least one of distance information for the actual distance mapped to the specific vehicle object on the predicted driving image and driving speed information of the calculated specific vehicle.

[0229] Meanwhile, the traffic control unit (137) can calculate the time when the specific vehicle is expected to reach a preset danger zone based on the calculated driving speed of the specific vehicle and the actual distance from the sound collection device (10) to the vehicle, in conjunction with the vehicle monitoring unit (136).

[0230] Additionally, the traffic control unit (137) may, when the estimated arrival time is below a preset threshold, generate an approach notification information and transmit it to a preset output device so that the approach notification information is output through the output device.

[0231] For example, if a sound collection device (10) installed in a section where jaywalking is frequent detects the approach of a vehicle and the vehicle monitoring unit (136) determines that the vehicle is currently approaching from a distance of 50m at a speed of 40km / h, the traffic control unit (137) calculates the estimated time for the vehicle to reach the danger zone as approximately 4.5 seconds. Since this estimated time of arrival is less than a preset threshold of 5 seconds, the traffic control unit (137) can generate approach notification information such as "vehicle entering" and control it to be output through speakers or electronic display devices installed along the roadside.

[0232] Through this, accidents can be prevented by providing pedestrians with advance notification (pre-warning) of approaching vehicles.

[0233] As described above, the service providing device (100) according to the embodiment of the present invention generates an acoustic image based on an audio signal received from an acoustic collection device (10), and generates a predicted driving image by applying it to a deep learning-based image prediction model to predict a driving image of the road, and then analyzes the road surface condition based on the predicted driving image through a road surface condition prediction model. This allows for accurate provision of visual information regarding the road environment and road surface condition information by providing the predicted driving image and predicted result information regarding the road surface condition, even in situations where there is no camera or the camera is malfunctioning, or in conditions where visibility is difficult to secure, such as at night or in bad weather. Furthermore, by supporting a manager to proactively detect risk factors such as black ice by verifying the predicted result information and to respond quickly to dangerous situations on the road, it is possible to prevent major traffic accidents and significantly improve road safety.

[0234] In addition, the present invention visualizes hazardous elements such as freezing sections or black ice using temperature-based color information through the generation of predictive driving images in the form of infrared images, and automatically identifies and marks low-temperature areas within the road, thereby supporting road managers in quickly recognizing and responding to hazardous road conditions regardless of the time of day or weather.

[0235] In addition, the present invention provides a function to extract the location and speed of a vehicle through audio waveform analysis, thereby enabling the detection of a vehicle's approach based solely on driving noise and the generation of an approach alert signal by calculating the estimated time of arrival at a danger zone, which can overcome the limitations of camera-based monitoring and significantly improve the efficiency and reliability of traffic safety management.

[0236] Meanwhile, the deep learning-based models described in the embodiments of the present invention may be composed of one or more neural network models.

[0237] In addition, the neural network model (or neural network) described in the present invention may be composed of an input layer, one or more hidden layers, and an output layer, and various types of neural networks such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Neural Network (CNN), Transformer, Attention mechanism, and Graph Neural Network may be applied to the neural network model.

[0238] The service provision method according to the embodiment of the present invention described in the above-described embodiment can be written as a computer program, and the codes and code segments constituting the computer program can be easily inferred by a computer programmer in the relevant field. In addition, the computer program can be stored on a computer-readable information storage medium and can be read and executed by a computer, a user terminal (20) according to the embodiment of the present invention, a service provision device (100), etc., thereby implementing the service provision method.

[0239] Information storage media include magnetic recording media and optical recording media. A computer program implementing a service provision method according to an embodiment of the present invention may be stored and installed in the built-in memory of a user terminal (20) and a service provision device (100). Alternatively, an external memory, such as a smart card, which stores and installs a computer program implementing a service provision method according to an embodiment of the present invention may be mounted on the user terminal (20) and the service provision device (100) through an interface.

[0240] The components described in the embodiments of the present invention may be implemented using one or more general-purpose computers or special-purpose computers, such as hardware including, for example, memory, processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPGA (Field Programmable Gate Array), PLU (programmable logic unit), microprocessor, logic gate, etc., software including an instruction set or a combination thereof, or any other device capable of executing and responding to instructions.

[0241] The various devices and components described in this specification may be implemented by hardware circuits (e.g., CMOS-based logic circuits), firmware, software, or a combination thereof. For example, they may be implemented using transistors, logic gates, and electronic circuits in the form of various electrical structures.

[0242] A person skilled in the art to which the present invention pertains will be able to make modifications and variations to the foregoing without departing from the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0243] 10: Acoustic acquisition device 20: User terminal 100: Service providing device 110: Communications Department 120: Storage section 130: Control unit 131: Sound processing unit 132: Image Prediction Unit 133: Road surface condition prediction unit 134: Result Provisioning Section 135: Warning handling unit 136: Vehicle Surveillance Unit 137: Traffic Control Department 138: Training Data Collection Section

Claims

Claim 1 A service providing device for road surface condition analysis based on driving noise, comprising: an acoustic processing unit that generates an acoustic image visualizing the waveform of an audio signal based on driving noise measurement information including an audio signal for a specific road section received from an acoustic acquisition device that measures driving noise and generates an audio signal; an image prediction unit that generates a predicted driving image corresponding to the current acoustic image through the image prediction model by applying the current acoustic image generated through the acoustic processing unit based on the driving noise measurement information to an image prediction model that learns the correlation between an acoustic image and a driving image based on a plurality of different learning data including an acoustic image and a driving image generated at the time of capturing the driving image, each corresponding to a plurality of different driving images captured for the specific road section; a road surface condition prediction unit that generates road surface condition prediction information by labeling road surface condition information for each of the plurality of driving images, training the road surface condition prediction model, and then applying the predicted driving image to the road surface condition prediction model that has completed training; and a result providing unit that generates and provides prediction result information including the predicted driving image and the road surface condition prediction information. Claim 2 A service providing device for road surface condition analysis based on driving noise, characterized in that, in claim 1, the road surface condition prediction information includes a type of road surface condition for at least one of a normal road surface condition, a freezing condition, a black ice occurrence condition, a wet road surface condition, a crack occurrence condition, and a pothole occurrence condition. Claim 3 A service providing device for road surface condition analysis based on driving noise according to claim 1, wherein the image prediction model is trained using training data including driving images taken during the day, excluding training data including driving images taken at night, and the image prediction unit generates and outputs a predicted driving image for the day corresponding to the current acoustic image through the image prediction model for the current acoustic image generated based on an audio signal measured at night. Claim 4 A service providing device for road surface condition analysis based on driving noise, wherein, in claim 1, the driving image is composed of an infrared driving image, which is an infrared image captured through an infrared camera; the image prediction model is learned based on a plurality of infrared driving images captured during the day and night, a plurality of acoustic images corresponding to each, and the plurality of infrared driving images; the image prediction unit generates a predicted driving image in the form of an infrared image corresponding to the current acoustic image through the image prediction model; the road surface condition prediction unit generates road surface condition prediction information corresponding to the predicted driving image through the road surface condition prediction model; and the result providing unit analyzes temperature-based color information for the predicted driving image and, if a danger area is identified, includes marking information marked in the danger area in the predicted driving image. Claim 5 A service providing device for road surface condition analysis based on driving noise, characterized in that, in claim 1, it further includes a warning processing unit that generates warning information when the type of road surface condition according to the road surface condition prediction information corresponds to a preset type of dangerous road surface condition, and transmits the generated warning information to a preset external server or user terminal. Claim 6 A service providing device for road surface condition analysis based on driving noise, wherein, in the case where an administrator who has verified the prediction result information modifies the road surface condition type according to the road surface condition prediction information because it does not match the actually observed road surface condition type, the result providing unit generates feedback learning data including the actual road surface condition type and a predicted driving image according to the prediction result information, and then improves the prediction accuracy of the road surface condition prediction model by additionally training the road surface condition prediction model using the feedback learning data through linkage with the road surface condition prediction unit. Claim 7 In claim 1, in conjunction with the image prediction unit, the predicted driving image is analyzed to identify one or more vehicle objects included within the predicted driving image, and a waveform section exceeding a predetermined amplitude threshold in the acoustic image is identified as a vehicle approach event; a slope for a preset time period according to the amplitude change in the identified vehicle approach event section is calculated, and the calculated slope value is applied to a speed prediction model in which the correlation between slope and speed is defined or learned to calculate the driving speed of a specific vehicle corresponding to the vehicle approach event; frequency characteristics and sound intensity corresponding to the vehicle approach event in the acoustic image are analyzed to calculate the actual distance from the acoustic collection device to the vehicle, and then the calculated actual distance is converted into a pixel distance on the predicted driving image through a preset distance conversion model in which the conversion relationship between the actual distance and the pixel distance on the predicted driving image is defined or learned; among the one or more identified vehicle objects, a specific vehicle object corresponding to the specific vehicle located closest to the converted pixel distance from the reference position of the acoustic collection device preset within the predicted driving image is determined, and distance information regarding the actual distance mapped to the specific vehicle object on the predicted driving image and the calculated specific vehicle A service providing device for road surface condition analysis based on driving noise, characterized by further including a vehicle monitoring unit that generates monitoring result information including at least one of driving speed information. Claim 8 A service providing device for road surface condition analysis based on driving noise according to claim 7, further comprising a traffic control unit that calculates the time at which a specific vehicle is expected to reach a preset danger zone based on the driving speed of the specific vehicle calculated above and the actual distance from the sound collection device to the vehicle, and if the expected time of arrival is below a preset threshold, generates an approach notification information and transmits it to a preset output device so that the approach notification information is output through the output device. Claim 9 A method for providing a service for road surface condition analysis based on driving noise of a service providing device, comprising: generating an acoustic image that visualizes the waveform of an audio signal based on driving noise measurement information including an audio signal for a specific road section received from an acoustic acquisition device that measures driving noise and generates an audio signal; applying a current acoustic image generated based on the driving noise measurement information to an image prediction model that learns the correlation between an acoustic image and a driving image based on a plurality of different learning data including an acoustic image and a driving image generated at the time of capturing the driving image, each corresponding to a plurality of different driving images captured for the specific road section, thereby generating a predicted driving image corresponding to the current acoustic image through the image prediction model; labeling road surface condition information for each of the plurality of driving images and training the road surface condition prediction model, and then applying the predicted driving image to the road surface condition prediction model that has completed training to generate road surface condition prediction information; and generating and providing prediction result information including the predicted driving image and the road surface condition prediction information. Claim 10 A computer-readable non-volatile recording medium having a computer program recorded thereon for enabling a computer to perform the method according to paragraph 9.

Images