A multimodal cognitive mechanism for road segment recognition.

A multi-modal recognition mechanism using acoustic spectral density analysis, spectrogram processing, and machine learning improves road segment recognition accuracy by reducing noise interference, enabling precise fault detection during vehicle testing.

JP2024545405AInactive Publication Date: 2024-12-06INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024529431
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-30
Filing Date
2022-11-25
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing methods for identifying road segments during vehicle testing are inaccurate due to interference from anomalous noise and limited accuracy of neural network architectures, making it difficult to reliably sense road sections and identify faults.

Method used

A multi-modal recognition mechanism that combines acoustic spectral density analysis, spectrogram image processing, and machine learning to identify road segment switching points, using a processor to generate acoustic spectral density maps, process spectrograms, and utilize a long short-term memory (LSTM) model for noise reduction and accurate road segment recognition.

Benefits of technology

Improves the accuracy of road segment recognition by reducing interference from vehicle noise and enhancing the precision of road segment identification, allowing for more effective fault detection during vehicle testing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024545405000001_ABST
    Figure 2024545405000001_ABST
Patent Text Reader

Abstract

In an approach for road segment recognition using a multimodal perception mechanism, a processor receives an audio signal from a road test. In a first mode, the processor processes the audio signal to generate an acoustic spectral density map and identify each of the at least one road segment switching points. In a second mode, the processor processes a spectrogram of the audio signal to identify each of the at least one road segment switching points. In a third mode, the processor uses a machine learning model to predict an expected sound for each frame of the audio signal, calculate a similarity between the expected sound and the actual sound, and identify each of the at least one road segment switching points when the similarity is less than a pre-set similarity threshold. The processor combines the results of the three modes to obtain a final set of road segment switching points.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates generally to the field of cognitive computing, and more particularly to a multi-modal cognitive mechanism for road segment recognition. [Background technology]

[0002] Car manufacturers test their cars in all types of environments. Many tests may be done on a closed track, but real world car testing needs to be done in real world conditions. By combining the data from the closed track with the information gathered from driving on public roads, car manufacturers use testing to create the vehicle they want and that will satisfy the market. Testing of cars during production covers everything from performance and comfort to reliability and safety. The intention behind car testing is to allow manufacturers to iron out all the defects and potential issues in the model before moving to full production.

[0003] The usual method for someone doing road testing is to drive on a special road and listen to any sound the car makes to observe if the car is faulty. With the current digitalization of business, some car manufacturers have begun to use voice signal processing technology and machine learning methods to identify faults to replace human subjective listening. For fault identification during vehicle road testing, identification of the current road section and selection of different models are necessary. Therefore, there is a need for automatic identification of the road section where road testing is being carried out. In general, there are two methods for identifying road sections. The first is to sense the sound volume and frequency based on pre-applied rules. However, when there are many road shapes, it may be impossible to accurately detect the road section based on this, and abnormal sounds may cause interference. The second is to use neural network architecture, but the accuracy of this method is limited due to irregular noise in the driving test process. Summary of the Invention

[0004] An aspect of an embodiment of the present invention discloses a method, computer program product, and computer system for road segment recognition using a multimodal perception mechanism. A processor receives an audio signal from a road test. In a first mode, the processor processes the audio signal to generate an acoustic spectral density map and identify each of the at least one road segment switching points. In a second mode, the processor processes a spectrogram of the audio signal to identify each of the at least one road segment switching points. In a third mode, the processor uses a machine learning model to predict an expected sound for each frame of the audio signal, calculates a similarity between the expected sound and the actual sound, and identifies each of the at least one road segment switching points when the similarity is less than a pre-set similarity threshold. The processor combines the results of the three modes to obtain a final set of road segment switching points.

[0005] Aspects of an embodiment of the present invention improve the accuracy of road segment recognition using a multi-modal perception mechanism that combines the results of three separate modes of road segment recognition.

[0006] In some aspects of embodiments of the present invention, processing the audio signal to generate an acoustic spectral density map and identifying each of the at least one road section switching point includes preprocessing the audio signal with a background sound filter and a low pass filter to produce a preprocessed audio sequence, generating the spectral density map using a sliding window process on the preprocessed audio sequence, and identifying at least one corresponding extreme point in the transition on the spectral density map.

[0007] In some aspects of embodiments of the present invention, identifying at least one corresponding extreme point in the transition on the spectral density distribution map includes finding at least one corresponding extreme point based on a change trend of the spectral density on a vertical axis of the spectral density distribution map, and filtering out outlier extreme points on the spectral density distribution map using a pre-set threshold.

[0008] In some aspects of embodiments of the present invention, processing a spectrogram of sound waves of the audio signal to identify each of the at least one road section switching point includes generating a spectrogram image of the audio signal using a frequency of the audio signal, preprocessing the spectrogram image of the audio signal using contrast enhancement and conversion from a color image to a grayscale image, filtering the preprocessed spectrogram image using binarization and converting the grayscale image to a binary image, completing image morphology processing on the preprocessed and filtered spectrogram image using kernels for corrosion and expansion operations to generate a fully processed spectrogram image, and identifying each of the at least one road section switching point by searching for a maximum connected domain in the fully processed spectrogram image.

[0009] In some aspects of embodiments of the present invention, a machine learning model is trained with historical audio signal data from road tests, and the machine learning model is a long short-term memory (LSTM) model.

[0010] In some aspects of embodiments of the present invention, the pre-set similarity threshold is based on the performance of the machine learning model on historical data.

[0011] In some aspects of embodiments of the present invention, the more similar the predicted sound is to the actual sound, the less similarity there is between the predicted sound and the actual sound. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a functional block diagram illustrating a distributed data processing environment in accordance with an embodiment of the present invention. [Diagram 2] 4 is a flow chart illustrating the operational steps of a multi-modal road recognition program running on a server in a distributed data processing environment for road segment recognition using a multi-modal perception mechanism in accordance with an embodiment of the present invention. [Diagram 3] FIG. 2 is a block diagram of server components of the distributed data processing environment of FIG. 1 for running a multi-mode road recognition program in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The embodiments of the present invention recognize that for fault identification during road testing of a vehicle, identification of the current road section and selection of different models are required. The embodiments of the present invention recognize the need for a method for automatically identifying the road section on which a road test is being performed. In general, there are two approaches to identifying the road section. The first is to sense the sound loudness and frequency based on pre-applied rules. However, when there are many road shapes, it may be impossible to accurately detect the road section based on this, and abnormal sounds may cause interference. The second is to use a neural network architecture, but the accuracy of this method is limited due to irregular noise in the driving test process.

[0014] An embodiment of the present invention provides a system and method for road segment recognition using a multi-modal perception mechanism based on noise recognition. The embodiment of the present invention utilizes three modes to identify road segments, and combines the results from the three modes to obtain a final set of road segment switching points and effectively remove the interference of vehicle disturbance sounds. The embodiment of the present invention (1) analyzes the acoustic wave spectral density, (2) finds the connected domain of the acoustic wave spectrogram, and (3) builds a time series prediction model.

[0015] Embodiments of the present invention recognize the advantages of this system and method: (1) removing interference from vehicle roughness by finding the maximally connected domain of the spectrogram in one mode and using a predictive model in another mode, and (2) improving the accuracy of road segment recognition using a multi-modal cognitive mechanism.

[0016] Implementations of embodiments of the present invention may take a variety of forms, and details of exemplary implementations are discussed below with reference to the figures.

[0017] FIG. 1 is a functional block diagram illustrating a distributed data processing environment, generally designated 100, in accordance with one embodiment of the present invention. As used herein, the term "distributed" refers to a computer system that includes multiple physically separate devices that operate together as a single computer system. FIG. 1 is merely illustrative of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by one skilled in the art without departing from the scope of the invention as defined by the claims.

[0018] The distributed data processing environment 100 includes a server 110 and a user computing device 130 interconnected via a network 105. The network 105 may be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the three, and may include wired, wireless, or fiber optic connections. The network 105 may include one or more wired and / or wireless networks capable of receiving and transmitting data, voice, or video signals, or combinations thereof, including multimedia signals including voice, data, and video information. In general, the network 105 may be any combination of connections and protocols that would support communications between the server 110, the user computing device 130, and other computing devices (not shown) in the distributed data processing environment 100.

[0019] Server 110 may be a standalone computing device, an administrative server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In other embodiments, server 110 may represent a server computing system utilizing multiple computers as a server system, such as in a cloud computing environment. In another embodiment, server 110 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smartphone, or any programmable electronic device capable of communicating with user computing devices 130 and other computing devices (not shown) in distributed data processing environment 100 via network 105. In another embodiment, server 110 represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that function as a single pool of seamless resources when accessed within distributed data processing environment 100. The server 110 includes a multi-mode road recognition program 112 and a database 114. The server 110 may include internal and external hardware components, as shown and described in further detail with respect to FIG.

[0020] The multi-modal road recognition program 112 operates as a method for road segment recognition using a multi-modal perception mechanism. In the illustrated embodiment described in FIG. 2, the multi-modal road recognition program 112 identifies a road segment by using three modes to find switch points between road segments. In some embodiments, the multi-modal road recognition program 112 completes the three modes simultaneously. In other embodiments, the multi-modal road recognition program 112 completes the three modes in unison. In the illustrated embodiment, the multi-modal road recognition program 112 is a stand-alone program. In another embodiment, the multi-modal road recognition program 112 may be integrated into another software product, such as, for example, an automotive testing software package. The multi-modal road recognition program 112 is illustrated and described in further detail with respect to FIG. 2.

[0021] In the first mode, the multi-modal road recognition program 112 uses the acoustic spectral density map to find the road segment switching points. In one embodiment, the multi-modal road recognition program 112 pre-processes the incoming audio signal with a background sound filter and a low-pass filter and outputs a processed audio sequence. In one embodiment, the multi-modal road recognition program 112 generates the spectral density map using a sliding window process on the audio sequence. For example, the multi-modal road recognition program 112 uses a window of length 1 to slide the window over the time series data with a step length of 0.5 seconds (s) (1), calculates the spectral density for every 1 s segment of the audio signal, and generates a graph of the change in the spectral density (i.e., the spectral density map) with the horizontal axis being time and the vertical axis being spectral density. In one embodiment, the multi-modal road recognition program 112 finds at least one corresponding extreme point (i.e., the road segment switching point) in the transition on the spectral density map. The multi-modal road recognition program 112 finds these extreme points based on the change trend of the spectral density on the vertical axis of the map. In some embodiments, the multi-modal road recognition program 112 uses a pre-set threshold to filter out extremely low points (i.e., outliers) that are too low.

[0022] In the second mode, the multi-mode road perception program 112 uses image processing techniques to process a spectrogram of the sound waves of the received audio signal to find road section switching points. In one embodiment, the multi-mode road perception program 112 uses the frequencies of the audio signal to generate a spectrogram of the audio signal. A spectrogram shows a visual representation of the frequency domain characteristics, i.e., the spectrum of frequencies (i.e., signal strength or "magnitude") of the received audio signal as the signal varies over time. In one embodiment, the multi-mode road perception program 112 pre-processes the spectrogram (also called a "spectrogram image") of the audio signal using, for example, contrast enhancement, or conversion from a color image to a grayscale image, or a combination thereof. In one embodiment, the multi-mode road perception program 112 filters the (pre-processed) spectrogram image using binarization, i.e., conversion from a grayscale image to a binary image. In some embodiments, the multi-modal road recognition program 112 uses one or more filtering methods on the spectrogram image, such as Otsu's method, local mean method, local Gaussian method, etc. In one embodiment, the multi-modal road recognition program 112 completes image morphology processing on the (pre-processed and filtered) spectrogram image using kernels of any shape and size for attenuation and magnification operations. In one embodiment, the multi-modal road recognition program 112 finds the locations of different road segments and the locations of road segment switching points by searching for maximal connected domains in the processed spectrogram image. The multi-modal road recognition program 112 finds all the connected domains (white domains) and from these identifies the maximal connected domains that are likely to be various road segments.

[0023] In the third mode, the multi-modal road perception program 112 uses a machine learning model to predict sounds for each frame of the received audio signal and calculates the similarity between the predicted value and the actual value, where the similarity is lower than a pre-set threshold at the road section switching point. In one embodiment, the multi-modal road perception program 112 uses a machine learning model trained on historical audio signal data from road tests. In one embodiment, the multi-modal road perception program 112 learns the spectral centroids using a long short-term memory (LSTM), which is an open-source machine learning model trained on historical data and used herein as a time series prediction model. In one embodiment, the multi-modal road perception program 112 predicts the next frame (i.e., a sampling point or moment in the sound sequence) of the current input audio signal by the LSTM based on the pre-frame of the audio signal. When the performance of the actual sound of the next moment does not match the prediction and the similarity is lower than a certain threshold, it is considered that an identifiable abnormal situation of the machine learning model has occurred. In one embodiment, the multi-modal road perception program 112 receives a pre-set loss threshold based on the performance of the historical data in the machine learning model. In one embodiment, the multi-modal road recognition program 112 calculates the loss between the predicted frame and the detected signal, and the more similar the predicted next frame is to the actual audio performance, the smaller the loss. In one embodiment, the multi-modal road recognition program 112 finds switching points for road segments where the loss exceeds a threshold.

[0024] The database 114 operates as a repository for data received, used, and / or output by the multi-mode road perception program 112. The data received, used, and / or generated may include, but is not limited to, audio signals received from a sound collection device, such as a sensor collecting data during a road test, and any other data received, used, and / or output by the multi-mode road perception program 112. The database 114 may be provided with any type of storage device capable of storing data and configuration files that can be accessed and utilized by the server 110, such as a hard disk drive, a database server, or flash memory. In one embodiment, the database 114 is accessed by the multi-mode road perception program 112 to store and / or access data. In the illustrated embodiment, the database 114 resides on the server 110. In another embodiment, the database 114 may reside on another computing device, server, cloud server, or may be spread across multiple devices at other locations (not shown) within the distributed data processing environment 100, provided that the multi-modal road recognition program 112 has access to the database 114.

[0025] The user computing device 130 operates as a computing device associated with a user through which the user can interact with the multi-modal road recognition program 112 through an application user interface. In the illustrated embodiment, the user computing device 130 includes an instance of a user interface 132. In one embodiment, the user computing device 130 can be a laptop computer, a tablet computer, a smartphone, a smartwatch, an e-reader, smart glasses, a wearable computer, or any programmable electronic device capable of communicating with various components and devices in the distributed data processing environment 100 over the network 105. In general, the user computing device 130 represents one or more programmable electronic devices, or a combination of programmable electronic devices, capable of executing machine-readable program instructions and communicating with other computing devices (not shown) in the distributed data processing environment 100 over a network, such as the network 105.

[0026] The user interface 132 provides an interface between the multi-mode road perception program 112 on the server 110 and a user of the user computing device 130. In one embodiment, the user interface 132 is mobile application software. Mobile application software, or "apps," are computer programs designed to run on smartphones, tablet computers, and other mobile computing devices. In one embodiment, the user interface 132 may be a graphical user interface (GUI) or web user interface (WUI) that can display text, documents, web browser windows, user options, application interfaces, and instructions for operation and can include information (such as graphics, text, and sound) that the program presents to the user, as well as control sequences employed by the user to control the program. The user interface 132 allows the user of the user computing device 130 to view and / or manage the output of the multi-mode road perception program 112.

[0027] 2 is a flow diagram 200 illustrating the operational steps of the multi-modal road recognition program 112 for road segment recognition using a multi-modal perception mechanism. It should be understood that the process illustrated in FIG. 2 illustrates one possible iteration of the multi-modal road recognition program 112.

[0028] In step 210, the multimodal road perception program 112 uses the acoustic spectral density map to locate road segment switching points. In one embodiment, the multimodal road perception program 112 uses the received audio signal to generate an acoustic spectral density map to locate road segment switching points. In one embodiment, the multimodal road perception program 112 preprocesses the incoming audio signal with a background sound filter and a low-pass filter and outputs a processed audio sequence. In one embodiment, the multimodal road perception program 112 generates the acoustic spectral density map using a sliding window process on the audio sequence. For example, the multimodal road perception program 112 uses a window of length 1 to slide the window over the time series data with a step length of 0.5 seconds (s) (1), calculates the spectral density for every 1 s segment of the audio signal, and generates a graph of the change in the spectral density (i.e., the spectral density map) with the horizontal axis being time and the vertical axis being spectral density. In one embodiment, the multimodal road perception program 112 finds at least one corresponding extreme point (i.e., the road segment switching point) in the transition on the spectral density map. The multi-modal road recognition program 112 finds these extreme points based on the change trend of the spectral density on the vertical axis of the diagram. In some embodiments, the multi-modal road recognition program 112 uses a pre-set threshold to filter out extremely low points (i.e., outliers) that are too low.

[0029] In step 220, the multi-mode road perception program 112 uses image processing techniques to process the spectrogram of the sound waves of the received audio signal to find road section switching points. In one embodiment, the multi-mode road perception program 112 uses image processing techniques to process the spectrogram of the sound waves of the received audio signal to find road section switching points. In one embodiment, the multi-mode road perception program 112 uses the frequency of the audio signal to generate a spectrogram of the audio signal. The spectrogram shows a visual representation of the frequency domain characteristics, i.e., the spectrum of the frequencies (i.e., the signal strength or "magnitude") of the received audio signal as the signal varies over time. In one embodiment, the multi-mode road perception program 112 pre-processes the spectrogram of the audio signal (also referred to as a "spectrogram image") using, for example, contrast enhancement, or conversion from a color image to a grayscale image, or a combination thereof. In one embodiment, the multi-mode road perception program 112 filters the (pre-processed) spectrogram image using binarization, i.e., conversion from a grayscale image to a binary image. In some embodiments, the multi-modal road recognition program 112 uses one or more filtering methods on the spectrogram image, such as Otsu's method, local mean method, local Gaussian method, etc. In one embodiment, the multi-modal road recognition program 112 completes image morphology processing on the (pre-processed and filtered) spectrogram image using kernels of any shape and size for attenuation and magnification operations. In one embodiment, the multi-modal road recognition program 112 finds the locations of different road segments and the locations of road segment switching points by searching for maximal connected domains in the processed spectrogram image. The multi-modal road recognition program 112 finds all connected domains (white domains) and from these identifies the maximal connected domains that are likely to be various road segments.

[0030] In step 230, the multimodal road perception program 112 uses a machine learning model to predict sounds for each frame of the received audio signal and calculates the similarity between the predicted value and the actual value, where the similarity is lower than a pre-set threshold at the road section switching point. In one embodiment, the multimodal road perception program 112 uses a machine learning model trained on historical audio signal data from road tests. In one embodiment, the multimodal road perception program 112 learns the spectral centroids using a long short-term memory (LSTM), which is an open source machine learning model trained on historical road test data and used herein as a time series prediction model. In one embodiment, the multimodal road perception program 112 predicts the next frame (i.e., a sampling point or moment in the sequence of sounds) of the received audio signal by the LSTM based on the pre-frame of the audio signal. When the performance of the actual sound at the next moment does not match the prediction and the similarity is lower than a certain threshold, it is considered that an abnormal situation has occurred that the machine learning model can identify and output. In one embodiment, the multimodal road perception program 112 receives a pre-set loss threshold based on the performance of the historical data in the machine learning model. In one embodiment, the multi-modal road recognition program 112 calculates the loss between the predicted frame and the detected signal, and the more similar the predicted next frame is to the actual audio performance, the smaller the loss. In one embodiment, the multi-modal road recognition program 112 finds switching points for road segments where the loss exceeds a threshold.

[0031] In step 240, the multi-mode road recognition program 112 combines the results of the three modes to obtain a set of switch points. In one embodiment, after combining the results, the multi-mode road recognition program 112 uses a pre-set time interval to determine a final set of switch points. In one embodiment, if there are two or more (i.e., two of the three) of the three modes (i.e., the modes identifying the road segment in steps 210, 220, or 230, or a combination thereof) that identified that a road segment switch point occurred within the time interval for a pre-set time interval, such as 3 seconds (s), the multi-mode road recognition program 112 determines that a road segment switch point occurred within the time interval. In one embodiment, the multi-mode road recognition program 112 determines whether two or more of the modes identified a road segment switch point within the pre-set time interval. If two or more of the modes identified a road segment switch point within the pre-set time interval, the multi-mode road recognition program 112 determines that a road segment switch point occurred within the pre-set time interval. If one of the modes or none of the modes identified a road segment switch within the preset time interval, the multi-mode road recognition program 112 determines that a road segment switch did not occur within the preset time interval.

[0032] In another embodiment, if all three of the modes (i.e., the modes identifying the road section in steps 210, 220, or 230, or a combination thereof) identify that a road section switch point occurs within the time interval for a preset time interval, such as 3 seconds (s), the multi-mode road recognition program 112 determines that a road section switch point occurs within the time interval. For example, the multi-mode road recognition program 112 combines the results of the three modes [5s, 50s, 55s, 59s, 64s, 69s], [4s, 50s, 56s, 58s, 64s, 70s], and [4s, 50s, 55s, 58s, 64s, 69s], respectively, to obtain a set of switch points. Using the preset time interval of 3s, the multi-mode road recognition program 112 determines the final set of switch points using the preset time interval. If 0 to 3s, the multi-mode road recognition program 112 identifies that there is no switch point. In the case of 2 to 5s, the multi-mode road recognition program 112 identifies 5s from the first mode, 4s from the second mode, and 4s from the third mode, and therefore the multi-mode road recognition program 112 identifies 4s as the final switching point. In the case of 48 to 51s, the multi-mode road recognition program 112 identifies 50s from the first mode, 50s from the second mode, and 50s from the third mode, and therefore the multi-mode road recognition program 112 identifies 50s as the final switching point. In the case of 54 to 57s, the multi-mode road recognition program 112 identifies 55s from the first mode, 56s from the second mode, and 55s from the third mode, and therefore the multi-mode road recognition program 112 identifies 55s as the final switching point. In the case of 56 to 59s, the multi-mode road recognition program 112 identifies 59s from the first mode, 58s from the second mode, and 58s from the third mode, and therefore, the multi-mode road recognition program 112 identifies 58s as the final switch point.For 62-65s, the multi-mode road recognition program 112 identifies 64s from the first mode, 64s from the second mode, and 64s from the third mode, and thus the multi-mode road recognition program 112 identifies 64s as the final switch point. For 68-71s, the multi-mode road recognition program 112 identifies 69s from the first mode, 70s from the second mode, and 69s from the third mode, and thus the multi-mode road recognition program 112 identifies 69s as the final switch point. For this example, the multi-mode road recognition program 112 identifies the final set of switch points as [4s, 50s, 55s, 58s, 64s, 69s]. This final output can also be expressed in the form of time intervals expressed as [[4s-50s], [55s-58s], [64s-69s]].

[0033] 3 illustrates a block diagram of components of a computing device 300 suitable for running the multi-mode road recognition program 112 by the server 110 in the distributed data processing environment 100 of FIG. 1 in accordance with an embodiment of the present invention. It should be understood that FIG. 3 is only intended to provide an illustration of one implementation and is not intended to suggest any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the illustrated environment may be made.

[0034] Computing device 300 includes a communications fabric 302 that provides communications between cache 316, memory 306, persistent storage 308, communications unit 310, and input / output (I / O) interface 312. Communications fabric 302 may be implemented with any architecture designed to communicate data and / or control information between processors (such as microprocessors, communications and network processors), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 302 may be implemented with one or more buses or crossbar switches.

[0035] Memory 306 and persistent storage 308 are computer readable storage media. In this embodiment, memory 306 includes random access memory (RAM). In general, memory 306 may include any suitable volatile or non-volatile computer readable storage medium. Cache 316 is a high-speed memory that enhances the performance of computer processor 304 by holding recently accessed data and data near the accessed data from memory 306.

[0036] The programs may be stored in persistent storage 308 and memory 306 for execution and / or access via cache 316 by one or more of the respective computer processors 304. In one embodiment, persistent storage 308 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage 308 may include a solid state hard drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0037] The media used by persistent storage 308 may also be removable. For example, a removable hard drive may be used for persistent storage 308. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 308.

[0038] The communications unit 310, in these examples, provides for communication with other data processing systems or devices. In these examples, the communications unit 310 includes one or more network interface cards. The communications unit 310 may communicate using either or both physical and wireless communications links. Programs may be downloaded to the persistent storage 308 through the communications unit 310.

[0039] The I / O interface 312 allows for the input and output of data with other devices that may be connected to the server 110. For example, the I / O interface 312 may provide a connection to an external device 318, such as a keyboard, a keypad, a touch screen, or some other suitable input device, or a combination thereof. The external device 318 may also include a portable computer-readable storage medium, such as, for example, a thumb drive, a portable optical or magnetic disk, and a memory card. Software and data used to practice embodiments of the present invention may be stored on such a portable computer-readable storage medium and loaded into the persistent storage 308 via the I / O interface 312. The I / O interface 312 also connects to a display 320.

[0040] Display 320 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0041] Programs described herein are identified based on the application in which they are executed in particular embodiments of the invention, but it should be understood that any particular program nomenclature herein is used merely as a matter of convenience, and thus the invention should not be limited to use in any particular application identified and / or suggested by such nomenclature.

[0042] The present invention may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions for causing a processor to carry out aspects of the present invention.

[0043] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge-in-groove structures having instructions recorded thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein should not be construed as being inherently transitory signals, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0044] The computer readable program instructions described herein can be downloaded from the computer readable storage medium to each computing / processing device or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage on the computer readable storage medium within the respective computing / processing device.

[0045] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language, or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, or partially on the user's computer, as a stand-alone software package, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic device, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electronic circuitry to perform aspects of the present invention.

[0046] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0047] These computer readable program instructions may be provided to a general purpose computer, special purpose computer, or other programmable data processing apparatus processor to produce a machine, the instructions of which execute via the processor of the computer, or other programmable data processing apparatus processor, to produce means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium, such that the computer readable storage medium having stored thereon an article of manufacture including instructions for performing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0048] Also, the computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to perform a series of operational steps on the computer, other programmable apparatus, or other device to produce a computer-executed process, such that the instructions execute on the computer, other programmable apparatus, or other device to perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0049] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may be performed out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order depending on the functionality involved. It is also noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be executed by a dedicated hardware-based system that performs the specified functions or acts or executes a combination of dedicated hardware and computer instructions.

[0050] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. The terminology used in this specification has been selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. receiving, by one or more processors, an audio signal from a road test; processing, by the one or more processors, in a first mode, the audio signal to generate an audio spectral density map and identify at least one respective road section switch; processing, by the one or more processors, in a second mode, a spectrogram of the sound waves of the audio signal to identify each of the at least one road section switch point; and in a third mode, using a machine learning model to predict an expected sound for each frame of the audio signal, calculate a similarity between the expected sound and an actual sound, and identify the respective at least one road switch point when the similarity is less than a pre-set similarity threshold. combining, by the one or more processors, results of the first mode, the second mode, and the third mode to obtain a final set of road section switching points.

4. A computer-implemented method comprising:

2. Processing the audio signal to generate the sound spectral density map and identifying each of the at least one road section switch points comprises: preprocessing, by the one or more processors, the audio signal with a background sound filter and a low pass filter to produce a preprocessed audio sequence; generating, by the one or more processors, a spectral density map using a sliding window process on the preprocessed audio sequence; and identifying, by the one or more processors, at least one corresponding limit point in the transition on the spectral density distribution map. The computer-implemented method of claim 1 , comprising:

3. Identifying the at least one corresponding limit point in the transition on the spectral density distribution map includes: finding, by the one or more processors, the at least one corresponding limit point based on a change trend of the spectral density on a vertical axis of the spectral density distribution map; and filtering outlier extreme points on the spectral density distribution map using a preset threshold by the one or more processors. The computer-implemented method of claim 2 , comprising:

4. processing the spectrogram of sound waves of the audio signal to identify each of the at least one road section switch points; generating, by the one or more processors, a spectrogram image of the audio signal using the frequencies of the audio signal; pre-processing, by the one or more processors, the spectrogram image of the audio signal using contrast enhancement and color to grayscale image conversion; filtering, by the one or more processors, the preprocessed spectrogram image using binarization to convert the grayscale image to a binary image; completing, by the one or more processors, image morphology processing on the preprocessed and filtered spectrogram image using attenuation and broadening computation kernels to generate a fully processed spectrogram image; and identifying, by the one or more processors, each of the at least one road segment switch point by searching for a maximum connected domain in the fully processed spectrogram image. The computer-implemented method of claim 1 , comprising:

5. the machine learning model is trained with historical audio signal data from road tests; The machine learning model is a long short-term memory (LSTM) model.

10. The computer-implemented method of claim 1.

6. The computer-implemented method of claim 1 , wherein the pre-set similarity threshold is based on performance of the machine learning model on historical data.

7. The computer-implemented method of claim 1 , wherein the more similar the predicted sound is to the actual sound, the less the similarity between the predicted sound and the actual sound.

8. one or more computer readable storage media; and program instructions collectively stored on the one or more computer readable storage media, the stored program instructions comprising: program instructions for receiving an audio signal from a road test; program instructions for processing the audio signal in a first mode to generate an audio spectral density map and identify each of at least one road segment switch; and program instructions for processing, in a second mode, a spectrogram of the sound waves of the audio signal to identify each of the at least one road section switch. and program instructions for, in a third mode, using a machine learning model to predict an expected sound for each frame of the audio signal, to calculate a similarity between the expected sound and an actual sound, and to identify the respective at least one road switch point when the similarity is less than a pre-defined similarity threshold; and program instructions for combining results of the first mode, the second mode, and the third mode to obtain a final set of road segment switch points; 2. A computer program product comprising:

9. The program instructions for processing the audio signal to generate the sound spectral density map and to identify each of the at least one road section switch points include: program instructions for preprocessing the audio signal with a background sound filter and a low pass filter to produce a preprocessed audio sequence; program instructions for generating a spectral density map using a sliding window process on the preprocessed audio sequence; and and program instructions for identifying at least one corresponding limit point in the transition on the spectral density distribution map.

9. The computer program product of claim 8, comprising:

10. The program instructions for identifying the at least one corresponding extreme point in the transition on the spectral density distribution map include: program instructions for finding the at least one corresponding limit point based on a change trend of the spectral density on a vertical axis of the spectral density distribution map; and and program instructions for filtering outlier limit points on said spectral density distribution plot using a pre-defined threshold.

10. The computer program product of claim 9, comprising:

11. The program instructions for processing the spectrogram of sound waves of the audio signal to identify the respective at least one road section switch point, comprising: program instructions for generating a spectrogram image of the audio signal using the frequencies of the audio signal; program instructions for pre-processing the spectrogram image of the audio signal using contrast enhancement and color to grayscale image conversion; program instructions for filtering the preprocessed spectrogram image using binarization to convert the grayscale image to a binary image; program instructions for completing image morphology processing on the preprocessed and filtered spectrogram image using kernels for attenuation and broadening operations to generate a fully processed spectrogram image; and program instructions for identifying each of said at least one road segment switch points by searching for a maximum connected domain in said fully processed spectrogram image; 9. The computer program product of claim 8, comprising:

12. the machine learning model is trained with historical audio signal data from road tests; The machine learning model is a long short-term memory (LSTM) model.

9. A computer program product according to claim 8.

13. 10. The computer program product of claim 8, wherein the pre-set similarity threshold is based on performance of the machine learning model on historical data.

14. 9. The computer program product of claim 8, wherein the more similar the predicted sound is to the actual sound, the less similarity between the predicted sound and the actual sound.

15. one or more computer processors; one or more computer readable storage media; and program instructions collectively stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors, the stored program instructions comprising: program instructions for receiving an audio signal from a road test; program instructions for processing the audio signal in a first mode to generate an audio spectral density map and identify each of at least one road segment switch; and program instructions for processing, in a second mode, a spectrogram of the sound waves of the audio signal to identify each of the at least one road section switch. and program instructions for, in a third mode, using a machine learning model to predict an expected sound for each frame of the audio signal, to calculate a similarity between the expected sound and an actual sound, and to identify the respective at least one road switch point when the similarity is less than a pre-defined similarity threshold; and program instructions for combining results of the first mode, the second mode, and the third mode to obtain a final set of road segment switch points; 1. A computer system comprising:

16. The program instructions for processing the audio signal to generate the sound spectral density map and to identify each of the at least one road section switch points include: program instructions for preprocessing the audio signal with a background sound filter and a low pass filter to produce a preprocessed audio sequence; program instructions for generating a spectral density map using a sliding window process on the preprocessed audio sequence; and and program instructions for identifying at least one corresponding limit point in the transition on the spectral density distribution map.

16. The computer system of claim 15, comprising:

17. The program instructions for identifying the at least one corresponding extreme point in the transition on the spectral density distribution map include: program instructions for finding the at least one corresponding limit point based on a change trend of the spectral density on a vertical axis of the spectral density distribution map; and and program instructions for filtering outlier limit points on said spectral density distribution plot using a pre-defined threshold.

20. The computer system of claim 16, comprising:

18. The program instructions for processing the spectrogram of sound waves of the audio signal to identify the respective at least one road section switch point, comprising: program instructions for generating a spectrogram image of the audio signal using the frequencies of the audio signal; program instructions for pre-processing the spectrogram image of the audio signal using contrast enhancement and color to grayscale image conversion; program instructions for filtering the preprocessed spectrogram image using binarization to convert the grayscale image to a binary image; program instructions for completing image morphology processing on the preprocessed and filtered spectrogram image using kernels for attenuation and broadening operations to generate a fully processed spectrogram image; and program instructions for identifying each of said at least one road segment switch points by searching for a maximum connected domain in said fully processed spectrogram image; 16. The computer system of claim 15, comprising:

19. the machine learning model is trained with historical audio signal data from road tests; The machine learning model is a long short-term memory (LSTM) model.

16. The computer system of claim 15.

20. 16. The computer system of claim 15, wherein the pre-set similarity threshold is based on performance of historical data on the machine learning model.

Citation Information

Patent Citations

  • Method for acoustic detection of road and tire conditions

    JP2017505430A

  • Vehicle noise canceller

    JP2018169525A

  • System for monitoring acoustic scene outside vehicle

    JP2020152372A

  • Road surface evaluation system and evaluation method

    JP2021014742A