Vehicle abnormal sound detection method and device, vehicle and storage medium

Abstract

The embodiment of the invention provides a vehicle abnormal sound detection method and device, a vehicle and a storage medium, and the method comprises the steps: obtaining an audio signal of the vehicle, and converting the audio signal into initial image data; judging whether a preset graph exists in the initial image data in a preset abnormal sound interval or not; when it is judged that the initial image data has the preset graph in the preset abnormal sound interval, a preset abnormal sound similarity detection model is adopted to perform graph similarity detection on the preset graph in the initial image data, and similarity information of the audio signals is determined, the abnormal sound similarity detection model comprises a first detection sub-model for carrying out similarity detection by adopting graphic comparison and / or a second detection sub-model for carrying out similarity detection by adopting RGB (Red, Green, Blue) comparison; and carrying out abnormal sound detection on the vehicle based on the similarity information. According to the embodiment of the invention, the abnormal sound similarity detection model is adopted to carry out abnormal sound recognition on the image converted from the audio signal, and compared with manual detection, the method is more objective and accurate.

Description

Technical Field

[0001] The present invention relates to the field of vehicle detection, and in particular to a vehicle abnormal noise detection method and device, a vehicle, and a storage medium. Background Art

[0002] The problem of abnormal noise in the whole vehicle is a key issue that customers can strongly perceive and affects the user experience. OEMs have invested a lot in vehicle testing to ensure quality. Inadequate abnormal noise detection will lead to major problems such as user complaints and brand damage in the market. Therefore, the accuracy of abnormal noise detection in the whole vehicle is particularly important.

[0003] At present, abnormal noise detection is traditionally performed through manual subjective judgment, which carries risks such as non-unique judgment criteria and problem omissions. Summary of the invention

[0004] In view of the above problems, a vehicle abnormal noise detection method and device, a vehicle, and a storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A vehicle abnormal noise detection method, the method comprising:

[0006] Acquire an audio signal of a vehicle, and convert the audio signal into initial image data;

[0007] Determining whether the initial image data contains a preset pattern within a preset abnormal sound interval;

[0008] When it is determined that the initial image data contains a preset pattern within the preset abnormal sound interval, a preset abnormal sound similarity detection model is used to perform pattern similarity detection on the pattern in the initial image data to determine similarity information of the audio signal, wherein the abnormal sound similarity detection model includes a first detection sub-model that uses pattern comparison to perform similarity detection and / or a second detection sub-model that uses RGB comparison to perform similarity detection;

[0009] Abnormal noise detection is performed on the vehicle based on the similarity information.

[0010] Optionally, the performing abnormal noise detection on the vehicle based on the similarity information includes:

[0011] determining whether the vehicle has an abnormal sound according to the similar information;

[0012] When it is determined that the vehicle has an abnormal noise, the type of the abnormal noise of the vehicle is determined to be an abnormal noise type corresponding to the preset abnormal noise interval.

[0013] Optionally, the adopting a preset abnormal sound similarity detection model to perform graphic similarity detection on a preset graphic in the initial image data to determine the similarity information of the audio signal includes:

[0014] The initial image data is subjected to color channel separation, and after extracting the V channel, the background is removed to obtain the first image data;

[0015] The first image data is input into the first detection sub-model that uses graphic comparison to perform similarity detection, and the first image data is graphic-compared with a preset image in the first detection sub-model to output first similarity information of the audio signal.

[0016] Optionally, the adopting a preset abnormal sound similarity detection model to perform graphic similarity detection on a preset graphic in the initial image data to determine the similarity information of the audio signal includes:

[0017] Determining a preset graphic in the initial image data;

[0018] The second image data composed of RGB data corresponding to the preset graphic is input into the second detection sub-model that uses RGB comparison to perform similarity detection. In the second detection sub-module, the second image data is compared with the RGB data of the preset image graphic, and the second similarity information of the audio signal is output.

[0019] Optionally, it also includes:

[0020] When it is determined based on the first similarity information that the vehicle does not have abnormal sound detection, second image data is determined according to RGB data corresponding to a preset pattern of the first image data;

[0021] The second image data is input into the second detection sub-model that uses RGB comparison to perform similarity detection, and second similarity information of the audio signal is output.

[0022] Optionally, the similarity information includes similarity of abnormal sound intervals, similarity of interference intervals, and similarity of normal intervals, and the abnormal sound detection on the vehicle based on the similarity information includes:

[0023] If the similarity of the abnormal sound interval is greater than a first threshold, determining that the vehicle has an abnormal sound;

[0024] If the similarity of the normal interval is greater than a second threshold, determining that the vehicle does not have an abnormal sound;

[0025] If the sum of the similarity of the interference section and the similarity of the normal section is greater than a third threshold, it is determined that the vehicle does not have an abnormal noise.

[0026] Optionally, it also includes:

[0027] A preset ROI region is extracted from the initial image data, and the preset ROI region is preprocessed.

[0028] A vehicle abnormal noise detection device, the device comprising:

[0029] An audio conversion module, used for acquiring an audio signal of a vehicle and converting the audio signal into initial image data;

[0030] An abnormal sound type prediction module, used to determine whether the initial image data has a preset pattern within a preset abnormal sound interval;

[0031] a similarity information determination module, configured to, when determining that the initial image data contains a preset pattern within the preset abnormal sound interval, perform pattern similarity detection on the pattern in the initial image data using a preset abnormal sound similarity detection model to determine the similarity information of the audio signal, wherein the abnormal sound similarity detection model includes a first detection sub-model that uses pattern comparison to perform similarity detection and / or a second detection sub-model that uses RGB comparison to perform similarity detection;

[0032] An abnormal sound detection module is used to detect abnormal sounds of the vehicle based on the similarity information.

[0033] A vehicle comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the vehicle abnormal noise detection method as described above when executed by the processor.

[0034] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the vehicle abnormal noise detection method as described above is implemented.

[0035] The embodiments of the present invention have the following advantages:

[0036] In an embodiment of the present invention, an audio signal of a vehicle is acquired and converted into initial image data; it is determined whether a preset pattern exists in the preset abnormal sound interval in the initial image data; when it is determined that the preset pattern exists in the preset abnormal sound interval in the initial image data, a preset abnormal sound similarity detection model is used to perform pattern similarity detection on the pattern in the initial image data, similarity information of the audio signal is determined, and abnormal sound detection is performed on the vehicle based on the similarity information, thereby realizing abnormal sound recognition of the image converted from the audio signal using the abnormal sound similarity detection model, which is more objective and accurate than manual detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0038] Figure 1a is a flowchart of a method for detecting abnormal noise in a vehicle provided by an embodiment of the present invention;

[0039] Figure 1b is a schematic diagram of image data provided by an embodiment of the present invention;

[0040] Figure 1c is another schematic diagram of image data provided by an embodiment of the present invention;

[0041] Figure 2 is a flowchart of another vehicle abnormal noise detection method provided by an embodiment of the present invention;

[0042] Figure 3 is a flowchart of another vehicle abnormal noise detection method provided by an embodiment of the present invention;

[0043] Figure 4 It is a schematic diagram of a vehicle abnormal noise processing process provided by an embodiment of the present invention;

[0044] Figure 5 It is a structural schematic diagram of a vehicle abnormal noise detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Reference Figure 1a , shows a flow chart of a method for detecting abnormal noise in a vehicle provided by an embodiment of the present invention, which may specifically include the following steps:

[0047] Step S101, acquiring an audio signal of a vehicle and converting the audio signal into initial image data;

[0048] Among them, audio signals can be obtained from multiple sources in the vehicle, such as engine sound, tire noise, wind noise, car audio system, etc. These signals are usually captured by microphones or sensors installed inside the vehicle.

[0049] In practical applications, audio signals can be collected by audio collection equipment installed in the vehicle, and then the audio signals can be converted into image data. Among them, the captured audio signal is a function of time, indicating the change of sound over time.

[0050] Specifically, the audio signal first needs to be processed to extract useful information. The collected audio signal is subjected to Fourier transformation, and then a time-frequency graph can be drawn with the horizontal axis as frequency, the vertical axis as time, and the color representing the amplitude, so as to realize the conversion of the audio signal into image data.

[0051] Among them, the Fourier transform is a mathematical transformation that converts a time domain signal (a signal that changes with time) into a frequency domain signal (a signal that changes with frequency). For non-stationary signals, the Short-Time Fourier Transform (STFT) is usually used, which applies the Fourier transform on different time periods of the signal to capture the time-localized frequency information.

[0052] After the STFT, a series of spectra are obtained, each corresponding to a short time period of the signal. These spectra can be arranged into a two-dimensional array, where one dimension represents frequency and the other represents time. This two-dimensional array can be visualized as a time-frequency graph, where the horizontal axis represents frequency, the vertical axis represents time, and the color or brightness represents the amplitude of the signal (i.e. the intensity of the frequency component). Then, the above two-dimensional array can be plotted into an image using graphics processing software or programming languages ​​(such as the Matplotlib library in Python).

[0053] In an image, high-amplitude frequency components can be represented by brighter colors or higher brightness, while low-amplitude frequency components can be represented by darker colors or lower brightness. At this time, the time-frequency diagram provides an intuitive way to observe the changes of the signal at different times and frequencies. The time-frequency diagram can simultaneously display the time and frequency characteristics of the signal, helping analysts quickly identify patterns and anomalies in the signal.

[0054] In one embodiment of the present invention, a preset ROI region is extracted from the initial image data, and the preset ROI region is preprocessed.

[0055] The preprocessing may include but is not limited to: any one of Gaussian filtering, histogram equalization, edge smoothing and binarization.

[0056] Among them, ROI (Region of Interest) refers to a part of an image or data that users or algorithms pay special attention to in image processing, computer vision or data analysis. The selection of ROI region is usually based on specific analysis requirements, such as object detection, image segmentation, feature extraction, pattern recognition, etc.

[0057] In image processing and computer vision, ROI area selection can be done in the following ways:

[0058] Manual selection: The user specifies the region of interest directly on the image, usually by drawing a rectangular, circular, or polygonal box to mark it.

[0059] Automatic selection: Use algorithms to automatically detect and select the ROI region. For example, edge detection, color segmentation, motion detection, or machine learning-based object detection algorithms can be used to identify specific objects or regions in the image.

[0060] Specifically, edge detection algorithms (such as Canny, Sobel), threshold segmentation algorithms (such as Otsu, adaptive threshold), region growing algorithms, deep learning-based target detection and segmentation algorithms (such as YOLO, Mask R-CNN) and other algorithms can be used to automatically select ROI areas. These algorithms and technologies can help automatically identify and select ROI areas in images according to different application scenarios and needs.

[0061] Based on prior knowledge: Use known prior information (such as the location, size, shape, etc. of the object) to define the ROI area.

[0062] In the embodiment of the present invention, selecting the ROI region can realize processing only the region of interest, which can reduce the amount of calculation and speed up the processing. Focusing on a specific region can reduce the interference of background noise and improve the accuracy of analysis. The selection of the ROI region can also simplify subsequent image processing steps, such as feature extraction and pattern recognition.

[0063] Step S102, determining whether the initial image data has a preset pattern within a preset abnormal sound interval;

[0064] The preset abnormal sound interval is a preset image area, which can be specifically determined according to the area of ​​the graphics used to represent the abnormal sound in the images generated by different abnormal sound signals. The preset graphics are graphics other than normal graphics in the image with the abnormal sound signal. The preset graphics are most likely formed due to the presence of abnormal sound, and may also be formed due to non-abnormal sound factors such as environmental interference sound sources.

[0065] After obtaining the initial image data, the image area corresponding to the preset abnormal sound interval is determined in the initial image data, and then it is determined whether there is a preset pattern in the image area. When the preset pattern exists in the image area, it can be determined that there is a high probability of an abnormal sound in the audio signal. Whether there is an abnormal sound needs further confirmation. When the preset pattern does not exist in the image area, it can be determined that there is no abnormal sound in the audio signal, and the abnormal sound detection is ended.

[0066] like Figure 1b As shown, it is a schematic diagram of image data in an embodiment of the present invention, wherein the left side is an image without abnormal sound, the right side is an image with abnormal sound, and the horizontal line part in the middle of the right image is the abnormal part, so that the area where the abnormal part is located can be set as the abnormal sound interval.

[0067] like Figure 1c As shown, it is another schematic diagram of image data in an embodiment of the present invention, wherein the left side is an image without abnormal sound, the right side is an image with abnormal sound, and the lower left corner area of ​​the right side image is the abnormal sound area.

[0068] Step S103, when it is determined that the initial image data has a preset pattern within the preset abnormal sound interval, a preset abnormal sound similarity detection model is used to perform pattern similarity detection on the pattern in the initial image data to determine similarity information of the audio signal;

[0069] The abnormal sound similarity detection model includes a first detection sub-model that uses graphic comparison to perform similarity detection and / or a second detection sub-model that uses RGB comparison to perform similarity detection.

[0070] When the initial image data contains a preset pattern within the preset abnormal sound interval, there is a high probability that an abnormal sound exists in the audio signal. At this time, the pattern similarity can be further determined in combination with a pre-trained abnormal sound similarity detection model, and then the abnormal sound detection of the vehicle can be performed according to the pattern similarity.

[0071] The training process of the abnormal sound similarity detection model is as follows: obtain the historical audio signals of the vehicle, wherein the historical audio signals include audio signals with abnormal sounds and audio signals without abnormal sounds. After converting the historical audio signals into image data, annotate the abnormal sound type for each image data, and determine the normal interval, abnormal sound interval and interference sound source interval of each abnormal sound type, and then select the initial model type of the abnormal sound similarity detection model, as well as model hyperparameters (such as the number of iterations, learning rate, batch size, etc.) and model parameters.

[0072] The processed image data can then be input into the initial model, and then, after being processed by the model, the graphic similarity data is output, the abnormal sound detection result is determined based on the graphic similarity data, and then the abnormal sound detection result is compared with the corresponding label, the accuracy of the model is statistically analyzed, and then the model parameters are adjusted based on feedback so that the result output by the model is close to the label. After multiple iterations of this method, a trained abnormal sound similarity detection model is obtained.

[0073] In one embodiment of the present invention, the first detection sub-model that uses graphic comparison to perform similarity detection and / or the second detection sub-model that uses RGB comparison to perform similarity detection in the abnormal sound similarity detection model can be trained using the above-mentioned model training method to obtain the final model for abnormal sound detection.

[0074] Specifically, the first detection sub-model can be trained using a first neural network, and the second detection sub-model can be trained using a second neural network. The first neural network model is suitable for performing graphic comparison to determine initial image data for graphic similarity, and the second neural network model is suitable for performing RGB value comparison to determine initial image data for graphic similarity.

[0075] In an embodiment of the present invention, the input data of the first detection sub-model is image data, and the output data is graphic similarity information. In the first detection sub-model, a graphic comparison method can be applied to determine the graphic similarity. In the graphic comparison method, after the coordinate matching of the image to be detected and the preset image is completed, the image can be compared with the preset image, and the extra graphics are regarded as abnormal. The preset image can include an image with abnormal sound signals, an image with interference, and a normal image. The image similarity information can be determined by image comparison, such as the similarity of abnormal intervals, the similarity of interference intervals, and the similarity of normal intervals.

[0076] The input data of the second detection submodule is image data, and the output data is graphic similarity information. The RGB comparison method can be applied in the second detection submodule to determine the similarity of the graphics. In the RGB comparison method, the RGB color of each graphic can be compared with the expected value range (the expected value range can be determined based on the RGB value of the preset image), and the graphics that exceed the range can be extracted and determined to be abnormal.

[0077] Among them, the first neural network model and the second neural network model can be selected from any of the following:

[0078] Feedforward Neural Networks (FNN): Data flows in only one direction, from the input layer to the output layer, without loops or feedback. For example, Multilayer Perceptron (MLP), used for handwritten digit recognition, etc.

[0079] Convolutional Neural Networks (CNN): designed to process data with a grid-like structure, such as images. For example, LeNet, AlexNet, VGGNet, GoogLeNet, ResNet, used for image classification, object detection, face recognition, etc.

[0080] Recurrent Neural Networks (RNN): Has internal state (memory) and can process sequential data, such as time series, text, etc. For example, Long Short-Term Memory Network (LSTM) and Gated Recurrent Unit (GRU) are used for language models, machine translation, speech recognition, etc.

[0081] Autoencoders: An unsupervised learning model used to learn a compact representation (encoding) of data. For example, variational autoencoders (VAEs) are used for data dimensionality reduction, feature learning, generative models, etc.

[0082] Generative Adversarial Networks (GAN): It consists of two networks: the generator and the discriminator, which generate new data samples through an adversarial process. For example, DeepFake technology, image super-resolution, artistic creation, etc.

[0083] Graph Neural Networks (GNN): used to process graph structure data, such as social networks, molecular structures, etc. For example, graph convolutional networks (GCN) and graph attention networks (GAT) are used for node classification, link prediction, graph classification, etc.

[0084] Reinforcement Learning Networks: Learn the optimal strategy through interaction with the environment to maximize the cumulative reward. For example: Deep Q-Network (DQN), Proximal Policy Optimization (PPO), used for game play, robot control, autonomous driving, etc.

[0085] Attention Mechanism Networks: Allows the model to focus on specific parts of the data when processing information, improving the performance of processing sequence data. For example, the Transformer model is used for machine translation, text summarization, question-answering systems, etc.

[0086] Each type of neural network has its unique advantages and applicable scenarios. In practical applications, the appropriate model type can be selected for training to improve the accuracy of vehicle abnormal noise detection.

[0087] In an embodiment of the present invention, when conducting model training data, it is also possible to pre-record external environmental interference sound sources (based on the site's traffic sounds, horn sounds, talking sounds, etc.), classify and label the graphics of the interference sound sources, and then determine the interference interval based on the graphics of the interference sound sources.

[0088] By adding an interfering sound source during the model training process, the model can learn the characteristics of the interfering sound, thereby improving the accuracy of the abnormal sound detection method in the embodiment of the present invention.

[0089] Step S104: abnormal noise detection is performed on the vehicle based on the similarity information.

[0090] After the similarity information is determined by using the abnormal sound similarity detection model, the abnormal sound detection of the vehicle can be performed according to the similarity to determine whether the audio signal of the vehicle has an abnormal sound.

[0091] In one embodiment of the present invention, the similarity information may include the similarity of abnormal sound intervals, the similarity of interference intervals and the similarity of normal intervals, and abnormal sound detection is performed on the vehicle based on the similarity information, including: if the similarity of the abnormal sound intervals is greater than a first threshold, it is determined that the vehicle has abnormal sound; if the similarity of the normal interval is greater than a second threshold, it is determined that the vehicle does not have abnormal sound; if the sum of the similarity of the interference interval and the similarity of the normal interval is greater than a third threshold, it is determined that the vehicle does not have abnormal sound.

[0092] In practical applications, the similarity can be divided into the similarity of the abnormal sound interval, the similarity of the interference interval and the similarity of the normal interval, so that the abnormal sound analysis can be performed in combination with the similarity of each interval.

[0093] Specifically, the abnormal sound judgment condition may be set based on the similarity of the abnormal sound interval, the similarity of the interference interval, and the similarity of the normal interval. For example, the abnormal sound judgment condition may be set as follows:

[0094] a. The similarity score of the abnormal sound label graphics in the abnormal sound interval is higher than the threshold;

[0095] b. The similarity score of the label graphics of the normal area and the interference sound source interval is higher than the threshold;

[0096] c. The similarity score of the interference sound source label graphic is higher than the threshold;

[0097] d. The graphic similarity score of the normal area is higher than the threshold;

[0098] If a|a&c are satisfied, it is judged as abnormal noise.

[0099] If b|d is satisfied, it is judged to be normal.

[0100] In one embodiment of the present invention, abnormal noise detection of a vehicle may include determining whether the vehicle has abnormal noise, and may also include what type of abnormal noise the vehicle has. Performing abnormal noise detection on a vehicle based on similarity information includes: determining whether the vehicle has abnormal noise based on similarity information; and when determining that the vehicle has abnormal noise, determining the type of abnormal noise of the vehicle to be the type of abnormal noise corresponding to a preset abnormal noise interval.

[0101] The abnormal noise type may include but is not limited to any one of interior resonance, motor whistling, friction noise, and impact noise. Different types of abnormal noises have different areas where the images show abnormal noises. Therefore, the abnormal noise intervals corresponding to different abnormal noise types can be determined in advance by analyzing a large number of vehicle audio signals. Furthermore, when the initial image data with the abnormal noise interval is detected using the abnormal noise similarity detection model, and the detection result shows that an abnormal noise exists, it can be determined that the garage has an abnormal noise corresponding to the abnormal noise interval.

[0102] In practical applications, the initial image data may include graphics corresponding to multiple abnormal sound intervals, and the abnormal sound similarity detection model may be used to perform similarity detection on each graphic respectively.

[0103] In one embodiment of the present invention, an abnormal sound similarity detection model can be trained for different abnormal sound types. The trained abnormal sound similarity detection model can perform graphic similarity detection on the graphics within the abnormal sound interval corresponding to the abnormal sound type. Specifically, when it is determined that the initial image data contains graphics within the preset abnormal sound interval, a target abnormal sound similarity detection model corresponding to the preset abnormal sound area can be determined from multiple abnormal sound similarity detection models.

[0104] In an embodiment of the present invention, an audio signal of a vehicle is acquired and converted into initial image data; it is determined whether a preset pattern exists in the preset abnormal sound interval in the initial image data; when it is determined that the preset pattern exists in the preset abnormal sound interval in the initial image data, a preset abnormal sound similarity detection model is used to perform pattern similarity detection on the pattern in the initial image data, similarity information of the audio signal is determined, and abnormal sound detection is performed on the vehicle based on the similarity information, thereby realizing abnormal sound recognition of the image converted from the audio signal using the abnormal sound similarity detection model, which is more objective and accurate than manual detection.

[0105] Reference Figure 2 , shows a flowchart of another vehicle abnormal noise detection method provided by an embodiment of the present invention, which may specifically include the following steps:

[0106] Step S201, acquiring an audio signal of a vehicle and converting the audio signal into initial image data;

[0107] Step S202, determining whether the initial image data has a preset pattern within a preset abnormal sound interval;

[0108] Step S203, when it is determined that the initial image data has a preset pattern within the preset abnormal sound interval, the initial image data is separated into color channels, and the background is removed after the V channel is extracted to obtain the first image data;

[0109] In the embodiment of the present invention, color channel separation refers to the process of separating the color channels of a color image and processing or analyzing them separately. In digital image processing, color image formats such as RGB (red, green, blue) and HSV (hue, saturation, brightness) contain multiple color channels.

[0110] In the RGB color model, an image consists of three channels:

[0111] Red channel (R): Contains the red component of all pixels in the image.

[0112] Green channel (G): Contains the green component of all pixels in the image.

[0113] Blue channel (B): Contains the blue component of all pixels in the image.

[0114] By separating these channels, you can process the red, green, and blue components separately. For example, you can enhance the contrast of a channel or analyze the characteristics of a channel.

[0115] In the HSV color model, an image consists of three channels:

[0116] Hue channel (H): Indicates the type of color, such as red, blue, etc.

[0117] Saturation channel (S): Indicates the purity or vividness of color.

[0118] Luminance channel (V): Indicates the brightness or lightness of a color.

[0119] Separating the HSV channels can help us better understand the color distribution and brightness distribution of the image, thereby performing more sophisticated image processing.

[0120] The color channel separation in the embodiment of the present invention improves the image quality by adjusting the contrast or brightness of a single channel. It can also detect a specific color in the image by analyzing the value of a specific channel. At the same time, the background of the image can be removed by separating and processing the brightness channel. In addition, image segmentation can also be performed by analyzing the distribution of color channels.

[0121] Step S204, inputting the first image data into a first detection sub-model that uses graphic comparison to perform similarity detection, performing graphic comparison between the first image data and a preset image in the first detection sub-model, and outputting first similarity information of the audio signal;

[0122] After obtaining the first image data, the first image data may be input into the first detection sub-model. The first detection sub-model uses a graphic comparison method to compare the first image data with a preset image, calculates the graphic similarity, and outputs the similarity data.

[0123] Step S205, performing abnormal noise detection on the vehicle based on the first similarity information;

[0124] After obtaining the first similarity information, the first similarity information may be compared with a preset abnormal noise judgment condition to implement abnormal noise detection of the vehicle.

[0125] In one embodiment of the present invention, when it is determined that there is no abnormal sound detection of the vehicle based on the first similarity information, the second image data can be determined according to the RGB data corresponding to the graphic of the first image data; the second image data is input into the second detection sub-model that uses RGB comparison to perform similarity detection, and the second similarity information of the audio signal is output. The abnormal sound detection of the vehicle is performed based on the second similarity information.

[0126] In practical applications, when it is determined based on the first similarity information that the vehicle has an abnormal noise, the abnormal noise detection result is outputted. When it is determined based on the first similarity information that the vehicle does not have an abnormal noise, the second detection sub-model can be further combined to perform vehicle abnormal noise detection. Specifically, the RGB data of the corresponding area can be determined based on the graphics in the first image data in the first detection sub-model, and then an RGB image is generated. The RGB image is the second image data, and then the second image data can be input into the second detection sub-model for RGB value comparison to determine the graphic similarity, that is, the second similarity information. Based on the second similarity information, the vehicle abnormal noise detection is performed to determine the abnormal noise detection result of the vehicle, that is, whether there is an abnormal noise and the type of abnormal noise.

[0127] By combining the first detection sub-model with the second detection sub-model, accurate detection of abnormal noises in vehicles can be achieved, and the second detection sub-model can further perform RGB value detection based on the graphic features that have been determined by the first detection sub-model. The area that requires RGB value comparison is reduced, which can reduce the data processing amount of the second detection sub-model and reduce the interference of irrelevant data, thereby improving the accuracy of similarity.

[0128] In the embodiment of the present invention, the first detection sub-model for similarity detection using graphic comparison can be used to identify abnormal sounds in images converted from audio signals, which is more objective and accurate than manual detection.

[0129] Reference Figure 3 , shows a flowchart of another vehicle abnormal noise detection method provided by an embodiment of the present invention, which may specifically include the following steps:

[0130] Step S301, acquiring an audio signal of a vehicle and converting the audio signal into initial image data;

[0131] Step S302, determining whether the initial image data has a preset pattern within a preset abnormal sound interval;

[0132] Step S303, when it is determined that the initial image data has a preset pattern within the preset abnormal sound interval, determining the preset pattern in the initial image data;

[0133] When it is determined that there is a preset pattern in the abnormal sound area, the preset pattern in the area can be extracted to further perform abnormal sound judgment.

[0134] Step S304, input the second image data composed of RGB data corresponding to the preset graphic into the second detection sub-model that uses RGB comparison to perform similarity detection, compare the RGB data of the second image data with the preset image graphic in the second detection sub-module, and output second similarity information of the audio signal.

[0135] After determining the graphic, the RGB data within the changed graphic area can be determined. The RGB data of the graphic constitutes the second image data. The second image data is input into the second detection sub-model. In the second detection sub-model, the RGB comparison method can be used to determine the graphic similarity information.

[0136] Step S305: abnormal noise detection is performed on the vehicle based on the second similarity information.

[0137] In the embodiment of the present invention, the second detection sub-model that performs similarity detection by RGB comparison can identify abnormal sounds in the image converted from the audio signal, which is more objective and accurate than manual detection.

[0138] Reference Figure 4 , is a schematic diagram of a vehicle abnormal noise processing process in an embodiment of the present invention, which may specifically include the following process:

[0139] (1) Original audio signal;

[0140] Audio signals can be collected in the vehicle based on the audio collection device.

[0141] (2) Fourier transform time-frequency diagram;

[0142] Perform Fourier transform on the collected audio signal, and then draw a time-frequency graph with the horizontal axis as frequency, the vertical axis as time, and the color representing the amplitude.

[0143] (3) ROI region extraction;

[0144] (4) Image preprocessing;

[0145] The image is preprocessed by Gaussian filtering, histogram equalization, edge smoothing and binarization to obtain a smooth and stable image. The image data is separated into color channels, the V channel is extracted from multiple color channels, and the background is removed.

[0146] (5) Establishment and application of abnormal noise model;

[0147] The process of establishing the abnormal noise model: after a large number of samples of images that are manually judged to have no abnormal noise, the obtained graphic area is used as the normal interval, and the images that are manually judged to have abnormal noise are matched with the normal image interval to obtain the abnormal noise interval; the different abnormal noises in the abnormal noise interval are classified (for example, interior resonance, motor whistling, friction noise, impact noise) and the corresponding abnormal noise image RGB and shape features are extracted for labeling. In order to improve the accuracy of vehicle abnormal noise monitoring, the external environment interference sound source can be pre-recorded at the test site (based on the site's traffic sound, horn sound, talking sound, etc.), and the graphics of the interference sound source can be classified and labeled. By introducing comprehensive analysis of the interference interval, a detection model that is more in line with the actual abnormal noise scene can be established.

[0148] Through a large amount of data statistics, an abnormal noise model library is established, and the normal range, abnormal noise range and interference sound source labels are added to the model library. A convolutional neural network is used for deep learning. The first neural network is trained for graphic shape comparison, and the second neural network is trained for RGB value comparison. The features of the normal range, abnormal noise range and interference sound source are extracted for similarity scoring to determine whether the vehicle's audio signal is an abnormal noise.

[0149] After repeatedly iterating the first neural network model or the second neural network model, the detection accuracy of the first neural network model or the second neural network model is higher than a preset value (set according to the actual scene). Then, the first neural network model or the second neural network model can be applied to perform actual audio signal detection.

[0150] It should be noted that in the embodiment of the present invention, the input data of the first neural network model is the image data after the V channel is separated from the color channel and the background is removed, and the input data of the second neural network model is RGB image data, such as the first image data after preprocessing.

[0151] The first neural network model and the second neural network model are suitable for detecting different types of abnormal sounds. After obtaining the image data, if the type of abnormal sound contained in the image data only matches the first neural network, the first neural network model is used to detect the similarity of abnormal sounds. If the type of abnormal sound contained in the image data only matches the second neural network, the second neural network model is used to detect the similarity of abnormal sounds. If there are multiple types of abnormal sounds contained in the image data, some of which are suitable for the first neural network model and some are suitable for the second neural network model, the image data is respectively applied to the first neural network model and the second neural network model for abnormal sound prediction.

[0152] (6) Presentation of abnormal noise results.

[0153] After being processed by the neural network model, the graphic similarity of the image data can be output, and then the presence of abnormal sound can be judged based on the graphic similarity. Specifically, the presence of abnormal sound can be judged in the following way:

[0154] a. The similarity score of the abnormal sound label graphics in the abnormal sound interval is higher than the threshold.

[0155] b. The similarity score of the label graphics of the normal area and the interference sound source interval is higher than the threshold.

[0156] c. The similarity score of the interference sound source label graphic is higher than the threshold.

[0157] d. The graphic similarity score of the normal area is higher than the threshold.

[0158] If a|a&c are satisfied, it is judged as abnormal noise.

[0159] If b|d is satisfied, it is judged to be normal.

[0160] In the embodiment of the present invention, the time domain signal is collected by the data acquisition device, and the time domain signal is subjected to Fourier transformation, and the time-frequency diagram is obtained as the sample image with the horizontal axis as the frequency, the vertical axis as the time, and the color representing the amplitude. The preset area information in the sample image is extracted, and the sample image is preprocessed by a series of means such as Gaussian filtering and edge smoothing. According to the abnormal sound failure type, the processed sample image is matched with the preset image to establish a model library through a convolutional neural network to obtain the test result. Abnormal sound can be quickly and accurately identified in complex and changeable scenes with more external interference.

[0161] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0162] Reference Figure 5 , shows a schematic diagram of the structure of a vehicle abnormal noise detection device provided by an embodiment of the present invention, which may specifically include the following modules:

[0163] The audio conversion module 501 is used to obtain the audio signal of the vehicle and convert the audio signal into initial image data;

[0164] The abnormal sound type prediction module 502 is used to determine whether the initial image data has a preset pattern within a preset abnormal sound interval;

[0165] a similarity information determination module 503, configured to, when determining that the initial image data contains a preset pattern within the preset abnormal sound interval, perform pattern similarity detection on the preset pattern in the initial image data using a preset abnormal sound similarity detection model to determine the similarity information of the audio signal, wherein the abnormal sound similarity detection model includes a first detection sub-model that uses pattern comparison to perform similarity detection and / or a second detection sub-model that uses RGB comparison to perform similarity detection;

[0166] The abnormal sound detection module 504 is used to perform abnormal sound detection on the vehicle based on the similarity information.

[0167] In an embodiment of the present invention, the abnormal sound detection module 504 may include:

[0168] An abnormal sound judgment submodule, used for judging whether the vehicle has an abnormal sound according to the similar information;

[0169] The abnormal noise type determination submodule is used to determine that the abnormal noise type of the vehicle is the abnormal noise type corresponding to the preset abnormal noise interval when it is determined that the vehicle has an abnormal noise.

[0170] In an embodiment of the present invention, the similarity information determination module 503 may include:

[0171] A first image data determination submodule is used to separate the color channels of the initial image data, extract the V channel and remove the background to obtain the first image data;

[0172] The first detection sub-model detection sub-module is used to input the first image data into the first detection sub-model that uses graphic comparison to perform similarity detection, perform graphic comparison between the first image data and a preset image in the first detection sub-model, and output first similarity information of the audio signal.

[0173] In an embodiment of the present invention, the similarity information determination module 503 may include:

[0174] A graphic determination submodule, used to determine a preset graphic in the initial image data;

[0175] The second detection model detection submodule is used to input the second image data composed of RGB data corresponding to the preset graphic into the second detection submodel that uses RGB comparison for similarity detection, compare the RGB data of the second image data with the preset image in the second detection submodule, and output the second similarity information of the audio signal.

[0176] In one embodiment of the present invention, the device further comprises:

[0177] A second image data generating module, configured to determine second image data according to RGB data corresponding to the graphic of the first image data when it is determined based on the first similarity information that the vehicle does not have abnormal sound detection;

[0178] The execution module is used to input the second image data into the second detection sub-model that uses RGB comparison to perform similarity detection, and output second similarity information of the audio signal.

[0179] In an embodiment of the present invention, the similarity information includes the similarity of the abnormal sound interval, the similarity of the interference interval and the similarity of the normal interval. The abnormal sound detection module 504 may include:

[0180] A first detection condition judgment submodule, configured to determine that the vehicle has an abnormal sound if the similarity of the abnormal sound interval is greater than a first threshold;

[0181] A second detection condition judgment submodule, configured to determine that the vehicle does not have an abnormal sound if the similarity of the normal interval is greater than a second threshold;

[0182] The third detection condition judgment submodule is used to determine that there is no abnormal noise in the vehicle if the sum of the similarity of the interference interval and the similarity of the normal interval is greater than a third threshold.

[0183] In one embodiment of the present invention, the device further comprises:

[0184] The preprocessing module is configured to extract a preset ROI area from the initial image data and perform preprocessing of the preset ROI area.

[0185] In an embodiment of the present invention, an audio signal of a vehicle is acquired and converted into initial image data; it is determined whether the initial image data has a preset graphic within a preset abnormal sound interval; when it is determined that the initial image data has a preset graphic within the preset abnormal sound interval, a preset abnormal sound similarity detection model is used to perform graphic similarity detection on the graphic in the initial image data, similarity information of the audio signal is determined, and abnormal sound detection is performed on the vehicle based on the similarity information, thereby realizing abnormal sound recognition of the image converted from the audio signal using the abnormal sound similarity detection model, which is more objective and accurate than manual detection.

[0186] An embodiment of the present invention further provides a vehicle, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the above vehicle abnormal noise detection method is implemented.

[0187] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above vehicle abnormal noise detection method is implemented.

[0188] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0189] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0190] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0191] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0192] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0193] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0194] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0195] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0196] The above is a detailed introduction to a vehicle abnormal noise detection method and device, electronic device, and storage medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a general technician in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A vehicle abnormal noise detection method, characterized in that: The method comprises: Acquire an audio signal of a vehicle, and convert the audio signal into initial image data; Determining whether the initial image data contains a preset pattern within a preset abnormal sound interval; When it is determined that the initial image data contains a preset pattern within the preset abnormal sound interval, a preset abnormal sound similarity detection model is used to perform pattern similarity detection on the pattern in the initial image data to determine similarity information of the audio signal, wherein the abnormal sound similarity detection model includes a first detection sub-model that uses pattern comparison to perform similarity detection and / or a second detection sub-model that uses RGB comparison to perform similarity detection; Abnormal noise detection is performed on the vehicle based on the similarity information.

2. The method according to claim 1, characterized in that The detecting abnormal noise of the vehicle based on the similarity information includes: determining whether the vehicle has an abnormal sound according to the similar information; When it is determined that the vehicle has an abnormal noise, the type of the abnormal noise of the vehicle is determined to be an abnormal noise type corresponding to the preset abnormal noise interval.

3. The method according to claim 1, characterized in that The adopting a preset abnormal sound similarity detection model to perform graphic similarity detection on the graphics in the initial image data to determine the similarity information of the audio signal includes: The initial image data is subjected to color channel separation, and after extracting the V channel, the background is removed to obtain the first image data; The first image data is input into the first detection sub-model that uses graphic comparison to perform similarity detection, and the first image data is graphic-compared with a preset image in the first detection sub-model to output first similarity information of the audio signal.

4. The method according to claim 1, characterized in that The adopting a preset abnormal sound similarity detection model to perform graphic similarity detection on the graphics in the initial image data to determine the similarity information of the audio signal includes: Determining a preset graphic in the initial image data; The second image data composed of RGB data corresponding to the preset graphic is input into the second detection sub-model that uses RGB comparison for similarity detection. In the second detection sub-module, the second image data is compared with the RGB data of the preset image graphic, and the second similarity information of the audio signal is output.

5. The method according to claim 3, characterized in that: Also includes: When it is determined based on the first similarity information that the vehicle does not have abnormal sound detection, second image data is determined according to RGB data corresponding to a preset pattern of the first image data; The second image data is input into the second detection sub-model that uses RGB comparison to perform similarity detection, and second similarity information of the audio signal is output.

6. The method according to any one of claims 1 to 5, characterized in that The similarity information includes the similarity of the abnormal sound interval, the similarity of the interference interval, and the similarity of the normal interval. The abnormal sound detection of the vehicle based on the similarity information includes: If the similarity of the abnormal sound interval is greater than a first threshold, determining that the vehicle has an abnormal sound; If the similarity of the normal interval is greater than a second threshold, determining that the vehicle does not have an abnormal sound; If the sum of the similarity of the interference section and the similarity of the normal section is greater than a third threshold, it is determined that the vehicle does not have an abnormal noise.

7. The method according to any one of claims 1 to 5, characterized in that Also includes: A preset ROI region is extracted from the initial image data, and the preset ROI region is preprocessed.

8. A vehicle abnormal noise detection device, characterized in that: The device comprises: An audio conversion module, used for acquiring an audio signal of a vehicle and converting the audio signal into initial image data; An abnormal sound type prediction module, used to determine whether the initial image data has a preset pattern within a preset abnormal sound interval; a similarity information determination module, configured to, when determining that the initial image data contains a preset graphic in the preset abnormal sound zone, perform graphic similarity detection on the graphic in the initial image data using a preset abnormal sound similarity detection model to determine the similarity information of the audio signal, wherein the abnormal sound similarity detection model includes a first detection sub-model that uses graphic comparison to perform similarity detection and / or a second detection sub-model that uses RGB comparison to perform similarity detection; An abnormal sound detection module is used to detect abnormal sounds of the vehicle based on the similarity information.

9. A vehicle, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the vehicle abnormal noise detection method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the vehicle abnormal noise detection method according to any one of claims 1 to 7 is implemented.

Images