Audio recognition method, apparatus, device, and storage medium

Abstract

This application relates to an audio recognition method, apparatus, device, and storage medium, and pertains to the field of vehicle technology. The method includes: in response to detecting an abnormal noise audio, determining an abnormal noise region based on the abnormal noise audio; if a fault code is detected, determining the faulty component corresponding to the fault code based on the fault code; if the region where the faulty component is located coincides with the abnormal noise region, determining that the abnormal noise audio is emitted by the faulty component. This is used to improve the accuracy of vehicle operation audio recognition.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, specifically to an audio recognition method, apparatus, device, and storage medium. Background Technology

[0002] With social development, cars have become a common means of transportation for people's daily travel, and the number of families owning cars is also quite considerable. The demand for rapid vehicle fault diagnosis is also constantly increasing.

[0003] Currently, abnormal vehicle noises are considered early signs of malfunctions, and if not detected and addressed promptly, they can significantly impact driving safety. Therefore, identifying vehicle malfunctions through sound is a common method, particularly when an unusual noise is detected, indicating a malfunction. For example, a vehicle noise recording device has been disclosed that stores sounds from different areas of the vehicle, allowing personnel to compare these sounds with the sounds associated with specific malfunctions to determine the cause. However, relying solely on unusual noises to diagnose vehicle malfunctions can lead to inaccurate fault identification. Summary of the Invention

[0004] The purpose of this application is to provide an audio recognition method, apparatus, device, and storage medium for improving the accuracy of vehicle operation audio recognition.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0006] According to a first aspect of this application, an audio recognition method is provided, applied to a vehicle. The method includes: in response to detecting an abnormal noise audio, determining an abnormal noise region based on the abnormal noise audio; if a fault code is detected, determining a faulty component corresponding to the fault code based on the fault code; and if the region where the faulty component is located is the same as the abnormal noise region, determining that the abnormal noise audio is emitted by the faulty component.

[0007] In one possible approach, the method further includes: determining that the components in the area of ​​the abnormal noise are functioning normally if no fault code is detected, and displaying the area of ​​the abnormal noise and an abnormal noise warning message on the vehicle's display screen. The abnormal noise warning message is used to indicate that an abnormal noise has occurred in the area and that the components in the area are functioning normally.

[0008] In one possible approach, the method further includes: acquiring operating audio via the vehicle's microphone and determining the operating audio features; calculating the similarity between the operating audio features and the abnormal noise audio features; and determining the operating audio as an abnormal noise audio if the similarity between the operating audio features and the abnormal noise audio features is greater than or equal to a similarity threshold.

[0009] In one possible approach, calculating the similarity between the operating audio feature and the abnormal noise audio feature includes: identifying the target component emitting the operating audio and acquiring the corresponding abnormal noise audio feature of the target component. The abnormal noise audio feature is collected when the target component is operating abnormally. The similarity between the operating audio feature and the corresponding abnormal noise audio feature of the target component is calculated. Determining that the operating audio is an abnormal noise audio includes: determining that the operating audio is an abnormal noise audio emitted by the target component.

[0010] In one possible approach, determining the target component that emits the operating audio includes: identifying the target component based on the frequency of the operating audio and a component correspondence. The component correspondence includes multiple components and the frequency corresponding to each component.

[0011] In one possible approach, the vehicle includes N microphones to determine the abnormal noise area based on the abnormal noise audio, including: determining the abnormal noise area based on the abnormal noise audio collected by M microphones. N ≥ M > 2.

[0012] In one possible approach, after determining that the abnormal noise is emitted by a faulty component, the method further includes: marking the location of the faulty component in the vehicle's 3D model on the vehicle's display screen and outputting a fault indication message. The fault indication message is used to indicate that the faulty component is malfunctioning.

[0013] According to a second aspect of this application, an audio recognition device is provided, applied to a vehicle, the device comprising: a determining unit. The determining unit is configured to, in response to detecting an abnormal noise audio, determine an abnormal noise region based on the abnormal noise audio. The determining unit is configured to, in the case of detecting a fault code, determine a faulty component corresponding to the fault code based on the fault code. The determining unit is configured to, if the region where the faulty component is located is the same as the abnormal noise region, determine that the abnormal noise audio is emitted by the faulty component.

[0014] In one possible embodiment, the audio recognition device further includes a display unit. The determining unit is also configured to determine that the components in the area of ​​the abnormal noise are functioning normally, provided no fault code is detected. The display unit is configured to display the area of ​​the abnormal noise and an abnormal noise warning message on the vehicle's display screen. The abnormal noise warning message is used to indicate that an abnormal noise has occurred in the area and that the components in the area are functioning normally.

[0015] In one possible embodiment, the audio recognition device further includes a data acquisition unit and a processing unit. The data acquisition unit is used to acquire operating audio through the vehicle's microphone. The determination unit is further used to determine the operating audio features. The processing unit is used to calculate the similarity between the operating audio features and the abnormal noise audio features. The determination unit is further used to determine that the operating audio is an abnormal noise audio if the similarity between the operating audio features and the abnormal noise audio features is greater than or equal to a similarity threshold.

[0016] In one possible approach, the processing unit is specifically configured to: determine the target component emitting the operating audio, and acquire the abnormal noise audio features corresponding to the target component. The abnormal noise audio features are collected when the target component is operating abnormally. The similarity between the operating audio features and the abnormal noise audio features corresponding to the target component is calculated. The determining unit is specifically configured to: determine that the operating audio is an abnormal noise audio emitted by the target component.

[0017] In one possible approach, the unit is determined by: identifying a target component based on the frequency of the running audio and the component correspondence. The component correspondence includes multiple components and the frequency corresponding to each component.

[0018] In one possible approach, the vehicle includes N microphones, and a determining unit is specifically used to: determine the abnormal noise area based on the abnormal noise audio collected by M microphones. N ≥ M > 2.

[0019] In one possible embodiment, the audio recognition device further includes an output unit. The display unit is also used to mark the location of the faulty component in the vehicle's 3D model on the vehicle's display screen. The output unit is used to output fault indication information. The fault indication information is used to indicate that the faulty component is malfunctioning.

[0020] According to a third aspect provided in this application, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the method of the first aspect described above and any possible implementation thereof.

[0021] According to a fourth aspect provided in this application, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of a processing device, enables the processing device to perform the methods described in the first aspect and any possible implementation thereof.

[0022] According to the fifth aspect provided in this application, a computer program product is provided, the computer program product including computer instructions that, when executed on a processing device, cause the processing device to perform the method described in the first aspect and any possible implementation thereof.

[0023] Therefore, the above-mentioned technical features of this application have the following beneficial effects:

[0024] In the case of shared computing power among vehicles, computational tasks are executed based on relevant data, execution results are obtained, and the shared result data, including the execution results, is inspected. Furthermore, if the shared result data passes inspection, the result data is sent. In this way, by inspecting the shared result data, abnormal data sent by the vehicles is prevented, thereby preventing the leakage of information such as vehicle system configuration files and ensuring the security of the vehicle system's configuration files. Attached Figure Description

[0025] Figure 1 A schematic diagram of the structure of an audio recognition system is shown in an exemplary embodiment provided in this application;

[0026] Figure 2 A schematic diagram illustrating a microphone layout as provided in an exemplary embodiment of this application;

[0027] Figure 3 One of the schematic flowcharts of an audio recognition method is shown in an exemplary embodiment provided in this application;

[0028] Figure 4 A schematic diagram illustrating a feature extraction process is provided for an exemplary embodiment of this application;

[0029] Figure 5 A schematic diagram illustrating the training process of a feature recognition model provided in this application is shown in an exemplary embodiment.

[0030] Figure 6 A schematic diagram illustrating the positioning of a microphone, provided for an exemplary embodiment of this application;

[0031] Figure 7 One of the schematic flowcharts of an audio recognition method is shown in an exemplary embodiment provided in this application;

[0032] Figure 8 A schematic diagram of an audio recognition device is shown as an exemplary embodiment provided in this application;

[0033] Figure 9 A schematic diagram of an electronic device is shown as an exemplary embodiment provided in this application. Detailed Implementation

[0034] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0035] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0036] With social development, cars have become a common means of transportation for people's daily travel, and the number of families owning cars is also quite considerable. The demand for rapid vehicle fault diagnosis is also constantly increasing.

[0037] Currently, abnormal vehicle noises are considered early signs of malfunctions, and if not detected and addressed promptly, they can significantly impact driving safety. Therefore, identifying vehicle malfunctions through sound is a common method, particularly when an unusual noise is detected, indicating a malfunction. For example, a vehicle noise recording device has been disclosed that stores sounds from different areas of the vehicle, allowing personnel to compare these sounds with the sounds associated with specific malfunctions to determine the cause. However, relying solely on unusual noises to diagnose vehicle malfunctions can lead to inaccurate fault identification.

[0038] To address the aforementioned technical problems, this application provides an audio recognition method applied to a vehicle. The method includes: in response to detecting an abnormal noise audio, determining the abnormal noise region based on the abnormal noise audio; if a fault code is detected, determining the faulty component corresponding to the fault code based on the fault code; and if the region where the faulty component is located coincides with the abnormal noise region, determining that the abnormal noise audio is emitted by the faulty component.

[0039] Based on the aforementioned technical methods, when an abnormal noise is detected, the corresponding abnormal noise region is determined. Further, it checks whether the vehicle has generated a diagnostic trouble code (DTC), and if a DTC is detected, the faulty component corresponding to the DTC is identified. Subsequently, if the region where the faulty component is located coincides with the abnormal noise region, the abnormal noise is determined to be emitted by the faulty component. In this way, by determining whether the abnormal noise is emitted by the faulty component based on the region where the faulty component is located and the abnormal noise region, the accuracy of fault detection is improved.

[0040] like Figure 1 As shown, Figure 1An audio recognition system 100 is provided as an embodiment of this application. The audio recognition system 100 may include a central processing unit 101, a sound processor 102, an image processor 103, a memory 104, a display 105, and multiple microphones. Figure 1 An exemplary diagram shows six microphones: microphone 1, microphone 2, microphone 3, microphone 4, microphone 5, and microphone 6 (in practical applications, there may be more or fewer microphones). The central processing unit 101 is connected to the sound processor 102, the image processor 103, the memory 104, and the display 105. The sound processor 102 is connected to multiple microphones.

[0041] The central processing unit 101 is configured to determine the area of ​​abnormal noise based on the detected abnormal noise audio. The central processing unit 101 is also configured to, in the event of a detected fault code, determine the faulty component corresponding to the fault code, and, if the area where the faulty component is located is the same as the area of ​​the abnormal noise, determine that the abnormal noise audio is emitted by the faulty component.

[0042] The central processing unit 101 is also used to generate an abnormal prompt message when it is determined that a faulty component is emitting an abnormal audio signal.

[0043] The sound processor 102 is used to process audio data captured by multiple microphones.

[0044] Multiple microphones are used to collect audio data about the vehicle's operation.

[0045] In some embodiments, the number of microphones is 13. Maintenance personnel place these 13 microphones in different locations to obtain audio data from different components.

[0046] For example, 13 microphones are installed in different parts of the vehicle (such as the body, brake disc, transmission, suspension, tires, motor, engine, steering wheel, seats, windows, and air conditioning) to form a multi-dimensional audio data acquisition space in the chassis and cabin, and to collect audio signals when the vehicle is running.

[0047] In some embodiments, two types of audio data are collected: 1. Audio data of the vehicle's body, brake discs, transmission, suspension, tires, motor, engine, steering wheel, seats, windows, and air conditioning operating normally under various scenarios such as acceleration, deceleration, turning, and braking; 2. Audio data of abnormal noises from the vehicle's body, brake discs, transmission, suspension, tires, motor, engine, steering wheel, seats, windows, and air conditioning operating abnormally under various scenarios such as acceleration, deceleration, turning, and braking.

[0048] In some embodiments, such as Figure 2 The diagram shows a schematic of a microphone layout. Figure 2 The image shows nine microphones: microphone 1, microphone 2, microphone 3, microphone 4, microphone 5, microphone 6, microphone 7, microphone 8, and microphone 9, along with noise and the positional relationship between the nine microphones and the noise.

[0049] Display 105 is used to display the area of ​​abnormal noise and abnormal prompts.

[0050] It should be noted that the display 105 includes a screen. The display 105 can be a head-up display, a central control display, or a passenger-side display; however, this application embodiment does not limit the specific type of display.

[0051] Memory 104 is used to store fault codes.

[0052] Image processor 103 is used to acquire scanning data of the vehicle by a three-dimensional laser scanning vibration meter and to obtain a three-dimensional model of the vehicle based on the scanning data.

[0053] The image processor 103 is also used to mark the target area in the 3D model and send the marked 3D model to the display 105 so that the display 105 can display the marked 3D model.

[0054] In some embodiments, when the vehicle is running, the audio recognition system 100 selects fault codes (DTCs) generated by the vehicle body system, brake disc system, transmission system, suspension system, tire (tire pressure) system, motor drive system, engine system, steering wheel (steering) system, seat system, window (control system) and air conditioning (thermal management) system, and filters out the fault types from all generated DTCs as mechanical faults, electrical faults, system-level communication bus faults and circuit faults, as potential influencing factors of abnormal noise.

[0055] Subsequently, when the vehicle produces an abnormal noise, the microphone transmits the audio data to the vehicle's sound processor 102. The sound processor 102 then transmits the audio data to the central processing unit 101 for comparative analysis and processing, and compares the analysis results with all DTCs (Disturbance Troubleshooting Codes) for the entire vehicle. If the central processing unit 101 identifies the abnormal noise audio and locks onto the abnormal noise area, but no components within the abnormal noise area generate DTCs, it is determined that only the abnormal noise exists in that area. If the central processing unit 101 identifies the abnormal noise audio and locks onto the abnormal noise area, and components within the abnormal noise area simultaneously generate DTCs, it is determined that the components within the abnormal noise area produce an abnormal noise. For example, if the air conditioner produces an abnormal noise of 200 Hz (approximately 45 dB), when the sound processor 102 identifies the abnormal noise area, the air intake grille of the air conditioning (thermal management) system generates a mechanical fault code (DTC), and the central processing unit determines that the air conditioner within the locked area is producing the abnormal noise.

[0056] For ease of understanding, the audio recognition method provided in this application will be described in detail below with reference to the accompanying drawings. In this embodiment, the audio recognition method uses an electronic device in a vehicle as an example.

[0057] Figure 3 A flowchart illustrating an audio recognition method according to an exemplary embodiment is shown. The audio recognition method is applied to a vehicle, and the method includes: S201-S203.

[0058] S201. In response to the detection of abnormal audio, determine the abnormal noise area based on the abnormal audio.

[0059] As one possible approach, the electronic device periodically or in real-time detects the audio captured by the microphone, and upon detecting an abnormal audio sound, determines the area of ​​the abnormal sound based on the abnormal audio sound.

[0060] In some embodiments, when an electronic device acquires a running audio, it detects the running audio to determine whether the running audio is an abnormal sound.

[0061] For example, an electronic device acquires the audio features of running audio and inputs these features into an audio detection model to obtain the detection results of the audio features, thus obtaining the detection result of the running audio. The detection result is used to indicate whether the audio features are abnormal audio features. As another example, the electronic device acquires the audio features of audio and calculates the similarity between the audio features and abnormal audio features. If the similarity between the audio features and abnormal audio features is greater than or equal to a similarity threshold, the running audio is determined to be abnormal audio.

[0062] In some embodiments, the audio characteristics are Mel frequency cepstral coefficients (MFCC) parameters.

[0063] In some embodiments, the electronic device acquires the operating audio of multiple components through microphones at different locations, and preprocesses the operating audio of the multiple components to obtain the operating audio of each component. Furthermore, the electronic device detects whether the operating audio of each component is an abnormal noise.

[0064] It should be noted that the audio detection model is a model pre-trained by the maintenance personnel. The audio detection model can be a neural network model. The similarity threshold is pre-configured in the electronic device by the maintenance personnel; this embodiment does not specifically limit the similarity threshold.

[0065] As one possible approach, an electronic device, in response to detecting an abnormal audio sound, locates the abnormal audio sound and obtains the area where the source of the abnormal audio sound is located.

[0066] In some embodiments, the electronic device identifies the target component corresponding to the abnormal noise and locates the target component to obtain its position. Further, the area where the target component is located is defined as the abnormal noise area.

[0067] In some embodiments, the electronic device determines the target component corresponding to the target abnormal noise audio, and determines the abnormal noise region based on the target component and the region correspondence. The region correspondence includes multiple components and the region corresponding to each component.

[0068] It is understood that the abnormal noise area is the area that emits abnormal noise audio, and the embodiments of this application do not specifically limit the size of the abnormal noise area.

[0069] S202. If a fault code is detected, determine the faulty component corresponding to the fault code based on the fault code.

[0070] In some embodiments, when an electronic device detects an abnormal audio noise, it determines whether a fault code has been detected, and if a fault code is detected, it identifies the faulty component corresponding to the fault code.

[0071] S203. If the area where the faulty component is located is the same as the area of ​​the abnormal noise, determine that the abnormal noise audio is emitted by the faulty component.

[0072] In some embodiments, when a faulty component is identified, the electronic device determines whether the area where the faulty component is located is the same as the area of ​​abnormal noise. If the area where the faulty component is located is the same as the area of ​​abnormal noise, the abnormal noise audio is determined to be the audio emitted by the faulty component.

[0073] Subsequently, the electronic device marks the faulty component in the 3D model and displays the 3D model of the marked faulty component on the display.

[0074] It should be noted that the area where the faulty component is located is the same as the area of ​​abnormal noise. This can be understood as: the area where the faulty component is located is the same as the area of ​​abnormal noise, the degree of overlap between the area where the faulty component is located and the area of ​​abnormal noise is greater than or equal to a preset threshold, or the area where the faulty component is located is located in the area of ​​abnormal noise. This application embodiment does not limit this.

[0075] The audio recognition method provided in this application provides at least the following advantages: When an abnormal noise is detected, the corresponding abnormal noise region is determined. Further, it detects whether the vehicle has generated a fault code, and if a fault code is detected, the faulty component corresponding to the DTC is identified. Subsequently, if the region where the faulty component is located is the same as the abnormal noise region, the abnormal noise is determined to be emitted by the faulty component. Thus, by determining whether the abnormal noise is emitted by the faulty component based on the region where the faulty component is located and the abnormal noise region, the accuracy of fault detection is improved.

[0076] In one possible design, in order to improve the accuracy of fault detection, the audio recognition method provided in this application embodiment further includes: S204-S205.

[0077] S204. If no fault code is detected, determine that the components in the area of ​​abnormal noise are operating normally.

[0078] As one possible approach, if an electronic device detects an abnormal audio noise but does not detect a fault code, it can determine that the component in the area of ​​the abnormal noise is functioning normally.

[0079] In some embodiments, when an electronic device detects an abnormal audio noise, it determines the duration of the abnormal noise, detects whether a fault code is generated during the duration of the abnormal noise, and determines that the components in the abnormal noise area are operating normally if no fault code is generated.

[0080] S205. Display the area of ​​abnormal noise and abnormal noise warning information on the vehicle's display screen.

[0081] The abnormal noise warning information is used to indicate that an abnormal noise has occurred in the abnormal noise area and that the components in the abnormal noise area are operating normally.

[0082] In some embodiments, when an abnormal noise is detected but no fault code is detected, the electronic device marks the area of ​​the abnormal noise in a 3D model of the vehicle and displays the marked 3D model on a display screen. Additionally, the electronic device generates and outputs an abnormal noise alert message. For example, the electronic device outputs the alert message via an audio jack or displays it on a display screen.

[0083] In another scenario, if the electronic device detects an unusual audio noise but no fault code, it outputs a diagnostic message. This message serves as a notification to the user regarding the unusual noise. For example, the electronic device might output the message "Unusual noise is occurring in the vehicle; please have it checked promptly" via the audio system.

[0084] Understandably, if an abnormal noise is detected but no fault code is found, it indicates that the vehicle's components are functioning normally. To avoid user concerns about vehicle safety, the system informs the user that the vehicle's components are normal and the vehicle can be driven normally, thereby improving the user experience.

[0085] In one possible design, in order to determine the abnormal audio, the audio recognition method provided in this application embodiment further includes: S206-S209.

[0086] S206. Acquire operating audio through the vehicle's microphone.

[0087] In some embodiments, the electronic device captures the operating audio of different components through microphones located at different positions.

[0088] For example, consider a vehicle equipped with 13 microphones, each microphone capturing the operating audio of a specific part of the vehicle. The electronic equipment uses each microphone to capture the operating audio of the corresponding part.

[0089] In other embodiments, the electronic device acquires the operating audio of multiple components via a microphone and determines the operating audio of each component based on the operating characteristics of each component.

[0090] For example, when an electronic device acquires the operating audio of six components, it determines the operating audio of a target component from the six operating audios based on the frequency of the operating audio of each component.

[0091] S207. Determine the running audio characteristics.

[0092] As one possible approach, when an electronic device acquires running audio, it preprocesses the running audio to obtain processed running audio, and then obtains the running audio characteristics based on a preset algorithm.

[0093] In some embodiments, the electronic device removes background noise and other irrelevant sounds from the operating audio to obtain the valid audio of the target component. For example, taking a gearbox as the target component, the electronic device removes background noise and other irrelevant sound signals from the gearbox's operating audio to obtain the actual operating audio of the gearbox.

[0094] In some embodiments, electronic devices improve the audio characteristics of the running audio through MFCC feature extraction.

[0095] Specifically, the electronic device preprocesses the running audio and then extracts features from the preprocessed audio. For example, the feature extraction process is as follows: Figure 4 As shown, it includes: S1-S5.

[0096] Since the human ear's perception of pitch is non-linear, the Mel scale is used to linearly characterize changes in audio data, improving the accuracy of audio features. Because the Mel scale is essentially a function of frequency, Hertz (Hz) is mapped to Mel: the formula for calculating the Mel scale and frequency is shown in Formula 1 below.

[0097] F mel (f) = 2595 × log 10 Formula 1 (1+f / 700)

[0098] Among them, F mel (f) is the perceptual frequency domain in megahertz, and f is the actual speech frequency in Hz.

[0099] S1, pre-emphasis, framing, and windowing processing.

[0100] When electronic devices receive pre-processed audio, they pre-emphasize, frame, and window the pre-processed audio to increase some high-frequency energy and reduce the impact of high-frequency loss in signal transmission.

[0101] The pre-emphasis of the running audio can be a first-order difference of the running audio, which is expressed by the following formula 2.

[0102] Formula 2: x[n]=x[n]-αx[n-1]

[0103] Where 0.9≤α≤1.0, and n is the audio of the nth frame.

[0104] Understandably, the range of α is limited to 0.9≤α≤1.0. This is to ensure that the processed signal is not over-enhanced or distorted, while maintaining the naturalness of the signal.

[0105] S2, Fast Fourier Transform.

[0106] In some embodiments, the electronic device performs a fast Fourier transform (FFT) on each frame of audio after framing and windowing to transform the audio signal to the frequency domain.

[0107] S3. Calculate the spectral line energy.

[0108] In some embodiments, the electronic device squares or takes the absolute value of each frame of signal after FFT transformation.

[0109] S4, Mel filter bank filtering.

[0110] In some embodiments, the electronic device multiplies the calculated energy spectrum of each frame's spectral line with a Mel filter bank to obtain the Mel spectrum.

[0111] S5, Discrete Cosine Transform.

[0112] In some embodiments, the electronic device performs a discrete cosine transform (DCT) on each frame of signal after filtering by a Mel filter, and takes the 2nd to 13th coefficients after the DCT as MFCC coefficients to obtain MFCC parameters. Further, the MFCC parameters are determined as feature values ​​for that frame of audio. In this way, the electronic device obtains the feature values ​​of each frame of the running audio, thereby obtaining the feature values ​​of the running audio.

[0113] In other embodiments, the electronic device will run an audio input audio feature extraction model to obtain running audio features.

[0114] It should be noted that the audio feature extraction model is a pre-trained model by operations and maintenance personnel, used to extract audio features from audio.

[0115] S208. Calculate the similarity between the running audio features and the abnormal noise audio features.

[0116] In some embodiments, when an electronic device acquires operational audio features, it calculates the similarity between the operational audio features and abnormal audio features in an audio feature database.

[0117] In some embodiments, the electronic device acquires the abnormal noise audio features corresponding to the running audio features and calculates the similarity between the running audio features and the abnormal noise audio features.

[0118] For example, the operating audio characteristics are the operating audio characteristics of the transmission. The electronic equipment acquires the abnormal noise audio characteristics of the transmission and calculates both the operating audio characteristics and the abnormal noise audio characteristics of the transmission.

[0119] In some embodiments, the electronic device acquires the operating conditions corresponding to the operating audio features, and determines the abnormal noise audio features corresponding to the operating conditions based on the operating conditions. Further, the electronic device calculates the similarity between the operating audio features and the abnormal noise audio features.

[0120] For example, the operating audio is obtained under acceleration conditions. The electronic equipment identifies the abnormal noise audio characteristics under acceleration conditions and calculates the similarity between the operating audio characteristics and the abnormal noise audio characteristics. Similarly, if the operating audio is obtained under turning conditions, the electronic equipment identifies the abnormal noise audio characteristics under turning conditions and calculates the similarity between the operating audio characteristics and the abnormal noise audio characteristics. As another example, if the operating audio is emitted by the brake disc under deceleration conditions, the electronic equipment acquires the abnormal noise audio characteristics of the brake disc under deceleration conditions and calculates the similarity between the operating audio characteristics and the abnormal noise audio characteristics.

[0121] In some embodiments, an abnormal noise audio feature dataset is determined before calculating the similarity between running audio features and abnormal noise audio features.

[0122] Specifically, abnormal noise sample audio is extracted using a trained feature recognition model. Before using the feature recognition model to extract abnormal noise sample audio, an initial feature recognition model is trained to obtain a trained feature recognition model. The training process for the initial feature recognition model is as follows: Figure 5 As shown, it includes: S21-S25.

[0123] S21, Data Preprocessing.

[0124] S22, Parameters for training the feature recognition model.

[0125] In some embodiments, the feature recognition model uses a backpropagation (BP) neural network, which includes forward propagation (to calculate the loss) and backpropagation (to backpropagate the error).

[0126] Forward propagation involves: given the initial weight values ​​W and bias term b of the initial feature recognition model, calculating the final output value and the loss value between the output value and the actual value. If the loss value is outside the preset range, backward propagation is performed. If the loss value is within the preset range, the updating of W and b is stopped.

[0127] For example, the calculation formula is as shown in Formula 3 below.

[0128] Formula 3: y = T(WX + b)

[0129] Where T represents the activation function, b represents the activation threshold, W represents the connection weight, X is the input value, and y is the output value.

[0130] Backpropagation involves propagating the output back through the hidden layers to the input layers layer by layer, distributing the error among all units in each layer to obtain the error signal for each unit. This error signal serves as the basis for adjusting the weights of each unit. The error function of the BP neural network is the mean square error function.

[0131]

[0132] Where m is the number of training samples and k is the number of output samples. Let y be the predicted output value for the i-th sample and the j-th output value. ij This corresponds to the actual value.

[0133] In some embodiments, audio feature data trained by a BP neural network is used for classification training, enabling it to identify normal audio features and abnormal audio features of different types of parts.

[0134] S23. Determine whether the training conditions are met.

[0135] If yes, proceed to S24. If no, proceed to S25.

[0136] S24. Obtain the trained feature recognition model.

[0137] In some embodiments, the operating audio of the target component is input into a trained feature recognition model to obtain a result indicating whether the operating audio of the target component is an abnormal noise.

[0138] S25. Modify the parameters of the feature recognition model.

[0139] In some embodiments, S22 is executed after modifying the parameters of the feature recognition model.

[0140] S209. If the similarity between the running audio features and the abnormal noise audio features is greater than or equal to the similarity threshold, the running audio is determined to be abnormal noise audio.

[0141] In some embodiments, when the electronic device obtains the similarity between the operating audio feature and the abnormal noise audio feature, it determines that the similarity between the operating audio feature and the abnormal noise audio feature is greater than or equal to a similarity threshold, and if the similarity between the operating audio feature and the abnormal noise audio feature is greater than or equal to the similarity threshold, it determines that the operating audio is an abnormal noise audio. Otherwise, it determines that the operating audio is an abnormal noise audio.

[0142] In one design, in order to provide accuracy in audio recognition, the above S208 includes: S2081-S2083.

[0143] S2081. Determine the target component that emits the operating audio.

[0144] S2082. Obtain the abnormal noise audio characteristics corresponding to the target component.

[0145] Among them, abnormal noise audio features are collected when the target component is operating abnormally.

[0146] S2083. Calculate the similarity between the running audio features and the abnormal noise audio features corresponding to the target component.

[0147] The above S209 includes: S2091.

[0148] S2091. Determine that the operating audio is an abnormal noise emitted by the target component.

[0149] In one design, the above S2081 includes: S301.

[0150] S301. Determine the target component based on the frequency of the running audio and the corresponding relationship between components.

[0151] The component correspondence includes multiple components and the frequency corresponding to each component.

[0152] In one design, the vehicle includes N microphones, and the above S201 includes: S2011.

[0153] S2011. Determine the abnormal noise area based on the abnormal noise audio collected by M microphones.

[0154] Where N≥M>2.

[0155] For example, such as Figure 6 As shown, noise points are located based on microphones 1, 2, and 3.

[0156] In some embodiments, to enable users to understand the vehicle's operating status in a timely manner, the audio recognition method provided in this application embodiment further includes: S210-S211.

[0157] S210. Mark the location of the faulty component in the vehicle's 3D model on the vehicle's display screen.

[0158] S211, Output fault message.

[0159] Among them, the fault indication information is used to indicate that the faulty component is malfunctioning.

[0160] In some embodiments, the electronic device stores abnormal audio.

[0161] Understandably, if only the area of ​​abnormal noise is identified, then the abnormal noise is determined to originate from that area. The location of the abnormal noise area is then displayed on the vehicle's monitor, showing the location of the abnormal noise in a 3D model, and the audio data of the abnormal noise is stored. If both the abnormal noise area and the component's fault code are identified simultaneously, then the component is determined to be the source of the abnormal noise. The abnormal noise area and component identifiers are then displayed on the vehicle's monitor, showing the specific location of the abnormal noise area and component, the cause of the abnormal noise corresponding to the fault code, and the audio data of the abnormal noise is stored.

[0162] In some embodiments, in order to determine audio characteristics

[0163] To better understand the audio recognition method provided in the embodiments of this application, such as Figure 7 As shown, an audio recognition process is illustrated, including: S401-S410.

[0164] S401, Acquire audio data.

[0165] S402, Extract audio features.

[0166] S403. Establish an audio feature set based on audio features.

[0167] In some embodiments, the electronic device marks the normal and abnormal audio characteristics of each component under different operating conditions to obtain an audio feature set.

[0168] For example, the collected audio data feature value set is labeled according to type and status, and the labeling results are mapped to the whole vehicle "3D modeling data" including "vehicle body, brake disc, gearbox, suspension, tire, motor, engine, steering wheel, seat, windows and air conditioning".

[0169] S404. Create a 3D model of the vehicle.

[0170] S405, Acquire running audio.

[0171] S406. In response to the input running audio, determine the running audio characteristics of the running audio.

[0172] S407. Determine whether the running audio characteristics are abnormal audio characteristics.

[0173] If yes, then execute S408. If no, then execute S405.

[0174] S408. Identify the abnormal noise area in the running audio.

[0175] S409, Obtain fault codes.

[0176] S410. If the area of ​​abnormal noise is the same as the area where the faulty component is located, display a three-dimensional model of the area marked with abnormal noise.

[0177] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the communication device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] Figure 8 This is an audio recognition device illustrated according to an exemplary embodiment, applied to a vehicle. (See reference...) Figure 8 The audio recognition device 50 includes: a determination unit 501.

[0179] The determining unit 501 is configured to determine the abnormal noise region based on the detected abnormal noise audio. The determining unit 501 is also configured to determine the faulty component corresponding to the fault code based on the detected fault code. Furthermore, the determining unit 501 is configured to determine that the abnormal noise audio is emitted by the faulty component if the area where the faulty component is located coincides with the abnormal noise region.

[0180] In one possible way, such as Figure 8 As shown, the audio recognition device 50 also includes a display unit 502. The determination unit 501 is further configured to determine that the components in the abnormal noise area are operating normally if no fault code is detected. The display unit 502 is configured to display the abnormal noise area and abnormal noise warning information on the vehicle's display screen. The abnormal noise warning information is used to indicate that an abnormal noise has occurred in the abnormal noise area and that the components in the abnormal noise area are operating normally.

[0181] In one possible way, such as Figure 8 As shown, the audio recognition device 50 also includes a acquisition unit 503 and a processing unit 504. The acquisition unit 503 is used to acquire operating audio through the vehicle's microphone. The determination unit 501 is further used to determine the operating audio features. The processing unit 504 is used to calculate the similarity between the operating audio features and the abnormal noise audio features. The determination unit 501 is further used to determine that the operating audio is an abnormal noise audio if the similarity between the operating audio features and the abnormal noise audio features is greater than or equal to a similarity threshold.

[0182] In one possible approach, processing unit 504 is specifically configured to: determine the target component emitting the operating audio, and acquire the abnormal noise audio features corresponding to the target component. The abnormal noise audio features are collected when the target component is operating abnormally. The similarity between the operating audio features and the abnormal noise audio features corresponding to the target component is calculated. Determination unit 501 is specifically configured to: determine that the operating audio is an abnormal noise audio emitted by the target component.

[0183] In one possible approach, the determining unit 501 is specifically used to: determine the target component based on the frequency of the running audio and the component correspondence. The component correspondence includes multiple components and the frequency corresponding to each component.

[0184] In one possible configuration, the vehicle includes N microphones, and the determining unit 501 is specifically used to: determine the abnormal noise area based on the abnormal noise audio collected by M microphones. N ≥ M > 2.

[0185] In one possible way, such as Figure 8 As shown, the audio recognition device 50 also includes an output unit 505. The display unit 502 is further used to mark the location of the faulty component in the vehicle's three-dimensional model on the vehicle's display screen. The output unit 505 is used to output fault indication information. The fault indication information is used to indicate abnormal operation of the faulty component.

[0186] Figure 9 This is a schematic diagram illustrating an electronic device according to an exemplary embodiment. Figure 9 As shown, the electronic device includes, but is not limited to, a processor 601 and a memory 602.

[0187] The memory 602 described above is used to store the executable instructions of the processor 601. It is understood that the processor 601 is configured to execute instructions to implement the model training method and SOC estimation method in the above embodiments.

[0188] It should be noted that those skilled in the art will understand that Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device; the electronic device may include, but is not limited to, other electronic devices. Figure 9This may indicate more or fewer components, or combinations of certain components, or different component arrangements.

[0189] Processor 601 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in memory 602, and by calling data stored in memory 602, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Processor 601 may include one or more processing units. Optionally, processor 601 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 601.

[0190] The memory 602 can be used to store software programs and various data. The memory 602 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required by at least one functional module (such as a determination unit, processing unit, etc.), etc. Furthermore, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0191] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, which can be executed by a processor 601 of an electronic device to implement the methods in the above embodiments.

[0192] Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0193] In an exemplary embodiment, this application also provides a computer program product including one or more instructions, which can be executed by a processor 601 of an electronic device to perform the methods described above.

[0194] It should be noted that when one or more instructions in the computer-readable storage medium or computer program product are executed by the processor of an electronic device, they implement the various processes of the above method embodiments and achieve the same technical effect as the above method. To avoid repetition, they will not be described again here.

[0195] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0197] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the classified units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0198] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0199] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, essentially, or the part that contributes to the prior art, or a complete or partial classification of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0200] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio recognition method, characterized in that, Applied to vehicles, the method includes: The vehicle's operating audio is collected using its microphones, and the characteristics of the operating audio are determined. There are multiple microphones located at different positions within the vehicle. The operating conditions corresponding to the operating audio features are obtained, and the abnormal noise audio features corresponding to the operating conditions are determined based on the operating conditions; the operating conditions are: acceleration condition, turning condition, or deceleration condition. Calculate the similarity between the running audio feature and the abnormal noise audio feature, and determine that the running audio is an abnormal noise audio if the similarity between the running audio feature and the abnormal noise audio feature is greater than or equal to a similarity threshold; In response to the detection of an abnormal audio sound, the abnormal noise area is determined based on the abnormal audio sound; If a fault code is detected, the faulty component corresponding to the fault code is determined based on the fault code; If the area where the faulty component is located is the same as the area of ​​the abnormal noise, then the abnormal noise audio is determined to be the audio emitted by the faulty component. A three-dimensional model of the faulty component is displayed on the vehicle's screen. If no fault code is detected, it is determined that the components in the abnormal noise area are operating normally, and the abnormal noise area, abnormal noise warning information, and a 3D model of the abnormal noise area are displayed on the vehicle's display screen; the abnormal noise warning information is used to indicate that an abnormal noise has occurred in the abnormal noise area and that the components in the abnormal noise area are operating normally.

2. The method according to claim 1, characterized in that, The calculation of the similarity between the running audio features and the abnormal noise audio features includes: Identify the target component emitting the operating audio and acquire the abnormal noise audio characteristics corresponding to the target component; the abnormal noise audio characteristics are collected when the target component is operating abnormally. Calculate the similarity between the running audio features and the abnormal noise audio features corresponding to the target component; The determination that the running audio is an abnormal sound includes: The operating audio is determined to be an abnormal noise emitted by the target component.

3. The method according to claim 2, characterized in that, The process of determining the target component that emits the operating audio includes: The target component is determined based on the frequency of the running audio and the component correspondence; the component correspondence includes multiple components and the frequency corresponding to each component.

4. The method according to any one of claims 1-3, characterized in that, The vehicle includes N microphones, and the step of determining the abnormal noise area based on the abnormal noise audio includes: The abnormal noise region is determined based on the abnormal noise audio collected by M microphones; where N ≥ M > 2.

5. The method according to any one of claims 1-3, characterized in that, After determining that the abnormal noise audio is emitted by the faulty component, the method further includes: The display screen of the vehicle marks the location of the faulty component in the three-dimensional model of the vehicle and outputs a fault prompt message; the fault prompt message is used to indicate that the faulty component is malfunctioning.

6. An audio recognition device, characterized in that, Applied to vehicles, the device includes: a determining unit, a display unit, an acquiring unit, a collection unit, and a processing unit; The acquisition unit is used to acquire operating audio through the vehicle's microphones; there are multiple microphones located at different positions within the vehicle. The determining unit is further configured to determine the running audio characteristics of the running audio; The acquisition unit is used to acquire the operating conditions corresponding to the operating audio features; the operating conditions are: acceleration, turning, or deceleration. The determining unit is further configured to determine the abnormal noise audio characteristics corresponding to the operating condition based on the operating condition; The processing unit is used to calculate the similarity between the running audio features and the abnormal noise audio features; The determining unit is further configured to determine that the running audio is abnormal audio when the similarity between the running audio feature and the abnormal audio feature is greater than or equal to a similarity threshold. The determining unit is used to determine the abnormal noise area based on the abnormal noise audio in response to the detection of the abnormal noise audio. The determining unit is used to determine the faulty component corresponding to the fault code based on the fault code when a fault code is detected. The determining unit is configured to determine, when the area where the faulty component is located is the same as the abnormal noise area, that the abnormal noise audio is the audio emitted by the faulty component. The display unit is used to display a three-dimensional model of the faulty component on the vehicle's display screen; The determining unit is also used to determine that the components in the abnormal noise area are operating normally if no fault code is detected; The display unit is also used to display the abnormal noise area, abnormal noise warning information, and a three-dimensional model of the abnormal noise area on the vehicle's display screen; the abnormal noise warning information is used to indicate that an abnormal noise has occurred in the abnormal noise area and that the components in the abnormal noise area are operating normally.

7. An electronic device, characterized in that, Including memory and processor; The memory and the processor are coupled; The memory is used to store computer program code, which includes computer instructions; When the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, When the computer-executable instructions stored in the computer-readable storage medium are executed by the processor of the processing device, the processing device is capable of performing the method as described in any one of claims 1-5.

Images