Audio recognition method and device, equipment and storage medium

Abstract

The invention relates to an audio recognition method and device, equipment and a storage medium, and relates to the technical field of vehicles. The method comprises the following steps: in response to detected abnormal sound audio, determining an abnormal sound area according to the abnormal sound audio; and under the condition that the fault code is detected, determining a fault component corresponding to the fault code according to the fault code. And under the condition that the area where the fault component is located is the same as the abnormal sound area, determining that the abnormal sound audio is the audio sent by the fault component. The accuracy of vehicle operation audio recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to an audio recognition method, device, equipment and storage medium. Background Art

[0002] With the development of society, cars have become a common means of transportation for people's daily travel, and the number of cars owned by families is also very considerable. The demand for rapid fault diagnosis of cars is also increasing.

[0003] At present, it is believed that abnormal sounds of vehicles are precursors to vehicle failures. If they cannot be discovered and resolved in time, they will have a great impact on driving safety. Therefore, it is a common method to identify whether a vehicle is faulty by the sound of the vehicle. In particular, when it is determined that the vehicle has an abnormal sound, it is determined that the vehicle has a fault. For example, a vehicle abnormal sound recording device discloses: the sounds of different areas of the vehicle are stored, and relevant personnel determine the vehicle's fault by comparing them with the sounds of specific faults. However, judging the fault of a vehicle based solely on abnormal sounds will result in inaccurate fault judgment. Summary of the invention

[0004] The purpose of this application is to provide an audio recognition method, device, equipment and storage medium for improving the accuracy of vehicle operation audio recognition.

[0005] In order to achieve the above purpose, the technical solution adopted in this application is as follows:

[0006] According to the first aspect provided by the present application, an audio recognition method is provided, which is applied to a vehicle, and the method includes: in response to detecting an abnormal sound audio, determining an abnormal sound area according to the abnormal sound audio. In the case where a fault code is detected, determining a faulty component corresponding to the fault code according to the fault code. In the case where the area where the faulty component is located is the same as the abnormal sound area, determining that the abnormal sound audio is the audio emitted by the faulty component.

[0007] In one possible manner, the method further includes: when no fault code is detected, determining that the components in the abnormal noise area are operating normally, and displaying the abnormal noise area and abnormal noise prompt information on a display screen of the vehicle. The abnormal noise prompt information is used to prompt that the abnormal noise occurs in the abnormal noise area and that the components in the abnormal noise area are operating normally.

[0008] In one possible manner, the method further includes: collecting operating audio through a microphone of the vehicle, and determining operating audio features of the operating audio, calculating similarity between the operating audio features and abnormal sound audio features, and determining that the operating audio is abnormal sound audio when the similarity between the operating audio features and the abnormal sound audio features is greater than or equal to a similarity threshold.

[0009] In one possible manner, calculating the similarity between the operating audio feature and the abnormal sound audio feature includes: determining a target component that emits the operating audio, and obtaining the abnormal sound audio feature corresponding to the target component. The abnormal sound audio feature is collected when the target component operates abnormally. Calculating the similarity between the operating audio feature and the abnormal sound audio feature corresponding to the target component. Determining that the operating audio is the abnormal sound audio includes: determining that the operating audio is the abnormal sound audio emitted by the target component.

[0010] In a possible manner, determining a target component that emits the operating audio includes: determining the target component according to the frequency of the operating audio and a component correspondence relationship, wherein the component correspondence relationship includes a plurality of components and a frequency corresponding to each component.

[0011] In a possible manner, the vehicle includes N microphones, and determining the abnormal sound area according to the abnormal sound audio includes: determining the abnormal sound area according to the abnormal sound audio collected by M microphones. N≥M>2.

[0012] In one possible manner, after determining that the abnormal sound audio is the audio emitted by the faulty component, the method further includes: marking the position of the faulty component in the three-dimensional model of the vehicle on the display screen of the vehicle, and outputting fault prompt information. The fault prompt information is used to indicate that the faulty component is operating abnormally.

[0013] According to a second aspect provided by the present application, an audio recognition device is provided, which is applied to a vehicle, and the device includes: a determination unit. The determination unit is used to determine an abnormal sound area according to the abnormal sound audio in response to detecting an abnormal sound audio. The determination unit is used to determine the faulty component corresponding to the fault code according to the fault code when a fault code is detected. The determination unit is used to determine that the abnormal sound audio is the audio emitted by the faulty component when the area where the faulty component is located is the same as the abnormal sound area.

[0014] In one possible manner, the audio recognition device further includes a display unit. The determination unit is further used to determine that the components in the abnormal noise area are operating normally when no fault code is detected. The display unit is used to display the abnormal noise area and abnormal noise prompt information on a display screen of the vehicle. The abnormal noise prompt information is used to prompt that an abnormal noise occurs in the abnormal noise area and that the components in the abnormal noise area are operating normally.

[0015] In one possible manner, the audio recognition device further includes a collection unit and a processing unit. The collection unit is used to collect the operating audio through the microphone of the vehicle. The determination unit is also used to determine the operating audio features of the operating audio. The processing unit is used to calculate the similarity between the operating audio features and the abnormal sound audio features. The determination unit is also used to determine that the operating audio is abnormal sound audio when the similarity between the operating audio features and the abnormal sound audio features is greater than or equal to a similarity threshold.

[0016] In one possible manner, the processing unit is specifically used to: determine the target component that emits the operating audio, and obtain the abnormal sound audio feature corresponding to the target component. The abnormal sound audio feature is collected when the target component operates abnormally. Calculate the similarity between the operating audio feature and the abnormal sound audio feature corresponding to the target component. The determination unit is specifically used to: determine that the operating audio is the abnormal sound audio emitted by the target component.

[0017] In a possible manner, the determination unit is specifically configured to determine the target component according to the frequency of the running audio and the component correspondence relationship, wherein the component correspondence relationship includes multiple components and the frequency corresponding to each component.

[0018] In a possible manner, the vehicle includes N microphones, and the determination unit is specifically configured to determine the abnormal sound area according to the abnormal sound audio collected by the M microphones. N≥M>2.

[0019] In one possible manner, the audio recognition device further includes an output unit. The display unit is further used to mark the position of the faulty component in the three-dimensional model of the vehicle on the display screen of the vehicle. The output unit is used to output fault prompt information. The fault prompt information is used to indicate that the faulty component is operating abnormally.

[0020] According to the third aspect provided by the present application, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute instructions to implement the method of the above-mentioned first aspect and any possible implementation manner thereof.

[0021] According to the fourth aspect provided by the present application, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of a processing device, the processing device is enabled to execute the method in the above-mentioned first aspect and any possible implementation method thereof.

[0022] According to the fifth aspect provided by the present application, a computer program product is provided, the computer program product comprising computer instructions, and when the computer instructions are executed on a processing device, the processing device executes the method of the above-mentioned first aspect and any possible implementation manner thereof.

[0023] Therefore, the above technical features of the present application have the following beneficial effects:

[0024] In the case of shared computing power among vehicles, computing tasks are performed according to relevant data to obtain execution results, and the shared result data including the execution results are tested. Furthermore, if the shared result data passes the test, the result data is sent. In this way, abnormal data sent by the vehicle is avoided by testing the shared result data, thereby avoiding leakage of information such as the configuration file of the vehicle system, thereby ensuring the security of the configuration file of the vehicle system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the structure of an audio recognition system shown as an exemplary embodiment provided by the present application;

[0026] Figure 2 A schematic diagram of a microphone layout shown in an exemplary embodiment provided by the present application;

[0027] Figure 3 One of the flowcharts of an audio recognition method shown in an exemplary embodiment provided by the present application;

[0028] Figure 4 A schematic diagram of a feature extraction process shown in an exemplary embodiment provided by the present application;

[0029] Figure 5 A schematic diagram of a training process of a feature recognition model shown in an exemplary embodiment provided by the present application;

[0030] Figure 6 A schematic diagram of positioning a microphone is shown for an exemplary embodiment provided in the present application;

[0031] Figure 7 One of the flowcharts of an audio recognition method shown in an exemplary embodiment provided by the present application;

[0032] Figure 8 A schematic diagram of an audio recognition device shown as an exemplary embodiment provided by the present application;

[0033] Fig. 9 The present invention is a schematic diagram of an electronic device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0036] With the development of society, cars have become a common means of transportation for people's daily travel, and the number of cars owned by families is also very considerable. The demand for rapid fault diagnosis of cars is also increasing.

[0037] At present, it is believed that abnormal sounds of vehicles are precursors to vehicle failures. If they cannot be discovered and resolved in time, they will have a great impact on driving safety. Therefore, it is a common method to identify whether a vehicle is faulty by the sound of the vehicle. In particular, when it is determined that the vehicle has an abnormal sound, it is determined that the vehicle has a fault. For example, a vehicle abnormal sound recording device discloses: the sounds of different areas of the vehicle are stored, and relevant personnel determine the vehicle's fault by comparing them with the sounds of specific faults. However, judging the fault of a vehicle based solely on abnormal sounds will result in inaccurate fault judgment.

[0038] In order to solve the above technical problems, the present application provides an audio recognition method, which is applied to a vehicle, and the method includes: in response to detecting an abnormal sound audio, determining an abnormal sound area according to the abnormal sound audio. In the case of detecting a fault code, determining the faulty component corresponding to the fault code according to the fault code. In the case where the area where the faulty component is located is the same as the abnormal sound area, determining that the abnormal sound audio is the audio emitted by the faulty component.

[0039] According to the above technical means, when an abnormal sound audio is detected, the abnormal sound area corresponding to the abnormal sound audio is determined. Further, it is detected whether the vehicle generates a diagnostic trouble code (DTC), and when the fault code is detected, the faulty component corresponding to the DTC is determined. Subsequently, when the area where the faulty component is located is the same as the abnormal sound area, it is determined that the abnormal sound audio is the audio emitted by the faulty component. In this way, according to the area where the faulty component is located and the abnormal sound area, it is determined whether the abnormal sound audio is the audio emitted by the faulty component, thereby improving the accuracy of fault detection.

[0040] like Figure 1 As shown, Figure 1An audio recognition system 100 is provided in an embodiment of the present application. The audio recognition system 100 may include a central processing unit 101, a sound processor 102, an image processor 103, a memory 104, a display 105, and a plurality of microphones ( Figure 1 Six microphones are shown as an example: microphone 1, microphone 2, microphone 3, microphone 4, microphone 5 and microphone 6. In actual applications, there may be more or fewer microphones. The central processor 101 is respectively connected to the sound processor 102, the image processor 103, the memory 104 and the display 105. The sound processor 102 is connected to multiple microphones.

[0041] The central processor 101 is used to determine the abnormal sound area according to the abnormal sound audio in response to detecting the abnormal sound audio. The central processor 101 is also used to determine the faulty component corresponding to the faulty code according to the faulty code when a fault code is detected, and to determine that the abnormal sound audio is the audio emitted by the faulty component when the area where the faulty component is located is the same as the abnormal sound area.

[0042] The central processor 101 is also used to generate abnormal prompt information when it is determined that the faulty component emits abnormal audio.

[0043] The sound processor 102 is used to process the audio data collected by multiple microphones.

[0044] Multiple microphones are used to collect the vehicle's operating audio data.

[0045] In some embodiments, the number of the plurality of microphones is 13. The operation and maintenance personnel place the 13 microphones at different locations to obtain audio data of different components.

[0046] Exemplarily, 13 microphones are installed in different parts of the vehicle (such as the body, brake disc, gearbox, suspension, tires, motor, engine, steering wheel, seat, window, air conditioner) to form a multi-dimensional audio data collection space for the chassis and the cabin to collect audio signals when the vehicle is running.

[0047] In some embodiments, two types of audio data are collected: 1. Audio data of normal operation of the vehicle's body, brake disc, gearbox, suspension, tires, motor, engine, steering wheel, seats, windows and air conditioner in various scenarios such as acceleration, deceleration, turning and braking; 2. Audio data of abnormal sound caused by abnormal operation of the vehicle's body, brake disc, gearbox, suspension, tires, motor, engine, steering wheel, seats, windows and air conditioner in various scenarios such as acceleration, deceleration, turning and braking.

[0048] In some embodiments, Figure 2 , a schematic diagram of a microphone layout is shown. Figure 2 , 9 microphones are shown: microphone 1, microphone 2, microphone 3, microphone 4, microphone 5, microphone 6, microphone 7, microphone 8, microphone 9, and a noise point, as well as the positional relationship between the 9 microphones and the noise point.

[0049] The display 105 is used to display the abnormal sound area and abnormal prompt information.

[0050] It should be noted that the display 105 includes a display screen. The display 105 can be a head-up display of the vehicle, a central control display, or a co-pilot display, which is not limited in the embodiment of the present application.

[0051] The memory 104 is used to store fault codes.

[0052] The image processor 103 is used to obtain scanning data of the vehicle from the three-dimensional laser scanning vibrometer, and obtain a three-dimensional model of the vehicle according to the scanning data.

[0053] The image processor 103 is further configured to mark the target area in the three-dimensional model and send the marked three-dimensional model to the display 105 so that the display 105 displays the marked three-dimensional model.

[0054] In some embodiments, when the vehicle is running, the audio recognition system 100 selects the fault codes DTC generated by the vehicle body system, brake disc system, gearbox system, suspension system, tire (tire pressure) system, motor drive system, engine system, steering wheel (steering) system, seat system, window (control system), and air conditioning (thermal management) system, and selects the fault types of mechanical faults, electrical faults, system-level communication bus faults, and circuit faults from all the generated DTCs as potential influencing factors for the generation of abnormal noises.

[0055] Subsequently, when the vehicle generates an abnormal sound, the microphone transmits the audio data to the vehicle sound processor 102, and the sound processor 102 transmits the audio data to the central processor 101 for comparative analysis and processing, and compares the analysis results with the full DTC of the vehicle. If the central processor 101 identifies the abnormal sound audio and locks the abnormal sound area, but no parts in the abnormal sound area generate DTC, it is determined that only abnormal sound exists in the area. If the central processor 101 identifies the abnormal sound audio and locks the abnormal sound area, and the parts in the abnormal sound area generate DTC at the same time, it is determined that the parts in the abnormal sound area have abnormal sound. For example, if the air conditioner generates an abnormal sound of about 45 decibels at 200 Hz, when the sound processor 102 identifies the abnormal sound area, the air intake grille of the air conditioning (thermal management) system generates a mechanical fault code (DTC), and the central processor determines that the air conditioner in the locked area generates an abnormal sound.

[0056] For ease of understanding, the audio recognition method provided by the present application is specifically introduced below in conjunction with the accompanying drawings. The audio recognition method in the embodiment of the present application takes an electronic device in a vehicle as an example.

[0057] Figure 3 According to a flowchart of an audio recognition method shown in an exemplary embodiment, the audio recognition method is applied to a vehicle, and the method includes: S201-S203.

[0058] S201. In response to detecting an abnormal sound audio, determine an abnormal sound area according to the abnormal sound audio.

[0059] As a possible implementation method, the electronic device periodically or in real time detects the audio collected by the microphone, and when abnormal sound audio is detected, in response to detecting the abnormal sound audio, determines the abnormal sound area according to the abnormal sound audio.

[0060] In some embodiments, when the electronic device obtains the operating audio, it detects the operating audio to determine whether the operating audio is abnormal sound audio.

[0061] For example, the electronic device obtains the audio features of the running audio, and inputs the audio features into the audio detection model to obtain the detection results of the audio features, thereby obtaining the detection results of the running audio. The detection results are used to indicate whether the audio features are abnormal sound audio features. For another example, the electronic device obtains the audio features of the audio, and calculates the similarity between the audio features and the abnormal audio features, and determines that the running audio is abnormal sound audio when the similarity between the audio features and the abnormal audio features is greater than or equal to the similarity threshold.

[0062] In some embodiments, the audio features are mel frequency cepstral coefficient (MFCC) parameters.

[0063] In some embodiments, the electronic device collects the operating audio of multiple components through microphones at different positions, and pre-processes the operating audio of the multiple components to obtain the operating audio of each component. Further, the electronic device detects whether the operating audio of each component is an abnormal sound audio.

[0064] It should be noted that the audio detection model is a model pre-trained by the operation and maintenance personnel. The audio detection model can be a neural network model. The similarity threshold is pre-configured in the electronic device by the operation and maintenance personnel, and the embodiment of the present application does not specifically limit the similarity threshold.

[0065] As a possible implementation method, in response to detecting the abnormal sound audio, the electronic device locates the abnormal sound audio and obtains the abnormal sound area where the sound source of the abnormal sound audio is located.

[0066] In some embodiments, the electronic device determines the target component corresponding to the abnormal sound audio, and locates the target component to obtain the position of the target component. Further, the area where the target component is located is determined as the abnormal sound area.

[0067] In some embodiments, the electronic device determines a target component corresponding to the target abnormal sound audio, and determines an abnormal sound area according to the target component and the area correspondence relationship. The area correspondence relationship includes multiple components and the area corresponding to each component.

[0068] It can be understood that the abnormal sound area is the area where the abnormal sound audio is emitted, and the embodiment of the present application does not specifically limit the size of the abnormal sound area.

[0069] S202: When a fault code is detected, determine a faulty component corresponding to the fault code according to the fault code.

[0070] In some embodiments, when the electronic device detects abnormal audio sound, it determines whether a fault code is detected, and when a fault code is detected, it determines the faulty component corresponding to the fault code according to the fault code.

[0071] S203: When the area where the faulty component is located is the same as the abnormal sound area, determine that the abnormal sound audio is the audio emitted by the faulty component.

[0072] In some embodiments, when the electronic device determines that a faulty component is present, it determines whether an area where the faulty component is present is the same as an area where the abnormal sound is present, and when the area where the faulty component is present is the same as the area where the abnormal sound is present, it determines that the abnormal sound audio is the audio emitted by the faulty component.

[0073] Subsequently, the electronic device marks the faulty component in the three-dimensional model, and displays the three-dimensional model with the marked faulty component on the display.

[0074] It should be noted that the area where the faulty component is located is the same as the abnormal sound area, which can be understood as: the area where the faulty component is located overlaps with the abnormal sound area, the overlap degree between the area where the faulty component is located and the abnormal sound area is greater than or equal to a preset threshold, or the area where the faulty component is located is located in the abnormal sound area, which is not limited to this embodiment of the present application.

[0075] The audio recognition method provided by the embodiment of the present application brings at least the following beneficial effects: when abnormal sound audio is detected, the abnormal sound area corresponding to the abnormal sound audio is determined. Further, it is detected whether the vehicle generates a fault code, and when the fault code is detected, the fault component corresponding to the DTC is determined. Subsequently, when the area where the faulty component is located is the same as the abnormal sound area, it is determined that the abnormal sound audio is the audio emitted by the faulty component. In this way, based on the area where the faulty component is located and the abnormal sound area, it is determined whether the abnormal sound audio is the audio emitted by the faulty component, thereby improving the accuracy of fault detection.

[0076] In a possible design, in order to improve the accuracy of fault detection, the audio recognition method provided in the embodiment of the present application further includes: S204-S205.

[0077] S204: When no fault code is detected, determine that the components in the abnormal noise area are operating normally.

[0078] As a possible implementation method, the electronic device determines that the components in the abnormal noise area are operating normally when the abnormal noise audio is detected and no fault code is detected.

[0079] In some embodiments, when the electronic device detects abnormal sound audio, it determines the abnormal sound time period of the abnormal sound audio, detects whether a fault code is generated during the abnormal sound time period, and when it is determined that no fault code is generated, it determines that the components in the abnormal sound area are operating normally.

[0080] S205: Displaying the abnormal sound area and abnormal sound prompt information on a display screen of the vehicle.

[0081] Among them, the abnormal noise prompt information is used to prompt that an abnormal noise occurs in the abnormal noise area and that the components in the abnormal noise area are operating normally.

[0082] In some embodiments, when the electronic device detects abnormal sound audio and does not detect a fault code, the abnormal sound area is marked in the three-dimensional model of the vehicle, and the marked three-dimensional model is displayed on the display screen. In addition, the electronic device generates abnormal sound prompt information and outputs the abnormal sound prompt information. For example, the electronic device outputs the abnormal sound prompt information through the speaker or displays the abnormal sound prompt information through the display screen.

[0083] In another case, the electronic device outputs the information to be detected when the abnormal sound audio is detected and the fault code is not detected. The information to be detected is used to remind the user that the vehicle has an abnormal sound. For example, the electronic device outputs the information to be detected through the audio, "The vehicle has an abnormal sound, please check it in time."

[0084] It can be understood that when abnormal audio is detected and no fault code is detected, it is determined that the current vehicle components are operating normally. In order to avoid the user's concerns about the safety of the vehicle, the user is prompted that the vehicle components are normal and can drive normally, thereby improving the user experience.

[0085] In a possible design, in order to determine abnormal sound audio, the audio recognition method provided in the embodiment of the present application also includes: S206-S209.

[0086] S206: Collect running audio through the microphone of the vehicle.

[0087] In some embodiments, the electronic device collects operating audio of different components through microphones at different locations.

[0088] For example, a vehicle is equipped with 13 microphones, and each microphone collects the operating audio of a part. The electronic device collects the operating audio of the part corresponding to each microphone through each microphone.

[0089] In other embodiments, the electronic device collects operating audio of multiple components through a microphone, and determines the operating audio of each component based on the operating characteristics of each component.

[0090] Exemplarily, when the electronic device acquires the operating audios of six components, the electronic device determines the operating audio of the target component from the six operating audios according to the frequency of the operating audio of each component.

[0091] S207: Determine the running audio features of the running audio.

[0092] As a possible implementation method, when acquiring the running audio, the electronic device preprocesses the running audio to obtain the processed running audio, and obtains the running audio features of the running audio according to a preset algorithm.

[0093] In some embodiments, the electronic device removes background noise and other irrelevant sounds from the operating audio to obtain the effective audio of the target component. For example, taking the target component as a gearbox, the electronic device removes background noise and other irrelevant sound signals from the operating audio of the gearbox to obtain the real operating audio of the gearbox.

[0094] In some embodiments, the electronic device improves the audio features of the running audio by extracting the MFCC features.

[0095] Specifically, the electronic device pre-processes the running audio and then extracts features from the pre-processed running audio. Exemplarily, the feature extraction process is as follows: Figure 4 As shown, including: S1-S5.

[0096] Since the human ear's perception of pitch is nonlinear, the Mel scale is used to linearly characterize the changes in audio data to improve the accuracy of audio features. Since the Mel scale is essentially a function of frequency, Hertz (Hz) is mapped to Mel (mel): The calculation formula of the Mel scale and frequency is shown in the following formula 1.

[0097] F mel (f) = 2595 × log 10 (1+f / 700) Formula 1

[0098] Among them, F mel (f) is the perceptual frequency domain in mel, and f is the actual speech frequency in Hz.

[0099] S1, pre-emphasis, framing and windowing processing.

[0100] When the electronic device obtains the preprocessed running audio, it pre-emphasizes, frames, and adds windows to the preprocessed running audio, thereby increasing part of the high-frequency energy and reducing the impact of high-frequency loss in signal transmission.

[0101] The pre-emphasis of the running audio may be a first-order difference of the running audio, and the first-order difference adopts the following formula 2.

[0102] x[n]=x[n]-αx[n-1] Formula 2

[0103] Wherein, 0.9≤α≤1.0, and n is the nth frame of audio.

[0104] It can be understood that the value range of α is limited to 0.9≤α≤1.0, in order to ensure that the processed signal will not be over-enhanced or distorted while maintaining the naturalness of the signal.

[0105] S2. Fast Fourier transform.

[0106] In some embodiments, the electronic device performs a fast Fourier transform (FFT) on each frame of audio after framing and windowing to transform the audio signal into the frequency domain.

[0107] S3. Calculate the energy of the spectral lines.

[0108] In some embodiments, the electronic device takes the square or the absolute value of each frame signal after FFT transformation.

[0109] S4, Mel filter bank filtering.

[0110] In some embodiments, the electronic device multiplies the obtained energy spectrum of each frame of spectral lines by the Mel filter bank to obtain a Mel spectrum.

[0111] S5. Discrete cosine transform.

[0112] In some embodiments, the electronic device performs a discrete cosine transform (DCT) on each frame of the signal after the Mel filter is filtered, and takes the 2nd to 13th coefficients after the DCT as MFCC coefficients to obtain MFCC parameters. Further, the MFCC parameters are determined as the characteristic values ​​of the frame of audio. In this way, the electronic device obtains the characteristic values ​​of each frame of the running audio, thereby obtaining the characteristic values ​​of the running audio.

[0113] In other embodiments, the electronic device inputs the running audio into an audio feature extraction model to obtain the running audio features.

[0114] It should be noted that the audio feature extraction model is a model pre-trained by operation and maintenance personnel and is used to extract audio features of audio.

[0115] S208: Calculate the similarity between the running audio feature and the abnormal sound audio feature.

[0116] In some embodiments, when the electronic device obtains the operating audio feature, it calculates the similarity between the operating audio feature and the abnormal sound audio feature in the audio feature database.

[0117] In some embodiments, the electronic device obtains an abnormal sound audio feature corresponding to the operating audio feature, and calculates the similarity between the operating audio feature and the abnormal sound audio feature.

[0118] For example, the operating audio feature is an operating audio feature of a gearbox. The electronic device obtains the abnormal sound audio feature of the gearbox, and calculates the operating audio feature of the gearbox and the abnormal sound audio feature of the gearbox.

[0119] In some embodiments, the electronic device obtains the operating condition corresponding to the operating audio feature, and determines the abnormal sound audio feature corresponding to the operating condition according to the operating condition. Further, the electronic device calculates the similarity between the operating audio feature and the abnormal sound audio feature.

[0120] For example, the operating audio is obtained under acceleration conditions. The electronic device determines the abnormal sound audio features under acceleration conditions, and calculates the similarity between the operating audio features and the abnormal sound audio features. For example, the operating audio is obtained under turning conditions. The electronic device determines the abnormal sound audio features under turning conditions, and calculates the similarity between the operating audio features and the abnormal sound audio features. For another example, the operating audio is emitted by the brake disc under deceleration conditions. The electronic device obtains the abnormal sound audio features of the brake disc under deceleration conditions, and calculates the similarity between the operating audio features and the abnormal sound audio features.

[0121] In some embodiments, before calculating the similarity between the running audio feature and the unusual sound audio feature, an unusual sound audio feature dataset is determined.

[0122] Specifically, the abnormal sound sample audio is extracted by the trained feature recognition model. Before using the feature recognition model to extract the abnormal sound sample audio, the initial feature recognition model is trained to obtain the trained feature recognition model. The initial feature recognition model training process is as follows: Figure 5 As shown, including: S21-S25.

[0123] S21. Data preprocessing.

[0124] S22, training parameters of feature recognition model.

[0125] In some embodiments, the feature recognition model is a backpropagation (BP) neural network, including forward propagation (calculating loss) and backpropagation (returning error).

[0126] Forward propagation is: given the initial weight value W of the initial feature recognition model and the value b of the bias term, calculate the final output value and the loss value between the output value and the actual value. If the loss value is not within the preset range, the back propagation process is performed. If the loss value is within the preset range, stop updating W and b.

[0127] For example, the calculation formula is as follows: Formula 3.

[0128] y=T(WX+b) Formula 3

[0129] Among them, T represents the activation function, b represents the activation threshold, W represents the connection weight, X represents the input value, and y represents the output value.

[0130] Back propagation is: the output is transmitted back through the hidden layer to the input layer layer by layer, and the error is distributed to all units in each layer, so as to obtain the error signal of each layer unit. This error signal is used as the basis for correcting the weights of each unit. The error function of the BP neural network is the mean square error function:

[0131]

[0132] Among them, m is the number of training samples, k is the number of outputs, is the jth output prediction value of the i-th sample, y ij is the corresponding true value.

[0133] In some embodiments, the audio feature data trained by the BP neural network is classified and trained so that it can identify normal audio features and abnormal audio features of different types of parts.

[0134] S23: Determine whether the training conditions are met.

[0135] If yes, then execute S24. If no, then execute S25.

[0136] S24, obtain a trained feature recognition model.

[0137] In some embodiments, the operating audio of the target component is input into a trained feature recognition model to obtain a result of whether the operating audio of the target component is an abnormal sound audio.

[0138] S25. Modify the parameters of the feature recognition model.

[0139] In some embodiments, after modifying the parameters of the feature recognition model, S22 is executed.

[0140] S209: When the similarity between the operating audio feature and the abnormal sound audio feature is greater than or equal to a similarity threshold, determine that the operating audio is the abnormal sound audio.

[0141] In some embodiments, when obtaining the similarity between the operating audio feature and the abnormal sound audio feature, the electronic device determines that the similarity between the operating audio feature and the abnormal sound audio feature is greater than or equal to a similarity threshold, and determines that the operating audio is abnormal sound audio when the similarity between the operating audio feature and the abnormal sound audio feature is greater than or equal to the similarity threshold. Otherwise, the operating audio is determined to be abnormal sound audio.

[0142] In one design, in order to improve the accuracy of audio recognition, the above S208 includes: S2081-S2083.

[0143] S2081. Determine the target component that emits the running audio.

[0144] S2082. Obtain abnormal sound audio features corresponding to the target component.

[0145] Among them, the abnormal sound audio feature is collected when the target component operates abnormally.

[0146] S2083. Calculate the similarity between the running audio feature and the abnormal sound audio feature corresponding to the target component.

[0147] The above S209 includes: S2091.

[0148] S2091. Determine that the running audio is the abnormal sound audio emitted by the target component.

[0149] In one design, the above-mentioned S2081 includes: S301.

[0150] S301, determining a target component according to the frequency of the running audio and the component correspondence.

[0151] The component correspondence relationship includes multiple components and the frequency corresponding to each component.

[0152] In one design, the vehicle includes N microphones, and the above S201 includes: S2011.

[0153] S2011. Determine an abnormal sound area according to the abnormal sound audio collected by M microphones.

[0154] Among them, N≥M>2.

[0155] For example, Figure 6 As shown, the noise point is located according to microphone 1, microphone 2 and microphone 3.

[0156] In some embodiments, in order to allow users to understand the operating status of the vehicle in a timely manner, the audio recognition method provided in the embodiments of the present application also includes: S210-S211.

[0157] S210: Mark the position of the faulty component in the three-dimensional model of the vehicle on a display screen of the vehicle.

[0158] S211. Output fault prompt information.

[0159] Among them, the fault prompt information is used to prompt that the faulty component is operating abnormally.

[0160] In some embodiments, the electronic device saves the abnormal noise audio.

[0161] It can be understood that if only the abnormal noise area is locked, it is determined that the abnormal noise area produces an abnormal noise, the location of the abnormal noise area is pushed to the vehicle display, and the location of the abnormal noise is displayed in a three-dimensional (3D) model, and the abnormal noise audio data is stored. If the abnormal noise area and the component fault code are locked at the same time, the component is determined to be an abnormal noise fault, and the abnormal noise area and component identification are pushed to the vehicle display to display the specific location of the abnormal noise area and the component, the abnormal noise cause corresponding to the fault code in the 3D model, and the abnormal noise audio data is stored.

[0162] In some embodiments, to determine the audio characteristics,

[0163] In order to better understand the audio recognition method provided in the embodiment of the present application, Figure 7 As shown, an audio recognition process is shown, including: S401-S410.

[0164] S401, collecting audio data.

[0165] S402: Extract audio features.

[0166] S403: Establish an audio feature set according to the audio features.

[0167] In some embodiments, the electronic device marks normal audio features and abnormal audio features of each component under different working conditions to obtain an audio feature set.

[0168] For example, the collected audio data feature value set is labeled according to type and status, and the labeled results are mapped with the "3D modeling data" of the entire vehicle including "vehicle body, brake disc, gearbox, suspension, tires, motor, engine, steering wheel, seat, window and air conditioning".

[0169] S404: Establish a three-dimensional model of the vehicle.

[0170] S405, collecting running audio.

[0171] S406: In response to the input running audio, determine the running audio features of the running audio.

[0172] S407: Determine whether the running audio feature is an abnormal audio feature.

[0173] If yes, then execute S408. If no, then execute S405.

[0174] S408: Determine an abnormal sound area of ​​the running audio.

[0175] S409: Obtain a fault code.

[0176] S410: When the abnormal sound area is the same as the area where the faulty component is located, display a three-dimensional model marking the abnormal sound area.

[0177] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, the communication device includes a hardware structure and / or software module corresponding to each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0178] Figure 8 An audio recognition device according to an exemplary embodiment is applied to a vehicle. Figure 8 , the audio recognition device 50 includes: a determination unit 501.

[0179] The determination unit 501 is used to determine the abnormal sound area according to the abnormal sound audio in response to detecting the abnormal sound audio. The determination unit 501 is used to determine the faulty component corresponding to the faulty code according to the faulty code when a faulty code is detected. The determination unit 501 is used to determine that the abnormal sound audio is the audio emitted by the faulty component when the area where the faulty component is located is the same as the abnormal sound area.

[0180] One possible approach is Figure 8 As shown, the audio recognition device 50 further includes a display unit 502. The determination unit 501 is also used to determine that the components in the abnormal sound area are operating normally when no fault code is detected. The display unit 502 is used to display the abnormal sound area and abnormal sound prompt information on the display screen of the vehicle. The abnormal sound prompt information is used to prompt that an abnormal sound occurs in the abnormal sound area and that the components in the abnormal sound area are operating normally.

[0181] One possible approach is Figure 8 As shown, the audio recognition device 50 also includes a collection unit 503 and a processing unit 504. The collection unit 503 is used to collect the running audio through the microphone of the vehicle. The determination unit 501 is also used to determine the running audio features of the running audio. The processing unit 501 is used to calculate the similarity between the running audio features and the abnormal sound audio features. The determination unit 501 is also used to determine that the running audio is an abnormal sound audio when the similarity between the running audio features and the abnormal sound audio features is greater than or equal to the similarity threshold.

[0182] In one possible manner, the processing unit 501 is specifically used to: determine the target component that emits the operating audio, and obtain the abnormal sound audio feature corresponding to the target component. The abnormal sound audio feature is collected when the target component operates abnormally. Calculate the similarity between the operating audio feature and the abnormal sound audio feature corresponding to the target component. The determination unit 501 is specifically used to: determine that the operating audio is the abnormal sound audio emitted by the target component.

[0183] In a possible manner, the determination unit 501 is specifically configured to determine a target component according to the frequency of the running audio and a component correspondence relationship, wherein the component correspondence relationship includes multiple components and a frequency corresponding to each component.

[0184] In a possible manner, the vehicle includes N microphones, and the determination unit 501 is specifically configured to determine the abnormal sound area according to the abnormal sound audio collected by the M microphones. N≥M>2.

[0185] One possible approach is Figure 8 As shown, the audio recognition device 50 further includes an output unit 505. The display unit 502 is also used to mark the position of the faulty component in the three-dimensional model of the vehicle on the display screen of the vehicle. The output unit 505 is used to output fault prompt information. The fault prompt information is used to indicate that the faulty component is operating abnormally.

[0186] Fig. 9 FIG. 1 is a schematic diagram of an electronic device according to an exemplary embodiment. Fig. 9 As shown, the electronic device includes but is not limited to: a processor 601 and a memory 602 .

[0187] The memory 602 is used to store executable instructions of the processor 601. It can be understood that the processor 601 is configured to execute instructions to implement the model training method and SOC estimation method in the above embodiment.

[0188] It should be noted that those skilled in the art can understand that Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include Fig. 9More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0189] The processor 601 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 601 may include one or more processing units. Optionally, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.

[0190] The memory 602 may be used to store software programs and various data. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one functional module (such as a determination unit, a processing unit, etc.), etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0191] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 602 including instructions, and the above instructions can be executed by a processor 601 of an electronic device to implement the method in the above embodiment.

[0192] Optionally, the computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0193] In an exemplary embodiment, the present application also provides a computer program product including one or more instructions, and the one or more instructions can be executed by the processor 601 of the electronic device to complete the method in the above embodiment.

[0194] It should be noted that when the instructions in the above-mentioned computer-readable storage medium or one or more instructions in the computer program product are executed by the processor of the electronic device, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect as the above-mentioned method can be achieved. To avoid repetition, they will not be repeated here.

[0195] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0196] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0197] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0198] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0199] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the full classification part or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute the full classification part or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or CD and other media that can store program code.

[0200] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An audio recognition method, characterized in that: Applied to a vehicle, the method comprises: In response to detecting an abnormal sound audio, determining an abnormal sound area according to the abnormal sound audio; In the case where a fault code is detected, determining a faulty component corresponding to the fault code according to the fault code; When the area where the faulty component is located is the same as the abnormal sound area, it is determined that the abnormal sound audio is the audio emitted by the faulty component.

2. The method according to claim 1, characterized in that The method further comprises: When no fault code is detected, it is determined that the components in the abnormal noise area are operating normally, and the abnormal noise area and abnormal noise prompt information are displayed on a display screen of the vehicle; the abnormal noise prompt information is used to prompt that an abnormal noise occurs in the abnormal noise area and that the components in the abnormal noise area are operating normally.

3. The method according to claim 1, characterized in that The method further comprises: collecting operating audio through a microphone of the vehicle, and determining operating audio features of the operating audio; The similarity between the operating audio feature and the abnormal sound audio feature is calculated, and when the similarity between the operating audio feature and the abnormal sound audio feature is greater than or equal to a similarity threshold, the operating audio is determined to be the abnormal sound audio.

4. The method according to claim 1, characterized in that: The calculating the similarity between the running audio feature and the abnormal sound audio feature includes: Determine a target component that emits the operating audio, and obtain an abnormal sound audio feature corresponding to the target component; the abnormal sound audio feature is collected when the target component operates abnormally; Calculating the similarity between the running audio feature and the abnormal sound audio feature corresponding to the target component; The determining that the running audio is an abnormal sound audio includes: The operating audio is determined to be an abnormal sound audio emitted by the target component.

5. The method according to claim 4, characterized in that The step of determining a target component that emits the running audio comprises: The target component is determined according to the frequency of the running audio and the component correspondence; the component correspondence includes multiple components and the frequency corresponding to each component.

6. The method according to any one of claims 1 to 5, characterized in that The vehicle includes N microphones, and determining the abnormal sound area according to the abnormal sound audio includes: The abnormal sound area is determined according to the abnormal sound audio collected by M microphones; wherein N≥M>2.

7. The method according to any one of claims 1 to 5, characterized in that After determining that the abnormal sound audio is the audio emitted by the faulty component, the method further includes: The position of the faulty component in the three-dimensional model of the vehicle is marked on the display screen of the vehicle, and fault prompt information is output; the fault prompt information is used to prompt that the faulty component is operating abnormally.

8. An audio recognition device, characterized in that: Applied to a vehicle, the device comprises: a determination unit; The determining unit is configured to determine an abnormal sound area according to the abnormal sound audio in response to detecting the abnormal sound audio; The determination unit is used to determine the fault component corresponding to the fault code according to the fault code when the fault code is detected; The determining unit is used to determine that the abnormal sound audio is the audio emitted by the faulty component when the area where the faulty component is located is the same as the abnormal sound area.

9. An electronic device, characterized in that: including memory and processor; The memory is coupled to the processor; The memory is used to store computer program code, wherein the computer program code includes computer instructions; When the processor executes the computer instructions, the electronic device performs the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the computer-executable instructions stored in the computer-readable storage medium are executed by a processor of a processing device, the processing device can perform the method according to any one of claims 1 to 7.

Images