Whole vehicle hearing method and system and medium

Abstract

The invention belongs to the technical field of intelligent automobiles, and particularly relates to a whole vehicle hearing method and system and a medium, and the whole vehicle hearing method comprises the steps: obtaining a to-be-trained sound; marking the to-be-trained sound to obtain a marked sound and a corresponding auditory label; training the marked sound and the corresponding auditory label in a preset model to obtain an auditory model; environment sound is collected and subjected to sound processing; and identifying the environment sound after sound processing based on the auditory model, and generating an auditory result. The problem that an existing whole automobile does not have human-like hearing ability is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent automobiles, and in particular relates to a whole-vehicle hearing method, system and medium. Background Art

[0002] With the development of automobile technology, the requirements for automobile intelligence are also increasing. In the existing technology, although there are technologies and corresponding supporting scenarios for answering by acquiring voice in the environment, the accuracy of existing voice recognition technology is low, and it is more dependent on existing voice recognition models or algorithms, and is only used for intelligent interaction, which cannot meet the needs of users.

[0003] Especially in the process of the increasing degree of automobile autonomous driving, how to assist the car to achieve autonomous driving and intelligent interactive services with users by sensing external sounds, and how to timely understand the car's own status and external status that is closely related to the car by sensing external sounds, so as to meet driving safety are problems that need to be urgently solved by existing technologies. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a whole vehicle hearing method, system and medium, which solves the problem that the existing whole vehicle does not have human-like hearing ability.

[0005] To achieve the above purpose, the present invention adopts the following technical solution.

[0006] A first aspect of the present invention provides a whole vehicle hearing method, comprising: Get the sound to be trained; Mark the training sound to obtain the marked sound and its corresponding auditory label; The marked sounds and the corresponding auditory labels are trained in a preset model to obtain an auditory model; Collect environmental sounds and process them; The processed environmental sound is recognized based on the auditory model to generate an auditory result.

[0007] As an embodiment of the present invention, the preset model is composed of a convolutional neural network and a gated recurrent neural network based on each running node, the convolutional neural network is applied to the first running node, and the gated recurrent neural network is applied to the second running node.

[0008] As an implementation mode of the present invention, the marked sound and the corresponding auditory label are trained in a preset model to obtain an auditory model, including: Inputting the labeled sound into a convolutional neural network; The output of the convolutional neural network model is converted into a sequence through the custom reshaping layer of the convolutional neural network, and input into the gated recurrent neural network, which outputs the auditory label and performs a preset number of trainings to obtain the auditory model.

[0009] As an implementation manner of the present invention, the sound to be trained includes at least: Noise, stone hitting, vehicle passing over bumps, car horns, human speech and human non-verbal sounds.

[0010] As an implementation mode of the present invention, the step of marking the training sound to obtain the marked sound and its corresponding auditory label includes: Marking the noise to obtain a marked sound as at least one of wind noise, road noise, abnormal noise and motor noise, wherein the auditory labels corresponding to the wind noise and the road noise are passive actions of the vehicle, and the auditory labels corresponding to the abnormal noise and the motor noise are active actions of the vehicle; and / or The stone hitting sound is marked, and the marked sound is at least one of the following: stone hitting glass sound, stone hitting sheet metal sound, and stone hitting guard plate sound, and the corresponding auditory labels are all vehicle passive actions; and / or The sound of a vehicle passing over a bump is marked, and the marked sound is at least one of the sound of a vehicle pressing down a manhole cover, the sound of a vehicle passing over a deep pit, and the sound of a vehicle passing over a speed bump, and the corresponding auditory labels are all passive actions of the vehicle.

[0011] As an implementation mode of the present invention, the collecting of environmental sound and performing sound processing on it includes: Filtering the collected ambient sound, and intercepting it according to preset time segments after filtering to obtain multiple ambient sound segments; The audio features in the multiple environmental sound segments are extracted through the Mel spectrum to obtain a spectrum image, and the spectrum image is the environmental sound after sound processing.

[0012] A second aspect of the present invention provides a vehicle hearing system, comprising: An acquisition unit, at least used to acquire the sound to be trained; A labeling unit, at least used to label the training sound to obtain the labeled sound and its corresponding auditory label; A training unit, at least used to train the marked sound and the corresponding auditory label in a preset model to obtain an auditory model; A collection unit, at least used to collect environmental sounds and perform sound processing on them; The generating unit is at least used to recognize the environmental sound after the sound processing based on the auditory model to generate an auditory result.

[0013] As an implementation mode of the present invention, the acquisition unit is connected to a cloud server and includes a vehicle-mounted microphone and a vehicle-mounted MCU; The on-board microphone collects ambient sound as a sensor for collecting ambient sound, and the on-board MCU collects ambient sound as a sound collector for collecting ambient sound. The on-board MCU transmits the collected ambient sound to the cloud server.

[0014] A third aspect of the present invention provides an electronic device, comprising: At least one processor; and at least one memory in communication with the processor, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the steps of the method described in the first aspect of the present invention.

[0015] A fourth aspect of the present invention provides a readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the steps of the method described in the first aspect of the present invention.

[0016] In summary, compared with the prior art, the present invention includes at least one of the following beneficial technical effects: 1. After marking the training sound, the present invention can match the collected sound with the preset auditory label to achieve the corresponding classification of the sound and its label; 2. The auditory model obtained through training in the present invention can recognize the collected environmental sounds, distinguish different sounds and their corresponding visual labels, and generate specific auditory outputs. For example, if the identified marker sound is wind noise, road noise, abnormal noise or motor noise, etc., it is conducive to triggering early warning behaviors or executing corresponding control behaviors in response to the identified sound events, thereby improving the user's driving experience and vehicle safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a flow chart of a whole vehicle hearing method according to a specific embodiment of the present invention.

[0019] Figure 2 It is a schematic diagram of dividing a marked sound according to a specific embodiment of the present invention.

[0020] Figure 3 It is a flow chart of another whole-vehicle auditory method according to a specific embodiment of the present invention.

[0021] Figure 4 It is a partial flow chart of a whole-vehicle auditory method according to a specific embodiment of the present invention.

[0022] Figure 5 It is a flow chart of another whole-vehicle auditory method according to a specific embodiment of the present invention.

[0023] Figure 6 It is a block diagram of a whole vehicle hearing system according to a specific embodiment of the present invention.

[0024] Figure 7 FIG. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. In addition, it should be understood that the specific implementation methods described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0026] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments of the present application. In addition, in the following embodiments, the description of each embodiment has its own emphasis, and for parts not described in detail in one embodiment, reference can be made to the relevant description of other embodiments.

[0027] The whole vehicle auditory method or system involves multiple areas such as vehicle safety, performance monitoring, environmental perception, user experience and regulatory compliance. Through this method or system, a more comprehensive and in-depth understanding of the vehicle and its surrounding environment can be achieved. Among them: With the improvement of the level of vehicle intelligence, real-time monitoring of vehicle operation status has become increasingly important. Auditory methods can be used as an effective monitoring method to help detect abnormal sounds of vehicles, thereby preventing potential failures and accidents; Auditory perception is an important component of the autonomous driving system, which can supplement the deficiencies of visual and radar perception, especially in obstacle recognition and perception under obstructed vision or bad weather conditions; Vehicles will produce various sounds during operation, which can reflect the health of the vehicle. By analyzing these sounds, vehicle problems, such as abnormal engine noise and brake system problems, can be discovered in time, so as to carry out repairs and maintenance; Vehicles will encounter various environmental sounds during driving, such as the horns of other vehicles and road condition sounds. Through auditory methods, vehicles can better understand the surrounding environment and improve driving safety; Through auditory methods, vehicles can identify the sounds of emergency vehicles (such as ambulances, fire trucks and police), and take corresponding avoidance measures, such as abnormal sounds outside the vehicle, such as stone hitting sounds, to alert the driver.

[0028] like Figure 1 and Figure 2 As shown, the first aspect of the present invention provides a whole vehicle hearing method, comprising the following steps.

[0029] Step S100: Acquire the sounds to be trained; specifically, in the process of shaping the car's hearing, in order to improve the intelligent interaction with the car, the sounds to be trained that need to be acquired include but are not limited to human language sounds and human non-speech sounds, wherein human language sounds are such as the interaction sounds between humans and the smart cockpit, such as the human voice of "help me turn on the air conditioner", and human non-speech sounds, such as coughing and sneezing sounds; and the sounds to be trained also include noise, stone hitting sounds, vehicle passing over bumps, car horns, etc., wherein the noise can be further divided into wind noise, The noise includes road noise, abnormal noise and motor noise, etc., and the abnormal noise includes abnormal brake noise, chassis noise and body noise, etc., and the motor noise includes air-conditioning noise, drive motor noise and water pump noise, etc.; the stone impact sound is further divided into stone impact glass sound, stone impact sheet metal sound and stone impact guard plate sound based on the location where the stone impact sound is generated; the vehicle passing over the bump sound is further divided into the vehicle pressing manhole cover sound, the vehicle passing over the deep pit sound and the vehicle passing over the speed bump sound based on the area where the vehicle passes over the bump; the car horn sound is further divided into police car sound, fire truck sound and ambulance sound based on the source of the received horn sound.

[0030] Step S200: Mark the training sound to obtain the marked sound and its corresponding auditory label; specifically, among the training sounds, such as human language sounds and human non-speech sounds, they represent the impact of hearing on intelligent interaction that needs to be improved and the experience enhanced, and such as noise, stone hitting sounds, vehicle crossing bump sounds, car horn sounds, etc., they represent the impact of hearing in external conditions that are closely related to the car. By marking the training sounds, the marked sound and its corresponding auditory label can be obtained. The marking process is: according to the category of the training sound, the training sound is divided into noise, stone hitting sounds, vehicle crossing bump sounds, car horn sounds, human language sounds, human non-speech sounds, etc. and their corresponding secondary branches, and the corresponding auditory label is designed for its smallest branch, and the auditory label includes active actions and passive actions, wherein the purpose of adding active actions and passive actions to the label is to provide training references for each training action and to provide a more accurate auditory information acquisition method for the subsequent whole vehicle hearing.

[0031] The present invention can provide corresponding clear feedback to the driver and passengers by designing active action labels and passive action labels, which can assist the driver and passengers or be input into the automatic driving system to judge or execute subsequent actions. In addition, the present invention can also provide a certain reference for automobile noise development by further refining the sound attributes and using them as the basis for sound logic judgment.

[0032] Step S300: training the marked sound and the corresponding auditory label in a preset model to obtain an auditory model.

[0033] Specifically, the preset model is composed of a convolutional neural network CNN and a gated recurrent neural network GRU based on each running node, the convolutional neural network is applied to the first running node, and the gated recurrent neural network is applied to the second running node, wherein the first node can be understood as the first time stage, i.e., the time stage using CNN, and the second running node can be understood as the second time stage, i.e., the time stage using GRU.

[0034] like Figure 3 As shown, in step S300, the marked sound and the corresponding auditory label are trained in a preset model to obtain an auditory model, including: Step S310: inputting the marked sound into a convolutional neural network; Step S320: The output of the convolutional neural network model is converted into a sequence through the custom reshaping layer of the convolutional neural network, and is input into the gated recurrent neural network, the auditory label is output, and a preset number of trainings are performed to obtain an auditory model.

[0035] Specifically, CNN has good performance in image processing, and it can capture local features and spatial hierarchical structures. In sound processing, CNN can effectively extract local features in time series data, such as frequency and time patterns, and has feature extraction capabilities; GRU is good at processing sequence data, and it can capture long-term dependencies and has sequence modeling capabilities. Therefore, CNN can process the spectrogram of sound signals (converting sound signals into time-frequency representations), while GRU can process these feature sequences and understand the dynamic changes of sounds over time. Combining CNN and GRU can directly learn the sound to be trained to the final labeled sound and its corresponding auditory label without manually designing a complex feature extraction process. In addition, the present invention uses a custom reshaping layer between CNN and GRU to effectively convert the output of CNN into a sequence format suitable for gated recurrent neural network processing, which helps the flow and integration of information between the two networks and improves the accuracy of the auditory model.

[0036] As a specific implementation method, Figure 4 As shown, in the present invention, the specific neural network model is established and the algorithm for running deep learning is as follows: Step 1: Define the convolutional neural network architecture, which mainly includes the following structures: Image input layer; used to specify the image size, the image height is 13 and the width is 50; The first convolutional layer: 2D convolution layer: convolution kernel size 3*3, number of convolution kernels 64; Batch Normalization Layer: Normalizes activation values ​​and gradient propagation in neural networks; Rectified linear unit; Max pooling layer: The convolution kernel size is 8*2 to reduce the spatial size of the feature map and remove redundant spatial information; The second convolutional layer: Two-dimensional convolution layer: convolution kernel size 3*3, number of convolution kernels 128; Batch Normalization Layer: Normalizes activation values ​​and gradient propagation in neural networks; Rectified Linear Unit: Max pooling layer: The convolution kernel size is 8*2 to reduce the spatial size of the feature map and remove redundant spatial information; The third convolutional layer: 2D convolution layer: convolution kernel size 3*3, number of convolution kernels 256; Batch Normalization Layer: Normalizes activation values ​​and gradient propagation in neural networks; Rectified Linear Unit: Max pooling layer: The convolution kernel size is 2*2 to reduce the spatial size of the feature map and remove redundant spatial information; The fourth convolutional layer: 2D convolution layer: convolution kernel size 3*3, number of convolution kernels 512; Batch Normalization Layer: Normalizes activation values ​​and gradient propagation in neural networks; Rectified Linear Unit: Max pooling layer: The convolution kernel size is 1*1 to reduce the spatial size of the feature map and remove redundant spatial information; Reshape layer: Define the recurrent neural network architecture: Sequence input layer: sequence length is 1024; Bidirectional gated recursive learning layer: The number of hidden layers is 1024 and 256 respectively; Bidirectional gated recursive learning layer: The number of hidden layers is 512 and 256 respectively; Bidirectional gated recursive learning layer: The number of hidden layers is 512 and 256 respectively; Fully connected layer: 1024 layers; Rectified linear unit layer; Fully connected layer: 1024 layers; Rectified linear unit layer; Fully connected layer: 1024 layers; Rectified linear unit layer; Fully connected layer: 1024 layers; Sigmoid activation layer: outputs sound actions, i.e. auditory results; Step ②: Transfer the data (labeled sounds and their corresponding auditory labels) to the constructed model, perform model training, and obtain the auditory model.

[0037] Optionally, by transmitting the auditory model to the vehicle computer, the vehicle computer can have human-like hearing.

[0038] Step S400: Collect ambient sound and perform sound processing on it. Specifically, the vehicle-mounted microphone can be used as a sensor for collecting ambient sound, and the vehicle-mounted MCU can be used as a sound collector for collecting ambient sound. Based on the current status of existing Internet of Vehicles technology and cloud technology, in one application scenario, the present invention transmits the collected ambient sound to a cloud server via a cellular network.

[0039] like Figure 5 As shown, in step S400, the collecting of environmental sound and performing sound processing on it include: Step S410: filtering the collected ambient sound, and intercepting it according to preset time segments after filtering to obtain multiple ambient sound segments; Step S420: extracting audio features from the multiple environmental sound segments through MFCC to obtain a spectrum image, where the spectrum image is the environmental sound after sound processing.

[0040] In an application scenario of the present invention, the cloud server performs necessary filtering processing on the collected ambient sound, intercepts the ambient sound into preset time segments such as 0.5s, and uses MFCC (Mel Frequency Cepstral Coefficients) to extract audio features to obtain a spectrum image, which is used as the input of the auditory model.

[0041] Step S500: Recognize the processed environmental sound based on the auditory model to generate an auditory result.

[0042] Specifically, the auditory model includes sound categories corresponding to environmental sounds, such as wind noise, and their corresponding auditory labels, such as passive actions. The present invention can further process the auditory results obtained, for example, for wind noise and road noise, the vehicle can be further analyzed for energy consumption within a preset time period, and for abnormal noise and motor noise, the vehicle can be further monitored for vehicle health and fault diagnosis within a preset time period.

[0043] In one embodiment of the present invention, the step of labeling the training sound to obtain the labeled sound and its corresponding auditory label includes: Marking the noise to obtain a marked sound as at least one of wind noise, road noise, abnormal noise and motor noise, wherein the auditory labels corresponding to the wind noise and the road noise are passive actions of the vehicle, and the auditory labels corresponding to the abnormal noise and the motor noise are active actions of the vehicle; and / or The stone hitting sound is marked, and the marked sound is at least one of the following: stone hitting glass sound, stone hitting sheet metal sound, and stone hitting guard plate sound, and the corresponding auditory labels are all vehicle passive actions; and / or The sound of a vehicle passing over a bump is marked, and the marked sound is at least one of the sound of a vehicle pressing down a manhole cover, the sound of a vehicle passing over a deep pit, and the sound of a vehicle passing over a speed bump, and the corresponding auditory labels are all passive actions of the vehicle.

[0044] It should be noted that Figure 2 Only one classification standard for the marker sounds in the present invention is shown. Those skilled in the art can choose to classify the marker sounds according to actual conditions, such as classifying the sound of stone hitting, the sound of vehicle passing over bumps, the sound of car horns, etc. into the branch of noise, etc. The present invention does not make any specific limitation on this.

[0045] In one embodiment of the present invention, before labeling the training sound to obtain the labeled sound and its corresponding auditory label, the method further includes: Get user voice; The human speech sound or human non-verbal sound inside and outside the vehicle is acquired based on the user's voice.

[0046] Furthermore, when the environmental sound after sound processing is identified based on the auditory model to generate an auditory result, if the marked sound in the generated auditory result is the corresponding human language sound or human non-language sound in the car, the human language sound or human non-language sound corresponding to the user is sent to the car control domain with voice interaction function to perform intelligent interactive voice; if the marked sound is the human language sound or human non-language sound corresponding to the human outside the car, the human language sound or human non-language sound corresponding to the human outside the car is sent to the car control domain with active noise control function to perform active noise control.

[0047] Optionally, for human language sounds or human non-language sounds corresponding to humans in the car, the corresponding auditory labels are active vehicle actions; for human language sounds or human non-language sounds corresponding to humans outside the car, the corresponding auditory labels are passive vehicle actions. Based on the active vehicle actions, i.e., the corresponding labeled sounds, the present invention can assist vehicle-machine interaction in achieving timely response to user intelligent interactions; based on the passive vehicle actions and the corresponding labeled sounds, the present invention can perform active noise control, or based on the passive vehicle actions and the corresponding labeled sounds, set recognition and filtering of key sensitive words such as human language sounds: "help", "help" and human non-language sounds: crying, shouting, etc., to improve the safety response capability to emergencies.

[0048] Optionally, in the present invention, the marked sound and the auditory label are bound. When the sound-processed ambient sound is identified based on the auditory model and an auditory result is generated, the marked sound corresponding to the ambient sound and its corresponding auditory label are generated accordingly. Based on the marked sound and its corresponding auditory label, corresponding warning behaviors or control behaviors for the vehicle can also be designed accordingly.

[0049] In an application scenario of the present invention, if the marked sound is the sound of a vehicle passing through a deep pothole, its corresponding auditory label is an active action of the vehicle, and the sound of the vehicle passing through a deep pothole in the auditory results obtained within a preset time exceeds a preset threshold, the vehicle suspension system, vehicle tires, and vehicle chassis are warned through the corresponding vehicle control domain, and a vehicle speed reference is provided to the user. The vehicle speed reference value is not restricted here, and it needs to be as slow as possible and within a safe threshold to ensure driving safety.

[0050] Similarly, in the present invention, when technical personnel in this field obtain various types of marking sounds contained in the auditory effects and their corresponding auditory labels, they can design corresponding warning behaviors or control behaviors for the vehicle in the above-mentioned manner, which should fall within the scope of protection of the present invention.

[0051] like Figure 6 As shown, the second unit of the present invention provides a whole vehicle hearing system, including: An acquisition unit 61, at least used to acquire a sound to be trained; A marking unit 62, at least used to mark the training sound to obtain the marked sound and its corresponding auditory label; A training unit 63, at least used to train the marked sound and the corresponding auditory label in a preset model to obtain an auditory model; A collection unit 64, at least used to collect environmental sounds and perform sound processing on them; The generating unit 65 is at least used to recognize the environmental sound after the sound processing based on the auditory model to generate an auditory result.

[0052] In one embodiment of the present invention, the acquisition unit 64 is connected to a cloud server and includes a vehicle-mounted microphone and a vehicle-mounted MCU; The on-board microphone collects ambient sound as a sensor for collecting ambient sound, and the on-board MCU collects ambient sound as a sound collector for collecting ambient sound. The on-board MCU transmits the collected ambient sound to the cloud server.

[0053] The present invention uses the acquired auditory results to, on the one hand, send the collected human language sounds and human non-language sounds that are beneficial to the user's intelligent interactive services to the voice service server to serve the vehicle-computer interaction; on the other hand, it further analyzes the environmental sounds that affect driving safety, such as stone hitting sounds, and based on the corresponding auditory labels, corresponding warning behaviors or control behaviors for the vehicle can also be designed to improve driving safety.

[0054] Based on the same idea as the method in the above embodiment, the whole vehicle auditory system provided by the present invention can implement the method in the above embodiment. For the convenience of explanation, the structural diagram of the system embodiment only shows the part related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the system, and may include more or fewer units than shown in the figure, or combine certain units, or arrange the units differently.

[0055] A third aspect of the present invention provides an electronic device, comprising: At least one processor; and at least one memory in communication with the processor, wherein: the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the steps of the method described in any one of the above embodiments.

[0056] A fourth aspect of the present invention discloses a readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the steps of the method described in any one of the above embodiments.

[0057] Computer-readable storage media may include: any entity or device capable of carrying a computer program, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc. A computer program includes a computer program code. The computer program code may be in source code form, object code form, an executable file, or some intermediate form, etc. A computer-readable storage medium may include: any entity or device capable of carrying a computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc.

[0058] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0059] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0060] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A whole vehicle auditory method, characterized in that: include: Get the sound to be trained; Mark the training sound to obtain the marked sound and its corresponding auditory label; The marked sounds and the corresponding auditory labels are trained in a preset model to obtain an auditory model; Collect environmental sounds and process them; The processed environmental sound is recognized based on the auditory model to generate an auditory result.

2. The whole vehicle hearing method according to claim 1, characterized in that: The preset model consists of a convolutional neural network and a gated recurrent neural network based on each running node. The convolutional neural network is applied to the first running node, and the gated recurrent neural network is applied to the second running node.

3. The whole vehicle hearing method according to claim 2, characterized in that: The marked sound and the corresponding auditory label are trained in a preset model to obtain an auditory model, including: Inputting the labeled sound into a convolutional neural network; The output of the convolutional neural network model is converted into a sequence through the custom reshaping layer of the convolutional neural network, and input into the gated recurrent neural network, which outputs the auditory label and performs a preset number of trainings to obtain the auditory model.

4. The whole vehicle hearing method according to claim 1, characterized in that: The sound to be trained includes at least: Noise, stone hitting, vehicle passing over bumps, car horns, human speech and human non-verbal sounds.

5. The whole vehicle hearing method according to claim 4, characterized in that: The step of marking the training sound to obtain the marked sound and its corresponding auditory label includes: Marking the noise to obtain a marked sound as at least one of wind noise, road noise, abnormal noise and motor noise, wherein the auditory labels corresponding to the wind noise and the road noise are passive actions of the vehicle, and the auditory labels corresponding to the abnormal noise and the motor noise are active actions of the vehicle; and / or The stone hitting sound is marked, and the marked sound is at least one of the following: stone hitting glass sound, stone hitting sheet metal sound, and stone hitting guard plate sound, and the corresponding auditory labels are all vehicle passive actions; and / or The sound of a vehicle passing over a bump is marked, and the marked sound is at least one of the sound of a vehicle pressing down a manhole cover, the sound of a vehicle passing over a deep pit, and the sound of a vehicle passing over a speed bump, and the corresponding auditory labels are all passive actions of the vehicle.

6. The whole vehicle hearing method according to claim 1, characterized in that: The collecting of environmental sound and performing sound processing on it includes: Filtering the collected ambient sound, and intercepting it according to preset time segments after filtering to obtain multiple ambient sound segments; The audio features in the multiple environmental sound segments are extracted through the Mel spectrum to obtain a spectrum image, and the spectrum image is the environmental sound after sound processing.

7. A vehicle hearing system, characterized in that: include: An acquisition unit, at least used to acquire the sound to be trained; A labeling unit, at least used to label the training sound to obtain the labeled sound and its corresponding auditory label; A training unit, at least used to train the marked sound and the corresponding auditory label in a preset model to obtain an auditory model; A collection unit, at least used to collect environmental sounds and perform sound processing on them; The generating unit is at least used to recognize the environmental sound after the sound processing based on the auditory model to generate an auditory result.

8. The vehicle hearing system according to claim 7, characterized in that: The acquisition unit is connected to the cloud server and includes a vehicle-mounted microphone and a vehicle-mounted MCU; The on-board microphone collects ambient sound as a sensor for collecting ambient sound, and the on-board MCU collects ambient sound as a sound collector for collecting ambient sound. The on-board MCU transmits the collected ambient sound to the cloud server.

9. An electronic device, characterized in that: include: at least one processor; And at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the steps of the whole vehicle auditory method as described in any one of claims 1-6.

10. A readable storage medium storing a computer program, characterized in that: The computer program is used by a processor to execute the steps of the whole vehicle auditory method as described in any one of claims 1-6.

Images