Automobile environment sound enhancement method and device, electronic equipment and storage medium

Abstract

The invention discloses an automobile environment sound enhancement method and device, electronic equipment and a storage medium, which are used for solving the technical problems of low automobile external environment sound recognition precision and poor environment adaptability in related technologies. The method comprises the following steps: acquiring audio data outside a vehicle and video data outside the vehicle; scene recognition is carried out according to the vehicle exterior audio data and the vehicle exterior video data, and the current environment sound type and the current driving environment are obtained; extracting visual information from the video data outside the vehicle, and extracting audio information from the audio data outside the vehicle; carrying out multi-modal data fusion based on comprehensive confidence in combination with the visual information and the audio information to obtain an effective scene judgment result; and based on the current driving environment, the current environment sound type and the effective scene judgment result, the external environment sound is enhanced through a sound unvarnished transmission enhancement strategy. Therefore, environmental sound recognition accuracy can be improved based on multi-modal data fusion, environmental sound enhancement is realized in combination with a sound unvarnished transmission enhancement strategy, and environmental adaptability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile intelligent cockpits, and in particular to a method, device, electronic equipment and storage medium for enhancing automobile ambient sound. Background Art

[0002] With the development of modern automobile intelligence, the sound insulation performance of vehicles is constantly improving, and the noise inside the car is significantly reduced. However, during the driving process, the driver may find it difficult to perceive the changes in the external environment because it is too quiet, such as missing important information such as alarms and horns. And driving in a very quiet environment for a long time can easily make people feel tired and bored. In order to improve the safety of driving and riding, and at the same time improve the auditory experience of passengers in the car, the current main method is to enhance or optimize the sound environment inside and outside the car through In-Car Ambient Sound Enhancement.

[0003] At present, most of the car ambient sound enhancement is achieved by using a multi-microphone combination system. Although the multi-microphone combination system can suppress background noise (such as wind noise, tire noise, engine noise of other vehicles, etc.) to a certain extent, the noise suppression effect in complex environments (such as high-speed driving, multiple vehicles in parallel, busy traffic, etc.) is limited, and it cannot provide high-quality audio signals, and the environmental adaptability is poor. And because it relies on audio signals for sound source positioning, in complex environments, it is difficult to accurately identify and locate the target sound source by audio signals alone, and the accuracy of ambient sound recognition is low, resulting in poor capture of effective scene sounds (such as emergency brake sounds, alarm sounds, etc.), affecting the performance of the system. Summary of the invention

[0004] The present invention provides a method, device, electronic device and storage medium for enhancing automobile ambient sound, which are used to solve or partially solve the technical problems of low recognition accuracy and poor environmental adaptability of automobile external ambient sound in related technologies.

[0005] The present invention provides a method for enhancing automobile ambient sound, the method comprising:

[0006] Collecting audio data and video data outside the vehicle;

[0007] Perform scene recognition based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment;

[0008] extracting visual information from the off-vehicle video data, and extracting audio information from the off-vehicle audio data;

[0009] Combining the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result;

[0010] Based on the current driving environment, the current ambient sound type and the effective scene judgment result, the ambient sound outside the vehicle is enhanced through a sound transmission enhancement strategy.

[0011] Optionally, performing scene recognition according to the external audio data and the external video data to obtain the current ambient sound type and the current driving environment includes:

[0012] Inputting the external audio data and the external video data into a pre-trained multimodal neural network; the multimodal neural network includes a driving environment classification model and an ambient sound classification model;

[0013] Extracting audio features from the external audio data using the ambient sound classification model, and classifying the ambient sound according to the audio features to identify the current ambient sound type of the vehicle;

[0014] The driving environment classification model is used to extract surrounding environment features from the off-vehicle video data, and the driving environment is classified according to the surrounding environment features to identify the current driving environment of the vehicle.

[0015] Optionally, extracting visual information from the off-vehicle video data includes:

[0016] Detecting and locating all faces in the off-vehicle video data using a face detection algorithm, and tracking the detected faces in real time based on a face tracking algorithm, and generating face detection information;

[0017] Based on the face detection information, head posture estimation is performed to obtain direction judgment information;

[0018] Extracting facial key point information from the face detection information, locating a lip region according to the facial key point information, extracting visual features of the lip region, and performing lip movement detection based on the visual features to obtain lip movement detection information;

[0019] Generate face detection confidence according to the face detection information, generate direction judgment confidence according to the direction judgment information, and generate lip movement detection confidence according to the lip movement detection information;

[0020] A weighted fusion calculation is performed based on the face detection confidence, the direction judgment confidence and the lip movement detection confidence to obtain visual information.

[0021] Optionally, extracting audio information from the external audio data includes:

[0022] Performing sound source positioning on the external audio data to determine the direction information of the sound source;

[0023] Performing voice recognition on the external audio data to obtain voice recognition information;

[0024] Generating a sound source direction confidence according to the sound source direction information, and generating a speech recognition confidence according to the speech recognition information;

[0025] A weighted fusion calculation is performed based on the sound source direction confidence and the speech recognition confidence to obtain audio information.

[0026] Optionally, the combining the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain a valid scene judgment result includes:

[0027] Performing multimodal data fusion according to the visual information and the audio information to obtain multimodal fusion information;

[0028] Solving the comprehensive confidence of the multimodal fusion information;

[0029] When the comprehensive confidence is greater than a preset confidence threshold, the external vehicle ambient sound scene is determined to be a valid scene;

[0030] When the comprehensive confidence is less than or equal to a preset confidence threshold, the exterior ambient sound scene is determined to be an invalid scene.

[0031] Optionally, the enhancing the ambient sound outside the vehicle through a sound transmission enhancement strategy based on the current driving environment, the current ambient sound type and the effective scene judgment result includes:

[0032] Dynamically adjust the transparent volume of the external ambient sound according to the current ambient sound type and its sound intensity;

[0033] When the effective scene judgment result indicates that the external ambient sound scene is a effective scene, the transparent volume of the external ambient sound is dynamically adjusted in combination with the current driving environment, and / or the audio of the external ambient sound is enhanced.

[0034] Optionally, a user display interface is provided in front of the cockpit of the vehicle, and the user display interface includes an ambient sound type display area and a sound intensity adjustment button; the ambient sound type display area displays several adjustable ambient sound types; the vehicle ambient sound enhancement method further includes:

[0035] When an ambient sound type selection operation for the ambient sound type display area is detected, determining a target adjustable ambient sound type from the plurality of adjustable ambient sound types;

[0036] In response to the adjustment operation on the sound intensity adjustment button, the transparent volume of the external vehicle ambient sound corresponding to the target adjustment ambient sound type is adjusted.

[0037] The present invention also provides a vehicle ambient sound enhancement device, comprising:

[0038] A data acquisition unit, used for acquiring audio data and video data outside the vehicle;

[0039] A scene recognition unit, configured to perform scene recognition based on the external audio data and the external video data to obtain a current ambient sound type and a current driving environment;

[0040] An information extraction unit, used to extract visual information from the off-vehicle video data, and extract audio information from the off-vehicle audio data;

[0041] A multimodal data fusion unit, used to combine the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result;

[0042] The vehicle exterior ambient sound enhancement unit is used to enhance the vehicle exterior ambient sound through a sound transmission enhancement strategy based on the current driving environment, the current ambient sound type and the effective scene judgment result.

[0043] The present invention also provides an electronic device, the device comprising a processor and a memory:

[0044] The memory is used to store program code and transmit the program code to the processor;

[0045] The processor is used to execute the automobile ambient sound enhancement method as described above according to the instructions in the program code.

[0046] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes, and the program codes are used to execute the automobile ambient sound enhancement method as described in any one of the above items.

[0047] It can be seen from the above technical solutions that the present invention has the following advantages:

[0048] A method for enhancing automobile ambient sound based on multimodal data fusion is provided. First, the audio data and video data outside the vehicle are collected; then, scene recognition is performed based on the audio data and video data outside the vehicle to obtain the current ambient sound type and the current driving environment; visual information is extracted from the video data outside the vehicle, and audio information is extracted from the audio data outside the vehicle; then, multimodal data fusion based on comprehensive confidence is performed in combination with the visual information and the audio information to obtain an effective scene judgment result; finally, based on the current driving environment, the current ambient sound type and the effective scene judgment result, the ambient sound outside the vehicle is enhanced through the sound transmission enhancement strategy. Therefore, by introducing multimodal fusion processing technology and using multimodal data fusion based on comprehensive confidence for effective scene judgment, the accuracy of ambient sound recognition can be improved. On this basis, the dynamic adjustment of ambient sound enhancement is realized in combination with the sound transmission enhancement strategy, which improves environmental adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 It is a structural schematic diagram of a car ambient sound enhancement system;

[0051] Figure 2 A flowchart of a method for enhancing vehicle ambient sound;

[0052] Figure 3 A display schematic diagram of a user display interface;

[0053] Figure 4 The overall process diagram of a method for enhancing the ambient sound of an automobile is shown;

[0054] Figure 5 The present invention is a structural block diagram of a vehicle ambient sound enhancement device. DETAILED DESCRIPTION

[0055] The embodiments of the present invention provide a method, device, electronic device and storage medium for enhancing automobile ambient sound, which are used to solve or partially solve the technical problems of low accuracy in identifying automobile external ambient sound and poor environmental adaptability in related technologies.

[0056] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] As an example, currently, most car ambient sound enhancement is achieved by using a multi-microphone combination system. Although the multi-microphone combination system can suppress background noise (such as wind noise, tire noise, engine noise of other vehicles, etc.) to a certain extent, the noise suppression effect in complex environments (such as high-speed driving, multiple vehicles in parallel, busy traffic, etc.) is limited, and it cannot provide high-quality audio signals, and the environmental adaptability is poor. And because it relies on audio signals for sound source positioning, in complex environments, it is difficult to accurately identify and locate the target sound source by audio signals alone, and the accuracy of ambient sound recognition is low, resulting in poor capture of effective scene sounds (such as emergency brake sounds, alarm sounds, etc.), affecting the performance of the system.

[0058] After further analysis, the present invention finds that the current audio processing algorithm lacks dynamic adjustment capabilities and cannot optimize the audio processing strategy in real time according to environmental changes, resulting in large performance differences in different environments. In addition, the currently used multi-microphone combination system only relies on audio information and cannot fully utilize the visual information provided by the camera. As a result, there are not only deficiencies in multimodal information fusion, which limits the robustness and accuracy of the system (especially in sound recognition and capture in complex environments), but also difficulty in accurately identifying complex driving scenarios, resulting in low flexibility and intelligence of the adjustment strategy. In addition, users can only perceive the environment through sound, lack visual intuition, and have limited user experience.

[0059] Therefore, one of the core invention points of the embodiment of the present invention is: in view of the shortcomings of the current technology, in order to solve or partially solve the problems of insufficient background noise suppression in the external environment of the car, low sound source positioning accuracy, poor environmental adaptability, and insufficient multimodal information fusion, a car environmental sound enhancement system based on multimodal fusion technology and its application method are provided. By introducing multimodal fusion perception technology, combining audio and video information, and using a variety of deep learning algorithms, the image data captured by the camera and the audio data captured by the microphone are jointly processed to generate richer multimodal features. Thereby, comprehensive recognition of various environmental sounds is achieved, the quality of audio signals and the capture effect of effective scene sounds are improved, and the accuracy of sound source positioning is improved. At the same time, through the sound transmission enhancement strategy, the audio processing algorithm is automatically adjusted according to environmental changes, and the clarity of effective scene sounds is enhanced by the algorithm while suppressing background noise. Thereby, effective scene sounds can be clearly captured even in complex environments. Especially in high-speed driving or complex traffic environments, it can continuously provide high-quality audio capture and enhancement effects, improve driving safety and user experience.

[0060] Reference Figure 1 , showing a schematic structural diagram of a car ambient sound enhancement system provided by an embodiment of the present invention.

[0061] Combination Figure 1 The automobile ambient sound enhancement system 100 may mainly include a central processing unit (CPU) 101, an external vehicle sound pickup module 102, a camera module 103, an audio playback module 104 and a user display interface 105 which are respectively connected to the central processing unit 101 for communication.

[0062] The vehicle exterior sound pickup module 102 may be a microphone (Microphone, Mic), which is mainly used to collect ambient sound outside the vehicle and transmit it as vehicle exterior audio data to the central processing unit 101 .

[0063] The camera module 103 may be a camera (Cam), which is mainly used to collect visual related information outside the vehicle and transmit it to the central processing unit 101 as outside vehicle video data.

[0064] In practical applications, multimodal fusion can be achieved by capturing visual information around the vehicle through a camera and combining it with the audio signal from a microphone. This fusion can more accurately identify and locate the sound source of an effective external scene and improve the accuracy of sound capture. For the detection and recognition of people outside the vehicle, the facial expressions, lip movements, and body language of the person speaking outside the vehicle can be captured by a camera. Combined with the audio signal captured by the microphone, the behavior and emotions of the person outside the vehicle can be more comprehensively analyzed to improve the accuracy of speech recognition. Especially for noisy environments or scenes with heavy accents, the performance of the speech recognition system can be significantly improved.

[0065] At the same time, the camera can also monitor environmental changes in real time, such as light, background noise, etc. Combined with the audio signal of the microphone, the audio processing strategy can be dynamically adjusted to achieve more effective noise suppression and effective sound source enhancement, improve the environmental adaptability of the system, and make the system more stable and reliable in different environments.

[0066] By combining microphones and cameras, the system solves the problems of noise interference and inaccurate speech recognition that exist in a single microphone, significantly improving audio quality and user experience. Even if the audio signal is interfered with, the video information can provide additional context to help the system better understand the audio content.

[0067] The central processing unit 101 receives data from the external sound pickup module 102 and the camera module 103, processes and analyzes the acquired data based on ambient sound and visual information using a deep learning algorithm, and executes a sound transmission enhancement strategy based on the data processing results.

[0068] The audio playing module 104 may be a speaker, which is mainly used to play ambient sound.

[0069] The user display interface 105 may be a touch screen, which mainly provides the user with the function of autonomously setting and controlling the sound transmission enhancement strategy. That is, the user is allowed to set the intensity and type of sound transmission according to personal needs. For example, the user may choose to transmit emergency vehicle alarms, children's voices, external commands and communication sounds, etc. according to actual needs.

[0070] Based on the above-mentioned car ambient sound enhancement system, refer to Figure 2 , shows a flowchart of a method for enhancing automobile ambient sound provided by an embodiment of the present invention, which may specifically include the following steps:

[0071] Step 201, collecting audio data and video data outside the vehicle;

[0072] Specifically, the ambient sound data outside the vehicle (external audio data) can be collected in real time through a microphone, and the external video data outside the vehicle can be collected in real time through a camera for subsequent processing.

[0073] Step 202, performing scene recognition according to the external audio data and the external video data to obtain the current ambient sound type and the current driving environment;

[0074] The multimodal data fusion in the embodiment of the present invention can actually be understood as fusing the external audio data collected by the microphone and the external video data collected by the camera to achieve comprehensive judgment.

[0075] In this step, scene recognition is mainly performed through deep learning models (such as multimodal neural networks), which can process audio and video data simultaneously and perform corresponding feature extraction operations and classification and recognition operations.

[0076] In a specific implementation, scene recognition is performed based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment, which can be:

[0077] First, the external audio data and the external video data are input into a pre-trained multimodal neural network, wherein the multimodal neural network may mainly include a driving environment classification model and an ambient sound classification model.

[0078] The driving environment classification model may be a deep learning model pre-trained based on driving environment classification, such as a convolutional neural network (CNN). The ambient sound classification model may be an audio classification model based on ambient sound classification, such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0079] Then, the ambient sound classification model is used to extract audio features (such as voiceprint features, spectrum features, etc.) from the external audio data, and the ambient sound is classified according to the audio features to identify the current ambient sound type of the vehicle. For example, a pre-trained audio classification model (such as a convolutional neural network CNN or a recurrent neural network RNN) is used to classify the external audio data and identify the current ambient sound type. The ambient sound types may include pedestrian sounds, traffic sign sounds, vehicle alarm sounds, background noise, and other types.

[0080] At the same time, the driving environment classification model is used to extract surrounding environment features (such as surrounding environment features, building features, roadside object features, etc.) from the vehicle's external video data, and the driving environment is classified according to the surrounding environment features to identify the vehicle's current driving environment. For example, the pre-trained driving environment classification model is used to classify the vehicle's external video data to identify the current driving environment. The driving environment can include urban, highway, rural, and other types.

[0081] Step 203, extracting visual information from the vehicle-external video data, and extracting audio information from the vehicle-external audio data;

[0082] Effective judgment of the ambient sound scene outside the vehicle requires the combination of visual information analysis and audio information analysis. This process can be seen as combining video data and audio data to make correlation feature judgments. Among them, correlation feature judgment can be understood as: when it is necessary to judge whether audio data is useful information, it is necessary to combine video data to make a judgment.

[0083] For example, if voice recognition determines that there is a voice in the audio data, but based on the video data, it is determined that the speaker is not facing the vehicle, then the external sound scene (specifically pointing to the voice information) is invalid. For another example, if voice recognition determines that there is an alarm in the audio data, but based on the video data analysis, there are no police cars, fire trucks, or ambulances around the vehicle, then the external sound scene (specifically pointing to the alarm) is also invalid.

[0084] Regarding the visual information analysis process, in some embodiments, the process of extracting visual information from the vehicle exterior video data includes the following sub-steps S01 to S05:

[0085] Step S01: Detect and locate all faces in the video data outside the vehicle using a face detection algorithm, and simultaneously track the detected faces in real time based on a face tracking algorithm, and generate face detection information;

[0086] Step S01 mainly implements face detection and tracking. Specifically, a face detection algorithm, such as Multi-Task Cascaded Convolutional Networks (MTCNN), can be used to detect all faces in the video, and a tracking algorithm, such as KCF (Kernelized Correlation Filters), can be used to track the detected faces in real time.

[0087] Among them, the multi-task cascade convolutional neural network MTCNN is an efficient face detection algorithm for face detection and facial feature point location. It mainly consists of three stages: PNet (Proposal Network), RNet (Refine Network) and ONet (Output Network). Each stage has a specific task to gradually improve the accuracy and efficiency of detection.

[0088] The main task of PNet is to generate candidate boxes. That is, to preliminarily detect areas that may contain faces. Its main workflow is: first input the image ; Then extract through the convolution layer features; then use an activation function (such as ReLU) to activate the features; finally, generate candidate boxes through a fully connected layer.

[0089] The calculation formula corresponding to the PNet processing flow is as follows:

[0090]

[0091] in, Represents the generated candidate box; represents the weight matrix; represents the bias vector; Represents an activation function (such as ReLU).

[0092] The main task of RNet is to filter candidate frames and remove most non-face areas. Its main workflow is: first, the candidate frames generated by PNet are As input; then through the convolution layer Further extract features; then use activation functions (such as ReLU) to activate the features; finally, generate more accurate candidate boxes through the fully connected layer.

[0093] The calculation formula corresponding to the RNet processing flow is as follows:

[0094]

[0095] in, Represents the candidate box after filtering; represents the weight matrix; represents the bias vector; Represents an activation function (such as ReLU).

[0096] The main task of ONet is to accurately locate the face and generate the key points of the face. Its main workflow is: first, the candidate box generated by RNet As input; then through the convolution layer Further extract features; then use activation functions (such as ReLU) to activate the features; finally, generate the final face frame and key points through the fully connected layer.

[0097] The calculation formula corresponding to the ONet processing flow is as follows:

[0098]

[0099] in, Represents the final face frame and key points; represents the weight matrix; represents the bias vector; Represents an activation function (such as ReLU).

[0100] Based on the above introduction, in the specific implementation, for face detection, firstly, continuous frames are extracted from the video data outside the vehicle; then, each frame image is Apply PNet to generate preliminary candidate boxes ; Then the candidate box generated by PNet Apply RNet to filter out more accurate candidate boxes ;Finally, the candidate box generated by RNet Apply ONet to accurately locate faces and generate key points .

[0101] Furthermore, for face tracking, we first perform face detection on the first frame image through the multi-task cascade convolutional neural network MTCNN to generate the initial face frame and key points; then use the tracking algorithm (such as KCF, DeepSort) to perform real-time continuous tracking on the detected face to obtain face detection information.

[0102] Step S02: Estimating the head posture based on the face detection information to obtain direction determination information;

[0103] Head posture estimation can be achieved in two main ways. One is to use face detection information as model input and use a posture estimation model such as OpenPose (a real-time multi-person posture estimation algorithm) to analyze the posture of the face to determine whether the face is facing the vehicle. The other is to determine whether the face is facing the vehicle by analyzing the head posture (such as the rotation angle of the head) based on face detection information. Through head posture estimation, the corresponding direction judgment information can be obtained.

[0104] Step S03: extracting facial key point information from the face detection information, locating the lip area according to the facial key point information, extracting visual features of the lip area, and performing lip movement detection based on the visual features to obtain lip movement detection information;

[0105] Lip movement detection can be seen as determining whether a person is speaking by analyzing facial expressions (such as evaluating the degree of mouth opening and closing based on facial key point information).

[0106] Specifically, first, facial key point information can be extracted from face detection information to locate the lip area; then, a pre-trained convolutional neural network model, such as VGG (Visual Geometry Group) or ResNet (Residual Network), is used to extract the visual features of the lip area; the extracted visual features are then input into a recurrent neural network RNN ​​that is pre-trained based on visual feature recognition to model the timing information of lip movement; finally, the output of the recurrent neural network RNN ​​passes through a fully connected layer and an activation layer (softmax function) to obtain the lip movement recognition result (whether there is speech), that is, the lip movement detection information.

[0107] Step S04: generating face detection confidence according to face detection information, generating direction judgment confidence according to direction judgment information, and generating lip movement detection confidence according to lip movement detection information;

[0108] In the embodiment of the present invention, it is necessary to convert the above-mentioned several judgment results into corresponding confidence levels before conducting subsequent comprehensive analysis. Therefore, the confidence level of each judgment result can be calculated. Specifically, the face detection confidence level is generated based on the face detection information and is defined as . The direction judgment confidence is generated according to the direction judgment information and defined as The lip movement detection confidence is generated based on the lip movement detection information and is defined as .

[0109] Step S05: Perform weighted fusion calculation based on the face detection confidence, direction judgment confidence and lip movement detection confidence to obtain visual information.

[0110] Visual Information The calculation formula is as follows:

[0111]

[0112] in, , , Both represent weights, which are used to balance the importance of different information in the visual information.

[0113] Regarding the audio information analysis process, in some embodiments, the implementation process of extracting audio information from the external audio data includes the following sub-steps S11 to S14:

[0114] Step S11: localizing the sound source of the external audio data to determine the direction of the sound source;

[0115] Microphone array technology (such as DOA (Direction of Arrival) algorithm) can be used to locate the sound source of the audio data outside the vehicle, determine the direction of the sound source, and determine whether the direction of the sound source is consistent with the direction of the vehicle to obtain the sound source direction information. If the direction of the sound source is consistent with the direction of the vehicle, the sound source direction information is considered valid.

[0116] Step S12: performing voice recognition on the external audio data to obtain voice recognition information;

[0117] The speech content in the audio data outside the vehicle can be recognized by a speech recognition algorithm (such as DeepSpeech), and the recognized speech content can be judged to be relevant to the vehicle (such as "stop" or "give way"), and speech recognition information can be generated. If the recognized speech content is relevant to the vehicle, the speech recognition information is considered valid.

[0118] Step S13: generating a sound source direction confidence according to the sound source direction information, and generating a speech recognition confidence according to the speech recognition information;

[0119] Specifically, the sound source direction confidence is generated according to the sound source direction information and is defined as . The speech recognition confidence is generated based on the speech recognition information and defined as .

[0120] Step S14: Perform weighted fusion calculation based on the sound source direction confidence and the speech recognition confidence to obtain audio information.

[0121] Audio Information The calculation formula is as follows:

[0122]

[0123] in, , Both represent weights, which are used to balance the importance of different information in the audio information.

[0124] Step 204, combining the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain a valid scene judgment result;

[0125] In some embodiments, the implementation process of combining visual information and audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result includes the following sub-steps S21 to S23:

[0126] Step S21: performing multimodal data fusion according to the visual information and the audio information to obtain multimodal fusion information;

[0127] Multimodal fusion information The calculation formula is as follows:

[0128]

[0129] For the convenience of calculation, and They are all normalized values, ranging from [0, 1]. , Both represent weights, which are used to balance the importance of visual and audio information.

[0130] The weights involved in the embodiments of the present invention can be adjusted according to actual applications. It is understandable that the present invention is not limited to this.

[0131] Step S22: solving the comprehensive confidence of the multimodal fusion information;

[0132] Comprehensive confidence It can be calculated by the following formula:

[0133]

[0134] in, Represents the activation function, where the Sigmoid function is usually used to map the output value to [0, 1]. Assume that the input of the Sigmoid function is , then the Sigmoid function can be defined as:

[0135]

[0136] The comprehensive confidence It can also be expressed as:

[0137]

[0138] Step S23: When the comprehensive confidence is greater than a preset confidence threshold, the external vehicle ambient sound scene is judged to be a valid scene; when the comprehensive confidence is less than or equal to the preset confidence threshold, the external vehicle ambient sound scene is judged to be an invalid scene.

[0139] To further improve the accuracy of judgment, a confidence threshold is set , only when the comprehensive confidence Exceeds confidence threshold Only when the external ambient sound scene is determined to be a valid scene.

[0140] The specific judgment formula for valid scenarios is as follows:

[0141]

[0142] For example, it is assumed that the face detection confidence is obtained by calculating , Direction judgment confidence , lip movement detection confidence , Confidence of sound source direction , Speech recognition confidence .

[0143] The weights are set to , , , , , , ; The confidence threshold is set to .

[0144] First, calculate the visual information :

[0145]

[0146] Calculate audio information at the same time :

[0147]

[0148] Then calculate the multimodal fusion information :

[0149]

[0150] Then calculate the comprehensive confidence :

[0151]

[0152] Finally, the threshold comparison judgment:

[0153] because , determine that the current external ambient sound scene is a valid scene.

[0154] Therefore, through the above calculation process, multimodal fusion and comprehensive confidence calculation can be achieved, thereby improving the accuracy and reliability of the intelligent cockpit ambient sound enhancement system.

[0155] Step 205: Based on the current driving environment, the current ambient sound type and the effective scene judgment result, the ambient sound outside the vehicle is enhanced through a sound transmission enhancement strategy.

[0156] In some embodiments, based on the current driving environment, the current ambient sound type and the effective scene judgment result, the implementation process of enhancing the external ambient sound through the sound transmission enhancement strategy can be: dynamically adjusting the transmission volume of the external ambient sound according to the current ambient sound type and its sound intensity; when the effective scene judgment result shows that the external ambient sound scene is a valid scene, dynamically adjusting the transmission volume of the external ambient sound in combination with the current driving environment, and / or performing audio enhancement on the external ambient sound.

[0157] The audio transmission enhancement strategy is mainly set based on dynamic volume adjustment and volume enhancement.

[0158] From the perspective of effective scenarios, the sound transmission enhancement strategy can mainly achieve the enhancement of the external environmental sound of effective scenarios (such as effective speaking sounds facing the vehicle, or alarm sounds), and reduce the external environmental sound of invalid scenarios (such as background noise). For example, assuming that someone is detected speaking in the direction of the vehicle, and the microphone array determines that the sound source comes from this direction, then the external environmental sound scene is judged to be a valid scene, and the sound picked up by the microphone is enhanced. For another example, assuming that an alarm sound is detected, and based on the video data analysis, it is determined that there is an alarm vehicle, then the external environmental sound scene is judged to be a valid scene, and the sound picked up by the microphone is enhanced.

[0159] In different driving environments, the adjustment of the effective scene external environmental sound will also be different. For example, when the driving environment is an urban environment, the noise around the vehicle is large, and when the driving environment is a rural environment, the noise around the vehicle is small and relatively quiet compared to the urban environment. The two have different ways of enhancing the intensity of the effective scene external environmental sound and reducing the intensity of background noise. For example, when the driving environment is an urban environment, assuming that someone is detected directing traffic, the intensity of the sound picked up by the enhanced microphone can be appropriately increased to better obtain the traffic command voice content. If necessary, the background noise can also be appropriately reduced.

[0160] From the perspective of the ambient sound itself, the sound transmission enhancement strategy can dynamically adjust the transmission volume based on the ambient sound type and its volume (intensity), to avoid the problem of poor experience for people in the car due to excessive volume of the ambient sound outside the car, and to avoid the situation where people in the car miss the sound due to too low volume of the ambient sound outside the car.

[0161] Dynamic volume adjustment can dynamically adjust the transparent volume according to the type and intensity of the ambient sound. For example, the volume of emergency vehicle alarms can be increased, while the volume of background noise can be reduced.

[0162] Audio enhancement is mainly achieved through audio enhancement algorithms (such as noise suppression and echo cancellation) to improve the sound quality of effective scenes. Specifically, deep learning models (such as deep neural networks (DNN)) or traditional signal processing methods (such as spectral subtraction) can be used to suppress background noise to achieve the purpose of noise suppression. Echo cancellation algorithms (such as acoustic echo cancellation (AEC)) can also be used to eliminate the impact of the sound played by the speakers in the car on the audio collected by the microphone, achieving the purpose of echo cancellation and enhancing the ambient sound outside the car.

[0163] In some embodiments, in combination with the above related introduction, the automobile ambient sound enhancement system may further include a user display interface. That is, a user display interface is provided in front of the cockpit in the vehicle. For example, the display of the user display interface is shown as follows: Figure 3 As shown. Combined Figure 3 The user display interface may mainly include an ambient sound type display area and a sound intensity adjustment button.

[0164] The ambient sound type display area displays several types of adjustable ambient sound, such as alarm sounds (corresponding to the ambient sounds emitted by police cars, ambulances, guide vehicles, etc.), command sounds (corresponding to traffic commands, reversing commands, and other command vehicle sounds), and external human voices (corresponding to human voices around the vehicle, etc.).

[0165] The sound intensity adjustment button can be further divided into several gears such as normal adjustment, medium adjustment, and enhanced adjustment. The sound intensity can be adjusted by clicking the sound intensity adjustment button with your finger. You can also adjust the sound intensity by pressing and holding the sound intensity adjustment button with your finger, sliding your finger up and down to pull the volume progress bar, and so on.

[0166] In a specific implementation, when the system detects an ambient sound type selection operation for the ambient sound type display area, a target adjustable ambient sound type is determined from several adjustable ambient sound types; in response to an adjustment operation for the sound intensity adjustment button, the transparent volume of the ambient sound outside the vehicle corresponding to the target adjustable ambient sound type is adjusted, thereby realizing user-oriented autonomous control of the sound transparent volume.

[0167] In an embodiment of the present invention, an application method of an automotive ambient sound enhancement system based on multimodal fusion technology is provided. By introducing multimodal fusion perception technology, combining audio and video information, and utilizing a variety of deep learning algorithms, the image data captured by the camera and the audio data captured by the microphone are jointly processed to generate richer multimodal features. This enables comprehensive recognition of a variety of ambient sounds, improves the quality of audio signals and the capture effect of effective scene sounds, and improves the accuracy of sound source positioning. At the same time, through the sound transmission enhancement strategy, the audio processing algorithm is automatically adjusted according to environmental changes, and the clarity of effective scene sounds is enhanced through the algorithm while suppressing background noise. Thus, effective scene sounds can be clearly captured even in complex environments. Especially in high-speed driving or complex traffic environments, it can continuously provide high-quality audio capture and enhancement effects, improving driving safety and user experience.

[0168] For better explanation, refer to Figure 4 , showing a schematic diagram of the overall process of a method for enhancing ambient sound in a car provided by an embodiment of the present invention. It should be noted that this embodiment only briefly describes the general process of enhancing ambient sound in a car, and the specific implementation process of each step can be understood by referring to the relevant content in the aforementioned embodiment, which will not be described here. It can be understood that the present invention does not limit this.

[0169] Step 401: Collecting audio data and video data outside the vehicle;

[0170] Step 402: Input the external audio data and the external video data into a pre-trained multimodal neural network for scene recognition to obtain the current ambient sound type and the current driving environment;

[0171] Step 403: extracting visual information from the vehicle exterior video data based on face detection and tracking and lip movement detection combined with confidence weighted fusion calculation;

[0172] Step 404: extracting audio information from the external audio data based on sound source localization and speech recognition combined with confidence weighted fusion calculation;

[0173] Step 405: Combining visual information and audio information to perform multimodal data fusion based on comprehensive confidence to obtain effective scene judgment results;

[0174] Step 406: Based on the current driving environment, the current ambient sound type and the effective scene judgment result, the ambient sound outside the vehicle is enhanced through the sound transmission enhancement strategy;

[0175] Step 407: when an ambient sound type selection operation on the user display interface is detected, a target adjustable ambient sound type is determined from a plurality of adjustable ambient sound types;

[0176] Step 408: In response to the adjustment operation on the sound intensity adjustment button, the transparent volume of the vehicle exterior ambient sound corresponding to the target adjustment ambient sound type is adjusted.

[0177] Reference Figure 5 , shows a structural block diagram of a vehicle ambient sound enhancement device provided by an embodiment of the present invention, which may specifically include:

[0178] The data acquisition unit 501 is used to acquire audio data and video data outside the vehicle;

[0179] A scene recognition unit 502 is used to perform scene recognition based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment;

[0180] An information extraction unit 503 is used to extract visual information from the vehicle-external video data and extract audio information from the vehicle-external audio data;

[0181] A multimodal data fusion unit 504, configured to combine the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain a valid scene judgment result;

[0182] The vehicle exterior ambient sound enhancement unit 505 is used to enhance the vehicle exterior ambient sound through a sound transmission enhancement strategy based on the current driving environment, the current ambient sound type and the effective scene judgment result.

[0183] In an optional embodiment, the scene recognition unit 502 includes:

[0184] A data input unit, used to input the external audio data and the external video data into a pre-trained multimodal neural network; the multimodal neural network includes a driving environment classification model and an ambient sound classification model;

[0185] An ambient sound classification unit, configured to extract audio features from the external audio data of the vehicle by using the ambient sound classification model, and classify the ambient sound according to the audio features to identify the current ambient sound type of the vehicle;

[0186] The driving environment classification unit is used to extract surrounding environment features from the off-vehicle video data through the driving environment classification model, and classify the driving environment according to the surrounding environment features to identify the current driving environment of the vehicle.

[0187] In an optional embodiment, the information extraction unit 503 includes:

[0188] A face detection unit, used to detect and locate all faces in the off-vehicle video data using a face detection algorithm, and simultaneously track the detected faces in real time based on a face tracking algorithm, and generate face detection information;

[0189] A head posture estimation unit, used to perform head posture estimation based on the face detection information to obtain direction determination information;

[0190] A lip movement detection unit, configured to extract facial key point information from the face detection information, locate a lip region according to the facial key point information, extract visual features of the lip region, and perform lip movement detection based on the visual features to obtain lip movement detection information;

[0191] A vision-related confidence generation unit, configured to generate a face detection confidence according to the face detection information, generate a direction judgment confidence according to the direction judgment information, and generate a lip movement detection confidence according to the lip movement detection information;

[0192] The visual information generating unit is used to perform weighted fusion calculation based on the face detection confidence, the direction judgment confidence and the lip movement detection confidence to obtain visual information.

[0193] In an optional embodiment, the information extraction unit 503 includes:

[0194] A sound source localization unit, used to localize the sound source of the external audio data and determine the direction information of the sound source;

[0195] A speech recognition unit, used for performing speech recognition on the external audio data to obtain speech recognition information;

[0196] An audio-related confidence generating unit, configured to generate a sound source direction confidence according to the sound source direction information, and generate a speech recognition confidence according to the speech recognition information;

[0197] The audio information generating unit is used to perform weighted fusion calculation based on the sound source direction confidence and the speech recognition confidence to obtain audio information.

[0198] In an optional embodiment, the multimodal data fusion unit 504 includes:

[0199] A multimodal fusion information calculation unit, used for performing multimodal data fusion according to the visual information and the audio information to obtain multimodal fusion information;

[0200] A comprehensive confidence solving unit, used for solving the comprehensive confidence of the multimodal fusion information;

[0201] A valid scene judgment unit, used for judging that the external vehicle ambient sound scene is a valid scene when the comprehensive confidence is greater than a preset confidence threshold;

[0202] The invalid scene judgment unit is used to judge that the external environmental sound scene is an invalid scene when the comprehensive confidence is less than or equal to a preset confidence threshold.

[0203] In an optional embodiment, the vehicle exterior ambient sound enhancement unit 505 is specifically used for:

[0204] Dynamically adjust the transparent volume of the external ambient sound according to the current ambient sound type and its sound intensity;

[0205] When the effective scene judgment result indicates that the external ambient sound scene is a effective scene, the transparent volume of the external ambient sound is dynamically adjusted in combination with the current driving environment, and / or the audio of the external ambient sound is enhanced.

[0206] In an optional embodiment, a user display interface is provided in front of the cockpit of the vehicle, and the user display interface includes an ambient sound type display area and a sound intensity adjustment button; the ambient sound type display area displays several adjustable ambient sound types; the vehicle ambient sound enhancement device also includes:

[0207] a target adjustment ambient sound type determining unit, configured to determine a target adjustment ambient sound type from the plurality of adjustable ambient sound types when an ambient sound type selection operation for the ambient sound type display area is detected;

[0208] The transparent volume adjustment unit is used to adjust the transparent volume of the external vehicle ambient sound corresponding to the target adjustment ambient sound type in response to the adjustment operation of the sound intensity adjustment button.

[0209] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the aforementioned method embodiment.

[0210] An embodiment of the present invention further provides an electronic device, the device comprising a processor and a memory:

[0211] The memory is used to store the program code and transmit the program code to the processor;

[0212] The processor is used to execute the automobile ambient sound enhancement method of any embodiment of the present invention according to the instructions in the program code.

[0213] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the method for enhancing the ambient sound of an automobile according to any embodiment of the present invention.

[0214] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0215] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0216] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0218] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0219] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for enhancing automobile ambient sound, characterized in that: include: Collecting audio data and video data outside the vehicle; Perform scene recognition based on the external audio data and the external video data to obtain the current ambient sound type and the current driving environment; extracting visual information from the off-vehicle video data, and extracting audio information from the off-vehicle audio data; Combining the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result; Based on the current driving environment, the current ambient sound type and the effective scene judgment result, the ambient sound outside the vehicle is enhanced through a sound transmission enhancement strategy.

2. The method for enhancing automobile ambient sound according to claim 1, characterized in that: The performing scene recognition according to the external audio data and the external video data to obtain the current ambient sound type and the current driving environment includes: Inputting the external audio data and the external video data into a pre-trained multimodal neural network; the multimodal neural network includes a driving environment classification model and an ambient sound classification model; Extracting audio features from the external audio data using the ambient sound classification model, and classifying the ambient sound according to the audio features to identify the current ambient sound type of the vehicle; The driving environment classification model is used to extract surrounding environment features from the off-vehicle video data, and the driving environment is classified according to the surrounding environment features to identify the current driving environment of the vehicle.

3. The method for enhancing automobile ambient sound according to claim 1, characterized in that: The extracting visual information from the off-vehicle video data comprises: Detecting and locating all faces in the off-vehicle video data using a face detection algorithm, and tracking the detected faces in real time based on a face tracking algorithm, and generating face detection information; Based on the face detection information, head posture estimation is performed to obtain direction judgment information; Extracting facial key point information from the face detection information, locating the lip area according to the facial key point information, extracting visual features of the lip area, and performing lip movement detection based on the visual features to obtain lip movement detection information; Generate face detection confidence according to the face detection information, generate direction judgment confidence according to the direction judgment information, and generate lip movement detection confidence according to the lip movement detection information; A weighted fusion calculation is performed based on the face detection confidence, the direction judgment confidence and the lip movement detection confidence to obtain visual information.

4. The method for enhancing the automobile ambient sound according to claim 3, characterized in that: The extracting audio information from the external audio data comprises: Performing sound source positioning on the external audio data to determine the direction information of the sound source; Performing voice recognition on the external audio data to obtain voice recognition information; Generating a sound source direction confidence according to the sound source direction information, and generating a speech recognition confidence according to the speech recognition information; A weighted fusion calculation is performed based on the sound source direction confidence and the speech recognition confidence to obtain audio information.

5. The method for enhancing automobile ambient sound according to claim 4, characterized in that: The combining the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result includes: Performing multimodal data fusion according to the visual information and the audio information to obtain multimodal fusion information; Solving the comprehensive confidence of the multimodal fusion information; When the comprehensive confidence is greater than a preset confidence threshold, the external vehicle ambient sound scene is determined to be a valid scene; When the comprehensive confidence is less than or equal to a preset confidence threshold, the exterior ambient sound scene is determined to be an invalid scene.

6. The method for enhancing automobile ambient sound according to claim 1, characterized in that: The enhancing the ambient sound outside the vehicle through a sound transmission enhancement strategy based on the current driving environment, the current ambient sound type and the effective scene judgment result includes: Dynamically adjust the transparent volume of the external ambient sound according to the current ambient sound type and its sound intensity; When the effective scene judgment result indicates that the external ambient sound scene is a effective scene, the transparent volume of the external ambient sound is dynamically adjusted in combination with the current driving environment, and / or the audio of the external ambient sound is enhanced.

7. The method for enhancing the automobile ambient sound according to any one of claims 1 to 6, characterized in that: A user display interface is provided in front of the cockpit in the vehicle, and the user display interface includes an ambient sound type display area and a sound intensity adjustment button; the ambient sound type display area displays several adjustable ambient sound types; the vehicle ambient sound enhancement method also includes: When an ambient sound type selection operation for the ambient sound type display area is detected, determining a target adjustable ambient sound type from the plurality of adjustable ambient sound types; In response to the adjustment operation on the sound intensity adjustment button, the transparent volume of the external vehicle ambient sound corresponding to the target adjustment ambient sound type is adjusted.

8. A car ambient sound enhancement device, characterized in that: include: A data acquisition unit, used for acquiring audio data and video data outside the vehicle; A scene recognition unit, configured to perform scene recognition based on the external audio data and the external video data to obtain a current ambient sound type and a current driving environment; An information extraction unit, used to extract visual information from the off-vehicle video data, and extract audio information from the off-vehicle audio data; A multimodal data fusion unit, used to combine the visual information and the audio information to perform multimodal data fusion based on comprehensive confidence to obtain an effective scene judgment result; The vehicle exterior ambient sound enhancement unit is used to enhance the vehicle exterior ambient sound through a sound transmission enhancement strategy based on the current driving environment, the current ambient sound type and the effective scene judgment result.

9. An electronic device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the automobile ambient sound enhancement method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the automobile ambient sound enhancement method according to any one of claims 1 to 7.

Images