Real-time localization and visualization system for vehicle-mounted noise sources based on distributed microphone array

Abstract

This invention discloses a real-time vehicle noise source localization and visualization system based on a distributed microphone array. The system creation includes the following steps: 1) Automated generation of three-dimensional curved surfaces inside and outside the vehicle body: Based on collected RGBD data of the vehicle body and interior, referencing a multi-view geometric correspondence algorithm to generate a vehicle body curve model, combined with depth point clouds, and using surface generation, stitching, and optimization algorithms to generate a G2 continuous three-dimensional curved surface model template, further utilizing topological consistency rules to quickly generate the corresponding three-dimensional curved surface model from the new curve model. This invention utilizes a distributed microphone array for in-vehicle noise source identification; visualization of noise sources on the three-dimensional curved surfaces inside and outside the vehicle body; and a more intelligent in-vehicle system. The identification of in-vehicle noise sources helps drivers assess the in-vehicle noise situation and further resolve and handle faults based on the visualized location of abnormal noise sources.

Description

Technical Field

[0001] This invention relates to the fields of acoustics and computer imaging technology, and in particular to a real-time localization and visualization system for vehicle-mounted noise sources based on a distributed microphone array. Background Technology

[0002] Microphone array technology for sound source identification and localization is widely used in many fields. However, the enclosed environment of a vehicle is not suitable for integrated microphone arrays that occupy a large space. Therefore, this patent proposes to use a distributed microphone array placed at eight apex positions inside the vehicle to identify and locate in-vehicle noise sources. To better adapt to the current state of vehicle connectivity, a multi-image three-dimensional surface reconstruction technology for the interior and exterior of the vehicle body is proposed to further visualize the noise sources on the three-dimensional curved surfaces inside and outside the vehicle body.

[0003] For example, CN113689852A provides a vehicle-mounted voice control method and system based on sound source localization, which uses a time difference method to determine the location of the voice source inside the vehicle, addressing the competition between voice commands from drivers and passengers. CN115158197A provides a control system for in-vehicle intelligent cockpit entertainment based on sound source localization, characterized by including a detection module, a localization module, and a control module. However, both of the above patents focus on detecting in-vehicle voice commands. This patent attempts to identify and locate different noise sources inside the vehicle, especially abnormal noise sources, and visualizes the noise sources by combining the reconstruction of the three-dimensional curved surfaces inside and outside the vehicle body. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a real-time localization and visualization system for vehicle noise sources based on a distributed microphone array. This invention, based on a distributed microphone array, is beneficial to the further development of the intelligent and connected automotive industry.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A real-time vehicle noise source localization and visualization system based on a distributed microphone array is created through the following steps:

[0007] 1) Automated generation of 3D curved surfaces inside and outside the vehicle body: Based on the collected RGBD data of the vehicle body and interior, referencing the multi-view geometric correspondence algorithm for generating vehicle body curve models, and combining depth point clouds, use surface generation, stitching and optimization algorithms to generate G2 continuous 3D curved surface model templates. Further utilize topology consistency rules to quickly generate corresponding 3D curved surface models for new curve models.

[0008] 2) Noise feature identification based on deep learning: By placing microphones at different locations inside the vehicle to collect noise data at different locations, a database of noise sources under various operating conditions is created. Combined with deep learning, various types of vehicle body noise are classified and their features are analyzed.

[0009] 3) Localization of abnormal noise sources in the vehicle using a distributed microphone array: Establish an in-vehicle distributed microphone array, and place several microphones at the top corner of the interior space of the vehicle to collect real-time voice data in the vehicle. Using the concept of blind source separation, separate and locate the abnormal noise sources in the vehicle from the data acquired by the microphone array.

[0010] 4) Establishment of vehicle-mounted finite element three-dimensional mesh acoustic holographic model: Mesh division is performed based on the three-dimensional curved surfaces inside and outside the vehicle body, and mesh templates are generated. Finally, a three-dimensional mesh model can be generated based on the template for any given curve model. The sound field distribution of abnormal noise sources on the closed vehicle interior curved surface S is calculated based on the various noise sources. The sound field at any point in the curved surface S can be calculated based on Kirchhoff integrals.

[0011] 5) Build an intelligent audiovisual fusion cockpit system platform: Based on the three-dimensional mesh acoustic holographic model of the noise source inside the vehicle, develop a software system and build an intelligent audiovisual fusion cockpit system that can reproduce the sound of noise sources at different locations and visualize their location and energy.

[0012] Preferably, the three-dimensional surface model is composed of 64 cubic Bezier curves, and the three-dimensional surface model can be matched with different vehicle models.

[0013] Preferably, the noise data includes, but is not limited to, engine compartment noise, wind noise, and tire noise.

[0014] Preferably, the system includes noise source localization based on a distributed microphone array.

[0015] Preferably, the system includes the establishment of a three-dimensional mesh acoustic holographic model of the vehicle body, both inside and out.

[0016] Preferably, the system includes the construction of an in-vehicle noise visualization cockpit system.

[0017] Preferably, the system includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the vehicle-mounted noise source real-time localization and visualization system based on a distributed microphone array.

[0018] Preferably, the system includes a readable storage medium storing a computer program, which, when executed, enables a real-time location and visualization system for vehicle-mounted noise sources based on a distributed microphone array.

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] 1. Identification of in-vehicle noise sources using a distributed microphone array;

[0021] 2. Visual display of the noise source on the three-dimensional curved surfaces inside and outside the vehicle body;

[0022] 3. A more intelligent in-vehicle system, with the ability to identify in-vehicle noise sources, helps drivers assess the noise level inside the vehicle and further resolve and handle faults based on the visualized location of abnormal noise sources. Attached Figure Description

[0023] To illustrate the technical solutions in the embodiments of the present invention or the prior art more specifically and intuitively, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0024] Figure 1 This is a flowchart illustrating the steps of an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of a three-dimensional wireframe of the vehicle body's outer surface according to an embodiment of the present invention;

[0026] Figure 3 This is the geometric correspondence of multiple views of the vehicle body exterior surface in an embodiment of the present invention;

[0027] Figure 4 This is an illustration of the generation of a three-dimensional curved surface on the exterior of a vehicle body according to an embodiment of the present invention;

[0028] Figure 5 This is an illustration of the generation of a three-dimensional mesh on the exterior of the vehicle body according to an embodiment of the present invention;

[0029] Figure 6 This is a diagram illustrating the arrangement of a distributed microphone array according to an embodiment of the present invention;

[0030] Figure 7 This is a diagram of the channel acquisition experimental device according to an embodiment of the present invention;

[0031] Figure 8 This is a diagram of the oscilloscope channel interface according to an embodiment of the present invention;

[0032] Figure 9 This is a diagram of the acoustic signals acquired in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0034] Reference Figure 1A real-time vehicle noise source localization and visualization system based on a distributed microphone array. The system creation includes the following steps:

[0035] 1) Automated generation of 3D curved surfaces inside and outside the vehicle body: Based on the collected RGBD data of the vehicle body and interior, referencing the multi-view geometric correspondence algorithm for generating vehicle body curve models, and combining depth point clouds, use surface generation, stitching and optimization algorithms to generate G2 continuous 3D curved surface model templates. Further utilize topology consistency rules to quickly generate corresponding 3D curved surface models for new curve models.

[0036] a. The 3D wireframe model of the vehicle's outer surface is attached. Figure 2 As shown, the model consists of 64 cubic Bezier curves. The curve numbers and definitions of the specific three-dimensional curve model are shown in Table 1. Any car can be simply represented by this model. Therefore, there is topological consistency between different models based on the correspondence of lines.

[0037] Table 1. Curve Numbering and Definitions for the 3D Curve Model of an Automobile

[0038]

[0039] b. As attached Figure 3 As shown, by using the multi-view geometric correspondence method, it is possible to draw... Figure 3 The three-dimensional wireframe model of the 64 cubic Bezier curves shown;

[0040] c. As attached Figure 4 As shown, the template is divided into patches based on the geometric features of the vehicle body, using curves as boundaries and auxiliary lines. Each patch has 3 or 4 curves as boundaries. Then, the surface of each patch can be generated by sweeping the surface. Finally, the surfaces of all patches are integrated to obtain the surface model of the entire vehicle body.

[0041] d. Based on the consistency of corresponding lines between models, a corresponding 3D surface model can be quickly generated for a new curve model.

[0042] 2) Noise feature identification based on deep learning: By placing microphones in different locations inside the vehicle to collect noise data from different locations, such as the engine compartment, near the wheels, outside the roof, and outside the windows, noise from the engine compartment, wind noise, tire noise, etc. can be collected, creating a noise source database under various operating conditions. Combined with deep learning, various types of vehicle body noise are classified and their features are analyzed.

[0043] a. Use microphones to collect noise data from different locations, especially engine compartment and wind noise, and create a noise source database for various operating conditions;

[0044] b. Use end-to-end deep learning methods to classify and learn features of different types of noise.

[0045] 3) Localization of in-vehicle abnormal noise sources using a distributed microphone array: Establish an in-vehicle distributed microphone array, placing 8 microphones at the top corners of the vehicle's interior space, as shown in the attached diagram. Figure 6 The diagram showing the arrangement of the four microphones at the front is used to collect real-time voice data inside the vehicle. Utilizing the concept of blind source separation, the data acquired by the microphone array is used to separate and locate abnormal noise sources within the vehicle. The accuracy of the algorithm can be evaluated using specialized software and equipment, such as... Figure 7-9 As shown;

[0046] 4) Establishment of vehicle-mounted finite element three-dimensional mesh acoustic holographic model: Mesh division is performed based on the three-dimensional curved surfaces inside and outside the vehicle body, and mesh templates are generated. Finally, a three-dimensional mesh model can be generated based on the template for any given curve model. The sound field distribution of abnormal noise sources on the closed vehicle interior curved surface S is calculated based on the various noise sources. The sound field at any point in the curved surface S can be calculated based on Kirchhoff integrals.

[0047] 5) Build an intelligent audiovisual fusion cockpit system platform: Based on the three-dimensional mesh acoustic holographic model of the noise source inside the vehicle, develop a software system and build an intelligent audiovisual fusion cockpit system that can reproduce the sound of noise sources at different locations and visualize their location and energy.

[0048] The system includes a processor and a memory. The memory stores program instructions, and the processor calls the stored instructions in the memory to execute the real-time localization and visualization system for vehicle-mounted noise sources based on a distributed microphone array.

[0049] The system includes a readable storage medium on which a computer program is stored. When the computer program is executed, it realizes a real-time positioning and visualization system for vehicle-mounted noise sources based on a distributed microphone array.

[0050] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A real-time vehicle noise source localization and visualization system based on a distributed microphone array, characterized in that, The creation of this system includes the following steps: 1) Automated generation of 3D curved surfaces inside and outside the vehicle body: Based on the collected RGBD data of the vehicle body and interior, referencing the multi-view geometric correspondence algorithm for generating vehicle body curve models, and combining depth point clouds, use surface generation, stitching and optimization algorithms to generate G2 continuous 3D curved surface model templates. Further utilize topology consistency rules to quickly generate corresponding 3D curved surface models for new curve models. 2) Noise feature identification based on deep learning: By placing microphones at different locations inside the vehicle to collect noise data at different locations, a database of noise sources under various operating conditions is created. Combined with deep learning, various types of vehicle body noise are classified and their features are analyzed. 3) Localization of abnormal noise sources in the vehicle using a distributed microphone array: Establish an in-vehicle distributed microphone array, and place several microphones at the top corner of the interior space of the vehicle to collect real-time voice data in the vehicle. Using the concept of blind source separation, separate and locate the abnormal noise sources in the vehicle from the data acquired by the microphone array. 4) Establishment of vehicle-mounted finite element three-dimensional mesh acoustic holographic model: Mesh division is performed based on the three-dimensional curved surfaces inside and outside the vehicle body, and a mesh template is generated. Finally, a three-dimensional mesh model can be generated based on the template for any given curve model. The sound field distribution of abnormal noise sources on the closed vehicle interior curved surface S is calculated based on the various noise sources separated, and the sound field at any point in the curved surface S is calculated based on Kirchhoff integral. 5) Build an intelligent audiovisual fusion cockpit system platform: Based on the three-dimensional mesh acoustic holographic model of the noise source in the vehicle, develop a software system and build an intelligent audiovisual fusion cockpit system that can reproduce the sound of noise sources at different locations and visualize their location and energy.

2. The vehicle-mounted noise source real-time localization and visualization system based on a distributed microphone array according to claim 1, characterized in that, The three-dimensional surface model consists of 64 cubic Bezier curves, and the three-dimensional surface model can be matched with different vehicle models.

3. The vehicle-mounted noise source real-time localization and visualization system based on a distributed microphone array according to claim 2, characterized in that, The noise data includes engine compartment noise, wind noise, and tire noise.

4. The vehicle-mounted noise source real-time localization and visualization system based on a distributed microphone array according to claim 3, characterized in that, The system includes noise source localization based on a distributed microphone array.

5. The vehicle-mounted noise source real-time localization and visualization system based on a distributed microphone array according to claim 4, characterized in that, The system includes the creation of a three-dimensional mesh acoustic holographic model of the vehicle body, both inside and out.

6. The vehicle-mounted noise source real-time localization and visualization system based on a distributed microphone array according to claim 5, characterized in that, The system includes the construction of an in-vehicle noise visualization cockpit system.

7. The vehicle-mounted noise source real-time localization and visualization system based on a distributed microphone array according to claim 6, characterized in that, The system includes a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the vehicle noise source real-time localization and visualization system based on a distributed microphone array as described in claim 6.

8. The vehicle-mounted noise source real-time localization and visualization system based on a distributed microphone array according to claim 7, characterized in that, The system includes a readable storage medium on which a computer program is stored. When the computer program is executed, it implements the vehicle noise source real-time localization and visualization system based on a distributed microphone array as described in claim 7.

Images