Map-guided scene-aware few-shot acoustic modeling method, device, equipment and medium
By using a map-guided scene perception method to extract and fuse semantic features, the limitations of acoustic learning performance in existing technologies are solved, enabling more accurate RIR prediction and spatial information understanding.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2024-12-11
- Publication Date
- 2026-07-28
AI Technical Summary
Existing technologies cannot fully understand and utilize limited sample data, which limits the performance of acoustic learning, especially in the fields of VR and AR, where they cannot effectively understand and predict the acoustic features of unknown scenes.
By using a map-guided scene perception method, semantic features of image data are extracted, and scene features are aligned and fused using depth mapping and pose information. Combined with Gaussian distance weighted calculation, the target RIR is predicted.
It significantly improves the model's prediction accuracy for RIR and enhances the understanding of spatial information about building structure, layout, and object semantics, especially in feature-sparse regions.
Smart Images
Figure CN119942543B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of virtual modeling, and more particularly to a map-guided scene-aware few-sample acoustic modeling method, apparatus, device, and medium. Background Technology
[0002] Few-shot acoustic modeling aims to predict the acoustic characteristics of unknown scenes, unknown regions, and unknown poses using a limited number of environmental audiovisual observations. It has broad application value in VR, AR, and other fields. The core objective of this task is to understand the entire acoustic space and subsequently predict the room impulse response function (RIR) at any specified location, providing powerful spatial awareness capabilities for downstream tasks.
[0003] Existing technical solutions cannot fully understand and utilize the limited amount of example data provided, thus limiting the performance of acoustic learning. They either perceive the environment from raw RGB-D images and predict the target RIR in an end-to-end manner, or they predict the target RIR directly by implicitly modeling local features through Nerf. The former weakens pose information through sinusoidal position encoding and ignores the connections between each position, while the latter cannot utilize complete visual information, especially semantic information, limiting the model's ability to understand space and thus failing to generalize well to unknown scenes. Summary of the Invention
[0004] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a map-guided scene perception few-sample acoustic modeling method, device, equipment and medium.
[0005] The first technical solution adopted in this invention is:
[0006] A map-guided scene-aware few-shot acoustic modeling method includes the following steps:
[0007] Acquire image data, extract visual features from the image data, and extract scene semantic features from the visual features;
[0008] Based on the extracted scene semantic features, the scene features obtained from different poses are aligned and fused to obtain a scene feature map;
[0009] Obtain the query coordinates, obtain the query features based on the query coordinates and the scene feature map, and obtain the target RIR based on the query features.
[0010] Furthermore, the extraction of scene semantic features from visual features includes:
[0011] The U-Net model, pre-trained on a semantic segmentation task, is used to extract acoustically relevant semantic features. The output of the U-Net model is a final layer of latent features that fuses low-level and high-level features, thus providing a comprehensive representation of environmental details.
[0012] v = f V (V)
[0013] In the formula, V represents the RGB image in the image data, and f V denoted as the pre-trained U-Net feature extraction model, and v represents the extracted pixel-level acoustic semantic features that are consistent with the length and width of the input image.
[0014] Furthermore, the step of aligning and fusing scene features obtained from different poses based on the extracted scene semantic features to obtain a scene feature map includes:
[0015] Using depth mapping techniques and depth maps in image data, semantic features are mapped onto their corresponding top-down views:
[0016] T=D P (v,D)
[0017] In the formula, T represents the top view corresponding to the feature, and D P denoted as depth mapping, D represents the depth map in the observation information, and v represents the extracted pixel-level acoustic semantic features that are consistent with the input image in length and width;
[0018] By utilizing the attitude information from each observation point, the top view is rotated and translated to achieve a unified world coordinate system.
[0019] T ′ =T R (T,P)
[0020] In the formula, T ′ T represents the top view corresponding to each observed feature after alignment. R This indicates a rotation and translation operation; P represents the attitude information (including coordinates and orientation information) corresponding to the observation point; and T represents the top view corresponding to each observation feature.
[0021] The aligned top-down views are then merged to retain the results with the most significant features (i.e., the largest feature values), thus obtaining a feature map of the entire scene:
[0022] M = F(T) i )
[0023] In the formula, M represents the merged top view, F represents the merging operation, and T represents the merging operation. ′ This represents the top view corresponding to each observed feature after alignment.
[0024] Further, the steps of obtaining query coordinates, obtaining query features based on the query coordinates and the scene feature map, and obtaining the target RIR based on the query features include:
[0025] Obtain the query coordinates and calculate the Gaussian distance between the query coordinates and each point on the scene feature map;
[0026] Using the Gaussian distance between each point as weight, the feature values are weighted and summed to obtain the query features under the query coordinates;
[0027] The query features, combined with audio features, are input into the autoencoder to predict the RIR of the query coordinates.
[0028] Furthermore, the formula for calculating the Gaussian distance is:
[0029]
[0030] In the formula, G represents the Gaussian distance, (x,y) represents the coordinates of each point on the scene feature map, and (x,y) represents the coordinates of each point on the scene feature map. q ,y q ) represents the query coordinates, and σ represents a trainable parameter.
[0031] Furthermore, the calculation formula for the query feature is as follows:
[0032]
[0033] In the formula, v q This represents the query features at the query coordinates, k represents the number of points on the scene map, G represents the Gaussian distance, (x i ,y i (x) represents the coordinates of the i-th point on the scene feature map. q ,y q ) represents the coordinates to be queried, M i This represents the feature of the i-th point on the scene feature map.
[0034] The second technical solution adopted in this invention is:
[0035] A map-guided scene-aware few-shot acoustic modeling device, comprising:
[0036] The semantic feature extraction module is used to acquire image data, extract visual features from the image data, and extract scene semantic features from the visual features;
[0037] The scene feature alignment module is used to align and fuse scene features obtained from different poses based on the extracted scene semantic features to obtain a scene feature map.
[0038] The prediction module is used to obtain query coordinates, obtain query features based on query coordinates and scene feature map, and obtain the target RIR based on query features.
[0039] The third technical solution adopted in this invention is:
[0040] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a map-guided scene-aware few-sample acoustic modeling method as described above.
[0041] The fourth technical solution adopted in this invention is:
[0042] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a map-guided scene-aware few-sample acoustic modeling method as described above.
[0043] The fifth technical solution adopted in this invention is:
[0044] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the method described above.
[0045] The beneficial effects of this invention are as follows: This invention proposes a map-guided scene perception scheme. By extracting acoustically relevant semantic features from different visual observations and using depth mapping for feature alignment and fusion, it obtains the feature information of the entire scene, enhancing the model's understanding of spatial information such as building structure, building layout, and object semantics. Furthermore, this invention utilizes a Gaussian distance-weighted calculation method to fully leverage the features of each point in the scene feature map, significantly improving the model's prediction accuracy for RIR. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 This is a flowchart of the steps of a map-guided scene perception few-sample acoustic modeling method in an embodiment of the present invention;
[0048] Figure 2 This is a schematic diagram of a map-guided scene perception few-sample acoustic modeling algorithm in an embodiment of the present invention. Detailed Implementation
[0049] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0050] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.
[0051] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.
[0052] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.
[0053] Example 1
[0054] like Figure 1 and Figure 2 As shown, this embodiment provides a map-guided scene-aware few-shot acoustic modeling method. In the field of virtual modeling such as AR and VR, this method solves the problem of weak alignment ability of existing audiovisual feature representations and enhances the model's understanding of spatial information such as building structure, building layout, and object semantics. The method specifically includes the following steps:
[0055] S1. Acquire image data, extract visual features from the image data, and extract scene semantic features from the visual features.
[0056] As one implementation method, a U-Net model pre-trained on a semantic segmentation task is used to extract acoustically relevant semantic features. The model outputs a final layer of latent features that fuses low-level and high-level features, thereby providing a comprehensive representation of environmental details.
[0057] v = f V (V)
[0058] Where V represents the RGB image in visual observation, f V denoted as the pre-trained U-Net feature extraction model, and v represents the extracted pixel-level acoustic semantic features that are consistent with the length and width of the input image.
[0059] S2. Based on the extracted scene semantic features, align and fuse the scene features obtained from different poses to obtain a scene feature map.
[0060] See Figure 2 In some embodiments, step S2 specifically includes the following steps:
[0061] S2-1: Using depth mapping techniques and depth maps from visual observation, semantic features are mapped onto a top-down view:
[0062] T=D P (v,D)
[0063] Where T represents the top view corresponding to the feature, and D P denoted as depth mapping, D represents the depth map in the observation information, and v represents the extracted pixel-level acoustic semantic features that are consistent with the input image in length and width.
[0064] S2-2: Using the attitude information of each observation point, perform rotation and translation operations on the top view to give it a unified world coordinate system.
[0065] T ′ =T R (T,P)
[0066] Among them, T ′ T represents a top view aligned with world coordinates. R This indicates a rotation and translation operation. P represents the attitude information (including coordinates and orientation information) corresponding to the observation point, and T represents the top view corresponding to each observation feature.
[0067] S2-3: Merge the aligned top views, retaining the results with the most significant features (i.e., the largest feature values), thus obtaining the feature map of the entire scene:
[0068] m=F(T ′ )
[0069] Where m represents the merged top view, F represents the merging operation, and T... ′ This represents the top view corresponding to each observed feature after alignment.
[0070] S3. Obtain the query coordinates, obtain the query features based on the query coordinates and the scene feature map, and obtain the target RIR based on the query features.
[0071] See Figure 2 In some embodiments, step S3 specifically includes the following steps:
[0072] S3-1: Calculate the Gaussian distance between the query coordinates and points on the scene feature map:
[0073]
[0074] Where G represents the Gaussian distance, (x,y) represents the coordinates of each point on the scene feature map, and (x,y) represents the coordinates of each point on the scene feature map. q ,y q ) represents the query coordinates, and σ represents a trainable parameter.
[0075] S3-2: Using the Gaussian distance values of each point as weights, perform a weighted sum of the feature values to obtain the query features under the query coordinates:
[0076]
[0077] Among them, v q This represents the query features at the query coordinates, k represents the number of points on the scene map, G represents the Gaussian distance, (x i ,y i (x) represents the coordinates of the i-th point on the scene feature map. q ,y q ) represents the coordinates to be queried, M i This represents the feature of the i-th point on the scene feature map.
[0078] S3-3: Input the query features and audio features into the autoencoder to predict the RIR of the query coordinates.
[0079] In summary, existing methods still cannot fully understand and utilize the limited amount of example data provided, thus limiting the performance of acoustic learning. They either perceive the environment from raw RGB-D images and predict the target RIR in an end-to-end manner, or they directly predict the target RIR by implicitly modeling local features through Nerf. The former weakens pose information through sinusoidal position encoding and ignores the connections between each position, while the latter fails to utilize complete visual information, especially semantic information, limiting the model's ability to understand space and thus failing to generalize well to unknown scenes. In contrast, this invention innovatively proposes a map-guided scene perception scheme. By extracting acoustically relevant semantic features from different visual observations and using depth mapping for feature alignment and fusion, it obtains the feature information of the entire scene, enhancing the model's understanding of spatial information such as building structure, building layout, and object semantics. Furthermore, this invention utilizes a Gaussian distance weighted calculation method to fully leverage the features of each point in the scene feature map, significantly improving the model's prediction accuracy for RIR.
[0080] In summary, by employing feature extraction fusion and sampling methods, this invention significantly improves RIR prediction accuracy in few-shot acoustic modeling tasks, particularly in feature-sparse regions. The model can simultaneously utilize features from the entire scene map, resulting in more accurate and comprehensive predictions.
[0081] Example 2
[0082] This embodiment provides a map-guided scene-aware few-shot acoustic modeling device, including:
[0083] The semantic feature extraction module is used to acquire image data, extract visual features from the image data, and extract scene semantic features from the visual features;
[0084] The scene feature alignment module is used to align and fuse scene features obtained from different poses based on the extracted scene semantic features to obtain a scene feature map.
[0085] The prediction module is used to obtain query coordinates, obtain query features based on query coordinates and scene feature map, and obtain the target RIR based on query features.
[0086] Since this device is a map-guided scene-aware few-sample acoustic modeling device according to an embodiment of the present invention, and the principle of the device in solving the problem is similar to that of the method, the implementation of this device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.
[0087] Example 3
[0088] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 1 This paper presents a map-guided method for scene-aware, few-sample acoustic modeling.
[0089] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.
[0090] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.
[0091] Since this electronic device is an electronic device corresponding to a map-guided scene perception few-sample acoustic modeling method according to an embodiment of the present invention, and the principle of solving the problem by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.
[0092] Example 4
[0093] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 1 This paper presents a map-guided method for scene-aware, few-sample acoustic modeling.
[0094] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.
[0095] Since this storage medium is the storage medium corresponding to a map-guided scene perception few-sample acoustic modeling method in this embodiment of the invention, and the principle of this storage medium in solving the problem is similar to that of this method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.
[0096] Example 5
[0097] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a map-guided scene-aware few-shot acoustic modeling method according to various exemplary embodiments of this application as described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.
[0098] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0099] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0100] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.
Claims
1. A map-guided scene-aware few-shot acoustic modeling method, characterized in that, Includes the following steps: Acquire image data, extract visual features from the image data, and extract scene semantic features from the visual features; Based on the extracted scene semantic features, the scene features obtained from different poses and obtained through depth mapping are aligned and fused to obtain a scene feature map; Obtain the query coordinates, obtain the query features based on the query coordinates and the scene feature map, and obtain the target RIR based on the query features; The process of obtaining query coordinates, obtaining query features based on the query coordinates and the scene feature map, and obtaining the target RIR based on the query features includes: Obtain the query coordinates and calculate the Gaussian distance between the query coordinates and each point on the scene feature map; Using the Gaussian distance of each point as the weight, the features of each point on the scene feature map are weighted and summed to obtain the query features under the query coordinates; The query features, combined with audio features, are input into the autoencoder to predict the RIR of the query coordinates; The formula for calculating the Gaussian distance is: In the formula, Represents Gaussian distance, This represents the coordinates of each point on the scene feature map. Indicates the coordinates to be queried. This represents a trainable parameter; The formula for calculating the query features is: In the formula, Indicates the query features under the query coordinates. This indicates the number of points on the scene map. Represents Gaussian distance, The first scene feature map represents the first scene feature map. The coordinates of the points Indicates the coordinates to be queried. The first scene feature map represents the first scene feature map. Features of each point.
2. The map-guided scene-aware few-shot acoustic modeling method according to claim 1, characterized in that, The extraction of scene semantic features from visual features includes: The U-Net model, pre-trained on a semantic segmentation task, is used to extract acoustically relevant semantic features. The output of the U-Net model is a final layer of latent features that fuses low-level and high-level features, thus providing a comprehensive representation of environmental details. In the formula, Represents the RGB image in the image data. This represents the pre-trained U-Net feature extraction model. This represents the extracted pixel-level acoustic semantic features that maintain the same length and width as the input image.
3. The map-guided scene-aware few-shot acoustic modeling method according to claim 1, characterized in that, The step of aligning and fusing scene features obtained from different poses and obtained through depth mapping based on the extracted scene semantic features to obtain a scene feature map includes: Using depth mapping techniques and depth maps in image data, semantic features are mapped onto their corresponding top-down views: In the formula, This represents the top view corresponding to the feature. Represents depth mapping, This represents the depth map in the observation information. This represents the extracted pixel-level acoustic semantic features that maintain the same length and width as the input image; By utilizing the attitude information from each observation point, the top view is rotated and translated to achieve a unified world coordinate system. In the formula, This represents the top view corresponding to each observed feature after alignment. This indicates a rotation and translation operation. This represents the attitude information corresponding to the observation point. This shows the top view corresponding to each observed feature; The aligned top views are then merged to retain the most significant features, resulting in a feature map of the entire scene: In the formula, This represents the merged top view. Indicates a fusion operation. This represents the top view corresponding to each observed feature after alignment.
4. A map-guided scene-aware few-shot acoustic modeling device, used to implement the method described in any one of claims 1 to 3, characterized in that, include: The semantic feature extraction module is used to acquire image data, extract visual features from the image data, and extract scene semantic features from the visual features; The scene feature alignment module is used to align and fuse scene features obtained from different poses and obtained through depth mapping based on the extracted scene semantic features to obtain a scene feature map. The prediction module is used to obtain query coordinates, obtain query features based on query coordinates and scene feature map, and obtain the target RIR based on query features.
5. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 3.
6. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1 to 3.