Panoramic sound imaging method and device, electronic equipment and storage medium

Abstract

The invention provides a panoramic sound imaging method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a sound signal of a to-be-imaged area through a spherical microphone array; acquiring a frequency domain signal corresponding to the sound signal, performing spherical harmonic transformation on the frequency domain signal, and decomposing the sound signal into a spherical harmonic coefficient; extracting a spherical harmonic domain feature corresponding to the spherical harmonic coefficient, and generating a panoramic sound image of the to-be-imaged area according to the spherical harmonic domain feature and the panoramic space grid; and collecting a panoramic optical image of a to-be-imaged area through cameras distributed in a surrounding manner, and fusing the panoramic sound image with the panoramic optical image to generate target panoramic light and a sound image of the to-be-imaged area. The requirement for rapid and real-time panoramic sound imaging can be met, and meanwhile the spatial resolution of imaging is high.

Description

Technical Field

[0001] The present disclosure relates to the field of environmental perception technology, and more specifically, to a panoramic sound imaging method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of MEMS sensing technology, artificial intelligence technology and intelligent processors, the perception range of local vision and omnidirectional hearing can no longer meet people's perception needs of the real world. As the next generation of environmental perception technology, panoramic sound imaging can not only provide panoramic light and sound images at the same time, but also has the advantages of high resolution, no blind spots, miniaturization and multi-dimensional intelligent perception. Combined with artificial intelligence methods, this technology can lay the foundation for the next generation of smart sensors to achieve 360° all-round and multi-dimensional environmental perception.

[0003] At present, in the process of generating panoramic sound images, the sound images of segmented areas are mainly obtained through a planar microphone array, and then stitched together to generate panoramic sound images. However, in the actual dynamically changing acoustic environment, the image registration mechanism and segmented stitching technology used in this method cannot achieve single-frame fast panoramic sound imaging, which is difficult to meet the needs of fast and real-time panoramic sound imaging, and the spatial resolution of the imaging is low. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a method, device, electronic device and storage medium for panoramic sound imaging, which can meet the needs of fast and real-time panoramic sound imaging while having high spatial resolution of imaging.

[0005] The present disclosure provides a panoramic sound imaging method, including:

[0006] Acquiring acoustic signals of the area to be imaged by means of a spherical microphone array;

[0007] Acquire a frequency domain signal corresponding to the acoustic signal, perform spherical harmonic transformation on the frequency domain signal, and decompose the acoustic signal into spherical harmonic coefficients;

[0008] Extracting spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generating a panoramic sound image of the area to be imaged according to the spherical harmonic domain features and a panoramic spatial grid;

[0009] The panoramic optical image of the area to be imaged is collected by cameras distributed around it, and the panoramic sound image is fused with the panoramic optical image to generate a target panoramic light and sound image of the area to be imaged.

[0010] In an optional implementation, performing spherical harmonic transformation on the frequency domain signal to decompose the acoustic signal into spherical harmonic coefficients specifically includes:

[0011] Constructing a receiving signal model corresponding to the spherical microphone array according to the sound source sound pressure signal vector corresponding to the sound signal, the steering matrix and the noise vector corresponding to the microphone;

[0012] Determining a spherical harmonic representation of the steering matrix based on a spherical harmonic basis function matrix and a spherical harmonic basis function matrix;

[0013] The steering matrix in the received signal model is replaced by a spherical harmonic representation of the steering matrix to transform the sound source sound pressure signal vector into a spherical harmonic domain, and the spherical harmonic coefficients are determined.

[0014] In an optional implementation, the steering matrix represents the directional relationship from the sound source corresponding to the sound signal to the microphone;

[0015] The spherical harmonic basis function matrix is ​​used to represent the microphone position;

[0016] The spherical harmonic basis function matrix is ​​used to represent the direction of the sound source.

[0017] In an optional implementation, extracting the spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generating a panoramic sound image of the area to be imaged according to the spherical harmonic domain features and a panoramic space grid, specifically includes:

[0018] Inputting the spherical harmonic coefficients into a fully convolutional neural network, and extracting the spherical harmonic domain features of different orders through a convolution kernel in the frequency domain;

[0019] Mapping the spherical harmonic domain features onto the panoramic spatial grid to determine a space-frequency pseudo-spectrum;

[0020] An averaging process is performed on the space-frequency pseudo-spectrum in each sub-frequency band to generate the panoramic sound image.

[0021] In an optional implementation, in the spherical microphone array, a plurality of unidirectional vector microphones are uniformly distributed in a spherical array.

[0022] In an optional implementation manner, mapping the spherical harmonic domain features onto the panoramic space grid to determine the space-frequency pseudo spectrum specifically includes:

[0023] Rotating the spherical harmonic domain features output by the last layer in the fully convolutional neural network along the frequency axis to transpose the feature dimension and the spatial dimension;

[0024] After transposing the feature dimension and the spatial dimension, the space-frequency pseudo-spectrum is determined through processing of the convolution layer and the activation function.

[0025] The present disclosure also provides a panoramic sound imaging device, including:

[0026] An acoustic signal acquisition module, used for acquiring acoustic signals of the area to be imaged through a spherical microphone array;

[0027] A spherical harmonic coefficient determination module is used to obtain a frequency domain signal corresponding to the acoustic signal, perform a spherical harmonic transformation on the frequency domain signal, and decompose the acoustic signal into spherical harmonic coefficients;

[0028] An audio-visual generation module, used for extracting the spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generating a panoramic audio-visual image of the area to be imaged according to the spherical harmonic domain features and a panoramic spatial grid;

[0029] The light and sound image generation module is used to collect a panoramic optical image of the area to be imaged through cameras distributed around it, and fuse the panoramic sound image with the panoramic optical image to generate a target panoramic light and sound image of the area to be imaged.

[0030] The present disclosure also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the above-mentioned panoramic sound imaging method or steps in any possible implementation of the above-mentioned panoramic sound imaging method are performed.

[0031] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for panoramic sound imaging or steps in any possible implementation of the method for panoramic sound imaging are executed.

[0032] The present disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned panoramic sound imaging method, or steps in any possible implementation of the above-mentioned panoramic sound imaging method.

[0033] The embodiments of the present disclosure provide a panoramic sound imaging method, device, electronic device and storage medium, which collects the sound signal of the area to be imaged through a spherical microphone array; obtains the frequency domain signal corresponding to the sound signal, performs spherical harmonic transformation on the frequency domain signal, and decomposes the sound signal into spherical harmonic coefficients; extracts the spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generates a panoramic sound image of the area to be imaged based on the spherical harmonic domain features and a panoramic spatial grid; collects a panoramic optical image of the area to be imaged through surrounding cameras, and fuses the panoramic sound image with the panoramic optical image to generate a target panoramic light and sound image of the area to be imaged. It can meet the needs of fast and real-time panoramic sound imaging, and the spatial resolution of imaging is high.

[0034] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0036] Figure 1 A flow chart of a panoramic sound imaging method provided by an embodiment of the present disclosure is shown;

[0037] Figure 2 A flow chart of a method for generating panoramic sound and image provided by an embodiment of the present disclosure is shown;

[0038] Figure 3 A schematic diagram of a panoramic sound imaging device provided by an embodiment of the present disclosure is shown;

[0039] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0041] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0042] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0043] Research has found that, at present, in the process of generating panoramic sound images, the sound images of segmented areas are mainly obtained through a planar microphone array, and then stitched together to generate panoramic sound images. However, in the actual dynamically changing acoustic environment, the image registration mechanism and segmented stitching technology used in this method cannot achieve single-frame fast panoramic sound imaging, which is difficult to meet the needs of fast and real-time panoramic sound imaging, and the spatial resolution of the imaging is low.

[0044] Based on the above research, the present disclosure provides a panoramic sound imaging method, device, electronic device and storage medium, which collects the sound signal of the area to be imaged through a spherical microphone array; obtains the frequency domain signal corresponding to the sound signal, performs spherical harmonic transformation on the frequency domain signal, and decomposes the sound signal into spherical harmonic coefficients; extracts the spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generates a panoramic sound image of the area to be imaged according to the spherical harmonic domain features and a panoramic spatial grid; collects a panoramic optical image of the area to be imaged through surrounding distributed cameras, and fuses the panoramic sound image with the panoramic optical image to generate a target panoramic light and sound image of the area to be imaged. It can meet the needs of fast and real-time panoramic sound imaging, and the spatial resolution of imaging is high.

[0045] To facilitate understanding of this embodiment, a panoramic sound imaging method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the panoramic sound imaging method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the panoramic sound imaging method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0046] See also Figure 1 FIG. 1 is a flowchart of a panoramic sound imaging method provided by an embodiment of the present disclosure, wherein the method includes steps S101 to S104, wherein:

[0047] S101 , collecting acoustic signals of a region to be imaged by using a spherical microphone array.

[0048] In a specific implementation, the spherical microphone array is a system composed of multiple unidirectional vector microphones, which are evenly distributed on a rigid sphere to form a spherical array. Each unidirectional vector microphone has a fixed directivity and can sense sound waves in a specific direction.

[0049] Here, the more unidirectional vector microphones there are in the array, the higher the accuracy of spatial sampling. The coordinates of each unidirectional vector microphone can be expressed in a spherical coordinate system as:

[0050] r q =[rcos(θ q )sin(φ q ), rsin(θ q )sin(φ q ), rcos(φ q )] T

[0051] Among them, r q represents the coordinates of the unidirectional vector microphone; r represents the radius of the sphere; θ q represents the azimuth measured counterclockwise from the x-axis; φ q Represents the elevation angle measured downward from the z-axis.

[0052] Specifically, the acoustic signal received by the spherical microphone array is the superposition of multiple far-field sound sources:

[0053] r l =[rcos(θ l )sin(φ l ), rsin(θ l )sin(φ l ), rcos(φ l )] T

[0054] Among them, r l represents the coordinates of the sound source; r represents the radius of the sphere; θ q represents the azimuth measured counterclockwise from the x-axis; φ q represents the elevation angle measured downward from the z-axis; l=1,..., L represents the number of the sound source.

[0055] S102: Acquire a frequency domain signal corresponding to the acoustic signal, perform spherical harmonic transformation on the frequency domain signal, and decompose the acoustic signal into spherical harmonic coefficients.

[0056] In a specific implementation, when the sound wave reaches the spherical microphone array, each microphone receives a sound pressure signal at a specific moment. These sound signals are converted into frequency domain signals after short-time Fourier transform (STFT), which is expressed as a function of frequency.

[0057] Here, in order to further analyze and process the sound waves, the spherical harmonic transform (SHT) is used to convert the sound signal into the spherical harmonic domain. Spherical harmonics are the basis functions that describe the distribution of the sound field in three-dimensional space, which can effectively capture the intensity and direction information of sound waves in different directions. Through the spherical harmonic transform, the spatial signal can be deconstructed into spherical harmonic coefficients, which reflect the contribution of each spherical harmonic mode, thereby describing the directionality of the sound source in three-dimensional space.

[0058] Specifically, the determination method of the spherical harmonic coefficients can be achieved through the following steps 1 to 3:

[0059] Step 1: construct a receiving signal model corresponding to the spherical microphone array according to the sound source sound pressure signal vector corresponding to the sound signal, the steering matrix and the noise vector corresponding to the microphone.

[0060] Step 2: Determine the spherical harmonic representation of the steering matrix according to the spherical harmonic basis function matrix and the spherical harmonic basis function matrix.

[0061] Step 3: Use the spherical harmonic representation of the steering matrix to replace the steering matrix in the received signal model to convert the sound source sound pressure signal vector into the spherical harmonic domain and determine the spherical harmonic coefficients.

[0062] In a specific implementation, sound waves propagate through space and reach the unidirectional vector microphone of the spherical microphone array. Assuming that there are L far-field sound sources in space, and each microphone is also affected by sensor noise when receiving the signal from the sound source, the receiving signal model of the spherical microphone array can be expressed as:

[0063] p(k)=A(k)S(k)+z(k)

[0064] Where p(k) represents the sound pressure signal vector received by the spherical microphone array, p(k) = [p(k, r1), p(k, r2), ..., p(k, r Q )] T , p(k, r q ) is the acoustic signal received by the qth microphone; A(k) is the steering matrix, A(k) = [a(k, r1), a(k, r2), ..., a(k, r L )] T , a(k, rl ) represents the guiding vector of the lth sound source; S(k) is the sound pressure signal vector of the sound source, S(k) = [s1(k), s2(k), ..., s L (k)] T ,s l (k) represents; z(k) is the sensor noise vector.

[0065] Here, for each sound source l, its steering vector a(k, r l ) can be expressed as:

[0066]

[0067] Among them, k l is the wave vector of the sound wave and can be defined as:

[0068] k l =-k[cos(θ l )sin(φ l ), sin(θ l )sin(φ l ), cos(φ l )] T ,i 2 =-1

[0069] Here, according to the spherical harmonic transformation theory, the plane wave The decomposition can be approximated using truncated Nth-order spherical harmonics:

[0070]

[0071] Among them, b n (kr) is the mode strength for a unidirectional microphone, defined as b n (kr)=4πi n [j n (kr)-ij′ n (kr)]; are the spherical harmonic basis functions:

[0072]

[0073] in, is the associated Legendre polynomial, j n is a spherical Bessel function of the first kind.

[0074] By using the orthogonality principle of spherical harmonics, the steering matrix A(k) can be expressed as:

[0075] A(k)=Y(Ω)B(kr)Y H (Γ)

[0076] Where Y(Ω) is a Q×(N+1) 2 The matrix of , representing the spherical harmonic basis function matrix of the microphone position:

[0077] Y(Ω)=[y T (θ1, φ1), ..., y T (θ Q ,φ Q )] T

[0078]

[0079] Among them, Y H (Γ) is a L×(N+1) 2 The matrix represents the spherical harmonic basis function matrix in the direction of the sound source:

[0080] Y H (Γ)=[y T (θ1, φ1), ..., y T (θ L ,φ L )]

[0081] Among them, B(kr) is a (N+1) 2 ×(N+1) 2 The diagonal matrix of is defined as:

[0082] B(kr)=diag{b0(kr), b1(kr), b1(kr),..., b N (kr)}

[0083] Furthermore, by using the orthogonality of spherical harmonics, the sound pressure signal representation in the spherical harmonics domain can be obtained:

[0084] p nm (k) = B(kr)Y H (Γ)S(k)+Z nm (k)

[0085] where p nm (k) = Y H (Ω)p(k) is the sound pressure signal in the spherical harmonics domain, z nm (k) is the noise term.

[0086] It should be noted that for broadband sound sources, the short-time Fourier transform (STFT) can be used to smooth the selected time-frequency domain to achieve acoustic imaging under reverberation conditions. Therefore, the spherical harmonic domain model can be expressed as:

[0087]

[0088] In this way, through the spherical microphone array and spherical harmonic transform, the spatial sound source signal can be represented in the form of spherical harmonic coefficients, thereby obtaining the three-dimensional spatial information of the sound source, providing an effective mathematical model for panoramic sound imaging and spatial sound processing.

[0089] S103, extracting spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generating a panoramic sound image of the area to be imaged according to the spherical harmonic domain features and a panoramic space grid.

[0090] In specific implementation, the generation of panoramic sound images starts with the capture of sound signals. A spherical microphone array is used to capture all-round sound source information. The sound data captured by each microphone is filtered to obtain a sound pressure signal in the time-frequency domain. The sound pressure signal is decomposed in the spatial frequency domain through spherical harmonic transformation. The sound field captured by the spherical microphone array is represented as spherical harmonic coefficients to represent the intensity distribution of the sound source in different directions.

[0091] For details, see Figure 2 FIG. 1 is a flowchart of a method for generating panoramic sound and image provided by an embodiment of the present disclosure, the method comprising steps S1031 to S1033, wherein:

[0092] S1031. Input the spherical harmonic coefficients into a fully convolutional neural network, and extract the spherical harmonic domain features of different orders through a convolution kernel in the frequency domain.

[0093] S1032: Map the spherical harmonic domain features onto the panoramic space grid to determine a space-frequency pseudo-spectrum.

[0094] S1033: Perform averaging processing on the space-frequency pseudo-spectrum on each sub-frequency band to generate the panoramic sound image.

[0095] In a specific implementation, usually, reverberation noise (i.e., coherent sound source) does not shift in frequency as the spatial position changes, so the spatial information and frequency information can be considered to be relatively independent. In an embodiment of the present application, a fully convolutional network is used to process acoustic signals in the spherical harmonic domain to achieve high-resolution acoustic imaging and reconstruction of the spatial sound field.

[0096] Here, in order to extract frequency features, the fully convolutional neural network uses a convolution kernel size of (N+1) 2 / 2×1, which ensures the independence of frequency features and can only extract features at different spherical harmonic orders, avoiding the mixing of frequency features.

[0097] Among them, the fully convolutional neural network achieves the mapping between features and space by increasing the number of feature channels. Multiple convolutional layers are used in the network to extract different spatial features, and the dimensions of the features are enhanced through convolution operations in these layers to achieve spatial dimensionality increase.

[0098] Specifically, the convolutional layer in the fully convolutional neural network extracts spatial features through layer-by-layer convolution operations. The feature map after each convolution layer is nonlinearly transformed through the ReLU activation function to enhance the expressiveness of the model. In the last layer of the convolutional feature extraction layer, the number of neurons is ensured to be consistent with the number of panoramic spatial grids. Then, the feature dimension and the spatial dimension are transposed to prepare for further convolution operations.

[0099] It should be noted that the panoramic space grid is a discretized grid used to represent the position of the sound source in space in panoramic sound imaging. It divides the three-dimensional spherical space into multiple small cells, each of which represents a specific direction and spatial position, and is used to accurately describe the position of the sound source.

[0100] Here, the acoustic signal in the spherical harmonic domain is used as the input of the fully convolutional neural network. The convolution layer mainly extracts the spherical harmonic features of different orders through the convolution kernel in the frequency domain to ensure that the features of each spherical harmonic order are fully extracted in an independent channel. After multiple layers of convolution and feature extraction, the output layer of the network is mapped to a panoramic spatial grid. At this time, each grid point corresponds to a three-dimensional spatial direction.

[0101] Furthermore, the spherical harmonic domain features output by the last layer in the fully convolutional neural network are rotated along the frequency axis to transpose the feature dimension and the spatial dimension; after transposing the feature dimension and the spatial dimension, the space-frequency pseudo-spectrum is determined by processing the convolution layer and the activation function.

[0102] Here, in order to ensure the matching of features and spatial information, the number of channels of the final convolution output is set to be consistent with the number of grids in the panoramic space. The feature dimension and the spatial dimension are rotated by the transposition operation at the end of the convolution layer to ensure that the model output can be mapped to each grid in the three-dimensional space.

[0103] Furthermore, after transposing the feature and spatial dimensions, the model uses a 1×1 convolutional layer for feature fusion. This operation helps to combine features at different spherical harmonic orders and achieve further feature processing. The output feature tensor of the last convolutional layer is rotated 90° along the frequency axis to achieve the conversion of feature dimensions and spatial dimensions. Then, a 1×1 convolutional layer and a sigmoid activation function are used to generate a space×frequency pseudo-spectrum.

[0104] Finally, the sound images in each sub-band are averaged to obtain the final panoramic sound image. This process integrates the information in the space × frequency pseudo-spectrum into a panoramic sound image, providing a high-resolution spatial sound field image.

[0105] In this way, the panoramic sound imaging model effectively improves the resolution and real-time performance of sound imaging through the full convolution method. The model extracts the spatial and frequency features in the spherical harmonic domain by designing the convolution kernel size and the number of feature channels. Through the dimensional transformation and fusion of the features, a high-quality panoramic sound image is finally generated. This method not only solves the resolution and real-time problems in traditional sound imaging technology, but also makes full use of the independence of frequency information and spatial information.

[0106] S104, collecting a panoramic optical image of the area to be imaged by surrounding distributed cameras, and fusing the panoramic sound image with the panoramic optical image to generate a target panoramic light and sound image of the area to be imaged.

[0107] In the specific implementation, four cameras distributed in a surround are used to capture panoramic images. Each camera is located at a different angle and can cover different directions to ensure seamless capture of image information of the entire scene. These cameras are installed around the spherical microphone array to form a 360-degree field of view.

[0108] Here, when the cameras capture scene images, their image data is processed and stored. In order to achieve the generation of panoramic images, the images captured by the four cameras need to be stitched together. Image stitching uses image stitching technology to seamlessly connect multiple images by detecting the overlapping areas between images, using key point detection algorithms to find the correspondence between different images, and then stitching these images into a complete panoramic image through geometric transformation.

[0109] Finally, the panoramic image information based on image stitching technology is combined with the panoramic sound and image to generate the final panoramic light and sound image.

[0110] The embodiment of the present disclosure provides a panoramic sound imaging method, which collects the sound signal of the area to be imaged through a spherical microphone array; obtains the frequency domain signal corresponding to the sound signal, performs spherical harmonic transformation on the frequency domain signal, and decomposes the sound signal into spherical harmonic coefficients; extracts the spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generates a panoramic sound image of the area to be imaged based on the spherical harmonic domain features and a panoramic spatial grid; collects a panoramic optical image of the area to be imaged through surrounding distributed cameras, and fuses the panoramic sound image with the panoramic optical image to generate a target panoramic light and sound image of the area to be imaged. It can meet the needs of fast and real-time panoramic sound imaging, and the spatial resolution of imaging is high.

[0111] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0112] Based on the same inventive concept, an ambient sound imaging device corresponding to the ambient sound imaging method is also provided in the embodiments of the present disclosure. Since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to the above-mentioned ambient sound imaging method in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0113] See also Figure 3 , Figure 3 Schematic diagram of a panoramic sound imaging device provided by an embodiment of the present disclosure. Figure 3 As shown in , the panoramic sound imaging device 300 provided by the embodiment of the present disclosure includes:

[0114] The acoustic signal collection module 310 is used to collect the acoustic signal of the area to be imaged through a spherical microphone array.

[0115] The spherical harmonic coefficient determination module 320 is used to obtain a frequency domain signal corresponding to the acoustic signal, perform a spherical harmonic transformation on the frequency domain signal, and decompose the acoustic signal into spherical harmonic coefficients.

[0116] The sound image generation module 330 is used to extract the spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generate a panoramic sound image of the area to be imaged according to the spherical harmonic domain features and the panoramic space grid.

[0117] The light and sound image generation module 340 is used to collect a panoramic optical image of the area to be imaged through cameras distributed around, and fuse the panoramic sound image with the panoramic optical image to generate a target panoramic light and sound image of the area to be imaged.

[0118] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0119] The present disclosure provides a panoramic sound imaging device, which collects sound signals of the area to be imaged through a spherical microphone array; obtains frequency domain signals corresponding to the sound signals, performs spherical harmonic transformation on the frequency domain signals, and decomposes the sound signals into spherical harmonic coefficients; extracts spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generates a panoramic sound image of the area to be imaged based on the spherical harmonic domain features and a panoramic spatial grid; collects a panoramic optical image of the area to be imaged through surrounding cameras, and fuses the panoramic sound image with the panoramic optical image to generate a target panoramic light and sound image of the area to be imaged. The requirements of fast and real-time panoramic sound imaging can be met, and the spatial resolution of imaging is high.

[0120] Corresponds to Figure 1 The panoramic sound imaging method in the present disclosure also provides an electronic device 400, such as Figure 4FIG. 4 is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present disclosure, including:

[0121] Processor 41, memory 42, and bus 43; memory 42 is used to store execution instructions, including memory 421 and external memory 422; memory 421 here is also called internal memory, which is used to temporarily store operation data in processor 41 and data exchanged with external memory 422 such as hard disk. Processor 41 exchanges data with external memory 422 through memory 421. When the electronic device 400 is running, the processor 41 communicates with the memory 42 through bus 43, so that the processor 41 executes Figure 1 Steps of the panoramic sound imaging method.

[0122] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the panoramic sound imaging method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0123] The present disclosure also provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, the steps of the panoramic sound imaging method described in the above method embodiment can be executed. For details, please refer to the above method embodiment, which will not be repeated here.

[0124] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0125] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0127] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0128] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0129] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A panoramic sound imaging method, characterized in that: include: Acquiring acoustic signals of the area to be imaged by means of a spherical microphone array; Acquire a frequency domain signal corresponding to the acoustic signal, perform spherical harmonic transformation on the frequency domain signal, and decompose the acoustic signal into spherical harmonic coefficients; Extracting spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generating a panoramic sound image of the area to be imaged according to the spherical harmonic domain features and a panoramic spatial grid; The panoramic optical image of the area to be imaged is collected by cameras distributed around it, and the panoramic sound image is fused with the panoramic optical image to generate a target panoramic light and sound image of the area to be imaged.

2. The method according to claim 1, characterized in that Performing spherical harmonic transformation on the frequency domain signal to decompose the acoustic signal into spherical harmonic coefficients specifically includes: Constructing a receiving signal model corresponding to the spherical microphone array according to the sound source sound pressure signal vector corresponding to the sound signal, the steering matrix and the noise vector corresponding to the microphone; Determining a spherical harmonic representation of the steering matrix based on a spherical harmonic basis function matrix and a spherical harmonic basis function matrix; The steering matrix in the received signal model is replaced by a spherical harmonic representation of the steering matrix to transform the sound source sound pressure signal vector into a spherical harmonic domain, and the spherical harmonic coefficients are determined.

3. The method according to claim 2, characterized in that: The steering matrix represents the directional relationship from the sound source corresponding to the sound signal to the microphone; The spherical harmonic basis function matrix is ​​used to represent the microphone position; The spherical harmonic basis function matrix is ​​used to represent the direction of the sound source.

4. The method according to claim 1, characterized in that: Extracting the spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generating a panoramic sound image of the area to be imaged according to the spherical harmonic domain features and a panoramic space grid, specifically comprising: Inputting the spherical harmonic coefficients into a fully convolutional neural network, and extracting the spherical harmonic domain features of different orders through a convolution kernel in the frequency domain; Mapping the spherical harmonic domain features onto the panoramic spatial grid to determine a space-frequency pseudo-spectrum; An averaging process is performed on the space-frequency pseudo-spectrum in each sub-frequency band to generate the panoramic sound image.

5. The method according to claim 1, characterized in that: In the spherical microphone array, a plurality of unidirectional vector microphones are uniformly distributed in a spherical array.

6. The method according to claim 4, characterized in that Mapping the spherical harmonic domain features onto the panoramic spatial grid to determine a space-frequency pseudo-spectrum specifically includes: Rotating the spherical harmonic domain features output by the last layer in the fully convolutional neural network along the frequency axis to transpose the feature dimension and the spatial dimension; After transposing the feature dimension and the spatial dimension, the space-frequency pseudo-spectrum is determined through processing of the convolution layer and the activation function.

7. A panoramic sound imaging device, characterized in that: include: An acoustic signal acquisition module, used for acquiring acoustic signals of the area to be imaged through a spherical microphone array; A spherical harmonic coefficient determination module is used to obtain a frequency domain signal corresponding to the acoustic signal, perform a spherical harmonic transformation on the frequency domain signal, and decompose the acoustic signal into spherical harmonic coefficients; An audio-visual generation module, used for extracting the spherical harmonic domain features corresponding to the spherical harmonic coefficients, and generating a panoramic audio-visual image of the area to be imaged according to the spherical harmonic domain features and a panoramic spatial grid; The light and sound image generation module is used to collect a panoramic optical image of the area to be imaged through cameras distributed around it, and fuse the panoramic sound image with the panoramic optical image to generate a target panoramic light and sound image of the area to be imaged.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the panoramic sound imaging method as described in any one of claims 1 to 6 are performed.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the panoramic sound imaging method according to any one of claims 1 to 6 are executed.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the panoramic sound imaging method according to any one of claims 1 to 6 are implemented.

Images