Panoramic sound imaging method and device, electronic equipment and readable storage medium

Abstract

The invention provides a panoramic sound imaging method and device, electronic equipment and a readable storage medium, and the method comprises the steps: determining the spherical harmonic domain feature coefficient of each order corresponding to a sound signal based on the sound signal obtained by a spherical unidirectional microphone array; inputting the spherical harmonic domain characteristic coefficient of each order into a hierarchical multi-task model, and distributing a panoramic space grid with resolution from low to high corresponding to frequency hierarchies with frequency bands from low to high for the extracted spherical harmonic domain characteristics of each order according to the order from low to high, so as to obtain a panoramic sound image of the sound signal at each frequency hierarchies; and superposing the panoramic sound image of the sound signal at each frequency level to generate a broadband panoramic sound image of the sound signal. Therefore, the spherical harmonic domain characteristics of different orders are allocated to the frequency levels of different frequency bands through the level multi-task model, so that panoramic sound imaging of different frequency levels corresponding to different resolutions is realized, and the accuracy of broadband panoramic sound imaging is further improved.

Description

Technical Field

[0001] The present application relates to the field of environmental perception technology, and in particular to a panoramic sound imaging method, device, electronic device and readable storage medium. Background Art

[0002] Vision and hearing are the main ways for humans to perceive the world, providing spatial information and sound field information respectively. However, traditional visual perception has limited field of view, and traditional auditory perception has low spatial recognition, which makes it difficult for people to meet their all-round perception needs of the real world. As a new imaging technology that integrates panoramic vision and panoramic hearing, panoramic sound imaging technology can provide all-round vision and spatial directional auditory perception, with the advantages of high resolution, no blind spots and multi-modal intelligent perception.

[0003] With the continuous development of new sensing technologies and intelligent processing devices, panoramic sound imaging technology based on large-scale visual and auditory sensor arrays has gradually become feasible. However, due to the limitations of the Rayleigh limit, the spatial sampling of the sound field is often restricted, making it a huge challenge for small-aperture microphone arrays to reconstruct high-resolution broadband sound fields. Existing panoramic sound imaging methods usually use a planar microphone array to obtain the sound and image of the segmented area and splice them to generate a panoramic sound image. However, in the lower frequency bands, higher-order frequency domain features may have pathological problems, and it is impossible to guarantee that the panoramic sound image is accurately imaged in each frequency band, resulting in low accuracy of broadband panoramic sound imaging. Summary of the invention

[0004] In view of this, the embodiments of the present application at least provide a panoramic sound imaging method, device, electronic device and readable storage medium, which allocates spherical harmonic domain features of different orders to frequency levels in different frequency bands through a hierarchical multi-task model, thereby realizing panoramic sound imaging of different frequency levels at corresponding different resolutions, thereby improving the accuracy of broadband panoramic sound imaging.

[0005] This application mainly includes the following aspects:

[0006] In a first aspect, an embodiment of the present application provides a panoramic sound imaging method, the method comprising:

[0007] Based on the acoustic signal acquired by the spherical unidirectional microphone array, determining the spherical harmonic domain characteristic coefficients of each order corresponding to the acoustic signal;

[0008] Inputting the spherical harmonic domain feature coefficients of each order into the hierarchical multi-task model, allocating panoramic spatial grids with resolutions from low to high corresponding to frequency levels from low to high frequency bands to the extracted spherical harmonic domain features of each order in order from low to high orders, and obtaining panoramic sound images of the sound signal at each frequency level;

[0009] The panoramic sound images of the sound signal at each frequency level are superimposed to generate a broadband panoramic sound image of the sound signal.

[0010] In a second aspect, an embodiment of the present application further provides a panoramic sound imaging device, the panoramic sound imaging device comprising:

[0011] A parameter determination module, used to determine the spherical harmonic domain characteristic coefficients of various orders corresponding to the acoustic signal based on the acoustic signal acquired by the spherical unidirectional microphone array;

[0012] The sound image generation module is used to input the spherical harmonic domain feature coefficients of each order into the hierarchical multi-task model, and allocate panoramic spatial grids with resolutions from low to high corresponding to frequency levels from low to high frequency bands to the extracted spherical harmonic domain features of each order in order from low to high order, so as to obtain the panoramic sound image of the sound signal at each frequency level;

[0013] The sound and image superposition module is used to superimpose the panoramic sound and image of the sound signal at each frequency level to generate a broadband panoramic sound and image of the sound signal.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the panoramic sound imaging method as described above.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the panoramic sound imaging method as described above are executed.

[0016] The embodiments of the present application provide a method, device, electronic device and readable storage medium for panoramic sound imaging, wherein the method includes: determining the spherical harmonic domain characteristic coefficients of each order corresponding to the sound signal based on the sound signal obtained by the spherical unidirectional microphone array; inputting the spherical harmonic domain characteristic coefficients of each order into the hierarchical multi-task model, and assigning the panoramic space grids with resolutions from low to high corresponding to the frequency levels of the frequency bands from low to high to the extracted spherical harmonic domain characteristics of each order in the order from low to high, so as to obtain the panoramic sound image of the sound signal at each frequency level; superimposing the panoramic sound image of the sound signal at each frequency level to generate a broadband panoramic sound image of the sound signal. In this way, the hierarchical multi-task model is used to assign spherical harmonic domain features of different orders to the frequency levels of different frequency bands, thereby realizing the panoramic sound imaging of different frequency levels at corresponding different resolutions, thereby improving the accuracy of broadband panoramic sound imaging.

[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flow chart of a panoramic sound imaging method provided by an embodiment of the present application is shown;

[0020] Figure 2 The overall process diagram of the panoramic sound imaging method in the embodiment of the present application is shown;

[0021] Figure 3 A schematic diagram of the structure of a spherical unidirectional microphone array in an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of the structure of a hierarchical multi-task model in an embodiment of the present application is shown;

[0023] Figure 5 One of the functional module diagrams of a panoramic sound imaging device provided in an embodiment of the present application is shown;

[0024] Figure 6 A second functional module diagram of a panoramic sound imaging device provided in an embodiment of the present application is shown;

[0025] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0027] The following is a detailed description of a panoramic sound imaging method provided by an embodiment of the present application. Figure 1 , Figure 1 This is a flow chart of a panoramic sound imaging method provided by an embodiment of the present application. Figure 1 As shown, the method includes:

[0028] S101, based on an acoustic signal acquired by a spherical unidirectional microphone array, determining characteristic coefficients of spherical harmonic domains of various orders corresponding to the acoustic signal.

[0029] Here, traditional panoramic sound imaging technology is usually achieved through the registration and stitching of sound and images collected in a plane, which not only faces the problem of stitching mismatch between multiple sound images, but also fails to meet the needs of high-resolution broadband sound and images. The embodiment of the present application acquires sound signals through a spherical unidirectional microphone array, and determines the characteristic coefficients of the spherical harmonic domain of each order corresponding to the sound signal based on the acquired sound signal, which can not only achieve the consistency of spatial directivity and support single-shot rapid panoramic imaging, but also make the characteristic directivity more precise as the order increases.

[0030] S102, inputting the spherical harmonic domain feature coefficients of each order into the hierarchical multi-task model, assigning panoramic spatial grids with resolutions from low to high corresponding to frequency levels from low to high frequency bands to the extracted spherical harmonic domain features of each order in order from low to high, and obtaining the panoramic sound image of the sound signal at each frequency level.

[0031] Here, since higher-order frequency domain features may have pathological problems in lower frequency bands and it is difficult to simultaneously meet the requirements of high-resolution panoramic acoustic imaging in different frequency bands, the embodiment of the present application introduces a hierarchical multi-task model to process spherical harmonic domain features of various orders, and the spherical harmonic domain feature coefficients of various orders are input into the hierarchical multi-task model. Panoramic spatial grids with resolutions from low to high corresponding to frequency levels from low to high frequency bands are allocated to the extracted spherical harmonic domain features of various orders in order from low to high, so as to obtain the panoramic sound image of the sound signal at various frequency levels.

[0032] Among them, the core of the hierarchical multi-task model is to allocate panoramic space grids of different resolutions for spherical harmonic domain features of different orders and different frequency levels. Specifically, the frequency range is divided into multiple different frequency levels from low to high according to the frequency band, and the panoramic space grids with different resolutions are divided for each frequency level according to the computational complexity, and the frequency level corresponding to the lower frequency band is allocated with a panoramic space grid with a lower resolution, and the frequency level corresponding to the higher frequency band is allocated with a panoramic space grid with a higher resolution; at the same time, the frequency level corresponding to the lower frequency band is allocated with a lower-order spherical harmonic domain feature, and the frequency level corresponding to the higher frequency band is allocated with a higher-order spherical harmonic domain feature. In this way, the imaging tasks of each frequency level are processed in the panoramic space grids at different resolutions, so as to realize the panoramic sound imaging of different resolutions at different frequency levels, and solve the pathological problems that may occur in the frequency domain features of higher orders in the lower frequency band. Here, since the acoustic signal acquisition in the embodiment of the present application adopts a spherical unidirectional microphone array, the panoramic space grid in the embodiment of the present application adopts a spatial spherical uniform sampling grid.

[0033] S103: superimpose the panoramic sound images of the sound signal at each frequency level to generate a broadband panoramic sound image of the sound signal.

[0034] Here, the panoramic sound images of the sound signal at each frequency level are superimposed and fused to generate a wideband panoramic sound image of the sound signal. Specifically, the panoramic sound image with lower resolution at a lower frequency level is interpolated and superimposed and fused to the panoramic sound image with higher resolution at a higher frequency level to obtain a wideband panoramic sound image of the sound signal.

[0035] Furthermore, S101 specifically includes:

[0036] Step a1: converting the acoustic signal acquired by the spherical unidirectional microphone array into a frequency domain signal according to short-time Fourier transform.

[0037] Specifically, see Figure 2 , Figure 2 Schematic diagram of the overall process of the panoramic sound imaging method in the embodiment of the present application. Figure 2 As shown, after the spherical unidirectional microphone array acquires the acoustic signal, the acoustic signal acquired by the spherical unidirectional microphone array is converted into a frequency domain signal according to short-time Fourier transform (STFT).

[0038] Specifically, see Figure 3 , Figure 3 Schematic diagram of the structure of a spherical unidirectional microphone array in an embodiment of the present application. Figure 3As shown in the figure, Q unidirectional microphones are evenly distributed on a rigid sphere with a radius of r. The spatial coordinates of any unidirectional microphone q can be expressed as:

[0039] r q =[rcos(θ q )sin(φ q ),rsin(θ q )sin(φ q ),rcos(φ q )] T ;

[0040] Where, q=1,…,Q; θ q is the direction measured counterclockwise from the x-axis, φ q is the elevation angle measured downward from the z-axis.

[0041] Assuming that there are L far-field sound sources in space, the coordinates of any far-field sound source l can be expressed as:

[0042] r l =[rcos(φ l )sin(φ l ),rsin(θ l )sin(φ l ),rcos(φ l )] T ;

[0043] Where, l = 1,…,L;

[0044] In this way, the received signal model of the spherical monodirectional microphone array can be expressed as:

[0045] p(k)=A(k)S(k)+z(k);

[0046] Where k is the wave number, i.e. k = 2πf / c, f is the frequency, and c is the speed of sound. p(k) is the sound pressure signal vector, A(k) is the steering matrix, S(k) is the sound source sound pressure signal vector, and z(k) is the sensor noise vector. p(k), A(k) and S(k) are expressed as:

[0047] p(k)=[p(k,r1),p(k,r2),…,p(k,r Q )] T ;

[0048] A(k)=[a(k,r1),a(k,r2),…,a(k,r L )] T ;

[0049] S(k)=[s1(k),s2(k),…,s L (k)] T;

[0050] Among them, p(k,r q ) is the acoustic signal received by the qth microphone, a(k,r l ) and s l (k) are the guiding vector and sound pressure intensity of the lth sound source respectively. l ) can be expressed as:

[0051]

[0052] Among them, the wave vector k of the lth sound source l It can be expressed as:

[0053] k l =-k[cos(θ l )sin(φ l ), sin(θ l )sin(φ l ),cos(φ l )] T ;i 2 = -1;

[0054] According to the Spherical Harmonic Transform (SHT), It can be approximately decomposed into truncated N-order spherical harmonics:

[0055]

[0056] Among them, b n (kr) is the mode strength for a unidirectional microphone:

[0057] b n (kr)=4πi n [j n (kr)-ij′ n (kr)];

[0058] is the spherical harmonic basis function:

[0059]

[0060] is the Legendre function; j n is a spherical Bessel function of the first kind, so the steering vector A(k) can be expressed as:

[0061] A(k)=Y(Ω)B(kr)Y H (Γ);

[0062] Where Y(Ω) is Q×(N+1) 2The microphone angle coordinate matrix is:

[0063] Y(Ω)=[y T (θ1,φ1),…,y T (θ Q ,φ Q )] T ;

[0064]

[0065] From the spherical harmonics of L sound source directions, we can get L×(N+1) 2 The sound source angle coordinate matrix Y H (Γ):

[0066] Y H (Γ)=[y T (θ1,φ1),…,y T (θ L ,φ L )];

[0067] and (N+1) 2 ×(N+1) 2 The diagonal matrix B(kr) is:

[0068] b(kr)=diag{b0(kr),b1(kr),b1(kr),…,b N (kr)};

[0069] From this, we can get the expression of the sound pressure signal vector p(k):

[0070] p(k)=Y(Ω)B(kr)Y H (Γ)S(k)+z(k);

[0071] like Figure 2 As shown, for a broadband sound source, the sound imaging under reverberation conditions can be achieved by smoothing in the selected time-frequency domain through short-time Fourier transform, and the time-domain sound pressure signal vector p(t) is transformed into the frequency-domain sound pressure signal vector p(τ,v).

[0072] Step a2, performing spherical harmonic transformation on the frequency domain signal to obtain characteristic coefficients of spherical harmonic domains of various orders corresponding to the acoustic signal.

[0073] In this step, due to the orthogonality principle of spherical harmonics, for approximately uniform spherical space sampling, Y h (Ω)Y(Ω)=I, we can get the sound pressure signal vector p converted to the spherical harmonic domain after spherical harmonic transformation nm (k) is:

[0074] p nm (k) = B(kr)YH (Γ)S(k)+z nm (k);

[0075] where p nm (k) = Y H (Ω)p(k); n and m are the spherical harmonic order (Degree) and the secondary quantum number (Order) at which the spherical harmonic ends, respectively.

[0076] According to the above formula derivation process, for a given spherical microphone array configuration information, we can get the spherical harmonic characteristic coefficient q nm (k):

[0077]

[0078] in, is the sensor noise vector in the spherical harmonic domain after the spherical harmonic transformation.

[0079] Then, the spherical harmonic domain characteristic coefficients q of each order after spherical harmonic transformation of the frequency domain signal after short-time Fourier transformation are obtained. nm (τ,v) can be expressed as:

[0080]

[0081] in is the time-frequency space of the sound signal.

[0082] In the embodiment of the present application, the spherical harmonic domain characteristic coefficient q can be established by transforming the above formula by transposing the diagonal matrix nm (τ,v) and the spherical harmonic domain sound pressure signal vector p after spherical harmonic transformation nm The corresponding relationship of (τ, v) is as follows:

[0083] q nm (τ,v)=B -1 (v,r)p nm (τ,v);

[0084] Furthermore, if Figure 2 As shown, the frequency domain signal p(τ,v) is transformed into spherical harmonics to obtain the characteristic coefficients q of each order of spherical harmonics corresponding to the acoustic signal. nm (τ,v).

[0085] Furthermore, the hierarchical multi-task model includes a convolutional neural network and multiple multi-layer perception networks, and S102 specifically includes:

[0086] Step b1, the spherical harmonic domain characteristic coefficients located in the corresponding frequency band of each frequency level are independently input into the convolutional neural network to obtain spherical harmonic domain characteristics of each order.

[0087] Here, please see Figure 4 , Figure 4 Schematic diagram of the structure of the hierarchical multi-task model in the embodiment of the present application. Figure 4 As shown in Figure 1, the hierarchical multi-task model includes a convolutional neural network (CNN) and multiple multi-layer perception networks (MLPN). First, according to the number of feature orders in the spherical harmonic domain N Seg , the frequency range is divided into multiple frequency intervals, each frequency interval corresponds to a frequency level. The spherical harmonic domain feature coefficients in the frequency bands corresponding to each frequency level, i.e., the frequency intervals, are independently input into the convolutional neural network to extract the spherical harmonic domain features of each order.

[0088] Step b2, assigning a multi-layer perception network corresponding to a frequency band from low to high for each order of spherical harmonic domain features in order from low to high, inputting each order of spherical harmonic domain features into the multi-layer perception network of the corresponding frequency level, so that each order of spherical harmonic domain features is respectively mapped to the corresponding panoramic space grid with a resolution from low to high in order from low to high, so as to obtain the panoramic sound image of the sound signal at each frequency level.

[0089] Here, the spherical harmonic domain features of each order are assigned to the multi-layer perception network corresponding to the frequency level from low to high in the order from low to high, and the spherical harmonic domain features of each order are input into the multi-layer perception network of the corresponding frequency level, so that the spherical harmonic domain features of each order are mapped to the corresponding panoramic space grids with low to high resolutions in the order from low to high, and the panoramic sound image of the sound signal at each frequency level is obtained. Among them, any multi-layer perception network includes multiple linear layers, and the number of linear layers N layer The spherical harmonic domain features of each frequency level are mapped to the panoramic space grid of the corresponding level respectively. layer After being processed by the linear layer and the rectified linear unit (ReLU) activation function, the space × frequency pseudo-spectrum, i.e. the panoramic sound image at each frequency level, is generated in the last layer through the sigmoid activation function.

[0090] Furthermore, the convolutional neural network includes multiple layers of convolution kernels, and step b1 specifically includes:

[0091] Step c1, extracting spherical harmonic domain features of each order in each frequency level from the spherical harmonic domain feature coefficients input each time according to the multi-layer convolution kernel.

[0092] Here, the convolutional neural network includes a multi-layer convolution kernel. In the embodiment of the present application, the size of the multi-layer convolution kernel is 2×1, corresponding to the spherical harmonic domain characteristic coefficient dimension and the frequency dimension respectively. Such a design can ensure the independence between frequency features and can effectively extract features on the spherical harmonic domain characteristic coefficient. Specifically, according to the multi-layer convolution kernel, spherical harmonic domain features of each order in each frequency level are extracted from the spherical harmonic domain characteristic coefficient input each time.

[0093] Step c2, merging the spherical harmonic domain features of the same order in each frequency level to obtain the spherical harmonic domain features of each order.

[0094] Here, the spherical harmonic domain features of the same order extracted in each frequency level are fused together to obtain the spherical harmonic domain features of each order.

[0095] Furthermore, when extracting spherical harmonic domain features of each order in each frequency level from the spherical harmonic domain feature coefficients input each time, the method further includes:

[0096] For any order of spherical harmonic domain features in each frequency level, the number of feature channels is adjusted so that the spatial dimension of the spherical harmonic domain features of the order is the same as the spatial dimension of the panoramic space grid corresponding to the multi-layer perception network assigned to the spherical harmonic domain features of the order.

[0097] Here, before the spherical harmonic domain features of each order are input into the corresponding multi-layer perception network, it is necessary to ensure that the spatial dimension of the spherical harmonic domain features is the same as the spatial dimension of the panoramic space grid corresponding to the assigned multi-layer perception network, so that panoramic sound imaging can be performed in the panoramic space grid of the corresponding resolution. Therefore, when extracting the spherical harmonic domain features of each order in each frequency level from the spherical harmonic domain feature coefficients input each time, it is necessary to spatially upgrade the spherical harmonic domain features of each order to match the spatial dimension of the panoramic space grid. In an embodiment of the present application, for the spherical harmonic domain features of any order in each frequency level, the number of feature channels is adjusted so that the spatial dimension of the spherical harmonic domain features of this order is the same as the spatial dimension of the panoramic space grid corresponding to the multi-layer perception network assigned to the spherical harmonic domain features of this order. Here, the mapping relationship with the panoramic space grid can be achieved by increasing the number of feature channels of the two-dimensional convolution. Specifically, the feature dimension and the spatial dimension are transposed through the rotation of the convolutional network; then the spatial dimension is increased through feature dimension increase; then in the last layer of the convolutional neural network, the number of neurons is ensured to be consistent with the number of panoramic spatial grids; finally, the feature and spatial dimensions are transposed and input into the 1×1 convolutional layer, so that the spatial dimensions of the spherical harmonic domain features of each order are the same as the spatial dimensions of the panoramic spatial grid corresponding to the multi-layer perception network assigned to the spherical harmonic domain features of the corresponding order.

[0098] Furthermore, the method further comprises:

[0099] Step d1, acquiring image information collected by each surround camera on the spherical unidirectional microphone array, and generating a panoramic optical image based on image stitching.

[0100] Here, if Figure 2 As shown, a plurality of surround cameras are also installed on the spherical unidirectional microphone array. The panoramic sound imaging method of the embodiment of the present application can also obtain image information collected by each surround camera, and through image stitching, stitch the images collected by each surround camera together to generate a panoramic optical image.

[0101] Step d2: superimpose and fuse the panoramic optical image with the broadband panoramic sound image to generate a broadband panoramic sound and light image of the sound signal.

[0102] Here, if Figure 2 As shown in the figure, the panoramic optical image and the broadband panoramic sound image are superimposed and fused to generate a broadband panoramic sound and light image of the sound signal. The broadband panoramic sound and light image includes both acoustic information and optical information, which can more intuitively display the spatial distribution of the sound signal and further improve the accuracy of broadband panoramic sound imaging.

[0103] The embodiment of the present application provides a panoramic sound imaging method, including: determining the spherical harmonic domain characteristic coefficients of each order corresponding to the sound signal based on the sound signal obtained by the spherical unidirectional microphone array; inputting the spherical harmonic domain characteristic coefficients of each order into the hierarchical multi-task model, and assigning the panoramic space grids with resolutions from low to high corresponding to the frequency levels of the frequency bands from low to high to the extracted spherical harmonic domain characteristics of each order in the order from low to high, so as to obtain the panoramic sound image of the sound signal at each frequency level; superimposing the panoramic sound image of the sound signal at each frequency level to generate a broadband panoramic sound image of the sound signal. In this way, the hierarchical multi-task model is used to assign spherical harmonic domain features of different orders to the frequency levels of different frequency bands, thereby realizing the panoramic sound imaging of different frequency levels at corresponding different resolutions, thereby improving the accuracy of broadband panoramic sound imaging.

[0104] Based on the same application concept, an embodiment of the present application also provides an ambient sound imaging device corresponding to the ambient sound imaging method provided in the above embodiment. Since the principle of solving the problem by the device in the embodiment of the present application is similar to that of the ambient sound imaging method in the above embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0105] See also Figure 5 , Figure 5 This is one of the functional module diagrams of a panoramic sound imaging device provided in an embodiment of the present application. Figure 5As shown, the panoramic sound imaging device 500 includes:

[0106] The parameter determination module 510 is used to determine the spherical harmonic domain characteristic coefficients of various orders corresponding to the acoustic signal based on the acoustic signal acquired by the spherical unidirectional microphone array.

[0107] The sound image generation module 520 is used to input the spherical harmonic domain feature coefficients of each order into the hierarchical multi-task model, and allocate panoramic spatial grids with resolutions from low to high corresponding to frequency levels from low to high frequency bands to the extracted spherical harmonic domain features of each order in order from low to high, so as to obtain the panoramic sound image of the sound signal at each frequency level.

[0108] The sound and image superposition module 530 is used to superimpose the panoramic sound and image of the sound signal at each frequency level to generate a broadband panoramic sound and image of the sound signal.

[0109] Further, when the parameter determination module 510 is used to determine the characteristic coefficients of each order of spherical harmonic domain corresponding to the acoustic signal based on the acoustic signal acquired by the spherical unidirectional microphone array, the parameter determination module 510 is specifically used to:

[0110] According to short-time Fourier transform, the acoustic signal acquired by the spherical unidirectional microphone array is converted into a frequency domain signal;

[0111] The frequency domain signal is subjected to spherical harmonic transformation to obtain characteristic coefficients of spherical harmonic domains of various orders corresponding to the acoustic signal.

[0112] Further, the hierarchical multi-task model includes a convolutional neural network and a plurality of multi-layer perceptual networks. The sound image generation module 520 is used to input the spherical harmonic domain feature coefficients of each order into the hierarchical multi-task model, and to allocate panoramic space grids with resolutions from low to high corresponding to frequency levels with frequency bands from low to high to the extracted spherical harmonic domain features of each order in order from low to high, so as to obtain the panoramic sound image of the sound signal at each frequency level. The sound image generation module 520 is specifically used to:

[0113] The spherical harmonic domain characteristic coefficients located in the corresponding frequency bands of each frequency level are independently input into the convolutional neural network to obtain spherical harmonic domain characteristics of each order;

[0114] A multi-layer perception network corresponding to a frequency band from low to high is allocated to each order of spherical harmonic domain features in order from low to high, and each order of spherical harmonic domain features is input into the multi-layer perception network of the corresponding frequency level, so that each order of spherical harmonic domain features is mapped to a corresponding panoramic space grid with a resolution from low to high in order from low to high, and a panoramic sound image of the sound signal at each frequency level is obtained.

[0115] Furthermore, the convolutional neural network includes multiple layers of convolution kernels. When the sound image generation module 520 is used to independently input the spherical harmonic domain characteristic coefficients located in the frequency band corresponding to each frequency level into the convolutional neural network to obtain spherical harmonic domain characteristics of each order, the sound image generation module 520 is specifically used to:

[0116] According to the multi-layer convolution kernel, spherical harmonic domain features of each order in each frequency level are extracted from the spherical harmonic domain feature coefficients input each time;

[0117] The spherical harmonic domain features of the same order in each frequency level are fused to obtain the spherical harmonic domain features of each order.

[0118] Furthermore, when the sound image generation module 520 is used to extract the spherical harmonic domain features of each order in each frequency level from the spherical harmonic domain feature coefficients input each time, the sound image generation module 520 is also used to:

[0119] For any order of spherical harmonic domain features in each frequency level, the number of feature channels is adjusted so that the spatial dimension of the spherical harmonic domain features of the order is the same as the spatial dimension of the panoramic space grid corresponding to the multi-layer perception network assigned to the spherical harmonic domain features of the order.

[0120] Furthermore, any of the multi-layer perception networks includes multiple linear layers, and the number of the linear layers is positively correlated with the number of panoramic space grids corresponding to the frequency levels.

[0121] For further information, see Figure 6 , Figure 6 This is a second functional module diagram of a panoramic sound imaging device provided in an embodiment of the present application. Figure 6 As shown, the panoramic sound imaging device 500 further includes:

[0122] The optical image generation module 540 is used to obtain image information collected by each surround camera on the spherical unidirectional microphone array, and generate a panoramic optical image based on image stitching.

[0123] The sound-light image superposition module 550 is used to superimpose and fuse the panoramic optical image with the broadband panoramic sound image to generate a broadband panoramic sound-light image of the sound signal.

[0124] The embodiment of the present application provides a panoramic sound imaging device, including: a parameter determination module, which is used to determine the spherical harmonic domain characteristic coefficients of each order corresponding to the sound signal based on the sound signal obtained by the spherical unidirectional microphone array; a sound image generation module, which is used to input the spherical harmonic domain characteristic coefficients of each order into the hierarchical multi-task model, and assign the panoramic space grids with resolutions from low to high corresponding to the frequency levels of the frequency bands from low to high to the extracted spherical harmonic domain characteristics of each order in the order from low to high, so as to obtain the panoramic sound image of the sound signal at each frequency level; a sound image superposition module, which is used to superimpose the panoramic sound images of the sound signal at each frequency level to generate a broadband panoramic sound image of the sound signal. In this way, the hierarchical multi-task model is used to assign spherical harmonic domain features of different orders to the frequency levels of different frequency bands, thereby realizing the panoramic sound imaging of different frequency levels at corresponding different resolutions, thereby improving the accuracy of broadband panoramic sound imaging.

[0125] Based on the same application idea, please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 includes a processor 710 , a memory 720 and a bus 730 .

[0126] The memory 720 stores machine-readable instructions executable by the processor 710. When the electronic device 700 is running, the processor 710 communicates with the memory 720 through the bus 730. When the processor 710 is running, the machine-readable instructions execute the steps of the panoramic sound imaging method provided in the above embodiment. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0127] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the panoramic sound imaging method provided in the above embodiment are executed. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0132] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0133] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0134] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A panoramic sound imaging method, characterized in that: The method comprises: Based on the acoustic signal acquired by the spherical unidirectional microphone array, determining the spherical harmonic domain characteristic coefficients of each order corresponding to the acoustic signal; Inputting the spherical harmonic domain feature coefficients of each order into the hierarchical multi-task model, allocating panoramic spatial grids with resolutions from low to high corresponding to frequency levels from low to high frequency bands to the extracted spherical harmonic domain features of each order in order from low to high orders, and obtaining panoramic sound images of the sound signal at each frequency level; The panoramic sound images of the sound signal at each frequency level are superimposed to generate a broadband panoramic sound image of the sound signal.

2. The panoramic sound imaging method according to claim 1, characterized in that: The method of determining characteristic coefficients of spherical harmonic domains of various orders corresponding to the acoustic signal based on the acoustic signal obtained by the spherical unidirectional microphone array comprises: According to short-time Fourier transform, the acoustic signal acquired by the spherical unidirectional microphone array is converted into a frequency domain signal; The frequency domain signal is subjected to spherical harmonic transformation to obtain characteristic coefficients of spherical harmonic domains of various orders corresponding to the acoustic signal.

3. The panoramic sound imaging method according to claim 1, characterized in that: The hierarchical multi-task model includes a convolutional neural network and a plurality of multi-layer perceptual networks. The spherical harmonic domain feature coefficients of each order are input into the hierarchical multi-task model, and panoramic spatial grids with resolutions from low to high corresponding to frequency levels from low to high frequency bands are allocated to the extracted spherical harmonic domain features of each order in order from low to high order, so as to obtain panoramic sound images of the sound signal at each frequency level, including: The spherical harmonic domain characteristic coefficients located in the corresponding frequency bands of each frequency level are independently input into the convolutional neural network to obtain spherical harmonic domain characteristics of each order; A multi-layer perception network corresponding to a frequency band from low to high is allocated to each order of spherical harmonic domain features in order from low to high, and each order of spherical harmonic domain features is input into the multi-layer perception network of the corresponding frequency level, so that each order of spherical harmonic domain features is mapped to a corresponding panoramic space grid with a resolution from low to high in order from low to high, and a panoramic sound image of the sound signal at each frequency level is obtained.

4. The panoramic sound imaging method according to claim 3, characterized in that: The convolutional neural network includes multiple layers of convolution kernels, and the spherical harmonic domain characteristic coefficients located in the frequency band corresponding to each frequency level are independently input into the convolutional neural network to obtain spherical harmonic domain characteristics of each order, specifically including: According to the multi-layer convolution kernel, spherical harmonic domain features of each order in each frequency level are extracted from the spherical harmonic domain feature coefficients input each time; The spherical harmonic domain features of the same order in each frequency level are fused to obtain the spherical harmonic domain features of each order.

5. The panoramic sound imaging method according to claim 4, characterized in that: When extracting spherical harmonic domain features of each order in each frequency level from the spherical harmonic domain feature coefficients input each time, the method further includes: For any order of spherical harmonic domain features in each frequency level, the number of feature channels is adjusted so that the spatial dimension of the spherical harmonic domain features of the order is the same as the spatial dimension of the panoramic space grid corresponding to the multi-layer perception network assigned to the spherical harmonic domain features of the order.

6. The panoramic sound imaging method according to claim 3, characterized in that: Any of the multi-layer perception networks includes multiple linear layers, and the number of the linear layers is positively correlated with the number of panoramic space grids corresponding to the frequency levels.

7. The panoramic sound imaging method according to claim 1, characterized in that: The method further comprises: Acquire image information collected by each surround camera on the spherical unidirectional microphone array, and generate a panoramic optical image based on image stitching; The panoramic optical image is superimposed and fused with the broadband panoramic sound image to generate a broadband panoramic sound and light image of the sound signal.

8. A panoramic sound imaging device, characterized in that: The panoramic sound imaging device comprises: A parameter determination module, used to determine the spherical harmonic domain characteristic coefficients of various orders corresponding to the acoustic signal based on the acoustic signal acquired by the spherical unidirectional microphone array; The sound image generation module is used to input the spherical harmonic domain feature coefficients of each order into the hierarchical multi-task model, and allocate panoramic spatial grids with resolutions from low to high corresponding to frequency levels from low to high frequency bands to the extracted spherical harmonic domain features of each order in order from low to high order, so as to obtain the panoramic sound image of the sound signal at each frequency level; The sound and image superposition module is used to superimpose the panoramic sound and image of the sound signal at each frequency level to generate a broadband panoramic sound and image of the sound signal.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the panoramic sound imaging method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the panoramic sound imaging method according to any one of claims 1 to 7 are executed.

Images