Attitude and depth map estimation method and system based on audiovisual knowledge distillation

Abstract

The invention discloses a pose and depth map estimation method and system based on audiovisual knowledge distillation, and is applied to the technical field of audiovisual multi-modality, and the method comprises the steps: obtaining an audio and a video, and carrying out the preprocessing of the audio and the video, and obtaining an audio frame-image pair; mapping the sound wave information of the audio frame; inputting the mapped sound wave information into a convolutional layer to carry out feature extraction, outputting a feature map, and then carrying out up-sampling; the feature map after up-sampling is subjected to pose estimation and depth map estimation respectively through a pose decoding module and an FPN (Fabry-Perot Networking); according to the method, the images in the audio frame-image pair are predicted through the mature visual network to generate the corresponding pseudo labels for learning of the auditory network.

Description

Technical Field

[0001] The present invention relates to the field of audio-visual multimodal technology, and in particular to a method and system for estimating a pose and depth map based on audio-visual knowledge distillation. Background Art

[0002] Knowledge distillation is a technique for training neural networks in which the knowledge of a large "teacher" model is transferred to a smaller "student" model, with the goal of the student model learning to mimic the behavior and predictive power of the teacher model.

[0003] Pose estimation and depth map estimation are two important tasks in the field of computer vision. Pose estimation refers to determining the position and orientation of an object or scene in three-dimensional space by extracting features from images or sensor data. Depth map estimation refers to inferring the depth information of each pixel in the scene from images or sensor data.

[0004] The main representatives of existing audio-visual learning deep network models are: SoundNet and Binaural SoundNet.

[0005] SoundNet jointly trains the proposed auditory network with ImageNet, Place CNN, etc. First, each video in the video dataset is split into two parts, one for audio and the other for RGB image frames. The visual recognition network uses the pre-trained VGG model on two large image datasets, ImageNet and Places. Both ImageNet and Places are large datasets in the field of image recognition, of which Imagenet is an image target classification dataset and Places is an image scene classification dataset. The RGB frames segmented from the video are input into the pre-trained VGG model, and the output results are used as the supervision information of the sound recognition network. The sound recognition network adopts an 8-layer fully convolutional structure, uses the sound time series extracted from the video as the input of the network, and uses KL-divergence as the loss function.

[0006] Binaural SoundNet jointly trains the auditory network with DeepLabv3+model, MonoDepth2 model, FlowNet2 model, etc. Unlike SounNet, which divides video data into audio and video data, it uses a separate multi-directional microphone device to receive sound during the initial data collection to obtain the three main clues that humans use to locate the sound source: interaural time difference (ITD), interaural level difference (ILD), and head-related transfer function (HRTF). Therefore, the auditory network can obtain more prior information during training to improve the performance of the network.

[0007] However, the existing auditory perception network is only implemented using convolutional modules, while ignoring the timing in pose estimation and the scale characteristics in depth map estimation. These shortcomings can be improved by using transformer and FPN in the decoder part respectively in this application. Summary of the invention

[0008] The purpose of this application is to provide a method and system for pose and depth map estimation based on audio-visual knowledge distillation, aiming to solve the problem that existing visual methods have great limitations for occlusion problems and the prediction estimation accuracy will drop significantly in the case of occlusion.

[0009] To achieve the above objectives, this application provides the following technical solutions:

[0010] The present application provides a method for estimating a pose and depth map based on audio-visual knowledge distillation, including:

[0011] Acquire audio and video and perform preprocessing to obtain audio frame-image pairs;

[0012] Mapping the sound wave information of the audio frame;

[0013] The mapped sound wave information is input into the convolution layer for feature extraction, and up-sampled after outputting the feature map;

[0014] The upsampled feature map is passed through the pose decoding module and the FPN network to obtain pose estimation and depth map estimation respectively.

[0015] Furthermore, in the step of mapping the sound wave information of the audio frame, the following steps are specifically included:

[0016] The sound wave information of the audio frame is mapped into a corresponding frequency function f(t), a loudness function l(t) and a hidden code z(t);

[0017] The frequency function f(t) is extracted by the pre-trained CREPE model; the loudness function l(t) is extracted by the audio; the amplitude attenuation weight tensor corresponding to the frequency tensor is calculated using the A-weighted power spectrum, and then the weight tensor is calculated in decibel scale, and the volume vector is normalized in the data set, and finally MFCCs are extracted and input into the normalization layer, the 512-unit CPU and the 512-unit linear layer to obtain the hidden code z(t).

[0018] Furthermore, in the step of inputting the mapped sound wave information into the convolution layer for feature extraction and outputting the feature map and then performing upsampling, the following steps are specifically included:

[0019] The mapped sound wave information is fused after four 4*4 convolutions, and a 1*1 convolution with 64 filters and a 3*3 convolution with expansion coefficients of 6, 12, and 18 are used. Finally, a 1*1 convolution is used to obtain the feature map of the corresponding audio.

[0020] The feature map is upsampled using a bilinear interpolation method.

[0021] Furthermore, in the step of obtaining pose estimation and depth map estimation respectively through the pose decoding module and the FPN network of the upsampled feature map, the following steps are specifically included:

[0022] The upsampled feature map is input into the pose decoding module to obtain the pose estimation;

[0023] The upsampled feature map is input into the FPN network to obtain the depth map estimation.

[0024] Furthermore, in the step of inputting the upsampled feature map into the pose decoding module to obtain pose estimation, the following steps are specifically included:

[0025] The upsampled feature map is input into the pose decoding module as query, and the sparse grid code obtained by encoding the audio frame information at adjacent moments is used as key and value;

[0026] The formula for pose decoding is:

[0027]

[0028]

[0029] where f i t is the output of t transformer modules, A scaling factor to enhance sparsity, is the code encoding, CrAtt is the cross attention function;

[0030] The transformer multi-head attention formula is:

[0031]

[0032] f i t+1 =MLP(softmax(Q·K)·V)

[0033] are the multi-head attention matrices at the current time step t, f i t+1is the hidden variable in the middle layer of the network, MLP() is the multi-layer perceptron function, and softmax() is the exponential normalization function;

[0034] In the last layer of the transformer module, the MLP network is used to calculate the spatial position coordinates, and the formula is:

[0035]

[0036] where f i T It is the last layer output of the transformer module;

[0037] Finally, the PnP method is used to establish a 2D-3D equation by mapping the spatial position coordinates and the feature map, and the pose estimation is obtained by solving it.

[0038] Furthermore, in the step of inputting the upsampled feature map into the FPN network to obtain a depth map estimation, the following steps are specifically included:

[0039] The feature map is transformed to the target frame coordinate system through homography, and the homography transformation matrix is:

[0040]

[0041] Where K i is the internal parameter matrix, R i is the posture, C i is the coordinate representation of the camera optical center in the world coordinate system, and d is the depth of the target frame;

[0042] The feature map after homography transformation is input into the FPN network, and the depth map estimation is obtained based on the scale information.

[0043] The present application provides a pose and depth map estimation system based on audio-visual knowledge distillation, comprising:

[0044] Acquisition module: acquires audio and video and performs preprocessing to obtain audio frame-image pairs;

[0045] Mapping module: mapping the sound wave information of the audio frame;

[0046] Output module: input the mapped sound wave information into the convolution layer for feature extraction, output the feature map and then perform upsampling;

[0047] Decoding module: The upsampled feature map is passed through the pose decoding module and FPN network to obtain pose estimation and depth map estimation respectively.

[0048] The present application provides a device, which includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a pose and depth map estimation method based on audio-visual knowledge distillation; the processor is used to execute the program instructions stored in the memory to implement pose and depth map estimation based on audio-visual knowledge distillation.

[0049] The present application provides a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute a pose and depth map estimation method based on audio-visual knowledge distillation.

[0050] The present application provides a method and system for estimating a pose and depth map based on audio-visual knowledge distillation, which has the following beneficial effects:

[0051] By acquiring audio and video and performing preprocessing, audio frame-image pairs for training are generated. After mapping the sound wave information of the audio frame, a common convolutional network is used as the backbone to extract and encode the feature map, and a transformer is used as a decoder for pose estimation, and an improved FPN network is used as a decoder for depth map estimation. The use of transformer as a decoder for pose estimation takes into account the temporal nature of pose estimation, and the use of FPN network as a decoder for depth map estimation takes into account the multi-scale features of depth map estimation, effectively improving the accuracy of pose estimation and depth estimation. The present application predicts the image in the audio frame-image pair through a mature visual network to generate corresponding pseudo-labels for learning the auditory network. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of a method for estimating a pose and depth map based on audio-visual knowledge distillation according to Embodiment 1 of the present application;

[0053] Figure 2 This is a schematic diagram of the convolutional coding operation of Example 1 of the present application;

[0054] Figure 3 This is a schematic diagram of posture estimation decoding of Example 1 of the present application;

[0055] Figure 4 This is a schematic diagram of depth estimation decoding according to Embodiment 1 of the present application;

[0056] Figure 5 This is a diagram of an algorithm framework for pose and depth map estimation based on audio-visual knowledge distillation according to Example 1 of the present application;

[0057] Figure 6 This is a structural schematic diagram of a method for estimating a pose and depth map based on audio-visual knowledge distillation according to Embodiment 2 of the present application;

[0058] Figure 7 A schematic diagram of the device structure provided in Example 3 of the present invention;

[0059] Figure 8 A schematic diagram of the storage medium structure provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0062] See also Figure 1 , which is a flowchart of a method for estimating a pose and depth map based on audio-visual knowledge distillation proposed in this application; the steps include:

[0063] S1: Acquire audio and video and perform preprocessing to obtain audio frame-image pairs.

[0064] In this embodiment, the acquired audio and video are segmented into units of two seconds. During training, frame pairs at the middle moments of the segments are usually selected as audio frame-image pairs.

[0065] S2: Mapping the sound wave information of the audio frame.

[0066] In this embodiment, the sound wave information of the audio frame is mapped to the corresponding frequency function f(t), loudness function l(t) and hidden code z(t);

[0067] The frequency function f(t) is extracted by the pre-trained CREPE model; the loudness function l(t) is extracted by the audio; the amplitude attenuation weight tensor corresponding to the frequency tensor is calculated by using the A-weighted power spectrum to emphasize higher frequencies; the weight tensor is then calculated in decibel scale, and the volume vector is normalized in the dataset to have the same scale on all samples; finally, the MFCCs are extracted and input into the normalization layer, the 512-unit CPU and the 512-unit linear layer to obtain the hidden code z(t).

[0068] S3: The mapped sound wave information is input into the convolution layer for feature extraction, and up-sampling is performed after the feature map is output.

[0069] See also Figure 2, which is a schematic diagram of the convolutional coding operation of Example 1 of the present application.

[0070] In this embodiment, the mapped sound wave information is fused after four 4*4 convolutions, and a 1*1 convolution with a filter number of 64 and a 3*3 convolution with expansion coefficients of 6, 12, and 18 are used, and finally a 1*1 convolution is used to obtain the feature map of the corresponding audio;

[0071] The size of the feature map output by the convolutional coding will be different from the final prediction estimate obtained by vision; therefore, it is necessary to use the bilinear interpolation method to upsample the feature map to achieve consistency with the input image size.

[0072] S4: The upsampled feature map is passed through the pose decoding module and the FPN network to obtain pose estimation and depth map estimation respectively.

[0073] In this embodiment, in the steps of obtaining pose estimation and depth map estimation respectively through the pose decoding module and the FPN network of the upsampled feature map, the following steps S41 to S42 are specifically included, and the implementation method of each step is described in detail below.

[0074] S41: Input the upsampled feature map into the pose decoding module to obtain pose estimation.

[0075] See also Figure 3 , which is a schematic diagram of posture estimation decoding of Example 1 of the present application.

[0076] The upsampled feature map is input into the pose decoding module as query, and the sparse grid code obtained by encoding the audio frame information at adjacent moments is used as key and value;

[0077] The formula for pose decoding is:

[0078]

[0079]

[0080] where f i t is the output of t transformer modules, A scaling factor to enhance sparsity, is the code encoding, CrAtt is the cross attention function;

[0081] The transformer multi-head attention formula is:

[0082]

[0083] f it+1 =MLP(softmax(Q·K)·V)

[0084] are the multi-head attention matrices at the current time step t, f i t+1 is the hidden variable in the middle layer of the network, MLP() is the multi-layer perceptron function, and softmax() is the exponential normalization function;

[0085] In the last layer of the transformer module, the MLP network is used to calculate the spatial position coordinates, and the formula is:

[0086]

[0087] where f i T It is the last layer output of the transformer module;

[0088] Finally, the PnP method is used to establish a 2D-3D equation by mapping the spatial position coordinates and the feature map, and the pose estimation is obtained by solving it.

[0089] S42: Input the upsampled feature map into the FPN network to obtain a depth map estimate.

[0090] See also Figure 4 , which is a schematic diagram of depth estimation decoding of Example 1 of the present application.

[0091] The feature map is transformed to the target frame coordinate system through homography, and the homography transformation matrix is:

[0092]

[0093] Where K i is the internal parameter matrix, R i is the posture, C i is the coordinate representation of the camera optical center in the world coordinate system, and d is the depth of the target frame;

[0094] The feature map after homography transformation is input into the FPN network, and the depth map estimation is obtained based on the scale information.

[0095] See also Figure 5 , is an algorithm framework diagram of pose and depth map estimation based on audio-visual knowledge distillation in Example 1 of the present application; the specific contents are as follows:

[0096] The original audio is encoded by differentiable digital signal processing (DDSP) to obtain the corresponding frequency function f(t), loudness function l(t) and latent code z(t), and then input into a convolutional network for further feature extraction to obtain a feature map; the feature map is decoded by Transformer to obtain spatial mapping three-dimensional points and pyramid network (FPN) to obtain depth map estimation; the spatial mapping three-dimensional points obtained by Transformer decoding need to be further transformed by PnP to obtain the final pose estimation; at the same time, the predicted estimate generated by the audio needs to be compared with the label generated by the image frame aligned with the audio time through the visual depth network (depthNet) and the pose network (poseNet), and the comparison error is fed back for training, and finally a network that can generate corresponding depth and pose estimates by inputting audio signals is obtained.

[0097] In summary, Example 1 of the present application obtains audio and video and performs preprocessing to generate audio frame-image pairs for training, maps the sound wave information of the audio frame, and then uses a common convolutional network as the backbone to extract and encode the feature map, and uses the transformer as a decoder for pose estimation, and the improved FPN network as a decoder for depth map estimation; the use of the transformer as the decoder for pose estimation takes into account the temporal nature of the pose estimation, and the use of the FPN network as the decoder for depth map estimation takes into account the multi-scale features of the depth map estimation, effectively improving the accuracy of pose estimation and depth estimation; the present application predicts the image in the audio frame-image pair through a mature visual network to generate corresponding pseudo-labels for learning the auditory network.

[0098] Example 2

[0099] See also Figure 6 , which is a structural diagram of a method for estimating a pose and depth map based on audio-visual knowledge distillation according to Embodiment 2 of the present application; comprising:

[0100] Acquisition module: acquires audio and video and performs preprocessing to obtain audio frame-image pairs;

[0101] Mapping module: mapping the sound wave information of the audio frame;

[0102] Output module: input the mapped sound wave information into the convolution layer for feature extraction, output the feature map and then perform upsampling;

[0103] Decoding module: The upsampled feature map is passed through the pose decoding module and FPN network to obtain pose estimation and depth map estimation respectively.

[0104] In summary, Example 2 of the present application obtains audio and video through an acquisition module, and obtains an audio frame-image pair after preprocessing; the sound wave information of the audio frame is mapped and input into the convolution layer for feature extraction, and a feature map is output; the transformer in the decoding module is used as a decoder for pose estimation, and the improved FPN network is used as a decoder for depth map estimation; the use of transformer as a decoder for pose estimation takes into account the temporal nature of pose estimation, and the use of FPN network as a decoder for depth map estimation takes into account the multi-scale features of depth map estimation, effectively improving the accuracy of pose estimation and depth estimation; the present application predicts the image in the audio frame-image pair through a mature visual network to generate corresponding pseudo-labels for learning the auditory network.

[0105] Example 3

[0106] See also Figure 7 , is a schematic diagram of the device structure of Embodiment 3 of the present application. The device 50 includes a processor 51 and a memory 52 coupled to the processor 51 .

[0107] The memory 52 stores program instructions for implementing the above-mentioned method for estimating a pose and depth map based on audio-visual knowledge distillation.

[0108] The processor 51 is used to execute program instructions stored in the memory 52 to implement pose and depth map estimation based on audio-visual knowledge distillation.

[0109] The processor 51 may also be referred to as a CPU (Central Processing Unit).

[0110] The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0111] Example 4

[0112] See also Figure 8, which is a schematic diagram of the structure of the storage medium of Example 4 of the present application. The storage medium of the embodiment of the present application stores a program file 61 that can implement all the above methods, wherein the program file 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a computer, a server, a mobile phone, a tablet, and other devices.

[0113] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0114] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

[0115] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the appended claims and their equivalents.

[0116] Of course, the present invention may have many other implementations. Based on this implementation, other implementations obtained by ordinary technicians in this field without any creative work are all within the scope of protection of the present invention.

Claims

1. A pose and depth map estimation method based on audio-visual knowledge distillation, characterized in that: include: Acquire audio and video and perform preprocessing to obtain audio frame-image pairs; Mapping the sound wave information of the audio frame; The mapped sound wave information is input into the convolution layer for feature extraction, and up-sampled after outputting the feature map; The upsampled feature map is passed through the pose decoding module and the FPN network to obtain pose estimation and depth map estimation respectively.

2. The method for estimating a pose and depth map based on audio-visual knowledge distillation according to claim 1, characterized in that: The step of mapping the sound wave information of the audio frame specifically includes the following steps: The sound wave information of the audio frame is mapped into a corresponding frequency function f(t), a loudness function l(t) and a hidden code z(t); The frequency function f(t) is extracted by the pre-trained CREPE model; the loudness function l(t) is extracted by the audio; the amplitude attenuation weight tensor corresponding to the frequency tensor is calculated using the A-weighted power spectrum, and then the weight tensor is calculated in decibel scale, and the volume vector is normalized in the data set, and finally MFCCs are extracted and input into the normalization layer, the 512-unit CPU and the 512-unit linear layer to obtain the hidden code z(t).

3. The method for estimating a pose and depth map based on audio-visual knowledge distillation according to claim 1, characterized in that: The steps of inputting the mapped sound wave information into the convolution layer for feature extraction and upsampling after outputting the feature map specifically include the following steps: The mapped sound wave information is fused after four 4*4 convolutions, and a 1*1 convolution with 64 filters and a 3*3 convolution with expansion coefficients of 6, 12, and 18 are used. Finally, a 1*1 convolution is used to obtain the feature map of the corresponding audio. The feature map is upsampled using a bilinear interpolation method.

4. The method for estimating a pose and depth map based on audio-visual knowledge distillation according to claim 1, characterized in that: In the step of obtaining pose estimation and depth map estimation respectively through the pose decoding module and the FPN network of the upsampled feature map, the following steps are specifically included: The upsampled feature map is input into the pose decoding module to obtain the pose estimation; The upsampled feature map is input into the FPN network to obtain the depth map estimation.

5. The method for estimating a pose and depth map based on audio-visual knowledge distillation according to claim 4, characterized in that: In the step of inputting the upsampled feature map into the pose decoding module to obtain the pose estimation, the following steps are specifically included: The upsampled feature map is input into the pose decoding module as a query, and the sparse grid code obtained by encoding the audio frame information at adjacent moments is used as the key and value; The formula for pose decoding is: where f i t is the output of t transformer modules, A scaling factor to enhance sparsity, is the code encoding, CrAtt is the cross attention function; The transformer multi-head attention formula is: f i t+1 =MLP(softmax(Q·K)·V) are the multi-head attention matrices at the current time step t, f i t+1 is the hidden variable in the middle layer of the network, MLP() is the multi-layer perceptron function, and softmax() is the exponential normalization function; In the last layer of the transformer module, the MLP network is used to calculate the spatial position coordinates, and the formula is: where f i T It is the last layer output of the transformer module; Finally, the PnP method is used to establish a 2D-3D equation by mapping the spatial position coordinates and the feature map, and the pose estimation is obtained by solving it.

6. The method for estimating a pose and depth map based on audio-visual knowledge distillation according to claim 4, characterized in that: In the step of inputting the upsampled feature map into the FPN network to obtain the depth map estimation, the following steps are specifically included: The feature map is transformed to the target frame coordinate system through homography, and the homography transformation matrix is: Where K i is the internal parameter matrix, R i is the posture, C i is the coordinate representation of the camera optical center in the world coordinate system, and d is the depth of the target frame; The feature map after homography transformation is input into the FPN network, and the depth map estimation is obtained based on the scale information.

7. A system for a pose and depth map estimation method based on audio-visual knowledge distillation according to any one of claims 1 to 6, comprising: Acquisition module: acquires audio and video and performs preprocessing to obtain audio frame-image pairs; Mapping module: mapping the sound wave information of the audio frame; Output module: input the mapped sound wave information into the convolution layer for feature extraction, output the feature map and then perform upsampling; Decoding module: The upsampled feature map is passed through the pose decoding module and FPN network to obtain pose estimation and depth map estimation respectively.

8. A device, characterized in that: The device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a pose and depth map estimation method based on audio-visual knowledge distillation as described in any one of claims 1-6; the processor is used to execute the program instructions stored in the memory to implement pose and depth map estimation based on audio-visual knowledge distillation.

9. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute a pose and depth map estimation method based on audio-visual knowledge distillation as described in any one of claims 1-6.

Images