Pose and depth map estimation method and system based on audiovisual knowledge distillation

Abstract

The application discloses a pose and depth map estimation method and system based on audio-visual knowledge distillation, which is applied to the field of audio-visual multi-modal technology, and the method comprises the following steps: acquiring audio and video and performing pretreatment to obtain an audio frame-image pair; mapping sound wave information of the audio frame; inputting the mapped sound wave information into a convolution layer to perform feature extraction, and performing up-sampling after outputting a feature map; and obtaining pose estimation and depth map estimation through a pose decoding module and an FPN network respectively after the up-sampling of the feature map; and the application generates corresponding pseudo labels for the learning of an auditory network by predicting the image in the audio frame-image pair through a mature visual network.

Description

Technical Field

[0001] This invention relates to the field of audiovisual multimodal technology, and in particular to a pose and depth map estimation method and system based on audiovisual knowledge distillation. Background Technology

[0002] Knowledge distillation is a technique for training neural networks in which the knowledge of a large “teacher” model is transferred to a smaller “student” model. The goal of this process is to enable the student model to learn to mimic the behavior and predictive abilities of the teacher model.

[0003] Pose estimation and depth map estimation are two important tasks in the field of computer vision. Pose estimation refers to determining the position and orientation of an object or scene in three-dimensional space by extracting features from image or sensor data. Depth map estimation refers to inferring the depth information of each pixel in a scene from image or sensor data.

[0004] The main representatives of existing deep network models for audiovisual learning are SoundNet and Binaural SoundNet.

[0005] SoundNet jointly trains its proposed auditory network with ImageNet, PlaceCNN, and other datasets. First, each video in the video dataset is segmented into two parts: audio and RGB image frames. The visual recognition network uses a pre-trained VGG model on two large image datasets: ImageNet and Places. ImageNet and Places are both large datasets in the field of image recognition; ImageNet is a dataset for object classification, and Places is a dataset for scene classification. The RGB frames segmented from the video are input into the pre-trained VGG model, and the output is used as supervision information for the sound recognition network. The sound recognition network employs an 8-layer fully convolutional structure, using the audio time series extracted from the video as the network input, and employing KL-divergence as the loss function.

[0006] Binaural SoundNet trains the auditory network jointly with DeepLabv3+, MonoDepth2, and FlowNet2 models. Unlike SoundNet, which divides video data into audio and video data, it uses a separate multi-directional microphone device for sound reception during initial data collection to obtain the three main cues humans use to locate sound sources: interaural time difference (ITD), interaural sound level difference (ILD), and head-related transfer function (HRTF). Therefore, the auditory network can obtain more prior information during training, thereby improving network performance.

[0007] However, existing auditory perception networks only utilize convolutional modules, neglecting the temporal nature of pose estimation and the scale features of depth map estimation. By using transformer and FPN in the decoder part in this application, these shortcomings can be improved. Summary of the Invention

[0008] The purpose of this application is to provide a pose and depth map estimation method and system based on audiovisual knowledge distillation, which aims to solve the problem that existing vision methods have great limitations in dealing with occlusion problems, and the prediction and estimation accuracy will drop significantly under occlusion conditions.

[0009] To achieve the above objectives, this application provides the following technical solution:

[0010] This application provides a pose and depth map estimation method based on audiovisual knowledge distillation, including:

[0011] Acquire audio and video data and preprocess them to obtain audio frame-image pairs;

[0012] Map the sound wave information of the audio frame;

[0013] The mapped acoustic wave information is input into a convolutional layer for feature extraction, and the output feature map is then upsampled.

[0014] After upsampling, the feature map is used to obtain pose estimation and depth map estimation through the pose decoding module and FPN network, respectively.

[0015] Furthermore, the step of mapping the sound wave information of the audio frame specifically includes the following steps:

[0016] The sound wave information of the audio frame is mapped to the corresponding frequency function f(t), loudness function l(t), and hidden code z(t);

[0017] The frequency function f(t) is extracted using a pre-trained CREPE model; the loudness function l(t) is extracted using the audio; the amplitude attenuation weight tensor corresponding to the frequency tensor is calculated using A-weighted power spectrum, and then the weight tensor is calculated using a decibel scale. The volume vector is normalized in the dataset, and finally the MFCCs are extracted and input to the normalization layer, the 512-unit CPU, and the 512-unit linear layer to obtain the hidden code z(t).

[0018] Furthermore, the step of upsampling the output feature map after inputting the mapped acoustic wave information into the convolutional layer for feature extraction specifically includes the following steps:

[0019] The mapped sound wave information is fused after passing through four 4*4 convolutions, and then a 1*1 convolution with 64 filters is used, followed by 3*3 convolutions with dilation coefficients of 6, 12, and 18. Finally, a 1*1 convolution is used to obtain the feature map of the corresponding audio.

[0020] The feature map is upsampled using a bilinear interpolation method.

[0021] Furthermore, the steps of obtaining pose estimation and depth map estimation from the upsampled feature map through the pose decoding module and FPN network respectively include the following steps:

[0022] The upsampled feature map is input into the pose decoding module to obtain the pose estimate;

[0023] The upsampled feature map is input into the FPN network to obtain a depth map estimate.

[0024] Furthermore, the step of inputting the upsampled feature map into the pose decoding module to obtain the pose estimate specifically includes the following steps:

[0025] The upsampled feature map is input into the pose decoding module as a query, and the sparse grid code is obtained by encoding the audio frame information of adjacent time steps and used as the key and value.

[0026] The formula for pose decoding is:

[0027]

[0028]

[0029] Where f i t For the output after passing through t transformer modules, To enhance sparsity scaling factor, For code encoding, CrAtt is the cross-attention function;

[0030] The transformer multi-head attention formula is:

[0031]

[0032] f i t+1 =MLP(softmax(Q·K)·V)

[0033] These are the multi-head attention matrices at the current time step t, f i t+1For hidden variables in the intermediate layers of the network, MLP() is the multilayer perceptron function, and softmax() is the exponential normalization function;

[0034] In the last layer of the transformer module, the spatial coordinates are calculated using an MLP network, with the following formula:

[0035]

[0036] Where f i T This is the output of the last layer of the transformer module;

[0037] Finally, the PnP method is used to establish a 2D-3D equation by mapping spatial position coordinates and feature maps, and the pose estimate is obtained by solving the equation.

[0038] Furthermore, the step of inputting the upsampled feature map into the FPN network to obtain the depth map estimate specifically includes the following steps:

[0039] The feature map is homography transformed to the target frame coordinate system. The homography transformation matrix is:

[0040]

[0041] Where K i Let R be the intrinsic parameter matrix. i For the pose, C i Let d be the coordinate representation of the camera's optical center in the world coordinate system, and d be the depth of the target frame.

[0042] The feature map after homography is input into the FPN network, and the depth map is estimated based on the scale information.

[0043] This application provides a pose and depth map estimation system based on audiovisual knowledge distillation, including:

[0044] Acquisition module: Acquires audio and video and performs preprocessing to obtain audio frame-image pairs;

[0045] Mapping module: Maps the sound wave information of the audio frame;

[0046] Output module: Inputs the mapped acoustic wave information into the convolutional layer for feature extraction, and then upsamples the output feature map;

[0047] Decoding module: The upsampled feature map is processed by the pose decoding module and the FPN network to obtain pose estimation and depth map estimation, respectively.

[0048] This application provides an apparatus comprising a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a pose and depth map estimation method based on audiovisual knowledge distillation; the processor is configured to execute the program instructions stored in the memory to implement pose and depth map estimation based on audiovisual knowledge distillation.

[0049] This application provides a storage medium storing processor-executable program instructions for executing a pose and depth map estimation method based on audiovisual knowledge distillation.

[0050] This application provides a pose and depth map estimation method and system based on audiovisual knowledge distillation, which has the following beneficial effects:

[0051] By acquiring and preprocessing audio and video, audio frame-image pairs are generated for training. After mapping the sound wave information of the audio frames, a common convolutional network is used as the backbone to extract and encode feature maps. A transformer is used as the decoder for pose estimation, and an improved FPN network is used as the decoder for depth map estimation. Using a transformer as the decoder for pose estimation takes into account the temporal nature of pose estimation, while using an FPN network as the decoder for depth map estimation takes into account the multi-scale features of depth map estimation, effectively improving the accuracy of pose estimation and depth estimation. This application uses a mature visual network to predict the images in the audio frame-image pairs to generate corresponding pseudo-labels for the learning of the auditory network. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a pose and depth map estimation method based on audiovisual knowledge distillation according to Embodiment 1 of this application.

[0053] Figure 2 This is a schematic diagram of the convolutional coding operation in Embodiment 1 of this application;

[0054] Figure 3 This is a schematic diagram of pose estimation decoding in Embodiment 1 of this application;

[0055] Figure 4 This is a schematic diagram of depth estimation decoding in Embodiment 1 of this application;

[0056] Figure 5 This is a schematic diagram of the algorithm framework for pose and depth map estimation based on audiovisual knowledge distillation in Embodiment 1 of this application;

[0057] Figure 6 This is a schematic diagram of the structure of a pose and depth map estimation method based on audiovisual knowledge distillation according to Embodiment 2 of this application;

[0058] Figure 7 This is a schematic diagram of the device structure provided in Embodiment 3 of the present invention;

[0059] Figure 8 This is a schematic diagram of the storage medium structure provided in Embodiment 4 of the present invention. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0062] Please see Figure 1 This is a flowchart illustrating a pose and depth map estimation method based on audiovisual knowledge distillation proposed in this application; the steps include:

[0063] S1: Acquire audio and video and preprocess them to obtain audio frame-image pairs.

[0064] In this embodiment, the acquired audio and video are segmented in units of two seconds. During training, the frame pairs at the middle of the segment are usually selected as audio frame-image pairs.

[0065] S2: Map the sound wave information of the audio frame.

[0066] In this embodiment, the sound wave information of the audio frame is mapped to the corresponding frequency function f(t), loudness function l(t), and hidden code z(t);

[0067] The frequency function f(t) is extracted using a pre-trained CREPE model; the loudness function l(t) is extracted using the audio data; the amplitude attenuation weight tensor corresponding to the frequency tensor is calculated using A-weighted power spectrum, which can emphasize higher frequencies; then the weight tensor is calculated on a decibel scale, and the volume vector is normalized in the dataset to make it have the same scale across all samples; finally, the MFCCs are extracted and input to the normalization layer, the 512-unit CPU, and the 512-unit linear layer to obtain the hidden code z(t).

[0068] S3: Input the mapped acoustic wave information into the convolutional layer for feature extraction, and then perform upsampling on the output feature map.

[0069] Please see Figure 2This is a schematic diagram of the convolutional coding operation in Embodiment 1 of this application.

[0070] In this embodiment, the mapped sound wave information is fused after passing through four 4*4 convolutions, and then a 1*1 convolution with 64 filters is used, followed by 3*3 convolutions with dilation coefficients of 6, 12, and 18. Finally, a 1*1 convolution is used to obtain the feature map of the corresponding audio.

[0071] The size of the feature map output by convolutional coding will differ from the size of the final prediction obtained by vision; therefore, bilinear interpolation is required to upsample the feature map to make it consistent with the size of the input image.

[0072] S4: The upsampled feature map is used to obtain pose estimation and depth map estimation through the pose decoding module and FPN network, respectively.

[0073] In this embodiment, the steps of obtaining pose estimation and depth map estimation by the pose decoding module and FPN network after upsampling of the feature map specifically include the following steps S41 to S42. The implementation of each step is described in detail below.

[0074] S41: Input the upsampled feature map into the pose decoding module to obtain the pose estimate.

[0075] Please see Figure 3 This is a schematic diagram of pose estimation decoding in Embodiment 1 of this application.

[0076] The upsampled feature map is input into the pose decoding module as a query, and the sparse grid code is obtained by encoding the audio frame information of adjacent time steps and used as the key and value.

[0077] The formula for pose decoding is:

[0078]

[0079]

[0080] Where f i t For the output after passing through t transformer modules, To enhance sparsity scaling factor, For code encoding, CrAtt is the cross-attention function;

[0081] The transformer multi-head attention formula is:

[0082]

[0083] f it+1 =MLP(softmax(Q·K)·V)

[0084] These are the multi-head attention matrices at the current time step t, f i t+1 For hidden variables in the intermediate layers of the network, MLP() is the multilayer perceptron function, and softmax() is the exponential normalization function;

[0085] In the last layer of the transformer module, the spatial coordinates are calculated using an MLP network, with the following formula:

[0086]

[0087] Where f i T This is the output of the last layer of the transformer module;

[0088] Finally, the PnP method is used to establish a 2D-3D equation by mapping spatial position coordinates and feature maps, and the pose estimate is obtained by solving the equation.

[0089] S42: Input the upsampled feature map into the FPN network to obtain a depth map estimate.

[0090] Please see Figure 4 This is a schematic diagram of depth estimation decoding in Embodiment 1 of this application.

[0091] The feature map is homography transformed to the target frame coordinate system. The homography transformation matrix is:

[0092]

[0093] Where K i Let R be the intrinsic parameter matrix. i For the pose, C i Let d be the coordinate representation of the camera's optical center in the world coordinate system, and d be the depth of the target frame.

[0094] The feature map after homography is input into the FPN network, and the depth map is estimated based on the scale information.

[0095] Please see Figure 5 This is a diagram illustrating the algorithm framework for pose and depth map estimation based on audiovisual knowledge distillation in Embodiment 1 of this application; the specific content is as follows:

[0096] The original audio is encoded using Differentiable Digital Signal Processing (DDSP) to represent the corresponding frequency function f(t), loudness function l(t), and hidden code z(t). This is then input into a convolutional network for further feature extraction to obtain a feature map. The feature map is then decoded using a Transformer to obtain spatially mapped 3D points, and a pyramid network (FPN) is used to obtain a depth map estimate. The spatially mapped 3D points obtained after Transformer decoding require further PnP transformation to obtain the final pose estimate. Simultaneously, the predicted estimate generated from the audio is compared with labels generated from image frames aligned with the audio time using a visual depth network (depthNet) and a pose network (poseNet). The comparison error is then backpropagated for training, ultimately resulting in a network that can generate corresponding depth and pose estimates from the input audio signal.

[0097] In summary, Embodiment 1 of this application acquires and preprocesses audio and video to generate audio frame-image pairs for training. After mapping the sound wave information of the audio frames, a common convolutional network is used as the backbone to extract and encode feature maps. A transformer is used as the decoder for pose estimation, and an improved FPN network is used as the decoder for depth map estimation. Using a transformer as the decoder for pose estimation takes into account the temporal nature of pose estimation, while using an FPN network as the decoder for depth map estimation takes into account the multi-scale features of depth map estimation, effectively improving the accuracy of pose estimation and depth estimation. This application uses a mature visual network to predict the images in the audio frame-image pairs to generate corresponding pseudo-labels for the learning of the auditory network.

[0098] Example 2

[0099] Please see Figure 6 This is a schematic diagram of a pose and depth map estimation method based on audiovisual knowledge distillation according to Embodiment 2 of this application; including:

[0100] Acquisition module: Acquires audio and video and performs preprocessing to obtain audio frame-image pairs;

[0101] Mapping module: Maps the sound wave information of the audio frame;

[0102] Output module: Inputs the mapped acoustic wave information into the convolutional layer for feature extraction, and then upsamples the output feature map;

[0103] Decoding module: The upsampled feature map is processed by the pose decoding module and the FPN network to obtain pose estimation and depth map estimation, respectively.

[0104] In summary, Embodiment 2 of this application acquires audio and video through an acquisition module, and obtains audio frame-image pairs after preprocessing. The sound wave information of the audio frames is mapped and input into a convolutional layer for feature extraction, outputting a feature map. The decoding module uses a transformer as a decoder for pose estimation and an improved FPN network as a decoder for depth map estimation. Using a transformer as the pose estimation decoder considers the temporal nature of pose estimation, while using an FPN network as the depth map estimation decoder considers the multi-scale features of depth map estimation, effectively improving the accuracy of both pose and depth estimation. This application also uses a mature visual network to predict images in the audio frame-image pairs, generating corresponding pseudo-labels for the learning of the auditory network.

[0105] Example 3

[0106] Please see Figure 7 This is a schematic diagram of the device structure in Embodiment 3 of this application. The device 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0107] The memory 52 stores program instructions for implementing the above-described pose and depth map estimation method based on audiovisual knowledge distillation.

[0108] The processor 51 is used to execute program instructions stored in the memory 52 to implement pose and depth map estimation based on audiovisual knowledge distillation.

[0109] The processor 51 can also be referred to as a CPU (Central Processing Unit).

[0110] Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0111] Example 4

[0112] Please see Figure 8This is a schematic diagram of the storage medium in Embodiment 4 of this application. The storage medium in this embodiment stores a program file 61 capable of implementing all the above methods. This program file 61 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or devices such as computers, servers, mobile phones, and tablets.

[0113] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0114] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0115] Although embodiments of this application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the appended claims and their equivalents.

[0116] Of course, the present invention may have many other embodiments. Based on this embodiment, other embodiments obtained by those skilled in the art without any creative effort are all within the scope of protection of the present invention.

Claims

1. A pose and depth map estimation method based on audiovisual knowledge distillation, characterized in that, include: Acquire audio and video data and preprocess them to obtain audio frame-image pairs; The prediction estimates generated from the audio need to be compared with the labels generated by the visual depth network and pose network from the image frames aligned with the audio time. The comparison error is then backpropagated for training to obtain a network that generates the corresponding depth and pose estimates by inputting the audio signal. Map the sound wave information of the audio frame; The mapped acoustic wave information is input into a convolutional layer for feature extraction, and the output feature map is then upsampled. After upsampling, the feature map is used to obtain pose estimation and depth map estimation through the pose decoding module and FPN network, respectively. The step of mapping the sound wave information of the audio frame specifically includes the following steps: The sound wave information of the audio frame is mapped to the corresponding frequency function. Loudness function and implicit coding ; The frequency function It is obtained through a pre-trained CREPE model; the loudness function The hidden code is obtained through audio extraction; the amplitude attenuation weight tensor corresponding to the frequency tensor is calculated using A-weighted power spectrum, and then the weight tensor is calculated on a decibel scale. The volume vector is normalized in the dataset, and finally, MFCCs are extracted and input to the normalization layer, the 512-unit CPU, and the 512-unit linear layer to obtain the hidden code. ; The process of inputting the mapped acoustic wave information into a convolutional layer for feature extraction, and then upsampling the output feature map, specifically includes the following steps: The mapped sound wave information is fused after passing through four 4*4 convolutions, and then a 1*1 convolution with 64 filters is used, followed by 3*3 convolutions with dilation coefficients of 6, 12, and 18. Finally, a 1*1 convolution is used to obtain the feature map of the corresponding audio. The feature map is upsampled using a bilinear interpolation method. The process of obtaining pose estimation and depth map estimation from the upsampled feature map through the pose decoding module and FPN network, respectively, specifically includes the following steps: The upsampled feature map is input into the pose decoding module to obtain the pose estimate; The upsampled feature map is input into the FPN network to obtain a depth map estimate; The step of inputting the upsampled feature map into the pose decoding module to obtain the pose estimate specifically includes the following steps: The upsampled feature map is input into the pose decoding module as a query, and the sparse grid code is obtained by encoding the audio frame information of adjacent time steps and used as the key and value. The formula for pose decoding is: in For the output after passing through t transformer modules, To enhance sparsity, Encode the code. For cross-attention functions; The transformer multi-head attention formula is: These are the bullish attention matrices at the current time step t. For implicit variables in the intermediate layer of the network, For multilayer perceptron functions, It is the exponential normalization function; In the last layer of the transformer module, the spatial coordinates are calculated using an MLP network, with the following formula: in This is the output of the last layer of the transformer module; Finally, the PnP method is used to establish 2D-3D equations by mapping spatial position coordinates and feature maps, and the pose estimation is obtained by solving the equations. The process of inputting the upsampled feature map into the FPN network to obtain the depth map estimate specifically includes the following steps: The feature map is homography transformed to the target frame coordinate system. The homography transformation matrix is: in This is the intrinsic parameter matrix. For position, Let be the coordinate representation of the camera's optical center in the world coordinate system. The depth of the target frame; The feature map after homography is input into the FPN network, and the depth map is estimated based on the scale information.

2. A system for a pose and depth map estimation method based on audiovisual knowledge distillation according to claim 1, comprising: Acquisition module: Acquires audio and video and performs preprocessing to obtain audio frame-image pairs; Mapping module: Maps the sound wave information of the audio frame; Output module: Inputs the mapped acoustic wave information into the convolutional layer for feature extraction, and then upsamples the output feature map; Decoding module: The upsampled feature map is processed by the pose decoding module and the FPN network to obtain pose estimation and depth map estimation, respectively.

3. A device, characterized in that, The device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing the pose and depth map estimation method based on audiovisual knowledge distillation as described in claim 1; the processor is used to execute the program instructions stored in the memory to implement pose and depth map estimation based on audiovisual knowledge distillation.

4. A storage medium, characterized in that, The device stores processor-executable program instructions for performing the pose and depth map estimation method based on audiovisual knowledge distillation as described in claim 1.

Images