Audio and video processing method and device, audio and video equipment and computer storage medium

Abstract

The invention relates to an audio and video processing method and device, audio and video equipment and a computer storage medium, and the method comprises the steps: obtaining sound data and video data, carrying out the target recognition of the video data, determining the image types, carrying out the audio separation of the sound data based on the image types, and determining the audio data corresponding to each image type; and according to the spatial position information of each image category and the corresponding audio data, a spatial audio is obtained, and the spatial audio is used for playing in combination with the video data or playing independently. Therefore, the spatial audio obtained by combining the image category and the spatial position information can accurately correspond to different categories of images and restore the sound orientation during recording, so that scene restoration and sound restoration can be realized when the spatial audio is played, and the reliability of spatial audio playing is improved.

Description

Technical Field

[0001] The present application relates to the field of audio and video technology, and in particular to an audio and video processing method, device, audio and video equipment, and computer storage medium. Background Art

[0002] With the development of audio and video technology, the trend of future development has shifted from the traditional audio processing from the sound dimension to the integration of audio and video with the video dimension.

[0003] In the traditional audio and video production process, from shooting the video to playing the video, the sound is recorded and played in stereo. The recorded sound is mixed with the sound of real life when recording on the recording end, and the sound is played back according to the original recorded sound when it is played back on the playback end.

[0004] However, in this case, all the sounds are mixed together during playback, and the degree of restoration of the original recorded sounds is not high. Summary of the invention

[0005] Based on this, it is necessary to provide an audio and video processing method, device, audio and video equipment and computer storage medium that can improve the degree of restoration of recorded sound in response to the above technical problems.

[0006] In a first aspect, the present application provides an audio and video processing method, the method comprising:

[0007] Acquiring audio and video data; the audio and video data includes sound data and video data;

[0008] Performing target recognition on the video data to determine the image category;

[0009] Performing audio separation on the sound data based on the image categories to determine audio data corresponding to each image category;

[0010] Spatial audio is obtained according to the spatial position information of each of the image categories and the corresponding audio data; the spatial audio is used for playing in combination with the video data or playing alone.

[0011] In one embodiment,

[0012] The obtaining of spatial audio according to the spatial position information of each image category and the corresponding audio data includes:

[0013] Acquire spatial position information of each of the image categories;

[0014] Inputting the spatial position information of each of the image categories as a key parameter into a head transfer function to obtain a head-related impulse response;

[0015] Spatial audio is obtained according to the head-related impulse response and the audio data corresponding to each of the image categories.

[0016] In one embodiment, the head-related impulse response includes a head-related impulse response to a left ear and a head-related impulse response to a right ear, the spatial audio includes an output audio signal of the left ear and an output audio signal of the right ear, and obtaining the spatial audio according to the head-related impulse response and the audio data corresponding to each of the image categories includes:

[0017] Convolving the audio data corresponding to each of the image categories with the head-related impulse response to the left ear, and then performing a downmixing algorithm to obtain an output audio signal for the left ear;

[0018] The audio data corresponding to each of the image categories and the head-related impulse response to the right ear are convolved respectively, and then a down-mixing algorithm is performed to obtain an output audio signal for the right ear.

[0019] In one embodiment, the performing target recognition on the video data to determine the image category includes:

[0020] The video data is input into an image recognition model to obtain the image category of each image in the video data; the image recognition model is trained based on image samples of each category.

[0021] In one embodiment, performing audio separation on the sound data based on the image category to determine the audio data corresponding to each image category includes:

[0022] The identified image categories and the sound data are input into a sound separation model to obtain audio data corresponding to each image category; the sound separation model is obtained by training based on image samples of each category and corresponding sounds.

[0023] In one embodiment, obtaining spatial audio according to the spatial position information of each image category and the corresponding audio data includes:

[0024] Obtaining intermediate audio data according to the spatial position information of each of the image categories and the corresponding audio data;

[0025] A reverberation parameter is added to the intermediate audio data to obtain spatial audio.

[0026] In one of the embodiments, after obtaining the spatial audio according to the spatial position information of each of the image categories and the corresponding audio data, the method further includes:

[0027] The angle in the spatial audio is adjusted according to the deviation angle of the user's head, so that the playback position of the adjusted spatial audio does not change with the deviation of the user's head.

[0028] In one of the embodiments, after obtaining the spatial audio according to the spatial position information of each of the image categories and the corresponding audio data, the method further includes:

[0029] Share the spatial audio.

[0030] In one embodiment, the spatial position information is the position information of each of the image categories in the video data, or,

[0031] The spatial position information is user-defined position information.

[0032] In a second aspect, the present application also provides an audio and video processing device. The device comprises:

[0033] An audio and video acquisition module, used to acquire audio and video data; the audio and video data includes sound data and video data;

[0034] An image recognition module, used to perform target recognition on the video data and determine the image category;

[0035] A sound separation module, used to perform audio separation on the sound data based on the image category, and determine the audio data corresponding to each image category;

[0036] The spatial audio reconstruction module is used to obtain spatial audio according to the spatial position information of each image category and the corresponding audio data; the spatial audio is used to be played in combination with the video data or played alone.

[0037] In a third aspect, the present application also provides an audio and video device, including a recorder and a processor, wherein the recorder is connected to the processor, the recorder is used to record audio and video data and send it to the processor, and the processor is used to perform audio and video processing according to the method of any of the above embodiments.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0039] Acquiring audio and video data; the audio and video data includes sound data and video data;

[0040] Performing target recognition on the video data to determine the image category;

[0041] Performing audio separation on the sound data based on the image categories to determine audio data corresponding to each image category;

[0042] Spatial audio is obtained according to the spatial position information of each of the image categories and the corresponding audio data; the spatial audio is used for playing in combination with the video data or playing alone.

[0043] The above-mentioned audio and video processing method, device, audio and video equipment and computer storage medium, after acquiring sound data and video data, perform target recognition on the video data, determine the image category, perform audio separation on the sound data based on the image category, determine the audio data corresponding to each image category, and then obtain spatial audio based on the spatial position information of each image category and the corresponding audio data. The spatial audio is used to be played in combination with video data or played alone. Therefore, the spatial audio obtained by combining the image category and spatial position information can accurately correspond to images of different categories, restore the sound orientation during recording, so that scene restoration and sound restoration can be achieved when playing the spatial audio, thereby improving the reliability of spatial audio playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 1 is a flow chart of an audio and video processing method in one embodiment;

[0045] Figure 2 is a schematic diagram of the location of spatial location information of an image category in one embodiment;

[0046] Figure 3 is a schematic flow chart of a step of obtaining spatial audio in one embodiment;

[0047] Figure 4 is a schematic flow chart of a process of constructing a head transfer function in one embodiment;

[0048] Figure 5 A schematic diagram of a flow chart of a step of obtaining spatial audio in another embodiment;

[0049] Figure 6 A schematic diagram of a flow chart of an audio and video processing method in another embodiment;

[0050] Figure 7 A schematic diagram of a flow chart of an audio and video processing method in another embodiment;

[0051] Figure 8 Detailed flowchart of an audio and video processing method in one embodiment;

[0052] Fig. 9 A schematic diagram of an application scenario of a head transfer function in an embodiment;

[0053] Fig.10 FIG. 4 is a structural block diagram of an audio and video processing device in an embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] The audio and video processing method provided in the embodiment of the present application is used to process the audio and video data collected by the device to form spatial audio. Among them, the device for collecting audio and video data can be a single device, for example, a device capable of taking a video such as a mobile phone, a camera, glasses, etc. can be used to record video, and a built-in microphone can be used to record sound. Alternatively, the device for collecting audio and video data can also be two or more devices, for example, a device carrying a camera such as a mobile phone or glasses can be used to record video, and a wearable device (such as headphones, glasses, etc.) can be used to record sound.

[0056] The audio and video processing method can be executed by a server or a terminal. After the device collects the audio and video data, it is sent to the server or the terminal, and the server or the terminal processes the audio and video data to form spatial audio. Among them, the terminal can be a device for collecting audio and video data, or it can be other devices. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers.

[0057] In one embodiment, Figure 1 As shown, an audio and video processing method is provided, which is described by taking the method applied to a terminal as an example, and includes the following steps:

[0058] Step 102, obtaining audio and video data.

[0059] The audio and video data includes audio data and video data. The audio data is the data recorded when recording the video.

[0060] After the device collects the audio and video data, it sends it to the server or terminal, and the server or terminal obtains the audio and video data. The sound data may be a mono audio file or a dual-channel audio file. Among them, the device for collecting audio and video data can be a single device, for example, a device that can record video such as a mobile phone, camera, glasses, etc. can be used to record video, and its built-in microphone can be used to record sound. Alternatively, the device for collecting audio and video data can also be two or more devices, for example, a device with a camera such as a mobile phone, glasses, etc. can be used to record video, and a wearable device (such as headphones, glasses, etc.) can be used to record sound.

[0061] Step 104: perform target recognition on the video data to determine the image category.

[0062] Object recognition is a technology in the field of computer vision, which is mainly used to automatically extract and identify target objects of interest from digital images or video data.

[0063] The steps of target recognition in video data mainly include: denoising, smoothing, image enhancement and other processing of the images in the video data to reduce image interference and improve target visibility. Then, the features related to the target in the image are extracted, such as color, shape, texture, edge, etc., and mathematical methods are usually used to describe these features. Then, the known template or sample features are matched with the target features to be identified to determine the image category in the video data. For example, the identified image categories include three categories: boys, girls and dogs. Alternatively, in a concert scenario, target recognition can be understood as identifying the image categories on the live concert stage as various musical instruments and vocals, such as lead singer, guitar, piano, violin, etc.

[0064] Step 106 , performing audio separation on the sound data based on the image category to obtain audio data corresponding to each image category.

[0065] Among them, audio separation is a method of decomposing multiple mixed sound data into single audio data.

[0066] After determining the image category in the video data, audio separation is performed on the sound data in combination with the image category, that is, the sound corresponding to each image category is separated to obtain the audio data corresponding to each image category. The image category here refers to the image category that emits sound. For example, the mixed audio of a boy, a girl, and a dog is separated into the audio data of the boy, the girl, and the dog through audio separation technology; in a concert scene, the mixed audio of the recorded concert performance is separated into the audio data of the lead singer, guitar, piano, and violin through instrument separation technology.

[0067] Step 108: obtaining spatial audio according to the spatial position information of each image category and the corresponding audio data.

[0068] The spatial position information of each image category can be expressed as (L, α, β), where L is the distance of each image category from the user, α is the horizontal angle of each image category from the user, and β is the vertical angle of each image category from the user. The schematic diagram of the spatial position information can be found in Figure 2 .

[0069] The spatial position information of each image category can be obtained according to the position of each image category in the video data. The spatial position information of each image type in the video data can be obtained by analyzing the video data. After obtaining the spatial position information of each image category and the corresponding audio data, the spatial audio of each image category can be obtained, thereby obtaining all the spatial audio.

[0070] After obtaining the spatial audio, the spatial audio is sent to a wearable device for playback, such as headphones, to better restore the sound. Both the spatial audio and video data can also be sent to a device for playback, such as a mobile phone or other terminal device, to restore the real scene of the sound when the video was recorded, so as to achieve scene and sound restoration.

[0071] In this embodiment, after obtaining sound data and video data, target recognition is performed on the video data to determine the image category, audio separation is performed on the sound data based on the image category, and the audio data corresponding to each image category is determined. Then, spatial audio is obtained based on the spatial position information of each image category and the corresponding audio data. The spatial audio is used to be played in combination with the video data or played alone. Therefore, the spatial audio obtained by combining the image category and the spatial position information can accurately correspond to images of different categories and restore the sound orientation during recording, so that scene restoration and sound restoration can be achieved when playing the spatial audio, thereby improving the reliability of spatial audio playback.

[0072] In one embodiment, Figure 3 As shown, step 108 includes steps 302 to 306 .

[0073] Step 302: Acquire spatial location information of each image category.

[0074] The spatial position information of each image category can be obtained based on the position of each image category in the video data. The spatial position information of each image type in the video data can be obtained by analyzing the video data. Among them, the spatial position information of each image category can be expressed as (L, α, β), where L is the distance of each image category from the user, α is the horizontal angle of each image category from the user, and β is the vertical angle of each image category from the user. The schematic diagram of the spatial position information can be seen in Figure 2 .

[0075] Step 304: input the spatial position information of each image category into the head transfer function as a key parameter to obtain a head-related impulse response.

[0076] Among them, the head transfer function (HRTF) is a signal processing function used to simulate the listening space. It is determined by the influence of factors such as the head, ears and body on the propagation and reception of sound, including the influence of sound waves passing through the head, the front and back of the ears, and the reflection and refraction of body parts such as hair, shoulders and chest. HRTF can describe the way sound reaches the listener at different directions and distances, and can be used to calculate the filter response between the sound source and the listener's ears.

[0077] After the spatial position information of each image category is input as a key parameter into the head transfer function, the head transfer function can simulate the human ear receiving sound source signals from different directions and positions to obtain the head-related impulse response. The head-related impulse response is the representation of the head transfer function in the time domain. After the spatial position information of each image category is input as a key parameter into the head transfer function, the head transfer function will contain many angles (horizontal angles, vertical angles) and many distances (L). At this time, it may be a three-dimensional data. After determining the image category and the corresponding spatial position information, two angles (horizontal angles, vertical angles) and a distance (the distance of the image category from the user) corresponding to the determined image category can be obtained. In this way, a curve corresponding to the output can be obtained as the head-related impulse response.

[0078] Step 306: Obtain spatial audio according to the head-related impulse response and the audio data corresponding to each image category.

[0079] Furthermore, spatial audio can be obtained based on the head-related impulse response containing the spatial position information of each image category and the audio data corresponding to each image category. The processing method can be: convolving the head-related impulse response with the audio data corresponding to each image category to obtain spatial audio, simulate the sound source position in the three-dimensional space, generate stereo or surround sound effects, and thus provide a realistic stereo effect for the listener.

[0080] In this embodiment, after obtaining the spatial position information of each image category, spatial audio is obtained using the spatial position information of each image category, the head-related impulse response and the audio data corresponding to each image category, which can simulate the position of the sound source in the three-dimensional space and provide a realistic stereo effect for the listener.

[0081] In an extensible manner, the head transfer function can be pre-constructed and stored, for example, in a memory of a terminal. When the head transfer function is needed, the user's auricle and other parts can be feature-recognized, and according to the feature parameters identified, the head transfer function corresponding to the identified feature parameters can be matched in the stored head transfer functions. Alternatively, a head transfer function can be assigned to the user according to the default settings. Further, if Figure 4 As shown, the process of constructing the head transfer function includes step 402 and step 404 .

[0082] Step 402: Under preset environmental conditions, measure the designated user to obtain measurement data.

[0083] The preset environmental conditions refer to environmental conditions with known environmental parameters, such as in an anechoic room or in a concert hall.

[0084] Designated users refer to users determined according to needs. Measuring designated users means measuring the influence of various positions of designated users on sound and obtaining measurement data. For example, the reflection and refraction of sound waves passing through the head, front and back of the ears, and hair, shoulders, chest and other body parts to obtain measurement data.

[0085] Step 404: Obtain the head transfer function according to the measurement data.

[0086] After obtaining the measurement data, the measurement data is analyzed and processed to obtain the head transfer function.

[0087] Specifically, data acquisition devices are used for measurement and recording. These devices usually include an artificial head model and an array of multiple microphones or microphones. Under preset environmental conditions, sound is transmitted to the array from different directions and distances, and the signal captured by each microphone is recorded as measurement data. With these data, the head transfer function in each direction can be calculated.

[0088] In this embodiment, by constructing a head transfer function based on measurement data obtained by measuring a designated user under preset environmental conditions, the head transfer function can be made to better fit the characteristics of the designated user.

[0089] It is understandable that when determining the head transfer function, the public data can be used to match the user's head transfer function. Alternatively, the HRTFs in the database are first clustered, and then the user's personalized auricle features are used to select the most matching HRTF from the public database as the final head transfer function. The above methods can be selected according to actual needs and are not limited here.

[0090] In one embodiment, the head-related impulse response includes a head-related impulse response to a left ear and a head-related impulse response to a right ear, and the spatial audio includes an output audio signal for the left ear and an output audio signal for the right ear, such as Figure 5 As shown, step 306 includes step 502 and step 504 .

[0091] Step 502 , convolve the audio data corresponding to each image category and the head-related impulse response to the left ear respectively, and then perform a downmixing algorithm to obtain an output audio signal for the left ear.

[0092] The head-related impulse response to the left ear refers to the transfer function of the sound of a certain image category to the left ear. For example, if a person or an animal makes a sound, the head-related impulse response to the left ear includes the transfer function of the person's sound to the left ear and the transfer function of the animal's sound to the left ear.

[0093] Step 504 , convolve the audio data corresponding to each image category and the head-related impulse response to the right ear respectively, and then perform a downmixing algorithm to obtain an output audio signal for the right ear.

[0094] The head-related impulse response to the right ear refers to the transfer function of the sound of a certain image category to the right ear. For example, if a person or an animal makes a sound, the head-related impulse response to the right ear includes the transfer function of the person's sound to the right ear and the transfer function of the animal's sound to the right ear.

[0095] Since some audio devices have only two channels, for example, headphones have only left and right channels, therefore, when forming spatial audio, the output audio signal of the left ear is obtained based on the head-related impulse response to the left ear and the audio data corresponding to each image category, and played at the left ear channel, and the output audio signal of the right ear is obtained based on the head-related impulse response to the right ear and the audio data corresponding to each image category, and played at the right ear channel, so as to restore the spatial reality of the sound.

[0096] Exemplarily, when there are three image categories, or three target image categories of interest, such as boys, girls and dogs, when playing back with headphones, since the headphones have only two channels, a downmixing algorithm is required here. If the sounds of boys, girls and dogs are x1, x2 and x3 respectively, hll and hlr represent the head-related impulse response of the sound of x1 to the left ear and the head-related impulse response to the right ear respectively, hcl and hcr represent the head-related impulse response of the sound of x2 to the left ear and the head-related impulse response to the right ear respectively, hrl and hrr represent the head-related impulse response of the sound of x3 to the left ear and the head-related impulse response to the right ear respectively, and the left and right ear sound signals heard by the human ear are yl and yr respectively, then:

[0097] yl=x1*hll+x2*hcl+x3*hrl;

[0098] yr=x1*hlr+x2*hcr+x3*hrr;

[0099] After the above processing, users can hear the same spatial audio as when recording through the headphones.

[0100] In one embodiment, Figure 6 As shown, step 104 includes step 602 .

[0101] Step 602: Input the video data into an image recognition model to obtain the image category of each image in the video data.

[0102] The image recognition model is a pre-trained model, which is trained based on image samples of various categories and contains features of different image categories.

[0103] The training process of the image recognition model includes: obtaining a database about image recognition and video target monitoring, dividing it into training, testing, and validation sets, and then using deep learning and other methods to train the model. Then, the trained model can be deployed on mobile phones, or hardware devices such as headphones and glasses. Among them, the models for video target recognition include deep learning models of computer vision such as MANet (multi-scale aggregation network), Association LSTM (long short-term memory association network), and T-CNN (temporal convolutional network). Specifically, MANet is a neural network architecture that combines multi-scale features in a single model to improve the accuracy of target detection. It uses pyramid pooling to construct multi-scale feature maps and uses lightweight networks to aggregate information at different scales. Association LSTM is a recursive neural network architecture that learns to associate objects by encoding temporal dependencies between object features in video sequences. It can track multiple objects simultaneously and has shown good results in several tracking benchmarks. T-CNN is a three-dimensional convolutional neural network architecture that can process temporal and spatial data such as videos. It uses temporal convolution to capture temporal dependencies between frames and spatial convolution to capture spatial features, demonstrating state-of-the-art performance for action recognition on benchmark datasets.

[0104] After the video data is input into the image recognition model, the features corresponding to different image types stored in the image recognition model are matched based on the features of the images in the video data, so as to obtain the image category of each image in the video data.

[0105] In this embodiment, the video data is input into the image recognition model, so that the image category of each image in the video data can be obtained quickly and accurately.

[0106] In one embodiment, Figure 6 As shown, step 106 includes step 604 .

[0107] Step 604: input the identified image categories and sound data into a sound separation model to obtain audio data corresponding to each image category.

[0108] The sound separation model is a pre-trained model, which is trained based on image samples of various categories and corresponding sounds, and includes sounds corresponding to image samples of multiple categories.

[0109] The training process of the sound separation model includes: obtaining a database about audio separation, including sound and label databases of various categories such as boys, girls, dogs, pianos, violins, etc., and dividing the training, testing, and validation sets, and using deep learning and other methods to train the model. After the training is completed, the trained model can be deployed on hardware devices such as mobile phones, glasses, and headphones. Among them, the audio recognition model includes deep learning models used in the field of audio separation such as conv-tasNet, Demucs, and D3net. Specifically, conv-tasNet is an end-to-end deep learning model based on a time-domain speech separation network, which uses convolutional layers and loss functions to process audio signals. The model can separate multiple mixed audio signals into corresponding single audio signals, and it has shown good performance in many audio separation tasks. Demucs is an audio separation model based on a deep neural network, which uses a bidirectional long short-term memory network (BLSTM) encoder and decoder, and performs downsampling and upsampling operations through a time-extended convolutional neural network (TDCN). D3net is a novel deep learning model designed to achieve blind source separation of multi-channel audio signals. This model combines standard convolutional layers with deconvolutional layers to improve the quality and speed of audio separation.

[0110] After the identified image categories and sound data are input into the sound separation model, the feature parameters of the sounds corresponding to the different image categories stored in the sound separation model are matched based on the feature parameters of the sound data of different image categories, so as to obtain the audio data corresponding to each image category.

[0111] In this embodiment, the recognized image categories and sound data are input into the sound separation model, so that the audio data corresponding to each image category can be obtained quickly and accurately.

[0112] In one embodiment, Figure 6 As shown, step 108 includes step 606 and step 608 .

[0113] Step 606: Obtain intermediate audio data according to the spatial position information of each image category and the corresponding audio data.

[0114] The spatial position information of each image category can be obtained according to the position of each image category in the video data. After obtaining the spatial position information of each image category and the corresponding audio data, the spatial audio of each image category can be obtained as the intermediate audio data.

[0115] Step 608: adding reverberation parameters to the intermediate audio data to obtain spatial audio.

[0116] Then, reverberation parameters are added to the intermediate audio data to obtain spatial audio, which can simulate the real listening effect under specific environmental conditions, making the obtained spatial audio closer to reality.

[0117] For example, to obtain the intermediate audio data based on the spatial position information of each image category and the corresponding audio data, it is usually necessary to combine the head transfer function. Most head transfer functions are laboratory head transfer functions. Since most of these are measured in an anechoic chamber, a lot of spatial information will be lost. Therefore, by adding reverberation parameters to the intermediate audio data through the reverberation algorithm, the early reflection and late reflection of a specific environment can be increased through the algorithm, thereby achieving a real listening effect that simulates a specific environment.

[0118] Furthermore, the reverberation can be added in the form of room impulse response HRIR or artificial reverberation. Room impulse response HRIR refers to the sound signal received by the microphone when an instantaneous sound source makes a sound in a room. By recording the room impulse response at different locations and convolving it with the original sound signal, a reverberation effect with the corresponding room characteristics can be obtained. This method requires measuring the room first to obtain the room impulse response data at each location, and the impulse response of each room is different. Artificial reverberation is a simulation reverberation method based on digital signal processing. It can generate the reverberation effect of a virtual room by simulating the propagation, reflection and attenuation of sound waves in space. Artificial reverberation usually uses digital filters to simulate the reflection and attenuation of sound waves in different frequency ranges, as well as preset parameters such as reverberation time and decay time to control the reverberation effect.

[0119] In this embodiment, reverberation parameters are added to the intermediate audio data obtained according to the spatial position information of each image category and the corresponding audio data to obtain spatial audio, which can simulate the real listening effect under specific environmental conditions, making the obtained spatial audio closer to reality.

[0120] In one embodiment, Figure 7 As shown, after step 108 , the audio and video processing method further includes step 702 .

[0121] Step 702: adjust the angle of the spatial audio according to the deviation angle of the user's head, so that the playback position of the adjusted spatial audio does not change with the deviation of the user's head.

[0122] Specifically, the gyroscope set in the wearable device can be used to track the user's head deviation angle. After the terminal obtains the head deviation angle, it adjusts the angle in the spatial audio according to the user's head deviation angle to offset the influence of the head deviation angle, so that the playback position of the adjusted spatial audio does not change with the user's head deviation. For example, when recording audio and video data at a concert, the position of the sound data during recording is the stage directly in front. After obtaining the spatial audio, when playing the spatial audio and video data, the user's head deviation angle is obtained, and the spatial audio is adjusted according to the user's head deviation angle, so that when playing the spatial audio, no matter how the user's head turns, the playback position of the spatial audio is directly in front. That is, if the user wears headphones to record sound, the recorded sound is directly in front of the user, then after the user turns his head to the left, the playback position of the spatial audio heard by the user will be on the right of the user; after the user turns his head to the right, the playback position of the spatial audio heard by the user will be on the left of the user. Therefore, no matter how the user's head turns, the playback position of the adjusted spatial audio does not change with the user's head deviation, and remains in a fixed position, thereby obtaining spatial audio based on head tracking.

[0123] In one embodiment, Figure 7 As shown, after step 108 , the audio and video processing method further includes step 704 .

[0124] Step 704, share the spatial audio.

[0125] After obtaining spatial audio, it can be played alone without video data. In addition, users can also share spatial audio for other users to use. Alternatively, after obtaining spatial audio, it can be played with video data, and both spatial audio and video data can be shared with other users for other users to use.

[0126] There is no unique way to share spatial audio and / or video data. For example, LE audio technology can be used. LE audio technology refers to the next-generation audio transmission standard based on Bluetooth low energy technology. It is a new technology developed by Bluetooth SIG (Bluetooth Special Interest Group) that can provide a better audio experience and more efficient audio transmission.

[0127] In this embodiment, after obtaining the spatial audio, the spatial audio can also be shared for use by other users.

[0128] In one embodiment, the spatial position information is the position information of each image category in the video data, or the spatial position information is user-defined position information.

[0129] Specifically, the spatial position information of each image type in the video data can be obtained by analyzing the video data. Alternatively, the spatial position information can also be customized position information set by the user as required.

[0130] In this embodiment, when the spatial position information is the position information of each image category in the video data, the spatial audio obtained based on the spatial position information can restore the real scene when the sound and video were recorded. When the spatial position information is user-defined position information, the spatial audio obtained according to the spatial position information can meet the personalized needs of the user. For example, in a concert scene, while recording video and sound, the sound can be decomposed into multi-channel sound sources such as singers, pianos, guitars, and drums through the music source separation algorithm. The decomposed multi-channel can be freely placed by the user according to his or her ideas through the customized spatial position information, and spatial audio can be generated without following the setting of the concert hall (such as putting the singer in the middle, the piano at the back, the drums on the left, and the guitar on top), which has a wide range of applications.

[0131] In order to better understand the above embodiment, a detailed explanation is given below in conjunction with a specific embodiment. Figure 8 As shown, when implementing the audio and video processing method, mobile phones, glasses and other devices with cameras are mainly used to record videos, and wearable devices (such as headphones, glasses, etc.) are used to record sounds. When recording, image recognition technology is used to identify the categories in the picture. After recording the sound, the sound is separated according to these categories. The separated sound is made into spatial audio according to the direction and distance of the picture, and given to the headphones for playback, giving users true scene restoration and sound restoration. The headphone end can also be combined with a gyroscope to achieve a spatial sound follow effect. The audio and video processing method specifically includes the following aspects:

[0132] On the recording end, you can use video-capable devices such as mobile phones, cameras, glasses, etc. to record the screen, and use their built-in microphones to record the sound. If the device does not have a built-in microphone, you can use headphones or other hardware with a microphone to record the sound. The recorded sound may be a mono audio file or a dual-channel audio file.

[0133] Then, target recognition is performed on the recorded video to determine the image category. Target recognition is a form of computer vision used to identify objects in images or videos. For example, it is recognized that the video recorded in the picture has three categories: boys, girls, and dogs. In the concert scene, target recognition can be understood as identifying various musical instruments and voices on the live concert stage, such as lead singers, guitars, pianos, violins, etc.

[0134] The target recognition of video data can adopt deep learning methods, and the steps include: there is a database related to image recognition and video target monitoring, and it is divided into training, testing, and verification sets. The model is trained using deep learning and other methods. The models for solving the video target recognition include MANet, Association LSTM, T-CNN, etc. Then the trained model is deployed on mobile phones, headphones, glasses and other hardware devices.

[0135] Next, the recorded sound data is audio separated. Specifically, for the image category identified by the target, the single and dual channels of the recorded audio are separated, that is, the mixed audio of boys, girls and dogs is separated into boys, girls and dogs through audio separation technology. In the concert scene, the mixed audio of the recorded concert performance is separated into lead singer, guitar, piano and violin through audio separation technology.

[0136] The steps for audio separation using deep learning are as follows:

[0137] 1. There is a database for audio separation, including the sound and label databases of various categories such as boys, girls, dogs, piano, violin, etc., and divided into training, test, and verification sets;

[0138] 2. Use deep learning and other methods to train the model. Common models for solving audio recognition include conv-tasNet, Demucs, D3net, etc.

[0139] 3. Deploy the trained model on hardware devices such as mobile phones, glasses, and headphones;

[0140] After audio separation, spatial audio processing is performed. The head transfer function is used to obtain the spatial positioning capability (including direction and distance) in the horizontal plane of the space and even in the entire sphere centered on the human. It can also be combined with the spatial sound field and some indoor acoustics or spatial acoustic reverberation algorithms to construct a more realistic sense of space.

[0141] Specifically, the audio file separated in the previous step is used as a multi-channel sound source, and then binauralization is performed. Binauralization is the core of audio and video processing. After obtaining a multi-channel sound source, it can actually only be played in stereo on headphones. In binauralization technology, HRTF technology can be used to fix the sound source in a way that simulates wearing headphones like not wearing headphones (if it is a stereo sound source, it simulates the feeling of two speakers placed in front of the eyes, and if it is a multi-channel sound source, it simulates the feeling of multiple speakers placed around).

[0142] There are many ways to determine the head transfer function HRTF, such as using public data to match the HRTF of a certain person's head; or first clustering the head transfer HRTFs in the database, and then using the user's personalized auricle features to select the most matching HRTF from the public database as the final function; or measuring the HRTF functions of the user's various positions in an anechoic room; or measuring the HRTF functions of the user's various positions in a concert hall.

[0143] After binauralization, the reverberation algorithm is executed. This part is mainly an important part of increasing the spatial sense of the sound effect. Generally speaking, HRTF functions are based on public laboratory HRTF functions, but since most of these are measured in anechoic chambers, a lot of spatial information will be lost. The reverberation algorithm can increase the early reflection and late reflection of the room through the algorithm, thereby achieving the real listening effect of simulating the room. Common ways to add reverberation include room impulse response HRIR, artificial reverberation, etc.

[0144] The above sound effect function that can represent the sense of sound orientation is HRTF (L, α, β), where L is the distance between the sound source and the user, α is the horizontal angle between the sound source and the user, and β is the vertical angle between the sound source and the user. The angle is defined as follows: Figure 2 As shown. Users can determine L, α, β of each image category according to the position of each object in the video data. Alternatively, users can also place the separated audio signals according to their own needs, that is, users define L, α, β of each image category. After definition, they can select the corresponding HRTF in the HRTF library (for example, a boy has hll and hlr, which represent the transfer function of his voice to the left ear and to the right ear respectively), and then convolve with the audio signal to obtain spatial audio. If necessary, users can also add different reverberations to enhance the spatial sense of spatial audio.

[0145] like Fig. 9 , there are three target sound sources. When playing back with headphones, since headphones have only two channels, a down-mixing algorithm is needed here. If the voices of boys, girls, and dogs are x1, x2, and x3 respectively, hll and hlr represent the transfer functions of the sound of x1 to the left ear and to the right ear respectively, hcl and hcr represent the transfer functions of the sound of x2 to the left ear and to the right ear respectively, and hrl and hrr represent the transfer functions of the sound of x3 to the left ear and to the right ear respectively. The left and right ear sound signals yl and yr heard by the human ear are:

[0146] yl=x1*hll+x2*hcl+x3*hrl;

[0147] yr=x1*hlr+x2*hcr+x3*hrr;

[0148] After the above processing, users can hear the same spatial audio as when recording through the headphones.

[0149] It can also be expanded to combine various G-sensors, gyroscopes, etc. to determine the user's head rotation angle. The built-in gyroscope of the headset can track the user's head deviation angle in real time. No matter how the user turns, the sound source always maintains the original position, and the direction of the sound will not change due to head changes, thus obtaining spatial audio based on head tracking;

[0150] In addition, users can change spatial audio by customizing L, α, and β. For example, in a concert scene, while recording video and sound, the original stereo sound can be decomposed into multi-channel sound sources such as singers, piano, guitar, and drums through audio separation technology. The decomposed multi-channel sound sources can be freely placed by the user according to their ideas, and spatial audio can be generated without following the settings of the concert hall (such as putting the singer in the middle, the piano at the back, the drums on the left, and the guitar on the top).

[0151] The generated spatial audio can be played alone without video playback. If the user wants to share his DIY work with his friends, he can use LE audio technology to share the multi-track sound file. Alternatively, the generated spatial audio can also be played with the video. For example, if the user wants to share the spatial audio of the concert scene with his friends, he can use LE audio technology to share the video file and the multi-track sound file.

[0152] The audio processing method provided in the embodiment of the present application can construct a multi-track target sound source on an ordinary recording file through audio separation technology, forming a spatial audio that can restore the scene and reality. Through the combined audio and video presentation method, the scene can be reproduced to a large extent, achieving an immersive and realistic effect.

[0153] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0154] Based on the same inventive concept, the embodiment of the present application also provides an audio and video processing device for implementing the audio and video processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more audio and video processing device embodiments provided below can refer to the limitations of the audio and video processing method above, and will not be repeated here.

[0155] In one embodiment, Fig.10 As shown, an audio and video processing device is provided, including: an audio and video acquisition module 1002, an image recognition module 1004, a sound separation module 1006 and a spatial audio reconstruction module 1008, wherein:

[0156] The audio and video acquisition module 1002 is used to acquire audio and video data; the audio and video data includes sound data and video data.

[0157] The image recognition module 1004 is used to perform target recognition on the video data and determine the image category.

[0158] The sound separation module 1006 is used to perform audio separation on the sound data based on the image category, and determine the audio data corresponding to each image category.

[0159] The spatial audio reconstruction module 1008 is used to obtain spatial audio according to the spatial position information of each image category and the corresponding audio data; the spatial audio is used to be played in combination with the video data or played alone.

[0160] In one embodiment, the spatial audio reconstruction module is also used to obtain the spatial position information of each image category, input the spatial position information of each image category as a key parameter into the head transfer function, obtain the head-related impulse response, and obtain the spatial audio based on the head-related impulse response and the audio data corresponding to each image category.

[0161] In one embodiment, the head-related impulse response includes a head-related impulse response to the left ear and a head-related impulse response to the right ear, the spatial audio includes an output audio signal for the left ear and an output audio signal for the right ear, and the spatial audio reconstruction module is further used to convolve the audio data corresponding to each image category with the head-related impulse response to the left ear, and then perform a downmixing algorithm to obtain the output audio signal for the left ear; and convolve the audio data corresponding to each image category with the head-related impulse response to the right ear, and then perform a downmixing algorithm to obtain the output audio signal for the right ear.

[0162] In one embodiment, the image recognition module is further used to input the video data into an image recognition model to obtain the image category of each image in the video data; the image recognition model is trained based on image samples of each category.

[0163] In one embodiment, the sound separation module is also used to input the identified image category and sound data into a sound separation model to obtain audio data corresponding to each image category; the sound separation model is obtained by training based on image samples of each category and corresponding sounds.

[0164] In one embodiment, the spatial audio reconstruction module is further used to obtain intermediate audio data according to the spatial position information of each image category and the corresponding audio data, and add reverberation parameters to the intermediate audio data to obtain spatial audio.

[0165] In one embodiment, the audio and video processing device also includes a head tracking module, which is used to adjust the angle of the spatial audio according to the user's head deviation angle after the spatial audio reconstruction module obtains the spatial audio according to the spatial position information of each image category and the corresponding audio data, so that the playback position of the adjusted spatial audio does not change with the user's head deviation.

[0166] In one embodiment, the audio and video processing device further includes a sharing module, which is used to share the spatial audio after the spatial audio reconstruction module obtains the spatial audio according to the spatial position information of each image category and the corresponding audio data.

[0167] In one embodiment, the spatial position information is the position information of each image category in the video data, or the spatial position information is user-defined position information.

[0168] Each module in the above-mentioned audio and video processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0169] In one embodiment, an audio and video device is provided, which includes a recorder and a processor, wherein the recorder is connected to the processor, the recorder is used to record audio and video data and send it to the processor, and the processor is used to perform audio and video processing according to the method of any of the above embodiments.

[0170] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0171] It should be noted that the user information (including but not limited to recorded audio and video data) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0172] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0173] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An audio and video processing method, characterized in that: The method comprises: Acquiring audio and video data; the audio and video data includes sound data and video data; Performing target recognition on the video data to determine the image category; Performing audio separation on the sound data based on the image categories to determine audio data corresponding to each image category; Spatial audio is obtained according to the spatial position information of each of the image categories and the corresponding audio data; the spatial audio is used for playing in combination with the video data or playing alone.

2. The method according to claim 1, characterized in that The obtaining of spatial audio according to the spatial position information of each image category and the corresponding audio data includes: Acquire spatial position information of each of the image categories; Inputting the spatial position information of each of the image categories as a key parameter into a head transfer function to obtain a head-related impulse response; Spatial audio is obtained according to the head-related impulse response and the audio data corresponding to each of the image categories.

3. The method according to claim 2, characterized in that The head-related impulse response includes a head-related impulse response to a left ear and a head-related impulse response to a right ear, the spatial audio includes an output audio signal for a left ear and an output audio signal for a right ear, and obtaining the spatial audio according to the head-related impulse response and the audio data corresponding to each of the image categories includes: Convolving the audio data corresponding to each of the image categories with the head-related impulse response to the left ear, and then performing a downmixing algorithm to obtain an output audio signal for the left ear; The audio data corresponding to each of the image categories and the head-related impulse response to the right ear are convolved respectively, and then a down-mixing algorithm is performed to obtain an output audio signal for the right ear.

4. The method according to claim 1, characterized in that: The performing target recognition on the video data to determine the image category includes: The video data is input into an image recognition model to obtain the image category of each image in the video data; the image recognition model is trained based on image samples of each category.

5. The method according to claim 1, characterized in that: The performing audio separation on the sound data based on the image category to determine the audio data corresponding to each image category includes: The identified image categories and the sound data are input into a sound separation model to obtain audio data corresponding to each image category; the sound separation model is obtained by training based on image samples of each category and corresponding sounds.

6. The method according to claim 1, characterized in that The obtaining of spatial audio according to the spatial position information of each image category and the corresponding audio data includes: Obtaining intermediate audio data according to the spatial position information of each of the image categories and the corresponding audio data; A reverberation parameter is added to the intermediate audio data to obtain spatial audio.

7. The method according to claim 1, characterized in that After obtaining the spatial audio according to the spatial position information of each image category and the corresponding audio data, the method further includes: The angle in the spatial audio is adjusted according to the deviation angle of the user's head, so that the playback position of the adjusted spatial audio does not change with the deviation of the user's head.

8. The method according to claim 1, characterized in that After obtaining the spatial audio according to the spatial position information of each image category and the corresponding audio data, the method further includes: Share the spatial audio.

9. The method according to any one of claims 1 to 8, characterized in that The spatial position information is the position information of each of the image categories in the video data, or, The spatial position information is user-defined position information.

10. An audio and video processing device, characterized in that: The device comprises: An audio and video acquisition module, used to acquire audio and video data; the audio and video data includes sound data and video data; An image recognition module, used to perform target recognition on the video data and determine the image category; A sound separation module, used to perform audio separation on the sound data based on the image category, and determine the audio data corresponding to each image category; The spatial audio reconstruction module is used to obtain spatial audio according to the spatial position information of each image category and the corresponding audio data; the spatial audio is used to be played in combination with the video data or played alone.

11. An audio and video device, characterized in that: It comprises a recorder and a processor, wherein the recorder is connected to the processor, the recorder is used to record audio and video data and send them to the processor, and the processor is used to perform audio and video processing according to the method described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images