Audio processing method, device, electronic device and computer readable storage medium

By combining the data collected by the two audio devices for audio noise reduction processing, the problem of high noise when the audio device is far away from the sound source is solved, and a more efficient audio noise reduction effect is achieved.

CN115442710BActive Publication Date: 2024-11-08ZHUHAI PROMETHEUS VISION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210940071.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-05
Publication Date
2024-11-08
Estimated Expiration
2042-08-05

AI Technical Summary

Technical Problem

When the distance between the audio device and the sound source is farther, the collected audio contains a lot of noise, making it difficult to effectively reduce noise through the prior art, resulting in poor audio quality.

Method used

By obtaining the audio data collected by two audio devices different from the sound source and inputting it into the audio noise reduction model, the audio noise reduction data trained by the model is used for processing to improve the audio noise reduction effect.

Benefits of technology

Compared with the noise reduction processing of a single audio data, the noise reduction effect of the audio is significantly improved by combining the processing methods of two audio data, and can more effectively remove noise and obtain purer audio.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115442710B_ABST
    Figure CN115442710B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses an audio processing method, device, electronic device and computer-readable storage medium, the method comprising: obtaining first audio data collected by a first audio device and second audio data collected by a second audio device, wherein a first collection distance between the first audio device and a clean sound source is less than a second collection distance between the second audio device and the clean sound source, and a first interference distance between the first audio device and a noise source is greater than a second interference distance between the second audio device and the noise source; inputting the first audio data and the second audio data into an audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data. By collecting the first audio data and the second audio data at different distances from the sound source, and performing audio noise reduction processing using the first audio data and the second audio data, the audio noise reduction effect is improved compared to the noise reduction processing of a single audio data in the related art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and specifically to an audio processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the rapid development of computer technology, audio processing technology has also been improved. For example, the collected audio can be denoised to obtain purer audio to improve audio quality.

[0003] However, when the distance between the audio device and the sound source is far, the collected audio contains more noise, which puts a burden on the audio noise reduction process and makes it difficult to obtain relatively pure audio. Summary of the invention

[0004] The embodiments of the present application provide an audio processing method, device, electronic device, and computer-readable storage medium, which can perform noise reduction on collected audio to obtain relatively pure audio.

[0005] In a first aspect, an embodiment of the present application provides an audio processing method, comprising:

[0006] Acquire first audio data collected by a first audio device and second audio data collected by a second audio device, wherein a first collection distance between the first audio device and a clean sound source is smaller than a second collection distance between the second audio device and the clean sound source, and a first interference distance between the first audio device and a noise source is larger than a second interference distance between the second audio device and the noise source;

[0007] Inputting the first audio data and the second audio data into an audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data;

[0008] The audio noise reduction model is obtained by taking the sample audio data obtained after the first audio device, the second audio device and the third audio device collect sample clean sound sources and sample noise sources as input, and is trained by taking the sample audio data collected by the third audio device as the fitting target, wherein a third collection distance between the third audio device and the sample clean sound source is less than the first collection distance.

[0009] In a second aspect, an embodiment of the present application further provides an audio processing device, including:

[0010] an acquisition module, configured to acquire first audio data collected by a first audio device and second audio data collected by a second audio device, wherein a first acquisition distance between the first audio device and a clean sound source is smaller than a second acquisition distance between the second audio device and the clean sound source, and a first interference distance between the first audio device and a noise source is larger than a second interference distance between the second audio device and the noise source;

[0011] A processing module, configured to input the first audio data and the second audio data into an audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data;

[0012] The audio noise reduction model is obtained by taking the sample audio data obtained after collecting the sample clean sound source and the sample noise source by the first audio device, the second audio device and the third audio device as input, and the model is trained with the sample audio data collected by the third audio device as the fitting target, wherein the third collection distance between the third audio device and the sample clean sound source is less than the first collection distance.

[0013] In some embodiments of the present application, the device further includes a training module, and the training module includes:

[0014] an acquisition unit, configured to acquire first sample audio data collected by a first sample audio device, second sample audio data collected by a second sample audio device, and third sample audio data collected by a third sample audio device, wherein a third sample acquisition distance between the third sample audio device and a sample clean sound source is smaller than a first sample acquisition distance between the first sample audio device and the sample clean sound source, the first sample acquisition distance is smaller than a second sample acquisition distance between the second sample audio device and the sample clean sound source, and a first sample interference distance between the first sample audio device and the sample noise source is larger than a second sample acquisition distance between the second sample audio device and the sample noise source;

[0015] A training unit is used to use the first sample audio data, the second sample audio data and the third sample audio data as inputs of a preset audio processing model, and use the third sample audio data as a fitting target to perform model training on the preset audio processing model to obtain the audio noise reduction model.

[0016] In some embodiments of the present application, the acquisition unit includes:

[0017] A first acquisition subunit is used to acquire third sample audio data collected by a third sample audio device;

[0018] a playing subunit, configured to play the audio corresponding to the third sample audio data at different locations of the studio, and / or to adjust the playing volume of the third sample audio data and play the audio corresponding to the adjusted third sample audio data;

[0019] The second acquisition subunit is used to collect audio through the first sample audio device to obtain first sample audio data during the playback process, and at the same time, collect audio through the second sample audio device to obtain second sample audio data.

[0020] In some embodiments of the present application, the first audio device is located in the studio, the second audio device is located outside the studio, the clean sound source is located in the studio, the noise source is located outside the studio, the clean sound source includes the object of the volumetric video, and the noise source includes the processing equipment of the volumetric video.

[0021] In some embodiments of the present application, the first audio device includes at least two located at different positions in the studio, and the acquisition module includes:

[0022] A collection unit, configured to collect audio through the at least two first audio devices to obtain at least two local audio data;

[0023] The synthesis unit is used to synthesize the at least two local audio data into one audio data to obtain the first audio data.

[0024] Wherein, in some embodiments of the present application, the synthesis unit includes:

[0025] a determination subunit, configured to determine, for each of the local audio data, a synthesis weight of the local audio data according to a position distance relationship between a first audio device corresponding to the local audio data and the photographed object;

[0026] The synthesis subunit is used to synthesize each local audio data according to the synthesis weight corresponding to each local audio data to obtain the first audio data.

[0027] In some embodiments of the present application, the processing module includes:

[0028] A first determining unit, used to determine the timbre type of the photographed object;

[0029] A second determining unit, configured to determine a target audio noise reduction model according to the timbre type;

[0030] The first processing unit is used to input the first audio data and the second audio data into the target audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data.

[0031] In some embodiments of the present application, the device further includes an audio playback module, and the audio playback module includes:

[0032] A second processing unit is used to perform stereoscopic transformation processing on the noise reduction audio data according to the angle between the viewer and the photographed object when the volumetric video is played, so as to obtain processed noise reduction audio data;

[0033] A playing unit is used to play audio based on the processed noise reduction audio data.

[0034] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned audio processing method when executing the computer program.

[0035] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned audio processing method are implemented.

[0036] Among them, after acquiring the first audio data collected by the first audio device at different distances from the sound source and the second audio data collected by the second audio device, the embodiment of the present application inputs the first audio data and the second audio data into the audio noise reduction model to obtain the noise reduction audio data corresponding to the first audio data, thereby realizing the function of using the model to reduce the noise of the collected audio data. Among them, by collecting the first audio data and the second audio data at different distances from the sound source, and using the first audio data and the second audio data to perform audio noise reduction processing, compared with the noise reduction processing of a single audio data in the related art, the audio noise reduction effect is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 is a schematic diagram of a scenario of an audio processing method provided in an embodiment of the present application;

[0039] Figure 2 is a flowchart of an audio processing method provided in an embodiment of the present application;

[0040] Figure 3 It is a schematic diagram of the process of audio acquisition and noise reduction based on a camera studio provided in an embodiment of the present application;

[0041] Figure 4 It is a flowchart of the audio noise reduction model training provided in the embodiment of the present application;

[0042] Figure 5 It is a schematic diagram of the process of audio collection and noise reduction based on volumetric video shooting provided by an embodiment of the present application;

[0043] Figure 6 is a structural diagram of an audio processing device provided in an embodiment of the present application;

[0044] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0046] The embodiments of the present application provide an audio processing method, device, electronic device and computer-readable storage medium. Specifically, the embodiments of the present application provide an audio processing device suitable for electronic devices, wherein the electronic device includes a terminal device or a server, wherein the terminal device can be a recorder, a mobile phone, a microphone, a computer or a tablet and other devices, and the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, and the server can be directly or indirectly connected via wired or wireless communication.

[0047] The audio processing method in the embodiment of the present application can be executed by the terminal device alone, or by the server alone, or by the terminal device and the server together. Figure 1 , taking the case where the terminal device and the server jointly execute the audio processing method as an example, the specific execution process of the audio processing method is as follows:

[0048] The first audio data is obtained through the terminal device 10, and the second audio data is obtained through the terminal device 11, and the obtained first audio data and second audio data are sent to the server 12, and the server 12 processes the first audio data and the second audio data.

[0049] After acquiring the first audio data and the second audio data, the server 12 inputs the first audio data and the second audio data into the audio noise reduction model of the server 12, and outputs the noise reduction audio data corresponding to the first audio data.

[0050] In the embodiment of the present application, the distances between the terminal device 10 and the terminal device 11 and the sound source are different, specifically including: the first collection distance between the first audio device and the clean sound source is smaller than the second collection distance between the second audio device and the clean sound source, and the first interference distance between the first audio device and the noise source is larger than the second interference distance between the second audio device and the noise source. In the embodiment of the present application, the clean sound source is the source of pure sound, that is, it represents an ideal and effective sound source, and the noise source is the source of noise, that is, it represents a sound source that interferes with the audio of the clean sound source.

[0051] Among them, the audio noise reduction processing based on the audio noise reduction model provided in the embodiment of the present application involves machine learning in the field of artificial intelligence; the embodiment of the present application can improve the efficiency of audio noise reduction and the audio noise reduction effect.

[0052] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Among them, artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0053] Among them, machine learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0054] For example, in an embodiment of the present application, the audio noise reduction model is pre-trained based on sample audio data, and the noise reduction processing of the audio data is implemented by the trained audio noise reduction model.

[0055] Among them, after obtaining the first audio data collected by the first audio device at different distances from the sound source and the second audio data collected by the second audio device, the embodiment of the present application inputs the first audio data and the second audio data into the audio noise reduction model to obtain the noise reduction audio data corresponding to the first audio data, thereby realizing the function of using the model to reduce the noise of the collected audio data. Among them, by collecting the first audio data and the second audio data at different distances from the sound source, and using the first audio data and the second audio data to perform audio noise reduction processing, compared with the noise reduction processing of a single audio data in the related art, the audio noise reduction effect is improved.

[0056] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0057] See also Figure 2 , Figure 2 This is a flow chart of the audio processing method provided in the embodiment of the present application. The specific flow of the audio processing method can be as follows:

[0058] 101. Obtain first audio data collected by a first audio device and second audio data collected by a second audio device, wherein a first collection distance between the first audio device and a clean sound source is smaller than a second collection distance between the second audio device and the clean sound source, and a first interference distance between the first audio device and a noise source is larger than a second interference distance between the second audio device and the noise source.

[0059] In the embodiment of the present application, the first audio data is collected by the first audio device, and the second audio data is collected by the second audio device. The first audio device and the second audio device can be the same audio collection device or different audio collection devices. For example, the audio collection device can include a recorder, a microphone, or a mobile phone.

[0060] Among them, in the embodiment of the present application, the first audio device and the second audio device can both collect the audio of the clean sound source and the audio of the noise source, but since the first audio device and the second audio device are at different distances from the clean sound source and the noise source, the first audio data and the second audio data are different. For example, the first audio data contains more audio of the clean sound source and a small amount of audio of the noise source, and the second audio data contains more audio of the noise source and a small amount of audio of the clean sound source. For example, when the distance between the first audio device and the noise source is far enough, the first audio data may not contain the audio corresponding to the noise source. Correspondingly, when the distance between the second audio device and the clean sound source is far enough, the second audio data may not contain the audio corresponding to the clean sound source.

[0061] Among them, the embodiment of the present application obtains audio data by respectively collecting two audio devices at different distances from the sound source, which facilitates audio noise reduction processing based on the two audio data. Compared with the noise reduction processing of a single audio data, the effect of audio noise reduction can be improved.

[0062] 102. Input the first audio data and the second audio data into an audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data; wherein the audio noise reduction model uses the sample audio data obtained after the first audio device, the second audio device and the third audio device collect sample clean sound sources and sample noise sources as input, and uses the sample audio data collected by the third audio device as a fitting target for model training, wherein a third collection distance between the third audio device and the sample clean sound source is less than the first collection distance.

[0063] Among them, in the embodiment of the present application, since the distances between the first audio device and the second audio device and the sound source are different, it can be understood that the ratio of the audio of the pure sound source and the audio of the noise source contained in the first audio data is different from the ratio of the aforementioned two audios contained in the second audio data. Therefore, by inputting the first audio data and the second audio data into the audio noise reduction model, the model analyzes the two audio data with different degrees of mixing, and outputs audio data corresponding to relatively pure noise reduction audio.

[0064] In the embodiment of the present application, the audio noise reduction model is obtained by training based on sample data, wherein the sample data includes audio data collected by the first audio device, audio data collected by the second audio device, and audio data collected by the third audio device, wherein, since the distance between the third audio device and the clean sound source is closer than the distance between the first audio device and the second audio device and the clean sound source, the audio collected by the third audio device can be used as the fitting target of the model, that is, as the true value (GT, GroundTruth) of model learning, and the audio data collected by the first audio device and the second audio device are used as input to train the model, so that after training, the model has the ability to input the audio collected by the first audio device and the second audio device, and output the audio data after noise reduction for the input audio. In the embodiment of the present application, since the distance between the first audio device and the clean sound source is closer than the distance between the second audio device and the clean sound source, the audio data output by the audio noise reduction model in this scheme can be understood as the audio data after noise reduction processing of the audio collected by the first audio device.

[0065] In the embodiment of the present application, when training the model, sample audio data can be collected through a sample device, and the model can be trained based on the sample audio data, wherein each sample device has a different distance from the clean sound source and the noise source, that is, optionally, in some embodiments of the present application, before the step of "inputting the first audio data and the second audio data into the audio noise reduction model to obtain the noise reduction audio data corresponding to the first audio data", the method also includes:

[0066] Acquire first sample audio data collected by a first sample audio device, second sample audio data collected by a second sample audio device, and third sample audio data collected by a third sample audio device, wherein a third sample collection distance between the third sample audio device and a sample clean sound source is smaller than a first sample collection distance between the first sample audio device and the sample clean sound source, the first sample collection distance is smaller than a second sample collection distance between the second sample audio device and the sample clean sound source, and a first sample interference distance between the first sample audio device and the sample noise source is larger than a second sample collection distance between the second sample audio device and the sample noise source;

[0067] The first sample audio data, the second sample audio data and the third sample audio data are used as inputs of a preset audio processing model, and the third sample audio data is used as a fitting target to perform model training on the preset audio processing model to obtain a trained audio noise reduction model.

[0068] Among them, audio data is collected by audio devices at different distances from the clean sound source and the noise source, and the audio data collected by these audio devices can be used for model training. When the model is trained with sample data, the closer the distance to the clean sound source, the purer the collected audio and the more in line with expectations. Therefore, the audio data collected by the device closest to the clean sound source is used as the fitting target, and the other two audio data are used as input to train the model.

[0069] Among them, in the actual audio collection process, in order to reduce the interference of noise on the audio, a studio can be used to collect audio. However, due to the limited noise blocking effect of the studio, the audio collected in the studio cannot be guaranteed to be pure enough, where the noise mainly comes from the operating sound of the equipment outside the studio. In addition, due to the characteristics of the audio equipment itself, such as the size, it cannot be kept at a clean sound source (such as the user's collar) for a long time. Therefore, it is difficult to continuously obtain relatively pure audio. Therefore, this solution can be used to perform further noise reduction on the audio collected in the studio to obtain purer audio. For example, the audio collected by the audio equipment in the studio is subjected to further noise reduction to obtain pure audio close to the audio at the user's collar. Specifically:

[0070] The first audio device can be set inside the studio to collect sound from the inside of the studio, wherein the first audio device is not set at the clean sound source, therefore, there is still a certain spatial distance between the first audio device and the clean sound source, and the spatial distance will still cause noise in the collected audio, wherein the first audio data can also be understood as audio data to be denoised; the second audio device can be set outside the studio to collect sound from the outside of the studio, wherein, due to the limited sound insulation effect of the studio, the audio collected by the first audio device may contain noise outside the studio, and similarly, the audio collected by the second audio device may also contain user audio inside the studio, accordingly, the distance between the first audio device and the clean sound source in the studio satisfies the requirement of being less than the distance between the second audio device located outside the studio and the clean sound source, and the distance between the first audio device and the noise source satisfies the requirement of being greater than the distance between the second audio device and the noise source.

[0071] Among them, after the first audio data is denoised according to the second audio data and through the audio noise reduction model, the obtained audio data is purer than the first audio data and is audio data closer to the audio at the user's collar.

[0072] Among them, in the embodiments of the present application, audio collection can also be applied in the process of shooting volumetric video, to collect the audio of the object of the volumetric video, and to perform noise reduction on the audio, so as to dub the volumetric video using the audio after noise reduction, thereby improving the audio effect of the volumetric video. That is, optionally, in some embodiments of the present application, the first audio device is located in the studio, the second audio device is located outside the studio, the net sound source is located in the studio, the noise source is located outside the studio, the net sound source includes the object of the volumetric video, and the noise source includes the processing equipment of the volumetric video, wherein the processing equipment also includes equipment for processing audio.

[0073] Among them, volumetric video (also known as volumetric video, spatial video, volumetric 3D video or 6-DOF video, etc.) is a technology that captures information in 3D space (such as depth information and color information, etc.) and generates a 3D model sequence. Compared with traditional videos, volumetric video adds the concept of space to the video, using 3D models to better restore the real 3D world, rather than using 2D flat video plus camera movement to simulate the sense of space in the real 3D world. Since volumetric video is essentially a 3D model sequence, users can adjust it to any viewing angle as they like, which has a higher degree of restoration and immersion than 2D flat video.

[0074] Optionally, in the present application, the three-dimensional model used to form the volumetric video can be reconstructed in the following manner:

[0075] First, color images and depth images of the object at different perspectives, as well as camera parameters corresponding to the color images, are obtained; then, a neural network model that implicitly expresses the three-dimensional model of the object is trained based on the obtained color images and their corresponding depth images and camera parameters, and isosurface extraction is performed based on the trained neural network model to achieve three-dimensional reconstruction of the object and obtain a three-dimensional model of the object.

[0076] It should be noted that the present application embodiment does not specifically limit the neural network model architecture, and can be selected by those skilled in the art according to actual needs. For example, a multilayer perceptron (MLP) without a normalization layer can be selected as the basic model for model training.

[0077] The three-dimensional model reconstruction method provided by the present application will be described in detail below.

[0078] First, multiple color cameras and depth cameras can be used simultaneously to shoot the target object (the target object is the shooting object) that needs to be reconstructed in three dimensions from multiple perspectives to obtain color images and corresponding depth images of the target object at multiple different perspectives, that is, at the same shooting time (the difference between the actual shooting time is less than or equal to the time threshold, which means that the shooting time is the same), the color camera of each perspective will shoot the color image of the target object at the corresponding perspective, and correspondingly, the depth camera of each perspective will shoot the depth image of the target object at the corresponding perspective. It should be noted that the target object can be any object, including but not limited to living objects such as people, animals and plants, or non-living objects such as machinery, furniture, and dolls.

[0079] In this way, the color images of the target object at different viewing angles all have corresponding depth images, that is, when shooting, the color camera and the depth camera can adopt the configuration of a camera group, and the color camera of the same viewing angle cooperates with the depth camera to synchronously shoot the same target object. For example, a studio can be built, the central area of ​​the studio is the shooting area, and around the shooting area, multiple groups of color cameras and depth cameras are paired and arranged at certain angles in the horizontal and vertical directions. When the target object is in the shooting area surrounded by these color cameras and depth cameras, the color images of the target object at different viewing angles and the corresponding depth images can be obtained by shooting with these color cameras and depth cameras.

[0080] In addition, the camera parameters of the color camera corresponding to each color image are further obtained. The camera parameters include the internal and external parameters of the color camera, which can be determined by calibration. The camera internal parameters are parameters related to the characteristics of the color camera itself, including but not limited to the focal length, pixels and other data of the color camera. The camera external parameters are parameters of the color camera in the world coordinate system, including but not limited to the position (coordinates) of the color camera and the rotation direction of the camera.

[0081] As described above, after obtaining color images of the target object from multiple different perspectives at the same shooting moment and their corresponding depth images, the target object can be reconstructed in three dimensions based on these color images and their corresponding depth images. Different from the method of converting depth information into point clouds for three-dimensional reconstruction in the related art, the present application trains a neural network model to realize the implicit expression of the three-dimensional model of the target object, thereby realizing the three-dimensional reconstruction of the target object based on the neural network model.

[0082] Optionally, the present application uses a multilayer perceptron (MLP) without a normalization layer as the basic model and trains it in the following manner:

[0083] Convert each pixel in the color image into a ray based on the corresponding camera parameters;

[0084] Sampling multiple sampling points on the ray, and determining the first coordinate information of each sampling point and the SDF value of each sampling point from the pixel point;

[0085] The first coordinate information of the sampling point is input into the basic model to obtain the predicted SDF value and the predicted RGB color value of each sampling point output by the basic model;

[0086] Based on a first difference between the predicted SDF value and the SDF value, and a second difference between the predicted RGB color value and the RGB color value of the pixel, adjusting the parameters of the basic model until a preset stop condition is met;

[0087] The basic model that meets the preset stopping conditions is used as a neural network model to implicitly express the three-dimensional model of the target object.

[0088] First, based on the camera parameters corresponding to the color image, a pixel in the color image is converted into a ray, which can be a ray passing through the pixel and perpendicular to the color image surface; then, multiple sampling points are sampled on the ray, and the sampling process of the sampling points can be performed in two steps. First, some sampling points can be uniformly sampled, and then multiple sampling points can be further sampled at key points based on the depth value of the pixel point to ensure that as many sampling points as possible can be sampled near the model surface; then, the first coordinate information of each sampling point obtained by sampling in the world coordinate system and the signed distance (SDF) of each sampling point are calculated according to the camera parameters and the depth value of the pixel point. The SDF value may be the difference between the depth value of the pixel point and the distance between the sampling point and the camera imaging plane, and the difference is a signed value. When the difference is positive, it indicates that the sampling point is outside the three-dimensional model, when the difference is negative, it indicates that the sampling point is inside the three-dimensional model, and when the difference is zero, it indicates that the sampling point is on the surface of the three-dimensional model. Then, after completing the sampling of the sampling points and calculating the SDF value corresponding to each sampling point, the first coordinate information of the sampling point in the world coordinate system is further input into the basic model (the basic model is configured to map the input coordinate information into the SDF value and the RGB color value and then output), the SDF value output by the basic model is recorded as the predicted SDF value, and the RGB color value output by the basic model is recorded as the predicted RGB color value. Then, based on the first difference between the predicted SDF value and the SDF value corresponding to the sampling point, and the second difference between the predicted RGB color value and the RGB color value of the pixel corresponding to the sampling point, the parameters of the basic model are adjusted.

[0089] In addition, for other pixel points in the color image, sampling points are sampled in the same manner as described above, and then the coordinate information of the sampling points in the world coordinate system is input into the basic model to obtain the corresponding predicted SDF value and predicted RGB color value, which are used to adjust the parameters of the basic model until the preset stop condition is met. For example, the preset stop condition can be configured so that the number of iterations of the basic model reaches a preset number, or the preset stop condition can be configured so that the basic model converges. When the iteration of the basic model meets the preset stop condition, a neural network model that can accurately and implicitly express the three-dimensional model of the photographed object is obtained. Finally, the isosurface extraction algorithm can be used to extract the three-dimensional model surface of the neural network model to obtain the three-dimensional model of the photographed object.

[0090] Optionally, in some embodiments, an imaging plane of the color image is determined according to camera parameters; and a ray passing through a pixel point in the color image and perpendicular to the imaging plane is determined to be a ray corresponding to the pixel point.

[0091] The coordinate information of the color image in the world coordinate system, that is, the imaging plane, can be determined according to the camera parameters of the color camera corresponding to the color image. Then, the ray passing through the pixel point in the color image and perpendicular to the imaging plane can be determined as the ray corresponding to the pixel point.

[0092] Optionally, in some embodiments, the second coordinate information and the rotation angle of the color camera in the world coordinate system are determined according to the camera parameters; and the imaging plane of the color image is determined according to the second coordinate information and the rotation angle.

[0093] Optionally, in some embodiments, a first number of first sampling points are sampled at equal intervals on the ray; a plurality of key sampling points are determined according to the depth value of the pixel point, and a second number of second sampling points are sampled according to the key sampling points; and the first number of first sampling points and the second number of second sampling points are determined as a plurality of sampling points obtained by sampling on the ray.

[0094] Among them, firstly, n (i.e., the first number) first sampling points are uniformly sampled on the ray, where n is a positive integer greater than 2; then, according to the depth value of the aforementioned pixel point, a preset number of key sampling points closest to the aforementioned pixel point are determined from the n first sampling points, or a key sampling point whose distance to the aforementioned pixel point is less than a distance threshold is determined from the n first sampling points; then, according to the determined key sampling points, m second sampling points are sampled again, where m is a positive integer greater than 1; finally, the sampled n+m sampling points are determined as multiple sampling points sampled on the ray. Among them, sampling m more sampling points at the key sampling points can make the training effect of the model more accurate on the surface of the three-dimensional model, thereby improving the reconstruction accuracy of the three-dimensional model.

[0095] Optionally, in some embodiments, the depth value corresponding to the pixel is determined according to the depth image corresponding to the color image; the SDF value of each sampling point from the pixel is calculated based on the depth value; and the coordinate information of each sampling point is calculated according to the camera parameters and the depth value.

[0096] Among them, after sampling multiple sampling points on the ray corresponding to each pixel point, for each sampling point, the distance between the shooting position of the color camera and the corresponding point on the target object is determined according to the camera parameters and the depth value of the pixel point, and then the SDF value of each sampling point is calculated one by one based on the distance, and the coordinate information of each sampling point is calculated.

[0097] It should be noted that after completing the training of the basic model, for the coordinate information of any given point, the trained basic model can predict its corresponding SDF value. The predicted SDF value represents the positional relationship between the point and the three-dimensional model of the target object (inside, outside or on the surface), thereby realizing the implicit expression of the three-dimensional model of the target object and obtaining a neural network model for implicitly expressing the three-dimensional model of the target object.

[0098] Finally, isosurface extraction is performed on the above neural network model. For example, an isosurface extraction algorithm (MC, Marching cubes) can be used to draw the surface of the three-dimensional model to obtain the three-dimensional model surface, and then the three-dimensional model of the target object is obtained based on the three-dimensional model surface.

[0099] The three-dimensional reconstruction scheme provided by the present application uses a neural network to implicitly model the three-dimensional model of the target object, and adds depth information to improve the speed and accuracy of model training. By adopting the three-dimensional reconstruction scheme provided by the present application, the three-dimensional model of the photographed object is continuously reconstructed in time sequence, and the three-dimensional model of the photographed object at different times can be obtained. The three-dimensional model sequence composed of these three-dimensional models at different times in time sequence is the volumetric video obtained by photographing the photographed object. In this way, "volume video shooting" can be performed on any photographed object to obtain a volumetric video presenting specific content. For example, a volumetric video of a dancing subject can be shot to obtain a volumetric video of the subject's dance that can be viewed from any angle. A volumetric video of a teaching subject can be shot to obtain a volumetric video of the subject's teaching that can be viewed from any angle, and so on.

[0100] Among them, in an embodiment of the present application, based on the volumetric video shooting scenario, the fitting target in the audio noise reduction model training process can be the sound collected at the volumetric video shooting object. For example, the third audio device is set near the shooting object (such as the user's collar). Since the distance between the third audio device and the shooting object is closer, the collected third audio data is purer than the first audio data. Therefore, when the audio data collected by the third audio device is used as the fitting target of the model, the model can have the ability to reduce audio noise after training, thereby achieving audio optimization.

[0101] In the embodiments of the present application, the first audio data in the studio may be collected by arranging a plurality of audio devices in the studio, and the first audio data with good audio effect may be obtained by synthesizing the plurality of audio devices. That is, optionally, in some embodiments of the present application, the first audio device includes at least two audio devices located at different positions in the studio, and the step of “obtaining the first audio data collected by the first audio device” includes:

[0102] Collecting audio through the at least two first audio devices to obtain at least two local audio data;

[0103] The at least two local audio data are combined into one audio data to obtain first audio data.

[0104] In this embodiment of the present application, local audio data is audio data obtained by a single audio device after collecting audio, wherein each audio device corresponds to a local audio data.

[0105] Among them, since the position of the subject or the direction of the sound can change at any time, it is difficult for an audio device in a single fixed position to collect relatively accurate audio data. Therefore, the embodiment of the present application sets audio devices in multiple positions in the studio, and can collect local audio data from multiple positions respectively. After synthesizing multiple local audio data, audio data that is closer to the user's voice can be obtained.

[0106] In the audio synthesis process, each audio data may be synthesized based on a weight to increase the accuracy and authenticity of the synthesized audio data, that is, optionally, in an embodiment of the present application, the step of “synthesizing the at least two local audio data into one audio data to obtain the first audio data” includes:

[0107] For each of the local audio data, determining a synthesis weight of the local audio data according to a position distance relationship between a first audio device corresponding to the local audio data and the photographed object;

[0108] The local audio data are synthesized according to the synthesis weight corresponding to each local audio data to obtain the first audio data.

[0109] Among them, since the actual position of each audio device and the subject may be different, and the audio data collected by audio devices at different position distances are different, and the closer the distance to the subject, the closer the collected audio data is to the actual sound audio of the subject, therefore, for each local audio data, the synthesis weight of the local audio data can be determined according to the distance between the audio device corresponding to the local audio data and the subject, and for each local audio data, each local audio data can be synthesized according to the synthesis weight corresponding to each local audio data to obtain the final audio data, and the final audio data is used as the first audio data collected in the studio.

[0110] Among them, since the timbre types of different photographed objects are different, and the difference in timbre types makes the audio data different, therefore, in order to reduce the probability of error in the operation of the audio noise reduction model and improve the effect of audio noise reduction, corresponding audio noise reduction models can be trained for different timbres respectively. When noise reduction processing is performed on audio data of different timbres, the audio data can be processed by the corresponding audio noise reduction model, that is, optionally, in some embodiments of the present application, the step of "inputting the first audio data and the second audio data into the audio noise reduction model to obtain the noise reduction audio data corresponding to the first audio data" includes:

[0111] Determining the timbre type of the photographed object;

[0112] Determining a target audio noise reduction model according to the timbre type;

[0113] The first audio data and the second audio data are input into the target audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data.

[0114] Among them, after determining the timbre type of the photographed object, a target audio noise reduction model suitable for the timbre type can be screened out from multiple audio noise reduction models according to the timbre type, and the first audio data and the second audio data collected based on the timbre type can be input into the target audio noise reduction model to implement noise reduction processing for the audio data of the timbre type.

[0115] Among them, in an embodiment of the present application, after the collected audio data is subjected to noise reduction processing, the noise-reduced audio can be dubbed into the corresponding volumetric video to enhance the audio effect of the volumetric video. However, since the volumetric video can be viewed from multiple angles, when the viewing angle of the volumetric video changes, the angle of the object model (the three-dimensional model of the photographed object) in the volumetric video also changes. Therefore, in order to make the audio playback consistent with the angle of the object model and make the audio playback effect more realistic, the noise-reduced audio can be subjected to stereo transformation processing according to the angle between the viewer and the object model, so that the processed audio played can be more realistic. That is, optionally, in some embodiments of the present application, after the step of "inputting the first audio data and the second audio data into the audio noise reduction model to obtain the noise-reduced audio data corresponding to the first audio data", the method further includes:

[0116] According to the angle between the viewer and the object when the volumetric video is played, the noise reduction audio data is subjected to stereoscopic transformation processing to obtain processed noise reduction audio data;

[0117] Audio playback is performed based on the processed noise-reduced audio data.

[0118] Among them, the stereo transformation processing of the audio makes the audio more three-dimensional and directional. Therefore, after the audio is transformed based on the angles of the viewer and the subject, the transformed audio can be made more realistic and more like the actual sound of the subject.

[0119] In an embodiment of the present application, when training the model, the sample data may be augmented to increase the amount of sample data used for model training, thereby improving the noise reduction effect of the model. In an embodiment of the present application, the augmentation of the sample data may be achieved by repeatedly playing the audio of the same clean sound source. For example, the same audio of the clean sound source is repeatedly played, and the first audio data and the second audio data are collected at different distances and under different conditions. A sample group for model training is formed based on the obtained audio data to train the model. That is, optionally, in some embodiments of the present application, the step of "obtaining first sample audio data collected by a first sample audio device, second sample audio data collected by a second sample audio device, and third sample audio data collected by a third sample audio device" includes:

[0120] Acquire third sample audio data collected by a third sample audio device;

[0121] Playing the audio corresponding to the third sample audio data at different locations in the studio, and / or adjusting the playback volume of the third sample audio data and playing the audio corresponding to the adjusted third sample audio data;

[0122] During the playing process, the first sample audio device collects audio to obtain first sample audio data, and at the same time, the second sample audio device collects audio to obtain second sample audio data.

[0123] Among them, since the third sample audio device is set at the clean sound source, the collected audio data is relatively pure and meets the true value used as the model fitting target. Therefore, the audio corresponding to the third sample audio data can be repeatedly played in the studio, and when the audio is played in the studio, the corresponding audio data is collected through the first sample audio device and the second sample audio device to obtain the first sample audio data and the second sample audio data.

[0124] Among them, when the playback position of the audio corresponding to the third sample audio data is adjusted, or the volume of the audio is adjusted, the collection of different first sample audio data and second sample audio data can be realized, and the sample data can be augmented. The increase in the amount of sample data can improve the noise reduction effect of the trained model.

[0125] Among them, after acquiring the first audio data collected by the first audio device at different distances from the sound source and the second audio data collected by the second audio device, the embodiment of the present application inputs the first audio data and the second audio data into the audio noise reduction model to obtain the noise reduction audio data corresponding to the first audio data, thereby realizing the function of using the model to reduce the noise of the collected audio data. Among them, by collecting the first audio data and the second audio data at different distances from the sound source, and using the first audio data and the second audio data to perform audio noise reduction processing, compared with the noise reduction processing of a single audio data in the related art, the audio noise reduction effect is improved.

[0126] See also Figure 3 , Figure 3 : is a schematic diagram of a process of audio acquisition and noise reduction based on a camera studio provided in an embodiment of the present application, wherein the process of audio acquisition and noise reduction based on a camera studio specifically includes:

[0127] 201. Set up a clean sound source inside the studio and a noise source outside the studio;

[0128] 202. Acquire first audio data using a first audio device in the studio, and acquire second audio data using a second audio device outside the studio;

[0129] 203. Input the collected first audio data and second audio data into an audio noise reduction model, and output noise reduction audio data corresponding to the first audio data.

[0130] Among them, since there is still a certain spatial distance between the first audio device in the studio and the clean sound source, the first audio data collected by the first audio device is still different from the audio of the clean sound source, that is, there is still room for noise reduction, and since the sound insulation effect of the studio is limited, the first audio data may be mixed with the noise outside the studio. Similarly, the second audio data may also be mixed with the clean sound source in the studio. Therefore, the embodiment of the present application can input the first audio data and the second audio data into the audio noise reduction model, use the audio noise reduction model to analyze the clean sound information and the noise information, and then input the noise reduction audio data that is purer than the first audio data.

[0131] In the embodiment of the present application, before the audio noise reduction model outputs the noise reduction audio data corresponding to the first audio data, the audio noise reduction model may be trained to improve the noise reduction effect of the audio noise reduction model. For example, refer to Figure 4 , Figure 4 : is a flow chart of the audio noise reduction model training provided in an embodiment of the present application, wherein the process of the audio noise reduction model training specifically includes:

[0132] 211. Set up a clean sound source inside the studio and a noise source outside the studio;

[0133] 212. First audio data is obtained by collecting data using a first audio device in the studio, second audio data is obtained by collecting data using a second audio device outside the studio, and third audio data is obtained by collecting data using a third audio device at a clean sound source in the studio, wherein a distance between the third audio device and the clean sound source is smaller than a distance between the first audio device and the clean sound source;

[0134] 213. Use the first audio data, the second audio data, and the third audio data as inputs of a preset audio noise reduction model, and use the third audio data as a fitting target of the model to train the preset audio noise reduction model until a preset stop condition is met, thereby obtaining an audio noise reduction model.

[0135] In an embodiment of the present application, sample data for model training can be collected respectively by three audio devices located at different positions in the studio, and the model can be trained multiple times based on the sample data to improve the noise reduction effect of the model.

[0136] Among them, in an embodiment of the present application, the sound of the clean sound source can also be played repeatedly, and the sample data can be enriched by adjusting the position of the clean sound source in the studio, or the volume of the clean sound source. For example, the sound of the clean sound source is played at different positions, and audio data is collected separately through three audio devices. The three audio data obtained at the same time can be used as a training sample group. In this way, multiple groups of training sample groups can be obtained to enrich the sample data for model training.

[0137] See also Figure 5 , Figure 5 : is a schematic diagram of the process of audio collection and noise reduction based on the volumetric video shooting process provided by an embodiment of the present application, wherein, taking the volumetric video shooting object as a user as an example, the process of audio collection and noise reduction based on the volumetric video shooting process specifically includes:

[0138] 221. A first audio device is arranged at the top position in the middle of the studio;

[0139] 222. Install audio processing equipment or related equipment for processing volumetric video outside the studio;

[0140] 223. Arrange a second audio device near the audio processing device or a device related to processing volumetric video;

[0141] 224. The user moves to the interior of the studio;

[0142] 225. When recording a volumetric video according to a user, first audio data is collected by a first audio device, and second audio data is collected by a second audio device;

[0143] 226. Input the first audio data and the second audio data into an audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data.

[0144] Among them, due to factors such as the size and weight of the audio device, it cannot be worn on the user's collar for a long time (such as inconvenience in wearing or shooting, etc.), and therefore, it is difficult to directly collect the audio at the user's collar, and usually only audio data at a fixed position in the studio can be collected, for example, the audio data at the location of the first audio device. Since there is still a certain spatial distance between the first audio device and the user's collar, there is still a situation where it is interfered by noise outside the studio. Therefore, the audio data collected by the first audio device still has room for noise reduction optimization. In the embodiment of the present application, by collecting the first audio data and the second audio data in the studio containing the user audio and the operating noise of the equipment outside the studio, the pure sound at the user's collar can be fitted based on these two audios and through the audio noise reduction model.

[0145] Before using the audio noise reduction model, the audio noise reduction model can also be trained. For example, the audio at the user's collar clip is collected as the fitting target of the model, and the audio collected by the first audio device and the second audio device is used as the input of the model to train the model. Since the audio collected at the collar clip is purer than the audio collected by the first audio device, the trained model has the ability to reduce audio noise.

[0146] In order to better implement the audio processing method of the present application, the present application also provides an audio processing device based on the above audio processing method. The meaning of the third target word is the same as that in the above audio processing method, and the specific implementation details can refer to the description in the method embodiment.

[0147] See also Figure 6 , Figure 6 is a schematic diagram of the structure of an audio processing device provided in an embodiment of the present application, wherein the audio processing device may include:

[0148] An acquisition module 301 is used to acquire first audio data collected by a first audio device and second audio data collected by a second audio device, wherein a first collection distance between the first audio device and a clean sound source is smaller than a second collection distance between the second audio device and the clean sound source, and a first interference distance between the first audio device and a noise source is larger than a second interference distance between the second audio device and the noise source;

[0149] A processing module 302 is used to input the first audio data and the second audio data into an audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data;

[0150] The audio noise reduction model is obtained by taking the sample audio data obtained after collecting the sample clean sound source and the sample noise source by the first audio device, the second audio device and the third audio device as input, and the model is trained with the sample audio data collected by the third audio device as the fitting target, wherein the third collection distance between the third audio device and the sample clean sound source is less than the first collection distance.

[0151] In some embodiments of the present application, the device further includes a training module, and the training module includes:

[0152] an acquisition unit, configured to acquire first sample audio data collected by a first sample audio device, second sample audio data collected by a second sample audio device, and third sample audio data collected by a third sample audio device, wherein a third sample acquisition distance between the third sample audio device and a sample clean sound source is smaller than a first sample acquisition distance between the first sample audio device and the sample clean sound source, the first sample acquisition distance is smaller than a second sample acquisition distance between the second sample audio device and the sample clean sound source, and a first sample interference distance between the first sample audio device and the sample noise source is larger than a second sample acquisition distance between the second sample audio device and the sample noise source;

[0153] A training unit is used to use the first sample audio data, the second sample audio data and the third sample audio data as inputs of a preset audio processing model, and use the third sample audio data as a fitting target to perform model training on the preset audio processing model to obtain the audio noise reduction model.

[0154] In some embodiments of the present application, the acquisition unit includes:

[0155] A first acquisition subunit is used to acquire third sample audio data collected by a third sample audio device;

[0156] a playing subunit, configured to play the audio corresponding to the third sample audio data at different locations of the studio, and / or to adjust the playing volume of the third sample audio data and play the audio corresponding to the adjusted third sample audio data;

[0157] The second acquisition subunit is used to collect audio through the first sample audio device to obtain first sample audio data during the playback process, and at the same time, collect audio through the second sample audio device to obtain second sample audio data.

[0158] In some embodiments of the present application, the first audio device is located in the studio, the second audio device is located outside the studio, the clean sound source is located in the studio, the noise source is located outside the studio, the clean sound source includes the object of the volumetric video, and the noise source includes the processing equipment of the volumetric video.

[0159] In some embodiments of the present application, the first audio device includes at least two audio devices located at different positions in the studio, and the acquisition module 301 includes:

[0160] A collection unit, configured to collect audio through the at least two first audio devices to obtain at least two local audio data;

[0161] The synthesis unit is used to synthesize the at least two local audio data into one audio data to obtain the first audio data.

[0162] Wherein, in some embodiments of the present application, the synthesis unit includes:

[0163] a determination subunit, configured to determine, for each of the local audio data, a synthesis weight of the local audio data according to a position distance relationship between a first audio device corresponding to the local audio data and the photographed object;

[0164] The synthesis subunit is used to synthesize each local audio data according to the synthesis weight corresponding to each local audio data to obtain the first audio data.

[0165] In some embodiments of the present application, the processing module 302 includes:

[0166] A first determining unit, used to determine the timbre type of the photographed object;

[0167] A second determining unit, configured to determine a target audio noise reduction model according to the timbre type;

[0168] The first processing unit is used to input the first audio data and the second audio data into the target audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data.

[0169] In some embodiments of the present application, the device further includes an audio playback module, and the audio playback module includes:

[0170] A second processing unit is used to perform stereoscopic transformation processing on the noise reduction audio data according to the angle between the viewer and the photographed object when the volumetric video is played, so as to obtain processed noise reduction audio data;

[0171] A playing unit is used to play audio based on the processed noise reduction audio data.

[0172] In the embodiment of the present application, the acquisition module 301 acquires the first audio data collected by the first audio device and the second audio data collected by the second audio device. Then, the processing module 302 inputs the first audio data and the second audio data into the audio noise reduction model to obtain the noise reduction audio data corresponding to the first audio data.

[0173] Among them, after acquiring the first audio data collected by the first audio device at different distances from the sound source and the second audio data collected by the second audio device, the embodiment of the present application inputs the first audio data and the second audio data into the audio noise reduction model to obtain the noise reduction audio data corresponding to the first audio data, thereby realizing the function of using the model to reduce the noise of the collected audio data. Among them, by collecting the first audio data and the second audio data at different distances from the sound source, and using the first audio data and the second audio data to perform audio noise reduction processing, compared with the noise reduction processing of a single audio data in the related art, the audio noise reduction effect is improved.

[0174] In addition, the present application also provides an electronic device, such as Figure 7 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0175] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0176] The processor 401 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, object interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0177] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0178] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0179] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to object settings and function controls.

[0180] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby implementing the steps in any audio processing method provided in this application.

[0181] After acquiring the first audio data collected by the first audio device at different distances from the sound source and the second audio data collected by the second audio device, the embodiment of the present application inputs the first audio data and the second audio data into the audio noise reduction model to obtain the noise reduction audio data corresponding to the first audio data, thereby realizing the function of using the model to reduce the noise of the collected audio data. Among them, by collecting the first audio data and the second audio data at different distances from the sound source, and using the first audio data and the second audio data to perform audio noise reduction processing, compared with the noise reduction processing of a single audio data in the related art, the audio noise reduction effect is improved.

[0182] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0183] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any audio processing method provided in the present application.

[0184] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0185] Since the instructions stored in the computer-readable storage medium can execute the steps in any audio processing method provided in the present application, the beneficial effects that can be achieved by any audio processing method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0186] The above is a detailed introduction to an audio processing method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An audio processing method, characterized in that: include: Acquire first audio data collected by a first audio device and second audio data collected by a second audio device, wherein a first collection distance between the first audio device and a clean sound source is smaller than a second collection distance between the second audio device and the clean sound source, and a first interference distance between the first audio device and a noise source is larger than a second interference distance between the second audio device and the noise source; Inputting the first audio data and the second audio data into an audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data; The audio noise reduction model is obtained by taking the sample audio data obtained after collecting the sample clean sound source and the sample noise source by the first audio device, the second audio device and the third audio device as input, and the model is trained with the sample audio data collected by the third audio device as the fitting target, wherein the third collection distance between the third audio device and the sample clean sound source is less than the first collection distance.

2. The method according to claim 1, characterized in that Before inputting the first audio data and the second audio data into the audio noise reduction model to obtain the noise reduction audio data corresponding to the first audio data, the method further includes: Acquire first sample audio data collected by a first sample audio device, second sample audio data collected by a second sample audio device, and third sample audio data collected by a third sample audio device, wherein a third sample collection distance between the third sample audio device and a sample clean sound source is smaller than a first sample collection distance between the first sample audio device and the sample clean sound source, the first sample collection distance is smaller than a second sample collection distance between the second sample audio device and the sample clean sound source, and a first sample interference distance between the first sample audio device and the sample noise source is larger than a second sample collection distance between the second sample audio device and the sample noise source; The first sample audio data, the second sample audio data and the third sample audio data are used as inputs of a preset audio processing model, and the third sample audio data is used as a fitting target to perform model training on the preset audio processing model to obtain the audio noise reduction model.

3. The method according to claim 1, characterized in that The first audio device is located in the studio, the second audio device is located outside the studio, the clean sound source is located in the studio, the noise source is located outside the studio, the clean sound source includes the object of the volumetric video, and the noise source includes the processing equipment of the volumetric video.

4. The method according to claim 3, characterized in that The first audio devices include at least two located at different positions in the studio, and the step of obtaining first audio data collected by the first audio devices includes: Collecting audio through the at least two first audio devices to obtain at least two local audio data; The at least two local audio data are combined into one audio data to obtain first audio data.

5. The method according to claim 4, characterized in that The step of synthesizing the at least two local audio data into one audio data to obtain the first audio data comprises: For each of the local audio data, determining a synthesis weight of the local audio data according to a position distance relationship between a first audio device corresponding to the local audio data and the photographed object; The local audio data are synthesized according to the synthesis weight corresponding to each local audio data to obtain the first audio data.

6. The method according to claim 3, characterized in that The step of inputting the first audio data and the second audio data into an audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data includes: Determining the timbre type of the photographed object; Determining a target audio noise reduction model according to the timbre type; The first audio data and the second audio data are input into the target audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data.

7. The method according to claim 3, characterized in that After inputting the first audio data and the second audio data into the audio noise reduction model to obtain the noise reduction audio data corresponding to the first audio data, the method further includes: According to the angle between the viewer and the object when the volumetric video is played, the noise reduction audio data is subjected to stereoscopic transformation processing to obtain processed noise reduction audio data; Audio playback is performed based on the processed noise-reduced audio data.

8. The method according to claim 2, characterized in that: The obtaining of first sample audio data collected by the first sample audio device, second sample audio data collected by the second sample audio device, and third sample audio data collected by the third sample audio device includes: Acquire third sample audio data collected by a third sample audio device; Playing the audio corresponding to the third sample audio data at different locations in the studio, and / or adjusting the playback volume of the third sample audio data and playing the audio corresponding to the adjusted third sample audio data; During the playing process, the first sample audio device collects audio to obtain first sample audio data, and at the same time, the second sample audio device collects audio to obtain second sample audio data.

9. An audio processing device, characterized in that: include: an acquisition module, configured to acquire first audio data collected by a first audio device and second audio data collected by a second audio device, wherein a first acquisition distance between the first audio device and a clean sound source is smaller than a second acquisition distance between the second audio device and the clean sound source, and a first interference distance between the first audio device and a noise source is larger than a second interference distance between the second audio device and the noise source; A processing module, configured to input the first audio data and the second audio data into an audio noise reduction model to obtain noise reduction audio data corresponding to the first audio data; The audio noise reduction model is obtained by taking the sample audio data obtained after collecting the sample clean sound source and the sample noise source by the first audio device, the second audio device and the third audio device as input, and the model is trained with the sample audio data collected by the third audio device as the fitting target, wherein the third collection distance between the third audio device and the sample clean sound source is less than the first collection distance.

10. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the audio processing method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the audio processing method according to any one of claims 1 to 8 are implemented.