Audio processing method, device and storage medium
By identifying the sound source and position parameters in the audio file, and performing sound separation and mixing, the problem of low spatial realism in two-channel stereo audio content is solved, achieving more realistic audio space simulation and enhancing the sense of presence and spatial experience.
Patent Information
- Application Number
- CN202111566841.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-20
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-12-20
AI Technical Summary
Existing HRTF-based spatial rendering technology cannot preserve the accurate spatial location of sound sources in two-channel stereo audio content, resulting in low realism in the simulated sense of presence and spatiality.
By identifying the sound-producing objects and positional parameters in audio files, sound separation and mixing are performed to generate associated audio track data. Spatial rendering is then performed based on scene type and positional parameters to improve the spatial realism of audio files.
It significantly improves the realism of the simulated presence and spatial feel of stereo audio content, enhancing the immersive and interactive experience of audio playback.
Smart Images

Figure CN114286275B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio, and in particular to an audio processing method and device, and a storage medium. Background Art
[0002] In recent years, spatial sound technology has been widely used in electronic products. Spatial sound technology can create a real sense of presence and space when users enjoy movies or music. The main spatial sound technology currently used is the spatial rendering technology based on the Head Related Transfer Function (HRTF), and the HRTF-based spatial rendering technology relies on multi-channel or target-based sound formats to produce a real spatial effect. For two-channel stereo audio content, since it cannot retain the accurate spatial position of each sound source after mixing, it is necessary to simulate spatial sound through sound enhancement or virtual surround sound technology, which will lead to the problem of low authenticity of the simulated sense of presence and space. Summary of the invention
[0003] The embodiments of the present application provide an audio processing method and device, and a storage medium, which can improve the authenticity of simulated presence and spatial sense for stereo audio content.
[0004] The technical solution of this application is implemented as follows:
[0005] In a first aspect, an embodiment of the present application provides an audio processing method, the method comprising:
[0006] Determine the scene type of the audio scene corresponding to the audio file;
[0007] Identify a first sounding object and a second sounding object in the audio file;
[0008] respectively determining a first position parameter and a second position parameter of the first sound emitting object and the second sound emitting object in the audio scene;
[0009] Separate the audio file based on the object types of the first sound object and the second sound object to obtain first audio track data associated with the first sound object, second audio track data associated with the second sound object, and background audio track data associated with the audio scene;
[0010] The first audio track data, the second audio track data and the background audio track data are mixed based on the scene type and the first position parameter and the second position parameter to obtain audio data to be played.
[0011] In a second aspect, an embodiment of the present application provides an audio processing device, the device comprising:
[0012] an object recognition unit configured to recognize a first sound-emitting object and a second sound-emitting object in an audio file, and to respectively determine a first position parameter and a second position parameter of the first sound-emitting object and the second sound-emitting object in an audio scene;
[0013] a sound separation unit configured to perform sound separation on the audio file based on the respective object types of the first sound object and the second sound object, and obtain first audio track data associated with the first sound object, second audio track data associated with the second sound object, and background audio track data associated with the audio scene;
[0014] The audio track processing unit is configured to mix the first audio track data, the second audio track data and the background audio track data based on the scene type and the first position parameter and the second position parameter to obtain audio data to be played.
[0015] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the audio processing method as described above is implemented.
[0016] In a fourth aspect, an embodiment of the present application provides an application processor for processing an audio file, wherein the application processor is configured to:
[0017] Determining a scene type of an audio scene corresponding to the audio file;
[0018] Identify a first sounding object and a second sounding object in the audio file;
[0019] respectively determining a first position parameter and a second position parameter of the first sound emitting object and the second sound emitting object in the audio scene;
[0020] Separate the audio file based on the object types of the first sound object and the second sound object to obtain first audio track data associated with the first sound object, second audio track data associated with the second sound object, and background audio track data associated with the audio scene;
[0021] The first audio track data, the second audio track data and the background audio track data are mixed based on the scene type and the first position parameter and the second position parameter to obtain audio data to be played.
[0022] In a fifth aspect, an embodiment of the present application proposes an electronic device, comprising the above-mentioned application processor.
[0023] The embodiment of the present application provides an audio processing method and device, and a storage medium, the method comprising: determining the scene type of an audio scene corresponding to an audio file; identifying a first sound object and a second sound object in the audio file; respectively determining a first position parameter and a second position parameter of the first sound object and the second sound object in the audio scene; performing sound separation on the audio file based on the object types of the first sound object and the second sound object, obtaining first track data associated with the first sound object, second track data associated with the second sound object, and background track data associated with the audio scene; performing mixing processing on the first track data, the second track data, and the background track data based on the scene type and the first position parameter and the second position parameter, to obtain audio data to be played. By adopting the above implementation scheme, the audio file can be sound separated according to different sound objects, and the track data corresponding to different sound objects can be obtained. After that, the track data of different sound objects can be processed based on the scene type of the audio scene and the position parameters of the sound object in the audio scene, so that a real audio playback space can be simulated, which greatly improves the authenticity of the simulated sense of presence and sense of space. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 A flowchart of an audio processing method provided in an embodiment of the present application;
[0025] Figure 2 A schematic diagram of an exemplary configuration of parameters on a configuration interface provided in an embodiment of the present application;
[0026] Figure 3 A schematic diagram of an exemplary user-worn motion measurement device provided in an embodiment of the present application;
[0027] Figure 4 A schematic diagram of the structure of an exemplary audio processing device combining audio files and video images provided in an embodiment of the present application;
[0028] Figure 5 A flowchart of an exemplary audio processing method combining audio files and video images provided in an embodiment of the present application;
[0029] Figure 6 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application Figure 1 ;
[0030] Figure 7 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION
[0031] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.
[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0033] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0034] The present application embodiment provides an audio processing method, such as Figure 1 As shown, the method may include:
[0035] S101: Determine a scene type of an audio scene corresponding to an audio file.
[0036] An audio processing method provided in an embodiment of the present application is suitable for a scenario in which spatial sound effects are processed on an audio file in a stereo format.
[0037] In the embodiment of the present application, the scene type of the audio scene can be audio types such as movies and music, small rooms, medium-sized rooms, large rooms, concert halls, stadiums, outdoor spaces, etc. The scene type of the audio scene can be specifically subdivided according to actual conditions, and the embodiment of the present application does not make specific limitations.
[0038] S102: Identify a first sound object and a second sound object in the audio file.
[0039] In the embodiment of the present application, the object type of the sound object in the audio file can be sound source types such as people, musical instruments, environmental events, etc., among which people can be further subdivided into conversation state, singing state, etc., musical instruments can be further subdivided into drums, guitars, pianos, bass, etc., and environmental events can be further subdivided into car driving, collisions, explosions, etc. The specific ones can be selected according to actual conditions, and the embodiment of the present application does not make specific limitations.
[0040] S103: Determine first position parameters and second position parameters of the first sound emitting object and the second sound emitting object in the audio scene respectively.
[0041] It should be noted that different position parameters of different sound-emitting objects in the audio scene may be three-dimensional coordinates of the sound-emitting objects in the audio scene.
[0042] In an embodiment of the present application, the scene type of the audio scene corresponding to the audio file can be determined in the following three ways: identifying the first sound object and the second sound object in the audio file; and determining the first position parameter and the second position parameter of the first sound object and the second sound object in the audio scene respectively.
[0043] In an optional embodiment, a video image corresponding to an audio file is detected to obtain a scene type; a first sound object and a second sound object in the audio file are identified from the video image; positions of the first sound object and the second sound object in the video image are respectively determined; scene size information corresponding to the scene type is determined; and based on the scene size information and the positions of the first sound object and the second sound object in the video image, first position parameters and second position parameters of the first sound object and the second sound object in the audio scene are determined.
[0044] In an embodiment of the present application, a video image corresponding to an audio file is determined, and scene detection and target detection are performed on the video image respectively. Specifically, the scene type is determined through scene detection, and the object type of the first sound object, the object type of the second sound object, the position of the first sound object in the video image, and the position of the second sound object in the video image are determined through target detection. Thereafter, the scene size information corresponding to the determined scene type is searched from the correspondence between each pre-set scene type and scene size information, and the first position parameters and the second position parameters of the first sound object and the second sound object in the audio scene are determined based on the scene size information and the position of the first sound object in the video image and the position of the second sound object in the video image.
[0045] It should be noted that a deep neural network (DNN) can be used to implement scene detection of video images, wherein the DNN may include network layers such as a convolutional neural network (CNN) or a recurrent neural network (RNN), a long short-term memory network (LSTM), etc. Among them, one or more frames of video images are input into the DNN, and the probability that one or more frames of video images belong to each scene type is output, and then the scene type of the video image is determined based on each probability. The scene type can be used for subsequent target type recognition, such as for film scenes, the target types of people and environmental events are mainly recognized, and for music scenes, the target types of singing and musical instruments are mainly recognized.
[0046] It should be noted that DNN can be used to realize target detection in video images, wherein DNN can include network layers such as CNN or RNN, LSTM, etc. wherein one or more video frames are input into DNN, and the probability that one or more video frames contain each preset sound object and the first position of the sound object in the video image are output. For moving sound objects, position tracking is also required, which can be achieved through tracking algorithms based on methods such as Kalman filtering and particle filtering.
[0047] Exemplarily, scene detection is performed on the video image, and it is detected that the video image corresponds to a concert scene. At the same time, target detection is performed on the video image, and the detected sound objects include lead singer, guitar, bass, drum set, and piano, wherein the lead singer is located in the middle of the concert stage area in the video image, the guitar is located on the left side of the concert stage area in the video image, the bass is located on the right side of the concert stage area in the video image, and the piano and drum set are located behind the concert stage area in the video image.
[0048] In another optional embodiment, based on a first touch operation on a preset scene type in a configuration interface corresponding to the audio file, the scene type of the audio scene is determined; based on a second touch operation on a preset object type in the configuration interface, the first sound object and the second sound object in the audio file, as well as the positions of the first sound object and the second sound object in a simulated arrangement area are identified; the configuration interface includes a simulated arrangement area; scene size information corresponding to the scene type is determined; based on the scene size information and the positions of the first sound object and the second sound object in the simulated arrangement area, first position parameters and second position parameters of the first sound object and the second sound object in the audio scene are determined.
[0049] It should be noted that a configuration interface can be provided for the audio file, wherein the configuration interface includes multiple preset scene types, multiple preset object types and a simulated arrangement area. The user can configure the scene type of the audio scene through a first touch operation on the multiple preset scene types, and configure the object type of the sound object and the position of the sound object in the simulated arrangement area through a second touch operation on the multiple preset object types.
[0050] It should be noted that the operation types of the first touch operation and the second touch operation may include a touch operation, a press operation, a drag operation, and the like.
[0051] It should be noted that the second touch operation type is a drag operation, and the preset object type can be dragged into the simulated object arrangement area to determine the object type of the sound object and the position of the sound object in the simulated arrangement area.
[0052] For example, Figure 2 As shown, the configuration interface displays a scene selection module, a scene style selection module and an advanced setting module, wherein the scene type of the concert hall is selected by touch in the scene selection module, the automatic scene style type is selected by touch in the scene style selection module, and in the advanced setting module, the person is dragged to the virtual stage position 1, the guitar is dragged to the virtual stage position 2, the bass is dragged to the virtual stage position 3, the drum set is dragged to the virtual stage position 4, and the piano is dragged to the virtual stage position 5. Each sound object is an independent control, and the volume and sound field width of the sound object can be adjusted through each control.
[0053] In another optional embodiment, based on pre-configuration information corresponding to the audio file, the scene type of the audio scene corresponding to the audio file is determined; the first sound object and the second sound object in the audio file are identified; and the first position parameters and the second position parameters of the first sound object and the second sound object in the audio scene are respectively determined.
[0054] It should be noted that for pure audio files, the scene type of the audio scene, the object type of the sound object, and the position parameters of the sound object in the audio scene can be pre-configured according to the pure audio file. For example, considering that pure audio files are usually music-based, the preset sound objects can be configured as human voices and preset musical instruments; based on the recognition of the music style of the pure audio file, the preset scene type and the position parameters of the sound object in the preset scene can be configured.
[0055] S104, separating the sound of the audio file based on the object types of the first sound object and the second sound object, to obtain first audio track data associated with the first sound object, second audio track data associated with the second sound object, and background audio track data associated with the audio scene.
[0056] In an embodiment of the present application, the audio file is sound separated based on different object types to obtain audio track data corresponding to each object type and background audio track data that does not belong to any object type.
[0057] It should be noted that the DNN can be trained based on a large number of independent audio track data and the corresponding mixed audio data, and then the mixed stereo audio file and the object type to be separated are input into the DNN to obtain the audio track data corresponding to each object type and other unseparated audio track data. The DNN may include network layers such as CNN, RNN, and LSTM.
[0058] In the embodiment of the present application, the sound enhancement parameter can also be determined based on the object type; after the audio file is sound-separated based on the object types of the first sound-emitting object and the second sound-emitting object, and the first audio track data associated with the first sound-emitting object, the second audio track data associated with the second sound-emitting object, and the background audio track data associated with the audio scene are obtained, the first audio track data, the second audio track data, and the background audio track data are sound-enhanced using the sound enhancement parameter to obtain the enhanced first audio track data, the enhanced second audio track data, and the enhanced background audio track data. In this way, different sound enhancements can be performed on different audio track data separately, and the listening experience of each type of audio track data can be improved without affecting other audio track data.
[0059] It should be noted that the sound enhancement processing may include equalizer, reverberation, treble enhancement, bass enhancement, etc. The corresponding sound enhancement parameters may be set in advance for each object type. For example, for drum sounds, impact sounds, etc., bass enhancement, dynamic enhancement or reverberation processing may be performed to make the sound deeper and more powerful; for human voices, treble enhancement may be performed to make the sound clearer and brighter.
[0060] S105 , mixing the first audio track data, the second audio track data, and the background audio track data based on the scene type, the first position parameter, and the second position parameter to obtain audio data to be played.
[0061] Specifically, based on the first position parameter and the second position parameter, the first audio track data, the second audio track data and the background audio track data are spatially rendered to obtain the first spatial audio track data, the second spatial audio track data and the background spatial audio track data; the mixing gain is determined based on the scene type; and based on the mixing gain, the first spatial audio track data, the second spatial audio track data and the background spatial audio track data are mixed to obtain the audio data to be played.
[0062] It should be noted that in the process of spatial rendering of the first audio track data, the second audio track data and the background audio track data, the first audio track data, the second audio track data and the background audio track data can also be processed based on the reverberation characteristics corresponding to the scene category.
[0063] It should be noted that the mixing gain is used to adjust the volume ratio between the data of each audio track.
[0064] Furthermore, motion tracking data can also be determined based on the received inertial measurement data; then, based on the motion tracking data, the first position parameter and the second position parameter are adjusted to obtain adjusted first position parameters and adjusted second position parameters; based on the adjusted position parameters and the adjusted second position parameters, the first audio track data, the second audio track data and the background audio track data are spatially rendered to obtain first spatial audio track data, second spatial audio track data and background spatial audio track data.
[0065] It should be noted that the user's inertial measurement data can be detected in real time by an inertial measurement unit (IMU) such as an accelerometer and / or a gyroscope, and the movement trajectory of the user's head and hands can be calculated using a tracking algorithm such as a Kalman filter or a particle filter. Then, based on the movement trajectory of the head and hands, the relative position and relative direction between the user and the center point of the video are determined. The relative position and relative direction are the motion tracking data in this application. Afterwards, the position parameters are adjusted accordingly by moving or rotating them using the motion tracking data, so that the position parameters can be adjusted accordingly with the movement of the user's head and hands.
[0066] Optionally, the motion measurement device equipped with an IMU can be a True Wireless Stereo (TWS) headset, a mobile phone, etc. The specific one can be selected according to the actual situation, and the embodiment of the present application does not make specific limitations. Figure 3 As shown, the user wears a motion measurement device 301, that is, a headset, and the user holds a motion measurement device 302, that is, a mobile phone.
[0067] For example, if it is detected that the user turns his head to the left, the position of the corresponding audio track will change accordingly. Specifically, the position of the vocals will change from the front to the right, the sound of the guitar will change from the front left to the front right, and the sound of the bass will change from the front right to the back right. If the user moves the phone up, down, left, or right, the relative position of the entire stage and the user himself will change. For example, if the phone is moved upward, the position parameter needs to move upward with the movement of the phone, and the user will feel that the entire stage is rising. This design not only enhances the authenticity of the sense of presence, but also allows the sound to jump to the rhythm, enhancing the user's sense of interaction.
[0068] In an embodiment of the present application, during the process of spatial rendering of the audio track data, binaural filtering may be performed on the audio track data based on preset binaural filtering parameters of the HRTF.
[0069] It should be noted that ear pictures of both ears can be taken in advance, and then the ear pictures can be processed using DNN to obtain ear features and generate binaural filter parameters of the HRTF corresponding to the ear features.
[0070] In an embodiment of the present application, the spatial audio track data is gain-adjusted according to the mixing gain and added to obtain the final audio data to be played.
[0071] It can be understood that the sound of the audio file can be separated according to different sound objects to obtain the audio track data corresponding to different sound objects. Afterwards, the audio track data of different sound objects can be processed based on the scene type of the audio scene and the position parameters of the sound object in the audio scene, which can simulate the real audio playback space and greatly improve the authenticity of the simulated sense of presence and space.
[0072] Based on the above embodiments, the present application embodiment proposes an audio processing method combining audio files and video images, and the structure of the corresponding audio processing device is as follows: Figure 4As shown, it includes a video processing module, an audio processing module and one or more inertial measurement modules. These three types of modules can be the same or different. For example, headphones can be used as an inertial measurement module, while the mobile phone can be used as a video processing module and an audio processing module at the same time. Among them, the video processing module performs scene detection on the video image to obtain the scene type, uses the neural network accelerator to perform target detection on the video image, obtains the first sound object, the second sound object and determines the positions of the first sound object and the second sound object in the video image; the audio processing module performs audio separation on the audio file based on the object categories of the first sound object and the second sound object, and obtains the first audio track data associated with the first sound object, the second audio track data associated with the second sound object and the background audio track data associated with the audio scene; at the same time, the inertial measurement device performs motion tracking to determine the motion tracking data, and the audio processing module performs spatial rendering on the first audio track data, the second audio track data and the background audio track data based on the scene type, the object categories of the first sound object and the second sound object, the positions of the first sound object and the second sound object in the video image and the motion tracking data to obtain the audio data to be played, and plays the audio data to be played through headphones. At the same time, the corresponding video image can also be synchronously displayed on the mobile phone.
[0073] It should be noted that in order to increase the processing speed, neural network accelerators can be used in both the video processing module and the audio processing module to perform scene detection, object detection and audio separation processing. When the processing speed meets the real-time requirements, that is, the processing time for each frame of audio and video data is less than the corresponding playback time, the entire processing flow can be performed in real time while playing audio and video.
[0074] The process of the audio processing method combining audio files and video images is as follows Figure 5 As shown,
[0075] 501. Perform scene detection on the video image to obtain a scene type;
[0076] 502. Perform object detection on the video image to obtain object types corresponding to the first sound-emitting object and the second sound-emitting object, and positions of the first sound-emitting object and the second sound-emitting object in the video image;
[0077] 503. Determine a mixing gain, a first position parameter and a second position parameter of the first sound emitting object and the second sound emitting object in the audio scene according to the scene type and the positions of the first sound emitting object and the second sound emitting object in the video image;
[0078] 504. Separate the audio file according to the object types corresponding to the first sound object and the second sound object, and obtain first audio track data associated with the first sound object, second audio track data associated with the second sound object, and background audio track data associated with the audio scene;
[0079] 505. Determine a sound enhancement parameter according to the object type, and use the sound enhancement parameter to perform sound enhancement on the first audio track data, the second audio track data, and the background audio track data to obtain enhanced first audio track data, enhanced second audio track data, and enhanced background audio track data;
[0080] 506. Process the inertial measurement data collected by the motion measurement module to obtain motion tracking data;
[0081] 507. Based on the motion tracking data, the first position parameter and the second position parameter, spatial rendering is performed on the enhanced first audio track data, the enhanced second audio track data and the enhanced background audio track data to obtain first spatial audio track data, second spatial audio track data and background spatial audio track data;
[0082] 508. According to the mixing gain, the first spatial audio track data, the second spatial audio track data and the background spatial audio track data are mixed to obtain the audio data to be played, and the audio data to be played and the video image are played simultaneously.
[0083] It should be noted that by comparing the audio data to be played after the sound effect processing in this application with the sound in the original audio file, the sense of space is greatly enhanced, especially for different sound objects. There is no sense of direction in the original audio file, but after the audio processing of this application, it can be clearly felt from different directions or different positions in space. When the sound object moves in the video image, the sound in the original audio file does not change accordingly, but after the audio processing, the virtual sound source movement effect that matches the picture can be perceived. The audio data to be played obtained by the audio processing method of this application can make the position and movement of the virtual sound target match the video image, and at the same time, the position parameters of the sound object can be matched with the audio scene displayed by the video image, which can create a real sense of presence and space, making the user feel as if they are at the performance scene.
[0084] It is understandable that through scene detection and target detection of video images, combined with audio separation and spatial rendering, an independent sound image can be established for the audio track data corresponding to each sound-emitting object in the video image, and matched with the target position in the video. Combined with the inertial measurement data collected by one or more motion measurement modules, the position parameters are updated during spatial rendering. This technology can not only greatly enhance the sense of presence and space of audio and video, but also add more interaction and fun in the process of enjoying audio and video. For video content that only contains ordinary stereo format audio, this immersive experience is something that traditional sound enhancement and virtual surround technology cannot achieve.
[0085] This embodiment of the application provides an audio processing device 601. Figure 6 As shown, the audio processing device 601 includes:
[0086] The object recognition unit 10 is configured to determine a scene type of an audio scene corresponding to an audio file; identify a first sound object and a second sound object in the audio file, and respectively determine a first position parameter and a second position parameter of the first sound object and the second sound object in the audio scene;
[0087] A sound separation unit 11 is configured to perform sound separation on the audio file based on the object types of the first sound object and the second sound object, and obtain first audio track data associated with the first sound object, second audio track data associated with the second sound object, and background audio track data associated with the audio scene;
[0088] The audio track processing unit 12 is configured to mix the first audio track data, the second audio track data and the background audio track data based on the scene type and the first position parameter and the second position parameter to obtain audio data to be played.
[0089] Optionally, the audio processing device 601 further includes: a spatial rendering unit and a mixing unit;
[0090] The spatial rendering unit is configured to perform spatial rendering on the first audio track data, the second audio track data and the background audio track data based on the first position parameter and the second position parameter to obtain the first spatial audio track data, the second spatial audio track data and the background spatial audio track data;
[0091] The determining unit is further configured to determine a mixing gain based on the scene type;
[0092] The mixing unit is configured to determine a mixing gain based on the scene type; and based on the mixing gain, mix the first spatial audio track data, the second spatial audio track data and the background spatial audio track data to obtain the audio data to be played.
[0093] Optionally, the audio processing device 601 further includes: a position adjustment unit;
[0094] The determining unit is further configured to determine motion tracking data based on the received inertial measurement data;
[0095] The position adjustment unit is configured to adjust the first position parameter and the second position parameter based on the motion tracking data to obtain an adjusted first position parameter and an adjusted second position parameter;
[0096] The spatial rendering unit is also configured to perform spatial rendering on the first audio track data, the second audio track data and the background audio track data based on the adjusted position parameters and the adjusted second position parameters to obtain the first spatial audio track data, the second spatial audio track data and the background spatial audio track data.
[0097] Optionally, the audio processing device 601 further includes: a sound enhancement unit;
[0098] The determining unit is further used to determine a sound enhancement parameter based on the object type;
[0099] The sound enhancement unit is used to perform sound enhancement on the first audio track data, the second audio track data and the background audio track data using the sound enhancement parameters to obtain enhanced first audio track data, enhanced second audio track data and enhanced background audio track data;
[0100] The audio track processing unit 12 is further configured to perform mixing processing on the enhanced first audio track data, the enhanced second audio track data and the enhanced background audio track data based on the scene type and the first position parameter and the second position parameter to obtain the audio data to be played.
[0101] Optionally, the audio processing device 601 further includes: a video detection unit;
[0102] The video detection unit is configured to detect the video image corresponding to the audio file to obtain the scene type; identify the first sound object and the second sound object in the audio file from the video image; and respectively determine the positions of the first sound object and the second sound object in the video image;
[0103] The determination unit is further configured to determine scene size information corresponding to the scene type; and determine the first position parameters and the second position parameters of the first sound object and the second sound object in the audio scene based on the scene size information and the positions of the first sound object and the second sound object in the video image.
[0104] Optionally, the determination unit is further configured to determine the scene type of the audio scene based on a first touch operation on a preset scene type in a configuration interface corresponding to the audio file; identify the first sound object and the second sound object in the audio file, and the positions of the first sound object and the second sound object in a simulated arrangement area based on a second touch operation on a preset object type in the configuration interface; the configuration interface includes the simulated arrangement area; determine the scene size information corresponding to the scene type; and determine the first position parameter and the second position parameter of the first sound object and the second sound object in the audio scene based on the scene size information and the positions of the first sound object and the second sound object in the simulated arrangement area.
[0105] Optionally, the determination unit is further configured to determine the scene type of the audio scene corresponding to the audio file; identify the first sound object and the second sound object in the audio file; and respectively determine the first position parameter and the second position parameter of the first sound object and the second sound object in the audio scene.
[0106] An audio processing device provided in an embodiment of the present application determines the scene type of an audio scene corresponding to an audio file; identifies a first sound object and a second sound object in the audio file; respectively determines the first position parameter and the second position parameter of the first sound object and the second sound object in the audio scene; performs sound separation on the audio file based on the object types of the first sound object and the second sound object, and obtains first track data associated with the first sound object, second track data associated with the second sound object, and background track data associated with the audio scene; performs mixing processing on the first track data, the second track data, and the background track data based on the scene type and the first position parameter and the second position parameter, and obtains audio data to be played. It can be seen that the audio processing device proposed in this embodiment can perform sound separation on the audio file according to different sound objects, obtain track data corresponding to different sound objects, and then process the track data of different sound objects based on the scene type of the audio scene and the position parameters of the sound objects in the audio scene, so as to simulate a real audio playback space, greatly improving the authenticity of the simulated sense of presence and sense of space.
[0107] The embodiment of the present application further proposes an application processor 701 for processing an audio file.
[0108] The application processor is configured to: determine the scene type of the audio scene corresponding to the audio file; identify the first sound object and the second sound object in the audio file; respectively determine the first position parameter and the second position parameter of the first sound object and the second sound object in the audio scene; perform sound separation on the audio file based on the object types of the first sound object and the second sound object, and obtain first audio track data associated with the first sound object, second audio track data associated with the second sound object, and background audio track data associated with the audio scene; perform mixing processing on the first audio track data, the second audio track data, and the background audio track data based on the scene type and the first position parameter and the second position parameter, and obtain audio data to be played.
[0109] Figure 7 A schematic diagram of the structure of an electronic device 702 provided in an embodiment of the present application Figure 2 In practical applications, based on the same disclosure concept of the above embodiments, Figure 7 As shown, the electronic device 702 of this embodiment includes the above-mentioned application processor 701.
[0110] Optionally, the electronic device 702 further includes a neural network processing unit 703 configured to detect the video image corresponding to the audio file to obtain the scene type;
[0111] Identify a first sound object and a second sound object in the audio file from the video image; and determine positions of the first sound object and the second sound object in the video image respectively;
[0112] The audio file is sound separated based on the respective object types of the first sound-emitting object and the second sound-emitting object to obtain first audio track data associated with the first sound-emitting object, second audio track data associated with the second sound-emitting object, and background audio track data associated with the audio scene.
[0113] An embodiment of the present application provides a storage medium having a computer program stored thereon. The computer-readable storage medium stores one or more programs. The one or more programs can be executed by one or more processors and applied to an audio processing device. The computer program implements the audio processing method as described above.
[0114] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for enabling an image display device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.
[0116] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.
Claims
1. An audio processing method, It is characterized in that The method comprises: Determine the scene type of the audio scene corresponding to the audio file; Identify a first sounding object and a second sounding object in the audio file; respectively determining a first position parameter and a second position parameter of the first sound emitting object and the second sound emitting object in the audio scene; Separate the audio file based on the object types of the first sound object and the second sound object to obtain first audio track data associated with the first sound object, second audio track data associated with the second sound object, and background audio track data associated with the audio scene; Performing mixing processing on the first audio track data, the second audio track data and the background audio track data based on the scene type and the first position parameter and the second position parameter to obtain audio data to be played; The step of determining a scene type of an audio scene corresponding to an audio file; identifying a first sound object and a second sound object in the audio file; and respectively determining a first position parameter and a second position parameter of the first sound object and the second sound object in the audio scene includes: Detecting the video image corresponding to the audio file to obtain the scene type; Identify a first sound object and a second sound object in the audio file from the video image; and determine positions of the first sound object and the second sound object in the video image respectively; Determine scene size information corresponding to the scene type; Based on the scene size information and the positions of the first sound emitting object and the second sound emitting object in the video image, the first position parameter and the second position parameter of the first sound emitting object and the second sound emitting object in the audio scene are determined.
2. The method according to claim 1, It is characterized in that The mixing process of the first audio track data, the second audio track data and the background audio track data based on the scene type and the first position parameter and the second position parameter to obtain the audio data to be played includes: Based on the first position parameter and the second position parameter, spatially render the first audio track data, the second audio track data, and the background audio track data to obtain first spatial audio track data, second spatial audio track data, and background spatial audio track data; A mixing gain is determined based on the scene type; and based on the mixing gain, the first spatial audio track data, the second spatial audio track data and the background spatial audio track data are mixed to obtain the audio data to be played.
3. The method according to claim 2, It is characterized in that The method further comprises: determining motion tracking data based on the received inertial measurement data; The mixing process of the first audio track data, the second audio track data and the background audio track data based on the scene type and the first position parameter and the second position parameter to obtain the audio data to be played includes: Based on the motion tracking data, adjusting the first position parameter and the second position parameter to obtain an adjusted first position parameter and an adjusted second position parameter; Based on the adjusted position parameter and the adjusted second position parameter, the first audio track data, the second audio track data and the background audio track data are spatially rendered to obtain first spatial audio track data, second spatial audio track data and the background spatial audio track data.
4. The method according to claim 1, It is characterized in that The method further comprises: Based on the object type, determining a sound enhancement parameter; Performing sound enhancement on the first audio track data, the second audio track data and the background audio track data using the sound enhancement parameters to obtain enhanced first audio track data, enhanced second audio track data and enhanced background audio track data; The mixing process of the first audio track data, the second audio track data and the background audio track data based on the scene type and the first position parameter and the second position parameter to obtain the audio data to be played includes: The enhanced first audio track data, the enhanced second audio track data and the enhanced background audio track data are mixed based on the scene type and the first position parameter and the second position parameter to obtain the audio data to be played.
5. The method according to claim 1, It is characterized in that The step of determining a scene type of an audio scene corresponding to an audio file; identifying a first sound object and a second sound object in the audio file; and respectively determining a first position parameter and a second position parameter of the first sound object and the second sound object in the audio scene further includes: Determining a scene type of the audio scene based on a first touch operation on a preset scene type in a configuration interface corresponding to the audio file; Based on a second touch operation on a preset object type in the configuration interface, identifying a first sound object and a second sound object in the audio file, and positions of the first sound object and the second sound object in a simulated arrangement area; the configuration interface includes the simulated arrangement area; Determine scene size information corresponding to the scene type; Based on the scene size information and the positions of the first sound emitting object and the second sound emitting object in the simulated arrangement area, the first position parameter and the second position parameter of the first sound emitting object and the second sound emitting object in the audio scene are determined.
6. The method according to claim 1, It is characterized in that The step of determining a scene type of an audio scene corresponding to an audio file; identifying a first sound object and a second sound object in the audio file; and respectively determining a first position parameter and a second position parameter of the first sound object and the second sound object in the audio scene further includes: Based on the pre-configuration information corresponding to the audio file, determine the scene type of the audio scene corresponding to the audio file; identify the first sound object and the second sound object in the audio file; and respectively determine the first position parameter and the second position parameter of the first sound object and the second sound object in the audio scene.
7. An audio processing device, It is characterized in that The device comprises: an object recognition unit configured to recognize a first sound-emitting object and a second sound-emitting object in an audio file, and to respectively determine a first position parameter and a second position parameter of the first sound-emitting object and the second sound-emitting object in an audio scene; a sound separation unit configured to perform sound separation on the audio file based on the respective object types of the first sound object and the second sound object, and obtain first audio track data associated with the first sound object, second audio track data associated with the second sound object, and background audio track data associated with the audio scene; An audio track processing unit configured to perform mixing processing on the first audio track data, the second audio track data and the background audio track data based on the scene type and the first position parameter and the second position parameter to obtain audio data to be played; The audio processing device further includes: a video detection unit; The video detection unit is configured to detect the video image corresponding to the audio file to obtain the scene type; identify the first sound object and the second sound object in the audio file from the video image; and respectively determine the positions of the first sound object and the second sound object in the video image; The determination unit is further configured to determine scene size information corresponding to the scene type; and determine the first position parameter and the second position parameter of the first sound object and the second sound object in the audio scene based on the scene size information and the positions of the first sound object and the second sound object in the video image.
8. A storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
9. An electronic device, It is characterized in that include: An application processor is used to process the audio file, wherein the application processor is configured to: Determining a scene type of an audio scene corresponding to the audio file; Identify a first sounding object and a second sounding object in the audio file; respectively determining a first position parameter and a second position parameter of the first sound emitting object and the second sound emitting object in the audio scene; Separate the audio file based on the object types of the first sound object and the second sound object to obtain first audio track data associated with the first sound object, second audio track data associated with the second sound object, and background audio track data associated with the audio scene; Performing mixing processing on the first audio track data, the second audio track data and the background audio track data based on the scene type and the first position parameter and the second position parameter to obtain audio data to be played; The application processor is further configured to: Detecting the video image corresponding to the audio file to obtain the scene type; Identify a first sound object and a second sound object in the audio file from the video image; and determine positions of the first sound object and the second sound object in the video image respectively; Determine scene size information corresponding to the scene type; Based on the scene size information and the positions of the first sound emitting object and the second sound emitting object in the video image, the first position parameter and the second position parameter of the first sound emitting object and the second sound emitting object in the audio scene are determined.
10. The electronic device as claimed in claim 9, It is characterized in that The electronic device further includes a neural network processing unit configured to: Detecting the video image corresponding to the audio file to obtain the scene type; Identify a first sound object and a second sound object in the audio file from the video image; and determine positions of the first sound object and the second sound object in the video image respectively; The audio file is sound separated based on the respective object types of the first sound-emitting object and the second sound-emitting object to obtain first audio track data associated with the first sound-emitting object, second audio track data associated with the second sound-emitting object, and background audio track data associated with the audio scene.
