Audio processing method and related apparatus

Abstract

The present application provides an audio processing method and a related apparatus. An electronic device can determine sound image information and sound field environment information on the basis of a video image and an audio in a video. On the basis of the information above, the electronic device can provide an audio editing control on a user interface, so that a user can perform audio editing such as sound image editing, sound field editing, and audio mixing editing. The electronic device can process the audio on the basis of an audio editing operation of the user, so that after the processed audio is rendered, the sound heard by the user can conform to an adjustment effect desired by the user. The audio editing operation is simple, convenient, and visual, users can conveniently perform personalized editing on audios in audio / video recording scenes such as video recording scenes and video call scenes, and the user experience of audio / video recording is improved.

Description

Audio processing method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 12, 2023, with application number 202310691922.0, and priority to the Chinese patent application entitled “Audio Processing Method and Related Devices”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to an audio processing method and related devices. Background Art

[0003] With the advancement of audio and video technologies, sound reproduction techniques have evolved from mono to dual-channel, stereo, surround, and even three-dimensional sound. More and more audio formats are enabling users to experience a sense of surround and immersion. In audio recording or playback scenarios, users may need to personalize the audio to achieve and listen to audio that meets their individual needs.

[0004] Summary of the Invention

[0005] This application provides an audio processing method and related apparatus. Electronic devices can provide audio editing controls on a video screen, allowing users to perform audio editing, such as image and sound editing, sound field editing, and mixing editing. These simple and intuitive audio editing operations facilitate personalized audio editing in scenarios such as video recording and video calls, thereby enhancing the user's audio and video recording experience.

[0006] In a first aspect, the present application provides an audio processing method. A first electronic device may display a first video image of a first video, the first video image including a first sound image, and the first video including a first audio. The first electronic device may display a first sound image control on the screen, the first sound image control being used to adjust the sound characteristics of the first sound image, wherein adjusting the sound characteristics of the first sound image includes one or more of the following: adjusting the volume of the first sound image, adjusting the timbre of the first sound image, and / or adjusting the spatial position of the first sound image. In response to a first operation on the first sound image control, the first electronic device adjusts the sound characteristics of the first sound image in the first audio.

[0007] The first audio may include an audio signal of a first sound image. The electronic device may adjust the sound characteristics of the first sound image by editing the audio signal of the first sound image.

[0008] The first audio and video control can refer to one or more of the following controls in the embodiment of the present application: the volume bar 531A, the mute control 531B, the audio and video control 534 shown in Figure 5C and / or one or more controls in the audio and video editing box 541 shown in Figure 5F.

[0009] As can be seen from the above method, the first electronic device can provide relevant audio and video controls for the audio and video captured on the video image, so that the user can adjust the sound characteristics of the audio and video. The user can edit the audio and video of the captured sound directly on the video image. For example, the user can mute unnecessary audio and video, enhance the human voice of some important audio and video, change the timbre of one or more audio and video, and so on. The above audio and video editing operation is simple and intuitive, and easy for users to understand and get started. In this way, users can obtain audio and video that meets their personal needs during the video shooting process, or let the other party of the call listen to the sound characteristics of the audio and video adjusted by themselves in the video call scenario.

[0010] In conjunction with the first aspect, in some embodiments, the volume of the first sound image may be at a first volume at the first moment of capture. The first operation may be used to adjust the volume of the first sound image to a second volume. During playback of a video with the adjusted sound characteristics of the first sound image, when the video reaches the moment corresponding to the first moment, the volume of the first sound image is at the second volume.

[0011] In conjunction with the first aspect, in some embodiments, the timbre of the first sound image may be the first timbre when captured by the first device. The first operation may be used to adjust the timbre of the first sound image to a second timbre. During playback of a video with the adjusted sound characteristics of the first sound image, when the video reaches the moment corresponding to the first moment, the timbre of the first sound image is the second timbre.

[0012] In conjunction with the first aspect, in some embodiments, the spatial position of the first sound image can be the first position on the first device being captured. The first operation can be used to adjust the spatial position of the first sound image to the second position. During playback of a video with the sound characteristics of the first sound image adjusted, when the video reaches the moment corresponding to the first moment, the spatial position of the first sound image is the second position. In other words, the listener can watch the video and listen to the audio. When listening to the moment corresponding to the first moment, the listener can hear that the first sound image is located at the second position.

[0013] In combination with the first aspect, in some embodiments, the types of the first sound and image may include people and musical instruments.

[0014] In conjunction with the first aspect, in some embodiments, the first video image further includes a second sound image. The first electronic device displays a second sound image control on the screen, where the second sound image control is used to adjust the sound characteristics of the second sound image. Adjusting the sound characteristics of the second sound image includes one or more of the following: adjusting the volume of the second sound image, adjusting the timbre of the second sound image, and / or adjusting the spatial position of the second sound image. In response to a second operation on the second sound image control, the first electronic device adjusts the sound characteristics of the second sound image in the second audio.

[0015] The first electronic device may perform sound object tracking on the first audio to separate audio signals (ie, audio tracks) of different sound images contained in the first audio.

[0016] As can be seen from the above embodiments, the video image displayed by the first electronic device can include multiple captured audio and video images. The user can edit one or more of these audio and video images separately. This can improve the user's experience in recording audio and video and better meet the user's needs for audio editing.

[0017] In combination with the first aspect, in some embodiments, the first electronic device displays a first sound field control on the screen, and the first sound field control is used to adjust the first sound field in the first audio, and adjusting the first sound field includes one or more of the following: adjusting the type of room where the first sound field is located, adjusting the size of the room where the first sound field is located, and / or adjusting the reflective material of the room where the first sound field is located; in response to a third operation on the first sound field control, the first electronic device adjusts the first sound field.

[0018] The first audio may include an audio signal of a first sound field (i.e., a sound bed). In addition to separating the audio signal of the sound image, the first electronic device may also separate the audio signal of the first sound field from the first audio. The first electronic device may then adjust the first sound field by editing the audio signal of the first sound field.

[0019] The above-mentioned first sound field control can refer to one or more controls in the sound field editing box 551 shown in Figure 5I of the present application.

[0020] In combination with the first aspect, in some embodiments, the first electronic device displays a first mixing control on the screen, and the first mixing control is used to adjust the mixing parameters of the first audio, and the mixing parameters include one or more of the following: mixing template, reverberation duration, equalizer parameters, the size of the room where the sound field is located, and / or the reflective material of the room where the sound field is located; the first electronic device determines the first mixing parameter based on the fourth operation of the first mixing control, and mixes the audio signal of the first sound field in the first audio with the audio signals of one or more sound images according to the first mixing parameter.

[0021] The first audio mixing control mentioned above may refer to one or more controls in the audio mixing editing box 561 shown in FIG. 5K to FIG. 5M of the present application.

[0022] The audio signal of the first sound field mixed by the first electronic device may be an adjusted audio signal of the first sound field. The audio signal of one or more sound images mixed by the first electronic device may be an audio signal with adjusted sound characteristics of the one or more sound images.

[0023] As can be seen from the above embodiments, in addition to providing controls for editing audio and video, electronic devices can also provide controls for editing the sound field and / or controls for editing mixing parameters. In this way, users can perform multi-faceted editing of the audio in the video to obtain audio that meets their needs.

[0024] In combination with the first aspect, in some embodiments, the first electronic device receives a fifth operation for the first sound image and the second sound image, and the fifth operation is used to merge the first sound image and the second sound image; in response to the fifth operation, the first electronic device displays a third sound image control on the screen, and the third sound image control is used to adjust the sound characteristics of the first sound image and the second sound image.

[0025] The fifth operation mentioned above may refer to the operations shown in FIG. 6E and FIG. 6F of the present application.

[0026] As can be seen from the above embodiment, the user can merge one or more audio and video images on the video image, thereby simultaneously editing the sound characteristics of multiple audio and video images in one audio editing operation, which can improve the efficiency of the user's audio and video editing.

[0027] In combination with the first aspect, in some embodiments, the first audio further includes an audio signal corresponding to a third sound image, the third sound image is not in the first video image, and the first electronic device displays a fourth sound image control on the screen, and the fourth sound image control is used to adjust the sound characteristics of the third sound image.

[0028] As can be seen from the above embodiments, the camera's shooting angle is limited, and the video image may only contain a portion of the sound and image. The electronic device can provide a control for representing the sound and image that are not captured, so that the user can edit the audio of the sound and image that are not captured.

[0029] In conjunction with the first aspect, in some embodiments, the display position of the fourth audio / video control on the screen is correlated with the spatial position of the third audio / video. Thus, the user can determine which audio / video the fourth audio / video control corresponds to based on its display position, and can then edit the uncaptured audio / video as needed.

[0030] In combination with the first aspect, in some embodiments, the sound feature of the first sound image is the first sound feature, the sound feature of the second sound is the second sound feature, and the first electronic device receives a sixth operation for the first sound image and the second sound image, and the sixth operation is used to exchange the sound features of the first sound image and the second sound image; in response to the sixth operation, the first electronic device determines the sound feature of the first sound image in the first audio as the second sound feature, and determines the sound feature of the second sound image in the first audio as the first sound feature; when receiving an operation on the first sound image control, the first electronic device adjusts the second sound feature; when receiving an operation on the second sound image control, the first electronic device adjusts the first sound feature.

[0031] As can be seen from the above embodiment, when the user believes that the sound features of the first sound image and the second sound image are opposite, the user can swap the sound features of the first sound image and the second sound image, which can improve the accuracy of determining the sound features of the sound image.

[0032] In combination with the first aspect, in some embodiments, the first electronic device receives a seventh operation; in response to the seventh operation, the first electronic device identifies the main object in the first video image; when the main object in the first video image is identified as the first sound and image, the first electronic device removes the audio signals of other sound and images in the first audio except the first sound and image.

[0033] The seventh operation mentioned above may refer to the operations shown in FIG. 12A and FIG. 12B of the present application.

[0034] Optionally, when the first electronic device recognizes that the primary object in the video image of the first video has changed from the first sound image to the second sound image, the first electronic device may remove audio signals of the other sound images except the second sound image from the audio included in the first video. The first electronic device may remove audio signals of the other sound images except the second sound image from the audio starting from the moment the primary object is recognized to have changed to the second sound image.

[0035] As can be seen from the above embodiments, the electronic device can identify the primary subject in the video image, retain the primary subject's sound in the audio, and suppress all other sounds in the audio. This allows the user to quickly eliminate all sounds except the primary subject in the audio during a video call, allowing the user to clearly hear the primary subject's sound. Alternatively, the user can suppress all sounds other than the primary subject in the captured video, resulting in a purer audio quality.

[0036] In combination with the first aspect, in some embodiments, a display position of the first audio and video control on the screen is related to a display position of the first audio and video on the video image of the first video.

[0037] The display position of the first audio and video control can refer to the display position of the volume bar 531A and the mute control 531B shown in Figure 5C of this application. As can be seen, the controls for audio editing can be displayed on the video image, and in particular, the controls for audio and video editing can also be displayed near the audio and video captured in the image. This makes audio editing operations more simple and intuitive for users, improving the user experience of recording audio and video, as well as editing audio.

[0038] In conjunction with the first aspect, in some embodiments, before the first electronic device displays the first video image of the first video, the first electronic device receives an operation to launch a camera application and / or start video recording, captures an image via an image capture device, and captures audio via an audio capture device; wherein the image captured by the image capture device includes the video image of the first video, and the audio captured by the audio capture device includes the first audio; and the first electronic device displays the image captured by the image capture device in real time. The first video image may be an image displayed in real time by the first electronic device during the capture process.

[0039] As can be seen from the above embodiments, users can perform audio editing before starting to shoot a video (i.e., during the video pre-recording phase). Thus, when the video recording phase begins, the electronic device can edit the audio captured during the video recording phase according to the editing parameters determined by the audio editing operation during the video pre-recording phase, so that the recorded audio meets the user's needs. Optionally, users can also perform audio editing while shooting a video.

[0040] Among them, when an audio editing operation (such as audio and video editing operation, sound field editing operation, mixing editing operation, etc.) is detected during the video recording process, the electronic device can determine the corresponding audio editing parameters based on the audio editing operation, and use the audio editing parameters to perform corresponding editing on the audio collected from the moment of the audio editing operation.

[0041] In conjunction with the first aspect, in some embodiments, before the first electronic device displays a first video image of the first video, the first electronic device retrieves the first video from a memory of the first electronic device; and the first electronic device receives an operation to play the first audio. The first video image may be an image displayed by the first electronic device during video playback.

[0042] As can be seen from the above embodiments, users can edit the audio of videos stored in electronic devices. In other words, in addition to editing the audio during the video recording process, users can also perform secondary audio editing on the recorded video after the video recording is completed.

[0043] In combination with the first aspect, in some embodiments, a first electronic device establishes a communication connection with a listening device, and the first electronic device sends a second audio to the listening device for playback, where the second audio is the first audio after one or more of the following adjustments have been made to the first audio: the sound characteristics of one or more sound images in the first audio have been adjusted, the first sound field of the first audio has been adjusted, and / or the mixing parameters of the first audio have been adjusted.

[0044] As can be seen from the above embodiments, during the video recording process, the electronic device can mix and output the audio file in real time, and render the audio file in real time, so that the user can monitor the sound effects of the video recording in real time through the audio playback device. This makes it convenient for the user to promptly understand the sound effects of the audio editing, so that the audio editing can achieve the desired sound effect.

[0045] In combination with the first aspect, in some embodiments, a communication connection is established between the first electronic device and the monitoring device. After the first electronic device adjusts the sound characteristics of the first sound image in the first audio, the first electronic device sends the audio signal of the first sound image with the adjusted sound characteristics to the monitoring device for playback.

[0046] As can be seen from the above embodiment, when a user adjusts the sound characteristics of one or more sound images during video recording, the electronic device can play only the audio signals of the adjusted sound image or images. In this way, the user can listen to only the audio of the edited sound image to determine whether the edit is appropriate.

[0047] In conjunction with the first aspect, in some embodiments, a first electronic device and a second electronic device enter into a first video call. The first electronic device sends a video image and second audio of the first video to the second electronic device. The second audio is audio of the first audio that has undergone one or more of the following adjustments: adjustment of sound characteristics of one or more sound images in the first audio, adjustment of the first sound field of the first audio, and / or adjustment of mixing parameters of the first audio. The first video image may be an image captured by the first electronic device during the first video call.

[0048] In combination with the first aspect, in some embodiments, a first electronic device receives a second video from a second electronic device; the first electronic device displays a second video image of the second video, the second video image including a fourth sound image; the first electronic device displays a fifth sound image control on the screen, the fifth sound image control being used to adjust the sound characteristics of the fourth sound image, adjusting the sound characteristics of the fourth sound image including one or more of the following: adjusting the volume of the fourth sound image, adjusting the timbre of the fourth sound image and / or adjusting the spatial position of the fourth sound image.

[0049] In combination with the first aspect, in some embodiments, the first electronic device determines a first editing parameter based on an operation on a fifth audio and video control; the first electronic device sends a first editing instruction to the second electronic device, the first editing instruction including a first editing parameter; the first electronic device receives a third video from the second electronic device, the third video including a third audio, wherein the sound characteristics of the fourth audio and video in the third audio are adjusted according to the first editing parameter.

[0050] In combination with the first aspect, in some embodiments, a third electronic device is also connected to the first video call, and the third video is also sent to the third electronic device for playback.

[0051] As can be seen from the above embodiment, a user can edit their own audio during a video call, allowing the other party to hear the user-edited audio. Furthermore, a user can also edit the audio of the other party during a video call, allowing themselves and other parties to hear the edited audio of the other party. In this way, the user, as the host, can control the audio of multiple parties connected to the video call, ensuring a more efficient video call. The above embodiment allows users to easily and intuitively edit call audio during a video call, enhancing the user's video call experience.

[0052] In combination with the first aspect, in some embodiments, the first electronic device displays a second sound field control on the screen, and the second sound field control is used to adjust the second sound field in the audio contained in the video sent from the second electronic device, and adjusting the second sound field includes one or more of the following: adjusting the type of room where the second sound field is located, adjusting the size of the room where the second sound field is located, and / or adjusting the reflective material of the room where the second sound field is located.

[0053] In combination with the first aspect, in some embodiments, the first electronic device displays a second mixing control on the screen, and the second mixing control is used to adjust the mixing parameters of the audio contained in the video sent from the second electronic device, and the mixing parameters include one or more of the following: mixing template, reverberation duration, equalizer parameters, the size of the room where the sound field is located, and / or the reflective material of the room where the sound field is located.

[0054] As can be seen from the above embodiments, during a video call, the user can also adjust the sound field and mixing parameters of the other end of the call.

[0055] In combination with the first aspect, in some embodiments, a first electronic device and a second electronic device enter into a first video call, and before the first electronic device displays a first video image of the first video, the first electronic device receives the first video from the second electronic device.

[0056] In a second aspect, the present application provides an audio processing method. In which, the first electronic device can collect video images and audio. The first electronic device can determine audio and video information and sound field environment information based on the video image and audio. The audio and video information includes but is not limited to: one or more of the audio tracks of the audio and video contained in the audio, the spatial position of the audio and video, and / or the display position of the audio and video on the video image. The sound field environment information may include but is not limited to: one or more of the reverberation time, room type, room size and / or room reflective material. The first electronic device can receive an audio editing operation and determine the audio editing parameters based on the audio editing operation. The audio editing operation includes performing one or more of the following editing operations on the audio: audio and video editing, sound field editing and / or mixing editing. The first electronic device can determine the audio file based on the audio editing parameters, audio and video information, and sound field environment information.

[0057] The above-mentioned sound and image editing may include adjusting the sound characteristics of the sound image, such as adjusting the volume of the sound image, adjusting the timbre of the sound image, adjusting the spatial position of the sound image, etc. The above-mentioned sound field editing may include, but is not limited to, adjusting the type of room in which the sound field is located, adjusting the size of the room in which the sound field is located, and / or adjusting the reflective material of the room in which the sound field is located. The above-mentioned mixing editing may include adjusting mixing parameters. The above-mentioned mixing parameters may include, but are not limited to, mixing templates, reverberation duration, equalizer parameters, the size of the room in which the sound field is located, and / or the reflective material of the room in which the sound field is located.

[0058] The audio file may contain the audio signal of the sound field and each sound image, as well as metadata. The metadata may include the spatial position of the sound image, sound field environment information, and other data used to correctly render the audio signal.

[0059] As can be seen from the above method, the first electronic device can perform sound object tracking and sound field analysis on the audio, thereby providing multi-track editing capabilities for the audio. In this way, the user can personalize the sound characteristics, sound field, and mixing parameters of the sound objects in the audio in real time while the electronic device is recording audio and video to obtain audio that meets personal needs. In addition, the user can also personalize the sound characteristics, sound field, and mixing parameters of the sound objects in the audio after the electronic device completes the audio and video recording. The above method can improve the user experience of recording audio and video.

[0060] In combination with the second aspect, in some embodiments, when the first electronic device receives an operation to end video recording, it can save the video image and audio collected from the start of video recording to the end of video recording as a first video. When receiving an operation to play the first video, the first electronic device can obtain the video image of the first video and the first audio file. The first audio file may contain audio signals of the sound field and each sound image, as well as metadata. The metadata may include data such as the spatial position of the sound image, sound field environment information, etc. for enabling the audio signal to be correctly rendered. When the first electronic device receives an audio editing operation, it can determine audio parameters based on the audio editing operation. The audio editing operation may include one or more of the following editing operations on the audio: sound image editing, sound field editing, and / or mixing editing. The first electronic device can edit the first audio file based on the audio editing parameters to obtain a second audio file. The first electronic device can store the second audio file as the audio file of the first video.

[0061] The above-mentioned editing of the first audio file may include: editing the audio signal of the sound field in the first audio file (to adjust the sound field), editing the audio signal of one or more sound images in the first audio file (to adjust the sound characteristics of the sound image), and editing the metadata in the first audio file (to ensure that the adjusted sound image and / or sound field audio signal can be correctly rendered and played after adjustment).

[0062] As can be seen from the above embodiments, the user can edit an existing video in the first electronic device to change the playback effect of the audio in the video.

[0063] In conjunction with the second aspect, in some embodiments, the first electronic device may determine the location of the acoustic object displayed in the video image based on the spatial location of the acoustic object and the video image. The first electronic device may use an image processing algorithm to identify and track one or more targets in the video image, and determine which target in the video image emits various sounds based on the target type and location in the video image, as well as the spatial location of the acoustic object, thereby determining which acoustic object the target in the video image corresponds to.

[0064] Determining the display position of sound objects within an image facilitates intuitive audio editing of one or more sound objects within a video image. The electronic device can then process the corresponding audio track based on the sound objects selected by the user to meet the user's audio editing needs. This simple and intuitive audio editing process enhances the user's experience in recording audio and video, as well as audio editing.

[0065] In conjunction with the second aspect, in some embodiments, the audio file determined by the first electronic device based on audio editing parameters, audio-visual information, and sound field environment information may include audio files in multiple formats. For example, audio files in stereo format, audio files in surround sound format, and audio files in 3D sound format. In this way, rendering and playing audio files in different formats can enable users to experience different sound effects.

[0066] In conjunction with the second aspect, in some embodiments, the first electronic device may perform one or more of the following encoding on the audio file: channel encoding, sound object encoding, HOA encoding, and metadata encoding. The first electronic device may then transmit the encoded audio file to another device. This encoding can reduce information redundancy in the audio file and improve audio file transmission efficiency.

[0067] In a third aspect, the present application provides an audio processing method. This method can be applied to a communication system including a first electronic device and a second electronic device. The first electronic device and the second electronic device connect to a first call. After capturing video images and audio, the first electronic device can edit the audio based on a received audio editing operation to obtain a first audio file. The first audio file can include audio signals of a sound field and individual sound images, as well as metadata. The metadata can include data such as the spatial position of the sound image and sound field environment information, which is used to correctly render the audio signal. The first electronic device can render the audio file to obtain first audio data. The first electronic device can send the first audio data to a second electronic device. The second electronic device can then play the edited audio from the first electronic device based on the first audio data. The first electronic device receives the audio editing operation from the second electronic device and determines audio editing parameters. The first electronic device can send an audio editing instruction to the second electronic device. The audio editing instruction can include the audio editing parameters. The second electronic device can edit the captured audio based on the audio editing instruction to obtain a second audio file. The second electronic device can render the second audio file to obtain second audio data. The second electronic device can send the second audio data to the first electronic device. The first electronic device can play the edited audio from the second electronic device based on the second audio data.

[0068] As can be seen from the above method, a user can edit their own audio during a video call, allowing the other party to hear the user-edited audio. Furthermore, a user can also edit the audio of the other party during a video call, allowing them to hear the user-edited audio of the other party being edited. In this way, the user, as the host, can control the audio of multiple parties connected to the video call, ensuring a more efficient video call. The above embodiment allows users to easily and intuitively edit call audio during a video call, enhancing the user's video call experience.

[0069] In conjunction with the third aspect, in some embodiments, the communication system further includes a third electronic device. The third electronic device also accesses the first video call. The second electronic device may also send the second audio data to the third electronic device. The third electronic device may play the audio edited by the second electronic device based on the second audio data.

[0070] From the above embodiments, it can be seen that the user of the first electronic device can edit the audio of the second electronic device so that the other end of the call can listen to the audio of the other end of the call edited by the user of the first electronic device.

[0071] In combination with the third aspect, in some embodiments, the first electronic device can receive a video image and an audio file from the second electronic device. The audio file may contain the audio signal of the sound field and each sound image of the call counterpart, as well as metadata. When an operation to edit the audio of the call counterpart is received, the first electronic device can edit the above audio file to obtain a new audio file. The first electronic device can use the new audio file to render and obtain audio data. Then, the first electronic device can use the audio data obtained by the above rendering to play the audio of the call counterpart. In this way, the user on the first electronic device side can listen to the audio of the call counterpart that has been edited by himself.

[0072] In conjunction with the third aspect, in some embodiments, the communication system may further include a server. The server may be used to enable the first electronic device and the second electronic device to access the first video call. Upon receiving an operation to edit the audio of the other party, the first electronic device may send an audio editing instruction to the server. The audio editing instruction may include audio editing parameters for performing audio editing. The server may also receive a video image and an audio file from the other party. The audio file may include the audio signal of the sound field and each sound and image of the other party, as well as metadata. The server may edit the audio file of the other party according to the audio editing parameters of the first electronic device to obtain a new audio file. The server may then send the new audio file to the first electronic device, which will render and play the new audio file. Alternatively, the server may render the new audio file to obtain audio data, and then hand the audio data over to the first electronic device for playback. In this way, the user of the first electronic device can listen to the audio of the other party that has been edited by them.

[0073] As can be seen from the above embodiments, the first electronic device can support the local user to edit the audio of the other party in the call. The device that specifically edits the audio file can be the first electronic device, or the other party in the call (such as the second electronic device), or a server.

[0074] In a fourth aspect, the present application provides an electronic device, which may include a camera, a microphone array, a memory, and a processor. The camera may be used to capture images. The microphone array may be used to capture audio. The memory may be used to store a computer program. The processor may be used to invoke the computer program, causing the electronic device to perform any possible implementation method described in the first or second aspects.

[0075] In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, enables the electronic device to execute any possible implementation method in the first aspect or the second aspect.

[0076] In a sixth aspect, the present application provides a computer program product, which may include computer instructions. When the computer instructions are run on an electronic device, the electronic device executes any possible implementation method in the first aspect or the second aspect.

[0077] In a seventh aspect, the present application provides a chip, which is applied to an electronic device, and the chip includes one or more processors, which are used to call computer instructions to enable the electronic device to execute any possible implementation method as in the first aspect or the second aspect.

[0078] It is understandable that the electronic device provided in the fourth aspect, the computer-readable storage medium provided in the fifth aspect, the computer program product provided in the sixth aspect, and the chip provided in the seventh aspect are all used to execute the methods provided in the embodiments of the present application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] FIG1 is a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application;

[0080] FIG2 is a software structure block diagram of an electronic device 100 provided in an embodiment of the present application;

[0081] FIG3 is a schematic structural diagram of another electronic device 100 provided in an embodiment of the present application;

[0082] 4A to 4C are architecture diagrams of some communication systems provided in embodiments of the present application;

[0083] 5A to 5M are schematic diagrams of some audio editing scenarios provided by embodiments of the present application;

[0084] 6A to 6G are schematic diagrams of other audio editing scenarios provided by embodiments of the present application;

[0085] 7A to 7E are schematic diagrams of other audio editing scenarios provided by embodiments of the present application;

[0086] 8A to 8C are schematic diagrams of other audio editing scenarios provided by embodiments of the present application;

[0087] 9A to 9C are schematic diagrams of other audio editing scenarios provided by embodiments of the present application;

[0088] 10A to 10D are schematic diagrams of other audio editing scenarios provided by embodiments of the present application;

[0089] 11A to 11H are schematic diagrams of other audio editing scenarios provided by embodiments of the present application;

[0090] 12A to 12C are schematic diagrams of other audio editing scenarios provided by embodiments of the present application;

[0091] 13A and 13B are schematic diagrams of other audio editing scenarios provided by embodiments of the present application;

[0092] FIG14 is a flowchart of an audio processing method provided in an embodiment of the present application;

[0093] FIG15 is a flowchart of another audio processing method provided in an embodiment of the present application;

[0094] FIG16 is a flowchart of another audio processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0096] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0097] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0098] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application (APP) or an operating system (OS) and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0099] The present application provides an audio processing method. An electronic device may capture video images and audio, and determine information such as sound object information and sound field environment information based on the video images and audio. The sound object information may include one or more of the sound tracks of each sound object separated from the audio and / or the spatial location of the sound object. The sound field environment information may include one or more of the reverberation duration, room type, room dimensions, and / or room reflective material. Based on the above information, the electronic device may provide controls on a user interface for editing the audio of the sound objects and / or controls for editing the sound field. A user may select one or more sound objects for audio editing. For example, a user may adjust the volume of a sound object, change the timbre of a sound object, move the spatial location of a sound object, and so on. The user may also choose to edit the sound field, such as replacing the sound field environment. The electronic device may determine editing parameters based on the user's editing operations on the sound object and / or sound field, and then process the captured audio using the editing parameters. The electronic device may output audio files in multiple formats (such as stereo, ambisonic, surround sound, etc.) so that the processed audio can be rendered on various types of audio playback devices. When the processed audio is rendered, the sound effect heard by the user can achieve the effect adjusted by the user.

[0100] As can be seen from the above method, electronic devices can perform sound object tracking and sound field analysis on audio, thereby providing multi-track audio editing capabilities. In this way, users can personalize the audio tracks and / or sound fields of sound objects in the audio in real time while the electronic device is recording audio or video, obtaining audio that meets their personal needs. Furthermore, users can also personalize the audio tracks and / or sound fields of sound objects in the audio after the electronic device completes the audio or video recording. The above method can improve the user experience of recording audio and video.

[0101] Terminology

[0102] To facilitate understanding, some concepts involved in this application are introduced below.

[0103] 1. Sound Channel

[0104] Channels can refer to the audio signals collected or played back at different spatial locations during recording or playback. The number of channels can refer to the number of audio input devices (such as microphones) during recording, or the number of audio output devices (such as speakers) during playback.

[0105] Based on the number of channels, there can be mono, dual channels, 5.1 channels, 7.1 channels, etc. The more channels there are, the stronger the stereo and presence of the audio during playback.

[0106] 2. Sound Object

[0107] A sound object can represent an object that emits sound, such as a person, animal, musical instrument, or vehicle. The sound of a sound object can be perceived as a whole or as a sound source, independent of the surrounding environment. A sound object can also be called a sound image or a sound source.

[0108] 3. Audio Track

[0109] An audio track refers to a location for placing audio elements. These audio elements may include sound objects. Each sound object may correspond to one audio track. It is understood that during the recording of an audio segment, multiple sound objects may be present in the environment emitting sound. An electronic device can capture the sounds of these multiple sound objects and generate an audio file. In other words, an audio file can contain multiple audio tracks.

[0110] 4. Sound field

[0111] The sound field can refer to the area in the medium where sound waves exist.

[0112] As sound propagates, it is reflected by obstacles such as building walls and ceilings. For each sound object, we perceive both the direct sound we hear and the reflected sound or reverberation sound. The characteristics of reflected and reverberant sounds can be affected by the shape, size, and material of the building. The combination of these sounds—the direct sound transmitted to the ear, the reflected sound, and the reverberant sound—gives us the auditory experience of being in a specific location.

[0113] Therefore, the sound field can correspond to sound field environment information (such as reverberation duration, room type, room size, room reflective material, etc.). Electronic devices can adjust the audio sound effects by changing the above sound field environment information. Sound field environment information can also be called sound field parameters.

[0114] 5. Metadata

[0115] Metadata can be used to describe audio signals so that they can be correctly rendered, processed, or distributed. Electronic devices can generate corresponding metadata when recording audio.

[0116] The metadata of an audio file can include the definitions of the different tracks within the file. Specifically, metadata can include information such as the spatial position and / or intensity of a sound object. Metadata in an audio file can also include information about the sound field environment. During playback, electronic devices can use this metadata to map the sound objects (i.e., tracks) to one or more speakers or perform binaural rendering to headphones, thereby achieving the desired spatial audio effect.

[0117] In some embodiments, the format of the metadata may conform to a standard, such as ITU-R BS.2076. Under ITU-R BS.2076, the metadata may include one or more of the following elements: audioTrackFormat, audioStreamFormat, audioChannelFormat, audioBlockFormat, audioPackFormat, audioObject, audioContent, audioPraogramme, audioTrackUID, and / or audioFormatExtended.

[0118] audioTrackFormat can represent data related to an audio track and can be used to describe the format of the data so that the renderer can correctly decode the signal. audioTrackFormat can contain the track ID, track name, format description information, etc.

[0119] The audioStreamFormat can be used to define the combination of audio tracks (i.e., audio streams) required to successfully decode audio track data. An audio stream can be a combination of several audio tracks (or a single audio track) used to render a channel, sound object, higher order ambisonics (HOA) component, or package. The audioStreamFormat can contain the audio stream ID, the name of the audio stream, a description of the format, and so on.

[0120] audioChannelFormat can represent channel-related data. audioChannelFormat can include the channel name (such as left channel, right channel, etc.), channel ID, channel type, etc.

[0121] An audioBlockFormat is a time-domain subdivision of an audioChannelFormat. An audioBlockFormat (or audio block) represents a single sequence of audioChannelFormat samples with fixed parameters within a specified time interval. An audioChannelFormat can be further subdivided into one or more audioBlockFormats in the time domain. An audioBlockFormat can contain the audio block ID, the start time of the audio block, and its duration.

[0122] audioPackFormat can represent an audio package that combines one or more audioChannelFormats. audioPackFormat can contain the audio package ID, audio package name, channel type, audio package importance, etc.

[0123] The above ITU-R BS.2076 is only an example standard of this application, and the format of metadata can also be defined according to other standards. This embodiment of the application does not limit this.

[0124] 6. Stereo

[0125] Stereo refers to an audio format that uses two channels to carry correlated audio signals, typically played back through two symmetrical speakers or headphones positioned in front of the listener, giving the listener the perception of a wider sound field.

[0126] 7. Surround Sound

[0127] Surround sound refers to an audio format that uses multiple channels to carry multi-channel audio signals that form the complete audio content, and is reproduced through multiple speakers located around the listener at ear level, giving the listener the feeling of being surrounded by a sound field.

[0128] 8. 3D sound

[0129] 3D sound can refer to an audio format and can include channel-based 3D sound, object-based 3D sound, and sound field-based 3D sound.

[0130] Channel-based 3D sound refers to playing different sounds in different channels to achieve a stereo effect. Channel-based 3D sound can supplement the sound information in the space by adding channels, giving the listener an immersive sound field feeling.

[0131] Object-based 3D sound can be described as breaking down sound into multiple independent audio objects, each with its own properties such as position, direction, and volume. These objects are then mixed and processed to create a stereo effect. Object-based 3D sound can be localized to any point in space by assigning spatial coordinates to sound objects. Object-based 3D sound can be rendered and played without restrictions on reproduction requirements (such as the number of speakers).

[0132] Sound field-based 3D sound can be achieved by placing multiple speakers in a space, emitting sound in different directions and angles to achieve a stereo effect. Sound field-based 3D sound can reproduce the sound in a three-dimensional space by recording the sound pressure in that space. Its core underlying algorithm may include HOA.

[0133] In some embodiments, the three-dimensional sound may further include three-dimensional sound implemented based on a channel+object+sound field technology.

[0134] It can be seen that three-dimensional sound can produce a three-dimensional sense of orientation on a speaker array or headphones, making the listener feel that the sound source they hear is in different 3D positions in space.

[0135] Three-dimensional sound can also be called spatial audio or 3D audio.

[0136] 9. Speaker Rendering

[0137] Speaker rendering may refer to rendering an audio signal through a set of speakers.

[0138] For example, in a scene where speakers are used to render three-dimensional sound, the audio signals of different sound sources are mapped to the speaker positions through the reasonable placement of the speaker array (for example, using the speaker array conversion matrix to perform mapping operations with the positions of the sound sources) to achieve the spatial and directional sense of the played sound signals.

[0139] If the audio playback device is a multi-speaker device, such as a car computer, a large screen, or multiple speakers in a home theater, speaker rendering can be used.

[0140] 10. Binaural Rendering

[0141] Binaural rendering can refer to rendering the audio signal through headphones or stereo external speakers (such as mobile phones, tablets, laptops, etc.).

[0142] For example, in a scenario where binaural rendering of three-dimensional sound is used, the head-related transfer function (HRTF) is used to perform time-domain convolution with the audio signal carrying azimuth information, distance attenuation compensation is compensated, and reverberation is added to simulate natural sound waves, making the sound source seem to come from a point in three-dimensional space, so that the user feels that the sound source they hear is in different 3D positions in space.

[0143] Equipment structure and communication system

[0144] The following introduces the software and hardware structure of the electronic device 100 involved in this application.

[0145] FIG1 exemplarily shows a hardware structure diagram of an electronic device 100 provided in an embodiment of the present application.

[0146] As shown in Figure 1, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0147] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0148] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0149] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0150] Processor 110 may also include a memory for storing instructions and data. In some examples, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces processor 110 latency, and thus improves system efficiency.

[0151] In the present application, a computer program is stored in the memory, which is used to enable the controller or processor to implement the audio processing method of the present application through an interface or protocol.

[0152] Exemplarily, the computer program stored in the memory can be used to process the collected video images and audio to determine the sound object information (including determining the position of the sound object displayed in the video image, the sound track of the sound object in the audio, and the spatial position of the sound object) and the sound field environment information. Among them, the sound track of the above-mentioned sound object can be obtained by using the sound object track separation technology. The spatial position of the above-mentioned sound object can be obtained by using a positioning algorithm to process the digital sound signal after performing analog-to-digital conversion on the analog sound signal in the collected environment to obtain a digital sound signal. The position of the above-mentioned sound object displayed in the video image can be obtained by using an image recognition algorithm. The above-mentioned sound field environment information can be obtained by jointly modeling the audio and video images.

[0153] Optionally, the computer program stored in the memory can also be used to: determine editing parameters based on the user's editing operations on the sound object and / or sound field, and use the editing parameters to process the collected audio; encode the audio (such as channel encoding, sound object encoding, HOA encoding, metadata encoding, etc.); output the processed audio information such as the audio track, the spatial position of the sound object, etc. as an audio file in one or more audio formats (such as stereo, surround sound, three-dimensional sound, etc.).

[0154] USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. USB interface 130 can be used to connect a charger to charge electronic device 100, transfer data between electronic device 100 and peripheral devices, or connect headphones to play audio.

[0155] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While charging the battery 142, the charging management module 140 can also power the electronic device through the power management module 141.

[0156] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0157] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0158] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0159] The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low-noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves for radiation via the antenna 1.

[0160] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0161] The electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering.

[0162] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

[0163] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0164] The ISP is used to process data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then transmitted to the ISP for processing and converted into an image visible to the naked eye.

[0165] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0166] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0167] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0168] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0169] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0170] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0171] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some examples, the audio module 170 can be set in the processor 110, or some functional modules of the audio module 170 can be set in the processor 110. The speaker 170A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. The headphone jack 170D is used to connect wired headphones.

[0172] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. In some embodiments, the electronic device 100 can be provided with multiple microphones 170C (such as two, three, etc.) to achieve functions such as audio collection, noise reduction, and spatial position determination of sound objects. Microphone 170C can be either an omnidirectional microphone or a directional microphone. Microphone 170C can be a microphone built into the electronic device 100. Alternatively, microphone 170C can also be a microphone independent of the electronic device 100 and having a communication connection with the electronic device 100, such as a wireless microphone, etc.

[0173] The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a gravity sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0174] The gyroscope sensor can be used to determine the motion posture of the electronic device 100. In some embodiments, the gyroscope sensor can be used to determine the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes). The electronic device 100 can also use the gyroscope sensor to determine the offset angle of the electronic device 100.

[0175] The accelerometer can detect the magnitude of acceleration in all directions (generally three axes) of the electronic device 100. In some embodiments, the accelerometer can be used to identify the posture of the electronic device 100 and can be applied to applications such as landscape and portrait screen switching and pedometers.

[0176] The gravity sensor can be used to determine the tilt angle of the electronic device 100 relative to the horizontal plane. In some embodiments, the gravity sensor can be used to determine the screen state of the electronic device 100, so as to adjust the screen to keep it horizontal.

[0177] In some embodiments, the electronic device 100 may determine the movement distance of the electronic device 100 within a period of time through an acceleration sensor and a gravity sensor.

[0178] Buttons 190 include a power button, a volume button, etc. Motor 191 can generate vibration prompts. Indicator 192 can be an indicator light that can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0179] The SIM card interface 195 is used to connect a SIM card. A SIM card can be connected to and disconnected from the electronic device 100 by inserting or removing it from the SIM card interface 195. The electronic device 100 may support one or N SIM card interfaces, where N is a positive integer greater than 1. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some examples, the electronic device 100 uses an eSIM, or embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0180] The electronic device 100 may be a mobile phone, a tablet computer, a smart watch, a television, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present application does not limit the specific type of the electronic device.

[0181] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the electronic device 100 is exemplarily described.

[0182] FIG2 is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application.

[0183] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime (Android Runtime) and system library, and kernel layer.

[0184] The application layer can include a series of application packages.

[0185] As shown in Figure 2, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, and short messaging. The camera application can be used to provide functions such as taking photos and recording videos. During video recording, the electronic device 100 can capture video images through the camera and audio through the microphone (or other audio input device). Furthermore, the electronic device 100 can display the captured video images on the screen in real time. The call application can be used to provide voice and video call functions. In a video call scenario, the electronic device 100 can capture video images through the camera and audio through the microphone. The electronic device 100 can send the captured video images and audio to the other party in the video call, and receive video images and audio from the other party in the video call. The electronic device 100 can display the video images captured by the electronic device 100 and the video images sent by the other party in the video call in real time on the screen, and play the audio from the other party in the video call. In some embodiments, the video call function described above can allow multiple people to access a call simultaneously, for example, a video call with three, four, or more people.

[0186] The electronic device 100 may further include more or fewer applications. For example, the electronic device 100 may further include a video conferencing application. The functions provided by the video conferencing application may refer to the above-mentioned video call function.

[0187] The application framework layer provides APIs and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions.

[0188] As shown in FIG2 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, an activity manager, and the like.

[0189] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0190] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0191] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0192] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0193] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0194] The Notification Manager allows applications to display notification information in the status bar (such as the pull-down notification bar). It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the Notification Manager is used to notify the completion of downloads, message reminders, etc. The Notification Manager can also be used to display notifications in the form of icons or scrolling text in the status bar at the top of the system, such as notifications from applications running in the background, or notifications that appear on the screen in the form of dialog windows. For example, text messages can be displayed in the status bar, prompts can be sounded, electronic devices can vibrate, indicator lights can flash, etc.

[0195] The Activity Manager is responsible for managing activities, starting, switching, and scheduling components in the system, as well as managing and scheduling applications. The Activity Manager can be called by upper-level applications to open corresponding activities.

[0196] Android Runtime includes core libraries and a virtual machine. Android Runtime is responsible for scheduling and management of the Android system.

[0197] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0198] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0199] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0200] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0201] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0202] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0203] A 2D graphics engine is a drawing engine for 2D drawings.

[0204] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0205] The embodiment of the present application does not limit the operating system of the electronic device 100. For example, the operating system of the electronic device 100 may also be apple OS, etc.

[0206] FIG3 is a schematic structural diagram of another electronic device 100 according to an embodiment of the present application.

[0207] As shown in FIG3 , the electronic device 100 may include a collection unit 310, an information extraction unit 320, an editing unit 330, a rendering unit 340, and a coding and decoding unit 350.

[0208] The acquisition unit 310 may include an audio acquisition module 311 and an image acquisition module 312 .

[0209] The audio acquisition module 311 can be used to collect sound signals in the environment (ie, collect audio). The audio acquisition module 311 can specifically be a sound pickup array, for example, a microphone array. In some embodiments, the audio collected by the audio acquisition module 311 can be spatial audio.

[0210] The image acquisition module 312 can be used to capture images. Specifically, the image acquisition module 312 can be a camera, such as the front camera or rear camera of the electronic device 100. The electronic device 100 can use the image acquisition module 312 to continuously capture images over a period of time and concatenate the captured images in the order in which they were captured to save them as a video. The images saved as a video can also be referred to as video images.

[0211] In some embodiments, the camera may include a depth camera so as to locate the person or object being photographed based on the image captured by the depth camera. The embodiment of the present application does not limit the type of the camera.

[0212] In some embodiments, the above-mentioned audio acquisition module 311 and / or image acquisition module 312 may also be modules included in other devices independent of the electronic device 100. For example, the electronic device 100 may establish a communication connection with a video image acquisition device and an audio acquisition device. The video image acquisition device may include an image acquisition module. The audio acquisition device may include an audio acquisition module. The video image acquisition device and the audio acquisition device may simultaneously acquire data and send the data to the electronic device 100. The electronic device 100 may receive the video image acquired by the video image acquisition device and the audio acquired by the audio acquisition device. Then, the electronic device 100 may provide the user with the function of sound object and sound field editing according to the audio processing method provided in this application, and process the above-mentioned received audio according to the user's audio editing operation.

[0213] The information extraction unit 320 may include an acoustic object determination module 321 , a spatial position determination module 322 , and a sound field information determination module 323 .

[0214] The information extraction unit 320 can obtain the audio and video images collected by the collection unit 310.

[0215] The acoustic object determination module 321 can be used to perform acoustic object tracking to separate the audio tracks of one or more acoustic objects from the audio. In one possible implementation, when the collected audio is multi-channel, the audio signal of an acoustic object can exist in one or more channels. The acoustic object determination module 321 can determine the spectral similarity of each channel in the audio and obtain a set of spectral similarities. Based on the set of spectral similarities, the acoustic object determination module 321 can perform channel grouping to obtain a set of channel groups. A channel group can be associated with an acoustic object. The above-mentioned channel grouping method may include but is not limited to clustering algorithms such as hierarchical method, density method, grid method, etc. Based on the above-mentioned set of channel groups, the acoustic object determination module 321 can generate a probability vector associated with each channel group for each frame contained in the audio. The probability vector of a frame can be used to indicate the probability value of this frame belonging to a channel group. The acoustic object determination module 321 can aggregate associated probability vectors across frames to generate a probability matrix corresponding to each channel group. Based on the probability matrix, acoustic object synthesis between the channel groups across frames is performed to separate the audio tracks of the individual acoustic objects. The specific method used by the acoustic object determination module to perform acoustic object track separation is not limited in this embodiment of the present application.

[0216] In some embodiments, the acoustic object determination module 321 may also be used to identify noise in the environment. For example, noise may include but is not limited to the sound of a television, a car horn, a dog barking, applause, and the like.

[0217] It should be noted that the audio collected by the audio collection module 311 may include not only the audio tracks of various sound objects but also a sound bed. The sound bed may represent the audio signal of the sound field and may be used to carry ambient sound.

[0218] The spatial position determination module 322 can be used to determine the spatial position of a sound object and the position at which the sound object is displayed in a video image. The spatial position of a sound object can be used to describe the position of the sound object in the audio relative to the listener or the audio acquisition device in three-dimensional space. The electronic device 100 can determine where the audio signal of the sound object should be played in three-dimensional space based on the spatial position of the sound object, so that the listener can experience the corresponding spatial sense when listening to the audio. In other words, the spatial position of the sound object can indicate the position at which the listener can distinguish the sound object when the audio is played. The electronic device 100 can change the spatial position of the sound object by editing the audio signal of the sound object. When the spatial position of the sound object changes, the listener can hear the sound of the sound object emanating from the changed spatial position when listening to the audio.

[0219] In some embodiments, in audio technologies based on spatial audio, the spatial position of a sound object can represent the location of the sound object (or sound source) in a virtual three-dimensional space. The sound object can be freely placed anywhere in the virtual three-dimensional space and can also move along a pre-defined trajectory. The spatial position of a sound object can also be referred to as the sound image position, etc.

[0220] The spatial position of a sound object can be represented by spatial coordinates. The spatial position determination module 322 can establish a spatial coordinate system and describe the relative positional relationship between the sound object and the listener using the spatial coordinates of the sound object and the spatial coordinates of the listener. The spatial coordinate system can be a three-dimensional coordinate system that can represent a virtual three-dimensional space. The embodiments of the present application do not limit the origin of the spatial coordinate system or the method for establishing it. The spatial coordinates of the listener can be, for example, the coordinates of the location of the electronic device 100 or the coordinates of the location of headphones connected to the electronic device 100. The spatial coordinates of the listener are not limited in the embodiments of the present application.

[0221] In one possible implementation, the spatial position determination module 322 may process the audio (e.g., performing analog-to-digital conversion, Fourier transform, etc.) to obtain spectral characteristics of each sound signal. The spatial position determination module 322 may determine the spatial position of the sound object based on the spectral characteristics. The relative positional relationship between the sound object and the listener may change due to sound, and the spatial position determination module 322 may determine the spatial position of the sound object in real time. The spatial position determination module 322 may also determine the spatial position of each sound object using other localization algorithms. This is not limited in the present embodiment.

[0222] The spatial position determination module 322 can determine the position of the sound object displayed in the video image based on the spatial position of the sound object and the video image. In one possible implementation, the spatial position determination module 322 can use an image processing algorithm to identify and track one or more targets in the video image. Based on the target type, target location, and spatial location of the sound object in the video image, the spatial position determination module 322 determines which target in the video image emits various sounds in the environment, thereby determining which sound object the target in the video image corresponds to. For example, the spatial position of sound object 1 indicates that sound object 1 is located in front of the left of the listener. The video image contains target 1, and target 1 is located on the left side of the video image. Furthermore, the sound of sound object 1 is a male voice. Target 1 is also male. Therefore, the spatial position determination module 322 can determine that target 1 in the video image is sound object 1, thereby associating target 1 with the sound track of sound object 1.

[0223] Determining the display position of a sound object within an image facilitates intuitive audio editing of one or more sound objects within a video image. The electronic device 100 can then process the corresponding audio track based on the sound object selected by the user to be edited within the video image to meet the user's audio editing needs. This simple and intuitive audio editing operation enhances the user's experience in recording audio and video, as well as audio editing.

[0224] Optionally, when determining the position of the acoustic object displayed in the video image, the spatial position determination module 322 may further correct the spatial position of the acoustic object based on the video image to obtain a more accurate spatial position of the acoustic object.

[0225] In some embodiments, the camera's shooting angle is limited, and the video image may only contain a portion of the sound object. For example, when only the rear camera is turned on for shooting, the electronic device 100 may only be able to capture sound objects within a certain range in front of the user, but not the sound objects behind the user. When it is determined that the sound object is not displayed in the video image, the electronic device 100 can display a control on the user interface for representing the above-mentioned sound object that is not captured. In this way, in addition to being able to edit the audio of the sound object that is captured, the user can also edit the audio of the sound object that is not captured.

[0226] The above method for determining the spatial position of a sound object and the position of the sound object displayed in the video image is merely an example of the present application and should not be construed as limiting the present application. The spatial position determination module 322 may also use other methods to determine the spatial position of a sound object and the position of the sound object displayed in the video image.

[0227] The sound field information determination module 323 can be used to determine sound field environment information. The sound field environment information may include one or more of the following: reverberation duration, room type, room dimensions, room reflective materials, etc. In one possible implementation, the sound field information determination module 323 can perform joint modeling with audio and video images to determine the sound field environment information for the audio and video recording scenario. This embodiment of the present application does not limit the method for determining the above sound field environment information.

[0228] The information extraction unit 320 may send the audio track, spatial position, sound field environment information and other contents of the sound object to the editing unit 330 .

[0229] The editing unit 330 may include a sound object editing module 331 , a sound field environment editing module 332 , and a sound mixing module 333 .

[0230] The sound object editing module 331 can be used to edit the sound object's audio track based on the user's editing operations on the sound object in the user interface. These sound object editing operations may include, but are not limited to, adjusting volume, timbre, and spatial position. These timbre adjustments can include adjusting the timbre of characters or instruments. This facilitates the user's ability to enhance the sound of a human voice or instrument, making audio editing more enjoyable for the user. The present embodiment of the application does not limit the method for editing the audio track to achieve the corresponding effects of these editing operations.

[0231] In some embodiments, the sound object editing module 331 may include modules for editing sound objects, such as a volume adjustment module 331A, a timbre adjustment module 331B, and a spatial position adjustment module 331C. The volume adjustment module 331A may be used to adjust the volume of a sound object. The timbre adjustment module 331B may be used to adjust the timbre of a sound object. The spatial position adjustment module 331C may be used to adjust the spatial position of a sound object.

[0232] The sound field environment editing module 332 can be used to adjust the sound field environment information based on the user's editing operations on the sound field in the user interface to achieve the effect of adjusting the sound field. When the sound field environment information is adjusted, the sound bed in the audio also changes accordingly. For example, if the room where the sound field is located is a theater, the user can change the room type from theater to recording studio. In this way, even if the user's actual environment is not a recording studio, the user can still hear the sound effects that would be heard in a recording studio, thus achieving an immersive experience of "traveling through time and space."

[0233] In some embodiments, the sound field environment editing module 332 may include modules for editing the sound field, such as a sound field intensity adjustment module 332A and a sound field replacement module 332B. Specifically, the sound field intensity adjustment module 332A may be used to adjust the sound intensity within the sound field. The sound field replacement module 332B may be used to adjust the sound field type (also known as the room type), for example, changing the sound field type from a theater to a recording studio.

[0234] The mixing module 333 can be used to mix sounds from multiple sources (i.e., the sound bed and multiple tracks in the audio) into a stereo track or a mono track. When mixing, the electronic device 100 can adjust and balance the relative levels (i.e., volume) of each track and the sound bed, and can perform various audio processing such as equalization and compression on individual tracks and track groups. In some embodiments, the electronic device 100 can provide professional mixing mode, intelligent mixing mode, and template mixing mode for users to perform mixing and editing operations.

[0235] In the professional mixing mode, the electronic device 100 can provide the ability to edit one or more parameters such as sound field environment information, equalizer (EQ), compressor, delay, etc. The mixing module 333 can perform mixing based on the user's mixing editing operation. In this way, the user can customize and adjust parameters such as sound field environment information, EQ, compressor, delay, etc. to configure a mixing effect that meets personal needs.

[0236] In the smart mixing mode, the mixing module 333 can use the mixing algorithm to comprehensively process the sound object's track, spatial position, and sound field environment information to generate matching sound field environment information, EQ parameters, compressor parameters, delay parameters, etc., to achieve the effect of beautifying the sound effect. In this way, users can achieve one-click smart mixing through the smart mixing control corresponding to the smart mixing mode.

[0237] In the template mixing mode, the electronic device 100 can provide a variety of mixing templates for the user to choose from. For example, the above-mentioned mixing templates may include but are not limited to: a NIC template and an ethereal template. Among them, the NIC template can focus on adjusting the sound object to make the sound effect of the sound object more beautiful, clear, and three-dimensional. The ethereal template can focus on adjusting the sound field to make the sound field wider. The mixing module 333 can perform corresponding mixing processing according to the mixing template selected by the user. The embodiment of the present application does not limit the specific content of the above-mentioned mixing templates.

[0238] Optionally, the audio mixing module 333 may also be configured to recommend one or more audio mixing templates based on the audio.

[0239] It should be noted that the sound mixing module 333 can receive the sound bed and audio tracks in the audio and perform mixing based on the received sound bed and audio tracks. The sound bed received by the sound mixing module 333 can be a sound bed that has been edited by the sound field environment editing module 332. The audio tracks received by the sound mixing module 333 can be audio tracks that have been edited by the sound object editing module 331.

[0240] In some embodiments, the audio mixing module 333 can output an audio file. The audio file can include an audio signal and metadata. The output audio signal can include the audio signal of each sound object and the audio signal of the sound field. The output metadata can be used to describe the audio signal (such as describing the audio track corresponding to the sound object, the spatial position of the sound object, the volume of the sound object, the timbre of the sound object, the reverberation duration, the room type, the room size, the room reflective material, etc.), so that the audio signal can be rendered with the corresponding spatial audio effect.

[0241] In some embodiments, the audio mixing module 333 can output audio files in multiple formats. For example, the format of the audio file may include but is not limited to stereo, surround sound, three-dimensional sound, etc. The audio files in the above-mentioned different formats can be used for playing in different audio playback devices. For example, an audio file in a stereo format can be used for playing in headphones or a sound array comprising two speakers. For another example, an audio file in a 5.1-channel format in surround sound can be used for playing in a sound array comprising 5 speakers (two front speakers on the left and right, two rear speakers on the left and right, and a center speaker) and a subwoofer. The embodiment of the present application does not limit the implementation method of the above-mentioned output of audio files in multiple formats.

[0242] Not limited to the mixing module 333 , the editing unit 330 may also include other modules to output the above-mentioned audio file.

[0243] The editing unit 330 may send the audio file to the rendering unit 340 .

[0244] The rendering unit 340 can identify the type of audio playback device and render audio data suitable for playback by the audio playback device based on the corresponding audio file. The audio playback device may include the electronic device 100 itself. For example, the electronic device 100 is a stereo external speaker device such as a mobile phone. The electronic device 100 can play audio through the speakers included in the electronic device 100. The audio playback device may also include a device external to the electronic device 100 (or referred to as a peripheral device). For example, the peripheral device may include headphones (wired headphones, wireless headphones), speakers, and the like. The electronic device 100 can send the rendered audio data to the peripheral device, thereby playing the audio through the peripheral device.

[0245] For example, when the audio playback device is identified as headphones or a stereo speaker, the rendering unit 340 can perform binaural rendering. The headphones or stereo speaker play audio data based on the binaural rendering, allowing the listener to experience the spatial and directional sense of the sound.

[0246] Optionally, when the audio playback device is a headset, the electronic device 100 may also provide a head tracking function, a reverberation model matching function, and a personalized playback function. The head tracking function may involve using the headset to detect the user's head movement and adjust the spatial position of the sound object based on the user's head movement, so that the user's perceived sense of the sound changes with the movement of the user's head. The personalized playback function may include personalized HRTF rendering. The electronic device 100 may provide a variety of HRTF templates for the user to choose from.

[0247] When the audio playback device is identified as a speaker array or a sound array, the rendering unit 340 can perform speaker rendering. For example, scenarios corresponding to the speaker array or sound array may include audio playback from a car computer, a large-screen device, or a home theater. In one possible implementation, the rendering unit 340 can implement speaker rendering using methods such as vector base amplitude panning (VBAP) technology.

[0248] The embodiments of the present application do not limit the specific implementation methods of the above-mentioned binaural rendering and speaker rendering.

[0249] In some embodiments, the electronic device 100 can perform real-time audio rendering in an audio or video recording scenario, so that an audio playback device can play the audio in real time. For example, the electronic device 100 is connected to a monitoring device or headphones. During the video shooting process, the electronic device 100 can send the rendered audio data to the monitoring device or headphones in real time. In this way, the user can monitor the effect of their audio editing in real time through the monitoring device or headphones.

[0250] The codec unit 350 may be used to encode an audio file so that the electronic device 100 can send the encoded audio file to other devices. The codec unit 350 may also be used to decode the encoded audio file received by the electronic device 100 so that the electronic device 100 can use the decoded audio file for audio playback.

[0251] The codec unit 350 encodes the audio file to reduce information redundancy and improve the efficiency of the electronic device 100 in transmitting the audio file.

[0252] The encoding of the audio file by the codec unit 350 may include one or more of the following: channel encoding, acoustic object encoding, HOA encoding, metadata encoding, etc. The decoding of the audio file by the codec unit 350 may include one or more of the following: channel decoding, acoustic object decoding, HOA decoding, metadata decoding, etc.

[0253] Among them, the codec unit 350 can use a general full-rate audio coding tool or a lossless audio coding tool to encode the channel signal (i.e., channel coding), encode the sound object signal (i.e., sound object coding), and encode the HOA signal (i.e., HOA coding). The above-mentioned channel information can represent a channel-based audio signal. The above-mentioned sound object audio signal can be an audio signal based on a sound object. The above-mentioned HOA signal can be an audio signal based on an HOA. The general full-rate audio coding tool can perform one or more of the following processing on the audio signal: transient detection, window type judgment, time-frequency transformation, frequency domain noise shaping, time domain noise shaping, frequency band extension, downmixing, neural network coding, quantization and interval coding, etc., to encode the channel signal, sound object signal, and HOA signal into a bit stream. The above-mentioned bit stream can be an encoded representation of the audio signal.

[0254] The codec unit 350 may encode the metadata using a metadata encoding tool and encode the metadata into a bitstream.

[0255] It is understood that decoding the audio file by the codec unit 350 can be the inverse process of the above encoding. Decoding can be implemented to decode the above bitstream into channel signals, acoustic object signals, HOA signals, and metadata. The electronic device 100 can then perform corresponding rendering (such as speaker rendering, binaural rendering, etc.) on the decoded data and play it.

[0256] FIG3 is merely an exemplary illustration of the structure of the electronic device 100 and should not be construed as limiting the present invention. It is understood that the electronic device 100 may further include more or fewer units or modules, or combine or separate certain modules.

[0257] The communication system involved in this application is introduced below.

[0258] FIG4A exemplarily shows an architecture diagram of a communication system 40 provided in the present application.

[0259] As shown in FIG4A , a communication system 40 may include an electronic device 100 and an audio playback device 200. A communication connection is established between the electronic device 100 and the audio playback device 200. For example, a wired communication connection, a wireless communication connection, etc. This embodiment of the present application does not limit the communication connection method between the electronic device 100 and the audio playback device 200.

[0260] The structure of the electronic device 100 may refer to the description of the embodiments shown in FIG. 1 to FIG. 3 .

[0261] The audio playback device 200 can be used to play audio. For example, the audio playback device 200 can be a headset (wireless headset or wired headset), a speaker array, etc. The embodiment of the present application does not limit the type of the audio playback device 200.

[0262] It is understandable that the electronic device 100 is connected to the audio playback device 200. The audio playback device 200 can be equivalent to an extension device (or peripheral device) of the electronic device 100, and can play the audio on the electronic device 100.

[0263] In some embodiments, in a video shooting scenario, the electronic device 100 can capture video images and audio, and edit the audio according to the user's audio editing operation to obtain an audio file containing an audio signal and metadata. The electronic device 100 can select an appropriate audio rendering method (such as binaural rendering, speaker rendering, etc.) according to the type of the audio playback device 200, and render the audio file to obtain audio data. The electronic device 100 can send the audio data to the audio playback device 200, and the audio playback device 200 can play the audio.

[0264] Among them, the audio file used by the electronic device 100 for rendering can be an audio file containing the audio signals of all sound objects in the environment and the audio signals of the sound field. In this way, the user can monitor the overall playback sound effect of the collected audio after editing in real time during the process of shooting the video. Optionally, the audio file used by the electronic device 100 for rendering can also be an audio file containing the audio signals of one or more sound objects. That is, the audio file contains the audio tracks of these one or more sound objects and metadata for describing the audio tracks of these one or more sound objects. For example, these one or more sound objects can be sound objects selected by the user for editing. In this way, the user can monitor the individual sound effects after the audio editing of one or more sound objects by the user in real time during the process of shooting the video.

[0265] In some embodiments, when playing a video, the electronic device 100 can obtain the audio file of the video and render the audio file to obtain audio data. The electronic device 100 can play the audio based on the audio data (i.e., the electronic device 100 plays the audio locally). Alternatively, the electronic device 100 can send the audio data to the audio playback device 200, which then plays the audio.

[0266] In addition, when playing a video, the electronic device 100 can also edit the audio according to the user's audio editing operation to obtain an audio file. In other words, in addition to real-time audio editing during video recording, the user can also perform secondary audio editing on the recorded video.

[0267] In some embodiments, in a video call (or video conference) scenario, the electronic device 100 can receive video images and audio from the other end of the call. The audio received by the electronic device 100 can be rendered audio data. The electronic device 100 can play audio on its own end based on the received audio data, or send the audio data to the audio playback device 200, which plays the audio. Alternatively, the audio received by the electronic device 100 can be an audio file containing an audio signal and metadata. The electronic device 100 can render the audio file to obtain audio data, and then play the audio on its own end or have the audio playback device 200 play the audio. In this way, the user can hear the voice of the other end during the video call.

[0268] In the communication system 40 shown in FIG4A , the audio playback device 200 mainly plays the role of audio playback. The electronic device 100 can send the rendered audio data to the audio playback device 200 .

[0269] FIG4B exemplarily shows an architecture diagram of a communication system 41 provided in this application.

[0270] As shown in FIG4B , a communication system 41 may include an electronic device 100 and an electronic device 300. A communication connection may be established between the electronic device 100 and the electronic device 300. For example, the electronic device 100 and the electronic device 300 may communicate via a local area network or a wide area network. The embodiments of the present application do not limit the communication connection method between the electronic device 100 and the electronic device 300.

[0271] The structures of the electronic device 100 and the electronic device 300 may refer to the description of the embodiments shown in FIG. 1 to FIG. 3 .

[0272] In some embodiments, the electronic device 100 can send a video image and an audio file to the electronic device 300. The audio file can include audio signals (such as audio signals of individual sound objects and audio signals of the sound field) and metadata. The electronic device 100 can encode the audio file using metadata, as well as one or more of the following encoding methods: channel encoding, sound object encoding, HOA encoding, etc. The electronic device 100 can then send the encoded audio file to the electronic device 300.

[0273] The above-mentioned audio encoding can reduce information redundancy and improve the transmission efficiency of audio files.

[0274] After receiving the video image and the encoded audio file, the electronic device 300 can decode the video image and the encoded audio file to obtain the audio file. Based on the received video image and audio file, the electronic device 300 can provide a sound object and sound field editing function. The electronic device 300 can process the audio file according to the user's audio editing operation to achieve the audio editing effect desired by the user.

[0275] That is to say, in addition to providing the user with the function of editing the audio object and sound field on the electronic device 100, the electronic device 100 can also send the audio file to other devices, such as the electronic device 300. In this way, the user can also edit the audio object and sound field on the electronic device 300.

[0276] FIG4C exemplarily shows an architecture diagram of the communication system 42 provided in the present application.

[0277] As shown in FIG4C , the communication system 42 may include multiple electronic devices (such as electronic device 100, electronic device 101, electronic device 102, etc.) and a server 400. The multiple electronic devices may establish communication connections with the server 400. The embodiment of the present application does not limit the manner of the communication connection.

[0278] The structures of the above-mentioned multiple electronic devices can all refer to the description of the embodiments shown in the aforementioned Figures 1 to 3.

[0279] At least two of the plurality of electronic devices can conduct a video call or video conference through the server 400. This application specifically takes the video call scenario as an example for introduction.

[0280] For example, electronic device 100, electronic device 101, and electronic device 102 can access the same video call. During the video call, electronic device 100 can send its own video image and audio to electronic devices 101 and 102 via server 400, and receive video image and audio from electronic device 101 and video image and audio from electronic device 102 via server 400.

[0281] Among them, the audio sent by the electronic device 100 can be audio data obtained by rendering an audio file. In this way, the electronic device 101 and the electronic device 102 can play the audio after receiving the audio data of the electronic device 100. The user on the electronic device 100 side can perform sound object and / or sound field editing on the audio collected by the electronic device 100. The user on the electronic device 101 side and the user on the electronic device 102 side can listen to the sound on the electronic device 100 side (for example, the sound after the user on the electronic device 100 side has performed sound object and / or sound field editing).

[0282] Alternatively, the audio sent by the electronic device 100 may be an audio file containing an audio signal and metadata. Furthermore, the electronic device 100 may encode the audio file (e.g., channel encoding, sound object encoding, HOA encoding, metadata encoding) before sending it to the server 400. The server 400 may directly send the encoded audio file to the electronic devices 101 and 102. Both the electronic devices 101 and 102 may decode the encoded audio file (e.g., channel decoding, sound object decoding, HOA decoding, metadata decoding) and render audio data from the decoded audio file. Since the audio files received by the electronic devices 101 and 102 are from the electronic device 100 side, the electronic devices 101 and 102 may provide sound object editing and sound field editing functions for the audio from the electronic device 100 side. In this way, the user on the electronic device 101 side and the user on the electronic device 102 side can change the sound they hear from the electronic device 100 side. Alternatively, the server 400 may decode the encoded audio file sent by the electronic device 100 and render it to obtain audio data. The server 400 may transmit audio data to the electronic device 101 and the electronic device 102 so that they can play the audio.

[0283] The audio received by the electronic device 100 (for example, the audio from the electronic device 101 and the audio from the electronic device 102) can be audio data, or can be an audio file containing an audio signal and metadata. The electronic device 100 can play the audio according to the audio data, or render the audio file to obtain the audio data and then play the audio. The electronic device 100 can play the audio locally or through a connected audio playback device (such as the audio playback device 200).

[0284] When the electronic device 100 receives the audio file, the electronic device 100 can edit the sound objects and / or sound fields in the audio file to generate a new audio file. The electronic device 100 can render the new audio file before playing the audio.

[0285] In some embodiments, the user of electronic device 100 can edit the sound objects and / or sound fields of one or more call peers, so that all users connected to the call can hear the sound objects and / or sound fields edited by electronic device 100 for the one or more call peers. For example, in host mode / master control mode, the user of electronic device 100 can be the host of a video call. The user of electronic device 100 can have the authority to control other users in the call and can adjust the sound objects and sound fields of the call peers.

[0286] In one possible implementation, the electronic device 100 receives an operation to edit the sound object and / or sound field on the electronic device 101 side. The electronic device 100 can generate editing parameters based on the above-mentioned editing operation and send an editing instruction to the server 400. The editing instruction may include the above-mentioned editing parameters. The server 400 can send the editing instruction to the electronic device 101. The electronic device 101 can edit the sound object and / or sound field according to the editing parameters in the editing instruction to generate an audio file. The electronic device 101 can then send the audio file or the audio data rendered by the audio file to the call partner (such as the electronic device 100 or the electronic device 102) through the server 400.

[0287] Optionally, when the server 400 receives the audio file of the electronic device 101, the server 400 can edit the audio file according to the editing instruction sent by the electronic device 100 to adjust the sound object and / or sound field therein. The server 400 can edit the audio file of the electronic device 101 into a new audio file, and send the new audio file or the audio data rendered by the new audio file to the electronic device 100 and the electronic device 102. It is understandable that the server 400 can encode the audio file (for example, channel decoding, sound object decoding, HOA decoding, metadata decoding) before sending it.

[0288] That is to say, the electronic device 100 can adjust the sound object and sound field of the call counterpart by sending editing instructions to instruct the server 400 or the adjusted call counterpart (such as electronic device 101, electronic device 102, etc.) to perform specific audio editing processing.

[0289] The following introduces the audio processing scenario provided by this application based on the above-mentioned electronic device and communication system.

[0290] The audio processing method provided in this application can be applied to video recording and video calling scenarios, allowing users to perform sound object editing, sound field editing, and audio mixing editing on the audio captured by the electronic device 100 in these scenarios. The audio processing method provided in this application is not limited to video recording and video calling scenarios and can also be applied to other scenarios involving the capture of video images and audio. This application specifically uses video recording and video calling scenarios as examples for illustration.

[0291] Audio editing in video recording scenarios

[0292] The video recording scenario may include the following stages: a video pre-recording stage, a video recording stage, and after the video recording is completed. Among them, the video pre-recording stage may refer to the stage after the electronic device 100 turns on the camera application but before starting to shoot the video. The video images captured by the electronic device 100 in the video pre-recording stage will not be saved as a video. The video recording stage may refer to the stage from the start to the end of the video shooting of the electronic device 100. After the video recording is completed, it may refer to the stage after the electronic device 100 finishes shooting the video. After the video recording is completed, the user can view the video shot by the electronic device 100 in the gallery application of the electronic device 100.

[0293] During the video pre-recording stage, video recording stage, and after video recording is completed, users can edit the audio. Audio editing can include but is not limited to: sound object editing, sound field editing, and mixing editing.

[0294] Video pre-recording stage

[0295] 5A to 5M are schematic diagrams of some audio editing scenarios.

[0296] As shown in Figure 5A, the electronic device 100 can display a user interface 510. The user interface 510 may include one or more application icons. For example, a camera application icon 511. The above application icons can be used to trigger the electronic device 100 to open the corresponding application. In response to the operation of the camera application icon 511 shown in Figure 5A, the electronic device 100 can open the camera application and display the user interface 520 shown in Figure 5B. When the camera application is opened, the electronic device 100 can turn on the camera and display the image captured by the camera on the screen. In addition, when the camera application is opened, the electronic device 100 can also turn on the microphone to collect audio. The embodiment of the present application does not limit the specific time when the above-mentioned camera and camera application are turned on.

[0297] As shown in FIG. 5B , the user interface 520 may include one or more of an audio editing control 521 , a preview area 522 , a camera mode option 523 , a gallery shortcut control 524 , a shutter control 525 , and / or a camera flip control 526 .

[0298] The audio editing control 521 can be used to turn on or off the audio editing function of the electronic device 100. The above-mentioned audio editing function may refer to a function that allows the user to perform sound object editing and sound field editing on the audio. When the audio editing function is turned off, the display style of the audio editing control 521 may be When the audio editing function is in an off state, the electronic device 100 may turn on the audio editing function in response to an operation on the audio editing control 521 .

[0299] Preview area 522 can be used to display a preview image. This preview image is an image captured in real time by the camera of electronic device 100. The electronic device can refresh the displayed content in preview area 522 in real time to facilitate the user to preview the image currently captured by the camera. As shown in preview area 522 in FIG. 5B , the content currently captured by electronic device 100 includes a boy, a girl, and a violin.

[0300] One or more shooting mode options may be displayed in the camera mode options 523. These one or more shooting mode options may include: a large aperture mode option 523A, a portrait mode option 523B, a video recording mode option 523C, a photo taking mode option 523D, and more options 523E. These one or more shooting mode options may be displayed as text information on the interface, such as "large aperture", "portrait", "video recording", "photo taking", and "more". Not limited to this, these one or more video recording options may also be displayed as icons or other forms of interactive elements (IEs) on the interface. When a user operation acting on a shooting mode option is detected, the electronic device 100 may turn on the shooting mode selected by the user and display the shooting mode option as selected. As shown in Figure 5B, the video recording mode option 523C is in a selected state. Not limited to what is shown in Figure 5B, the camera mode options 523 may also include more or fewer shooting mode options. The user can browse other shooting mode options by swiping left / right in the camera mode options 523.

[0301] The gallery shortcut control 524 can be used to open the gallery application. In response to a user operation on the gallery shortcut control 524, such as a touch operation, the electronic device 100 can open the gallery application. In this way, the user can conveniently view the photos and videos taken without having to exit the camera application first and then open the gallery application. The gallery application is an image management application on electronic devices such as smartphones and tablets. It can also be called an "album". This embodiment does not limit the name of this application. The gallery application can support users to perform various operations on pictures stored on the electronic device 100, such as browsing, editing, deleting, selecting, and the like.

[0302] The shutter control 525 can be used to monitor user operations that trigger taking photos or recording videos. For example, when the video mode option 523C shown in FIG5B is selected, in response to the operation of the shutter control 525, the electronic device 100 can start shooting a video and enter the video recording phase.

[0303] The camera flip control 526 can be used to monitor user operations that trigger camera flipping. The electronic device 100 can detect user operations acting on the camera flip control 526, such as touch operations, and in response to the operations, the electronic device 100 can flip the camera used for shooting, such as switching the rear camera to the front camera, or switching the front camera to the rear camera.

[0304] As shown in Figure 5B, when the audio editing control 521 is displayed in a closed state, in response to an operation on the audio editing control 521 (such as a click operation), the electronic device 100 can turn on the audio editing function and display the user interface 530 shown in Figure 5C.

[0305] As shown in Figure 5C, when the audio editing function is turned on, the display style of the audio editing control 521 can be "AI Audio". When the audio editing function is turned on, in response to the operation of the audio editing control 521, the electronic device 100 can turn off the audio editing function. The embodiment of the present application does not limit the display style of the audio editing control 521. Optionally, the electronic device 100 can turn on the audio editing function without the user manually. For example, the audio editing function can be always on.

[0306] User interface 530 may include a preview area 522. Electronic device 100 may mark captured objects in preview area 522 and provide corresponding volume bars and mute controls. For example, electronic device 100 may identify three objects, or three acoustic objects (referred to as acoustic images), on the image in preview area 522: acoustic image 1, acoustic image 2, and acoustic image 3.

[0307] The electronic device 100 can display a mark box 531, a volume bar 531A, and a mute control 531B associated with sound image 1, a mark box 532, a volume bar 532A, and a mute control 532B associated with sound 2, and a mark box 533, a volume bar 533A, and a mute control 533B associated with sound 3 on the user interface 530.

[0308] Marking box 531 can be used to circle audio image 1 to indicate to the user the object represented by audio image 1. For example, audio image 1 is a male figure currently being captured by electronic device 100. Volume bar 531A can be used to adjust the volume of audio image 1. Mute control 531B can be used to mute audio image 1 (i.e., adjust the volume of audio image 1 to 0).

[0309] Marking box 532 can be used to circle audio image 2 to indicate to the user the object represented by audio image 2. For example, audio image 2 is a female figure currently being photographed by electronic device 100. Volume bar 532A can be used to adjust the volume of audio image 2. Mute control 532B can be used to mute audio image 2 (i.e., adjust the volume of audio image 2 to 0).

[0310] Marking box 533 can be used to circle sound image 3 to indicate to the user what sound image 3 represents. For example, sound image 3 is the violin currently being captured by electronic device 100. Volume bar 533A can be used to adjust the volume of sound image 3. Mute control 533B can be used to mute sound image 3 (i.e., adjust the volume of sound image 3 to 0).

[0311] The volume bars and mute controls associated with each panner allow for intuitive and quick adjustment of the panner's volume. For example, users can quickly enhance vocals or mute a sound object.

[0312] The user interface 530 may also include an audio / video control 534 , a sound field control 535 , and a mixing control 536 .

[0313] The sound image control 534 can be used to select the sound image and perform sound object editing such as adjusting the timbre.

[0314] The sound field control 535 can be used to edit the sound field so that the user can adjust the sound field environment information.

[0315] Mixing controls 536 may be used to adjust parameters for mixing.

[0316] It should be noted that when the audio editing function is turned on, the electronic device 100 can perform sound object tracking on the collected audio to obtain sound tracks of different sound objects. The electronic device 100 can also determine the spatial position of each sound object, as well as the position displayed in the video image (i.e., the image displayed in the preview area 522). In this way, the electronic device 100 can determine the sound track of sound image 1, the sound track of sound image 2, and the sound track of sound image 3 shown in Figure 5C. When a user's audio editing operation on a sound image is detected, the electronic device 100 can edit the sound track of this sound image and the descriptive information about the sound track of this sound image in the metadata to adjust the volume, timbre, or spatial position and other parameters of this sound object.

[0317] The electronic device 100 can also determine the sound bed (i.e., the audio signal of the sound field) and sound field environment information. Upon detecting a user audio editing operation on the sound field, the electronic device 100 can edit the sound bed and the metadata describing the sound field (i.e., the sound field environment information) to adjust parameters such as reverberation duration, room type, room dimensions, or room reflective materials.

[0318] In response to the upward sliding operation on volume bar 531A shown in FIG5C , electronic device 100 may move the slider on volume bar 531A upward to increase the volume of sound image 1. Electronic device 100 may edit the audio track of sound image 1 and the descriptive information about the audio track of sound image 1 in the metadata to increase the volume of sound image 1. In this way, the volume of sound image 1 is increased in the overall playback sound effect.

[0319] As shown in Figure 5D, the position of the slider on volume bar 531A is higher than the position of the slider on volume bar 531A shown in Figure 5C. In response to the operation on the audio and video control 534 shown in Figure 5D, the electronic device 100 can display the audio and video 1 option 534A, audio and video 2 option 534B, and audio and video 3 option 534C shown in Figure 5E.

[0320] 5E , the audio / video 1 option 534A, the audio / video 2 option 534B, and the audio / video 3 option 534C can be used to select audio / video 1, audio / video 2, and audio / video 3, respectively, so as to edit the selected audio / video. In response to the operation of the audio / video 1 option 534A shown in FIG5E , the electronic device 100 can display the user interface 540 shown in FIG5F .

[0321] As shown in FIG. 5F , the user interface 540 may include an audio / visual control 534 and an audio / visual editing box 541 .

[0322] The text content “audio image 1” may be displayed on the audio image control 534 to indicate that the audio image 1 is currently selected, and the object of audio image editing based on the audio image editing box 541 is audio image 1.

[0323] The audio and video editing box 541 may include multiple timbre options so that the user can adjust the timbre of the audio and video to the timbre corresponding to the selected timbre option. For example, the timbre options may include: a girl timbre option, a boy timbre option 541A, an angel timbre option, an elf timbre option, and so on. In one possible implementation, the above timbre options may correspond to preset timbre adjustment parameters. The timbre adjustment parameters can be used to adjust the timbre of the audio and video to the timbre corresponding to the timbre option. For example, the electronic device 100 adjusts the timbre of the audio and video according to the timbre adjustment parameters corresponding to the boy timbre option 541A, so that the timbre of the audio and video can be heard as a boy's voice.

[0324] The audio / video editing box 541 may further include a confirmation control 541B and a cancel control 541C. The confirmation control 541B may be used to trigger the electronic device 100 to adjust the timbre of the audio / video 1 according to the timbre option selected in the audio / video editing box 541. The cancel control 541C may be used to cancel the editing of the audio / video 1.

[0325] Not limited to the timbre options, the sound image editing box 541 may also include more controls for editing other parameters of the sound image 1. For example, the sound image editing box 541 may include controls for adjusting the volume of the sound image 1, controls for adjusting the spatial position of the sound image 1, and so on.

[0326] In response to the operation on the juvenile voice option 541A shown in FIG5F , the electronic device 100 may display the juvenile voice option 541A as a selected state as shown in FIG5G (eg, the display color is deepened).

[0327] As shown in Figure 5G, when the juvenile voice option 541A is selected, in response to the operation of the confirmation control 541B, the electronic device 100 can adjust the timbre of the sound image 1 to the timbre corresponding to the juvenile voice option 541A, so that the timbre of the sound image 1 sounds like a teenager's voice.

[0328] As shown in FIG. 5H , in response to the operation of the sound field control 535 , the electronic device 100 may display the user interface 550 shown in FIG. 5I .

[0329] As shown in Figure 5I, the user interface 550 may include a sound field editing box 551. The sound field editing box 551 may include multiple room type options to facilitate users to replace the room type of the sound field. For example, the room type options may include a conference room option 551A, a church option, a recording studio option, a living room option, a bedroom option, a theater option, and so on. In one possible implementation, the above-mentioned room type options may correspond to preset sound field adjustment parameters. The sound field adjustment parameters may include one or more of the following: reverberation time, room size, room reflective material, and so on. The sound field adjustment parameters can be used to adjust the sound field to the sound field corresponding to the sound field option. For example, the electronic device 100 edits the sound field according to the sound field adjustment parameters corresponding to the conference room option 551A, so that the user can feel as if they are in a conference room listening to the audio collected by the electronic device 100.

[0330] The sound field editing box 551 may further include a confirmation control 551B and a cancel control 551C. The confirmation control 551B may be used to trigger the electronic device 100 to adjust the sound field according to the room type option selected in the sound field editing box 551. The cancel control 551C may be used to cancel the editing of the sound field.

[0331] Not limited to the room type option, the sound field editing box 551 may also include more controls for editing other parameters of the sound field. For example, the sound field editing box 551 may include controls for adjusting the reverberation time of the sound field, controls for adjusting the room size of the sound field, controls for adjusting the room reflection material of the sound field, and so on. It can be understood that when the room type of the sound field is selected, the electronic device 100 can obtain preset sound field adjustment parameters (such as reverberation time, room size, room reflection material). However, the user can still change the value of one or more parameters in the sound field adjustment parameters as needed. For example, the room size in the preset sound field adjustment parameters of the conference room is 30m 2 The user can adjust the room size of the conference room to 50m 2 Equal sizes.

[0332] In response to an operation on conference room option 551A shown in FIG5I , electronic device 100 may display conference room option 551A in a selected state as shown in FIG5I . When conference room option 551A is selected, in response to an operation on confirmation control 551B, electronic device 100 may replace the audio sound field with the sound field corresponding to conference room option 551A, so that the user feels as if they are in the conference room and listening to the audio.

[0333] As shown in FIG5J , in response to an operation on the audio mixing control 536 , the electronic device 100 may display the user interface 560 shown in FIG5K .

[0334] As shown in FIG5K , the user interface 560 may include a sound mixing edit box 561. The sound mixing edit box 561 may include a smart sound mixing control 561A, a NICam control 561B, a vacant control 561C, an OK control 561E, and a cancel control 561F.

[0335] Smart mixing control 561A can be used to trigger electronic device 100 to enter smart mixing mode. In smart mixing mode, electronic device 100 can use a mixing algorithm to intelligently balance the volume and reverberate the audio track, spatial position, and sound field environment information of the audio image to achieve the effect of beautifying the sound effects. As can be seen, users can achieve one-click smart mixing using smart mixing control 561A.

[0336] It should be noted that, in the intelligent mixing mode, the electronic device 100 can combine the user's individual editing of the sound image and the sound field when performing mixing.

[0337] The NICam control 561B may be used to trigger the electronic device 100 to perform mixing using a NICam template.

[0338] The ethereal control 561C can be used to trigger the electronic device 100 to use the ethereal template for mixing. The above-mentioned NICam template and ethereal template can refer to the introduction of the previous embodiment. The electronic device 100 can also include more or fewer mixing templates. This embodiment of the application is not limited to this.

[0339] The confirmation control 561E may be used to trigger the electronic device 100 to perform mixing according to the selected control and the set parameters in the mixing edit box 561. The cancel control 561F may be used to cancel the editing of the mixing.

[0340] In response to the left sliding operation in the audio mix edit box 561 shown in FIG5K , the electronic device 100 may switch the content displayed in the audio mix edit box 561. For example, the electronic device 100 may display the equalizer control 562 shown in FIG5L in the audio mix edit box 561. The equalizer control 562 may be used to adjust the EQ of the audio.

[0341] In response to a left-swiping operation in the mix edit box 561 shown in FIG5L , the electronic device 100 may further switch the content displayed in the mix edit box 561. For example, the electronic device 100 may display the reverberation duration control 563, room size control 564, and room reflection material control 565 shown in FIG5M in the mix edit box 561. The reverberation duration control 563 may be used to adjust the reverberation duration of the sound field. The room size control 564 may be used to adjust the room size of the sound field. The room reflection material control 565 may be used to adjust the room reflection material of the sound field. For example, room reflection materials may include, but are not limited to, walls, glass, carpet, and metal.

[0342] The contents displayed in the audio mix editing box 561 shown in Figures 5K to 5M are not limited thereto. The audio mix editing box 561 may also include more spaces for other parameters for audio mix editing, such as controls for adjusting audio compression parameters (i.e., compressors), controls for adjusting audio delay parameters (i.e., delays), and so on.

[0343] In some embodiments, the electronic device 100 can edit the collected audio according to the editing parameters determined by the sound object editing operation, the sound field editing operation, and the mixing editing operation, and output an audio file containing the audio signal and metadata after mixing.

[0344] It should be noted that the controls for editing sound objects, sound fields, and mixing are merely exemplary of this application and should not be construed as limiting this application. The electronic device 100 may also provide controls in other display styles, or allow the user to edit sound objects and sound fields using gestures or other methods.

[0345] As can be seen from the scenarios shown in Figures 5A to 5M above, users can perform audio editing before shooting a video, for example, muting unnecessary audio and video, enhancing the human voice of some important audio and video, adjusting the sound field, etc. In this way, when entering the video recording stage, the electronic device 100 can edit the audio collected during the video recording stage according to the editing parameters determined by the audio editing operation in the video pre-recording stage, so that the recorded audio meets the user's needs.

[0346] Video recording stage

[0347] 6A to 6G exemplarily show schematic diagrams of other audio editing scenarios.

[0348] As shown in FIG6A , the electronic device 100 may display a user interface 530. User interface 530 may be described with reference to the preceding embodiments. In response to an operation on the shutter control 525 shown in FIG6A , such as a click operation, the electronic device 100 may enter a video recording phase, begin capturing a video, and display the user interface 610 shown in FIG6B .

[0349] As shown in FIG. 6B , the user interface 610 may include time information 611 , an audio editing indicator 612 , an image display area 613 , a photo taking control 614 , an end control 615 , and a pause control 616 .

[0350] The time information 611 may be used to indicate the duration of the video being shot.

[0351] The audio editing indicator 612 may be used to indicate that the audio editing function is enabled.

[0352] The image display area 613 may be used to display images captured by the electronic device 100 during the video shooting process.

[0353] The photo control 614 may be used to save a frame of image captured when an operation on the photo control 614 is detected to a gallery.

[0354] End control 615 may be used to end capturing the video.

[0355] The pause control 616 may be used to pause capturing the video.

[0356] In some embodiments, when entering the video recording stage, the electronic device 100 may cancel the display of the aforementioned controls for audio editing (such as a mark box, a volume bar, a mute control, an audio and video control, a sound field control, a mixing control, etc.). When a preset operation acting on the image is detected (such as clicking or long pressing the audio and video on the image, etc.), the electronic device 100 may display the controls for audio editing again. Afterwards, the electronic device 100 may hide the controls for audio editing again when no operation on the controls for audio editing is detected within a preset time period. This can reduce the situation in which the controls for audio editing block the video screen during the video recording stage, interfering with the user's shooting.

[0357] Optionally, when entering the video recording stage, the electronic device 100 may also always keep the controls for audio editing displayed on the shooting screen.

[0358] In response to an operation on the sound image 3 in the image shown in FIG6B , such as a click operation, the electronic device 100 may display a user interface 620 shown in FIG6C .

[0359] As shown in FIG6C , user interface 620 may display controls for audio editing: marker boxes, volume bars, and mute controls associated with each of Pan 1, Pan 2, and Pan 3, as well as pan controls 621, sound field controls 622, and mixing controls 623. For details about these controls, please refer to the description of FIG5C .

[0360] When the text content “audio and video 3” shown in FIG. 6C is displayed on the audio and video control 621 , in response to an operation on the audio and video control 621 , the electronic device 100 may display the user interface 630 shown in FIG. 6D .

[0361] As shown in Figure 6D, the user interface 630 may include a sound image editing box 631. The sound image editing box 631 can refer to the sound image editing box 541 shown in Figure 5F above. Comparing Figure 5F and Figure 6D, it can be seen that the timbre options in the sound image editing box 541 can be used to adjust the timbre of sound image 1. The timbre options in the sound image editing box 631 can be used to adjust the timbre of sound image 3. Among them, the electronic device 100 can identify whether the type of sound image is a person or a musical instrument. When the type of sound image is identified as a person (for example, sound image 1), the electronic device 100 can provide timbre options related to human voice in the sound image editing box 541. When the type of sound image is identified as a musical instrument (for example, sound image 3), the electronic device 100 can provide timbre options related to musical instrument sound in the sound image editing box 631.

[0362] In other words, in addition to supporting timbre adjustment for human voices, electronic device 100 can also support timbre adjustment for musical instruments to achieve instrument conversion. This is not limited to human voices and musical instruments; electronic device 100 can also perform timbre adjustment for other types of sound images. This is not limited to this embodiment of the present application.

[0363] As shown in FIG6D , the sound image editing box 631 may include a piano sound option 631A, a flute sound option, a guitar sound option, and a confirmation control 631B. In response to an operation on the piano sound option 631A, the electronic device 100 may display the piano sound option 631A in a selected state as shown in FIG6D . When the piano sound option 631A is selected, in response to an operation on the confirmation control 631B, the electronic device 100 may adjust the sound image 3 to the sound image corresponding to the piano sound option 631A, so that the sound image 3 sounds like a piano.

[0364] In some embodiments, the electronic device 100 may further provide a function of merging multiple audio and video images, so that the user can edit multiple audio and video images simultaneously through a single audio editing operation.

[0365] 6E , user interface 620 may include a mark box 624 associated with audio image 1 and a mark box 625 associated with audio image 2. In response to a sliding operation from the area where mark box 624 is located to the area where mark box 625 is located as shown in FIG6E , electronic device 100 may display a prompt box 626 as shown in FIG6F .

[0366] As shown in FIG6F , a prompt box 626 may be used to ask the user whether to merge the audio and video. Prompt box 626 may include a text prompt "Confirm to merge audio and video 1 and audio and video 2," an OK control 626A, and a cancel control 626B. OK control 626A may be used to trigger electronic device 100 to merge audio and video 1 and audio and video 2. Cancel control 626B may be used to cancel the merging of audio and video 1 and audio and video 2.

[0367] In one possible implementation, merging multiple sound images by the electronic device 100 may mean associating the audio tracks of the multiple sound images with the merged sound image. Upon detecting an editing operation (e.g., adjusting the volume, timbre, spatial position, etc.) on the merged sound image, the electronic device 100 may edit the audio tracks of the multiple sound images and the descriptive information about the audio tracks of the multiple sound images in the metadata.

[0368] For example, electronic device 100 separates Track 1 and Track 2 from the collected audio. Track 1 is the track of audio image 1 shown in FIG6E . Track 2 is the track of audio image 2 shown in FIG6E . When an operation for merging audio image 1 and audio image 2 is detected, electronic device 100 may merge audio image 1 and audio image 2 into audio image 1′ and determine that the audio tracks of audio image 1′ are Track 1 and Track 2. In this way, the user can edit audio image 1 and audio image 2 simultaneously by editing audio image 1′.

[0369] In response to the operation of the confirmation control 626A shown in Figure 6F, the electronic device 100 can display the user interface 620 shown in Figure 6G.

[0370] As shown in Figure 6G, the user interface 620 may include a marking box 627, a volume bar 627A, and a mute control 627B associated with the audio and video 1′. The marking box 627 can circle the audio and video 1 and the audio and video 2 before the audio and video are merged on the image, so as to prompt the user of the object represented by the audio and video 1′. The volume bar 627A can be used to adjust the volume of the audio and video 1′. The mute control 627B can be used to mute the audio and video 1′. For example, the audio and video 1′ is a boy and a girl currently captured by the electronic device. In response to the operation of the mute control 627B, the electronic device 100 can edit the audio track of the audio and video 1′ and the descriptive information about the audio track of the audio and video 1′ in the metadata, so that the volume of the audio and video 1′ (that is, a boy and a girl on the image shown in Figure 6G) in the audio collected by the electronic device 100 is 0. That is, the sound of the audio and video 1′ cannot be heard in the video captured by the electronic device 100.

[0371] The above user operation for merging audio and video is only an exemplary description of the present application and should not be construed as limiting the present application. The user can also merge audio and video through other operation methods.

[0372] It should be noted that the electronic device 100 can determine the editing parameters according to the received audio editing operation and edit the audio using the editing parameters. The effective time (or valid time) of the editing parameters can be determined by the time the audio editing operation is performed.

[0373] For example, at time t1 during the video recording phase, the electronic device 100 receives an operation to mute the audio and video 1′. Based on this operation, the electronic device 100 can determine editing parameter 1: the volume of the audio and video 1′ is 0. Then, the electronic device 100 can perform audio editing based on editing parameter 1, so that the audio and video 1′ in the video shot by the electronic device 100 remains silent from time t1. At time t2 during the video recording phase, the electronic device 100 receives an operation to increase the volume of the audio and video 1′ to volume 1. The above-mentioned time t2 is later than time t1. Based on this operation, the electronic device 100 determines editing parameter 2: the volume of the audio and video 1′ is volume 1. Then, the electronic device 100 can perform audio editing based on editing parameter 2, so that the volume of the audio and video 1′ in the video shot by the electronic device 100 remains silent from time t1 to time t2.

[0374] The durations of other editing parameters can be referenced to the durations of the volume-related editing parameters mentioned above. For example, you can adjust the timbre of a sound image so that it is timbre 1 in one time period and timbre 2 in another. For another example, you can adjust the sound field so that it is timbre 1 in one time period and timbre 2 in another.

[0375] In some embodiments, the camera has a limited viewing angle, and the video image may only contain a portion of the audio and video. The electronic device 100 may provide a control for indicating the audio and video that are not captured, so that the user can perform audio editing on the audio and video that are not captured.

[0376] 7A to 7E exemplarily show schematic diagrams of other audio editing scenarios.

[0377] As shown in Figure 7A, the electronic device 100 can display a user interface 710. The user interface 710 can be a video shooting interface of the electronic device 100 during the video recording stage. The user interface 710 can display an image captured by the electronic device 100. The electronic device 100 can recognize the sound and image on the image and provide controls for audio editing. For example, the controls for audio editing may include: mark boxes associated with each sound and image, a volume bar, a mute control, as well as an audio and video control 712, a sound field control 713, and a mixing control 714. The above-mentioned controls for users to perform audio editing can refer to the introduction of the aforementioned embodiments.

[0378] The user interface 710 may also include an off-screen sound and image display area 711. The off-screen sound and image display area 711 may include controls for representing sound and images that are not captured. For example, after the electronic device 100 performs sound object tracking on the collected audio, in addition to determining the audio tracks of the captured sound and images 1, 2, and 3 as shown in Figure 7A, it also separates the audio tracks of sound and images 4, 5, and 6. Among them, sound and images 4, 5, and 6 are all sound and images that are not captured. The electronic device 100 can display a sound and image 4 identifier 711A, a sound and image 5 identifier 711B, and a sound and image 6 identifier 711C in the off-screen sound and image display area 711.

[0379] In some embodiments, the electronic device 100 can identify the type of sound and image and determine the display style of the above-mentioned sound and image that was not captured based on the type of sound and image. For example, the above-mentioned sound image 4 and sound image 5 are people. The display style of sound image 4 identifier 711A and sound image 5 identifier 711B can be a person icon, indicating that the sound of sound image 4 and sound image 5 is a human voice. The above-mentioned sound image 6 is a cat. The display style of sound image 6 identifier 711C can be a cat icon, indicating that the sound of sound image 6 is a cat's meow. The embodiment of the present application does not limit the display style of the above-mentioned sound and image that was not captured.

[0380] In some embodiments, the electronic device 100 can determine the spatial position of the sound image that is not captured, and determine the display position of the control for representing the sound image that is not captured on the user interface 710 based on the spatial position. For example, when the spatial position of the sound image 4 indicates that the sound image 4 is located to the right of the sound image 1, the electronic device 100 can display the above-mentioned sound image 4 logo 711A to the right of the sound image 1 on the image. In this way, the user can determine which sound image it corresponds to based on the display position of the sound image logo, and then edit the sound image that is not captured as needed. The embodiment of the present application does not limit the display position of the above-mentioned sound image that is not captured.

[0381] In response to an operation on the audio image 4 identifier 711A shown in FIG7A , such as a click operation, the electronic device 100 may display the volume bar 715 shown in FIG7B . The volume bar 715 may be used to adjust the volume of the audio image 4. Upon detecting an operation to adjust the volume on the volume bar 715, the electronic device 100 may edit the audio track of the audio image 4 and the description of the audio track of the audio image 4 in the metadata to adjust the volume of the audio image 4.

[0382] 7B , in response to the operation of the audio and video control 712 , the electronic device 100 may display all audio and video options so that the user can select the audio and video to be edited. The above-mentioned all audio and video may include audio and video captured by the electronic device 100 and audio and video not captured.

[0383] As shown in FIG7C , all audio and video options displayed by electronic device 100 may include audio and video 1 option 716A, audio and video 2 option 716B, audio and video 3 option 716C, audio and video 4 option 716D, audio and video 5 option 716E, and audio and video 6 option 716F. In response to the operation of audio and video 4 option 716D shown in FIG7C , electronic device 100 may display user interface 720 shown in FIG7D .

[0384] As shown in FIG. 7D , the user interface 720 may include an audio / visual control 712 and an audio / visual editing box 721 .

[0385] The text content “audio image 4” may be displayed on the audio image control 712 to indicate that the audio image 4 is currently selected, and the object of the audio image editing based on the audio image editing box 721 is the audio image 4.

[0386] The audio and video editing box 721 can refer to the introduction of the audio and video editing box 541 shown in Figure 5F above, which will not be repeated here.

[0387] As shown in Figure 7E, the electronic device 100 can rotate to change the shooting angle. For example, when the shooting angle changes, the sound and images captured by the electronic device 100 change from sound images 1, 2, and 3 shown in Figure 7A to sound images 2, 4, and 5 shown in Figure 7E. When the shooting angle changes, the electronic device 100 can display a user interface 730. User interface 730 may include a mark box 731, a volume bar 731A, and a mute control 731B associated with sound image 4, and a mark box 732, a volume bar 732A, and a mute control 732B associated with sound image 5. This allows the user to intuitively adjust the volume of sound images 4 and 5.

[0388] User interface 730 may also include an off-screen audio and video display area 711. Due to the change in the shooting angle, audio images 1 and 3 are no longer captured. Therefore, electronic device 100 may display audio image 1 marker 711D, audio image 3 marker 711E, and audio image 6 marker 711C, which represents audio image 6, which is still not captured, in off-screen audio and video display area 711, as shown in FIG. The user can edit the audio of audio images 1 and 3 using audio image 1 marker 711D and audio image 3 marker 711E, respectively.

[0389] In some embodiments, the electronic device 100 is connected to an audio playback device such as a monitoring device or headphones. The monitoring device can be used to play audio. The electronic device 100 can mix and output audio files in real time, and render the audio files in real time, so that the user can monitor the sound effects of the video recording in real time through the audio playback device. This allows the user to promptly understand the sound effects of the audio editing, allowing the user to edit the audio to achieve the desired sound effect.

[0390] The electronic device 100 can render the audio files corresponding to the sound bed and the audio tracks of all sound objects, and pass the rendered audio data to an audio playback device for playback. This allows the user to listen to all the audio during the video recording process. Alternatively, the electronic device 100 can render only the audio files corresponding to the audio tracks of a single or multiple sound images, and pass the rendered audio data to an audio playback device for playback. This allows the user to listen to the audio of a single or multiple sound images. For example, the user can listen to only the audio of an edited sound image to determine whether the edits to that sound image are appropriate.

[0391] 8A to 8C exemplarily show schematic diagrams of other audio editing scenarios.

[0392] Here, the connection between the electronic device 100 and the monitoring device 801 shown in FIG. 8A is taken as an example for description.

[0393] As shown in Figure 8A, the electronic device 100 can display a user interface 810. The user interface 810 can refer to the introduction of the user interface 620 shown in Figure 6C above. The user interface 810 may include an audio and video control 811, a sound field control 812, and a mixing control 813. The electronic device 100 can output the collected audio as an audio file containing an audio signal and metadata after mixing, and render the audio file to obtain audio data. The electronic device 100 can send the audio data to the monitoring device 801. The monitoring device 801 can play all the audio during the video recording process based on the received audio data. In this way, the user can understand the overall playback sound effect of the video.

[0394] In a case where the text content displayed on the audio-visual control 811 is “audio-visual 1”, in response to the operation of the audio-visual control 811 shown in FIG8A , the electronic device 100 may display the user interface 820 shown in FIG8B .

[0395] As shown in FIG8B , user interface 820 may include an audio / video editing box 821. Audio / video editing box 821 may refer to audio / video editing box 541 shown in FIG5F . Audio / video editing box 821 may include an angel voice option 821A and a confirmation control 821B. In response to an operation on angel voice option 821A shown in FIG8B , electronic device 100 may display angel voice option 821A in a selected state (e.g., with a darker display color) as shown in FIG8C .

[0396] As shown in FIG8C , in response to the operation of the confirmation control 821B, the electronic device 100 may adjust the timbre of the sound image 1 to the timbre corresponding to the angel timbre option 821A, so that the timbre of the sound image 1 sounds like an angel.

[0397] In one possible implementation, upon detecting an operation to adjust the timbre of sound image 1, electronic device 100 may edit the audio track of sound image 1 and the descriptive information about the audio track of sound image 1 in the metadata. Electronic device 100 may generate an audio file of sound image 1 based on the audio track of sound image 1 and the metadata related to sound image 1. The metadata related to sound image 1 may include descriptive information about the audio track of sound image 1, such as the volume, timbre, spatial position, and other information that enables the correct rendering of the audio track of sound image 1. Electronic device 100 may render the audio file of sound image 1 to obtain audio data of sound image 1, and then transmit the audio data of sound image 1 to monitoring device 801 for playback. Monitoring device 801 may only play the audio of sound image 1. In this way, after adjusting the timbre of sound image 1, the user can monitor the sound effects of sound image 1 in real time through monitoring device 801 to determine whether the adjustment to the timbre of sound image 1 is appropriate.

[0398] Optionally, within a preset time period starting from the moment of performing the timbre adjustment on the audio image 2, if the electronic device 100 does not detect an audio editing operation again, the electronic device 100 may continue to instruct the monitoring device 801 to play all audio during the video recording process.

[0399] As will be appreciated, while monitoring device 801 is playing the audio of audio / video 1, if it detects an attempt to edit another audio / video, such as audio / video 2, electronic device 100 can edit the audio track of audio / video 2 and the metadata describing audio / video 2. Electronic device 100 can generate an audio file for audio / video 2 based on audio / video 2 and its associated metadata. Electronic device 100 can render the audio file for audio / video 2 to obtain audio data for audio / video 2, and then pass this audio data to monitoring device 801 for playback. The content played by monitoring device 801 can then switch from the audio of audio / video 1 to the audio of audio / video 2.

[0400] As can be seen from the above embodiment, after the electronic device 100 receives the audio editing operation, it can play the part of the audio that has changed due to the audio editing operation through the monitoring device 801. In this way, the user can monitor the sound effects of the part they edited in real time during the video shooting process to determine whether the sound effects of the separately edited part of the audio are appropriate. In addition, after the user stops audio editing, the electronic device 100 can play all the audio during the video recording process through the monitoring device 801, and the user can determine whether the overall playback sound effects of the video are appropriate. The above embodiment can improve the convenience of users performing audio editing when recording videos, and help users confirm whether the content of the audio editing is appropriate more quickly.

[0401] In some embodiments, the electronic device 100 may also, in response to a preset operation for one or more audio images, instruct the monitoring device 801 to play the audio of the one or more audio images. In other words, the user can actively choose to listen to the audio of a single or multiple audio images. The present embodiment of the application does not limit the method of the above-mentioned preset operation.

[0402] In some embodiments, the electronic device 100 may provide a function for adjusting the spatial position of the sound image. The user may change the spatial position of the sound image so that the sound of the sound image appears to be emitted from the changed spatial position.

[0403] 9A to 9C exemplarily show schematic diagrams of other audio editing scenarios.

[0404] As shown in Figure 9A, the electronic device 100 can display a user interface 910. The user interface 910 can refer to the user interface 620 shown in Figure 6C above. The user interface 910 may include a marking box 911 associated with the sound image 1. In response to the operation shown in Figure 9A acting on the area where the marking box 911 is located, such as a long press operation, the electronic device 100 can display the sound image 1 control 912 shown in Figure 9B. In response to the operation of dragging the sound image 1 control 912 shown in Figure 9B, the electronic device 100 can display the prompt box 921 shown in Figure 9C. The prompt box 921 can be used for the user to adjust the spatial position of the sound image 1.

[0405] As shown in Figure 9C, prompt box 921 may include a horizontal angle control 922, a pitch angle control 923, an OK control 924, and a Cancel control 925. Horizontal angle control 922 can be used to adjust the horizontal angle of sound image 1 relative to the listener. The horizontal angle can be used to reflect the horizontal position of sound image 1 relative to the listener, such as to the left, front left, front right, right, or back of the listener. Pitch angle control 923 can be used to adjust the pitch angle of sound image 1 relative to the listener. The pitch angle can be used to reflect the vertical position of sound image 1 relative to the listener, such as to above, below, or above the listener. The listener's position can be the location of electronic device 100, or the location of monitoring device 801 shown in Figure 8A. The present embodiment does not limit the listener's position.

[0406] The confirmation control 924 can be used to trigger the electronic device 100 to adjust the spatial position of the sound image 1 according to the values ​​of the horizontal angle and the pitch angle in the prompt box 921. The cancel control 925 can be used to cancel the adjustment of the spatial position of the sound image 1.

[0407] It is understood that the operations for adjusting the spatial position of the sound image shown in Figures 9A to 9C are merely exemplary descriptions of the present application and should not constitute a limitation on the present application. The electronic device 100 may also support the user to adjust the spatial position of the sound image through other operations.

[0408] In some embodiments, in addition to adjusting the horizontal angle and the pitch angle of the sound image relative to the listener, the electronic device 100 may also support the user to adjust the distance of the sound image relative to the listener and other azimuth information.

[0409] In some embodiments, the electronic device 100 can also support users swapping the audio tracks of two audio images. For example, after the electronic device 100 has separated the audio objects into tracks and determined the spatial locations of the audio images and their display positions on the video image, it can determine that object 1 on the image is audio image 1, with audio track 1, and that object 2 on the image is audio image 2, with audio track 2. Upon detecting an operation to swap audio image 1 and audio image 2, the electronic device 100 can determine that object 1 on the image is audio image 2, with audio track 2, and that object 2 on the image is audio image 1, with audio track 1. Following this swap, if the electronic device 100 detects an audio editing operation for object 1 on the image, it can edit audio track 2 and the metadata description of audio track 2. Similarly, if it detects an audio editing operation for object 2 on the image, the electronic device 100 can edit audio track 1 and the metadata description of audio track 1. For example, if the user determines that the audio tracks of audio image 1 and audio image 2 are opposite, the user can swap the audio tracks of audio image 1 and audio image 2. This can improve the accuracy of panning audio tracks.

[0410] During the video recording phase, the user can refer to the above description of the video pre-recording phase for sound field editing and mixing editing, and will not be described in detail here.

[0411] The various embodiments introduced in the video recording stage can also be applied to all scenarios in this application, such as the scenario in the video pre-recording stage, as well as the scenario after the video recording is completed and the video call scenario to be described later.

[0412] After the video recording is completed

[0413] In some embodiments, a user can edit the audio of a video already stored in the electronic device 100. The electronic device 100 can store the video image and an audio file containing the audio signal and metadata. Thus, the electronic device 100 can edit the audio file to achieve the effect of editing the sound object, sound field, and sound mix.

[0414] 10A to 10D exemplarily show schematic diagrams of other audio editing scenarios.

[0415] As shown in FIG10A , electronic device 100 may display user interface 510. User interface 510 may include a gallery application icon 512. Gallery application icon 512 may be used to open the gallery application. In response to an operation on gallery application icon 512 shown in FIG10A , electronic device 100 may display user interface 1010 shown in FIG10B .

[0416] As shown in FIG10B , the user interface 1010 may include a video control 1011 for Video 1. The aforementioned Video 1 may be, for example, the video recorded during the aforementioned video recording phase. The present embodiment of the application does not limit the content of Video 1. The electronic device 100 may store the video image and audio file of Video 1 so that Video 1 can be correctly rendered and played. The audio file may include the sound field and audio signals of each sound object as well as metadata.

[0417] In response to an operation on the video control 1011, such as a click operation, the electronic device 100 may display the user interface 1020 shown in FIG10C. The user interface 1020 may include a play control 1021 and an edit control 1022. The play control 1021 may be used to trigger the electronic device 100 to play the video 1. The edit control 1022 may be used to trigger the electronic device 100 to display a user interface for editing the video 1, so that the user can edit the video 1 (such as audio editing). In response to the operation on the edit control 1022 shown in FIG10C, the electronic device 100 may display the user interface 1030 shown in FIG10D.

[0418] 10D , the user interface 1030 may display the video image included in the video 1. The user interface 1030 may also include an audio editing control 1031 , a progress bar 1032 , an audio / video control 1033 , a sound field control 1034 , and a mixing control 1035 .

[0419] The audio editing control 1031 can be used to trigger the electronic device 100 to turn on or off the audio editing function. When the audio editing function is in the off state, in response to the operation of the audio editing control 1031, the electronic device 100 can display the audio editing control 1031 as the selected state shown in Figure 10D.

[0420] The progress bar 1032 can be used to adjust the playback progress of the video.

[0421] The audio-visual control 1033, the sound field control 1034, and the mixing control 1035 can be used for audio-visual editing, sound field editing, and mixing editing, respectively. For details, please refer to the introduction of the above embodiment.

[0422] When the audio editing function is enabled, the electronic device 100 may display the aforementioned audio and video controls 1033, sound field controls 1034, and audio mixing controls 1035 on the user interface 1030, as well as controls for audio editing such as a mark box associated with each audio and video on the video image, a volume bar, and a mute control. When the audio editing function is disabled, the electronic device 100 may remove the aforementioned controls for audio editing from the user interface 1030.

[0423] It should be noted that after the video recording is completed, the editing that the user can perform on the audio in the existing video can refer to the audio editing performed in the aforementioned video pre-recording stage and the video recording stage. For example, after the video recording is completed, the user can edit the audio in the existing video, including but not limited to: adjusting the volume of the sound and image, adjusting the timbre of the sound and image, adjusting the spatial position of the sound and image, swapping the audio tracks of two sound and images, adjusting the sound field, mixing and editing, etc. Optionally, when the existing video is played to a segment where there is some sound and image that is not captured, the electronic device 100 can also provide a control in the user interface 1030 for indicating the sound and image that is not captured, so as to facilitate editing of the sound and image that is not captured.

[0424] As can be seen from the above embodiments, users can perform audio editing operations before and during video shooting, adjusting the sound field and individual sound and image to make the video playback sound effects meet personalized needs. In addition, users can also perform secondary audio editing on the recorded video after the video is recorded.

[0425] Controls for audio editing can be displayed on the video image. In particular, controls for editing audio and video can be displayed near the captured audio and video. This makes audio editing easier and more intuitive for users, improving the user experience when recording audio and video, as well as editing audio.

[0426] Audio editing in video call scenarios

[0427] 11A to 11H exemplarily show schematic diagrams of other audio editing scenarios.

[0428] As shown in FIG11A , electronic device 100 may display user interface 510. User interface 510 may include a video call application icon 513. Video call application icon 513 may be used to trigger electronic device 100 to open a video call application. In response to an operation on video call application icon 513 shown in FIG11A , electronic device 100 may display user interface 1110 shown in FIG11B .

[0429] As shown in FIG11B , user interface 1110 may include a standard video call control 1111 and a spatial audio / video call control 1112. Standard video call control 1111 may be used to conduct a video call, while the audio editing function remains disabled. Spatial audio / video call control 1112 may also be used to conduct a video call, while the audio editing function remains enabled.

[0430] In some embodiments, during a video call, the electronic device 100 may provide an audio editing control to turn the audio editing function on or off. In this way, the user can turn the audio editing function on or off at any time during the video call. In other words, the user does not need to decide whether to use the video call with the audio editing function turned on or the video editing function turned off before making the video call. In other words, the ordinary video call control 1111 and the spatial audio and video call control 1112 shown in Figure 11B are optional. The embodiments of the present application do not limit the above-mentioned operation for turning on or off the audio editing function during a video call.

[0431] In some embodiments, electronic device 100 can enable the audio editing function during a video call, allowing the user of electronic device 100 to edit the audio captured by electronic device 100 and then send the edited audio to the other party. In this way, the audio heard by the other party of electronic device 100 can be the audio edited by the user of electronic device 100.

[0432] In some embodiments, the electronic device 100 enables the audio editing function during a video call, and can also support the user on the electronic device 100 side to edit the audio of the other party. For example, during a video call between user 1 and user 2 on the electronic device 100 side, user 1 can edit the audio of user 2 on the electronic device 100. In this way, user 1 can listen to the audio of user 2 that has been edited by user 1. For another example, during a video call between user 1 on the electronic device 100 side and multiple other parties (such as user 2, user 3, etc.), user 1 can edit the audio of user 2 on the electronic device 100. In this way, in addition to the edited call end, that is, user 2, user 1 and other call ends, such as user 3, etc., can listen to the audio of user 2 that has been edited by user 1.

[0433] Optionally, before the user on the electronic device 100 edits the audio of the other party, he or she may request the consent of the other party. If the other party agrees, the user on the electronic device 100 may edit the audio of the other party.

[0434] In response to the operation of the spatial audio and video call control 1112 shown in FIG. 11B , the electronic device 100 may enable the audio editing function during the video call.

[0435] As shown in FIG11C , electronic device 100 may display user interface 1120. User interface 1120 may be a video call interface. In this embodiment, electronic device 100 is initiating a video call with a call partner. Electronic device 100 may be the initiator of the video call or the recipient of the video call. This is not limited in this embodiment of the present application.

[0436] As shown in Figure 11C, user interface 1120 may display a video image sent by the other end of the call. The video image sent by the other end of the call may be the image shown in Figure 11C, including audio image a, audio image b, and audio image c. User interface 1120 may also include a mark box 1121 and volume control 1121A associated with audio image a, a mark box 1122 and volume control 1122A associated with audio image b, a mark box 1123 and volume control 1123A associated with audio image c, a floating window 1124, audio and video controls 1125, sound field controls 1126, audio mixing controls 1127, time information 1128, and an audio editing indicator 1129.

[0437] The display style of the volume control 1121A may be the display style of the non-mute state as shown in Figure 11C. This may indicate that the volume of the sound image a in the audio of the other end of the call is not 0 (ie, not muted).

[0438] The display style of volume control 1122A and volume control 1123A may be the display style of the mute state as shown in Figure 11C. This may indicate that the volume of audio image b and audio image c in the audio of the other end of the call is 0, that is, in the mute state.

[0439] In some embodiments, a sound image may include multiple objects. For example, the sound image c shown in FIG11C may include multiple objects (for example, three objects). It is understandable that when multiple objects are close to each other but are all far away from the audio and video acquisition device, the area occupied by these multiple objects on the acquired image is smaller, and the sounds of these multiple objects can be considered to be emitted from the same spatial position. Compared with the objects that are closer to the audio and video acquisition device, the sounds of the multiple objects that are farther away from the audio and video acquisition device are weaker and have less impact on the sounds of the objects that are closer to the audio and video acquisition device. Therefore, the multiple objects that are close to each other but are all far away from the audio and video acquisition device can be determined to be one sound image.

[0440] The floating window 1124 may display an image captured by the electronic device 100. For example, the image captured by the electronic device 100 through the front camera.

[0441] The time information 1128 may be used to indicate the duration of the video call.

[0442] The audio editing indicator 1129 may be used to indicate that the audio editing function is enabled. Optionally, when the audio editing function is disabled, the electronic device 100 may cancel the display of the audio editing indicator 1129 on the user interface 1120 .

[0443] As can be seen, since the audio editing function is enabled, electronic device 100 can display the mark boxes and volume controls associated with each audio and video as shown in Figure 11C, as well as audio and video controls 1125, sound field controls 1126, and mixing controls 1127 on the video image of the caller. This makes it easier for the user on electronic device 100 to edit the audio of the caller.

[0444] In one possible implementation, the electronic device 100 receives a video image and an audio file from the other end of the call. The audio file may contain the audio signal of the sound field and each sound image of the other end of the call, as well as metadata. When an operation to edit the audio of the other end of the call is received, the electronic device 100 can edit the above audio file to obtain a new audio file. The electronic device 100 can use the new audio file to render and obtain audio data. Then, the electronic device 100 can use the audio data obtained by the above rendering to play the audio of the other end of the call. In this way, the user on the electronic device 100 side can listen to the audio of the other end of the call that has been edited by himself.

[0445] In another possible implementation, when an operation to edit the audio of the other end of the call is received, the electronic device 100 may send an audio editing instruction to the other end of the call. The audio editing instruction may include audio editing parameters for performing audio editing. After receiving the audio editing instruction, the other end of the call may edit the collected audio according to the audio editing parameters to obtain an audio file. The other end of the call may render the audio file to obtain audio data. Then, the other end of the call may send the audio data to the electronic device 100. The electronic device 100 may play the audio of the other end of the call based on the received audio data. The audio data received by the electronic device 100 is rendered from an audio file edited according to the audio editing instruction of the electronic device 100. Therefore, the user on the electronic device 100 side can listen to the audio of the other end of the call that has been edited by himself.

[0446] In another possible implementation, upon receiving an operation to edit the audio of the other party, the electronic device 100 may send an audio editing instruction to the server. The audio editing instruction may include audio editing parameters for performing audio editing. The server may be a server for enabling a video call between the electronic device 100 and the other party. The server may also receive a video image and an audio file from the other party. The audio file may include the audio signal of the sound field and each sound and image of the other party, as well as metadata. The server may edit the audio file of the other party according to the audio editing parameters of the electronic device 100 to obtain a new audio file. The server may then send the new audio file to the electronic device 100, which will render and play the new audio file. Alternatively, the server may render the new audio file to obtain audio data, and then hand the audio data over to the electronic device 100 for playback. In this way, the user on the electronic device 100 can listen to the audio of the other party that has been edited by them.

[0447] That is to say, the electronic device 100 can support the local user to edit the audio of the other end of the call. The device that specifically edits the audio file can be the electronic device 100, or can be the other end of the call, or can be a server.

[0448] As shown in Figure 11C, in response to the operation of the volume control 1122A, the electronic device 100 may display a volume bar 1122B as shown in Figure 11D. The volume bar 1122B can be used to adjust the volume of the sound image b.

[0449] As shown in FIG11D , in response to an operation on the audio / video control 1125, the electronic device 100 may display an audio / video a option 1125A, an audio / video b option 1125B, and an audio / video c option 1125C as shown in FIG11E . The audio / video a option 1125A, the audio / video b option 1125B, and the audio / video c option 1125C may be used to select audio / video a, audio / video b, and audio / video c, respectively, so that the user can edit the selected audio / video.

[0450] In some embodiments, in addition to the above-mentioned audio and video corresponding options of the other end of the video call, the electronic device 100 can also provide audio and video corresponding options on the video image in the floating window 1124 (that is, audio and video corresponding options on the electronic device 100 side).

[0451] In response to an operation on the floating window 1124 shown in Figure 11E, such as a click operation, the electronic device 100 can swap the display positions of the video image captured by the electronic device 100 and the video image from the call partner. Specifically, the electronic device 100 can display the user interface 1130 shown in Figure 11F.

[0452] As shown in FIG11F , user interface 1130 may display a video image captured by electronic device 100. The video image captured by electronic device 100 may be the image shown in FIG11F , including audio image d and audio image e. User interface 1130 may also include a mark box 1131 and volume control 1131A associated with audio image d, a mark box 1132 and volume control 1132A associated with audio image e, and a floating window 1133. Other controls on user interface 1130 may refer to user interface 1120 shown in FIG11C .

[0453] The floating window 1133 may display a video image sent by the other party of the electronic device 100 .

[0454] User interface 1130 may include an audio / visual control 1125. Since the video image captured by electronic device 100 and the video image from the other end of the call are displayed in different positions, in response to operating audio / visual control 1125, electronic device 100 may display the options corresponding to the audio / visual on electronic device 100 as shown in FIG11F: audio / visual d option 1125D and audio / visual e option 1125E. This allows the user on electronic device 100 to select and edit the audio / visual on their end.

[0455] In the user interface 1130 shown in FIG11F , the user of the electronic device 100 can edit the audio of the local end. Specifically, the electronic device 100 can edit the audio collected by the electronic device 100 based on the user's audio editing operation on the local end, thereby obtaining an audio file containing an audio signal and metadata. The electronic device 100 can then send the audio file to the other end of the call, or send the audio data rendered from the audio file to the other end of the call. In this way, the other end of the call can listen to the audio edited by the user of the electronic device 100.

[0456] If electronic device 100 transmits an audio file to a caller, it may first encode the audio file. For example, this encoding may include one or more of the following: channel encoding, sound object encoding, HOA encoding, metadata encoding, etc. Electronic device 100 may then transmit the encoded audio file to the caller. This encoding can reduce information redundancy in the audio file and improve audio file transmission efficiency.

[0457] In some embodiments, the electronic device 100 can enlarge the floating window on the video call interface so that the user can select the audio and video on the video image in the floating window for audio and video editing.

[0458] 11G , in response to a two-finger swipe in opposite directions on the floating window 1133, the electronic device 100 may enlarge the floating window 1133 to display the floating window 1133 shown in FIG11H . It can be seen that the floating window 1133 shown in FIG11H is larger than the floating window shown in FIG11G .

[0459] As shown in Figure 11H, when the floating window 1133 is enlarged, the video image displayed in the floating window 1133 is also enlarged. The electronic device 100 can display the audio editing controls associated with each sound and image of the video image in the floating window to facilitate the user to edit the sound and image. It is understandable that when the floating window 1133 is small, the size of the sound and image presented in the floating window is also small, or it is possible that only a part of the captured sound and image is presented. This makes it inconvenient for the user to select the sound and image on the video image in the floating window for editing. Enlarging the floating window and the video image displayed therein allows the user to more conveniently and accurately select the sound and image for editing.

[0460] Optionally, without enlarging the floating window, the electronic device 100 may also provide audio editing controls associated with each sound and image of the video image in the floating window, so that the user can edit the sound and image displayed in the floating window.

[0461] It should be noted that when the content displayed in the floating window 1133 is a video image on the electronic device 100 side (refer to the floating window 1124 shown in Figure 11C), the electronic device 100 can also support the user to enlarge the floating window and edit the audio and video on the electronic device 100 side.

[0462] The audio editing performed by the user of the electronic device 100 can refer to the embodiment of audio editing in the aforementioned video recording scenario, which will not be described in detail here.

[0463] In some embodiments, the electronic device 100 can identify the primary subject in the video image and retain the primary subject's sound in the audio, while suppressing all other sounds in the audio. This allows the user to quickly eliminate all other sounds in the audio during a video call, allowing the user to clearly hear the primary subject's sound.

[0464] 12A to 12C exemplarily show schematic diagrams of other audio editing scenarios.

[0465] As shown in FIG12A , electronic device 100 may display user interface 1130. User interface 1130 may be described with reference to FIG11F . User interface 1130 may include a playback mode control 1134. Playback mode control 1134 may be used to select an audio playback mode. Audio playback modes may include a natural mode and a vocal mode.

[0466] In the natural mode, the electronic device 100 may retain all the sounds of the sound image in the audio collected by the electronic device 100 .

[0467] In the human voice mode, the electronic device 100 can identify the main object on the video image captured by the electronic device 100. For example, the main object can be the object that occupies the largest display area on the video image. The embodiment of the present application does not limit the method by which the electronic device 100 determines the main object on the image. When the main object is determined, the electronic device 100 can edit the audio tracks of other audio and video except the main object so that the volume of other audio and video except the main object is 0 (i.e., remains silent).

[0468] The text displayed on playback mode control 1134 is "Natural Mode," as shown in Figure 12A , indicating that the audio playback mode is currently in Natural Mode. As can be seen in Figure 12A , the display styles of volume control 1131A associated with audio image d and volume control 1132A associated with audio image e are both non-muted. This means that the audio signals of audio image d and audio image e can be captured and retained by electronic device 100. The other party on electronic device 100 can hear the sounds of audio image d and audio image e.

[0469] In response to the operation of the play mode control 1134 in FIG12A, the electronic device 100 may display the natural mode option 1134A and the vocal mode option 1134B shown in FIG12B. The natural mode option 1134A may be used to select the natural mode. The vocal mode option 1134B may be used to select the vocal mode.

[0470] As shown in FIG12B , in response to the operation of the vocal mode option 1134B, the electronic device 100 may change the text content displayed on the play mode control 1134 to “Voice Mode” as shown in FIG12C . This may indicate that the audio playback mode is currently in the vocal mode.

[0471] As shown in Figure 12C, when switching from natural mode to human voice mode, the electronic device 100 can identify the main object on the video image captured by the electronic device 100. For example, the main object is the sound image d shown in Figure 12C. The electronic device 100 can edit the audio tracks of other sound images other than the sound image d, such as the sound image e, so that the sound image e remains silent. It can be seen that since the electronic device 100 edits the sound image e to make it silent, the display style of the volume control 1132A associated with the sound image e can be changed from the display style of the non-muted state shown in Figure 12A to the display style of the muted state shown in Figure 12C. In other words, the electronic device 100 can suppress the audio signal of the sound image e. The other end of the call of the electronic device 100 will only hear the sound of the main object (i.e., the sound image d), and will not hear the sound of other sound images on the electronic device 100 side except the above-mentioned main object.

[0472] In some embodiments, when switching to the human voice mode, the electronic device 100 may provide one or more options corresponding to the sound and image, allowing the user to select a sound and image as the main object. In this way, the user can retain the sound of one sound and image as needed and mute the other sound and images.

[0473] It is understandable that, in addition to muting other audio and video except the main object by selecting the audio playback mode, the user can also manually edit one or more audio and video to mute the one or more audio and video.

[0474] In some embodiments, the aforementioned human voice mode can be applied not only to audio captured by electronic device 100 but also to audio from the other end of a call with electronic device 100. The electronic device 100 can receive a video image and an audio file from the other end of the call. This audio file can be rendered and played by the other end of the call. Upon detecting the application of the human voice mode to the audio from the other end of the call, the electronic device 100 can identify a primary object based on the video image from the other end of the call and edit the audio file based on the primary object. The electronic device 100 can retain the audio signal of the primary object in the audio file from the other end of the call and mute all other objects except the primary object. Alternatively, upon detecting the application of the human voice mode to the audio from the other end of the call, the electronic device 100 can send an audio editing instruction. This audio editing instruction can be used to instruct the electronic device 100 to mute all other audio and video from the other end of the call except the primary object. The device that performs the specific editing of the audio file from the other end of the call can be the device of the other end of the call or a server. The electronic device 100 can receive audio data from the other end of the call. This audio data can be audio data in which all other audio and video from the other end of the call except the primary object are muted. In this way, the user on the electronic device 100 side can only listen to the sound of the main object on the other end of the call, and cannot hear the sounds of other audio and video on the other end of the call except the main object.

[0475] As can be seen from the above embodiments, in a video call scenario, the user on the electronic device 100 side can suppress the sounds of other objects on the local side except the main object with one click. This makes it convenient for the user on the other end of the call with the electronic device 100 to only listen to the sound of the main object on the electronic device 100 side without being disturbed by other sounds and images on the electronic device 100 side. In addition, the user on the electronic device 100 side can also suppress the sounds and images of other objects on the other end of the call except the main object with one click. This makes it convenient for the user on the electronic device 100 side to only listen to the sound of the main object on the other end of the call without being disturbed by other sounds and images on the other end of the call. The above embodiments can improve the user's video call experience.

[0476] The embodiments shown in Figures 12A to 12C can also be applied to the aforementioned video recording scenario. In this way, the user can suppress the sounds of other than the main subject in the captured video with one click, making the captured audio purer.

[0477] In some embodiments, electronic device 100 can conduct video calls with multiple peers. Furthermore, electronic device 100 can edit the audio of one or more peers. When electronic device 100 edits the audio of one peer, the audio heard by the other peers may be the edited audio of the aforementioned peer.

[0478] 13A and 13B are schematic diagrams of other audio editing scenarios.

[0479] As shown in Figure 13A, electronic device 100 may display user interface 1310. User interface 1310 may be an interface for video calls between electronic device 100 and multiple call peers. This description uses a video call between electronic device 100 and three call peers as an example. The call peer where electronic device 100 is located may be referred to as Room A. The three call peers of electronic device 100 may be referred to as Room B1, Room B2, and Room B3, respectively. This embodiment of the application does not limit the names of the various call peers.

[0480] The user interface 1310 may include a display area 1311 , a display area 1312 , a display area 1313 , and a display area 1314 .

[0481] Display area 1311 can be used to display the video image sent by the call terminal corresponding to room B1. Display area 1311 can also include a mute control 1311A. Mute control 1311A can be used to mute or unmute room B1 as a whole. Muting room B1 as a whole means that all call terminals in the video call will not hear any sound in room B1.

[0482] Display area 1312 may be used to display the video image sent by the call terminal corresponding to room B2. Display area 1312 may include a mute control 1312A. Mute control 1312A may be used to mute or unmute the entire room B2.

[0483] Display area 1313 may be used to display the video image sent by the call terminal corresponding to room B3. Display area 1313 may include a mute control 1313A. Mute control 1313A may be used to mute or unmute the entire room B3.

[0484] Display area 1314 may be used to display the video image of room A, that is, the video image captured by electronic device 100. Display area 1314 may include a mute control 1314A. Mute control 1314A may be used to mute or unmute room A as a whole.

[0485] For example, the video image captured by electronic device 100 may include: audio image 11, audio image 12, and audio image 13. Electronic device 100 may display one or more of the following controls for audio editing in display area 1314: a volume bar and mute control associated with each audio image, audio and video controls 1315, sound field controls 1316, mixing controls 1317, and play mode controls 1318. The functions of the above-mentioned controls for audio editing can be referred to the description of the aforementioned embodiments. A user of electronic device 100 can use the controls in display area 1314 to perform audio editing, such as audio and video editing, sound field editing, and mixing editing, on the audio in room A.

[0486] When the audio in room A is edited, the three call counterparts of the electronic device 100 , namely, users in room B1 , room B2 , and room B3 , can all hear the edited audio in room A.

[0487] In response to an operation, such as a click operation, on display area 1311 shown in Figure 13A, electronic device 100 may display user interface 1320 shown in Figure 13B. Comparing Figures 13A and 13B, electronic device 100 may swap the display areas of the video image of room A and the video image of room B1.

[0488] As shown in FIG. 13B , the user interface 1320 may include a display area 1321 , a display area 1312 , a display area 1313 , and a display area 1322 .

[0489] The display area 1321 may be located at the position of the display area 1311 shown in FIG13A . The display area 1321 may be used to display the video image of room A.

[0490] Display area 1322 can be displayed in the location where display area 1314 shown in FIG13A was originally displayed. Display area 1322 can be used to display the video image transmitted by the call end corresponding to room B1. For example, the video image transmitted by the call segment corresponding to room B1 can include audio image 21 and audio image 22. Electronic device 100 can display controls for audio editing, such as volume bars and mute controls associated with each audio image, in display area 1322. The controls for audio editing in display area 1322 can refer to the controls for audio editing in display area 1314 shown in FIG13A. Electronic device 100 can use the controls for audio editing in display area 1322 to perform audio editing, such as audio and video editing, sound field editing, and mixing editing, on the audio in room B1.

[0491] When the user on the electronic device 100 side edits the audio in room B1, the electronic device 100 and the other two call partners of the electronic device 100, namely the users in room A, room B2, and room B3, can all hear the edited sound in room B1.

[0492] For example, during a video call, the call end corresponding to room B1 can determine the spatial position of each sound image in room B1 relative to a preset position. The above-mentioned preset position can be the position of the device used for audio and video calls in room B1. Here, the preset position is taken as position 0 in room B1 as an example for explanation. The embodiment of the present application does not limit the above-mentioned position 0. For example, the call end corresponding to room B1 can determine that the sound image 21 shown in Figure 13B is located at position 1 relative to position 0, and the sound image 22 is located at position 2 relative to position 0. The audio file generated by the call end corresponding to room B1 may contain the spatial positions of the above-mentioned sound image 21 and sound image 22. When the electronic device 100 plays the audio from the call end corresponding to room B1, the user on the electronic device 100 side (i.e., the user in room A) can listen to the sound of the sound image 21 and sound image 22 in room B1, and experience the spatial sense and direction of the sound in terms of hearing. For example, a user in room A can perceive themselves as being at position 0 in room B2, and can sense that the sound of sound image 21 is emanating from position 1, and that the sound of sound image 22 is emanating from position 2. Similarly, users at other endpoints (such as the endpoint corresponding to room B2 or room B3) can experience the same auditory experience as the user at electronic device 100 when listening to audio in room B1.

[0493] The user on the electronic device 100 side can adjust the spatial position of the sound image in room B1. For example, the spatial position of the sound image 22 can be moved from position 2 to position 3. Then, when the electronic device 100 plays the audio from the call terminal corresponding to room B1, the user in room A can be placed at position 0 in room B2 and feel that the sound of the sound image 21 is emanating from position 1, and the sound of the sound image 22 is emanating from position 3. Similarly, when users on other call terminals (such as the call terminal corresponding to room B2 and the call terminal corresponding to room B3) listen to the audio in room B1, they can also feel the direction of the sound image 22 moving from position 2 to position 3.

[0494] As can be seen from the audio editing scenario above, a user can edit their own audio during a video call, allowing the other party to hear the user-edited audio. Furthermore, a user can also edit the audio of the other party during a video call, allowing themselves and other parties to hear the edited audio of the other party. In this way, the user, as the host, can control the audio of multiple parties connected to the video call, ensuring a more efficient video call. The above embodiments facilitate easy and intuitive audio editing for users during video calls, enhancing the user's video call experience.

[0495] FIG14 exemplarily shows a flowchart of an audio processing method provided in an embodiment of the present application.

[0496] As shown in FIG14 , the audio processing method may include steps S1411 to S1414 , wherein steps S1411 to S1414 may be collectively referred to as step S1400 . The device executing steps S1411 to S1414 may be, for example, the electronic device 100 .

[0497] S1411. Collect video images and audio.

[0498] The process of the electronic device 100 collecting video images and audio can refer to the introduction of the collection unit 310 shown in Figure 3 above.

[0499] S1412: Determine audiovisual information and sound field environment information based on the video image and audio. The audiovisual information includes: the audio track of the audio image contained in the audio, the spatial position of the audio image, and the display position of the audio image on the video image.

[0500] The sound field environment information may include, but is not limited to: one or more of: reverberation duration, room type, room size and / or room reflective material.

[0501] The process of determining the audio-visual information and the sound field environment information may refer to the introduction of the information extraction unit 320 in FIG. 3 .

[0502] S1413: Receive an audio editing operation, and determine audio editing parameters according to the audio editing operation. The audio editing operation includes performing one or more of the following editing operations on the audio: audio and video editing, sound field editing, and mixing editing.

[0503] The above audio editing operations can be specifically referred to the audio editing operations described in the aforementioned video recording scenario and video call scenario. No further details will be given here.

[0504] S1414: Determine an audio file based on audio editing parameters, audio and video information, and sound field environment information.

[0505] An audio file can contain the audio signal of the sound field and each sound image, as well as metadata. The metadata can include the spatial position of the sound image, sound field environment information, and other data used to correctly render the audio signal.

[0506] In some embodiments, the electronic device 100 can generate audio files in multiple formats. For example, audio files in stereo format, audio files in surround sound format, and audio files in 3D sound format. Rendering and playing audio files in different formats can allow users to experience different sound effects.

[0507] In some embodiments, electronic device 100 can transmit audio files to other devices. Specifically, electronic device 100 can encode the audio files before transmitting them to other devices. Such encoding may include, but is not limited to, channel encoding, sound object encoding, HOA encoding, and metadata encoding. Such encoding can reduce information redundancy in audio files and improve audio file transmission efficiency.

[0508] FIG15 exemplarily shows a flowchart of another audio processing method provided in an embodiment of the present application.

[0509] As shown in FIG15 , the audio processing method may include steps S1511 to S1513 . The device that executes steps S1511 to S1513 and steps S1411 to S1414 may be, for example, the electronic device 100 .

[0510] S1511. Obtain the video image and audio file 1 of video 1.

[0511] The electronic device 100 may store a video 1. Alternatively, the electronic device 100 may receive a video 1 sent by another device. The video 1 may include a video image and an audio file 1.

[0512] The audio file 1 may include audio signals of the sound field and each sound image, as well as metadata.

[0513] S1512: Receive an audio editing operation, and determine audio editing parameters according to the audio editing operation. The audio editing operation includes performing one or more of the following editing operations on the audio: audio and video editing, sound field editing, and mixing editing.

[0514] S1513. Edit audio file 1 according to the audio editing parameters to obtain audio file 2, and store audio file 2 as the audio file of video 1.

[0515] Editing the audio file 1 according to the audio editing parameters may specifically include editing the audio signals (ie, audio tracks) of the sound field and each sound image according to the audio editing parameters, and editing the metadata.

[0516] The electronic device 100 can replace the audio file 1 in the video 1 with the audio file 2. In this way, when playing the video 1, the electronic device 100 can display the video image in the video 1 and play the audio according to the audio file 2.

[0517] As can be seen from the method shown in FIG. 15 , the user can edit an existing video in the electronic device 100 to change the playback effect of the audio in the video.

[0518] FIG16 exemplarily shows a flowchart of another audio processing method provided in an embodiment of the present application.

[0519] As shown in Figure 16, the audio processing method may include steps S1611 to S1623. The audio processing method may be applied to a video call scenario.

[0520] 1. (S1611-S1615) Edit the local audio.

[0521] S1611. Electronic devices 100, 101, and 102 access the same video call.

[0522] The electronic device 100 executes step S1400 shown in FIG. 14 to obtain an audio file.

[0523] S1612. The electronic device 100 renders the audio file to obtain audio data 1.

[0524] S1613 : Electronic device 100 sends audio data 1 to electronic device 101 and electronic device 102 .

[0525] S1614 , the electronic device 101 plays audio according to the audio data 1 .

[0526] S1615 . The electronic device 102 plays audio according to the audio data 1 .

[0527] It can be understood that, in addition to playing audio according to the audio data 1 , the electronic device 101 and the electronic device 102 can also display video images from the electronic device 100 .

[0528] In some embodiments, the electronic device 100 may send the audio file to the electronic device 101 and the electronic device 102. Then, the electronic device 101 and the electronic device 102 render the audio file to obtain the audio data 1, and play the audio according to the audio data 1.

[0529] In other embodiments, electronic device 100 may send the audio file to a server (e.g., server 400 shown in FIG. 4C ). The server may aggregate the audio files from each call end. The server may render the audio file from electronic device 100 to obtain the aforementioned audio data 1. The server may then send audio data 1 to electronic devices 101 and 102 for audio playback.

[0530] 2. (S1616-S1623) Edit the audio of the other end of the call.

[0531] S1616: The electronic device 100 receives an operation to edit the audio of the electronic device 101 and determines audio editing parameters.

[0532] S1617: The electronic device 100 sends an audio editing instruction to the electronic device 101, where the audio editing instruction includes audio editing parameters.

[0533] S1618. The electronic device 101 edits the collected audio according to the audio editing instruction to determine the audio file a.

[0534] The method for the electronic device 101 to determine the audio file a may refer to the method flowchart shown in the aforementioned FIG. 14 .

[0535] S1619. The electronic device 101 renders the audio file a to obtain audio data 2.

[0536] S1620 : The electronic device 101 sends the audio data 2 to the electronic device 100 .

[0537] S1621 . Electronic device 101 sends audio data 2 to electronic device 102 .

[0538] S1622 : The electronic device 100 plays audio according to the audio data 2 .

[0539] S1623 . The electronic device 102 plays audio according to the audio data 2 .

[0540] It can be understood that in addition to playing audio according to the audio data 2 , the electronic device 100 and the electronic device 102 can also display the video image from the electronic device 101 .

[0541] In some embodiments, the electronic device 100 may send an audio editing instruction to the server. The server may also receive the audio file b from the electronic device 101. The audio file b may be an audio file that has not yet been edited according to the audio editing instruction of the electronic device 100. The server may edit the audio file b according to the audio editing instruction of the electronic device 100 to obtain the audio file a. The method by which the server determines the audio file a from the audio file b may refer to the method flow chart shown in FIG15 . The server may then send the audio file a to the electronic device 100 and the electronic device 102. Alternatively, the server may render the audio file a to obtain the audio data 2, and send the audio data 2 to the electronic device 100 and the electronic device 102 for them to play the audio.

[0542] As shown in the method shown in Figure 16, a user can edit their own audio during a video call, allowing the other party to hear the user-edited audio. Furthermore, the user can also edit the audio of the other party during a video call, allowing themselves and other parties to hear the edited audio of the other party. In this way, the user, as the host, can control the audio of multiple parties connected to the video call, ensuring a more efficient video call. The above embodiments facilitate easy and intuitive audio editing for users during video calls, enhancing the user's video call experience.

[0543] It is understood that the various user interfaces described in the embodiments of this application are merely exemplary interfaces and do not limit the scope of this application. In other embodiments, the user interface may adopt a different interface layout, include more or fewer controls, and add or remove other functional options. As long as they are based on the same inventive concept provided by this application, they are all within the scope of protection of this application.

[0544] It should be noted that, without causing any contradiction or conflict, any feature in any embodiment of the present application, or any part of any feature, can be combined, and the combined technical solution is also within the scope of the embodiments of the present application.

[0545] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio processing method, It is characterized in that The method comprises: The first electronic device displays a first video image of a first video, the first video image includes a first sound image, and the first video includes a first audio; The first electronic device displays a first sound and image control on a screen, wherein the first sound and image control is used to adjust a sound feature of the first sound and image, wherein adjusting the sound feature of the first sound and image includes one or more of the following: adjusting a volume of the first sound and image, adjusting a timbre of the first sound and image, and / or adjusting a spatial position of the first sound and image; In response to a first operation on the first sound and image control, the first electronic device adjusts a sound feature of the first sound and image in the first audio.

2. The method according to claim 1, It is characterized in that The first video image also includes a second audio image, and the method further includes: The first electronic device displays a second sound and image control on the screen, wherein the second sound and image control is used to adjust a sound feature of the second sound and image, wherein adjusting the sound feature of the second sound and image includes one or more of the following: adjusting a volume of the second sound and image, adjusting a timbre of the second sound and image, and / or adjusting a spatial position of the second sound and image; In response to a second operation on the second sound and image control, the first electronic device adjusts sound characteristics of the second sound and image in the second audio.

3. The method according to claim 1 or 2, It is characterized in that The method further comprises: The first electronic device displays a first sound field control on the screen, wherein the first sound field control is used to adjust a first sound field in the first audio, wherein adjusting the first sound field includes one or more of the following: adjusting a type of a room where the first sound field is located, adjusting a size of a room where the first sound field is located, and / or adjusting a reflective material of a room where the first sound field is located; In response to a third operation on the first sound field control, the first electronic device adjusts the first sound field.

4. The method according to any one of claims 1 to 3, It is characterized in that The method further comprises: The first electronic device displays a first mixing control on the screen, the first mixing control being used to adjust mixing parameters of the first audio, the mixing parameters comprising one or more of the following: a mixing template, a reverberation time, an equalizer parameter, a size of a room where the sound field is located, and / or a reflective material of a room where the sound field is located; The first electronic device determines a first mixing parameter according to a fourth operation on the first mixing control, and mixes the audio signal of the first sound field in the first audio with the audio signals of one or more sound images according to the first mixing parameter.

5. The method according to any one of claims 2 to 4, It is characterized in that The method further comprises: The first electronic device receives a fifth operation for the first sound image and the second sound image, wherein the fifth operation is used to merge the first sound image and the second sound image; In response to the fifth operation, the first electronic device displays a third sound and image control on the screen, where the third sound and image control is used to adjust the sound characteristics of the first sound and image and the second sound and image.

6. The method according to any one of claims 1 to 5, It is characterized in that The first audio further includes an audio signal corresponding to a third sound image, and the third sound image is not in the first video image. The method further includes: The first electronic device displays a fourth sound and image control on the screen, where the fourth sound and image control is used to adjust the sound characteristics of the third sound and image.

7. The method according to claim 6, It is characterized in that The display position of the fourth sound and image control on the screen is related to the spatial position of the third sound and image.

8. The method according to any one of claims 2 to 7, It is characterized in that The sound feature of the first sound image is a first sound feature, the sound feature of the second sound is a second sound feature, and the method further includes: The first electronic device receives a sixth operation for the first sound image and the second sound image, wherein the sixth operation is used to exchange sound features of the first sound image and the second sound image; In response to the sixth operation, the first electronic device determines the sound feature of the first sound image in the first audio as the second sound feature, and determines the sound feature of the second sound image in the first audio as the first sound feature; When receiving an operation on the first audio-visual control, the first electronic device adjusts the second sound feature; When receiving an operation on the second audio-visual control, the first electronic device adjusts the first sound feature.

9. The method according to any one of claims 1 to 8, It is characterized in that The method further comprises: The first electronic device receives a seventh operation; In response to the seventh operation, the first electronic device identifies a main object in the first video image; When the main object in the first video image is identified as the first sound and image, the first electronic device removes audio signals of other sound and images in the first audio except for the first sound and image.

10. The method according to any one of claims 1 to 9, It is characterized in that The display position of the first audio and video control on the screen is related to the display position of the first audio and video on the video image of the first video.

11. The method according to any one of claims 1 to 10, It is characterized in that Before the first electronic device displays the first video image of the first video, the method further includes: The first electronic device receives an operation of opening a camera application and / or starting to record a video, captures an image through an image acquisition device, and captures audio through an audio acquisition device; wherein the image captured by the image acquisition device includes a video image of the first video, and the audio captured by the audio acquisition device includes the first audio; The first electronic device displays the image captured by the image acquisition device in real time, and the first video image is an image displayed in real time during the capturing process.

12. The method according to any one of claims 1 to 10, It is characterized in that Before the first electronic device displays the first video image of the first video, the method further includes: The first electronic device obtains the first video from a memory of the first electronic device; The first electronic device receives an operation of playing the first video, and the first video image is an image displayed during the video playback process.

13. The method according to claim 11 or 12, It is characterized in that The first electronic device establishes a communication connection with a monitoring device, and the method further includes: The first electronic device sends a second audio to the monitoring device for playback, where the second audio is the first audio after one or more of the following adjustments have been made to the first audio: sound characteristics of one or more sound images in the first audio have been adjusted, the first sound field of the first audio has been adjusted, and / or mixing parameters of the first audio have been adjusted.

14. The method according to claim 11 or 12, It is characterized in that The first electronic device establishes a communication connection with a monitoring device. After the first electronic device adjusts the sound feature of the first sound image in the first audio, the method further includes: The first electronic device sends the audio signal of the first sound image after the sound characteristics are adjusted to the monitoring device for playback.

15. The method according to any one of claims 1 to 10, It is characterized in that The first electronic device and the second electronic device enter into a first video call, and the method further includes: The first electronic device sends a video image of the first video and a second audio to the second electronic device, where the second audio is the audio of the first audio after one or more of the following adjustments are made: sound features of one or more sound images in the first audio are adjusted, a first sound field of the first audio is adjusted, and / or a mixing parameter of the first audio is adjusted; The first video image is an image captured by the first electronic device during the first video call.

16. The method according to claim 15, It is characterized in that The method further comprises: The first electronic device receives a second video from the second electronic device; The first electronic device displays a second video image of the second video, wherein the second video image includes a fourth sound and image; The first electronic device displays a fifth audio-visual control on the screen, wherein the fifth audio-visual control is used to adjust the sound of the fourth audio-visual Characteristics, wherein adjusting the sound characteristics of the fourth sound image includes one or more of the following: adjusting the volume of the fourth sound image, adjusting the timbre of the fourth sound image, and / or adjusting the spatial position of the fourth sound image.

17. The method according to claim 16, It is characterized in that The method further comprises: The first electronic device determines a first editing parameter according to an operation on the fifth audio-visual control; The first electronic device sends a first editing instruction to the second electronic device, where the first editing instruction includes the first editing parameter; The first electronic device receives a third video from the second electronic device, wherein the third video includes a third audio, wherein the sound feature of the fourth sound image in the third audio is adjusted according to the first editing parameter.

18. The method according to claim 17, It is characterized in that A third electronic device is also connected to the first video call, and the third video is also sent to the third electronic device for playback.

19. The method according to any one of claims 15 to 18, It is characterized in that The method further comprises: The first electronic device displays a second sound field control on the screen, and the second sound field control is used to adjust the second sound field in the audio contained in the video sent from the second electronic device, and adjusting the second sound field includes one or more of the following: adjusting the type of room where the second sound field is located, adjusting the size of the room where the second sound field is located, and / or adjusting the reflective material of the room where the second sound field is located.

20. The method according to any one of claims 15 to 19, It is characterized in that The method further comprises: The first electronic device displays a second mixing control on the screen, and the second mixing control is used to adjust the mixing parameters of the audio contained in the video sent from the second electronic device, and the mixing parameters include one or more of the following: mixing template, reverberation duration, equalizer parameters, the size of the room where the sound field is located, and / or the reflective material of the room where the sound field is located.

21. The method according to any one of claims 1 to 10, It is characterized in that The first electronic device and the second electronic device enter into a first video call, and before the first electronic device displays a first video image of the first video, the method further includes: The first electronic device receives the first video from the second electronic device.

22. An electronic device, It is characterized in that The electronic device includes a camera, a microphone array, a memory and a processor, wherein the camera is used to capture images; the microphone array is used to collect audio; the memory is used to store computer programs; and the processor is used to call the computer program so that the electronic device executes the method described in any one of claims 1 to 21.

23. A computer-readable storage medium comprising instructions, It is characterized in that When the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 21.

24. A computer program product, It is characterized in that The computer program product comprises computer instructions, and when the computer instructions are run on an electronic device, the electronic device executes the method according to any one of claims 1 to 21.

Images