Audio processing method and related device

Abstract

The invention provides an audio processing method and a related device. The electronic equipment can determine sound image information and sound field environment information according to video images and audios in videos. Based on the information, the electronic equipment can provide an audio editing control on a user interface, so that a user can carry out audio editing such as sound image editing, sound field editing and sound mixing editing. The electronic equipment can process the audio according to the audio editing operation of the user, so that after the processed audio is rendered, the sound effect heard by the user can reach the effect adjusted by the user. The audio editing operation is simple, convenient and visual, the user can conveniently carry out personalized editing on the audio in audio and video recording scenes such as a video recording scene and a video call scene, and the use experience of the user for recording the audio and the video is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an audio processing method and related devices. Background Art

[0002] With the development of audio and video technology, sound restoration technology has evolved from mono to dual-channel, stereo, surround, and three-dimensional sound. More and more audio can allow users to experience the surround and immersive feeling of sound. In audio recording or playback scenarios, users may need to edit the audio in a personalized way to obtain and listen to audio that meets their personal requirements. Summary of the invention

[0003] The present application provides an audio processing method and related devices. An electronic device can provide an audio editing control on a video screen for a user to perform audio editing such as audio and video editing, sound field editing, and mixing editing. The above audio editing operation is simple and intuitive, and can facilitate users to perform personalized editing of audio in video recording scenes such as video recording scenes and video call scenes, thereby improving the user's experience in recording audio and video.

[0004] In a first aspect, the present application provides an audio processing method. In which, a first electronic device can display a first video image of a first video, the first video image includes a first sound image, and the first video includes a first audio; the first electronic device can display a first sound image control on the screen, the first sound image control is used to adjust the sound characteristics of the first sound image, and adjusting the sound characteristics of the first sound image includes one or more of the following: adjusting the volume of the first sound image, adjusting the timbre of the first sound image, and / or adjusting the spatial position of the first sound image; in response to a first operation on the first sound image control, the first electronic device adjusts the sound characteristics of the first sound image in the first audio.

[0005] The first audio may include an audio signal of a first sound image. The electronic device may adjust the sound characteristics of the first sound image by editing the audio signal of the first sound image.

[0006] The first audio-visual control may refer to one or more of the following controls in the embodiment of the present application: Figure 5C The volume bar 531A, mute control 531B, pan control 534 and / or Fig. 5F One or more controls in the audio and video editing box 541 shown.

[0007] It can be seen from the above method that the first electronic device can provide relevant audio and video controls for the audio and video captured on the video image, so that the user can adjust the sound characteristics of the audio and video. The user can directly edit the audio and video of the captured sound on the video image. For example, the user can mute unnecessary audio and video, enhance the human voice of some important audio and video, change the timbre of one or more audio and video, and so on. The above audio and video editing operation is simple and intuitive, and it is easy for users to understand and get started. In this way, users can get audio and video that meet their personal needs during the video shooting process, or let the other end of the call listen to the sound characteristics of the audio and video adjusted by themselves in the video call scene.

[0008] In combination with the first aspect, in some embodiments, the volume of the first sound image may be the first volume at the first moment of being captured. The first operation may be used to adjust the volume of the first sound image to the second volume. During the process of playing the video after the sound feature of the first sound image is adjusted, when the video is played to the moment corresponding to the first moment, the volume of the first sound image is the second volume.

[0009] In combination with the first aspect, in some embodiments, the timbre of the first sound image may be the first timbre in the first device being filmed. The first operation may be used to adjust the timbre of the first sound image to a second timbre. During the process of playing the video after the sound feature of the first sound image is adjusted, when the video is played to the moment corresponding to the first moment, the timbre of the first sound image is the second timbre.

[0010] In combination with the first aspect, in some embodiments, the spatial position of the first sound image may be the first position in the first device being filmed. The first operation may be used to adjust the spatial position of the first sound image to the second position. In the process of playing the video after the sound characteristics of the first sound image are adjusted, when the video is played to the moment corresponding to the first moment, the spatial position of the first sound image is the second position. That is, the listener can watch the video and listen to the audio. When listening to the moment corresponding to the first moment, the listener can distinguish that the first sound image is located at the second position.

[0011] In combination with the first aspect, in some embodiments, the types of the first sound and image may include people and musical instruments.

[0012] In combination with the first aspect, in some embodiments, the first video image also includes a second sound image. The first electronic device displays a second sound image control on the screen, and the second sound image control is used to adjust the sound characteristics of the second sound image. Adjusting the sound characteristics of the second sound image includes one or more of the following: adjusting the volume of the second sound image, adjusting the timbre of the second sound image, and / or adjusting the spatial position of the second sound image; in response to the second operation on the second sound image control, the first electronic device adjusts the sound characteristics of the second sound image in the second audio.

[0013] The first electronic device may perform sound object tracking on the first audio to separate audio signals (ie, audio tracks) of different sound images contained in the first audio.

[0014] As can be seen from the above embodiments, the video image displayed by the first electronic device may include multiple captured audio and video. The user can edit one or more of the audio and video separately. This can improve the user's experience of recording audio and video and better meet the user's needs for audio editing.

[0015] In combination with the first aspect, in some embodiments, the first electronic device displays a first sound field control on a screen, and the first sound field control is used to adjust the first sound field in the first audio, and adjusting the first sound field includes one or more of the following: adjusting the type of room where the first sound field is located, adjusting the size of the room where the first sound field is located, and / or adjusting the reflective material of the room where the first sound field is located; in response to a third operation on the first sound field control, the first electronic device adjusts the first sound field.

[0016] The first audio may include an audio signal of the first sound field (i.e., a sound bed). In addition to separating the audio signal of the sound image, the first electronic device may also separate the audio signal of the first sound field from the first audio. Then, the first electronic device may adjust the first sound field by editing the audio signal of the first sound field.

[0017] The first sound field control mentioned above can refer to this application Fig.5I One or more controls in the sound field edit box 551 shown.

[0018] In combination with the first aspect, in some embodiments, the first electronic device displays a first mixing control on a screen, where the first mixing control is used to adjust mixing parameters of the first audio, where the mixing parameters include one or more of the following: a mixing template, reverberation duration, equalizer parameters, the size of the room where the sound field is located, and / or the reflective material of the room where the sound field is located; the first electronic device determines the first mixing parameter based on a fourth operation on the first mixing control, and mixes the audio signal of the first sound field in the first audio with the audio signals of one or more sound images according to the first mixing parameter.

[0019] The first mixing control mentioned above can refer to this application Figure 5K to Figure 5M One or more controls in the mix edit box 561 are shown.

[0020] The audio signal of the first sound field mixed by the first electronic device may be an adjusted audio signal of the first sound field. The audio signal of one or more sound images mixed by the first electronic device may be an audio signal with adjusted sound features of the one or more sound images.

[0021] As can be seen from the above embodiments, in addition to providing controls for editing audio and video, the electronic device can also provide controls for editing the sound field and / or controls for editing mixing parameters. In this way, the user can perform multi-faceted editing of the audio in the video to obtain audio that meets the user's needs.

[0022] In combination with the first aspect, in some embodiments, the first electronic device receives a fifth operation for the first sound image and the second sound image, and the fifth operation is used to merge the first sound image and the second sound image; in response to the fifth operation, the first electronic device displays a third sound image control on the screen, and the third sound image control is used to adjust the sound characteristics of the first sound image and the second sound image.

[0023] The fifth operation mentioned above can refer to this application Fig. 6E and Fig. 6F The operation shown.

[0024] As can be seen from the above embodiments, the user can merge one or more audio and video images on the video image, so that the sound features of multiple audio and video images can be edited simultaneously in one audio editing operation, which can improve the efficiency of audio and video editing for the user.

[0025] In combination with the first aspect, in some embodiments, the first audio also includes an audio signal corresponding to a third sound image, the third sound image is not in the first video image, and the first electronic device displays a fourth sound image control on the screen, and the fourth sound image control is used to adjust the sound characteristics of the third sound image.

[0026] As can be seen from the above embodiments, the camera has a limited shooting angle, and the video image may only contain a portion of the sound and image. The electronic device can provide a control for representing the sound and image that are not captured, so that the user can perform audio editing on the sound and image that are not captured.

[0027] In combination with the first aspect, in some embodiments, the display position of the fourth audio and video control on the screen is related to the spatial position of the third audio and video. In this way, the user can determine which audio and video the fourth audio and video control corresponds to based on the display position of the fourth audio and video control, and then edit the audio and video that has not been captured as required.

[0028] In combination with the first aspect, in some embodiments, the sound feature of the first sound image is the first sound feature, the sound feature of the second sound is the second sound feature, and the first electronic device receives a sixth operation for the first sound image and the second sound image, and the sixth operation is used to exchange the sound features of the first sound image and the second sound image; in response to the sixth operation, the first electronic device determines the sound feature of the first sound image in the first audio as the second sound feature, and determines the sound feature of the second sound image in the first audio as the first sound feature; when receiving an operation on the first sound image control, the first electronic device adjusts the second sound feature; when receiving an operation on the second sound image control, the first electronic device adjusts the first sound feature.

[0029] It can be seen from the above embodiment that when the user believes that the sound features of the first sound image and the second sound image are opposite, the user can exchange the sound features of the first sound image and the second sound image, which can improve the accuracy of determining the sound features of the sound image.

[0030] In combination with the first aspect, in some embodiments, the first electronic device receives a seventh operation; in response to the seventh operation, the first electronic device identifies the main object in the first video image; when the main object in the first video image is identified as the first sound and image, the first electronic device removes the audio signals of other sound and images in the first audio except the first sound and image.

[0031] The seventh operation mentioned above can refer to this application Fig. 12A and Fig. 12B The operation shown.

[0032] Optionally, when the first electronic device recognizes that the main object on the video image in the first video changes from the first sound image to the second sound image, the first electronic device can remove the audio signals of the other sound images except the second sound image from the audio contained in the first video. The first electronic device can remove the audio signals of the other sound images except the second sound image from the audio starting from the recognition that the main object changes to the second sound image.

[0033] As can be seen from the above embodiments, the electronic device can identify the main object on the video image, retain the sound of the main object in the audio, and suppress all sounds except the main object in the audio. In this way, the user can quickly eliminate all sounds except the main object in the audio during a video call, so that the sound of the main object can be clearly heard. Alternatively, the user can suppress the sound of other sound images except the main object in the captured video, so that the audio in the captured audio is purer.

[0034] In combination with the first aspect, in some embodiments, a display position of the first audio and video control on the screen is related to a display position of the first audio and video on the video image of the first video.

[0035] The display position of the first audio-visual control can refer to this application Figure 5C The display positions of the volume bar 531A and the mute control 531B are shown. It can be seen that the controls for audio editing can be presented on the video image, and in particular, the controls for editing the sound and image can also be displayed near the sound and image captured in the image. This allows users to perform audio editing operations more easily and intuitively, and improves the user experience of recording audio and video, and editing audio.

[0036] In combination with the first aspect, in some embodiments, before the first electronic device displays the first video image of the first video, the first electronic device receives an operation of opening a camera application and / or starting to record a video, captures an image through an image acquisition device, and captures audio through an audio acquisition device; wherein the image captured by the image acquisition device includes the video image of the first video, and the audio captured by the audio acquisition device includes the first audio; the first electronic device displays the image captured by the image acquisition device in real time. The first video image may be an image displayed in real time by the first electronic device during the shooting process.

[0037] As can be seen from the above embodiments, the user can perform audio editing before starting to shoot a video (i.e., the video pre-recording stage). In this way, when entering the video recording stage, the electronic device can edit the audio collected in the video recording stage according to the editing parameters determined by the audio editing operation in the video pre-recording stage, so that the recorded audio meets the user's needs. Optionally, the user can also perform audio editing while shooting a video.

[0038] Among them, when an audio editing operation (such as audio and video editing operation, sound field editing operation, mixing editing operation, etc.) is detected during the process of recording a video, the electronic device can determine the corresponding audio editing parameters based on the audio editing operation, and use the audio editing parameters to perform corresponding editing on the audio collected from the moment of the audio editing operation.

[0039] In conjunction with the first aspect, in some embodiments, before the first electronic device displays the first video image of the first video, the first electronic device obtains the first video from the memory of the first electronic device; and the first electronic device receives an operation to play the first audio. The first video image may be an image displayed by the first electronic device during video playback.

[0040] From the above embodiments, it can be seen that the user can edit the audio of the video stored in the electronic device. In other words, in addition to editing the audio during the video recording process, the user can also perform secondary audio editing on the recorded video after the video recording is completed.

[0041] In combination with the first aspect, in some embodiments, a first electronic device establishes a communication connection with a listening device, and the first electronic device sends a second audio to the listening device for playback, where the second audio is the first audio after one or more of the following adjustments have been made: sound characteristics of one or more sound images in the first audio have been adjusted, the first sound field of the first audio has been adjusted, and / or mixing parameters of the first audio have been adjusted.

[0042] As can be seen from the above embodiments, during the process of recording a video, the electronic device can mix and output an audio file in real time, and render the audio file in real time, so that the user can monitor the sound effects of the video recording in real time through the audio playback device. This makes it convenient for the user to understand the sound effects of the audio editing in a timely manner, so as to perform audio editing so that the sound effects meet the effects required by the user.

[0043] In combination with the first aspect, in some embodiments, a communication connection is established between the first electronic device and the monitoring device. After the first electronic device adjusts the sound characteristics of the first sound image in the first audio, the first electronic device sends the audio signal of the first sound image with the adjusted sound characteristics to the monitoring device for playback.

[0044] As can be seen from the above embodiments, during the process of recording a video, when the user adjusts the sound characteristics of a single or multiple sound images, the electronic device can only play the audio signal of the adjusted one or more sound images. In this way, the user can only listen to the audio of the edited sound image to determine whether the editing of the sound image is appropriate.

[0045] In combination with the first aspect, in some embodiments, a first electronic device and a second electronic device enter a first video call, and the first electronic device sends a video image and a second audio of the first video to the second electronic device, where the second audio is the audio of the first audio after one or more of the following adjustments: the sound characteristics of one or more sound images in the first audio are adjusted, the first sound field of the first audio is adjusted, and / or the mixing parameters of the first audio are adjusted. The first video image may be an image taken by the first electronic device during the first video call.

[0046] In combination with the first aspect, in some embodiments, a first electronic device receives a second video from a second electronic device; the first electronic device displays a second video image of the second video, the second video image including a fourth sound and image; the first electronic device displays a fifth sound and image control on the screen, the fifth sound and image control being used to adjust the sound characteristics of the fourth sound and image, adjusting the sound characteristics of the fourth sound and image including one or more of the following: adjusting the volume of the fourth sound and image, adjusting the timbre of the fourth sound and image, and / or adjusting the spatial position of the fourth sound and image.

[0047] In combination with the first aspect, in some embodiments, the first electronic device determines a first editing parameter based on an operation of a fifth audio and video control; the first electronic device sends a first editing instruction to a second electronic device, the first editing instruction including a first editing parameter; the first electronic device receives a third video from the second electronic device, the third video including a third audio, wherein the sound features of the fourth audio and video in the third audio are adjusted according to the first editing parameter.

[0048] In combination with the first aspect, in some embodiments, a third electronic device is also connected to the first video call, and the third video is also sent to the third electronic device for playback.

[0049] As can be seen from the above embodiments, the user can edit the audio of the local end during a video call so that the other end of the call can hear the audio edited by the user. In addition, the user can also edit the audio of the other end of the call during a video call so that he and other call ends can hear the audio of the edited call end after being edited by the user. In this way, the user can control the audio of multiple call ends connected to the video call as a host so that the video call can be carried out better. The above embodiments can facilitate users to edit call audio easily and intuitively during a video call, thereby improving the user's video call experience.

[0050] In combination with the first aspect, in some embodiments, the first electronic device displays a second sound field control on a screen, and the second sound field control is used to adjust the second sound field in the audio contained in the video sent from the second electronic device, and adjusting the second sound field includes one or more of the following: adjusting the type of room where the second sound field is located, adjusting the size of the room where the second sound field is located, and / or adjusting the reflective material of the room where the second sound field is located.

[0051] In combination with the first aspect, in some embodiments, the first electronic device displays a second mixing control on the screen, and the second mixing control is used to adjust the mixing parameters of the audio contained in the video sent from the second electronic device, and the mixing parameters include one or more of the following: mixing template, reverberation time, equalizer parameters, the size of the room where the sound field is located, and / or the reflective material of the room where the sound field is located.

[0052] It can be seen from the above embodiments that during a video call, the user can also adjust the sound field and mixing parameters of the other end of the call.

[0053] In combination with the first aspect, in some embodiments, a first electronic device and a second electronic device enter into a first video call, and before the first electronic device displays a first video image of the first video, the first electronic device receives the first video from the second electronic device.

[0054] In a second aspect, the present application provides an audio processing method. Among them, the first electronic device can collect video images and audio. The first electronic device can determine the audio and video information and the sound field environment information based on the video image and the audio. The audio and video information includes but is not limited to: one or more of the audio tracks of the audio and video contained in the audio, the spatial position of the audio and video, and / or the display position of the audio and video on the video image. The sound field environment information may include but is not limited to: one or more of the reverberation time, room type, room size and / or room reflection material. The first electronic device can receive an audio editing operation and determine the audio editing parameters based on the audio editing operation. The audio editing operation includes performing one or more of the following editing operations on the audio: audio and video editing, sound field editing and / or mixing editing. The first electronic device can determine the audio file based on the audio editing parameters, the audio and video information, and the sound field environment information.

[0055] The above-mentioned sound and image editing may include adjusting the sound characteristics of the sound and image, such as adjusting the volume of the sound and image, adjusting the timbre of the sound and image, adjusting the spatial position of the sound and image, etc. The above-mentioned sound field editing may include but is not limited to: adjusting the type of room where the sound field is located, adjusting the size of the room where the sound field is located, and / or adjusting the reflective material of the room where the sound field is located. The above-mentioned mixing editing may include adjusting mixing parameters. The above-mentioned mixing parameters may include but are not limited to: mixing template, reverberation time, equalizer parameters, the size of the room where the sound field is located, and / or the reflective material of the room where the sound field is located.

[0056] The above audio file may contain the audio signal of the sound field and each sound image, as well as metadata. The metadata may include the spatial position of the sound image, sound field environment information, and other data used to enable the audio signal to be correctly rendered.

[0057] It can be seen from the above method that the first electronic device can perform sound object track division and sound field analysis on the audio, thereby providing multi-track editing capabilities for the audio. In this way, the user can make personalized adjustments to the sound characteristics, sound field, and mixing parameters of the sound objects in the audio in real time during the process of recording audio and video by the electronic device to obtain audio that meets personal needs. In addition, the user can also make personalized adjustments to the sound characteristics, sound field, and mixing parameters of the sound objects in the audio after the electronic device completes the audio and video recording. The above method can improve the user experience of recording audio and video.

[0058] In conjunction with the second aspect, in some embodiments, the first electronic device receives an operation to end video recording, and can save the video image and audio collected from the start of video recording to the end of video recording as the first video. When receiving an operation to play the first video, the first electronic device can obtain the video image of the first video and the first audio file. The first audio file may contain audio signals of the sound field and each sound image, as well as metadata. The metadata may contain data such as the spatial position of the sound image, sound field environment information, etc. for enabling the audio signal to be correctly rendered. The first electronic device receives an audio editing operation, and can determine audio parameters according to the audio editing operation. The audio editing operation may include one or more of the following editing operations on the audio: sound image editing, sound field editing, and / or mixing editing. The first electronic device can edit the first audio file according to the audio editing parameters to obtain a second audio file. The first electronic device can store the second audio file as the audio file of the first video.

[0059] The above-mentioned editing of the first audio file may include: editing the audio signal of the sound field in the first audio file (to adjust the sound field), editing the audio signal of one or more sound images in the first audio file (to adjust the sound characteristics of the sound image), and editing the metadata in the first audio file (to ensure that the adjusted sound image and / or audio signal of the sound field can be correctly rendered and played after adjustment).

[0060] It can be seen from the above embodiments that the user can edit the existing video in the first electronic device to change the playing effect of the audio in the video.

[0061] In conjunction with the second aspect, in some embodiments, the first electronic device may determine the position of the acoustic object displayed in the video image based on the spatial position of the acoustic object and the video image. The first electronic device may use an image processing algorithm to identify and track one or more targets in the video image, and determine which target in the video image emits various sounds in the environment based on the type of the target in the video image, the position of the target, and the spatial position of the acoustic object, thereby determining which acoustic object the target in the video image corresponds to.

[0062] The above determination of the display position of the sound object in the image can facilitate the user to intuitively perform audio editing operations on one or more sound objects on the video image. Then, the electronic device can process the corresponding audio track according to the sound object selected by the user to be edited on the video image to meet the user's audio editing needs. The above audio editing operation is simple and intuitive, which can improve the user's experience in recording audio and video, and editing audio.

[0063] In conjunction with the second aspect, in some embodiments, the audio file determined by the first electronic device according to the audio editing parameters, the audio-visual information, and the sound field environment information may include audio files in multiple formats. For example, an audio file in a stereo format, an audio file in a surround sound format, and an audio file in a three-dimensional sound format. In this way, rendering and playing audio files in different formats can enable users to experience different sound effects.

[0064] In conjunction with the second aspect, in some embodiments, the first electronic device may perform one or more of the following encodings on the audio file: channel encoding, sound object encoding, HOA encoding, and metadata encoding. Then, the first electronic device may send the encoded audio file to other devices. The above encoding can reduce information redundancy in the audio file and improve the transmission efficiency of the audio file.

[0065] In a third aspect, the present application provides an audio processing method. The method can be applied to a communication system including a first electronic device and a second electronic device. The first electronic device and the second electronic device access a first call. After the first electronic device collects video images and audio, it can edit the audio according to the received audio editing operation to obtain a first audio file. The first audio file may include audio signals of the sound field and each sound image, as well as metadata. The metadata may include data such as the spatial position of the sound image, sound field environment information, etc. for enabling the audio signal to be correctly rendered. The first electronic device can render the audio file to obtain the first audio data. The first electronic device can send the first audio data to the second electronic device. The second point itself can play the audio edited by the first electronic device side according to the first audio data. The first electronic device receives the operation of editing the audio on the second electronic device side and determines the audio editing parameters. The first electronic device can send an audio editing instruction to the second electronic device. The audio editing instruction may include the above-mentioned audio editing parameters. The second electronic device can edit the collected audio according to the audio editing instruction to obtain a second audio file. The second electronic device renders the second audio file to obtain the second audio data. The second electronic device can send the second audio data to the first electronic device. The first electronic device can play the audio edited by the second electronic device side according to the second audio data.

[0066] It can be seen from the above method that the user can edit the audio of the local end during the video call so that the other end of the call can hear the audio edited by the user. In addition, the user can also edit the audio of the other end of the call during the video call so that he can hear the audio of the edited other end of the call after the user edits it. In this way, the user can control the audio of multiple other ends of the call connected to the video call as a host so that the video call can be carried out better. The above embodiment can facilitate users to edit the call audio easily and intuitively during the video call, thereby improving the user's video call experience.

[0067] In conjunction with the third aspect, in some embodiments, the communication system further includes a third electronic device. The third electronic device also accesses the first video call. The second electronic device may also send the second audio data to the third electronic device. The third electronic device may play the audio edited by the second electronic device according to the second audio data.

[0068] It can be seen from the above embodiments that the user of the first electronic device can edit the audio of the second electronic device so that other call ends can listen to the audio of the call partner edited by the user of the first electronic device.

[0069] In combination with the third aspect, in some embodiments, the first electronic device may receive a video image and an audio file from a second electronic device. The audio file may contain the sound field of the call counterpart and audio signals of each sound and image, as well as metadata. When an operation to edit the audio of the call counterpart is received, the first electronic device may edit the audio file to obtain a new audio file. The first electronic device may render the new audio file to obtain audio data. Then, the first electronic device may use the audio data obtained by the rendering to play the audio of the call counterpart. In this way, the user on the first electronic device side can listen to the audio of the call counterpart that has been edited by himself.

[0070] In conjunction with the third aspect, in some embodiments, the communication system may further include a server. The server may be used to enable the first electronic device and the second electronic device to access the first video call. When receiving an operation to edit the audio of the call counterpart, the first electronic device may send an audio editing instruction to the server. The audio editing instruction may include audio editing parameters for audio editing. The server may also receive a video image and an audio file from the call counterpart. The audio file may include an audio signal of the call counterpart sound field and each sound image, as well as metadata. The server may edit the audio file of the call counterpart according to the audio editing parameters of the first electronic device to obtain a new audio file. Then, the server may send the new audio file to the first electronic device, and the first electronic device may render and play the new audio file. Alternatively, the server may render the new audio file to obtain audio data, and hand over the audio data to the first electronic device for playback. In this way, the user on the first electronic device side can listen to the audio of the call counterpart edited by himself.

[0071] From the above embodiments, it can be known that the first electronic device can support the local user to edit the audio of the other end of the call. The device that specifically edits the audio file can be the first electronic device, or can be the other end of the call (such as the second electronic device), or can be a server.

[0072] In a fourth aspect, the present application provides an electronic device, which may include a camera, a microphone array, a memory, and a processor. The camera may be used to capture images. The microphone array may be used to collect audio. The memory may be used to store a computer program. The processor may be used to call the computer program so that the electronic device executes any possible implementation method in the first aspect or the second aspect.

[0073] In a fifth aspect, the present application provides a computer-readable storage medium, comprising instructions, which, when executed on an electronic device, enables the electronic device to execute any possible implementation method in the first aspect or the second aspect.

[0074] In a sixth aspect, the present application provides a computer program product, which may include computer instructions. When the computer instructions are run on an electronic device, the electronic device executes any possible implementation method in the first aspect or the second aspect.

[0075] In a seventh aspect, the present application provides a chip, which is applied to an electronic device, and the chip includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute any possible implementation method as in the first aspect or the second aspect.

[0076] It can be understood that the electronic device provided in the fourth aspect, the computer-readable storage medium provided in the fifth aspect, the computer program product provided in the sixth aspect, and the chip provided in the seventh aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 is a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application;

[0078] Figure 2 is a software structure block diagram of an electronic device 100 provided in an embodiment of the present application;

[0079] Figure 3 is a structural schematic diagram of another electronic device 100 provided in an embodiment of the present application;

[0080] Figure 4A to Figure 4C It is an architecture diagram of some communication systems provided in the embodiments of the present application;

[0081] Figure 5A to Figure 5M It is a schematic diagram of some audio editing scenarios provided in the embodiments of the present application;

[0082] Figure 6A to Figure 6G Schematic diagrams of other audio editing scenarios provided in embodiments of the present application;

[0083] Figure 7A to Figure 7E Schematic diagrams of other audio editing scenarios provided in embodiments of the present application;

[0084] Figure 8A to Figure 8C Schematic diagrams of other audio editing scenarios provided in embodiments of the present application;

[0085] Figures 9A to 9C Schematic diagrams of other audio editing scenarios provided in embodiments of the present application;

[0086] FIG. 10A to FIG. 10D Schematic diagrams of other audio editing scenarios provided in embodiments of the present application;

[0087] Figures 11A to 11H Schematic diagrams of other audio editing scenarios provided in embodiments of the present application;

[0088] Figures 12A to 12C Schematic diagrams of other audio editing scenarios provided in embodiments of the present application;

[0089] Fig.13A and Fig. 13B Schematic diagrams of other audio editing scenarios provided in embodiments of the present application;

[0090] Fig.14 is a flow chart of an audio processing method provided by an embodiment of the present application;

[0091] Fig.15 is a flowchart of another audio processing method provided by an embodiment of the present application;

[0092] Fig.16 It is a flowchart of another audio processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in a "or" relationship.

[0094] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0095] In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0096] The term "user interface (UI)" in the following embodiments of the present application refers to the medium interface for interaction and information exchange between an application (APP) or an operating system (OS) and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that can be recognized by the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0097] The present application provides an audio processing method. Among them, the electronic device can collect video images and audio, and determine the sound object information, sound field environment information and other information therein according to the video images and audio. The sound object information may include one or more of the tracks of each sound object separated from the audio and / or the spatial position of the sound object. The sound field environment information may include one or more of the reverberation time, room type, room size and / or room reflection material. Based on the above information, the electronic device can provide controls for editing the audio of the above sound object on the user interface, and / or controls for editing the above sound field. The user can select one or more sound objects for audio editing. For example, the user can adjust the volume of the sound object, change the timbre of the sound object, move the spatial position of the sound object, etc. The user can also choose to edit the sound field, such as replacing the sound field environment, etc. The electronic device can determine the editing parameters according to the user's operation of editing the sound object and / or the sound field, so as to process the collected audio using the editing parameters. The electronic device can output audio files in multiple formats (such as stereo, ambisonic, surround sound, etc.) so that the processed audio can be rendered on multiple types of audio playback devices. When the processed audio is rendered, the sound effect heard by the user can achieve the effect adjusted by the user.

[0098] It can be seen from the above method that the electronic device can perform sound object track division and sound field analysis on the audio, thereby providing multi-track editing capabilities for the audio. In this way, the user can perform personalized editing of the sound track and / or sound field of the sound object in the audio in real time during the process of recording audio and video by the electronic device to obtain audio that meets personal needs. In addition, the user can also perform personalized editing of the sound track and / or sound field of the sound object in the audio after the electronic device completes the audio and video recording. The above method can improve the user experience of recording audio and video.

[0099] Terminology

[0100] To facilitate understanding, some concepts involved in this application are introduced below.

[0101] 1. Channel

[0102] Channels can refer to audio signals collected or played back at different spatial locations when recording or playing sound. The number of channels can be the number of audio input devices (such as microphones) when recording sound, or the number of audio output devices (such as speakers) when playing sound.

[0103] Based on the number of channels, there can be mono, dual channels, 5.1 channels, 7.1 channels, etc. The more channels there are, the stronger the stereo and presence of the audio during playback.

[0104] 2. Sound Object

[0105] A sound object can represent an object that makes a sound, such as a person, animal, musical instrument, or vehicle. The sound of a sound object can be perceived as a whole sound or a sound source that is independent of the environment. A sound object can also be called a sound image or a sound source.

[0106] 3. Audio Track

[0107] An audio track may refer to a location for placing an audio element. The above audio element may include a sound object. One sound object may correspond to one audio track. It is understandable that during the process of recording an audio segment, there may be multiple sound objects in the environment making sounds. The electronic device may collect the sounds of these multiple sound objects and generate an audio file. That is, one audio file may contain multiple audio tracks.

[0108] 4. Sound field

[0109] The sound field can refer to the area in the medium where sound waves exist.

[0110] During the propagation process, sound will be reflected by obstacles such as the walls and ceilings of buildings. For the sounds of various sound objects, we can feel the sounds heard directly, as well as the reflected sounds or echoed sounds. The characteristics of reflected and echoed sounds can be affected by the shape, size, and material of the building. All the sounds directly transmitted to the ears, reflected sounds, echoed sounds, etc., together give us the auditory experience of being in a certain location.

[0111] Therefore, the sound field may correspond to sound field environment information (such as reverberation time, room type, room size, room reflective material, etc.). Electronic devices can adjust the audio effect by changing the above sound field environment information. The sound field environment information can also be called sound field parameters and the like.

[0112] 5. Metadata

[0113] Metadata can be used to describe audio signals so that the audio signals can be correctly rendered, processed or distributed. Electronic devices can generate corresponding metadata when recording audio.

[0114] The metadata of an audio file may include the definition of different tracks in the audio file. That is, the metadata may include information such as the spatial position and / or intensity of the sound object. The metadata in an audio file may also include sound field environment information. When playing back, the electronic device may map the sound object (i.e., the track) to one or more speakers or perform binaural rendering to headphones for playback based on the metadata to achieve the desired spatial audio effect.

[0115] In some embodiments, the format of the metadata may comply with a standard, such as ITU-R BS.2076. Under standard ITU-R BS.2076, the metadata may include one or more of the following elements: audioTrackFormat, audioStreamFormat, audioChannelFormat, audioBlockFormat, audioPackFormat, audioObject, audioContent, audioPraogramme, audioTrackUID, and / or audioFormatExtended.

[0116] audioTrackFormat can represent data related to the audio track and can be used to describe the format of the data so that the renderer can correctly decode the signal. audioTrackFormat can contain the ID of the audio track, the name of the audio track, description information of the format, etc.

[0117] audioStreamFormat can be used to successfully decode the audio track data required for the combination of audio tracks (i.e., audio streams). An audio stream can be a combination of several audio tracks (or a track) used to render a channel, sound object, higher order ambisonics (HOA) component or package. audioStreamFormat can contain the audio stream ID, audio stream name, format description information, etc.

[0118] audioChannelFormat can represent channel-related data. audioChannelFormat can include the name of the channel (such as left channel, right channel, etc.), the ID of the channel, the type of the channel, etc.

[0119] audioBlockFormat can be a subdivision of audioChannelFormat in the time domain. That is, audioBlockFormat (or audio block) can represent a single sequence of audioChannelFormat samples with fixed parameters within a specified time interval. audioChannelFormat can be further subdivided into one or more audioBlockFormat in the time domain. audioBlockFormat can contain the ID of the audio block, the start time and duration of the audio block.

[0120] audioPackFormat can represent an audio package that combines one or more audioChannelFormats. audioPackFormat can contain the ID of the audio package, the name of the audio package, the type of channel, the importance of the audio package, etc.

[0121] The above ITU-R BS.2076 is only an example standard of this application, and the format of metadata can also be defined according to other standards. This embodiment of the application does not limit this.

[0122] 6. Stereo

[0123] Stereo may refer to an audio format in which two channels are used to carry audio signals with a certain correlation, which are usually reproduced through two symmetrical speakers or headphones located in front of the listener, giving the listener a feeling of a wider sound field.

[0124] 7. Surround Sound

[0125] Surround sound may refer to an audio format in which multiple channels are used to carry multi-channel audio signals constituting complete audio content, and the signals are reproduced through multiple speakers located at the ear height level of the listener to surround the listener, giving the listener a feeling of being surrounded by a sound field.

[0126] 8. 3D sound

[0127] 3D sound can represent an audio format and can include: channel-based 3D sound, object-based 3D sound, and sound field-based 3D sound.

[0128] Among them, channel-based 3D sound can refer to playing different sounds in different channels to achieve a stereo effect. Channel-based 3D sound can supplement the sound information in the space by adding channels, giving the listener an immersive sound field feeling.

[0129] Object-based 3D sound can refer to the decomposition of sound into multiple independent audio objects, each of which has its own properties such as position, direction and volume. By mixing and processing these objects, a stereo effect is achieved. Object-based 3D sound can be positioned at any point in space by assigning spatial coordinates to sound objects. Object-based 3D sound is not restricted by sound reproduction conditions (such as the number of speakers) when rendering and playing.

[0130] 3D sound based on sound field can refer to the stereo effect achieved by placing multiple speakers in the space and emitting sound in different directions and angles. 3D sound based on sound field can reproduce the sound in the space by recording the sound pressure in the 3D space, and its core underlying algorithm can include HOA.

[0131] In some embodiments, the three-dimensional sound may also include three-dimensional sound implemented based on the channel+object+sound field technology.

[0132] It can be seen that three-dimensional sound can produce a three-dimensional sense of orientation on a speaker array or headphones, making the listener feel that the sound source they hear is in different 3D positions in space.

[0133] Three-dimensional sound can also be called spatial audio or 3D audio.

[0134] 9. Speaker Rendering

[0135] Speaker rendering may refer to rendering an audio signal through a set of speakers.

[0136] For example, in a scene where speakers are used to render three-dimensional sound, the audio signals of different sound sources are mapped to the speaker positions through the reasonable placement of the speaker array (for example, using the speaker array conversion matrix to perform mapping operations with the positions of the sound sources) to achieve a sense of space and orientation of the played sound signals.

[0137] When the audio playback device is a multi-speaker device, such as a car computer, a large screen, multiple speakers in a home theater, etc., speaker rendering can be used.

[0138] 10. Binaural Rendering

[0139] Binaural rendering can refer to rendering the audio signal through headphones or stereo external speakers (such as mobile phones, tablets, laptops, etc.).

[0140] For example, in a scene using binaural rendering of 3D sound, the head-related transfer function (HRTF) is used to perform time-domain convolution with the audio signal carrying the orientation information, distance attenuation compensation is performed, and reverberation is added to imitate natural sound waves, so that the sound source seems to come from a point in the three-dimensional space, so that the user feels that the sound source heard is in different 3D positions in the space.

[0141] Equipment structure and communication system

[0142] The software and hardware structure of the electronic device 100 involved in this application is introduced below.

[0143] Figure 1 The hardware structure diagram of the electronic device 100 provided in the embodiment of the present application is shown as an example

[0144] like Figure 1As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0145] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0146] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0147] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0148] The processor 110 may also be provided with a memory for storing instructions and data. In some examples, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0149] In the present application, a computer program is stored in the memory, which is used to enable the controller or processor to implement the audio processing method of the present application through an interface or protocol.

[0150] Exemplarily, the computer program stored in the memory can be used to: process the collected video images and audio to determine the sound object information (including determining the position of the sound object displayed in the video image, the sound track of the sound object in the audio, and the spatial position of the sound object) and the sound field environment information. Among them, the sound track of the above-mentioned sound object can be obtained by processing using the sound object track separation technology. The spatial position of the above-mentioned sound object can be obtained by using a positioning algorithm to process the digital sound signal after performing analog-to-digital conversion on the analog sound signal in the collected environment to obtain a digital sound signal. The position of the above-mentioned sound object displayed in the video image can be obtained by processing using an image recognition algorithm. The above-mentioned sound field environment information can be obtained by combining audio and video images for joint modeling.

[0151] Optionally, the computer program stored in the memory can also be used to: determine editing parameters based on the user's editing operations on the sound object and / or sound field, and use the editing parameters to process the collected audio; encode the audio (such as channel encoding, sound object encoding, HOA encoding, metadata encoding, etc.); output the processed audio information such as the audio track and the spatial position of the sound object into an audio file in one or more audio formats (such as stereo, surround sound, three-dimensional sound, etc.).

[0152] The USB interface 130 is an interface that complies with USB standard specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 may be used to connect a charger to charge the electronic device 100, or may be used to transmit data between the electronic device 100 and a peripheral device. It may also be used to connect headphones to play audio through the headphones.

[0153] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 charges the battery 142, it can also power the electronic device through the power management module 141.

[0154] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0155] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0156] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0157] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0158] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0159] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering.

[0160] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

[0161] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0162] The ISP is used to process data fed back by the camera 193. For example, when taking a photo, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, and the light signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye.

[0163] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0164] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0165] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0166] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0167] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0168] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0169] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some examples, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110. The speaker 170A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. The headphone jack 170D is used to connect wired headphones.

[0170] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. In some embodiments, the electronic device 100 may be provided with multiple microphones 170C (such as two, three, etc.) to achieve functions such as audio collection, noise reduction, and determination of the spatial position of sound objects. Microphone 170C may be either an omnidirectional microphone or a directional microphone. Microphone 170C may be a microphone built into the electronic device 100. Alternatively, microphone 170C may also be a microphone independent of the electronic device 100 and having a communication connection with the electronic device 100, such as a wireless microphone, etc.

[0171] The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a gravity sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0172] Among them, the gyroscope sensor can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor. The electronic device 100 can determine the offset angle of the electronic device 100 through the gyroscope sensor.

[0173] The acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes) of the electronic device 100. In some embodiments, the acceleration sensor can be used to identify the posture of the electronic device 100, and can be applied to applications such as horizontal and vertical screen switching and pedometers.

[0174] The gravity sensor can be used to determine the tilt angle of the electronic device 100 relative to the horizontal plane. In some embodiments, the screen state of the electronic device 100 can be determined by the gravity sensor, so that the screen is adjusted to keep it horizontal.

[0175] In some embodiments, the electronic device 100 may determine the moving distance of the electronic device 100 within a period of time by using an acceleration sensor and a gravity sensor.

[0176] The buttons 190 include a power button, a volume button, etc. The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power change, messages, missed calls, notifications, etc.

[0177] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some examples, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0178] The electronic device 100 may be a mobile phone, a tablet computer, a smart watch, a television, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present application does not limit the specific type of the electronic device.

[0179] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the electronic device 100 is exemplified.

[0180] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0181] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. The system is divided into four layers, from top to bottom: application layer, application framework layer, Android Runtime and system library, and kernel layer.

[0182] The application layer can include a series of application packages.

[0183] like Figure 2As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, short message, and other applications. Among them, the camera application can be used to provide functions such as taking pictures and video recording. During the video recording process, the electronic device 100 can capture video images through the camera and collect audio through the microphone (or other audio input device). In addition, the electronic device 100 can display the captured video images on the screen in real time. The call application can be used to provide voice call and video call functions. In the video call scenario, the electronic device 100 can capture video images through the camera and collect audio through the microphone. The electronic device 100 can send the captured video images and audio to the video call peer end, and receive the video images and audio from the video call peer end. The electronic device 100 can display the video images captured by the electronic device 100 and the video images sent by the video call peer end on the screen in real time, and play the audio from the video call peer end. In some embodiments, the above-mentioned video call function can allow multiple people to access a call at the same time, such as a video call accessed by 3, 4 or more people.

[0184] The electronic device 100 may also include more or fewer application programs. For example, the electronic device 100 may also include a video conferencing application. The functions provided by the video conferencing application may refer to the above-mentioned video call function.

[0185] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0186] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, an activity manager, and the like.

[0187] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0188] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0189] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0190] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0191] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0192] The notification manager enables applications to display notification information in the status bar (such as the pull-down notification bar). It can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0193] The activity manager is responsible for managing activities, starting, switching, and scheduling components in the system, as well as managing and scheduling applications. The activity manager can be called by upper-level applications to open corresponding activities.

[0194] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system.

[0195] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0196] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0197] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0198] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0199] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0200] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0201] A 2D graphics engine is a drawing engine for 2D drawings.

[0202] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0203] The embodiment of the present application does not limit the operating system of the electronic device 100. For example, the operating system of the electronic device 100 may also be apple OS, etc.

[0204] Figure 3 It is a schematic diagram of the structure of another electronic device 100 according to an embodiment of the present application.

[0205] like Figure 3 As shown, the electronic device 100 may include a collection unit 310, an information extraction unit 320, an editing unit 330, a rendering unit 340, and a coding and decoding unit 350. Among them:

[0206] The acquisition unit 310 may include an audio acquisition module 311 and an image acquisition module 312 .

[0207] The audio acquisition module 311 can be used to collect sound signals in the environment (ie, collect audio). The audio acquisition module 311 can specifically be a sound pickup array, for example, a microphone array. In some embodiments, the audio collected by the audio acquisition module 311 can be spatial audio.

[0208] The image acquisition module 312 can be used to acquire images. The image acquisition module 312 can be a camera, for example, a front camera, a rear camera, etc. of the electronic device 100. The electronic device 100 can continuously acquire images over a period of time through the image acquisition module 312, and save the acquired images in series as a video according to the order of acquisition. The above-mentioned images saved as videos can also be called video images.

[0209] In some embodiments, the camera may include a depth camera, so as to locate the photographed person or object according to the image captured by the depth camera. The embodiment of the present application does not limit the type of the camera.

[0210] In some embodiments, the above-mentioned audio acquisition module 311 and / or image acquisition module 312 may also be modules included in other devices independent of the electronic device 100. For example, the electronic device 100 may establish a communication connection with a video image acquisition device and an audio acquisition device. The video image acquisition device may include an image acquisition module. The audio acquisition device may include an audio acquisition module. The video image acquisition device and the audio acquisition device may simultaneously acquire data and send the data to the electronic device 100. The electronic device 100 may receive the video image acquired by the video image acquisition device and the audio acquired by the audio acquisition device. Then, the electronic device 100 may provide the user with the functions of sound object and sound field editing according to the audio processing method provided in the present application, and process the above-mentioned received audio according to the audio editing operation of the user.

[0211] The information extraction unit 320 may include an acoustic object determination module 321 , a spatial position determination module 322 , and a sound field information determination module 323 .

[0212] The information extraction unit 320 can obtain the audio and video images collected by the collection unit 310.

[0213] The acoustic object determination module 321 can be used to perform acoustic object sub-tracking to separate the audio tracks of one or more acoustic objects from the audio. In a possible implementation, when the collected audio is multi-channel, the audio signal of an acoustic object can exist in one or more channels. The acoustic object determination module 321 can determine the spectral similarity of each channel in the audio and obtain a set of spectral similarities. Based on the set of spectral similarities, the acoustic object determination module 321 can perform channel grouping to obtain a set of channel groups. A channel group can be associated with an acoustic object. The above-mentioned channel grouping method may include but is not limited to clustering algorithms such as hierarchical method, density method, grid method, etc. Based on the above-mentioned set of channel groups, the acoustic object determination module 321 can generate a probability vector associated with each channel group for each frame contained in the audio. The probability vector of a frame can be used to indicate the probability value that this frame belongs to a channel group. The acoustic object determination module 321 can generate a probability matrix corresponding to each channel group by aggregating the associated probability vectors across frames, and perform acoustic object synthesis between channel groups across frames according to the probability matrix, thereby separating the tracks of each acoustic object in the audio. The specific method of performing acoustic object track separation by the acoustic object determination module is not limited in the embodiment of the present application.

[0214] In some embodiments, the acoustic object determination module 321 may also be used to identify noise in the environment. For example, the noise may include but is not limited to the sound of a television, a car horn, a dog barking, applause, and the like.

[0215] It should be noted that the audio collected by the audio collection module 311 may include not only the audio tracks of various sound objects but also a sound bed. The sound bed may represent the audio signal of the sound field and may be used to carry the ambient sound.

[0216] The spatial position determination module 322 can be used to determine the spatial position of the sound object and the position of the sound object displayed in the video image. The spatial position of the sound object can be used to describe the position of the sound object in the audio relative to the listener or the audio acquisition device in the three-dimensional space. Among them, the electronic device 100 can determine where the audio signal of the sound object should be played in the three-dimensional space according to the spatial position of the sound object, so that the listener can experience the corresponding sense of space when listening to the audio. In other words, the spatial position of the sound object can indicate that when the audio is played, the listener can distinguish the position of the sound object. The electronic device 100 can change the spatial position of the sound object by editing the audio signal of the sound object. When the spatial position of the sound object changes, the listener can distinguish that the sound of the sound object is emitted from the spatial position after the above change when listening to the audio.

[0217] In some embodiments, in audio technology based on spatial audio, the spatial position of a sound object may represent the position of the sound object (or sound source) in a virtual three-dimensional space. The sound object may be freely placed at any position in the virtual three-dimensional space and may also move along a preset trajectory. The spatial position of a sound object may also be referred to as a sound image position or the like.

[0218] The spatial position of the sound object can be represented by spatial coordinates. Among them, the spatial position determination module 322 can establish a spatial coordinate system, and describe the relative position relationship between the sound object and the listener through the spatial coordinates of the sound object and the spatial coordinates of the listener. The above-mentioned spatial coordinate system can be a three-dimensional coordinate system, which can represent a virtual three-dimensional space. The embodiment of the present application does not limit the origin of the spatial coordinate system and the establishment method. The above-mentioned spatial coordinates of the listener can be, for example, the coordinates of the position of the electronic device 100, or the coordinates of the position of the headphones connected to the electronic device 100. The above-mentioned spatial coordinates of the listener in the embodiment of the present application are not limited.

[0219] In a possible implementation, the spatial position determination module 322 can process the audio (such as analog-to-digital conversion, Fourier transform, etc.) to obtain the spectral characteristics of each sound signal. The spatial position determination module 322 can determine the spatial position of the sound object based on the above-mentioned spectral characteristics. Among them, the relative position relationship between the sound object and the listener may change with sound, and the spatial position determination module 322 can determine the spatial position of the sound object in real time. The spatial position determination module 322 can also determine the spatial position of each sound object through other positioning algorithms. The embodiment of the present application is not limited to this.

[0220] The spatial position determination module 322 can determine the position of the sound object displayed in the video image according to the spatial position of the sound object and the video image. In a possible implementation, the spatial position determination module 322 can use an image processing algorithm to identify and track one or more targets in the video image. The spatial position determination module 322 determines which target in the video image emits various sounds in the environment according to the type of target in the video image, the position of the target, and the spatial position of the sound object, thereby determining which sound object the target in the video image corresponds to. For example, the spatial position of sound object 1 indicates that sound object 1 is located in front of the left of the listener. The video image contains target 1, and target 1 is located on the left side of the video image. In addition, the sound of sound object 1 is a male voice. Target 1 is also male. Then, the spatial position determination module 322 can determine that target 1 in the video image is sound object 1, so that target 1 can be associated with the sound track of sound object 1.

[0221] The above determination of the display position of the sound object in the image can facilitate the user to intuitively perform audio editing operations on one or more sound objects on the video image. Then, the electronic device 100 can process the corresponding audio track according to the sound object selected by the user to edit on the video image to meet the user's audio editing needs. The above audio editing operation is simple and intuitive, which can improve the user's experience in recording audio and video, and editing audio.

[0222] Optionally, when determining the position of the acoustic object displayed in the video image, the spatial position determination module 322 may also correct the spatial position of the acoustic object based on the video image to obtain a more accurate spatial position of the acoustic object.

[0223] In some embodiments, the camera has a limited shooting angle, and the video image may only contain a portion of the sound object. For example, when only the rear camera is turned on for shooting, the electronic device 100 may only be able to capture the sound objects within a certain range in front of the user, but not the sound objects behind the user. When it is determined that the sound object is not displayed in the video image, the electronic device 100 may display a control on the user interface for representing the above-mentioned sound object that is not captured. In this way, in addition to being able to edit the audio of the captured sound object, the user can also edit the audio of the sound object that is not captured.

[0224] The above method for determining the spatial position of the sound object and the position of the sound object displayed in the video image is only an exemplary description of the present application and should not be construed as limiting the present application. The spatial position determination module 322 can also use other methods to determine the spatial position of the sound object and the position of the sound object displayed in the video image.

[0225] The sound field information determination module 323 can be used to determine the sound field environment information. The sound field environment information may include one or more of the following: reverberation time, room type, room size, room reflective material, etc. In a possible implementation, the sound field information determination module 323 can be combined with audio and video images for joint modeling to determine the sound field environment information in the audio and video recording scene. The embodiment of the present application does not limit the method for determining the above-mentioned sound field environment information.

[0226] The information extraction unit 320 may send the audio track, spatial position, sound field environment information and other contents of the sound object to the editing unit 330 .

[0227] The editing unit 330 may include a sound object editing module 331 , a sound field environment editing module 332 , and a sound mixing module 333 .

[0228] The sound object editing module 331 can be used to edit the sound track of the sound object according to the user's editing operation on the sound object on the user interface. The above-mentioned editing operation on the sound object may include but is not limited to: volume adjustment operation, timbre adjustment operation, spatial position movement operation, etc. The above-mentioned timbre adjustment may include adjusting the timbre of a character and adjusting the timbre of a musical instrument. This can facilitate the user to beautify the human voice or the sound of a musical instrument and increase the interest of the user in audio editing. The method for editing the sound track to achieve the corresponding effect of the above-mentioned editing operation is not limited in the embodiment of the present application.

[0229] In some embodiments, the sound object editing module 331 may include a volume adjustment module 331A, a timbre adjustment module 331B, a spatial position adjustment module 331C, and other modules for editing the sound object. The volume adjustment module 331A may be used to adjust the volume of the sound object. The timbre adjustment module 331B may be used to adjust the timbre of the sound object. The spatial position adjustment module 331C may be used to adjust the spatial position of the sound object.

[0230] The sound field environment editing module 332 can be used to adjust the sound field environment information according to the user's editing operation on the sound field on the user interface to achieve the effect of adjusting the sound field. When the above sound field environment information is adjusted, the sound bed in the audio also changes accordingly. For example, if the room where the sound field is located is a theater, the user can change the room type from theater to recording studio. In this way, even if the user's real environment is not a recording studio, the user can listen to the sound effects that can be heard in a recording studio, thereby achieving an immersive experience of "traveling through time and space".

[0231] In some embodiments, the sound field environment editing module 332 may include a sound field strength adjustment module 332A, a sound field replacement module 332B, and other modules for editing the sound field. Among them, the sound field strength adjustment module 332A can be used to adjust the sound intensity in the sound field. The sound field replacement module 332B can be used to adjust the sound field type (also called room type), for example, replacing the sound field type from a theater to a recording studio.

[0232] The mixing module 333 can be used for mixing, integrating sounds from multiple sources (i.e., the sound bed and multiple tracks in the audio) into a stereo track or a mono track. When mixing, the electronic device 100 can adjust and balance the relative levels (i.e., volume) of each track and the sound bed, and can perform various audio processing such as equalization and compression on a single track or track group. In some embodiments, the electronic device 100 can provide a professional mixing mode, an intelligent mixing mode, and a template mixing mode for users to perform mixing and editing operations.

[0233] Among them, in the professional mixing mode, the electronic device 100 can provide the ability to edit one or more parameters such as sound field environment information, equalizer (EQ), compressor, delay, etc. The mixing module 333 can perform mixing according to the user's mixing editing operation. In this way, the user can customize and adjust parameters such as sound field environment information, EQ, compressor, delay, etc. to configure a mixing effect that meets personal needs.

[0234] In the intelligent mixing mode, the mixing module 333 can comprehensively process the sound track, spatial position and sound field environment information of the sound object through the mixing algorithm to generate matching sound field environment information, EQ parameters, compressor parameters, delay parameters, etc., so as to achieve the effect of beautifying the sound effect. In this way, the user can realize one-click intelligent mixing through the intelligent mixing control corresponding to the intelligent mixing mode.

[0235] In the template mixing mode, the electronic device 100 can provide a variety of mixing templates for users to choose from. For example, the above-mentioned mixing templates may include but are not limited to: a beautiful sound template and an ethereal template. Among them, the beautiful sound template can focus on adjusting the sound object so that the sound effect of the sound object is more beautiful, clear, and three-dimensional. The ethereal template can focus on adjusting the sound field so that the sound field is wider. The mixing module 333 can perform corresponding mixing processing according to the mixing template selected by the user. The specific content of the above-mentioned mixing template is not limited in the embodiment of the present application.

[0236] Optionally, the audio mixing module 333 may also be used to recommend one or more audio mixing templates based on the audio.

[0237] It should be noted that the mixing module 333 can receive the sound bed and the audio track in the audio, and then perform mixing based on the received sound bed and the audio track. The sound bed received by the mixing module 333 can be the sound bed edited by the sound field environment editing module 332. The audio track received by the mixing module 333 can be the audio track edited by the sound object editing module 331.

[0238] In some embodiments, the mixing module 333 can output an audio file. The audio file may include an audio signal and metadata. The output audio signal may include audio signals of various sound objects and audio signals of the sound field. The output metadata may be used to describe the audio signal (such as describing the audio track corresponding to the sound object, the spatial position of the sound object, the volume of the sound object, the timbre of the sound object, the reverberation time, the room type, the room size, the room reflection material, etc.), so that the audio signal can be rendered with the corresponding spatial audio effect.

[0239] In some embodiments, the audio mixing module 333 can output audio files in multiple formats. For example, the format of the audio file may include but is not limited to stereo, surround sound, three-dimensional sound, etc. The above-mentioned audio files in different formats can be used to play in different audio playback devices. For example, an audio file in a stereo format can be used to play in headphones or a sound array comprising two speakers. For another example, an audio file in a 5.1-channel format in surround sound can be used to play in a sound array comprising 5 speakers (two left and right front speakers, two left and right rear speakers, and a center speaker) and a subwoofer. The embodiment of the present application does not limit the implementation method of outputting the above-mentioned audio files in multiple formats.

[0240] Not limited to the mixing module 333, the editing unit 330 may also include other modules to output the above audio file.

[0241] The editing unit 330 may send the audio file to the rendering unit 340 .

[0242] The rendering unit 340 can identify the type of audio playback device and render audio data suitable for playback by the audio playback device according to the corresponding audio file. The audio playback device may include the electronic device 100 itself. For example, the electronic device 100 is a stereo external speaker device such as a mobile phone. The electronic device 100 can play audio through a speaker included in the electronic device 100. The audio playback device may also include a device (or peripheral device) external to the electronic device 100. For example, the peripheral device may include headphones (wired headphones, wireless headphones), speakers, and the like. The electronic device 100 may send the rendered audio data to the peripheral device, thereby playing the audio through the peripheral device.

[0243] Exemplarily, when the audio playback device is identified as a headset or a stereo speaker, the rendering unit 340 can perform binaural rendering. The headset or stereo speaker plays audio according to the binaurally rendered audio data, so that the listener can experience the spatial sense and orientation sense of the sound.

[0244] Optionally, when the audio playback device is a headset, the electronic device 100 may also provide a head tracking function, a matching reverberation model function, and a personalized playback function. Among them, the head tracking function may mean using a headset to detect the user's head movement, and adjusting the spatial position of the sound object according to the user's head movement, so that the sense of direction of the sound perceived by the user can change with the movement of the user's head. The personalized playback function may include personalized HRTF rendering. The electronic device 100 may provide a variety of HRTF templates for users to choose from.

[0245] When the audio playback device is identified as a speaker array or a sound array, the rendering unit 340 can perform speaker rendering. For example, the scenes corresponding to the above speaker array or sound array may include scenes of audio playback by a car computer, scenes of audio playback by a large screen device, scenes of audio playback by a home theater, etc. In a possible implementation, the rendering unit 340 can implement speaker rendering by methods such as vector base amplitude panning (VBAP) technology.

[0246] The embodiments of the present application do not limit the specific implementation methods of the above-mentioned binaural rendering and speaker rendering.

[0247] In some embodiments, the electronic device 100 can perform audio rendering in real time in the audio and video recording scene, so that the audio playback device can play the audio in real time. For example, the electronic device 100 is connected to a monitoring device or headphones. During the video shooting process, the electronic device 100 can send the rendered audio data to the monitoring device or headphones in real time. In this way, the user can monitor the effect of his audio editing in real time through the monitoring device or headphones.

[0248] The codec unit 350 may be used to encode an audio file so that the electronic device 100 can send the encoded audio file to other devices. The codec unit 350 may also be used to decode the encoded audio file received by the electronic device 100 so that the electronic device 100 can use the decoded audio file for audio playback.

[0249] The codec unit 350 encodes the audio file to reduce information redundancy and improve the efficiency of the electronic device 100 in transmitting the audio file.

[0250] The encoding of the audio file by the codec unit 350 may include one or more of the following: channel encoding, sound object encoding, HOA encoding, metadata encoding, etc. The decoding of the audio file by the codec unit 350 may include one or more of the following: channel decoding, sound object decoding, HOA decoding, metadata decoding, etc.

[0251] Among them, the codec unit 350 can use a general full-rate audio coding tool or a lossless audio coding tool to encode the channel signal (i.e., channel coding), encode the sound object signal (i.e., sound object coding), and encode the HOA signal (i.e., HOA coding). The above-mentioned channel information can represent an audio signal based on the channel. The above-mentioned sound object audio signal can be an audio signal based on the sound object. The above-mentioned HOA signal can be an audio signal based on HOA. The general full-rate audio coding tool can perform one or more of the following processing on the audio signal: transient detection, window type judgment, time-frequency transformation, frequency domain noise shaping, time domain noise shaping, frequency band extension, downmixing, neural network coding, quantization and interval coding, etc., and encode the channel signal, the sound object signal, and the HOA signal into a bit stream. The above-mentioned bit stream can be an encoded representation of the audio signal.

[0252] The codec unit 350 may encode the metadata using a metadata encoding tool and encode the metadata into a bit stream.

[0253] It can be understood that the decoding of the audio file by the codec unit 350 can be the inverse process of the above encoding. Decoding can realize decoding the above bit stream into channel signals, sound object signals, HOA signals, and metadata respectively. Then, the electronic device 100 can render the above decoded data accordingly (such as speaker rendering, binaural rendering, etc.) and play it.

[0254] Figure 3 This is only an exemplary description of the structure of the electronic device 100 of the present application and should not be construed as limiting the present application. It is understandable that the electronic device 100 may also include more or fewer units or modules, or combine some modules, or separate some modules.

[0255] The communication system involved in this application is introduced below.

[0256] Figure 4A The architecture diagram of the communication system 40 provided in the present application is exemplarily shown.

[0257] like Figure 4A As shown, the communication system 40 may include an electronic device 100 and an audio playback device 200. A communication connection is established between the electronic device 100 and the audio playback device 200. For example, a wired communication connection, a wireless communication connection, etc. The embodiment of the present application does not limit the communication connection method between the electronic device 100 and the audio playback device 200.

[0258] The structure of the electronic device 100 can refer to the aforementioned Figure 1 to Figure 3 Description of the illustrated embodiment.

[0259] The audio playback device 200 can be used to play audio. For example, the audio playback device 200 can be a headset (wireless headset or wired headset), a speaker array, etc. The embodiment of the present application does not limit the type of the audio playback device 200.

[0260] It can be understood that the electronic device 100 is connected to the audio playback device 200. The audio playback device 200 can be equivalent to an extension device (or a peripheral device) of the electronic device 100, and can play the audio on the electronic device 100.

[0261] In some embodiments, in the scene of shooting a video, the electronic device 100 can collect video images and audio, and edit the audio according to the user's audio editing operation to obtain an audio file containing an audio signal and metadata. The electronic device 100 can select a suitable audio rendering method (such as binaural rendering, speaker rendering, etc.) according to the type of the audio playback device 200, and render the audio file to obtain audio data. The electronic device 100 can send the audio data to the audio playback device 200, and the audio playback device 200 performs audio playback.

[0262] Among them, the audio file used by the electronic device 100 for rendering can be an audio file containing the audio signals of all sound objects in the environment and the audio signals of the sound field. In this way, the user can monitor the overall playback sound effect of the collected audio after editing in real time during the process of shooting the video. Optionally, the audio file used by the electronic device 100 for rendering can also be an audio file containing the audio signals of one or more sound objects. That is, the audio file contains the audio tracks of the one or more sound objects and the metadata used to describe the audio tracks of the one or more sound objects. For example, the one or more sound objects can be the sound objects selected by the user for editing. In this way, the user can monitor the individual sound effects after the audio editing of one or more sound objects in real time during the process of shooting the video.

[0263] In some embodiments, in the scenario of playing a video, the electronic device 100 can obtain the audio file of the video and render the audio file to obtain audio data. The electronic device 100 can play the audio according to the audio data (i.e., the electronic device 100 plays the audio itself). Alternatively, the electronic device 100 can send the audio data to the audio playback device 200, and the audio playback device 200 plays the audio.

[0264] In addition, in the scenario of playing a video, the electronic device 100 can also edit the audio according to the user's audio editing operation to obtain an audio file. That is, in addition to real-time audio editing during video recording, the user can also perform secondary audio editing on the recorded video.

[0265] In some embodiments, in a video call (or video conference) scenario, the electronic device 100 can receive video images and audio from the other end of the call. Among them, the audio received by the electronic device 100 can be rendered audio data. The electronic device 100 can play audio at the local end according to the received audio data, or send the audio data to the audio playback device 200, and the audio playback device 200 plays the audio. Alternatively, the audio received by the electronic device 100 can be an audio file containing an audio signal and metadata. The electronic device 100 can render the audio file to obtain audio data, and then play the audio at the local end or play the audio by the audio playback device 200. In this way, the user can hear the sound of the other end of the call during the video call.

[0266] In the above Figure 4A In the communication system 40 shown, the audio playing device 200 mainly plays the audio. The electronic device 100 can send the rendered audio data to the audio playing device 200 .

[0267] Figure 4B The architecture diagram of the communication system 41 provided in the present application is exemplarily shown.

[0268] like Figure 4B As shown, the communication system 41 may include an electronic device 100 and an electronic device 300. A communication connection may be established between the electronic device 100 and the electronic device 300. For example, the electronic device 100 and the electronic device 300 may communicate via a local area network or a wide area network. The embodiment of the present application does not limit the communication connection method between the electronic device 100 and the electronic device 300.

[0269] The structures of the electronic device 100 and the electronic device 300 can refer to the aforementioned Figure 1 to Figure 3 Description of the illustrated embodiment.

[0270] In some embodiments, the electronic device 100 may send a video image and an audio file to the electronic device 300. The audio file may include an audio signal (such as an audio signal of each sound object, an audio signal of a sound field) and metadata. The electronic device 100 may encode the audio file with metadata, and one or more of the following encodings: channel encoding, sound object encoding, HOA encoding, etc. Then, the electronic device 100 may send the encoded audio file to the electronic device 300.

[0271] The above-mentioned audio encoding can reduce information redundancy and improve the transmission efficiency of audio files.

[0272] After receiving the video image and the encoded audio file, the electronic device 300 can decode and obtain the audio file. Based on the received video image and audio file, the electronic device 300 can provide the editing function of the sound object and the sound field. The electronic device 300 can process the audio file according to the audio editing operation of the user to achieve the audio editing effect desired by the user.

[0273] That is to say, in addition to providing the user with the function of editing the audio object and the sound field at the local end, the electronic device 100 can also send the audio file to other devices, such as the electronic device 300. In this way, the user can also edit the audio object and the sound field on the electronic device 300.

[0274] Figure 4C The architecture diagram of the communication system 42 provided by the present application is exemplarily shown.

[0275] like Figure 4C As shown, the communication system 42 may include multiple electronic devices (such as electronic device 100, electronic device 101, electronic device 102, etc.) and a server 400. The multiple electronic devices may establish communication connections with the server 400. The embodiment of the present application does not limit the communication connection method.

[0276] The structures of the above-mentioned multiple electronic devices can all refer to the aforementioned Figure 1 to Figure 3 Description of the illustrated embodiment.

[0277] At least two of the above-mentioned multiple electronic devices can conduct a video call or a video conference through the server 400. This application specifically takes the scenario of a video call as an example for introduction.

[0278] For example, electronic device 100, electronic device 101, and electronic device 102 can access the same video call. During the video call, electronic device 100 can send its own video image and audio to electronic device 101 and electronic device 102 through server 400, and receive video image and audio from electronic device 101 and video image and audio from electronic device 102 through server 400.

[0279] Among them, the audio sent by the electronic device 100 can be audio data obtained by rendering an audio file. In this way, the electronic device 101 and the electronic device 102 can play the audio after receiving the audio data of the electronic device 100. The user on the electronic device 100 side can perform sound object and / or sound field editing on the audio collected by the electronic device 100. The user on the electronic device 101 side and the user on the electronic device 102 side can listen to the sound on the electronic device 100 side (for example, the sound after the user on the electronic device 100 side performs sound object and / or sound field editing).

[0280] Alternatively, the audio sent by the electronic device 100 may be an audio file containing an audio signal and metadata. Furthermore, the electronic device 100 may encode the audio file (e.g., channel encoding, acoustic object encoding, HOA encoding, metadata encoding) and then send it to the server 400. Among them, the server 400 may directly send the encoded audio file to the electronic device 101 and the electronic device 102. Both the electronic device 101 and the electronic device 102 may decode the encoded audio file (e.g., channel decoding, acoustic object decoding, HOA decoding, metadata decoding), and render the audio data from the decoded audio file. Since the electronic device 101 and the electronic device 102 receive the audio file on the electronic device 100 side, the electronic device 101 and the electronic device 102 may provide the audio on the electronic device 100 side with the acoustic object editing function and the sound field editing function. In this way, the user on the electronic device 101 side and the user on the electronic device 102 side may change the sound on the electronic device 100 side that they hear. Alternatively, the server 400 may decode the encoded audio file sent by the electronic device 100 and render the audio data. The server 400 may send audio data to the electronic device 101 and the electronic device 102 so that they can play the audio.

[0281] The audio received by the electronic device 100 (for example, the audio from the electronic device 101 and the audio from the electronic device 102) may be audio data, or may be an audio file containing an audio signal and metadata. The electronic device 100 may play the audio according to the audio data, or render the audio file to obtain the audio data and then play the audio. The electronic device 100 may play the audio locally or through an audio playback device (such as the audio playback device 200) connected thereto.

[0282] When the electronic device 100 receives the audio file, the electronic device 100 can edit the sound object and / or sound field in the audio file to generate a new audio file. The electronic device 100 can render the new audio file and then play the audio.

[0283] In some embodiments, the user on the electronic device 100 side can edit the sound object and / or sound field of one or more call peers, so that all users connected to the call can hear the sound after the electronic device 100 edits the sound object and / or sound field of the above one or more call peers. For example, in the host mode / master control mode, the user on the electronic device 100 side can be the host of the video call. The user on the electronic device 100 side can have the authority to control other users in the call and can adjust the sound object and sound field of the call peer.

[0284] In one possible implementation, the electronic device 100 receives an operation to edit the sound object and / or sound field on the electronic device 101 side. The electronic device 100 can generate editing parameters based on the above editing operation and send an editing instruction to the server 400. The editing instruction may include the above editing parameters. The server 400 can send the editing instruction to the electronic device 101. The electronic device 101 can edit the sound object and / or sound field according to the editing parameters in the editing instruction to generate an audio file. The electronic device 101 can then send the audio file or the audio data rendered by the audio file to the call partner (such as the electronic device 100, the electronic device 102) through the server 400.

[0285] Optionally, when the server 400 receives the audio file of the electronic device 101, the server 400 can edit the audio file according to the editing instruction sent by the electronic device 100 to adjust the sound object and / or sound field therein. The server 400 can edit the audio file of the electronic device 101 into a new audio file, and send the new audio file or the audio data rendered by the new audio file to the electronic device 100 and the electronic device 102. It can be understood that the server 400 can encode the audio file (for example, channel decoding, sound object decoding, HOA decoding, metadata decoding) before sending it.

[0286] That is to say, the electronic device 100 can adjust the sound object and sound field of the call counterpart by sending editing instructions to instruct the server 400 or the adjusted call counterpart (such as electronic device 101, electronic device 102, etc.) to perform specific audio editing processing.

[0287] The following introduces the audio processing scenario provided by this application based on the above-mentioned electronic device and communication system.

[0288] Among them, the audio processing method provided by this application can be applied to video recording scenarios and video call scenarios, so that users can perform sound object editing, sound field editing, and mixing editing on the audio collected by the electronic device 100 in the above scenarios. Not limited to video recording scenarios and video call scenarios, the audio processing method provided by this application can also be applied to other scenarios involving the collection of video images and audio. This application specifically uses video recording scenarios and video call scenarios as examples for explanation.

[0289] Audio editing in video recording scenarios

[0290] The video recording scene may include the following stages: a video pre-recording stage, a video recording stage, and after the video recording is completed. Among them, the video pre-recording stage may refer to the stage after the electronic device 100 turns on the camera application but before starting to shoot the video. The video images captured by the electronic device 100 in the video pre-recording stage will not be saved as a video. The video recording stage may refer to the stage from the start of video shooting to the end of video shooting of the electronic device 100. After the video recording is completed, it may refer to the stage after the electronic device 100 finishes shooting the video. After the video recording is completed, the user can view the video shot by the electronic device 100 in the gallery application of the electronic device 100.

[0291] During the video pre-recording stage, video recording stage, and after the video recording is completed, the user can edit the audio. Audio editing may include but is not limited to: sound object editing, sound field editing, and mixing editing.

[0292] Video pre-recording stage

[0293] Figure 5A to Figure 5M Schematic diagrams showing some audio editing scenarios.

[0294] like Figure 5A As shown, the electronic device 100 may display a user interface 510. The user interface 510 may include one or more application icons. For example, a camera application icon 511. The above application icons may be used to trigger the electronic device 100 to open a corresponding application. Figure 5A As shown in the operation of the camera application icon 511, the electronic device 100 can open the camera application and display Figure 5B The user interface 520 shown. When the camera application is turned on, the electronic device 100 can turn on the camera and display the image collected by the camera on the screen. In addition, when the camera application is turned on, the electronic device 100 can also turn on the microphone to collect audio. The specific time when the above-mentioned camera and camera application are turned on is not limited in the embodiment of the present application.

[0295] like Figure 5BAs shown, user interface 520 may include one or more of audio editing controls 521 , preview area 522 , camera mode options 523 , gallery shortcut controls 524 , shutter controls 525 , and / or camera flip controls 526 .

[0296] The audio editing control 521 can be used to turn on or off the audio editing function of the electronic device 100. The above-mentioned audio editing function may refer to a function that allows the user to edit the sound object and the sound field of the audio. When the audio editing function is turned off, the display style of the audio editing control 521 may be When the audio editing function is in an off state, the electronic device 100 may turn on the audio editing function in response to an operation on the audio editing control 521 .

[0297] The preview area 522 can be used to display a preview image. The preview image is an image captured by the electronic device 100 through the camera in real time. The electronic device can refresh the display content in the preview area 522 in real time to facilitate the user to preview the image currently captured by the camera. Figure 5B As can be seen from the preview area 522 shown, the content currently photographed by the electronic device 100 includes a boy, a girl, and a violin.

[0298] One or more shooting mode options may be displayed in the camera mode option 523. The one or more shooting mode options may include: large aperture mode option 523A, portrait mode option 523B, video recording mode option 523C, photo taking mode option 523D and more options 523E. The one or more shooting mode options may be displayed as text information on the interface, such as "large aperture", "portrait", "video recording", "photo taking", and "more". Not limited to this, the one or more camera options may also be displayed as icons or other forms of interactive elements (IE) on the interface. When a user operation on a shooting mode option is detected, the electronic device 100 may turn on the shooting mode selected by the user and display the shooting mode option as selected. Figure 5B As shown, the video mode option 523C is selected. Figure 5B As shown, more or fewer shooting mode options may also be included in the camera mode option 523. The user may browse other shooting mode options by sliding left / right in the camera mode option 523.

[0299] The gallery shortcut control 524 can be used to open the gallery application. In response to a user operation acting on the gallery shortcut control 524, such as a touch operation, the electronic device 100 can open the gallery application. In this way, the user can conveniently view the photos and videos taken without having to exit the camera application first and then open the gallery application. The gallery application is an application for picture management on electronic devices such as smartphones and tablets, and can also be called an "album". This embodiment does not limit the name of the application. The gallery application can support users to perform various operations on pictures stored on the electronic device 100, such as browsing, editing, deleting, selecting, and the like.

[0300] The shutter control 525 can be used to monitor the user operation that triggers taking a photo or recording a video. Figure 5B When the video recording mode option 523C is selected, the electronic device 100 can start shooting a video and enter the video recording stage in response to the operation of the shutter control 525.

[0301] The camera flip control 526 can be used to monitor the user operation that triggers the flipping of the camera. The electronic device 100 can detect the user operation acting on the camera flip control 526, such as a touch operation, and in response to the operation, the electronic device 100 can flip the camera used for shooting, such as switching the rear camera to the front camera, or switching the front camera to the rear camera.

[0302] like Figure 5B As shown, in the case where the audio editing control 521 is displayed in a closed state, in response to an operation (such as a click operation) on the audio editing control 521, the electronic device 100 can turn on the audio editing function and display Figure 5C User interface 530 is shown.

[0303] like Figure 5C As shown, when the audio editing function is in the on state, the display style of the audio editing control 521 may be "AI Audio". When the audio editing function is in the on state, in response to the operation of the audio editing control 521, the electronic device 100 may turn off the audio editing function. The embodiment of the present application does not limit the display style of the audio editing control 521. Optionally, the electronic device 100 may turn on the audio editing function without the user manually turning on the audio editing function. For example, the audio editing function may be always on.

[0304] The user interface 530 may include a preview area 522. The electronic device 100 may mark the captured target in the preview area 522 and provide a volume bar and a mute control corresponding to the target. For example, the electronic device 100 may determine three targets on the image in the preview area 522, that is, three sound objects (which may be referred to as sound images for short): sound image 1, sound image 2, and sound image 3.

[0305] The electronic device 100 can display a marking box 531, a volume bar 531A, and a mute control 531B associated with sound image 1, a marking box 532, a volume bar 532A, and a mute control 532B associated with sound 2, and a marking box 533, a volume bar 533A, and a mute control 533B associated with sound 3 on the user interface 530.

[0306] The marking box 531 can be used to circle the sound image 1 to prompt the user the object represented by the sound image 1. For example, the sound image 1 is a boy currently photographed by the electronic device 100. The volume bar 531A can be used to adjust the volume of the sound image 1. The mute control 531B can be used to mute the sound image 1 (that is, adjust the volume of the sound image 1 to 0).

[0307] The marking box 532 can be used to circle the sound image 2 to prompt the user the object represented by the sound image 2. For example, the sound image 2 is a girl currently photographed by the electronic device 100. The volume bar 532A can be used to adjust the volume of the sound image 2. The mute control 532B can be used to mute the sound image 2 (that is, adjust the volume of the sound image 2 to 0).

[0308] The marking box 533 can be used to circle the sound image 3 to prompt the user the object represented by the sound image 3. For example, the sound image 3 is a violin currently captured by the electronic device 100. The volume bar 533A can be used to adjust the volume of the sound image 3. The mute control 533B can be used to mute the sound image 3 (that is, adjust the volume of the sound image 3 to 0).

[0309] The volume bars and mute controls associated with each sound image can be used to adjust the volume of the sound image intuitively and quickly. For example, users can quickly enhance the human voice or mute the sound object.

[0310] The user interface 530 may also include an audio and video control 534 , a sound field control 535 , and a mixing control 536 .

[0311] The sound image control 534 can be used to select the sound image and perform sound object editing such as adjusting the timbre.

[0312] The sound field control 535 can be used to edit the sound field so that the user can adjust the sound field environment information.

[0313] Mixing controls 536 may be used to adjust parameters for mixing.

[0314] It should be noted that when the audio editing function is turned on, the electronic device 100 can separate the collected audio into sound objects to obtain tracks of different sound objects. The electronic device 100 can also determine the spatial position of each sound object and the position displayed in the video image (i.e., the image displayed in the preview area 522). In this way, the electronic device 100 can determine Figure 5CThe audio tracks of the audio image 1, the audio tracks of the audio image 2, and the audio tracks of the audio image 3 are shown. When detecting the user's audio editing operation on an audio image, the electronic device 100 can edit the audio track of the audio image and the description information about the audio track of the audio image in the metadata to adjust the volume, timbre, or spatial position and other parameters of the sound object.

[0315] The electronic device 100 may also determine the sound bed (i.e., the audio signal of the sound field) and the sound field environment information. When the user's audio editing operation on the sound field is detected, the electronic device 100 may edit the sound bed and the description information about the sound field in the metadata (i.e., the sound field environment information) to adjust the reverberation time, room type, room size, or room reflective material of the sound field and other parameters.

[0316] In response to Figure 5C As shown in the operation of sliding upward on the volume bar 531A, the electronic device 100 can move the slider on the volume bar 531A upward to increase the volume of the sound image 1. The electronic device 100 can edit the audio track of the sound image 1 and the description information about the audio track of the sound image 1 in the metadata to increase the volume of the sound image 1. In this way, the volume of the sound image 1 is increased in the overall playback sound effect.

[0317] like Figure 5D As shown, the position of the slider on the volume bar 531A is Figure 5C The slider on the volume bar 531A is shown to be positioned further up. Figure 5D The electronic device 100 can display the operation on the audio and video control 534. Figure 5E Shown are audio and video 1 option 534A, audio and video 2 option 534B, and audio and video 3 option 534C.

[0318] like Figure 5E As shown, the audio and video 1 option 534A, the audio and video 2 option 534B, and the audio and video 3 option 534C can be used to select audio and video 1, audio and video 2, and audio and video 3, respectively, so as to edit the selected audio and video. Figure 5E As shown in the operation of the audio and video 1 option 534A, the electronic device 100 can display Fig. 5F User interface 540 is shown.

[0319] like Fig. 5F As shown, the user interface 540 may include an audio and video control 534 and an audio and video editing box 541 .

[0320] The text content “audio image 1” may be displayed on the audio image control 534 to indicate that the audio image 1 is currently selected, and the object of audio image editing based on the audio image editing box 541 is the audio image 1.

[0321] The sound image editing box 541 may include multiple timbre options so that the user can adjust the timbre of the sound image to the timbre corresponding to the selected timbre option. For example, the timbre options may include: a girl timbre option, a boy timbre option 541A, an angel timbre option, an elf timbre option, and the like. In a possible implementation, the above timbre options may correspond to preset timbre adjustment parameters. The timbre adjustment parameters can be used to adjust the timbre of the sound image to the timbre corresponding to the timbre option. For example, the electronic device 100 adjusts the timbre of the sound image according to the timbre adjustment parameters corresponding to the boy timbre option 541A, so that the timbre of the sound image can sound like a boy's voice.

[0322] The audio and video editing box 541 may further include a confirmation control 541B and a cancel control 541C. The confirmation control 541B may be used to trigger the electronic device 100 to adjust the timbre of the audio and video 1 according to the timbre option selected in the audio and video editing box 541. The cancel control 541C may be used to cancel the editing of the audio and video 1.

[0323] Not limited to the timbre option, the sound image editing box 541 may also include more controls for editing other parameters of the sound image 1. For example, the sound image editing box 541 may include controls for adjusting the volume of the sound image 1, controls for adjusting the spatial position of the sound image 1, and so on.

[0324] In response to Fig. 5F The electronic device 100 may display the juvenile voice option 541A as Figure 5G The selected state shown (as shown in a darker color).

[0325] like Figure 5G As shown, when the juvenile voice option 541A is in a selected state, in response to the operation of the determination control 541B, the electronic device 100 can adjust the timbre of the sound image 1 to the timbre corresponding to the juvenile voice option 541A, so that the timbre of the sound image 1 sounds like a teenager's voice.

[0326] like Figure 5H As shown, in response to the operation of the sound field control 535, the electronic device 100 can display Fig.5I User interface 550 is shown.

[0327] like Fig.5IAs shown, the user interface 550 may include a sound field editing box 551. The sound field editing box 551 may include multiple room type options to facilitate the user to replace the room type of the sound field. For example, the room type options may include conference room option 551A, church option, recording studio option, living room option, bedroom option, theater option, etc. In one possible implementation, the above-mentioned room type options may correspond to preset sound field adjustment parameters. The sound field adjustment parameters may include one or more of the following: reverberation time, room size, room reflective material, etc. The sound field adjustment parameters can be used to adjust the sound field to the sound field corresponding to the sound field option. For example, the electronic device 100 edits the sound field according to the sound field adjustment parameters corresponding to the conference room option 551A, so that the user can feel as if he is in a conference room to listen to the audio collected by the electronic device 100.

[0328] The sound field editing box 551 may also include a confirmation control 551B and a cancel control 551C. The confirmation control 551B may be used to trigger the electronic device 100 to adjust the sound field according to the room type option selected in the sound field editing box 551. The cancel control 551C may be used to cancel the editing of the sound field.

[0329] Not limited to the room type option, the sound field editing box 551 may also include more controls for editing other parameters of the sound field. For example, the sound field editing box 551 may include controls for adjusting the reverberation time of the sound field, controls for adjusting the room size of the sound field, controls for adjusting the room reflection material of the sound field, and so on. It can be understood that when the room type of the sound field is selected, the electronic device 100 can obtain preset sound field adjustment parameters (such as reverberation time, room size, room reflection material). However, the user can still change the value of one or more parameters in the sound field adjustment parameters as needed. For example, the room size in the preset sound field adjustment parameters of the conference room is 30m 2 . Users can adjust the room size of the meeting room to 50m 2 Equal size.

[0330] In response to Fig.5I The electronic device 100 may display the conference room option 551A as Fig.5I When the conference room option 551A is in the selected state, in response to the operation of the confirmation control 551B, the electronic device 100 can replace the sound field of the audio with the sound field corresponding to the conference room option 551A, so that the user feels as if he is in the conference room to listen to the audio.

[0331] like Figure 5J As shown, in response to the operation of the mixing control 536, the electronic device 100 can display Figure 5K User interface 560 is shown.

[0332] like Figure 5K As shown, the user interface 560 may include a sound mixing edit box 561. The sound mixing edit box 561 may include a smart sound mixing control 561A, a nice sound control 561B, a vacant control 561C, a confirmation control 561E, and a cancel control 561F.

[0333] The smart mixing control 561A can be used to trigger the electronic device 100 to enter the smart mixing mode. In the smart mixing mode, the electronic device 100 can perform intelligent volume balancing and reverberation processing on the audio track, spatial position and sound field environment information of the sound image through the mixing algorithm to achieve the effect of beautifying the sound effect. It can be seen that the user can achieve one-click smart mixing through the smart mixing control 561A.

[0334] It should be noted that, in the intelligent mixing mode, the electronic device 100 can combine the content of the sound image and the sound field edited separately by the user when mixing.

[0335] The NICam control 561B may be used to trigger the electronic device 100 to perform mixing using a NICam template.

[0336] The ethereal control 561C can be used to trigger the electronic device 100 to use the ethereal template for mixing. The above-mentioned NIC template and ethereal template can refer to the introduction of the above-mentioned embodiment. The electronic device 100 can also include more or fewer mixing templates. This embodiment of the application is not limited to this.

[0337] The confirmation control 561E may be used to trigger the electronic device 100 to perform mixing according to the selected control and the set parameters in the mixing edit box 561. The cancel control 561F may be used to cancel the editing of the mixing.

[0338] In response to Figure 5K The electronic device 100 can switch the content displayed in the audio mixing edit box 561 by sliding left in the audio mixing edit box 561. For example, the electronic device 100 can display the content displayed in the audio mixing edit box 561. Figure 5L Equalizer controls 562 are shown. Equalizer controls 562 can be used to adjust the EQ of the audio.

[0339] In response to Figure 5L The electronic device 100 can further switch the content displayed in the audio mixing edit box 561 by sliding left in the audio mixing edit box 561. For example, the electronic device 100 can display the content displayed in the audio mixing edit box 561. Figure 5MThe reverberation time control 563, the room size control 564, and the room reflection material control 565 are shown. The reverberation time control 563 can be used to adjust the reverberation time of the sound field. The room size control 564 can be used to adjust the room size of the sound field. The room reflection material control 565 can be used to adjust the room reflection material of the sound field. For example, the room reflection material can include but is not limited to: wall, glass, carpet, metal.

[0340] Not limited to Figure 5K to Figure 5M The content displayed in the mixing edit box 561 is shown, and the mixing edit box 561 may also contain more space for other parameters for mixing edits, such as controls for adjusting compression parameters of audio (i.e., compressor), controls for adjusting delay parameters of audio (i.e., delay), and the like.

[0341] In some embodiments, the electronic device 100 can edit the collected audio according to the editing parameters determined by the sound object editing operation, the sound field editing operation, and the mixing editing operation, and output an audio file containing the audio signal and metadata after mixing.

[0342] It should be noted that the controls for editing sound objects, sound fields, and mixing are only exemplary descriptions of the present application and should not be construed as limiting the present application. The electronic device 100 may also provide controls in other display styles, or allow the user to edit sound objects and sound fields using gestures or the like.

[0343] By the above Figure 5A to Figure 5M As can be seen from the scenario shown, the user can perform audio editing before shooting a video, for example, muting unnecessary sound and images, enhancing the human voice of some important sound and images, adjusting the sound field, etc. In this way, when entering the video recording stage, the electronic device 100 can edit the audio collected in the video recording stage according to the editing parameters determined by the audio editing operation in the video pre-recording stage, so that the recorded audio meets the needs of the user.

[0344] Video recording stage

[0345] Figure 6A to Figure 6G Schematic diagrams of other audio editing scenarios are shown as examples.

[0346] like Fig. 6A As shown, the electronic device 100 can display a user interface 530. The user interface 530 can refer to the introduction of the above embodiment. Fig. 6A By performing an operation on the shutter control 525, such as a click operation, the electronic device 100 can enter the video recording phase, start shooting a video, and display Figure 6B User interface 610 is shown.

[0347] like Figure 6B As shown, the user interface 610 may include time information 611 , an audio editing mark 612 , an image display area 613 , a photo taking control 614 , an end control 615 , and a pause control 616 .

[0348] The time information 611 may be used to indicate the duration of shooting the video.

[0349] The audio editing indicator 612 may be used to indicate that the audio editing function is enabled.

[0350] The image display area 613 may be used to display images captured by the electronic device 100 during the process of shooting a video.

[0351] The photo control 614 may be used to save a frame of image captured when an operation on the photo control 614 is detected into a gallery.

[0352] End control 615 may be used to end capturing the video.

[0353] The pause control 616 may be used to pause capturing the video.

[0354] In some embodiments, when entering the video recording stage, the electronic device 100 may cancel the display of the aforementioned controls for audio editing (such as a mark box, a volume bar, a mute control, an audio and video control, a sound field control, a mixing control, etc.). When a preset operation acting on the image is detected (such as clicking or long pressing the audio and video on the image, etc.), the electronic device 100 may display the controls for audio editing again. Afterwards, the electronic device 100 may hide the controls for audio editing again when no operation on the controls for audio editing is detected within a preset time. This can reduce the situation in which the controls for audio editing block the video screen during the video recording stage and interfere with the user's shooting.

[0355] Optionally, when entering the video recording stage, the electronic device 100 may also always keep the controls for audio editing displayed on the shooting screen.

[0356] In response to Figure 6B As shown in the figure, for example, when operating the sound image 3 in the image, the electronic device 100 can display Figure 6C User interface 620 is shown.

[0357] like Figure 6C As shown, the user interface 620 may display controls for audio editing: mark boxes, volume bars, and mute controls associated with each of the audio image 1, audio image 2, and audio image 3, an audio image control 621, a sound field control 622, and a mixing control 623. The above controls may refer to the above description of Figure 5C Introduction.

[0358] The audio and video control 621 shows Figure 6C In the case of the text content "audio and video 3" shown in the figure, in response to the operation of the audio and video control 621, the electronic device 100 can display Fig.6D User interface 630 is shown.

[0359] like Fig.6D As shown, the user interface 630 may include an audio and video editing box 631. The audio and video editing box 631 may refer to the aforementioned Fig. 5F The audio and video editing box 541 is shown. Fig. 5F and Fig.6D It can be seen that the timbre options in the sound image editing box 541 can be used to adjust the timbre of sound image 1. The timbre options in the sound image editing box 631 can be used to adjust the timbre of sound image 3. Among them, the electronic device 100 can identify whether the type of the sound image is a person or a musical instrument. When the type of the sound image is identified as a person (for example, sound image 1), the electronic device 100 can provide the timbre options related to human voice in the sound image editing box 541. When the type of the sound image is identified as a musical instrument (for example, sound image 3), the electronic device 100 can provide the timbre options related to the sound of the musical instrument in the sound image editing box 631.

[0360] That is to say, in addition to supporting the timbre adjustment of human voice, the electronic device 100 can also support the timbre adjustment of musical instrument sound to achieve instrument conversion. Not limited to human voice and musical instrument sound, the electronic device 100 can also adjust the timbre of other types of sound images. This embodiment of the application is not limited to this.

[0361] like Fig.6D As shown in FIG. 1 , the sound image editing box 631 may include piano sound color options 631A, flute sound color options, guitar sound color options, and the like, and a confirmation control 631B. In response to the operation of the piano sound color option 631A, the electronic device 100 may display the piano sound color option 631A as Fig.6D When the piano timbre option 631A is selected, in response to the operation of the confirmation control 631B, the electronic device 100 can adjust the timbre of the sound image 3 to the timbre corresponding to the piano timbre option 631A, so that the timbre of the sound image 3 sounds like the timbre of a piano.

[0362] In some embodiments, the electronic device 100 may also provide a function of merging multiple audio and video. In this way, the user can edit multiple audio and video at the same time through one audio editing operation.

[0363] like Fig. 6E As shown, the user interface 620 may include a mark box 624 associated with the audio image 1 and a mark box 625 associated with the audio image 2. Fig. 6E As shown in the figure, the electronic device 100 can display the sliding operation from the area where the mark box 624 is located to the area where the mark box 625 is located. Fig. 6F Prompt box 626 is shown.

[0364] like Fig. 6F As shown, prompt box 626 can be used to ask the user whether to merge the sound image. Prompt box 626 can include a text prompt "Confirm to merge sound image 1 and sound image 2", a confirmation control 626A, and a cancel control 626B. Confirm control 626A can be used to trigger electronic device 100 to merge sound image 1 and sound image 2. Cancel control 626B can be used to cancel the merger of sound image 1 and sound image 2.

[0365] In a possible implementation, the electronic device 100 merging multiple sound images may represent associating the audio tracks of the multiple sound images with the merged sound image. When an operation of editing the merged sound image (e.g., an operation of adjusting the volume, adjusting the timbre, adjusting the spatial position, etc.) is detected, the electronic device 100 may edit the audio tracks of the multiple sound images and the description information about the audio tracks of the multiple sound images in the metadata.

[0366] Exemplarily, the electronic device 100 separates Track 1 and Track 2 from the collected audio. Track 1 is Fig. 6E The audio track of Pan 1 is shown. Track 2 is Fig. 6E When an operation for merging audio image 1 and audio image 2 is detected, the electronic device 100 may merge audio image 1 and audio image 2 into audio image 1′, and determine the audio tracks of audio image 1′ as audio track 1 and audio track 2. In this way, the user can edit audio image 1 and audio image 2 at the same time by editing audio image 1′.

[0367] In response to Fig. 6F As shown in the operation of the confirmation control 626A, the electronic device 100 can display Figure 6G User interface 620 is shown.

[0368] like Figure 6G As shown, the user interface 620 may include a marking box 627, a volume bar 627A, and a mute control 627B associated with the sound image 1′. The marking box 627 may circle the sound image 1 and the sound image 2 before the sound image is merged on the image, so as to prompt the user the object represented by the sound image 1′. The volume bar 627A can be used to adjust the volume of the sound image 1′. The mute control 627B can be used to mute the sound image 1′. For example, the sound image 1′ is a boy and a girl currently photographed by the electronic device. In response to the operation of the mute control 627B, the electronic device 100 can edit the audio track of the sound image 1′ and the descriptive information about the audio track of the sound image 1′ in the metadata, so that the sound image 1′ (that is, Figure 6G The volume of the two images (a boy and a girl) is 0. That is, the sound of the audio image 1′ cannot be heard in the video shot by the electronic device 100.

[0369] The above user operation for merging audio and video is only an exemplary description of the present application and should not be construed as limiting the present application. The user can also merge audio and video through other operation modes.

[0370] It should be noted that the electronic device 100 can determine the editing parameters according to the received audio editing operation, and use the editing parameters to edit the audio. The effect time (or effective time) of the above editing parameters can be determined by the time of the audio editing operation.

[0371] Exemplarily, at time t1 in the video recording stage, the electronic device 100 receives an operation to mute the sound and image 1′. The electronic device 100 can determine the editing parameter 1 based on the operation: the volume of the sound and image 1′ is 0. Then, the electronic device 100 can perform audio editing based on the editing parameter 1, so that the sound and image 1′ of the video shot by the electronic device 100 remains silent from time t1. At time t2 in the video recording stage, the electronic device 100 receives an operation to increase the volume of the sound and image 1′ to volume 1. The above-mentioned time t2 is later than time t1. The electronic device 100 can determine the editing parameter 2 based on the operation: the volume of the sound and image 1′ is volume 1. Then, the electronic device 100 can perform audio editing based on the editing parameter 2, so that the volume of the sound and image 1′ of the video shot by the electronic device 100 is volume 1 from time t2. It can be seen that the effective time of the editing parameter 1 is from time t1 to time t2. The sound and image 1′ in the video shot by the electronic device 100 remains silent from time t1 to time t2.

[0372] The effective time of other editing parameters can refer to the effective time of the above-mentioned volume-related editing parameters. For example, the user can adjust the timbre of a sound image so that the timbre of the sound image in the video is timbre 1 in one time period and timbre 2 in another time period. For another example, the user can adjust the sound field so that the sound field of the video is sound field 1 in one time period and sound field 2 in another time period.

[0373] In some embodiments, the camera has a limited shooting angle, and the video image may only contain a portion of the sound and image. The electronic device 100 may provide a control for representing the sound and image that are not captured, so that the user can perform audio editing on the sound and image that are not captured.

[0374] Figure 7A to Figure 7E Schematic diagrams of other audio editing scenarios are shown as examples.

[0375] like Fig. 7AAs shown, the electronic device 100 can display a user interface 710. The user interface 710 can be a video shooting interface of the electronic device 100 in the video recording stage. The user interface 710 can display an image captured by the electronic device 100. The electronic device 100 can recognize the sound and image on the image and provide controls for audio editing. For example, the controls for audio editing may include: mark boxes associated with each sound and image, a volume bar, a mute control, as well as an audio and video control 712, a sound field control 713, and a mixing control 714. The above-mentioned controls for user audio editing can refer to the introduction of the aforementioned embodiment.

[0376] The user interface 710 may also include an off-screen audio and video display area 711. The off-screen audio and video display area 711 may include controls for representing audio and video that are not captured. Fig. 7A The audio tracks of the captured sound images 1, 2 and 3 are shown, and the audio tracks of the separated sound images 4, 5 and 6 are also separated. Among them, the sound images 4, 5 and 6 are all sound images that are not captured. The electronic device 100 can display the sound image 4 mark 711A, the sound image 5 mark 711B, and the sound image 6 mark 711C in the off-screen sound image display area 711.

[0377] In some embodiments, the electronic device 100 can identify the type of sound and image, and determine the display style of the above-mentioned sound and image that was not captured according to the type of sound and image. For example, the above-mentioned sound and image 4 and sound and image 5 are people. The display style of the sound and image 4 mark 711A and the sound and image 5 mark 711B can be a person icon, indicating that the sound of the sound and image 4 and the sound and image 5 is a human voice. The above-mentioned sound and image 6 is a cat. The display style of the sound and image 6 mark 711C can be a cat icon, indicating that the sound of the sound and image 6 is a cat's cry. The embodiment of the present application does not limit the display style of the above-mentioned sound and image that was not captured.

[0378] In some embodiments, the electronic device 100 may determine the spatial position of the sound image that has not been captured, and determine the display position of the control for representing the sound image that has not been captured on the user interface 710 according to the spatial position. Exemplarily, when the spatial position of the sound image 4 indicates that the sound image 4 is located on the right side of the sound image 1, the electronic device 100 may display the above-mentioned sound image 4 identification 711A on the right side of the sound image 1 on the image. In this way, the user can determine which sound image corresponds to the sound image identification according to the display position of the sound image identification, and then edit the sound image that has not been captured according to the needs. The display position of the above-mentioned sound image that has not been captured is not limited in the embodiment of the present application.

[0379] In response to Fig. 7A As shown in FIG. 1 , for example, by clicking on the audio / video mark 711A, the electronic device 100 may display Figure 7BThe volume bar 715 is shown. The volume bar 715 can be used to adjust the volume of the sound image 4. When an operation of adjusting the volume on the volume bar 715 is detected, the electronic device 100 can edit the audio track of the sound image 4 and the description information about the audio track of the sound image 4 in the metadata to adjust the volume of the sound image 4.

[0380] like Figure 7B As shown, in response to the operation of the audio and video control 712, the electronic device 100 can display all audio and video options to facilitate the user to select the audio and video to be edited. The above-mentioned all audio and video can include audio and video captured by the electronic device 100 and audio and video not captured.

[0381] like Figure 7C As shown, all the audio and video options displayed by the electronic device 100 may include audio and video 1 option 716A, audio and video 2 option 716B, audio and video 3 option 716C, audio and video 4 option 716D, audio and video 5 option 716E, and audio and video 6 option 716F. Figure 7C For the operation of the audio and video 4 option 716D, the electronic device 100 can display Fig.7D User interface 720 is shown.

[0382] like Fig.7D As shown, the user interface 720 may include an audio and video control 712 and an audio and video editing box 721 .

[0383] The text content “audio image 4” may be displayed on the audio image control 712 to indicate that the audio image 4 is currently selected, and the object of the audio image editing based on the audio image editing box 721 is the audio image 4.

[0384] The audio and video editing box 721 can refer to the aforementioned Fig. 5F The description of the audio and video editing frame 541 is shown. It will not be repeated here.

[0385] like Fig. 7E As shown, the electronic device 100 can rotate to change the shooting angle. For example, when the shooting angle changes, the sound and image captured by the electronic device 100 changes from Fig. 7A The audio image 1, audio image 2, and audio image 3 shown in the figure change to Fig. 7E The audio image 2, audio image 4, and audio image 5 are shown. When the shooting angle changes, the electronic device 100 can display a user interface 730. The user interface 730 can include a mark box 731, a volume bar 731A, and a mute control 731B associated with the audio image 4, and a mark box 732, a volume bar 732A, and a mute control 732B associated with the audio image 5. In this way, the user can intuitively adjust the volume of the audio image 4 and the audio image 5.

[0386] The user interface 730 may also include an off-screen audio and video display area 711. Due to the change in the shooting angle, audio and video 1 and audio and video 3 become audio and video that are not captured. Therefore, the electronic device 100 may display the audio and video in the off-screen audio and video display area 711. Fig. 7E The illustrated audio image 1 mark 711D, the audio image 3 mark 711E, and the audio image 6 mark 711C which is still the audio image 6 that has not been captured. The user can edit the audio of the audio image 1 and the audio image 3 through the audio image 1 mark 711D and the audio image 3 mark 711E respectively.

[0387] In some embodiments, the electronic device 100 is connected to an audio playback device such as a monitoring device or headphones. The monitoring device can be used to play audio. The electronic device 100 can mix and output audio files in real time, and render the audio files in real time, so that the user can monitor the sound effects of the video recording in real time through the audio playback device. This makes it convenient for the user to understand the sound effects of the audio editing in a timely manner, so as to perform audio editing so that the sound effects reach the effects required by the user.

[0388] Among them, the electronic device 100 can render the audio files corresponding to the sound bed and the audio tracks of all sound objects, and pass the rendered audio data to the audio playback device for playback. In this way, the user can listen to all the audio during the video recording process. Alternatively, the electronic device 100 can also only render the audio files corresponding to the audio tracks of a single or multiple sound images, and pass the rendered audio data to the audio playback device for playback. In this way, the user can listen to the audio of a single or multiple sound images. For example, the user can only listen to the audio of the edited sound image in order to determine whether the editing of the sound image is appropriate.

[0389] Figure 8A to Figure 8C Schematic diagrams of other audio editing scenarios are shown as examples.

[0390] Here, the electronic device 100 and Fig. 8A The connection of the monitoring device 801 shown is taken as an example for explanation.

[0391] like Fig. 8A As shown, the electronic device 100 can display a user interface 810. The user interface 810 can refer to the aforementioned Figure 6C 801. The user interface 810 may include an audio-visual control 811, a sound field control 812, and a mixing control 813. The electronic device 100 may output the collected audio as an audio file including an audio signal and metadata after mixing, and render the audio file to obtain audio data. The electronic device 100 may send the audio data to the monitoring device 801. The monitoring device 801 may play all the audio during the video recording process according to the received audio data. In this way, the user can understand the overall playback sound effect of the video.

[0392] In the case where the text content displayed on the audio-visual control 811 is "audio-visual 1", in response to Fig. 8A As shown in the operation of the audio and video control 811, the electronic device 100 can display Figure 8B User interface 820 is shown.

[0393] like Figure 8B As shown, the user interface 820 may include an audio and video editing box 821. The audio and video editing box 821 may refer to the aforementioned Fig. 5F The audio and video editing box 541 shown. The audio and video editing box 821 may include an angel tone option 821A and a confirmation control 821B. Figure 8B The electronic device 100 may display the angel tone option 821A as Figure 8C The selected state shown (as shown in a darker color).

[0394] like Figure 8C As shown, in response to the operation of the determination control 821B, the electronic device 100 can adjust the timbre of the sound image 1 to the timbre corresponding to the angel timbre option 821A, so that the timbre of the sound image 1 sounds like the timbre of an angel.

[0395] In a possible implementation, when an operation of adjusting the timbre of the sound image 1 is detected, the electronic device 100 may edit the audio track of the sound image 1 and the description information about the audio track of the sound image 1 in the metadata. The electronic device 100 may generate an audio file of the sound image 1 based on the audio track of the sound image 1 and the metadata related to the sound image 1. The metadata related to the sound image 1 may include the description information about the audio track of the sound image 1, for example, the volume, timbre, spatial position, etc. of the sound image 1 so that the audio track of the sound image 1 can be correctly rendered. The electronic device 100 may render the audio file of the sound image 1 to obtain the audio data of the sound image 1, and hand over the audio data of the sound image 1 to the monitoring device 801 for playback. The monitoring device 801 may only play the audio of the sound image 1. In this way, after adjusting the timbre of the sound image 1, the user can understand the sound effect of the sound image 1 in real time through the monitoring device 801 to determine whether the adjustment of the timbre of the sound image 1 is appropriate.

[0396] Optionally, within a preset time period starting from the moment when the timbre of the sound image 2 is adjusted, if the electronic device 100 does not detect an audio editing operation again, the electronic device 100 may continue to instruct the monitoring device 801 to play all audios during the video recording process.

[0397] It can be understood that when the monitoring device 801 plays the audio of the sound image 1, when the operation of editing other sound images, such as the sound image 2, is detected, the electronic device 100 can edit the audio track of the sound image 2 and the description information about the sound image 2 in the metadata. The electronic device 100 can generate an audio file of the sound image 2 based on the sound image 2 and the metadata related to the sound image 2. The electronic device 100 can render the audio file of the sound image 2 to obtain the audio data of the sound image 2, and hand over the audio data of the sound image 2 to the monitoring device 801 for playback. Then, the content played by the monitoring device 801 can be switched from the audio of the above-mentioned sound image 1 to the audio of the sound image 2.

[0398] It can be seen from the above embodiment that after the electronic device 100 receives the audio editing operation, the part of the audio that causes the audio editing operation to change can be played through the monitoring device 801. In this way, the user can monitor the sound effects of the part he edited in real time during the process of shooting the video to determine whether the sound effects of the separately edited part of the audio are appropriate. In addition, after the user stops audio editing, the electronic device 100 can play all audio during the video recording process through the monitoring device 801, and the user can determine whether the overall playback sound effect of the video is appropriate. The above embodiment can improve the convenience of the user to perform audio editing when recording a video, and help the user to confirm whether the content of the audio editing is appropriate more quickly.

[0399] In some embodiments, the electronic device 100 can also respond to a preset operation for one or more sound images and instruct the monitoring device 801 to play the audio of the one or more sound images. In other words, the user can actively choose to listen to the audio of a single or multiple sound images. The embodiment of the present application does not limit the method of the above preset operation.

[0400] In some embodiments, the electronic device 100 may provide a function of adjusting the spatial position of the sound image. The user may change the spatial position of the sound image so that the sound of the sound image is heard as if it is emitted from the spatial position after the change.

[0401] Figures 9A to 9C Schematic diagrams of other audio editing scenarios are shown as examples.

[0402] like Fig. 9A As shown, the electronic device 100 can display a user interface 910. The user interface 910 can refer to the aforementioned Figure 6C The user interface 620 shown. The user interface 910 may include a marker box 911 associated with the audio image 1. Fig. 9A As shown, for an operation on the area where the mark box 911 is located, such as a long press operation, the electronic device 100 can display Fig. 9B The pan / tilt 1 control 912 is shown. Fig. 9B As shown in the operation of dragging the audio and video 1 control 912, the electronic device 100 can display Fig. 9C The prompt box 921 shown. The prompt box 921 can be used for the user to adjust the spatial position of the sound image 1.

[0403] like Fig. 9C As shown, the prompt box 921 may include a horizontal angle control 922, a pitch angle control 923, a confirmation control 924, and a cancel control 925. Among them, the horizontal angle control 922 can be used to adjust the horizontal angle of the sound image 1 relative to the listener. The horizontal angle can be used to reflect in which direction of the listener the sound image 1 is located in the horizontal direction, for example, on the left side of the listener, or in the front left, or in the front right, or on the right side, or on the back side, etc. The pitch angle control 923 can be used to adjust the pitch angle of the sound image 1 relative to the listener. The pitch angle can be used to reflect in which direction of the listener the sound image 1 is located in the vertical direction, for example, above, below, etc. The above-mentioned position of the listener may be the position of the electronic device 100, or may be the aforementioned Fig. 8A The position of the monitoring device 801 is shown. The embodiment of the present application does not limit the position of the listener.

[0404] The confirmation control 924 can be used to trigger the electronic device 100 to adjust the spatial position of the sound image 1 according to the values ​​of the horizontal angle and the pitch angle in the prompt box 921. The cancellation control 925 can be used to cancel the adjustment of the spatial position of the sound image 1.

[0405] Understandably, Figures 9A to 9C The operation for adjusting the spatial position of the sound image is only an exemplary description of the present application and should not be construed as limiting the present application. The electronic device 100 may also support the user to adjust the spatial position of the sound image through other operation modes.

[0406] In some embodiments, in addition to adjusting the horizontal angle and the pitch angle of the sound image relative to the listener, the electronic device 100 may also support the user to adjust the distance of the sound image relative to the listener and other azimuth information.

[0407] In some embodiments, the electronic device 100 can also support the user to exchange the audio tracks of two audio and video. Exemplarily, after the electronic device 100 divides the sound object into tracks, determines the spatial position of the audio and video, and the display position on the video image, it can determine that the target 1 on the image is determined as the audio and video 1, and the audio track is the audio track 1, and determines that the target 2 on the image is determined as the audio and video 2, and the audio track is the audio track 2. When the operation of exchanging the audio and video 1 and the audio and video 2 is detected, the electronic device 100 can determine the target 1 on the image as the audio and video 2, and the audio track is the audio track 2, and determine the target 2 on the image as the audio and video 1, and the audio track is the audio track 1. After the above exchange, after the electronic device 100 detects the operation of audio editing for the above-mentioned target 1 on the image, it can edit the audio track 2 and the description information about the audio track 2 in the metadata. Similarly, after the operation of audio editing for the above-mentioned target 2 on the image is detected, the electronic device 100 can edit the audio track 1 and the description information about the audio track 1 in the metadata. For example, when the user believes that the audio tracks of the above-mentioned audio and video 1 and the audio and video 2 are opposite, the user can exchange the audio tracks of the audio and video 1 and the audio and video 2. This can improve the accuracy of panning audio tracks.

[0408] In the video recording stage, the scenarios of sound field editing and mixing editing by the user can refer to the introduction of the video pre-recording stage above, which will not be described in detail here.

[0409] The various embodiments introduced in the video recording stage can also be applied to all scenarios in this application, such as the scenario in the video pre-recording stage, as well as the scenario after the video recording is completed and the video call scenario to be described later.

[0410] After the video recording is completed

[0411] In some embodiments, the user can edit the audio of the video stored in the electronic device 100. The electronic device 100 can store the video image in the video and the audio file containing the audio signal and metadata. In this way, the electronic device 100 can edit the audio file to achieve the effect of editing the sound object, the sound field and the mixing.

[0412] FIG. 10A to FIG. 10D Schematic diagrams of other audio editing scenarios are shown as examples.

[0413] like Fig. 10A As shown, electronic device 100 may display user interface 510. User interface 510 may include gallery application icon 512. Gallery application icon 512 may be used to open the gallery application. Fig. 10A As shown in the operation of the gallery application icon 512, the electronic device 100 can display Fig. 10B User interface 1010 is shown.

[0414] like Fig. 10BAs shown, the user interface 1010 may include a video control 1011 for video 1. The video 1 may be, for example, a video recorded in the aforementioned video recording stage. The embodiment of the present application does not limit the content of the video 1. The electronic device 100 may store the video image and audio file of the video 1 so that the video 1 can be correctly rendered and played. The audio file may include the audio signal of the sound field and each sound object and metadata.

[0415] In response to an operation on the video control 1011, such as a click operation, the electronic device 100 may display Fig. 10C The user interface 1020 shown. The user interface 1020 may include a play control 1021 and an edit control 1022. The play control 1021 may be used to trigger the electronic device 100 to play the video 1. The edit control 1022 may be used to trigger the electronic device 100 to display a user interface for editing the video 1, so that the user can edit the video 1 (such as audio editing). Fig. 10C For the operation of the editing control 1022, the electronic device 100 can display Fig. 10D User interface 1030 is shown.

[0416] like Fig. 10D As shown, the video image included in the video 1 may be displayed on the user interface 1030. The user interface 1030 may also include an audio editing control 1031, a progress bar 1032, an audio-visual control 1033, a sound field control 1034, and a mixing control 1035.

[0417] The audio editing control 1031 can be used to trigger the electronic device 100 to turn on or off the audio editing function. When the audio editing function is turned off, in response to the operation of the audio editing control 1031, the electronic device 100 can display the audio editing control 1031 as Fig. 10D Shown in selected state.

[0418] The progress bar 1032 can be used to adjust the playback progress of the video.

[0419] The audio-visual control 1033, the sound field control 1034, and the mixing control 1035 can be used for audio-visual editing, sound field editing, and mixing editing, respectively. For details, please refer to the introduction of the aforementioned embodiments.

[0420] When the audio editing function is turned on, the electronic device 100 can display the above-mentioned audio-visual control 1033, sound field control 1034, mixing control 1035, and the mark boxes, volume bars, mute controls and other controls for audio editing associated with each audio and video on the video image on the user interface 1030. When the audio editing function is turned off, the electronic device 100 can cancel the display of the above-mentioned controls for audio editing on the user interface 1030.

[0421] It should be noted that after the video recording is completed, the user can edit the audio in the existing video by referring to the audio editing performed in the aforementioned video pre-recording stage and the video recording stage. For example, after the video recording is completed, the user can edit the audio in the existing video, including but not limited to: adjusting the volume of the sound and image, adjusting the timbre of the sound and image, adjusting the spatial position of the sound and image, exchanging the audio tracks of two sound and images, adjusting the sound field, mixing and editing, etc. Optionally, when the existing video is played to a segment where there is a part of the sound and image that is not captured, the electronic device 100 can also provide a control for indicating the sound and image that is not captured in the user interface 1030, so as to facilitate editing of the sound and image that is not captured.

[0422] As can be seen from the above embodiments, the user can perform audio editing operations before and during the video shooting process, and adjust the sound field and the individual sound and image so that the video playback sound effects meet personalized needs. In addition, the user can also perform secondary audio editing on the recorded video after the video recording is completed.

[0423] The controls for audio editing can be presented on the video image, and in particular, the controls for audio and video editing can also be displayed near the audio and video captured in the image. This allows users to perform audio editing operations more easily and intuitively, and improves the user experience of recording audio and video, and editing audio.

[0424] Audio editing in video call scenarios

[0425] Figures 11A to 11H Schematic diagrams of other audio editing scenarios are shown as examples.

[0426] like Fig.11A As shown, the electronic device 100 may display a user interface 510. The user interface 510 may include a video call application icon 513. The video call application icon 513 may be used to trigger the electronic device 100 to open the video call application. Fig.11A As shown in the operation of the video call application icon 513, the electronic device 100 can display Fig. 11B User interface 1110 is shown.

[0427] like Fig. 11B As shown, user interface 1110 may include a normal video call control 1111 and a spatial audio and video call control 1112. Normal video call control 1111 may be used to make a video call, and the audio editing function remains turned off during the video call. Spatial audio and video call control 1112 may also be used to make a video call, but the audio editing function remains turned on during the video call.

[0428] In some embodiments, during a video call, the electronic device 100 may provide an audio editing control to turn on or off the audio editing function. In this way, the user may turn on or off the audio editing function at any time during the video call. In other words, the user does not need to decide whether to use the video call with the audio editing function turned on or the video editing function turned off before making a video call. In other words, Fig. 11B The common video call control 1111 and the spatial audio and video call control 1112 shown are optional. The embodiment of the present application does not limit the above operation for turning on or off the audio editing function during a video call.

[0429] In some embodiments, the electronic device 100 enables the audio editing function during a video call, and can support the user of the electronic device 100 to edit the audio collected by the electronic device 100, so as to send the edited audio to the other end of the call. In this way, the audio heard by the other end of the call of the electronic device 100 can be the audio edited by the user of the electronic device 100.

[0430] In some embodiments, the electronic device 100 enables the audio editing function during a video call, and can also support the user on the electronic device 100 side to edit the audio of the other end of the call. For example, during a video call between user 1 on the electronic device 100 side and user 2. User 1 can edit the audio of user 2 on the electronic device 100. In this way, user 1 can listen to the audio of user 2 that has been edited by user 1. For another example, during a video call between user 1 on the electronic device 100 side and multiple other ends of the call (such as user 2, user 3, etc.). User 1 can edit the audio of user 2 on the electronic device 100. In this way, except for the edited call end, that is, user 2, user 1 and other call ends, such as user 3, etc., can listen to the audio of user 2 that has been edited by user 1.

[0431] Optionally, before the user on the electronic device 100 edits the audio of the other party, the user may request the consent of the other party. When the other party agrees, the user on the electronic device 100 may edit the audio of the other party.

[0432] In response to Fig. 11B As shown in the operation of the spatial audio and video call control 1112, the electronic device 100 can enable the audio editing function during a video call.

[0433] like Fig. 11CAs shown, the electronic device 100 may display a user interface 1120. The user interface 1120 may be a video call interface. The electronic device 100 is in a video call with a call partner. The electronic device 100 may be the initiator of the video call or the receiver of the video call. This embodiment of the application is not limited to this.

[0434] like Fig. 11C As shown, the user interface 1120 may display a video image sent by the call partner. The video image sent by the call partner may be Fig. 11C The image shown includes images of audio image a, audio image b, and audio image c. The user interface 1120 may also include a mark box 1121 and a volume control 1121A associated with audio image a, a mark box 1122 and a volume control 1122A associated with audio image b, a mark box 1123 and a volume control 1123A associated with audio image c, a floating window 1124, an audio image control 1125, a sound field control 1126, a mixing control 1127, time information 1128, and an audio editing mark 1129.

[0435] The display style of the volume control 1121A can be Fig. 11C The display style of the non-muted state shown. This can indicate that the volume of the sound image a in the audio of the other end of the call is not 0 (that is, it is not muted).

[0436] The display styles of the volume control 1122A and the volume control 1123A can be Fig. 11C The display style of the mute state shown. This can indicate that the volume of the audio image b and the audio image c in the audio of the other end of the call is 0, that is, it is in the mute state.

[0437] In some embodiments, a sound image may contain multiple objects. Fig. 11C The sound image c shown may include multiple objects (for example, three objects). It is understandable that when multiple objects are close in distance but far away from the audio and video acquisition device, the area occupied by these multiple objects on the acquired image is smaller, and the sounds of these multiple objects can be considered to be emitted from the same spatial position. Compared with the objects that are closer to the audio and video acquisition device, the sounds of the multiple objects that are far away from the audio and video acquisition device are weaker and have less impact on the sounds of the objects that are closer to the audio and video acquisition device. Therefore, the multiple objects that are close in distance but far away from the audio and video acquisition device can be determined as one sound image.

[0438] The floating window 1124 may display an image captured by the electronic device 100. For example, the image captured by the electronic device 100 through a front camera.

[0439] The time information 1128 may be used to indicate the duration of the video call.

[0440] The audio editing mark 1129 may be used to indicate that the audio editing function is turned on. Optionally, when the audio editing function is turned off, the electronic device 100 may cancel the display of the audio editing mark 1129 on the user interface 1120 .

[0441] It can be seen that since the audio editing function is turned on, the electronic device 100 can display the video image of the other end of the call. Fig. 11C The marking boxes and volume controls associated with each audio and video are shown, as well as audio and video controls 1125, sound field controls 1126, and mixing controls 1127. This can facilitate the user on the electronic device 100 side to edit the audio of the other end of the call.

[0442] In one possible implementation, the electronic device 100 receives a video image and an audio file from the other end of the call. The audio file may contain the audio signal of the sound field and each sound and image of the other end of the call, as well as metadata. When receiving an operation to edit the audio of the other end of the call, the electronic device 100 can edit the above audio file to obtain a new audio file. The electronic device 100 can render the new audio file to obtain audio data. Then, the electronic device 100 can use the audio data obtained by the above rendering to play the audio of the other end of the call. In this way, the user on the electronic device 100 side can listen to the audio of the other end of the call that has been edited by himself.

[0443] In another possible implementation, when an operation to edit the audio of the other end of the call is received, the electronic device 100 may send an audio editing instruction to the other end of the call. The audio editing instruction may include audio editing parameters for performing audio editing. After receiving the audio editing instruction, the other end of the call may edit the collected audio according to the audio editing parameters to obtain an audio file. The other end of the call may render the audio file to obtain audio data. Then, the other end of the call may send the audio data to the electronic device 100. The electronic device 100 may play the audio of the other end of the call based on the received audio data. Among them, the audio data received by the electronic device 100 is rendered from an audio file edited according to the audio editing instruction of the electronic device 100. Therefore, the user on the electronic device 100 side can listen to the audio of the other end of the call that has been edited by himself.

[0444] In another possible implementation, when receiving an operation to edit the audio of the other end of the call, the electronic device 100 may send an audio editing instruction to the server. The audio editing instruction may include audio editing parameters for audio editing. The above-mentioned server may be a server for implementing a video call between the electronic device 100 and the other end of the call. The server may also receive a video image and an audio file from the other end of the call. The audio file may include the audio signal of the sound field and each sound and image of the other end of the call, as well as metadata. The server may edit the audio file of the other end of the call according to the audio editing parameters of the electronic device 100 to obtain a new audio file. Then, the server may send the new audio file to the electronic device 100, and the electronic device 100 renders and plays the above-mentioned new audio file. Alternatively, the server may render the above-mentioned new audio file to obtain audio data, and hand over the audio data to the electronic device 100 for playback. In this way, the user on the electronic device 100 side can listen to the audio of the other end of the call that has been edited by himself.

[0445] That is to say, the electronic device 100 can support the local user to edit the audio of the other end of the call. The device that specifically edits the audio file can be the electronic device 100, or can be the other end of the call, or can be a server.

[0446] like Fig. 11C As shown, in response to the operation of the volume control 1122A, the electronic device 100 may display Fig.11D Volume bar 1122B is shown. Volume bar 1122B can be used to adjust the volume of audio image b.

[0447] like Fig.11D As shown, in response to the operation of the audio and video control 1125, the electronic device 100 can display Fig.11E The audio image a option 1125A, the audio image b option 1125B, and the audio image c option 1125C are shown. The audio image a option 1125A, the audio image b option 1125B, and the audio image c option 1125C can be used to select the audio image a, the audio image b, and the audio image c, respectively, so that the user can edit the selected audio image.

[0448] In some embodiments, in addition to the above-mentioned options corresponding to the sound and image of the other end of the video call, the electronic device 100 can also provide options corresponding to the sound and image on the video image in the floating window 1124 (that is, options corresponding to the sound and image on the electronic device 100 side).

[0449] In response to Fig.11E By operating the floating window 1124, such as clicking, the electronic device 100 can exchange the display positions of the video image captured by the electronic device 100 and the video image from the call partner. Fig.11FUser interface 1130 is shown.

[0450] like Fig.11F As shown, the user interface 1130 may display a video image captured by the electronic device 100. The video image captured by the electronic device 100 may be Fig.11F The user interface 1130 may also include a mark box 1131 and a volume control 1131A associated with the sound image d, a mark box 1132 and a volume control 1132A associated with the sound image e, and a floating window 1133. Other controls on the user interface 1130 may refer to the aforementioned Fig. 11C User interface 1120 is shown.

[0451] The floating window 1133 may display a video image sent by the call partner of the electronic device 100 .

[0452] The user interface 1130 may include an audio-visual control 1125. Since the display positions of the video image captured by the electronic device 100 and the video image from the call partner are swapped, in response to the operation of the audio-visual control 1125, the electronic device 100 may display Fig.11F The options corresponding to the audio and video on the electronic device 100 are: audio and video d option 1125D, audio and video e option 1125E. In this way, the user on the electronic device 100 can select the audio and video on the local side for editing.

[0453] exist Fig.11F In the user interface 1130 shown, the user on the electronic device 100 side can edit the audio of the local end. Among them, the electronic device 100 can edit the audio collected by the electronic device 100 according to the user's operation of editing the audio on the local end, and obtain an audio file containing an audio signal and metadata. Then, the electronic device 100 can send the audio file to the call partner, or send the audio data rendered by the audio file to the call partner. In this way, the call partner can listen to the audio edited by the user on the electronic device 100 side.

[0454] If the electronic device 100 sends an audio file to the other end of the call, the electronic device 100 may first encode the audio file. For example, the above encoding may include one or more of the following: channel encoding, sound object encoding, HOA encoding, metadata encoding, etc. The electronic device 100 may send the encoded audio file to the other end of the call. The above encoding may reduce information redundancy in the audio file and improve the transmission efficiency of the audio file.

[0455] In some embodiments, the electronic device 100 can enlarge the floating window on the video call interface so that the user can select the audio and video on the video image in the floating window for audio and video editing.

[0456] For example, Fig.11G As shown, in response to the operation of sliding two fingers in opposite directions on the floating window 1133, the electronic device 100 can enlarge the floating window 1133 and display Fig.11H The floating window 1133 shown. It can be seen that Fig.11H The floating window 1133 shown is Fig.11G The floating window shown is large.

[0457] like Fig.11H As shown, when the floating window 1133 is enlarged, the video image displayed in the floating window 1133 is also enlarged. The electronic device 100 can display the audio editing controls associated with each sound and image of the video image in the floating window so that the user can edit the sound and image. It can be understood that when the floating window 1133 is small, the size of the sound and image presented in the floating window is also small, or it is possible that only a part of the captured sound and image is presented. This is not convenient for the user to select the sound and image on the video image in the floating window for editing. Enlarging the floating window and the video image displayed therein allows the user to more conveniently and accurately select the sound and image for editing.

[0458] Optionally, without enlarging the floating window, the electronic device 100 may also provide audio editing controls associated with each sound and image of the video image in the floating window, so that the user can edit the sound and image displayed in the floating window.

[0459] It should be noted that, in the case where the content displayed in the floating window 1133 is a video image on the electronic device 100 side (refer to Fig. 11C The electronic device 100 may also support the user to enlarge the floating window and edit the audio and video on the electronic device 100 side.

[0460] The audio editing performed by the user of the electronic device 100 can refer to the embodiment of audio editing in the aforementioned video recording scenario, which will not be described in detail here.

[0461] In some embodiments, the electronic device 100 can identify the main object on the video image, retain the sound of the main object in the audio, and suppress all sounds except the main object in the audio. In this way, the user can quickly eliminate all sounds except the main object in the audio during a video call, so that the sound of the main object can be clearly heard.

[0462] Figures 12A to 12C Schematic diagrams of other audio editing scenarios are shown as examples.

[0463] like Fig. 12A As shown, the electronic device 100 can display a user interface 1130. The user interface 1130 can refer to the aforementioned Fig.11FThe user interface 1130 may include a play mode control 1134. The play mode control 1134 may be used to select a play mode of the audio. The play mode of the audio may include a natural mode and a vocal mode.

[0464] In the natural mode, the electronic device 100 can retain the sounds of all sound images in the audio collected by the electronic device 100.

[0465] In the human voice mode, the electronic device 100 can identify the main object on the video image captured by the electronic device 100. For example, the main object can be an object that occupies the largest display area on the video image. The embodiment of the present application does not limit the method by which the electronic device 100 determines the main object on the image. When the main object is determined, the electronic device 100 can edit the audio tracks of other audio and video except the main object so that the volume of other audio and video except the main object is 0 (i.e., remains silent).

[0466] The text displayed on the play mode control 1134 is: Fig. 12A The "Natural Mode" shown in the figure indicates that the audio playback mode is currently in the Natural Mode. Fig. 12A It can be seen that the display styles of the volume control 1131A associated with the sound image d and the volume control 1132A associated with the sound image e are both in the non-mute state. That is, the audio signals of the sound image d and the sound image e can be collected and retained by the electronic device 100. The other end of the call of the electronic device 100 can hear the sound of the sound image d and the sound image e.

[0467] In response to Fig. 12A For the operation of the play mode control 1134, the electronic device 100 may display Fig. 12B The natural mode option 1134A and the vocal mode option 1134B are shown. The natural mode option 1134A can be used to select the natural mode. The vocal mode option 1134B can be used to select the vocal mode.

[0468] like Fig. 12B As shown, in response to the operation of the vocal mode option 1134B, the electronic device 100 may change the text content displayed on the play mode control 1134 to Fig. 12C The "Vocal Mode" is shown. This indicates that the audio playback mode is currently in the vocal mode.

[0469] like Fig. 12C As shown in FIG. 1 , when switching from the natural mode to the human voice mode, the electronic device 100 can identify the main object on the video image captured by the electronic device 100. For example, the main object is Fig. 12CThe electronic device 100 can edit the audio tracks of other audio images other than the audio image d, such as the audio image e, so that the audio image e remains silent. It can be seen that since the electronic device 100 edits the audio image e to make it silent, the display style of the volume control 1132A associated with the audio image e can be changed from Fig. 12A The display style changes for the non-muted state shown as Fig. 12C The display style of the mute state is shown. That is, the electronic device 100 can suppress the audio signal of the sound image e. The other end of the call of the electronic device 100 will only hear the sound of the main object (i.e., the sound image d), and cannot hear the sound of other sound images on the electronic device 100 side except the above main object.

[0470] In some embodiments, when switching to the human voice mode, the electronic device 100 may provide one or more options corresponding to the sound image, allowing the user to select a sound image as the main object. In this way, the user can keep the sound of a sound image as needed and mute the other sound images.

[0471] It can be understood that, in addition to muting other sound images except the main object by selecting the audio playback mode, the user can also manually edit one or more sound images to mute the one or more sound images.

[0472] In some embodiments, the above-mentioned human voice mode can be applied to the audio collected by the electronic device 100 as well as the audio of the other end of the call of the electronic device 100. Among them, the electronic device 100 can receive the video image and audio file of the other end of the call. The audio file can realize the audio playback of the other end of the call after rendering. When the operation of applying the human voice mode to the audio of the other end of the call is detected, the electronic device 100 can identify the main object according to the video image of the other end of the call, and edit the audio file of the other end of the call according to the main object. The electronic device 100 can retain the audio signal of the main object in the audio file of the other end of the call, and adjust the other sound objects of the other end of the call except the main object to be muted. Alternatively, when the operation of applying the human voice mode to the audio of the other end of the call is detected, the electronic device 100 can send an audio editing instruction. The audio editing instruction can be used to instruct to mute other sound and images of the other end of the call of the electronic device 100 except the main object. The device that performs specific editing processing on the audio file of the other end of the call can be the device of the other end of the call or can be a server. The electronic device 100 can receive audio data of the other end of the call. The audio data can be audio data in which other sound and images of the other end of the call except the main object are muted. In this way, the user on the electronic device 100 side can only listen to the sound of the main object on the other end of the call, but cannot hear the sound of other audio and video on the other end of the call except the main object.

[0473] It can be seen from the above embodiments that in the scenario of a video call, the user on the electronic device 100 side can suppress the sound of other sounds and images on this side except the main object with one click. This can facilitate the user on the other end of the call with the electronic device 100 to only listen to the sound of the main object on the electronic device 100 side without being disturbed by other sounds and images on the electronic device 100 side. In addition, the user on the electronic device 100 side can also suppress the sound and images of other sounds and images on the other end of the call except the main object with one click. This can facilitate the user on the electronic device 100 side to only listen to the sound of the main object on the other end of the call without being disturbed by other sounds and images on the other end of the call. The above embodiments can improve the user's video call experience.

[0474] Above Figures 12A to 12C The embodiment shown can also be applied to the aforementioned video recording scene. In this way, the user can suppress the sound of other sound images except the main object in the captured video with one click, making the audio in the captured audio more pure.

[0475] In some embodiments, the electronic device 100 can make video calls with multiple call peers. The electronic device 100 can edit the audio of one or more call peers. When the electronic device 100 edits the audio of one call peer, the audio of the above-mentioned call peer heard by other call peers may be the audio edited by the electronic device 100.

[0476] Fig.13A and Fig. 13B Schematic diagrams of other audio editing scenarios are shown as examples.

[0477] like Fig.13A As shown, the electronic device 100 can display a user interface 1310. The user interface 1310 can be an interface for video calls between the electronic device 100 and multiple call peers. Here, a video call between the electronic device 100 and three call peers is used as an example for explanation. Among them, the call end where the electronic device 100 is located can be called room A. The three call peers of the electronic device 100 can be respectively called: room B1, room B2, and room B3. The embodiment of the present application does not limit the name of each call end.

[0478] The user interface 1310 may include a display area 1311 , a display area 1312 , a display area 1313 , and a display area 1314 .

[0479] Display area 1311 may be used to display the video image sent by the call end corresponding to room B1. Display area 1311 may include a mute control 1311A. Mute control 1311A may be used to mute or unmute the entire room B1. Among them, the entire muting of room B1 may mean that all call ends in the video call will not hear any sound in room B1.

[0480] Display area 1312 may be used to display the video image sent by the call end corresponding to room B2. Display area 1312 may include a mute control 1312A. Mute control 1312A may be used to mute or unmute the entire room B2.

[0481] Display area 1313 may be used to display the video image sent by the call end corresponding to room B3. Display area 1313 may include a mute control 1313A. Mute control 1313A may be used to mute or unmute the entire room B3.

[0482] The display area 1314 may be used to display the video image of room A, that is, the video image captured by the electronic device 100. The display area 1314 may include a mute control 1314A. The mute control 1314A may be used to mute or unmute the entire room A.

[0483] For example, the video image captured by the electronic device 100 may include: audio and video 11, audio and video 12, and audio and video 13. The electronic device 100 may display one or more of the following controls for audio editing in the display area 1314: volume bars and mute controls associated with each audio and video, audio and video controls 1315, sound field controls 1316, mixing controls 1317, and play mode controls 1318. The functions of the above-mentioned controls for audio editing can refer to the introduction of the above-mentioned embodiments. The user on the electronic device 100 side can perform audio editing such as audio and video editing, sound field editing, and mixing editing on the audio in room A through the controls in the above-mentioned display area 1314.

[0484] When the audio in room A is edited, the three call counterparts of the electronic device 100, namely, users in room B1, room B2, and room B3, can all listen to the edited audio in room A.

[0485] In response to Fig.13A As shown in FIG. 1 , for an operation performed on the display area 1311, such as a click operation, the electronic device 100 may display Fig. 13B User interface 1320 is shown. Fig.13A and Fig. 13B It can be seen that the electronic device 100 can exchange the display areas of the video image of room A and the video image of room B1.

[0486] like Fig. 13B As shown, user interface 1320 may include display area 1321 , display area 1312 , display area 1313 , and display area 1322 .

[0487] The display area 1321 may be located at Fig.13A The location of display area 1311 is shown. Display area 1321 can be used to display the video image of room A.

[0488] The display area 1322 may be displayed on the original display Fig.13A The display area 1314 is shown in FIG. 1322. The display area 1322 can be used to display the video image sent by the call end corresponding to room B1. For example, the video image sent by the call segment corresponding to room B1 can include audio image 21 and audio image 22. The electronic device 100 can display controls for audio editing such as volume bars and mute controls associated with each audio image in the display area 1322. The controls for audio editing in the display area 1322 can refer to FIG. Fig.13A The control for performing audio editing in the display area 1314 is shown. The electronic device 100 can perform audio editing such as sound and image editing, sound field editing, and mixing editing on the audio in the room B1 through the control for performing audio editing in the display area 1322.

[0489] When the user on the electronic device 100 side edits the audio in room B1, the electronic device 100 and the other two call partners of the electronic device 100, namely the users in room A, room B2 and room B3, can all listen to the edited sound in room B1.

[0490] For example, during a video call, the call end corresponding to room B1 can determine the spatial position of each sound image in room B1 relative to a preset position. The preset position may be the position of a device used for making an audio or video call in room B1. Here, the preset position is taken as position 0 in room B1. The present application embodiment does not limit the position 0. For example, the call end corresponding to room B1 can determine Fig. 13B The sound image 21 shown is located at position 1 relative to position 0, and the sound image 22 is located at position 2 relative to position 0. The spatial positions of the above-mentioned sound image 21 and sound image 22 may be included in the audio file generated by the call end corresponding to room B1. When the electronic device 100 plays the audio from the call end corresponding to room B1, the user on the electronic device 100 side (i.e., the user in room A) can listen to the sound of the sound image 21 and sound image 22 in room B1, and experience the spatial sense and sense of orientation of the sound. For example, the user in room A can be placed at position 0 in room B2 in terms of hearing, and feel that the sound of the sound image 21 is emitted from the above-mentioned position 1, and the sound of the sound image 22 is emitted from the above-mentioned position 2. Similarly, when users of other call ends (such as the call end corresponding to room B2 and the call end corresponding to room B3) listen to the audio in room B1, they can have the same sense of hearing as the user on the electronic device 100 side.

[0491] The user on the electronic device 100 side can adjust the spatial position of the sound image in the room B1. For example, the spatial position of the sound image 22 is moved from the above position 2 to position 3. Then, when the electronic device 100 plays the audio from the call end corresponding to the room B1, the user in the room A can be placed at the position 0 in the room B2 in terms of hearing, and feel that the sound of the sound image 21 is emitted from the above position 1, and the sound of the sound image 22 is emitted from the above position 3. Similarly, when the users of other call ends (such as the call end corresponding to the room B2 and the call end corresponding to the room B3) listen to the audio in the room B1, they can also feel the direction of the sound image 22 moving from the above position 2 to the position 3.

[0492] From the above audio editing scenario, it can be seen that the user can edit the audio of the local end during the video call, so that the other end of the call can hear the audio edited by the user. In addition, the user can also edit the audio of the other end of the call during the video call, so that he and other call ends can hear the audio of the edited call end after the user edits it. In this way, the user can control the audio of multiple call ends connected to the video call as a host, so that the video call can be carried out better. The above embodiment can facilitate users to edit call audio easily and intuitively during video calls, thereby improving the user's video call experience.

[0493] Fig.14 The following is a flowchart of an audio processing method provided by an embodiment of the present application.

[0494] like Fig.14 As shown, the audio processing method may include steps S1411 to S1414. Steps S1411 to S1414 may be collectively referred to as step S1400. The execution device of steps S1411 to S1414 may be, for example, the electronic device 100.

[0495] S1411. Collect video images and audio.

[0496] The process of the electronic device 100 collecting video images and audio can refer to the aforementioned Figure 3 An introduction to the acquisition unit 310 is shown.

[0497] S1412: Determine audiovisual information and sound field environment information according to the video image and audio. The audiovisual information includes: the audio track of the audio and video contained in the audio, the spatial position of the audio and video, and the display position of the audio and video on the video image.

[0498] The sound field environment information may include, but is not limited to: one or more of: reverberation duration, room type, room size and / or room reflective material.

[0499] The above process of determining the audio-visual information and the sound field environment information can refer to the above Figure 3 Introduction of information extraction unit 320.

[0500] S1413. Receive an audio editing operation, and determine audio editing parameters according to the audio editing operation. The audio editing operation includes performing one or more of the following editing operations on the audio: audio and video editing, sound field editing, and mixing editing.

[0501] The above audio editing operation can be specifically referred to the audio editing operation introduced in the aforementioned video recording scenario and video call scenario, which will not be repeated here.

[0502] S1414. Determine an audio file according to audio editing parameters, audio-visual information, and sound field environment information.

[0503] The audio file may contain the audio signal of the sound field and each sound image, as well as metadata. The metadata may include the spatial position of the sound image, sound field environment information, and other data used to enable the audio signal to be correctly rendered.

[0504] In some embodiments, the electronic device 100 can generate audio files in multiple formats, such as stereo audio files, surround audio files, and 3D audio files. Rendering and playing audio files in different formats can allow users to experience different sound effects.

[0505] In some embodiments, the electronic device 100 may send the audio file to other devices. The electronic device 100 may encode the audio file before sending it to other devices. The above encoding may include but is not limited to: channel encoding, sound object encoding, HOA encoding, and metadata encoding. The above encoding can reduce information redundancy in the audio file and improve the transmission efficiency of the audio file.

[0506] Fig.15 The following is a flowchart of another audio processing method provided by an embodiment of the present application.

[0507] like Fig.15 As shown, the audio processing method may include steps S1511 to S1513. The execution device of steps S1511 to S1513 and steps S1411 to S1414 may be, for example, the electronic device 100.

[0508] S1511, obtain the video image and audio file 1 of video 1.

[0509] The electronic device 100 may store a video 1. Alternatively, the electronic device 100 may receive a video 1 sent by another device. The video 1 may include a video image and an audio file 1.

[0510] The audio file 1 may contain audio signals of the sound field and each sound image, as well as metadata.

[0511] S1512: Receive an audio editing operation, and determine audio editing parameters according to the audio editing operation. The audio editing operation includes performing one or more of the following editing operations on the audio: audio and video editing, sound field editing, and mixing editing.

[0512] S1513. Edit audio file 1 according to the audio editing parameters to obtain audio file 2, and store audio file 2 as the audio file of video 1.

[0513] Editing the audio file 1 according to the audio editing parameters may specifically include editing the audio signals (ie, audio tracks) of the sound field and each sound image according to the audio editing parameters, and editing the metadata.

[0514] The electronic device 100 can replace the audio file 1 in the video 1 with the audio file 2. In this way, when playing the video 1, the electronic device 100 can display the video image in the video 1 and play the audio according to the audio file 2.

[0515] By the above Fig.15 As can be seen from the method shown, the user can edit the existing video in the electronic device 100 to change the playback effect of the audio in the video.

[0516] Fig.16 The following is a flowchart of another audio processing method provided by an embodiment of the present application.

[0517] like Fig.16 As shown, the audio processing method may include steps S1611 to S1623. The audio processing method may be applied to a video call scenario.

[0518] 1. (S1611-S1615) Edit the local audio.

[0519] S1611. Electronic device 100, electronic device 101, and electronic device 102 access the same video call.

[0520] The electronic device 100 executes the aforementioned Fig.14 In step S1400 shown, an audio file is obtained.

[0521] S1612. The electronic device 100 renders the audio file to obtain audio data 1.

[0522] S1613 . Electronic device 100 sends audio data 1 to electronic device 101 and electronic device 102 .

[0523] S1614 , the electronic device 101 plays audio according to the audio data 1 .

[0524] S1615 , the electronic device 102 plays audio according to the audio data 1 .

[0525] It can be understood that, in addition to playing audio according to the audio data 1 , the electronic device 101 and the electronic device 102 can also display video images from the electronic device 100 .

[0526] In some embodiments, the electronic device 100 may send the audio file to the electronic device 101 and the electronic device 102. Then, the electronic device 101 and the electronic device 102 render the audio file to obtain the above-mentioned audio data 1, and play the audio according to the audio data 1.

[0527] In some other embodiments, the electronic device 100 may send the audio file to a server (eg, Figure 4C The server 400 shown in the figure). The server can aggregate the audio files of each call end. The server can render the audio file of the electronic device 100 to obtain the above-mentioned audio data 1. Then, the server can send the audio data 1 to the electronic device 101 and the electronic device 102 for them to play audio.

[0528] 2. (S1616-S1623) Edit the audio of the other end of the call.

[0529] S1616: The electronic device 100 receives an operation to edit the audio of the electronic device 101 and determines audio editing parameters.

[0530] S1617: The electronic device 100 sends an audio editing instruction to the electronic device 101, where the audio editing instruction includes audio editing parameters.

[0531] S1618. The electronic device 101 edits the collected audio according to the audio editing instruction to determine the audio file a.

[0532] The method for electronic device 101 to determine audio file a can refer to the aforementioned Fig.14 The method flow chart is shown.

[0533] S1619. The electronic device 101 renders the audio file a to obtain audio data 2.

[0534] S1620 , electronic device 101 sends audio data 2 to electronic device 100 .

[0535] S1621 . Electronic device 101 sends audio data 2 to electronic device 102 .

[0536] S1622 : The electronic device 100 plays audio according to the audio data 2 .

[0537] S1623. The electronic device 102 plays audio according to the audio data 2.

[0538] It can be understood that, in addition to playing audio according to the audio data 2 , the electronic device 100 and the electronic device 102 can also display the video image from the electronic device 101 .

[0539] In some embodiments, the electronic device 100 may send an audio editing instruction to the server. The server may also receive an audio file b from the electronic device 101. The audio file b may be an audio file that has not been edited according to the audio editing instruction of the electronic device 100. The server may edit the audio file b according to the audio editing instruction of the electronic device 100 to obtain the audio file a. The method for the server to determine the audio file a from the audio file b may refer to the aforementioned Fig.15 Then, the server can send the audio file a to the electronic device 100 and the electronic device 102. Alternatively, the server can render the audio file a to obtain the audio data 2, and send the audio data 2 to the electronic device 100 and the electronic device 102 for audio playback.

[0540] Depend on Fig.16 As can be seen from the method shown, the user can edit the audio of the local end during the video call so that the other end of the call can hear the audio edited by the user. In addition, the user can also edit the audio of the other end of the call during the video call so that he and other call ends can hear the audio of the edited call end after being edited by the user. In this way, the user can control the audio of multiple call ends connected to the video call as a host so that the video call can be carried out better. The above embodiments can facilitate users to edit call audio easily and intuitively during video calls, thereby improving the user's video call experience.

[0541] It is understandable that the various user interfaces described in the embodiments of the present application are only example interfaces and do not limit the present application scheme. In other embodiments, the user interface may adopt a different interface layout, may include more or fewer controls, and may increase or decrease other functional options, as long as they are based on the same inventive concept provided by the present application, they are all within the scope of protection of the present application.

[0542] It should be noted that, without causing any contradiction or conflict, any feature in any embodiment of the present application, or any part of any feature, can be combined, and the combined technical solution is also within the scope of the embodiments of the present application.

[0543] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio processing method, characterized in that: The method comprises: The first electronic device displays a first video image of a first video, the first video image includes a first sound image, and the first video includes a first audio; The first electronic device displays a first sound and image control on a screen, wherein the first sound and image control is used to adjust a sound feature of the first sound and image, wherein adjusting the sound feature of the first sound and image includes one or more of the following: adjusting a volume of the first sound and image, adjusting a timbre of the first sound and image, and / or adjusting a spatial position of the first sound and image; In response to a first operation on the first sound and image control, the first electronic device adjusts a sound feature of the first sound and image in the first audio.

2. The method according to claim 1, characterized in that The first video image also includes a second audio image, and the method further includes: The first electronic device displays a second sound and image control on the screen, wherein the second sound and image control is used to adjust a sound feature of the second sound and image, wherein adjusting the sound feature of the second sound and image includes one or more of the following: adjusting a volume of the second sound and image, adjusting a timbre of the second sound and image, and / or adjusting a spatial position of the second sound and image; In response to a second operation on the second sound and image control, the first electronic device adjusts sound characteristics of the second sound and image in the second audio.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The first electronic device displays a first sound field control on the screen, wherein the first sound field control is used to adjust a first sound field in the first audio, wherein adjusting the first sound field includes one or more of the following: adjusting a type of a room where the first sound field is located, adjusting a size of a room where the first sound field is located, and / or adjusting a reflective material of a room where the first sound field is located; In response to a third operation on the first sound field control, the first electronic device adjusts the first sound field.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The first electronic device displays a first mixing control on the screen, the first mixing control being used to adjust mixing parameters of the first audio, the mixing parameters comprising one or more of the following: a mixing template, a reverberation time, an equalizer parameter, a size of a room where the sound field is located, and / or a reflective material of a room where the sound field is located; The first electronic device determines a first mixing parameter according to a fourth operation on the first mixing control, and mixes the audio signal of the first sound field in the first audio with the audio signals of one or more sound images according to the first mixing parameter.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: The first electronic device receives a fifth operation for the first sound image and the second sound image, wherein the fifth operation is used to merge the first sound image and the second sound image; In response to the fifth operation, the first electronic device displays a third sound and image control on the screen, where the third sound and image control is used to adjust the sound characteristics of the first sound and image and the second sound and image.

6. The method according to any one of claims 1 to 5, characterized in that The first audio further includes an audio signal corresponding to a third sound image, and the third sound image is not in the first video image. The method further includes: The first electronic device displays a fourth sound and image control on the screen, where the fourth sound and image control is used to adjust the sound characteristics of the third sound and image.

7. The method according to claim 6, characterized in that The display position of the fourth sound and image control on the screen is related to the spatial position of the third sound and image.

8. The method according to any one of claims 2 to 7, characterized in that: The sound feature of the first sound image is a first sound feature, the sound feature of the second sound is a second sound feature, and the method further includes: The first electronic device receives a sixth operation for the first sound image and the second sound image, wherein the sixth operation is used to exchange sound features of the first sound image and the second sound image; In response to the sixth operation, the first electronic device determines the sound feature of the first sound image in the first audio as the second sound feature, and determines the sound feature of the second sound image in the first audio as the first sound feature; When receiving an operation on the first audio-visual control, the first electronic device adjusts the second sound feature; When receiving an operation on the second audio-visual control, the first electronic device adjusts the first sound feature.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: The first electronic device receives a seventh operation; In response to the seventh operation, the first electronic device identifies a main object in the first video image; When the main object in the first video image is identified as the first sound and image, the first electronic device removes audio signals of other sound and images in the first audio except for the first sound and image.

10. The method according to any one of claims 1 to 9, characterized in that The display position of the first audio and video control on the screen is related to the display position of the first audio and video on the video image of the first video.

11. The method according to any one of claims 1 to 10, characterized in that Before the first electronic device displays the first video image of the first video, the method further includes: The first electronic device receives an operation of opening a camera application and / or starting to record a video, captures an image through an image acquisition device, and captures audio through an audio acquisition device; wherein the image captured by the image acquisition device includes a video image of the first video, and the audio captured by the audio acquisition device includes the first audio; The first electronic device displays the image captured by the image acquisition device in real time, and the first video image is an image displayed in real time during the capturing process.

12. The method according to any one of claims 1 to 10, characterized in that Before the first electronic device displays the first video image of the first video, the method further includes: The first electronic device obtains the first video from a memory of the first electronic device; The first electronic device receives an operation of playing the first video, and the first video image is an image displayed during the video playback process.

13. The method according to claim 11 or 12, characterized in that: The first electronic device establishes a communication connection with a monitoring device, and the method further includes: The first electronic device sends a second audio to the monitoring device for playback, where the second audio is the first audio after one or more of the following adjustments have been made to the first audio: sound characteristics of one or more sound images in the first audio have been adjusted, the first sound field of the first audio has been adjusted, and / or mixing parameters of the first audio have been adjusted.

14. The method according to claim 11 or 12, characterized in that: The first electronic device establishes a communication connection with a monitoring device. After the first electronic device adjusts the sound feature of the first sound image in the first audio, the method further includes: The first electronic device sends the audio signal of the first sound image after the sound characteristics are adjusted to the monitoring device for playback.

15. The method according to any one of claims 1 to 10, characterized in that The first electronic device and the second electronic device enter into a first video call, and the method further includes: The first electronic device sends a video image of the first video and a second audio to the second electronic device, where the second audio is the audio of the first audio after one or more of the following adjustments are made: sound features of one or more sound images in the first audio are adjusted, a first sound field of the first audio is adjusted, and / or a mixing parameter of the first audio is adjusted; The first video image is an image captured by the first electronic device during the first video call.

16. The method according to claim 15, characterized in that The method further comprises: The first electronic device receives a second video from the second electronic device; The first electronic device displays a second video image of the second video, wherein the second video image includes a fourth sound and image; The first electronic device displays a fifth sound and image control on the screen, and the fifth sound and image control is used to adjust the sound characteristics of the fourth sound and image, and the adjustment of the sound characteristics of the fourth sound and image includes one or more of the following: adjusting the volume of the fourth sound and image, adjusting the timbre of the fourth sound and image, and / or adjusting the spatial position of the fourth sound and image.

17. The method according to claim 16, characterized in that The method further comprises: The first electronic device determines a first editing parameter according to an operation on the fifth audio-visual control; The first electronic device sends a first editing instruction to the second electronic device, where the first editing instruction includes the first editing parameter; The first electronic device receives a third video from the second electronic device, wherein the third video includes a third audio, wherein the sound feature of the fourth sound image in the third audio is adjusted according to the first editing parameter.

18. The method according to claim 17, characterized in that A third electronic device is also connected to the first video call, and the third video is also sent to the third electronic device for playback.

19. The method according to any one of claims 15 to 18, characterized in that The method further comprises: The first electronic device displays a second sound field control on the screen, and the second sound field control is used to adjust the second sound field in the audio contained in the video sent from the second electronic device, and adjusting the second sound field includes one or more of the following: adjusting the type of room where the second sound field is located, adjusting the size of the room where the second sound field is located, and / or adjusting the reflective material of the room where the second sound field is located.

20. The method according to any one of claims 15 to 19, characterized in that The method further comprises: The first electronic device displays a second mixing control on the screen, and the second mixing control is used to adjust the mixing parameters of the audio contained in the video sent from the second electronic device, and the mixing parameters include one or more of the following: mixing template, reverberation duration, equalizer parameters, the size of the room where the sound field is located, and / or the reflective material of the room where the sound field is located.

21. The method according to any one of claims 1 to 10, characterized in that The first electronic device and the second electronic device enter into a first video call, and before the first electronic device displays a first video image of the first video, the method further includes: The first electronic device receives the first video from the second electronic device.

22. An electronic device, characterized in that: The electronic device includes a camera, a microphone array, a memory and a processor, wherein the camera is used to capture images; the microphone array is used to collect audio; the memory is used to store computer programs; and the processor is used to call the computer program so that the electronic device executes the method described in any one of claims 1 to 21.

23. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 21.

24. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are run on an electronic device, the electronic device executes the method according to any one of claims 1 to 21.

Images