Audio processing method and device

Abstract

The embodiment of the present application provides an audio processing method and device, which relates to the field of electronic technology. It can record multiple video images and multiple audios in a multi-channel recording mode in combination with characteristic values ​​such as zoom factors, and can play different audios during video playback to improve the user's audio experience of multi-channel recording. The specific scheme is: the electronic device enters the multi-channel recording mode, and after detecting the user's shooting operation, the first video image and the second video image are recorded simultaneously through the first camera and the second camera, and the audio of multiple sound channels are recorded, and the audio of the multiple sound channels includes panoramic audio, the first audio corresponding to the first video image and the second audio corresponding to the second video image; wherein the electronic device records the first audio in the audio of multiple sound channels, including: the electronic device records the first audio according to the characteristic values ​​such as the zoom factor corresponding to the first display area. The embodiment of the present application is used for multi-channel recording.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of electronic technology, and in particular to an audio processing method and device. Background Art

[0002] As the computing power and hardware capabilities of electronic devices such as mobile phones or tablet computers improve, the recording functions of electronic devices are becoming more and more powerful. For example, some electronic devices can support multi-channel recording, also known as multi-view recording.

[0003] In the existing multi-channel recording mode, the electronic device can record audio and multi-channel video images, for example, the electronic device can record a panoramic video image and a close-up video image respectively. During video playback, the electronic device can play audio and multi-channel video images. Summary of the invention

[0004] The embodiments of the present application provide an audio processing method and device, which can record multi-channel video images and multi-channel audio in combination with characteristic values ​​such as zoom ratio in multi-channel recording mode, and can play different audios during video playback to improve the user's audio experience of multi-channel recording.

[0005] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0006] On the one hand, an embodiment of the present application provides an audio processing method applied to an electronic device, wherein the electronic device includes a first camera and a second camera. The method includes: after the electronic device detects the operation of a user opening a camera application, the shooting preview interface is displayed. Then, the electronic device enters a multi-channel recording mode. After the electronic device detects the user's shooting operation, the shooting interface is displayed, and the shooting interface includes a first display area and a second display area, the first display area is used to display a first video screen captured by the first camera, and the second display area is used to display a second video screen captured by the second camera. Then, the electronic device simultaneously records the first video screen and the second video screen through the first camera and the second camera. The electronic device records audio of multiple sound channels, and the audio of the multiple sound channels includes panoramic audio, a first audio corresponding to the first video screen, and a second audio corresponding to the second video screen. Among them, the electronic device records the first audio in the audio of multiple sound channels, including: the electronic device records the first audio according to the characteristic value corresponding to the first display area, and the characteristic value includes the zoom factor of the first camera.

[0007] In this solution, in multi-channel recording mode, the electronic device can record the audio of multiple sound channels while recording multiple video images. The audio of the multiple sound channels includes panoramic audio and audio corresponding to each video image. Among them, the audio corresponding to the video image is related to the characteristic values ​​such as the zoom factor corresponding to the display area where the video image is located. In this way, when playing back the video, the user can choose to play the panoramic audio or the audio corresponding to the video image they are interested in, thereby improving the user's audio experience.

[0008] In a possible design, the characteristic value also includes a front / rear attribute parameter corresponding to the first display area, and the front / rear attribute parameter is used to indicate whether the first video picture is a video picture taken by the front camera or a video picture taken by the rear camera.

[0009] That is to say, the electronic device can also record the audio corresponding to the first video picture according to the front / rear attribute parameters corresponding to the first display area.

[0010] In another possible design, the characteristic value also includes the distance between the first camera and the frame of the electronic device and / or the focal range of the first camera.

[0011] That is to say, the electronic device can also record the audio corresponding to the first video image according to the distance between the first camera and the frame of the electronic device and / or the focal length range of the first camera.

[0012] In another possible design, before the electronic device records audio of multiple sound channels, the method further includes: the electronic device collects sound signals in all directions and obtains initial audio data according to the sound signals. The electronic device records the first audio according to the characteristic value corresponding to the first display area, including: the electronic device determines the first sound pickup range corresponding to the first display area according to the characteristic value corresponding to the first display area; the electronic device records the first audio according to the first sound pickup range and the initial audio data.

[0013] That is to say, the electronic device can determine the first sound pickup range according to the characteristic value corresponding to the first display area, so as to record the first audio according to the first sound pickup range.

[0014] In another possible design, the audio of multiple sound channels also includes a third audio corresponding to the first video picture.

[0015] That is to say, the electronic device can record the audio of multiple sound channels corresponding to the first video picture.

[0016] In another possible design, the electronic device records the third audio, including: the electronic device determines a reference first sound pickup range based on the first sound pickup range; and records the third audio based on the reference first sound pickup range and initial audio data.

[0017] That is, the electronic device can determine a nearby reference first sound pickup range according to the first sound pickup range, so as to record another audio corresponding to the first video picture according to the reference first sound pickup range.

[0018] In another possible design, the audio of multiple sound channels also includes a fourth audio, the fourth audio includes a fifth audio of a first duration and a sixth audio of a second duration, the fifth audio is a panoramic audio, and the sixth audio includes an audio corresponding to the first zoom factor of the first camera; wherein, within the first duration, the first camera uses the second zoom factor for shooting, and within the second duration, the first camera uses the first zoom factor for shooting.

[0019] In this solution, when the user adjusts the zoom factor in the first display area, the user may want to focus on the video image and sound pickup range corresponding to the adjusted zoom factor. Therefore, the electronic device can synthesize the panoramic audio with the subsequent corresponding audio of the first display area into a dynamically changing panoramic audio, thereby highlighting the audio corresponding to the user adjusting the zoom factor and enhancing the user's experience of audio changes after zooming.

[0020] In another possible design, the first display area also includes a front / rear switching control, and the front / rear switching control is used to switch the front / rear camera.

[0021] In this way, the electronic device can also switch the front camera and the rear camera used in the first display area.

[0022] In another possible design, the electronic device further includes a third camera, and the first display area includes a camera switching control. The method further includes: the electronic device displays a video image captured by the third camera in the first display area in response to an operation on the camera switching control; wherein the first camera is a rear-mounted wide-angle camera, and the third camera is a rear-mounted telephoto camera.

[0023] In this solution, the electronic device can directly switch to the corresponding target camera according to the camera switching control indicated by the user, and use the target camera to capture video images.

[0024] In another possible design, the first display area also includes a first zoom adjustment control, which is used to adjust the zoom ratio of the first camera; the second display area also includes a second zoom adjustment control, which is used to adjust the zoom ratio of the second camera.

[0025] In this way, the first display area and the second display area can both adjust their respective zoom factors.

[0026] In another possible design, the method also includes: when the electronic device determines that the third zoom factor set by the user for the first display area exceeds the focal range of the first camera, the electronic device prompts the user to switch to the fourth camera; and the electronic device displays the video image captured by the fourth camera according to the third zoom factor in the first display area.

[0027] In this solution, when the third zoom ratio exceeds the focal range of the first camera corresponding to the first display area, the electronic device can switch to the fourth camera and directly use the third zoom ratio to capture the video image in the first display area to achieve continuous change of zoom ratio.

[0028] In another possible design, the first display area corresponds to the first focal length range, the second display area corresponds to the second focal length range, and the first focal length range is different from the second focal length range. The method also includes: when the electronic device determines that the fourth zoom multiple set by the user for the first display area exceeds the first focal length range and enters the second focal length range, prompting the user to perform a zoom operation on the second display area; the electronic device displays the video image captured by the second camera according to the fourth zoom multiple in the second display area.

[0029] In this solution, when the fourth zoom ratio exceeds the focal length range corresponding to the first display area and enters the focal length range corresponding to the second display area, the electronic device directly uses the fourth zoom ratio to capture video images in the second display area to achieve continuous change of zoom ratio between different display areas.

[0030] In another possible design, after the electronic device detects the user's shooting operation, the method further includes: after the electronic device detects the user's operation of stopping shooting, generating multiple video recordings, the multiple video recordings include a first video screen, a second video screen, and audio of multiple sound channels. After the electronic device detects the user's operation of playing the multiple video recordings, the video playback interface is displayed, the video playback interface includes the first video screen and the second video screen. The electronic device automatically plays the panoramic audio.

[0031] That is to say, during video playback, the electronic device can play multiple video images and panoramic audio by default.

[0032] In another possible design, the video playback interface also includes an audio switching control. After the electronic device automatically plays the panoramic audio, the method also includes: after the electronic device detects the user's operation on the audio switching control, the first audio or the second audio is played; and the electronic device stops playing the panoramic audio.

[0033] In this way, based on the audio switching control, the electronic device can play corresponding audio according to the user's instructions.

[0034] In another possible design, after the electronic device displays the video playback interface, the method also includes: after the electronic device detects the user's operation of playing the first video screen, it displays the first video screen on the video playback interface and stops displaying the second video screen; the electronic device automatically plays the first audio corresponding to the first video screen.

[0035] In this solution, the electronic device can play a single-channel video screen according to the user's instructions, and automatically play the audio corresponding to the video screen.

[0036] In another possible design, the electronic device displays the first video picture on the video playback interface, including: the electronic device enlarges and displays the first video picture in full screen on the video playback interface.

[0037] In this way, when the user instructs to play only the first video frame, the electronic device can focus on and highlight the first video frame to improve the user's visual experience.

[0038] In another possible design, the first video frame and the second video frame included in the multi-channel recorded videos are saved as the same video file.

[0039] In another possible design, the first video frames included in the multiple-channel recorded videos are saved as a first video file, and the second video frames included in the multiple-channel recorded videos are saved as a second video file.

[0040] In this way, during video playback, the electronic device can play only the first video frame based on the first video file, and can also play only the second video frame based on the second video file.

[0041] In another possible design, before the electronic device displays the shooting interface, the method further includes: the electronic device determines the number of display areas. The electronic device displays the shooting interface, including: the electronic device displays the shooting interface according to the number of display areas.

[0042] In this way, the electronic device can display a corresponding number of display areas on the shooting interface according to the determined number of display areas.

[0043] On the other hand, an embodiment of the present application provides an electronic device, comprising: multiple microphones for collecting sound signals; a first camera and a second camera for collecting video images; a screen for displaying an interface; an audio playback component for playing audio; one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, cause the electronic device to perform the following steps: after detecting an operation of a user opening a camera application, display a shooting preview interface; enter a multi-channel recording mode; after detecting a user's shooting operation, display a shooting interface, the shooting interface including a first display area and a second display area, the first display area being used to display a first video image captured by the first camera, and the second display area being used to display a second video image captured by the second camera; recording the first video image and the second video image simultaneously through the first camera and the second camera; recording audio of multiple sound channels, the audio of the multiple sound channels including panoramic audio, first audio corresponding to the first video image, and second audio corresponding to the second video image; wherein recording the first audio in the audio of the multiple sound channels includes: recording the first audio according to a feature value corresponding to the first display area, the feature value including the zoom factor of the first camera.

[0044] In a possible design, the characteristic value also includes a front / rear attribute parameter corresponding to the first display area, and the front / rear attribute parameter is used to indicate whether the first video picture is a video picture taken by the front camera or a video picture taken by the rear camera.

[0045] In another possible design, the characteristic value also includes the distance between the first camera and the frame of the electronic device and / or the focal range of the first camera.

[0046] In another possible design, when the instruction is executed by the electronic device, it also causes the electronic device to perform the following steps: before recording audio of multiple sound channels, collect sound signals from all directions and obtain initial audio data based on the sound signals; record the first audio based on the characteristic value corresponding to the first display area, including: determining the first sound pickup range corresponding to the first display area based on the characteristic value corresponding to the first display area; and recording the first audio based on the first sound pickup range and the initial audio data.

[0047] In another possible design, the audio of multiple sound channels also includes a third audio corresponding to the first video picture.

[0048] In another possible design, recording the third audio includes: determining a reference first sound pickup range according to the first sound pickup range; and recording the third audio according to the reference first sound pickup range and the initial audio data.

[0049] In another possible design, the audio of multiple sound channels also includes a fourth audio, the fourth audio includes a fifth audio of a first duration and a sixth audio of a second duration, the fifth audio is a panoramic audio, and the sixth audio includes an audio corresponding to the first zoom factor of the first camera; wherein, within the first duration, the first camera uses the second zoom factor for shooting, and within the second duration, the first camera uses the first zoom factor for shooting.

[0050] In another possible design, the first display area also includes a front / rear switching control, and the front / rear switching control is used to switch the front / rear camera.

[0051] In another possible design, the electronic device further includes a third camera, and the first display area includes a camera switching control. When the instruction is executed by the electronic device, the electronic device is further caused to perform the following steps: in response to an operation on the camera switching control, displaying a video image captured by the third camera in the first display area; wherein the first camera is a rear wide-angle camera, and the third camera is a rear telephoto camera.

[0052] In another possible design, the first display area also includes a first zoom adjustment control, which is used to adjust the zoom ratio of the first camera; the second display area also includes a second zoom adjustment control, which is used to adjust the zoom ratio of the second camera.

[0053] In another possible design, when the instruction is executed by the electronic device, it also causes the electronic device to perform the following steps: when it is determined that the third zoom factor set by the user for the first display area exceeds the focal range of the first camera, prompt the user to switch to the fourth camera; and display the video image captured by the fourth camera according to the third zoom factor in the first display area.

[0054] In another possible design, the first display area corresponds to a first focal length range, the second display area corresponds to a second focal length range, and the first focal length range is different from the second focal length range. When the instruction is executed by the electronic device, the electronic device is also caused to perform the following steps: when it is determined that the fourth zoom factor set by the user for the first display area exceeds the first focal length range and enters the second focal length range, the user is prompted to perform a zoom operation on the second display area; in the second display area, the video image captured by the second camera according to the fourth zoom factor is displayed.

[0055] In another possible design, when the instruction is executed by the electronic device, it also causes the electronic device to perform the following steps: after detecting the user's shooting operation and detecting the user's stop shooting operation, generating multi-channel video recordings, the multi-channel video recordings including a first video screen, a second video screen and audio of multiple sound channels; after detecting the user's operation of playing the multi-channel video recordings, displaying a video playback interface, the video playback interface including a first video screen and a second video screen; automatically playing panoramic audio.

[0056] In another possible design, the video playback interface also includes an audio switching control. When the instruction is executed by the electronic device, it also causes the electronic device to perform the following steps: after automatically playing the panoramic audio and detecting the user's operation on the audio switching control, play the first audio or the second audio; stop playing the panoramic audio.

[0057] In another possible design, when the instruction is executed by the electronic device, it also causes the electronic device to perform the following steps: after displaying the video playback interface and detecting the user's operation of playing the first video screen, displaying the first video screen on the video playback interface and stopping displaying the second video screen; and automatically playing the first audio corresponding to the first video screen.

[0058] In another possible design, displaying the first video picture on the video playback interface includes: enlarging and displaying the first video picture on the video playback interface or displaying the first video picture in full screen.

[0059] In another possible design, the first video frames included in the multiple-channel recorded videos are saved as a first video file, and the second video frames included in the multiple-channel recorded videos are saved as a second video file.

[0060] In another possible design, when the instruction is executed by the electronic device, it also causes the electronic device to perform the following steps: before displaying the shooting interface, determine the number of display areas; display the shooting interface, including: displaying the shooting interface according to the number of display areas.

[0061] On the other hand, an embodiment of the present application provides an audio processing device, which is included in an electronic device. The device has the function of realizing the behavior of the electronic device in any of the above aspects and possible designs, so that the electronic device executes the audio processing method executed by the electronic device in any of the possible designs of the above aspects. The function can be implemented by hardware, or by hardware executing the corresponding software implementation. The hardware or software includes at least one module or unit corresponding to the above function. For example, the device may include a multi-channel video recording module, an audio hardware module, an audio processing module, a video playback module, etc.

[0062] In another aspect, an embodiment of the present application provides an electronic device, including: one or more processors; and a memory, wherein a code is stored in the memory. When the code is executed by the electronic device, the electronic device executes the audio processing method executed by the electronic device in any possible design of the above aspects.

[0063] On the other hand, an embodiment of the present application provides a computer-readable storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the audio processing method in any possible design of the above aspects.

[0064] On the other hand, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the audio processing method executed by the electronic device in any possible design of the above aspects.

[0065] On the other hand, an embodiment of the present application provides a chip system, which is applied to an electronic device. The chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the interface circuit is used to receive a signal from a memory of the electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the audio processing method in any possible design of the above aspects.

[0066] The beneficial effects corresponding to the above-mentioned other aspects can be found in the description of the beneficial effects of the method, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of the layout of a microphone provided in an embodiment of the present application;

[0069] Figure 3 A flow chart of an audio processing method provided in an embodiment of the present application;

[0070] Figure 4 A set of interface schematic diagrams provided for embodiments of the present application;

[0071] Figure 5 Another set of interface schematic diagrams provided for embodiments of the present application;

[0072] Figure 6 Another set of interface schematic diagrams provided for embodiments of the present application;

[0073] Figure 7 An audio recording flow chart provided in an embodiment of the present application;

[0074] Figure 8 A schematic diagram of an interface provided in an embodiment of the present application;

[0075] Fig. 9 A schematic diagram of a sound pickup range provided in an embodiment of the present application;

[0076] Fig.10 Another sound pickup range schematic diagram provided for an embodiment of the present application;

[0077] Fig.11A Another set of interface schematic diagrams provided for embodiments of the present application;

[0078] Fig. 11B Another set of interface schematic diagrams provided for embodiments of the present application;

[0079] Fig. 11C Another set of interface schematic diagrams provided for embodiments of the present application;

[0080] Fig.11D Another set of interface schematic diagrams provided for embodiments of the present application;

[0081] Fig.12 Another set of interface schematic diagrams provided for embodiments of the present application;

[0082] Fig.13 Another set of interface schematic diagrams provided for embodiments of the present application;

[0083] Fig.14 Another interface schematic diagram provided for an embodiment of the present application;

[0084] Fig.15 Another set of interface schematic diagrams provided for embodiments of the present application;

[0085] Fig.16 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application;

[0086] Fig.17 A flowchart of another audio processing method provided in an embodiment of the present application;

[0087] Fig.18 A diagram of the interaction relationship between various modules inside an electronic device provided in an embodiment of the present application;

[0088] Fig.19 A flowchart of another audio processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0090] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0091] In the multi-channel video recording (or multi-view video recording) mode, the electronic device can record multi-channel video images during the video recording process, that is, record video images of multiple lines.

[0092] An embodiment of the present application provides an audio processing method in a multi-channel recording mode, where the electronic device can simultaneously record multiple video images and multiple audios in the multi-channel recording mode; when playing back the multi-channel recorded videos (hereinafter referred to as video playback), the electronic device can play different audios, thereby improving the user's audio experience of multi-channel recording.

[0093] For example, in multi-channel recording mode, the electronic device can record multi-channel video images and multi-channel audio in combination with characteristic values ​​such as zoom ratio. The multi-channel audio recorded by the electronic device may include panoramic audio and at least one audio corresponding to each display area. Among them, the panoramic audio records the sound within the panoramic range in all directions around the electronic device. Each video on the display screen of the electronic device corresponds to a display area, and the at least one audio corresponding to each display area can be recorded according to the characteristic values ​​such as zoom ratio currently corresponding to the display area, so that the audio corresponding to each display area corresponds to the video image in the display area.

[0094] In this way, when playing back a video, the electronic device can play the audio that the user is interested in according to the user's instructions, thereby playing the audio corresponding to the video picture displayed in the display area that the user is interested in, thereby improving the user's audio experience.

[0095] In the existing multi-channel recording mode, the electronic device only records one audio channel, and can only play this audio channel during video playback. The playback cannot be switched, and the audio content cannot match the video images in different display areas, and cannot match the video images that the user is paying attention to in real time, resulting in a poor user audio experience.

[0096] The audio processing method provided in the embodiment of the present application can be applied to electronic devices. For example, the electronic device can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) or a special camera (such as a SLR camera, a card camera), etc. The embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0097] For example, Figure 1 A schematic diagram of the structure of the electronic device 100 is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0098] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0099] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0100] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0101] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0102] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0103] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0104] The mobile communication module 150 may provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0105] The wireless communication module 160 can provide wireless communication solutions for application in the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0106] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0107] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0108] In an embodiment of the present application, the display screen 194 can display a shooting preview interface, a video preview interface and a shooting interface in a multi-channel video recording mode, and can also display a video playback interface during video playback, etc.

[0109] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0110] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193. For example, in an embodiment of the present application, ISP can control the photosensitive element to expose and take pictures according to the shooting parameters.

[0111] Camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.

[0112] In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1. Among them, the camera 193 may be located in the edge area of ​​the electronic device, may be an under-screen camera, or may be a liftable camera. The camera 193 may include a rear camera and may also include a front camera. The specific position and form of the camera 193 are not limited in the embodiments of the present application. The electronic device 100 may include cameras of various focal lengths. For example, cameras of different focal lengths may include a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, or a panoramic camera.

[0113] In the embodiment of the present application, in the multi-channel recording mode, different cameras can be used to collect video images corresponding to different zoom factors. For example, a wide-angle camera, an ultra-wide-angle camera, or a panoramic camera can collect video images corresponding to a smaller zoom factor, and a telephoto camera can collect video images corresponding to a larger zoom factor.

[0114] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0115] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0116] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0117] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 .

[0118] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as images, audio data, phone books, etc. collected by the electronic device 100), etc.

[0119] In an embodiment of the present application, the processor 110 can record multiple video images and multiple audios in a multi-channel recording mode by running instructions stored in the internal memory 121, so that the audio corresponding to the video images in different display areas can be played during video playback, so that the played audio matches the video image that the user is paying attention to.

[0120] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the plurality of microphones 170C, the headphone jack 170D, and the application processor.

[0121] The audio module 170 is used to convert digital audio data into analog audio electrical signals for output, and also to convert analog audio electrical signals into digital audio data. The audio module 170 may include an analog / digital converter and a digital / analog converter. For example, the audio module 170 is used to convert the analog audio electrical signals output by the microphone 170C into digital audio data.

[0122] The audio module 170 may also include an audio processing module. The audio processing module is used to perform audio processing on digital audio data in a multi-channel recording mode, thereby generating multi-channel audio. For example, the audio processing module may be used to determine the pickup range of the audio corresponding to the display area according to characteristic values ​​such as the zoom factor corresponding to the display area, enhance the sound within the pickup range, and weaken the sound outside the pickup range, and then perform multi-sound effect processing, etc., so as to record and generate an audio file corresponding to the display area.

[0123] The audio module 170 may also be used to encode and decode audio data.

[0124] In some embodiments, the audio module 170 may be disposed in the processor 110 , or some functional modules of the audio module 170 may be disposed in the processor 110 .

[0125] The speaker 170A, also called a "speaker", is used to convert analog audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A. In an embodiment of the present application, when playing back multi-channel recorded videos, the speaker 170A can be used to play the audio corresponding to the video images in different display areas.

[0126] The receiver 170B, also called a "earpiece", is used to convert analog audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0127] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into analog audio electrical signals. When making a call or sending a voice message, the user can speak by approaching the microphone 170C with his mouth to input the sound signal into the microphone 170C.

[0128] In an embodiment of the present application, the electronic device 100 may include at least three microphones 170C, which can collect sound signals from all directions and convert the collected sound signals into analog audio electrical signals. It can also achieve noise reduction, identify sound sources, or directional recording functions, etc.

[0129] For example, the layout of the microphone 170C on the electronic device 100 can be seen in Figure 2 .like Figure 2As shown, the front cdhg of the electronic device 100 is the plane where the display screen 194 is located, the microphone A is located on the plane ahe at the top of the electronic device 100, the microphone B is located on the plane bcgf at the bottom of the electronic device 100, and the microphone C is located on the back abfe of the electronic device 100. The combination of microphones AC can collect sound signals from all directions around the electronic device 100. It is understandable that the electronic device 100 can also include a larger number of microphones, for example, it can also include a microphone set on the front of the screen, etc.

[0130] It should be noted that the microphone 170C may be a built-in component of the electronic device 100 , or may be an external accessory of the electronic device 100 .

[0131] In some embodiments, the microphone 170C may be a directional microphone that can collect sound signals in a specific direction.

[0132] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0133] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0134] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0135] For example, in an embodiment of the present application, the electronic device 100 may detect an operation of a user instructing to start and / or stop recording through the touch sensor 180K.

[0136] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0137] In an embodiment of the present application, in a multi-channel video recording mode, the display screen 194 can display a shooting preview interface, a video recording preview interface, and a shooting interface during video recording. The camera 193 can be used to collect multi-channel video images. Multiple microphones 170C can be used to collect sound signals and generate analog audio electrical signals. The audio module 170 can generate digital audio data from analog audio electrical signals and generate multi-channel audio based on the digital audio data. During video playback, the display screen 194 can display a video playback interface. The processor 110 can control the speaker 170A to play the audio corresponding to the video image in the display area that the user is concerned about according to the user's choice by running the instructions stored in the internal memory 121, thereby enhancing the user's audio experience of multi-channel video recording.

[0138] The following will be based on electronic equipment Figure 1 Taking the mobile phone with the structure shown in the figure as an example, the audio processing method provided by the embodiment of the present application is described. Figure 3 , the method may include:

[0139] 301. After the mobile phone detects that the user opens the camera application, it displays a shooting preview interface.

[0140] After the mobile phone detects that the user has opened the camera application, it can start the camera application (hereinafter referred to as the camera) and display the shooting preview interface. Among them, the user can open the camera in various ways. For example, when the mobile phone detects that the user clicks Figure 4 After operating the camera icon 401 shown in (a) in FIG. 1 , the camera application can be started and displayed. Figure 4 The shooting preview interface shown in (b) in FIG.

[0141] For example, after the mobile phone detects the user's voice instruction operation to open the camera, the camera application can be started and the display Figure 4 The shooting preview interface shown in (b) in FIG.

[0142] 302. The mobile phone enters a multi-channel video recording mode and displays a video recording preview interface, where the video recording preview interface includes at least two display areas, each of which is used to display a preview image of one channel.

[0143] There are many ways for a mobile phone to enter the multi-channel recording mode. For example, in some implementations, after the mobile phone starts the camera application, it enters a non-multi-channel recording mode such as a photo taking mode or a recording mode (i.e., a single-channel recording mode) by default. After the mobile phone detects a preset operation 1 instructing the user to enter the multi-channel recording mode, it enters the multi-channel recording mode. Exemplarily, the preset operation 1 may be that the user clicks Figure 4 Operation of the control 402 shown in (b) in FIG. 4. For example, in the multi-channel video recording mode, the video preview interface displayed by the mobile phone can be referred to as Figure 4 (c) in .

[0144] In other implementations, in non-multi-channel recording modes such as photo mode or video mode (i.e., single-channel recording mode), the mobile phone enters multi-channel recording mode after detecting that the user draws a preset track 1 (e.g., an "M" track) on the touch screen.

[0145] In other implementations, in the video recording mode (i.e., the single-channel video recording mode), the mobile phone can prompt the user on the video preview interface whether to enter the multi-channel video recording mode. The mobile phone can enter the multi-channel video recording mode according to the user's instructions.

[0146] In some other implementations, after the mobile phone starts the camera application, it enters the multi-channel video recording mode by default.

[0147] In some other embodiments, the mobile phone may not execute step 301, but directly execute step 302, thereby directly entering the multi-channel recording mode. For example, when the screen is bright and the desktop is displayed, or when the screen is black, if the mobile phone detects that the user has drawn a preset track (such as a "CM" track) on the touch screen, the camera application is started and the multi-channel recording mode is directly entered.

[0148] In some embodiments, after the mobile phone enters the multi-channel recording mode, the number of display areas included in the recording preview interface can be a preset number, the number pre-set by the user through the setting interface of the mobile phone before entering the multi-channel recording mode, or the number of display areas included when the multi-channel recording mode was used last time. Among them, each display area on the recording preview interface is used to display one preview screen. Then, the mobile phone can also adjust the number of display areas according to the user's instructions, that is, adjust the number of lines of the preview screen.

[0149] For example, after the mobile phone enters the multi-channel recording mode, see Figure 5 (a) and Figure 5 In (b), the video preview interface may include two display areas and two preview screens, namely, display area 501 and an internal preview screen, and display area 502 and an internal preview screen. After the mobile phone detects that the user selects the operation of the three-channel screen, Figure 5 As shown in (c) in FIG. 1 , the mobile phone displays three display areas and three preview screens, namely, display area 503 and an internal preview screen, display area 504 and an internal preview screen, and display area 505 and an internal preview screen.

[0150] For example, after the mobile phone enters the multi-channel video recording mode, Figure 4 As shown in (c) of FIG. 1 , the video preview interface may include two display areas and two preview screens. The mobile phone detects that the user clicks Figure 4After the control 403 shown in (c) in the figure is pressed, the shooting setting interface can be displayed. The mobile phone responds to the user's operation on the shooting setting interface and displays the following Figure 5 (a) or Figure 5 The interface for setting the number of display areas shown in (b) etc.

[0151] In some other embodiments, after the mobile phone enters the multi-channel video recording mode, an interface for prompting the user to set the number of display areas is displayed. The mobile phone can set the number of display areas according to the user's instructions.

[0152] In some other embodiments, after entering the multi-channel video recording mode, the video preview interface may include a control for setting the number of display areas. After the mobile phone detects that the user clicks the control, it may display a setting interface for the number of display areas. The mobile phone may set the number of display areas according to the user's instructions.

[0153] Among them, after entering the multi-channel recording mode, the zoom ratio corresponding to each display area on the recording preview interface can be a preset zoom ratio, a zoom ratio used in the multi-channel recording mode last time, or a zoom ratio indicated in advance by the user. In addition, the zoom ratio corresponding to each display area can also change according to the user's instructions. The screen range and screen size of the video screen in the display area will change with the change of the zoom ratio.

[0154] Moreover, the preview screen in each display area can switch the front / rear camera capture screen. Figure 6 The video preview interface shown in (a) includes a display area 601 and a display area 602. The preview images in the display area 601 and the display area 602 are both images captured by the rear camera. The display area 601 also includes a front / rear switching control 603, which is used to switch the front / rear camera. The preview images captured by the front / rear camera are displayed in the display area 601. The display area 602 also includes a front / rear switching control 604, which is also used to switch the front / rear camera. The preview images captured by the front / rear camera are displayed in the display area 602. Exemplarily, after the mobile phone detects that the user clicks the control 604, the rear camera can be switched to the front camera. Accordingly, the preview image displayed in the display area 602 is changed by Figure 6 The preview image captured by the rear camera shown in (a) is switched to Figure 6 (b) shows the preview image captured by the front camera.

[0155] 303. After the mobile phone detects a shooting operation instructed by the user, it displays a shooting interface, where the shooting interface includes at least two display areas, each of which is used to display a video image.

[0156] For example, the user indicates that the action of taking a photo can be a user clicking Figure 6 The operation of the shooting control 600 shown in (a), the user voice instruction operation, or other operations are not limited in the embodiments of the present application. After the mobile phone detects the user's instruction to shoot, it collects multiple video images and displays the corresponding shooting interface, which has at least two display areas, each of which is used to display one video picture.

[0157] Among them, there can be multiple layout formats of the display area and multi-channel video images on the shooting interface, such as left / right splicing format, left / center / right splicing format, top / bottom splicing format, top / center / bottom splicing format, or picture-in-picture format.

[0158] For example, the schematic diagram of the shooting interface can be seen in Figure 6 (c) in the figure. The shooting interface includes a first display area 601 and a second display area 602. The first display area 601 and the second display area 602 both display video images shot by the rear camera. The first display area 601 displays images of a larger shooting range shot by the wide-angle camera, including objects such as a little boy, a little girl, a toy car, and a puppy; the second display area 602 displays close-up images of the little girl shot by other cameras. Figure 6 As shown in (c) in FIG. 8 , the shooting interface may further include an indicator for indicating the current video recording time.

[0159] Exemplarily, the first display area 601 and the second display area 602 can respectively display video images shot by different cameras, for example, the first display area 601 displays the video images shot by the rear wide-angle camera, and the second display area 602 displays the video images shot by the rear telephoto camera. Alternatively, the first display area 601 displays the video images shot by the rear wide-angle camera, and the second display area 602 displays the video images shot by the front camera.

[0160] 304. Record multiple video clips with your mobile phone.

[0161] After the mobile phone detects the user's instruction to shoot, it can record multiple video images at the same time, and can perform video encoding and other processing on the collected multiple video images to generate and save video files.

[0162] In some embodiments, the multiple video images recorded simultaneously are saved as one video file. In other embodiments, each video image is saved as a separate video file; thus, during subsequent video playback, the mobile phone can play one of the video images separately.

[0163] 305. After detecting the user's instruction to shoot, the mobile phone records multiple channels of audio while recording multiple channels of video images. The multiple channels of audio include panoramic audio and at least one channel of audio corresponding to each display area.

[0164] Among them, one audio channel may include audio of one or more sound channels. For example, one audio channel may be audio of a left channel, or may be audio of a right channel, or may include audio of a left channel and audio of a right channel, or may be stereo audio after the left channel and the right channel are mixed.

[0165] That is to say, after the mobile phone detects the user's instruction to shoot, it can record audio of multiple sound channels while recording multiple video images. The audio of multiple sound channels includes panoramic audio and audio of at least one sound channel corresponding to each display area.

[0166] See also Figure 7 , step 305 may include:

[0167] 701. The mobile phone collects panoramic sound signals and generates initial audio data according to the collected panoramic sound signals.

[0168] Among them, the mobile phone can use multiple microphones (for example Figure 2 The mobile phone collects sound signals from all directions, that is, collects panoramic sound signals. The mobile phone converts the panoramic sound signals collected by multiple microphones into initial audio data, which contains sound information from all directions.

[0169] Then, the mobile phone can record the panoramic audio and at least one audio corresponding to each display area according to the initial audio data. The following describes the recording process of different audios respectively:

[0170] (1) Panoramic audio

[0171] The initial audio data contains sound information from all directions. The mobile phone can fuse the audio data of each microphone in the initial audio data to record panoramic audio, which includes sounds from all directions around. Figure 7 As shown, step 305 may also include:

[0172] 702. The mobile phone records the panoramic audio according to the initial audio data.

[0173] Among them, since the pickup ranges of the three microphones may partially overlap, for the same direction, the sound quality of the sound signal collected by one microphone may be better (for example, higher signal-to-noise ratio, less peak noise and burr noise, etc.), while the sound quality of the sound signal collected by another microphone may be poor. Therefore, the mobile phone can select audio data with better sound quality in each direction from the initial audio data for fusion processing, and then generate panoramic audio with better effect based on the processed audio data recording.

[0174] The following description will be made by taking the shooting interface including the first display area and the second display area as an example. It is understandable that the shooting interface may also include more than two display areas.

[0175] In some embodiments, when the mobile phone is recording multiple audio channels corresponding to multiple display areas, a recording prompt message may be displayed in each display area to remind the user that the audio corresponding to each display area is currently being recorded. For example, the recording prompt message may be Figure 8 The microphones shown are labeled 801-802.

[0176] (2) At least one audio channel corresponding to the first display area

[0177] The following description takes recording of at least one audio channel corresponding to the first display area as an example, and the process of recording at least one audio channel corresponding to the second display area is similar.

[0178] In an embodiment of the present application, the mobile phone can obtain the first characteristic value corresponding to the first display area, and generate at least one audio corresponding to the first display area according to the first characteristic value and the initial audio data. Figure 7 , the above step 305 may also include:

[0179] 703. The mobile phone obtains a first characteristic value corresponding to the first display area.

[0180] 704. The mobile phone determines a first sound pickup range corresponding to the first display area according to the first characteristic value.

[0181] For example, the first feature value corresponding to the first display area may include a front / rear attribute parameter of the video screen corresponding to the first display area, and the front / rear attribute parameter is used to indicate whether the video screen displayed in the first display area is a video screen taken by the front camera (for ease of description, also referred to as the front video screen in this article) or a video screen taken by the rear camera (for ease of description, also referred to as the rear video screen in this article). The front / rear attribute parameter can be used to determine whether the first sound pickup range is within the 180-degree range of the rear camera of the mobile phone or within the 180-degree range of the front camera. Exemplarily, the sound pickup range corresponding to the rear video screen can be referred to Fig. 9The range indicated by the ellipse 901 in FIG. 1 and the sound pickup range corresponding to the front video screen can be referred to as Fig. 9 The range indicated by the ellipse 902 in FIG.

[0182] For another example, the first characteristic value corresponding to the first display area may also include the zoom factor currently corresponding to the first display area. Different zoom factors have different shooting ranges, and the picture ranges displayed on the shooting interface are also different. Moreover, different zoom factors have different picture ranges that users pay attention to, and different sound pickup ranges that users pay attention to. The sound pickup range that users pay attention to is usually the sound within the picture range that users pay attention to. For example, when other parameters are the same, if the zoom factor is increased to 2 times the original, the sound pickup range may be reduced to 1 / 3 of the original; if the zoom factor is increased to 3 times the original, the sound pickup range may be reduced to 1 / 6 of the original. Therefore, the mobile phone can determine the first sound pickup range according to the zoom factor corresponding to the first display area.

[0183] For example, for the first display area, when the zoom factor is N (ie, N-fold zoom), the corresponding sound pickup range can be seen as follows: Fig. 9 The range represented by the ellipse 903 in FIG. 1 and the corresponding sound pickup range when the zoom factor is X (i.e., X-fold zoom) can be seen in FIG. Fig. 9 The dotted oval 904 in the figure indicates a range where N is less than X.

[0184] For example, when the zoom factor is N (i.e., N-fold zoom), the corresponding sound pickup range can be found in Fig.10 , this pickup range is part of the panoramic audio pickup range.

[0185] That is, the mobile phone can determine the first sound pickup range corresponding to the first display area according to the front / rear attribute parameter corresponding to the first display area and the zoom factor. For example, when the front / rear attribute parameter corresponding to the first display area indicates rear, and the zoom factor is N, the first sound pickup range corresponding to the first display area can be determined by referring to Fig. 9 The range indicated by the ellipse 903 in FIG.

[0186] For another example, the first characteristic value corresponding to the first display area may also include the position of the first camera used to capture the video screen in the first display area. The position is used to represent the coordinates of the first camera on the mobile phone. For example, the position may be the position of the first camera relative to the upper left corner of the mobile phone, or the distance between the first camera and the frame of the mobile phone. The position of the first camera is different, and the shooting range corresponding to the same zoom factor is also different, the picture range displayed on the shooting interface is also different, and the sound pickup range corresponding to the picture range is also different. The first camera may be one or more, and the embodiments of the present application are not limited thereto.

[0187] For another example, the first characteristic value corresponding to the first display area may also include the type of the first camera used to capture the video image in the first display area, and different types of first cameras correspond to different focal length ranges. For example, the type includes a wide-angle camera type, a telephoto camera type, or an ultra-wide-angle camera type. Different types of first cameras may have different shooting ranges corresponding to the same zoom factor and the same camera position, and the picture range displayed on the shooting interface may also be different, and the sound pickup range corresponding to the picture range may also be different.

[0188] In this way, the mobile phone can determine the first sound pickup range corresponding to the first display area according to the front / rear attribute parameters and zoom ratio corresponding to the first display area, and the first characteristic values ​​such as the position and / or type of the first camera.

[0189] 705. The mobile phone records at least one audio channel corresponding to the first display area according to the first sound pickup range and the initial audio data.

[0190] After the mobile phone determines the first sound pickup range corresponding to the first display area, it can enhance the sound within the first sound pickup range in the initial audio data, suppress (or weaken) the sound outside the first sound pickup range, and then record the processed audio data to obtain an audio channel corresponding to the first display area.

[0191] In this way, the audio corresponding to the first display area records the sound within the first sound pickup range, and the first sound pickup range is a sound pickup range determined according to the first characteristic value such as the zoom factor corresponding to the first display area, so the first sound pickup range can match the screen range that the user is paying attention to at the zoom factor indicated by the user in the first display area. In other words, the audio corresponding to the first display area matches the video screen in the first display area that the user is paying attention to.

[0192] In some embodiments, since the first sound pickup range of the first display area determined by the mobile phone according to the first characteristic value may have a certain error with the image range displayed in the first display area, the mobile phone can determine one or more reference first sound pickup ranges near the first sound pickup range. Exemplarily, the first sound pickup range is an angle range of 0-60 degrees, and the reference first sound pickup range can be an angle range of 350-50 degrees, an angle range of 5-65 degrees, or an angle range of 5-70 degrees near the first sound pickup range. In addition, the mobile phone can process the initial audio data to enhance the sound within the reference first sound pickup range and suppress the sound outside the reference first sound pickup range, and then record the processed audio data to obtain one or more audio channels corresponding to the first display area.

[0193] In this way, the mobile phone can record and obtain multiple audio channels that match the first characteristic value of the first display area and the video screen in the first display area according to the first sound pickup range and the reference first sound pickup range, so that the user can choose to play them later. Each audio data corresponding to the first display area can be saved as an audio file, and the first display area can correspond to multiple audio files.

[0194] When the mobile phone records multi-channel audio corresponding to the first display area according to the first sound pickup range and reference to the first sound pickup range, the multi-channel audio provides the user with more audio within different sound pickup ranges, has a greater possibility of accurately matching the video screen in the first display area that the user is concerned about, and the user has greater selectivity in audio playback.

[0195] In the embodiment of the present application, during the multi-channel video recording process, similar to the way in which the user adjusts the zoom factor on the video preview interface, see Fig.11A In (a)-(c), the zoom ratio corresponding to each display area on the shooting interface can be changed according to the user's instructions. In other words, the zoom ratio in the first characteristic value can be changed in real time. The mobile phone can determine the first sound pickup range corresponding to the first display area according to the current real-time zoom ratio in the first characteristic value.

[0196] Among them, Fig.11A In the case shown in (a), the first display area includes a zoom adjustment control 1101 for increasing the zoom ratio and a zoom adjustment control 1102 for decreasing the zoom ratio, and the user can click these controls to adjust the zoom ratio corresponding to the first display area. Similarly, the second display area can also include a zoom adjustment control for increasing and decreasing the zoom ratio.

[0197] exist Fig.11A In the case shown in (b), the first display area and the second display area include a zoom adjustment control 1103 and a zoom adjustment control 1104 respectively, and the user can drag on the zoom adjustment control to adjust the zoom ratio.

[0198] exist Fig.11A In the case shown in (c), the user can adjust the zoom ratio by pinching the fingers in the first display area and the second display area.

[0199] For example, Solution 1: Each display area and each video screen corresponds to a set of front and rear cameras with full focal lengths. For example, cameras of different focal lengths include wide-angle cameras and telephoto cameras (and may also include ultra-wide-angle cameras, etc.), the focal length range corresponding to the wide-angle camera is [P, Q], and the focal length range corresponding to the telephoto camera is [R, S]. Among them, S is greater than Q, and R is less than or equal to Q, that is, the focal length ranges of cameras with different focal lengths may overlap. [P, S] can be called the full focal length range.

[0200] Table 1

[0201]

[0202] In solution 1, the zoom ratio in each display area can be changed within the full focal length range [P, S]. The full focal length ranges [P, S] corresponding to different display areas may be the same or different. For each display area, the mobile phone can switch the camera corresponding to the display area to meet the user's demand for zoom ratio.

[0203] Taking the first display area as an example, in the rear-facing scene, when the zoom ratio switches from [P, Q] to [Q, S], the first camera can switch from the rear-facing wide-angle camera (i.e., camera 1) to the rear-facing telephoto camera (i.e., camera 2); when the zoom ratio switches from [R, S] to [P, R], the first camera can switch from the rear-facing telephoto camera (i.e., camera 2) to the rear-facing wide-angle camera (i.e., camera 1). The zoom processing in the front-facing scene is similar to that in the rear-facing scene, and the zoom processing in the second display area is similar to that in the first display area, which will not be described in detail here.

[0204] For example, in Fig.11A In the scenario shown in (a) in FIG. 1 , the first camera used in the first display area on the left is a rear wide-angle camera. After the mobile phone detects that the user has increased the zoom factor, and the increased zoom factor exceeds the focal length range of the rear wide-angle camera, the first camera is switched to the rear telephoto camera. Fig. 11B As shown in (a) of FIG. 1 , the image range in the first display area is reduced, and the video image in the first display area is enlarged and displayed. Fig.11A In the scenario shown in (a) in FIG. 1 , the second camera used in the second display area on the right is a rear telephoto camera. After the mobile phone detects that the user has reduced the zoom factor, and the reduced zoom factor exceeds the focal length range of the rear telephoto camera, the second camera is switched to the rear wide-angle camera. Fig. 11B As shown in (a) in the figure, the image range in the second display area is enlarged, and the video image in the second display area is reduced and displayed.

[0205] Solution 2: The mobile phone includes multiple cameras, and each display area shares the multiple cameras. For example, when the user indicates that the zoom ratio in the first display area changes in a certain direction and another camera needs to be used, if the other camera is occupied by the second display area, the first display area will not be able to switch to use the other camera, and the zoom ratio in the first display area will not be able to continue to change in that direction.

[0206] Exemplarily, referring to Table 2, the first display area and the second display area can share cameras 1 to 4 .

[0207] Table 2

[0208]

[0209] Solution 3: Each display area corresponds to at least one camera, and the zoom ratio in the display area can change within the focal length range of the corresponding camera. For example, when the zoom ratio in the first display area indicated by the user changes in a certain direction and the camera corresponding to another display area needs to be used, the first display area will not be able to switch to use the camera corresponding to another display area, and the zoom ratio in the first display area will not be able to continue to change in that direction.

[0210] Exemplarily, referring to Table 3, the first display area corresponds to camera 1 and camera 3, and the second display area corresponds to camera 2 and camera 4.

[0211] Table 3

[0212]

[0213] In Solution 2 and Solution 3, the zoom ratio in the same display area cannot be changed continuously within the full focal length range. The mobile phone can achieve continuous change of the zoom ratio within the full focal length range through the coordination of multiple display areas. For example, when the zoom ratio indicated by the user in a certain display area exceeds the focal length range of the camera available in the display area, the mobile phone can prompt the user to switch to another display area to continue zooming.

[0214] For example, in Fig. 11B In the scenario shown in (b), the first camera used in the first display area on the left is a rear wide-angle camera, and the second camera used in the second display area on the right is a rear telephoto camera. The focal range of the rear wide-angle camera is [0, 4], and the focal range of the rear telephoto camera is [2, 10]. The current zoom factor of the first display area is 4. After the mobile phone detects that the user increases the zoom factor in the first display area, it determines that the increased zoom factor exceeds the focal range of the wide-angle camera. At this time, see Fig. 11BIn (b), the mobile phone can prompt the user: the zoom has exceeded the focal length range of the left area, do you want to continue zooming in the right area? Or, the mobile phone can prompt the user: the zoom has exceeded the focal length range of the left area, do you want to perform a zoom operation on the zoom adjustment control of the right area? After the mobile phone detects that the user clicks the "OK" control 1105, if Fig. 11B As shown in (c) in the figure, the zoom factor in the second display area on the right automatically changes to 4 (or a value greater than 4), and the video image corresponding to the zoom factor is displayed in the second display area. After the mobile phone detects that the user clicks the "Cancel" control, the zoom factor in the first display area on the left is 4, and the zoom factor in the second display area on the right remains unchanged. In this way, the mobile phone can achieve continuous zoom of the rear video during multi-channel recording.

[0215] As another example, the first camera used in the first display area on the left is a front wide-angle camera, and the second camera used in the second display area on the right is a front telephoto camera. The focal range of the front wide-angle camera is [0, 4], and the focal range of the front telephoto camera is [2, 10]. The current zoom factor of the second display area is 2. After the mobile phone detects that the user has reduced the zoom factor in the second display area, it determines that the reduced zoom factor exceeds the focal range of the telephoto camera. At this time, the mobile phone can prompt the user: the zoom has exceeded the focal range of the right area, please continue zooming in the left area. In addition, the zoom factor in the second display area on the left automatically changes to 2 or a value less than 2. In this way, the mobile phone can achieve continuous zoom of the front video during multi-channel video recording.

[0216] As another example, on the shooting interface, the first camera used in the first display area on the left is a rear wide-angle camera, and the second camera used in the second display area on the right is a front telephoto camera. The focal range of the rear wide-angle camera is [0, 4], and the focal range of the front telephoto camera is [2, 10]. The current zoom factor of the first display area is 4. After the mobile phone detects that the user has increased the zoom factor in the first display area, it determines that the increased zoom factor exceeds the focal range of the rear wide-angle camera. At this time, the mobile phone can prompt the user: the zoom has exceeded the focal range of the rear wide-angle camera (or rear wide-angle mode), do you want to switch to the rear telephoto camera (or rear telephoto mode) in the left area to continue shooting? After the mobile phone detects that the user instructs to switch to the rear telephoto camera (rear telephoto mode) in the left area to continue shooting, the picture shot by the rear telephoto camera begins to be displayed in the first display area on the left, and the zoom factor of the rear telephoto camera automatically changes to 4 (or a value greater than 4). In this way, after the mobile phone switches to another camera in the same display area, it does not need to start zooming from the boundary value of the focal range of the other camera, but can continue zooming from the zoom multiple indicated by the user, thereby achieving continuous zoom and improving the user experience. For example, the boundary values ​​of the focal range [2, 10] of the rear telephoto camera are 2 and 10.

[0217] In some embodiments of the present application, on the shooting interface, each display area may also include a front / rear switching control for switching the front / rear video screen in the display area during the video recording process. The front / rear attribute parameters in the first feature value may change in real time according to the user's front / rear switching instruction. The mobile phone may determine the first sound pickup range corresponding to the first display area according to the current real-time front / rear attribute parameters in the first feature value.

[0218] For example, the shooting interface in the multi-channel video mode can be seen in Fig. 11C In (a), the shooting interface includes a first display area on the left and a second display area on the right. The first display area and the second display area both include a front / rear switching control. When the mobile phone detects that the user clicks the front / rear switching control in the second display area, Fig. 11C As shown in (b), the mobile phone can switch the rear video picture in the second display area to the front video picture.

[0219] In some other embodiments of the present application, the shooting interface may further include some mode controls, so that the user can directly switch the camera used in the first display area or the second display area through the mode controls, and directly switch the zoom factor to the focal length range corresponding to the camera. In other words, the mode controls can facilitate the user to directly switch to the target camera and the target focal length range that the user wants to use.

[0220] Each mode control corresponds to a shooting mode, which may include a rear wide-angle mode, a rear telephoto mode, a front wide-angle mode, or a front telephoto mode. The rear wide-angle mode corresponds to the rear wide-angle camera, the rear telephoto mode corresponds to the rear telephoto camera, the front wide-angle mode corresponds to the front wide-angle camera, and the front telephoto mode corresponds to the front telephoto camera.

[0221] In some embodiments, the mode control can also be called a camera switching control, and each camera switching control corresponds to a type of camera; after the mobile phone detects the user's operation on a camera switching control, the target type of camera corresponding to the camera switching control is used to capture the video image in the display area, and the zoom ratio corresponding to the display area is within the focal length range of the target camera. For example, the target type camera includes a rear wide-angle camera, a rear telephoto camera, a front wide-angle camera, or a front telephoto camera.

[0222] For example, see Fig.11D In (a), the first display area and the second display area respectively include: a rear wide-angle control, a rear telephoto control, a front wide-angle control, and a front telephoto control. Among them, the front wide-angle control in the first display area is currently selected, indicating that the first camera used in the first display area on the left is the front wide-angle camera. The front telephoto control in the second display area is currently selected, indicating that the second camera used in the second display area on the right is the front telephoto camera.

[0223] If the phone detects that the user clicks Fig.11D The operation of the rear wide-angle control 1106 in the first display area shown in (a) is as follows Fig.11D As shown in (b) of FIG. 1 , the rear wide-angle control 1106 is selected, and the mobile phone can switch the first camera used in the first display area to the rear wide-angle camera, so as to use the rear wide-angle camera to capture video images and display the captured video images in the first display area. Fig.11D The operation of the rear telephoto control 1107 in the second display area shown in (a) is as follows Fig.11D As shown in (b), the rear telephoto control 1107 is selected, and the mobile phone can switch the second camera to the rear telephoto camera, thereby using the rear telephoto camera to capture video images and display the captured video images in the second display area.

[0224] (3) At least one audio channel corresponding to the second display area

[0225] Similar to recording at least one audio channel corresponding to the first display area, the mobile phone can record at least one audio channel corresponding to the second display area according to the second characteristic value and the initial audio data. Figure 7, the above step 305 may also include:

[0226] 706. The mobile phone obtains a second characteristic value corresponding to the second display area.

[0227] 707. The mobile phone determines a second sound pickup range of the audio corresponding to the second display area according to the second characteristic value.

[0228] For example, when the zoom factor corresponding to the second display area is X, the second sound pickup range corresponding to the second display area can be referred to as Fig. 9 and Fig.10 .

[0229] 708. The mobile phone records at least one audio channel corresponding to the second display area according to the second sound pickup range and the initial audio data.

[0230] Among them, the description of steps 706-708 can refer to the relevant description of the above steps 703-705, which will not be repeated here.

[0231] Similarly, the mobile phone can also determine one or more reference second sound pickup ranges according to the second sound pickup range, so as to record one or more audio channels corresponding to the second display area according to the reference second sound pickup ranges. The mobile phone generates audio files of each audio channel corresponding to the second display area.

[0232] For example, in Figure 6 In the multi-channel video recording scene shown in (c), the panoramic audio recorded by the mobile phone includes sounds from all directions. The audio corresponding to the first display area 601 includes the sounds of objects such as the little boy, the little girl and the puppy, and the audio corresponding to the second display area 602 is mainly the voice of the little girl.

[0233] In another example, the shooting interface includes a first display area and a second display area. The video screen in the first display area includes the performance screen of each instrument player in the concert, and the video screen in the second display area includes the performance screen of the first violin player. In this scenario, the panoramic audio recorded by the mobile phone includes sounds from all directions. The audio corresponding to the first display area includes the ensemble sound composed of the instruments played by the players in the band, and the audio corresponding to the second display area mainly includes the sound played by the first violin player.

[0234] In another example, the shooting interface includes a first display area and a second display area. The video screen in the first display area includes the performance screen of each member of the band, and the video screen in the second display area includes the performance screen of the lead guitarist. In this scenario, the panoramic audio recorded by the mobile phone includes sounds from all directions. The audio corresponding to the first display area includes the performance sounds of each member of the band, and the audio corresponding to the second display area mainly includes the singing sound of the lead guitarist and the sound of the guitar playing.

[0235] In some embodiments, the audio recorded by the mobile phone may also include:

[0236] (4) Panoramic Audio+

[0237] During multi-channel recording, if the mobile phone determines that the zoom factor of a certain display area has changed, the mobile phone can also replace the subsequent part of the panoramic audio with the subsequent corresponding audio of the display area, thereby generating panoramic audio +. Figure 7 , step 305 may also include:

[0238] 709. When the zoom ratio of the mobile phone in a certain display area changes, record panoramic audio +.

[0239] When a user adjusts the zoom ratio in a certain display area, the user may want to focus on the video image and sound pickup range corresponding to the adjusted zoom ratio. Therefore, the mobile phone can replace the subsequent part of the panoramic audio with the subsequent audio corresponding to the display area, thereby generating a dynamically changing panoramic audio, highlighting the audio corresponding to the user adjusting the zoom ratio in the panoramic audio, and enhancing the user's experience of audio changes after zooming.

[0240] For example, the shooting interface includes a first display area and a second display area. During the multi-channel video recording process, the mobile phone records panoramic audio, at least one audio corresponding to the first display area, and at least one audio corresponding to the second display area. If the mobile phone determines that the zoom factor corresponding to the first display area has increased after recording for 00:09, the mobile phone displays the enlarged video image in the first display area according to the increased zoom factor, giving the user a visual experience of zooming in.

[0241] For audio, the mobile phone can record panoramic audio, at least one audio channel corresponding to the first display area, and at least one audio channel corresponding to the second display area, while also recording panoramic audio+. The panoramic audio+ includes two parts, namely the audio part within the duration of 00:00-00:09, and the audio part after 00:09. Among them, the audio part within the duration of 00:00-00:09 in the panoramic audio+ is consistent with the audio part within the duration of 00:00-00:09 in the panoramic audio, and the audio part after 00:09 in the panoramic audio+ is consistent with the audio part after 00:09 in the audio channel corresponding to the first display area. In other words, the panoramic audio+ can synthesize the audio part corresponding to 00:00-00:09 in the panoramic audio and the audio part after 00:09 in the audio channel corresponding to the first display area into panoramic audio+.

[0242] In this way, when playing Panoramic Audio+, the user can hear the panoramic sound within the playback time of 00:00-00:09; after 00:09, corresponding to the user increasing the zoom factor corresponding to the first display area, the user can hear the sound within the pickup range corresponding to the increased zoom factor in the first display area, thereby giving the user a dynamic auditory experience of closer sound.

[0243] In other words, the mobile phone can play panoramic audio + to achieve the following: play panoramic sound by default, and if the zoom factor changes, highlight the sound corresponding to the pickup range after the zoom factor changes, giving users an audio playback experience with dynamic changes in sound distance, size and range.

[0244] In other words, panoramic audio + includes audio a of a first duration and audio b of a second duration, audio a is panoramic audio, and audio b includes audio corresponding to the first zoom multiple of the first camera; wherein, during the first duration, the first camera uses the second zoom multiple for shooting, and during the second duration, the first camera uses the first zoom multiple for shooting.

[0245] In this way, the mobile phone can record panoramic audio, panoramic audio + and the audio corresponding to each display area, so that users can choose to play different audio later, thereby improving the user's audio experience of multi-channel recording.

[0246] In addition, the mobile phone can record the audio corresponding to each display area, and record the audio that matches the video screen in each display area in real time, so that it can subsequently switch to play the audio corresponding to different display areas, so that the played audio matches the video screen in the display area that the user is paying attention to in real time, thereby improving the user's audio experience.

[0247] Moreover, the mobile phone can record the audio corresponding to the characteristic values ​​such as the zoom multiple or the front / rear attribute parameters that change in real time in the display area. Therefore, when playing back the video, the mobile phone can play the audio corresponding to the characteristic values ​​such as the zoom multiple or the front / rear attribute parameters that change in real time in the display area, so that the audio matches the video picture that changes according to the characteristic values ​​such as the zoom multiple or the front / rear attribute parameters in real time, thereby improving the user's audio experience.

[0248] In some embodiments of the present application, before generating audio files of each audio channel, the mobile phone can perform multiple audio effect processing on each audio channel, so that the recorded audio has higher audio quality and better audio processing effect. For example, the audio effect processing may include: Dolby audio effect, Histen audio effect, sound retrieval system (SRS) audio effect, bass enhanced engine (BBE) audio effect, or dynamic bass enhanced engine (DBEE) audio effect, etc.

[0249] 306. After the mobile phone detects an operation of the user instructing to stop filming, it stops recording the video image and audio, and generates multiple recorded videos.

[0250] For example, the user's instruction to stop shooting may be that the user clicks Figure 8 The operation of the control 800 shown, the operation of the user's voice instruction to stop shooting, or other operations such as gestures are not limited in the embodiments of the present application.

[0251] After the mobile phone detects the user's instruction to stop shooting, it generates multiple video recordings and returns to the video preview interface or the shooting preview interface. The multiple video recordings include multiple video images and multiple audios. For example, the thumbnails of the multiple video recordings generated by the mobile phone can be found in Fig.12 Thumbnail 1201 shown in (a) of FIG. 1 , or Fig.12 Thumbnail 1202 shown in (b) in FIG.

[0252] In some embodiments, for a multi-channel video recorded using the method provided in the embodiments of the present application, the mobile phone can prompt the user that the video has multi-channel audio. For example, the multi-channel video thumbnail or the detailed information of the multi-channel video can include prompt information for indicating the multi-channel audio. For example, the prompt information can be Fig.12 The mark 1203 of multiple speakers shown in (b), other forms of marks, or text information, etc.

[0253] In other embodiments, the mobile phone can save the video files corresponding to each recorded video. The gallery can also display the video files corresponding to each recorded video. Fig.12 In (c), the files identified by the thumbnails 1204 in the gallery include files corresponding to the video images of each channel and files corresponding to the audio of each channel. Fig.12In (c), the files identified by thumbnail 1205 in the gallery include the video images and audio files corresponding to the wide-angle perspective of line 1; the files identified by thumbnail 1206 in the gallery include the video images and audio files corresponding to the wide-angle perspective of line 2.

[0254] 307. After the mobile phone detects the user's instruction to play the multiple-channel recorded videos, the mobile phone plays the video images and audios of the multiple-channel recorded videos.

[0255] For example, the user instructs to play multiple recorded videos, and the user can click Fig.12 1201 in the video preview interface shown in (a). In another exemplary embodiment, the user indicates the operation of playing multiple video recordings, which can be performed by the user clicking Fig.12 Operation of thumbnails 1202 in the gallery shown in (b).

[0256] After the mobile phone detects the user's instruction to play the multi-channel video recording operation, it plays the multi-channel video recording according to the multi-channel video images and multi-channel audio recorded in the multi-channel video recording process. That is, during video playback, the mobile phone plays the video images and audio recorded in the multi-channel video recording process.

[0257] When playing back a video, the mobile phone can display a video playback interface. In some embodiments, the video playback interface can include various video images, that is, the mobile phone can play various video images. For audio playback, in some technical solutions, when playing back a video, the mobile phone can play panoramic audio by default, and then switch to play other audio according to the user's instructions.

[0258] For example, when playing back a video, the video playback interface may include multiple audio switching controls, each of which corresponds to an audio channel recorded during the multi-channel video recording process. After the mobile phone detects that the user clicks on an audio switching control, it plays the audio channel corresponding to the audio switching control.

[0259] For example, during video playback, the mobile phone may display the following Fig.13 The video playback interface shown in (a) of FIG. 1 includes a video screen 1 in the first display area on the left (which may be referred to as the left video screen) and a video screen 2 in the second display area on the right (which may be referred to as the right video screen). The video playback interface may also include audio switching controls 1301-1307. Fig.13 As shown in (a) of FIG. 1 , the mobile phone currently selects the audio switching control 1301, and the panoramic audio is played by default. Fig.13In (b), after the mobile phone detects that the user clicks the audio switching control 1306, the right audio 1 corresponding to the audio switching control 1306, that is, the audio 1 corresponding to the second display area, can be played. The right audio 1 is the voice of the little girl in the second display area.

[0260] For example, during video playback, the mobile phone may display the following Fig.14 The video playback interface shown in the figure includes a left video screen in the first display area on the left and a right video screen in the second display area on the right. The video playback interface may also include multiple audio switching controls. After the mobile phone detects that the user clicks the audio switching control 1401, the right audio 2 corresponding to the audio switching control 1401, that is, the audio 2 corresponding to the second display area, can be played. The right audio 2 is the sound of the lead guitarist singing and the sound of the guitar playing in the second display area.

[0261] For example, during video playback, the video playback interface may include a preset control for calling out the audio switching control. After the mobile phone detects that the user clicks on the preset control, the following information may be displayed: Fig.13 (a) in Fig.13 (b) of Fig.14 After the mobile phone detects that the user clicks on an audio switching control, it plays the audio corresponding to the audio switching control.

[0262] In other technical solutions, when playing back a video, the mobile phone can display a video playback interface without playing the audio first; after detecting the user's instruction operation, the mobile phone plays the audio instructed by the user.

[0263] In some other embodiments, during video playback, the mobile phone may only play one video screen in one display area. For example, during video playback, the mobile phone displays each video screen in each display area by default, and plays panoramic audio by default. After the mobile phone detects that the user indicates that only one video screen in a certain display area is displayed, it can only display the video screen, such as enlarging or displaying the video screen in full screen, and automatically play the audio corresponding to the video screen. Among them, the audio automatically played by the mobile phone can be a video randomly selected by the mobile phone from the multiple audios corresponding to the display area. Alternatively, the mobile phone can sort the audio quality of the multiple audios corresponding to the display area, and the audio automatically played by the mobile phone is the audio with the best audio quality. Then, when the mobile phone detects that the user indicates another audio corresponding to the display area, the mobile phone switches to playing the audio indicated by the user.

[0264] For example, during video playback, see Fig.15In (a), the mobile phone plays the video images and the right audio 1. After the mobile phone detects that the user clicks the full screen display control 1501 in the second display area on the right, Fig.15 As shown in (b) of FIG. 1 , the mobile phone displays the right video image corresponding to the second display area in full screen, and automatically plays the audio corresponding to the right video image. Subsequently, after the mobile phone detects that the user clicks on control 1502, it can play the left video image in the first display area and the audio corresponding to the first display area in full screen; or, after the mobile phone detects that the user clicks on control 1503 or control 1504, it can restore the display as shown in FIG. Fig.15 The multi-channel video screen shown in (a) in the figure and the panoramic audio are automatically played.

[0265] In the scheme described in the above embodiment, during video playback, the mobile phone can switch to play panoramic audio, panoramic audio +, or audio corresponding to different display areas, giving users a variety of audio playback options, realizing the adjustability of audio, and improving the user's audio experience of multi-channel recordings.

[0266] In addition, the mobile phone can switch to play audio corresponding to different display areas, so that the played audio matches the video screen in the display area that the user is paying attention to in real time, improving the user's audio experience.

[0267] Moreover, the mobile phone can play audio corresponding to characteristic values ​​such as zoom ratio or front / rear attribute parameters that change in real time in the display area, so that the audio can be matched with the video pictures that change according to characteristic values ​​such as zoom ratio or front / rear attribute parameters in real time, thereby improving the user's audio experience.

[0268] In addition, the phone detects that the user clicks Fig.12 After operating the thumbnail 1205 shown in (c) in the figure, you can play Fig.13 The first display area shown in (a) corresponds to a single video image and a single audio. The mobile phone detects that the user clicks Fig.12 After operating the thumbnail 1206 shown in (c) in the figure, you can play Fig.13 The second display area shown in (a) corresponds to a single video image and a single audio image. The played video image can be as follows: Fig.15 As shown in (b) in .

[0269] It is understandable that in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0270] In this embodiment, the electronic device can be divided into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0271] For example, in one division, see Fig.16 , the electronic device may include a multi-channel video recording module, an audio hardware module, an audio processing module, a video playback module, etc. Among them, the multi-channel video recording module may include a camera application package (such as a camera Android application package (APK)), a first display area preview module, a second display area preview module, and other display area preview modules, etc. The audio hardware module may include components such as a microphone and an analog / digital conversion circuit.

[0272] The following will be Fig.16 The audio processing method provided by the embodiment of the present application is described from the perspective of each module of the electronic device shown in FIG. It can be understood that the operations and functions performed by each module of the electronic device are the operations and functions performed by the electronic device itself. Among them, the audio processing flow based on each module can be referred to in Fig.17 , the interactive relationship diagram between each module can be found in Fig.18 .

[0273] See also Fig.17 , the method may include: after the user instructs to open the camera, the camera application package starts the camera application. Then, after the user instructs to enter the multi-channel recording mode, the camera application package enters the multi-channel recording mode. After entering the multi-channel recording mode, the first display area preview module can display the preview screen in real time in the first display area, and the second display area preview module can display the preview screen in real time in the second display area. In addition, after entering the multi-channel recording mode, the camera application package can also query the microphone capability to determine whether the electronic device has the capability of recording multi-channel audio.

[0274] See also Fig.17 The method may further include: after the user instructs to start recording, the first display area preview module may display the video screen of the recording process in real time in the first display area, and the second display area preview module may display the video screen of the recording process in real time in the second display area. The camera application package generates video files corresponding to the multiple recording screens. In addition, during the recording process, the zoom ratio corresponding to the first display area and / or the second display area may also change (i.e., zoom).

[0275] See also Fig.17 , the method may also include: after the user instructs to start recording, the microphone in the audio hardware module can collect sound signals from all directions around (i.e., collect sound) and convert the sound signals into analog audio electrical signals. The analog / digital conversion circuit in the audio hardware module can convert the analog audio electrical signal into initial audio data. The audio hardware module can send the initial audio data to the audio processing module. The audio processing module generates an audio file of the panoramic audio based on the initial audio data.

[0276] See also Fig.17 , the method may also include: the first preview module may send the zoom factor in the first eigenvalue corresponding to the first display area to the audio processing module. The camera application package may send other parameters in the first eigenvalue to the audio processing module. The audio processing module determines the first sound pickup range according to the first eigenvalue, and enhances the sound within the first sound pickup range according to the initial audio data, and weakens the sound outside the first sound pickup range. In addition, the audio processing module may also determine one or more reference first sound pickup ranges according to the first sound pickup range, and enhance the sound within the reference first sound pickup range according to the initial audio data, and weaken the sound outside the reference first sound pickup range. The audio processing module may also perform multi-sound effect processing on the obtained audio data. Then, the audio processing module may generate an audio file of at least one audio channel corresponding to the first display area.

[0277] Similarly, the second preview module can send the zoom factor in the second eigenvalue corresponding to the second display area to the audio processing module. The camera application package can send other parameters in the second eigenvalue to the audio processing module. The audio processing module determines the second sound pickup range according to the second eigenvalue, and enhances the sound within the second sound pickup range according to the initial audio data, and weakens the sound outside the second sound pickup range. In addition, the audio processing module can also determine one or more reference second sound pickup ranges according to the second sound pickup range, and enhance the sound within the reference second sound pickup range according to the initial audio data, and weaken the sound outside the reference second sound pickup range. The audio processing module can also perform multi-sound effect processing on the obtained audio data. Then, the audio processing module can generate an audio file of at least one audio channel corresponding to the second display area.

[0278] In addition, when the zoom factor in the first display area or the second display area changes, the audio processing module may also generate panoramic audio+.

[0279] The audio processing module may send the generated audio file of the multi-channel audio to the camera application package.

[0280] In this way, the electronic device can record panoramic audio, panoramic audio+, and the audio corresponding to each display area, so that the user can subsequently choose to play different audios, thereby improving the user's audio experience of multi-channel recording.

[0281] In addition, the electronic device can record the audio corresponding to each display area, and record the audio that matches the video screen in each display area in real time, so that it can subsequently switch to play the audio corresponding to different display areas, so that the played audio matches the video screen in the display area that the user is paying attention to in real time, thereby improving the user's audio experience.

[0282] Moreover, the electronic device can record audio corresponding to the characteristic values ​​such as zoom multiple or front / rear attribute parameters that change in real time in the display area. Therefore, when playing back the video, the electronic device can play the audio corresponding to the characteristic values ​​such as zoom multiple or front / rear attribute parameters that change in real time in the display area, so that the audio matches the video picture that changes according to the characteristic values ​​such as zoom multiple or front / rear attribute parameters in real time, thereby improving the user audio experience.

[0283] See also Fig.17 The method may further include: the camera application package synthesizes the video files corresponding to the multi-channel video images and the audio files of the multi-channel audio into a multi-channel video file. After the user instructs to play the multi-channel video, the video playback module can play the video images and the default audio. After the user selects the audio, the video playback module can play the video selected by the user.

[0284] In this way, during video playback, the electronic device can switch to play panoramic audio, panoramic audio +, or audio corresponding to different display areas, giving users a variety of audio playback options, achieving audio adjustability, and improving the user's audio experience of multi-channel recordings.

[0285] Furthermore, the electronic device can switch to play audio corresponding to different display areas, so that the played audio matches the video screen in the display area that the user is paying attention to in real time, thereby improving the user's audio experience.

[0286] Moreover, the electronic device can play audio corresponding to characteristic values ​​such as zoom ratio or front / rear attribute parameters that change in real time within the display area, so that the audio matches the video pictures that change according to characteristic values ​​such as zoom ratio or front / rear attribute parameters in real time, thereby improving the user's audio experience.

[0287] In some other embodiments, the microphone in the audio hardware module is rotatable, and the electronic device can adjust the sound pickup direction by rotating the microphone, and determine the sound pickup range in combination with the audio processing module. In this case, the audio corresponding to the display area can be picked up by only part of the microphones of the electronic device, without using all the microphones of the electronic device.

[0288] In some other embodiments, the electronic device may include more than three microphones, so that it can collect sounds from more directions and more areas, and the sound collection range is more accurate, so as to subsequently produce better sounds in different pickup ranges. In addition, when the electronic device includes more than three microphones, different types of audio (for example, panoramic audio, or audio corresponding to different display areas) can be picked up by a combination of different microphones to pick up sounds within the corresponding ranges.

[0289] In some other embodiments, the audio processing method provided in the embodiments of the present application can also be combined with an image recognition algorithm. The electronic device can further process the recognized object according to the conventional audio characteristics and conventional audio model of the object, thereby enhancing the audio effect. Fig.19 The solution can generally include: using a camera to capture video images, identifying objects based on an image recognition algorithm, obtaining conventional audio models of objects (such as musical instruments, natural scenery (wind, rain, thunder), animal calls, etc.), and then superimposing the audio processing method in the above-mentioned embodiment of the present application, and then optimizing the audio effect.

[0290] For example, based on an image recognition algorithm, an electronic device can recognize an object in a video screen and recognize the sound of the object, thereby retaining the sound of the object recognized in the sound collected by the microphone and suppressing other sounds, performing audio optimization processing, etc. Fig.14 In the band performance scene shown, for the second display area on the right, the mobile phone can recognize that the objects on the video screen include the guitarist and the guitar, so the mobile phone can recognize and retain the guitarist's singing and the guitar's sound from the sound signals in the second pickup range corresponding to the second display area, and suppress other sounds. In addition, if the guitarist's singing and the guitar's sound retained by the mobile phone are distorted or incomplete, they can be optimized according to the human voice model and the guitar's sound model.

[0291] Among them, due to the accuracy problems of image algorithms, and the fact that conventional audio models may be more practical for fixed devices, but have poor effects on irregular sounds produced by abnormalities, and there is no fixed audio model to match, etc.; for audio segments that cannot be processed, in order to prevent distortion, they can be left unprocessed and recognized by a precise fixed audio model recognition algorithm, thereby enhancing noise reduction optimization processing. For unrecognizable sounds, the original sound is kept unprocessed. This solution can bring users a better listening experience.

[0292] In some other embodiments, in the traditional recording mode (i.e., single-channel recording mode), the electronic device can record the audio corresponding to the characteristic values ​​such as the zoom factor that changes in real time while recording the video. During video playback, the electronic device can play the video and audio that match the characteristic values ​​such as the zoom factor in real time. In this way, the audio recorded by the electronic device matches the video within the shooting range that the user is concerned about in real time, which can improve the user's audio experience. Alternatively, the electronic device can record panoramic audio and audio corresponding to characteristic values ​​such as the zoom factor that changes in real time. Alternatively, the electronic device can record panoramic audio, panoramic audio+, and audio corresponding to characteristic values ​​such as the zoom factor that changes in real time. During video playback, the user can switch to play different audio to improve the audio playback experience.

[0293] The embodiment of the present application also provides an electronic device, including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, and the computer program codes include computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the above-mentioned related method steps to implement the audio processing method in the above-mentioned embodiment.

[0294] An embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the audio processing method in the above-mentioned embodiment.

[0295] The embodiments of the present application further provide a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the audio processing method executed by the electronic device in the above-mentioned embodiments.

[0296] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the audio processing method executed by the electronic device in the above-mentioned method embodiments.

[0297] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0298] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0299] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0300] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0301] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0302] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0303] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An audio processing method applied to an electronic device, the electronic device comprising a first camera and a second camera, characterized in that: The method comprises: After the electronic device detects that the user opens the camera application, the electronic device displays a shooting preview interface; The electronic device enters a multi-channel video recording mode; After the electronic device detects the shooting operation of the user, the shooting interface is displayed, and the shooting interface includes a first display area and a second display area, the first display area is used to display the first video picture collected by the first camera, and the second display area is used to display the second video picture collected by the second camera; The electronic device simultaneously records the first video image and the second video image through the first camera and the second camera; The electronic device collects sound signals from all directions and obtains initial audio data according to the sound signals; The electronic device records audio of multiple sound channels, where the audio of the multiple sound channels includes panoramic audio, first audio corresponding to the first video screen, and second audio corresponding to the second video screen; The step of recording, by the electronic device, the first audio among the audios of the multiple sound channels includes: The electronic device determines a first sound pickup range corresponding to the first display area according to a characteristic value corresponding to the first display area; the characteristic value includes a zoom factor of the first camera and a distance between the first camera and a frame of the electronic device; The electronic device records the first audio according to the first sound pickup range and the initial audio data by enhancing the sound within the first sound pickup range and suppressing the sound outside the first sound pickup range in the initial audio data.

2. The method according to claim 1, characterized in that The characteristic value also includes a front / rear attribute parameter corresponding to the first display area, and the front / rear attribute parameter is used to indicate whether the first video picture is a video picture taken by a front camera or a video picture taken by a rear camera.

3. The method according to claim 1 or 2, characterized in that: The characteristic value also includes the focal length range of the first camera.

4. The method according to claim 1, characterized in that: The audio of the multiple sound channels also includes a third audio corresponding to the first video picture.

5. The method according to claim 1, characterized in that: The audio of the multiple sound channels also includes a fourth audio, the fourth audio includes a fifth audio of a first duration and a sixth audio of a second duration, the fifth audio is the panoramic audio, and the sixth audio includes an audio corresponding to a first zoom factor of the first camera; wherein, within the first duration, the first camera uses the second zoom factor for shooting, and within the second duration, the first camera uses the first zoom factor for shooting.

6. The method according to claim 1, characterized in that The first display area also includes a front / rear switching control, and the front / rear switching control is used to switch the front / rear camera.

7. The method according to claim 1, characterized in that The electronic device further includes a third camera, the first display area includes a camera switching control, and the method further includes: The electronic device displays the video image captured by the third camera in the first display area in response to the operation on the camera switching control; Among them, the first camera is a rear wide-angle camera, and the third camera is a rear telephoto camera.

8. The method according to claim 1, characterized in that The first display area further includes a first zoom adjustment control, and the first zoom adjustment control is used to adjust the zoom ratio of the first camera; The second display area also includes a second zoom adjustment control, and the second zoom adjustment control is used to adjust the zoom ratio of the second camera.

9. The method according to claim 1, characterized in that: The method further comprises: When the electronic device determines that the third zoom factor set by the user for the first display area exceeds the focal range of the first camera, prompting the user to switch to the fourth camera; The electronic device displays the video images captured by the fourth camera according to the third zoom factor in the first display area.

10. The method according to claim 1, characterized in that The first display area corresponds to a first focal length range, the second display area corresponds to a second focal length range, the first focal length range is different from the second focal length range, and the method further includes: When the electronic device determines that the fourth zoom multiple set by the user for the first display area exceeds the first focal length range and enters the second focal length range, the electronic device prompts the user to perform a zoom operation on the zoom adjustment control of the second display area; The electronic device displays, in the second display area, a video image captured by the second camera according to the fourth zoom factor.

11. The method according to claim 1, characterized in that: After the electronic device detects a shooting operation of the user, the method further includes: After the electronic device detects that the user stops shooting, it generates a multi-channel video recording, wherein the multi-channel video recording includes the first video screen, the second video screen and the audio of the multiple sound channels; After the electronic device detects the user's operation of playing the multiple recorded videos, the electronic device displays a video playing interface, where the video playing interface includes the first video screen and the second video screen; The electronic device automatically plays the panoramic audio.

12. The method according to claim 11, characterized in that The video playback interface further includes an audio switching control. After the electronic device automatically plays the panoramic audio, the method further includes: After the electronic device detects the user's operation on the audio switching control, the electronic device plays the first audio or the second audio; The electronic device stops playing the panoramic audio.

13. The method according to claim 11 or 12, characterized in that: After the electronic device displays the video playback interface, the method further includes: After the electronic device detects the user's operation of playing the first video picture, the electronic device displays the first video picture on the video playing interface and stops displaying the second video picture; The electronic device automatically plays the first audio corresponding to the first video picture.

14. The method according to claim 13, characterized in that The electronic device displays the first video picture on the video playback interface, including: The electronic device displays the first video image in an enlarged manner or in full screen on the video playback interface.

15. The method according to claim 11, characterized in that The first video frames included in the multiple recorded videos are saved as a first video file, and the second video frames included in the multiple recorded videos are saved as a second video file.

16. The method according to claim 1, characterized in that Before the electronic device displays the shooting interface, the method further includes: The electronic device determines the number of the display areas; The electronic device displays the shooting interface, including: The electronic device displays the shooting interface according to the number of the display areas.

17. An electronic device, characterized in that: include: Multiple microphones for collecting sound signals; Multiple cameras for capturing video images; Screen, used to display the interface; An audio playing component, used for playing audio; one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the audio processing method as described in any one of claims 1-16.

18. The electronic device according to claim 17, characterized in that: The plurality of microphones include at least three, and at least one microphone is respectively disposed on the top, bottom and back of the electronic device.

19. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on a computer, enable the computer to execute the audio processing method according to any one of claims 1 to 16.

20. A computer program product, characterized in that When the computer program product runs on a computer, the computer is enabled to execute the audio processing method according to any one of claims 1 to 16.

Images