Audio processing method and electronic device

Abstract

This application discloses an audio processing method and an electronic device, relating to the field of terminals. The method includes: the electronic device receiving a user operation that allows the user to select to enter a main character mode for recording. In response to the user operation, the electronic device obtains configuration information for an audio transmission path. This configuration information may include a value for the number of audio channels. The electronic device can verify the number of audio channels in the configuration information based on audio policy management information. Upon successful verification, the electronic device can configure the audio transmission path based on the configuration information. The audio transmission path includes a first number of audio channels. This first number is the same as the value of the number of audio channels in the aforementioned configuration information. During video recording, the electronic device can acquire audio signals in real time through a microphone. The electronic device can process the audio signals based on the audio transmission path to generate a first target audio signal and a second target audio signal.

Description

Technical Field

[0001] This application relates to the field of terminals, and more particularly to an audio processing method and an electronic device. Background Technology

[0002] With the development of terminal technology, users are increasingly using mobile phones and other electronic devices to take photos and / or videos to record their daily lives. The shooting modes achievable by electronic devices are also becoming increasingly diverse. For example, electronic devices can implement an autofocus-tracking shooting mode. In this mode, the electronic device can shoot based on a user-selected object, ensuring the camera's focus remains on that object throughout the shooting process. After shooting, the electronic device can output a video with the selected object centered on it.

[0003] However, in the current autofocus mode, when the electronic device plays a video with the selected object as the center, it outputs all the sounds of the surrounding environment, including the selected object. Therefore, the electronic device cannot accurately represent the characteristics of the selected object in its audio output, and the output quality of the selected object's sound signal is also poor, significantly impacting the user experience. Summary of the Invention

[0004] This application provides an audio processing method and an electronic device that configures four channels in the audio transmission path, enabling the electronic device to reproduce the protagonist's voice characteristics in the audio output effect. Furthermore, the electronic device can improve the output quality of the protagonist's voice signal. Simultaneously, the electronic device can also save all sounds in the protagonist's environment based on a panoramic video file and output different audio signals based on different video files to achieve different user intentions.

[0005] In a first aspect, this application provides an audio processing method applied to an electronic device, the electronic device including a display screen, a camera, and a microphone, comprising: the electronic device displaying a first interface, the first interface including a first screen and a first shooting mode control, the first screen being an image captured in real time by the electronic device through the camera, the first screen including a first shooting object; responding to a first operation acting on the first shooting mode control, the electronic device entering a first shooting mode and configuring an audio transmission path, wherein the audio transmission path includes a first number of channels, the first number of channels including a first channel, a second channel, a third channel, and a fourth channel; after the electronic device receives a first input selecting the first shooting object as the main subject, the electronic device starting recording and displaying a second interface, and capturing audio in real time through the microphone. The signal, wherein the second interface includes a first window and a second window, the first window including a first video frame captured in real time by the electronic device through the camera, and the second window including a second video frame cropped based on the position of the first subject in the first video frame; the electronic device obtains a first target audio signal from the audio signal based on the first channel audio data and the second channel audio data; the electronic device obtains a second target audio signal from the audio signal based on the third channel audio data and the fourth channel audio data; the electronic device receives a second input; in response to the second input, the electronic device obtains a first video file based on the first video frame and the first target audio signal; the electronic device obtains a second video file based on the second video frame and the second target audio signal.

[0006] In one possible implementation, in response to a first operation applied to the first shooting mode control, the electronic device enters a first shooting mode and configures an audio transmission path, specifically including: in response to the first operation, the electronic device obtains configuration information of the audio transmission path; the electronic device verifies the number of channels in the configuration information based on audio policy management information; when the configuration information passes the verification, the electronic device configures the audio transmission path based on the configuration information.

[0007] In one possible implementation, the configuration information includes one or more of the following: number of channels, sample size of audio sampling points, total size of buffer block, and number of frames in the buffer.

[0008] In one possible implementation, the number of channels in the configuration information is a first value; the electronic device verifies the configuration information based on audio policy management information, specifically including: the electronic device determining whether the audio policy management information includes a first identifier corresponding to the first value based on the first value; when the electronic device determines that the audio policy management information includes the first value, the electronic device determines that the configuration information passes the verification; when the electronic device determines that the audio policy management information does not include the first value, the electronic device determines that the configuration information fails the verification.

[0009] In one possible implementation, after the configuration information passes verification, the electronic device configures the audio transmission path based on the configuration information, specifically including: when the electronic device determines that the audio policy management information includes the first identifier, the electronic device configures the audio transmission path based on the configuration information; wherein, the audio transmission path includes a first number of channels, and the first number is the same as the first value.

[0010] In one possible implementation, in response to a first operation applied to the first shooting mode control, the method further includes: the electronic device displaying a first tracking mark in the area of ​​the first shooting object and a second tracking mark in the area of ​​the second shooting object; wherein the first tracking mark and the second tracking mark are used to mark the shooting object identified by the electronic device based on the first image.

[0011] In one possible implementation, after the electronic device receives a first input selecting the first subject as the main subject, the color and / or shape of the first tracking identifier changes.

[0012] In one possible implementation, the first quantity is 4.

[0013] In a second aspect, embodiments of this application provide an electronic device, including: one or more processors, one or more memories, and a display screen; the one or more memories are coupled to one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when the one or more processors execute the computer instructions, cause the electronic device to perform the method in any possible implementation of the first aspect described above.

[0014] Thirdly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method in any possible implementation of the first aspect described above.

[0015] Fourthly, embodiments of this application provide a chip or chip system, including a processing circuit and an interface circuit. The interface circuit is used to receive code instructions and transmit them to the processing circuit. The processing circuit is used to execute the code instructions to perform the method in any possible implementation of the first aspect described above.

[0016] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the method in any possible implementation of the first aspect described above. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0018] Figure 2 A schematic diagram of a software structure framework provided in an embodiment of this application;

[0019] Figure 3 A user interface diagram provided for an embodiment of this application;

[0020] Figures 4A-4B A set of user interface diagrams provided for embodiments of this application;

[0021] Figure 5 A user interface diagram provided for an embodiment of this application;

[0022] Figures 6A-6C A set of user interface diagrams provided for embodiments of this application;

[0023] Figures 7A-7D A set of user interface diagrams provided for embodiments of this application;

[0024] Figures 8A-8B A set of user interface diagrams provided for embodiments of this application;

[0025] Figures 9A-9C A set of user interface diagrams provided for embodiments of this application;

[0026] Figures 10A-10C A set of user interface diagrams provided for embodiments of this application;

[0027] Figure 11 A user interface diagram provided for an embodiment of this application;

[0028] Figure 12 A schematic diagram of a user interface provided for an embodiment of this application;

[0029] Figures 13A-13BA set of user interface diagrams provided for embodiments of this application;

[0030] Figure 14 A schematic diagram of a user interface provided for an embodiment of this application;

[0031] Figure 15 This application provides a schematic diagram of a module interaction process as an embodiment.

[0032] Figure 16 A schematic diagram illustrating a microphone for acquiring sound signals, provided as an embodiment of this application;

[0033] Figure 17 A flowchart illustrating an audio processing method provided in an embodiment of this application; Detailed Implementation

[0034] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to including one or more of the listed prominent features, any or all possible combinations. In the embodiments of this application, the terms “first” and “second” are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, “a plurality” means two or more.

[0035] In some implementations, when an electronic device takes a picture using a camera, it can receive a user-selected object based on the captured image. The electronic device can ensure that the camera's focus remains on the selected object throughout the recording process. After recording, the electronic device can output a video centered on the selected object. However, when the electronic device outputs the audio for this video, it outputs all sounds from the surrounding environment, including the selected object. Therefore, the electronic device cannot accurately represent the characteristics of the selected object in its audio output, and the output quality of the selected object's audio signal is poor, significantly impacting the user experience.

[0036] Therefore, this application provides an audio processing method that can be applied to electronic devices such as mobile phones and tablets. Hereinafter, electronic device 100 will be used to refer to the aforementioned electronic device.

[0037] Not limited to mobile phones and tablets, electronic device 100 can also be a desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. The specific type of terminal is not particularly limited in the embodiments of this application.

[0038] Specifically, in the audio processing method provided in this application, the electronic device 100 can process the raw audio signals collected in real time by multiple microphones based on the object selected by the user on the display interface to generate two video files. The multiple microphones can form a microphone array, which can be arranged according to a specified arrangement rule. For example, the multiple microphones can be arranged linearly, in a triangle, or in a circle. The two video files can be referred to as a panoramic video file and a close-up video file, respectively. The object selected by the user can be referred to as the main character. The panoramic video file can include: a panoramic video frame and a target audio signal 1. The close-up video file can include: a close-up video frame and a target audio signal 2. The panoramic video frame can be: the raw image stream collected by the electronic device 100 based on the camera during video recording. This raw image stream can include: the raw image captured by the camera at each moment during video recording. The raw image captured by the camera at any moment can be referred to as a frame. The close-up video frame can be: a frame cropped from the position of the main character in the panoramic video frame. The target audio signal 1 is all the sound signals in the environment where the main character is located, obtained by the electronic device 100 based on the raw audio signal processing. Target audio signal 2 is the sound signal where the protagonist's voice is clearer and more prominent than the voices of other speakers. That is, the protagonist's voice in target audio signal 2 is more prominent than the protagonist's voice in target audio signal 1. The panoramic video frame can be referred to as the first video frame, and the close-up video frame can be referred to as the second video frame.

[0039] In this way, the electronic device 100 can reflect the protagonist's voice characteristics in the audio output effect. Furthermore, the electronic device 100 can improve the output quality of the protagonist's voice signal. Simultaneously, the electronic device 100 can also save all the sounds in the protagonist's environment based on the panoramic video file, and output different audio signals based on different video files to achieve different user intentions.

[0040] Next, the hardware structure of an electronic device 100 provided in the embodiments of this application will be introduced.

[0041] Please refer to Figure 1 , Figure 1 An exemplary schematic diagram of the hardware structure of an electronic device 100 is shown.

[0042] Electronic device 100 may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. The embodiments of this application do not impose any special restrictions on the specific type of electronic device 100.

[0043] like Figure 1 As shown, the electronic device 100 may include a processor 101, a memory 102, a wireless communication module 103, a display screen 104, a camera 105, an audio module 106, and a microphone 107.

[0044] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0045] Processor 101 may include one or more processor units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0046] The processor 101 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 101 is a cache memory. This memory can store instructions or data that the processor 101 has just used or that are used repeatedly. If the processor 101 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 101, and thus improves the efficiency of the system.

[0047] In some embodiments, the processor 101 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a USB interface, etc.

[0048] The memory 102 is coupled to the processor 101 and is used to store various software programs and / or multiple sets of instructions. In specific implementations, the memory 102 may include volatile memory, such as random access memory (RAM); it may also include non-volatile memory, such as ROM, flash memory, hard disk drive (HDD), or solid-state drive (SSD); the memory 102 may also include combinations of the above types of memory. The memory 102 may also store some program code so that the processor 101 can call the program code stored in the memory 102 to implement the implementation method of the present application embodiment in the electronic device 100. The memory 102 may store an operating system, such as uCOS, VxWorks, RTLinux, or other embedded operating systems.

[0049] In this embodiment, the memory 102 can be used to store panoramic video files and close-up video files generated by the electronic device 100. Detailed descriptions can be found in subsequent embodiments, and will not be repeated here.

[0050] The wireless communication module 103 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 103 can be one or more devices integrating at least one communication processing module. The wireless communication module 103 receives electromagnetic waves via an antenna, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to the processor 101. The wireless communication module 103 can also receive signals to be transmitted from the processor 101, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via the antenna. In some embodiments, the electronic device 100 can also communicate via the Bluetooth module in the wireless communication module 103 (… Figure 1 (not shown), WLAN module ( Figure 1(Not shown) The device transmits signals to detect or scan devices near electronic device 100 and establishes wireless communication connections with those devices to transmit data. The Bluetooth module can provide solutions for one or more Bluetooth communication methods, including basic rate / enhanced data rate (BR / EDR) or Bluetooth Low Energy (BLE), and the WLAN module can provide solutions for one or more WLAN communication methods, including Wi-Fi direct, Wi-Fi LAN, or Wi-Fi softAP.

[0051] The display screen 104 can be used to display images, videos, etc. The display screen 104 may include a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 104, where N is a positive integer greater than 1.

[0052] In this embodiment, the display screen can show controls related to the main character mode, a preview screen, a recording screen, one or more shooting objects, and tracking icons associated with the shooting objects. During video recording, the display screen can also display the aforementioned panoramic video and close-up video. For details on the specific interface elements displayed on the display screen, please refer to the subsequent user interface examples; they will not be elaborated upon here.

[0053] Camera 105 can be used to capture still images or videos. An object is projected onto a photosensitive element through a lens, generating an optical image. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some examples, electronic device 100 may include one or N cameras 105, where N is a positive integer greater than 1.

[0054] The audio module 106 can be used to convert digital audio information into analog audio signal output, and can also be used to convert analog audio input into digital audio signal. The audio module 106 can also be used to encode and decode audio signals. In some embodiments, the audio module 106 can also be disposed in the processor 101, or some functional modules of the audio module 106 can be disposed in the processor 101.

[0055] In this embodiment, the audio module 106 can configure an audio transmission path including a first number of channels based on the configuration information of the audio transmission path, and process the audio stream (i.e., audio signal) collected by the microphone 107. Detailed explanations can be found in subsequent embodiments, and will not be repeated here.

[0056] Microphone 107, also known as a "microphone" or "voice transducer," is used to collect sound signals from the environment surrounding the electronic device. This sound signal is then converted into an electrical signal, which undergoes a series of processing steps, such as analog-to-digital conversion, to obtain a digital audio signal that can be processed by the processor 101 of the electronic device. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to the microphone 107, inputting the sound signal into the microphone 107. The electronic device 100 may have at least one microphone 107. In some embodiments, the electronic device 100 may have two microphones 107, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, the electronic device 100 may have three, four, or more microphones 107, enabling sound signal collection, noise reduction, sound source identification, and directional recording, among other functions.

[0057] Electronic device 100 may also include a sensor module ( Figure 1 (not shown) and / or touch sensor ( Figure 1(Not shown). A touch sensor can also be called a "touch device". The touch sensor can be disposed on the display screen 104, and the touch sensor and the display screen 104 together form a touch screen, also called a "touchscreen". The touch sensor can be used to detect touch operations applied to or near it. Optionally, the sensor module may also include a gyroscope sensor (…). Figure 1 (not shown), accelerometer ( Figure 1 (Not shown), etc. Among them, the gyroscope sensor can be used to determine the motion attitude of the electronic device 100. In some embodiments, the electronic device 100 can determine its angular velocity around three axes (i.e., the x, y, and z axes) using the gyroscope sensor. The accelerometer sensor can be used to detect the magnitude of the acceleration of the electronic device 100 in various directions (generally the x, y, and z axes), and can also detect the magnitude and direction of gravity when the electronic device 100 is stationary.

[0058] It should be noted that, Figure 1 The electronic device 100 shown is merely an illustrative explanation of the hardware structure of the electronic device provided in this application and does not constitute a specific limitation on this application.

[0059] The following describes a software structure framework for an electronic device 100 provided by an embodiment of this application.

[0060] Please refer to Figure 2 , Figure 2 An exemplary schematic diagram of the software architecture framework of an application electronic device 100 is shown.

[0061] like Figure 2 As shown, this software architecture framework may include: an application layer, an application framework layer, a hardware abstraction layer, and a hardware layer. Wherein:

[0062] The application layer may include a CameraApp module. The CameraApp module may include a UI module, a focus tracking module, and a camera management module. The UI module exemplifies modules such as a photo-taking module, a video recording module, and a protagonist mode module. The UI module can provide different UI elements (e.g., controls, text, icons) based on these different modules. For example, it might provide UI elements for the photo-taking interface based on the photo-taking module, UI elements for the video recording interface based on the video recording module, and UI elements for the protagonist mode interface based on the protagonist mode module. The focus tracking module includes a camera focus tracking module and an audio focus tracking module. The camera focus tracking module can identify the position of the selected protagonist in the image and display the protagonist in the protagonist window. The audio focus tracking module can extract target audio signal 1 and target audio signal 2 from the buffer block of the audio transmission path. The camera management module may include a device management module, a session management module, and a surface management module. These modules can be referenced from corresponding modules in existing camera frameworks and will not be elaborated further here.

[0063] The application framework layer can include the camera framework CameraFWK and the audio service. The CameraFWK module can include the camera management module CameraManager, the camera device module CameraDevice, the session module Session, and the output module output. These modules are similar to their counterparts in existing camera frameworks and will not be elaborated upon here. The audio service (audio_server) module can include the audio management (AudioManager) module, the audio stream (Stream), the audio device (AudioDevice) module, and the audio policy (policy) module. The AudioManager and AudioDevice modules are similar to their counterparts in existing audio frameworks and will not be elaborated upon here. The audio stream is used to transmit the audio stream captured by the microphone, and the audio policy module is used to verify the configuration information of the audio transmission path.

[0064] The hardware abstraction layer can include a camera hardware abstraction layer (CameraHAL) and an audio abstraction layer (AudioHAL). CameraHAL can include a frame stream module (Stream), an image capture module (Capture), a camera device (camera3_device) management module, and a subject recognition module. The frame stream module can be used to buffer image frame streams captured by the camera. The Capture module can be used to initiate camera capture of images. Camera device management can be used to initiate camera photography or video recording, etc. The subject recognition module stores a subject recognition algorithm (e.g., the ReID algorithm), which can be used to identify the subject's position in the image (including the aforementioned facial coordinate information). The audio abstraction layer (AudioHAL) can include an audio device (AudioDeivce) module, two audio stream modules, and an audio tracking module (Track). The AudioDeivce and Stream modules can be referenced from corresponding modules in existing audio frameworks, and will not be elaborated upon here. The audio tracking module can be used to configure the audio transmission path based on the configuration information of the audio transmission path, and to process the audio stream captured by the microphone in response to the user-selected protagonist, so as to generate all the sound signals in the protagonist's environment (i.e., the aforementioned target audio signal 1) and the sound signal in which the protagonist's voice is clearer and more prominent than other sound sources (i.e., the aforementioned target audio signal 2).

[0065] The hardware layer can include a camera (also known as a webcam) and a microphone. The camera is used to capture images of the scene within the field of view, and the microphone is used to collect sound from the shooting environment.

[0066] Specifically, the interaction process between the modules will be explained in detail in subsequent embodiments, and will not be repeated here.

[0067] The following is a schematic diagram of the user interface of the electronic device 100 implementing the audio processing method provided in this application.

[0068] The audio processing method provided in this application embodiment can be applied in conjunction with a video recording method provided in this application embodiment. In this video recording method, the electronic device 100 can provide a main character mode video recording function, which will be described below in conjunction with... Figure 3 The main character mode involved in the embodiments of this application is further explained.

[0069] To facilitate the description of the recording method provided in the embodiments of this application, the definitions of terms used in the embodiments of this application are explained below. The "protagonist mode" can be understood as a mode in which an additional portrait tracking video can be generated when the electronic device 100 is recording video. The portrait in this portrait tracking video can be understood as the "protagonist" that the user is interested in. The method for generating the video corresponding to the "protagonist" can be: cropping the video content corresponding to the "protagonist" from the video conventionally recorded by the electronic device 100. It is understood that the protagonist mode of the electronic device 100 can provide a preview mode and a recording mode. In preview mode, the preview interface can be displayed on the screen of the electronic device 100. In recording mode, the recording interface can be displayed on the screen of the electronic device 100. It is understood that the large window described later can be referred to as the first window, and the small window can be referred to as the second window. The large window includes the panoramic video image captured in real time by the electronic device through the camera, and the small window includes the close-up video image obtained by cropping based on the position of the protagonist in the panoramic video image.

[0070] It should be noted that the interface displayed by the electronic device 100 in both preview mode (before recording) and recording mode (during recording) can be referred to as the preview interface; the screen displayed in the preview interface in preview mode (before recording) will not generate a video and be saved; the screen displayed in the preview interface in recording mode (during recording) can generate a video and be saved. For ease of distinction, in the following text, the preview interface in preview mode (before recording) will be referred to as the preview interface; and the preview interface in recording mode (during recording) will be referred to as the recording interface.

[0071] The preview interface may include a large window and a small window. The large window can be a window with dimensions equal to or slightly smaller than the display screen, and it displays the image captured by the camera. The image displayed in the large window in preview mode can be defined as the preview screen of the large window. The small window can be a window with dimensions smaller than the large window, and it displays the image of the user-selected tracking object. The electronic device 100 can select the tracking object based on a tracking identifier associated with it, and the image displayed in the small window in preview mode can be defined as the preview screen of the small window. It is understood that in preview mode, the electronic device 100 can display the image captured by the camera in the large window and the image of the tracking object in the small window, but the electronic device 100 may not generate video or save the content displayed in the large and small windows.

[0072] The recording interface may include a large window and a small window. The large window can be a window with dimensions equal to or slightly smaller than the display screen, displaying the image captured by the camera. The image displayed in the large window during recording mode can be defined as the recording screen of the large window. The small window can be a window with dimensions smaller than the large window, displaying the image of the user-selected focus target. The image displayed in the small window during recording mode can be defined as the recording screen of the small window. It is understood that in recording mode, the electronic device 100 can not only display the recording screens of the large and small windows, but also generate the large window video and the small window video recorded after starting the recording mode. The electronic device 100 can save the video generated in the large window when recording in the large window ends, and save the video generated in the small window when recording in the small window ends. This application embodiment does not limit the naming of the preview mode and the recording mode.

[0073] It should be noted that the preview interface described in this embodiment can be understood as the preview mode of the camera application of the electronic device 100 in the main subject mode; the recording interface can be understood as the recording mode of the camera application of the electronic device 100 in the main subject mode. Further details will not be provided hereafter.

[0074] For example, the aforementioned protagonist mode functionality can be implemented in a camera application (also known as a camera or camera app). For instance, in a preview scene, the preview interface of protagonist mode on electronic device 100 can be as follows: Figure 3 As shown in Figure a, the preview interface may include a large window 301, a small window 302, and multiple buttons. These buttons may include a start recording button 303, a first landscape / portrait switching button 304, a second landscape / portrait switching button 305, a small window close button 306, and an exit protagonist mode button 307. Optionally, the buttons may also include a recording settings button 308, a flash button 309, and a zoom button 310, etc.

[0075] The electronic device 100 can display a preview screen in a large window 301, which may include multiple people. When the electronic device 100 detects the presence of people in the preview screen of the large window, a tracking icon associated with the person can be displayed in the preview screen. For example, the tracking icon can be a tracking box (e.g., tracking box 311 and tracking box 312) displayed at the corresponding position of the person. For example, a male person in the preview screen can correspond to tracking box 311, and a female person can correspond to tracking box 312. The tracking box can prompt the user that the corresponding person can be set as the focus target or can be switched to the focus target. When the electronic device 100 recognizes N people, M (M≤N) tracking boxes can be displayed in the large window. The electronic device 100 can set any person as the focus target to generate video content for that focus target. This application embodiment does not limit the "protagonist", where the "protagonist" can be a living being such as a person or an animal, or a non-living being such as a vehicle. It is understood that any item that can be identified based on an algorithm model can be used as the "protagonist" in this application embodiment. In this embodiment, the "protagonist" can be defined as the focus tracking object. The focus tracking object can also be called the protagonist object, the tracking target, the tracking object and the focus tracking target, etc. This embodiment uses a person as the "protagonist" for illustrative purposes, but this embodiment does not limit the concept of "protagonist".

[0076] In some embodiments, the tracking identifiers can also be other forms of tracking identifiers. For example, when the electronic device 100 identifies multiple focusable objects, a large window displays a tracking identifier corresponding to the focusable object near the focusable object. The tracking identifier can be numbers, letters, graphics, etc. When a user clicks a tracking identifier, the electronic device 100 responds to the click operation and selects the focusable object. As another example, multiple focusable objects in the large window are marked with numbers, graphics, user images, or other tracking identifiers. The electronic device 100 can arrange multiple tracking identifiers at the edge or other locations of the large window display area, and the user can click on the tracking identifiers in the large window to select the focusable object. This application embodiment uses a tracking frame as an example to illustrate the recording method, but this application embodiment does not limit the form of the tracking identifier.

[0077] It should be noted that the electronic device 100 in this application embodiment can mark the corresponding tracking box for the person when it recognizes two or more people; the electronic device 100 can also mark the corresponding tracking box for a single person when it recognizes a single person, or it can choose not to mark the tracking box, and there is no limitation here.

[0078] Optionally, the N figures displayed in the large window can be trackable objects, with the selected "main character" being the trackable object and the unselected characters being other objects. The tracking frame of the trackable object (e.g., tracking frame 311) can display different styles from the tracking frames of other objects (e.g., tracking frame 312). This makes it easier for users to distinguish the tracked characters.

[0079] In some embodiments, the shape, color, size, and position of the tracking frame are adjustable. For example, the tracking frame 311 for the object being tracked can be a dashed frame. The tracking frame 312 for other objects can be a combination of a dashed frame and a "+" sign. Besides different shapes, embodiments of this application can also set the color of the tracking frame; for example, tracking frame 311 and tracking frame 312 can be different colors. This allows for a direct distinction between the object being tracked and other objects. It is understood that the tracking frame can also have other display forms, as long as it satisfies the function of being triggered by the user to track the focusable object.

[0080] The tracking frame can be marked at any position on the trackable object; this application embodiment does not impose specific limitations. In a possible implementation, to avoid visual interference with the preview of the trackable object in a large window, the tracking frame can avoid the face of the trackable object; for example, the tracking frame can be marked at a relatively central position on the body of the trackable object. The electronic device 100 can perform face recognition and body recognition. When the electronic device 100 recognizes a face, it can display the tracking frame. The electronic device 100 can determine the display position of the tracking frame based on face recognition and body recognition, and the tracking frame is displayed at the center of the body.

[0081] It should be noted that in some embodiments, the following scenario may occur: the preview screen in the large window includes N people, of which M (M≤N) are trackable objects with marked tracking boxes and NM people are not recognized by the electronic device 100. In actual shooting, the electronic device 100 can display tracking boxes based on face recognition technology. When the electronic device 100 cannot capture the face of a person (e.g., the back view of a person), the electronic device 100 will not mark a tracking box for that person. This application embodiment does not limit the method for implementing the tracking box display.

[0082] In the preview scene, the small window 302 displays a preview image of the object being tracked. The preview image in the small window can be a portion of the preview image in the large window. In a possible implementation, the preview image in the small window is obtained by cropping the preview image in the large window based on the object being tracked. The electronic device 100 can crop the image in the large window according to an algorithm, and the small window obtains a portion of the image from the large window. In some embodiments, when the cropping calculation takes a long time, the image displayed in the small window in real time may be a cropped image of the first few frames displayed in real time in the large window. This application embodiment does not specifically limit the image displayed in the small window.

[0083] When the focus target changes, the person in the preview displayed in small window 302 changes accordingly. For example, if the focus target changes from a male to a female person, the preview displayed in small window 302 will also change accordingly. This will be discussed later. Figure 5The scenarios in which electronic device 100 selects or switches the focus target will be described in detail, but will not be repeated here.

[0084] In some embodiments, the size, position, and landscape / portrait display mode of the small window are adjustable. Users can adjust the style of the small window according to their recording habits, which will be discussed later. Figures 6A-6C The details of adjusting the small window style are explained in detail, but will not be repeated here.

[0085] The preview interface also includes several buttons, the functions of which are explained below.

[0086] The Start Recording button 303 is used to control the electronic device 100 to start recording in both the large and small windows.

[0087] The first landscape / portrait switching button 304 can be displayed in the large window and is used to adjust the landscape and portrait display of the small window.

[0088] The second landscape / portrait switching button 305 can be displayed in a small window and is also used to adjust the landscape and portrait display of the small window.

[0089] Small window close button 306, used by electronic device 100 to close small windows.

[0090] The Exit Pro Mode button 307 is used by electronic device 100 to exit Pro Mode and enter regular recording mode.

[0091] Optionally, the buttons on the preview screen may also include:

[0092] The video recording settings button 308 is used to display the camera settings interface on the electronic device 100, where the user can adjust various shooting parameters of the camera.

[0093] The flash button 309 is used to set the flash effect. The flash button can be expanded into buttons with multiple flash functions, including controlling the flash to be forced on, forced off, turned on when taking a picture, and turned on according to the environment.

[0094] The zoom button 310 is used to adjust the zoom in the preview interface. It offers wide-angle, 1X, and 2X zoom functions. The zoom button can be used to adjust the size of the preview scene in the large window, while the preview scene in the small window can be scaled proportionally to the size of the large window.

[0095] Understandably, in a preview scenario, the preview interface may include a large window and a small window. The preview screen in the large window includes the trackable object. When the electronic device 100 selects the trackable object, the preview screen in the small window can display the trackable object in the center. In some scenarios, the trackable object may be in a moving state. When the trackable object moves but does not leave the lens, the preview screen in the small window can continue to display the trackable object in the center. For example, if the trackable objects in the preview interface include male and female figures, and the electronic device 100 responds to the user's click operation on the tracking box for the male figure, the electronic device 100 selects the male figure as the trackable object and enters the preview screen as follows: Figure 3 The interface shown in Figure a. Figure 3 In interface A, the preview screen in the small window displays the male character in the center, positioned to the right of the female character. As the male character moves, the electronic device 100 continuously tracks and focuses on him, centering his image in the small window. When the male character moves to the left of the female character, the interface of the electronic device 100 can be as follows... Figure 3 As shown in b. Figure 3 In the B interface, the preview window still displays the male character in the center, with the male character to the left of the female character.

[0096] For example, in a recording scenario, the recording interface of the protagonist mode on the electronic device 100 can be as follows: Figure 3 As shown in c. The recording interface may include a large window 301, a small window 302, multiple buttons, and a recording time. Among them, the buttons may include a pause recording button 313, an end recording button 314, and an end small window recording button 315.

[0097] Unlike the preview scene, in the recording scene, the small window 302 displays the recorded image of the object being tracked. During recording, the electronic device 100 can generate an additional video stream recorded in the small window, in addition to the video recorded in the large window. Similar to the preview process, the recorded image in the small window can be a portion of the recorded image in the large window. In a possible implementation, the recorded image in the small window is obtained by cropping the recorded image of the large window based on the object being tracked. The two video streams are stored independently in the electronic device 100. In this way, the video corresponding to the object being tracked can be obtained without subsequent manual editing of the entire video, making the operation simple and convenient and improving the user experience.

[0098] The recording interface may include multiple buttons, the functions of which are explained below.

[0099] The pause recording button 313 is used to pause video recording. Recording in both the large and small windows can be paused simultaneously. When the recording interface does not include the small window, the pause recording button 313 will only pause the recording in the large window.

[0100] The End Recording button 314 is used to end video recording. Recording of the large window and the small window can be ended simultaneously. When the recording interface does not include the small window, the End Recording button 314 can end the recording of only the large window.

[0101] The "End Small Window Recording" button 315 is used to end the recording of the small window video. The electronic device 100 can end the recording of the small window using the "End Small Window Recording" button 315, without affecting the recording of the large window.

[0102] Recording time indicates the duration of the currently recorded video. The recording time for the large window can be the same as or different from that for the small window.

[0103] Understandably, in a recording scenario, the recording interface may include a large window and a small window. The recording screen in the large window includes the focusable object. When the electronic device 100 selects the focusable object, the recording screen in the small window can display the focusable object in the center. In some scenarios, the focusable object may be in a moving state. When the focusable object moves but does not leave the lens, the focus moves with the focusable object, and the recording screen in the small window can continue to display the focusable object in the center. For example, the trackable objects in the recording interface include male and female figures. In response to the user's click operation on the tracking box for the male figure, the electronic device 100 selects the male figure as the focusable object and enters the recording screen as follows: Figure 3 The interface shown in c. Figure 3 In the c interface, the recording screen in the small window displays the male figure centered, positioned to the right of the female figure. The focus is on the male figure's face, located slightly to the right of the center of the frame. As the male figure moves, the electronic device 100 continuously tracks and records him, displaying him centered in the small window. When the male figure moves to the left of the female figure, the interface of the electronic device 100 can be adjusted as follows: Figure 3 As shown in b. Figure 3 In the b interface, the recording screen in the small window still displays the male figure in the center, to the left of the female figure. At this time, the focus is on the male figure's face area, which is located in the middle left part of the screen.

[0104] In this application embodiment, the shooting mode that generates an additional tracking video based on the tracking object is defined as the main subject mode. This shooting mode can also be called tracking mode, etc., and this application embodiment does not limit it.

[0105] There are several ways to enter protagonist mode when recording.

[0106] For example, the following is combined with Figure 4A and Figure 4B The method for entering the protagonist mode in the embodiments of this application will be described.

[0107] In one possible implementation, electronic device 100 is in Figure 4A The main interface shown in Figure a allows the electronic device 100 to enter the camera application 401 when it detects that the user has opened the camera application. Figure 4A The photo preview interface shown in b is as follows. This photo preview interface may include a preview screen and a shooting mode selection button. The preview screen can display the scene captured by the camera of the electronic device 100 in real time. The shooting mode selection button includes, but is not limited to: a "Portrait" button, a "Take Photo" button, a "Record Video" button 402, a "Pro" button, and a "More" button 403.

[0108] When the electronic device 100 detects that the user has clicked the "record" button 402, the electronic device 100 switches from the photo preview interface to the current one. Figure 4A The video preview interface shown in c; the video preview interface may include, but is not limited to: a protagonist mode button 404 for receiving triggers to enter protagonist mode, a recording settings button for receiving triggers to enter settings, a filter button for receiving triggers to enable filter effects, and a flash button for setting flash effects.

[0109] Electronic device 100 can enter protagonist mode via the protagonist mode button 404 in the video preview interface. For example, when a user clicks the protagonist mode button 404 in the interface, electronic device 100 responds to the click operation and enters the protagonist mode. Figure 4A The preview interface shown in d; in the preview interface, there can be multiple shooting objects in the large window. The electronic device 100 can identify the multiple shooting objects based on the image content of the large window. The multiple shooting objects can be used as trackable objects. The preview interface of the electronic device 100 can mark tracking boxes for each trackable object.

[0110] In another possible implementation, electronic device 100 is in Figure 4B The main interface shown in Figure a. When the electronic device 100 detects that the user has opened the camera application 401, the electronic device 100 can enter... Figure 4B The photo preview interface is shown in b. This photo preview interface may include multiple buttons, such as a "Portrait" button, a "Take Photo" button, a "Video Recording" button 402, a "Pro" button, and a "More" button 403. When the electronic device 100 detects that the user clicks the "More" button 403, the electronic device 100 enters the following... Figure 4B The interface shown in c is... Figure 4BThe interface of C can be expanded to display multiple function buttons under the "More" button. For example, the "More" button may include, but is not limited to: "Pro" button, "Panorama" button, "High Dynamic Range (HDR)" button, "Time-lapse" button, "Watermark" button, "Document Correction" button, "High Pixel" button, "Mini-Movie" button, and Protagonist Mode button 405. Alternatively, it can be understood that Protagonist Mode can be hidden within the "More" button.

[0111] When the user clicks Figure 4B When the main character mode button 405 is pressed in the interface of device c, the electronic device 100 responds to the click operation and can enter the following mode: Figure 4B The preview interface shown in d contains, Figure 4B The relevant content of the preview interface shown in d can be found in the description of the preview interface shown in d of 4A, and will not be repeated here.

[0112] It is understood that the embodiments of this application are based on Figure 4A and Figure 4B Two methods for an electronic device 100 to enter protagonist mode are shown, but the embodiments of this application are not limited to the above two methods. The protagonist mode button, in addition to the above two methods, may also include... Figure 4A c and Figure 4B The icon shown as 'c' can also be in other forms. The protagonist mode button can also be placed in other hidden menus or other window display locations; this application embodiment does not limit this.

[0113] After the electronic device 100 enters the main character mode, it can simultaneously record in a large window and generate an additional video clip of the user-selected "main character" displayed in a small window. It is understood that when the electronic device 100 is recording in main character mode, it can be in either landscape or portrait mode. The principle of implementing main character mode is similar in both landscape and portrait modes. To facilitate the explanation of the recording method in this application embodiment, the following embodiments will use landscape recording by the electronic device 100 as an example to describe multiple application scenarios of main character mode.

[0114] In this application, the embodiments are combined with Figures 5-8B This document provides a detailed description of the scenes in the preview mode of the protagonist mode, and combines this application's embodiments with... Figures 9A-14 This section provides a detailed explanation of the scenes in the recording mode within the protagonist mode. First, it introduces the scenes in the preview mode.

[0115] For example, Figure 5 This diagram illustrates the interface for selecting the focus object in the preview mode of the main character mode. Figure 5 As shown:

[0116] Electronic device 100 enters the preview mode of the main character mode, such as... Figure 5 As shown in Figure a, the electronic device 100 can display a preview interface for the main character mode. The preview interface includes multiple focusable objects, each of which can be marked with its own tracking box (for example, tracking box 311 for male characters and tracking box 312 for female characters).

[0117] The electronic device 100 can determine the object selected for focus tracking based on the user's click operation on the tracking frame. For example, if a user wants to preview the focused image of a male figure in a small window on the electronic device 100, they can click on the tracking frame 311 corresponding to the male figure. The electronic device 100 responds to this click operation and enters the following window: Figure 5 The interface shown in b.

[0118] like Figure 5 As shown in Figure b, when the electronic device 100 selects a male figure as the focus target, a small window appears floating in the large window of the preview interface. The small window displays the image corresponding to the position of the focus target in the large window. In the small window, the focus target can be centered, highlighting its "protagonist" status. Optionally, after the tracking frame of the focus target is triggered, the color of the tracking frame can change, for example, becoming lighter, darker, or changing to another color. The shape of the tracking frame can also change; for example, the tracking frame 311 for the male figure is a dashed frame, and the tracking frame 312 for the female figure is a combination of a dashed frame and a "+". In this embodiment, the tracking frame styles of the focus target and other objects can be any combination of different colors, sizes, and shapes to facilitate user differentiation between the focus target and other objects in the large window. Optionally, after the tracking frame of the focus target is triggered, the corresponding tracking frame can disappear, preventing the user from repeatedly selecting the already selected focus target.

[0119] Understandably, in the preview mode of the main character mode, users can manually change the focus object after selecting it, such as... Figure 5 As shown in the interface (b), when the electronic device 100 receives the user's click on the tracking box 312 of the female figure, it enters the following... Figure 5 The interface shown in Figure c. At this point, the focus target in the small window switches from the male character to the female character. The tracking frame state of the character changes; for example, the color and shape of the female character's tracking frame 312 change, while the male character's tracking frame 311 reverts to its unselected style. The changes in the tracking frame style can be seen in [reference needed]. Figure 5 The relevant descriptions in the interface shown in b are not repeated here.

[0120] Optionally, the electronic device 100 switches the focus target in preview mode, and the object displayed in the preview window changes from the original focus target to the new focus target. To make the switching process smoother, this embodiment also provides a dynamic effect for switching the focus target. For example, the design of the dynamic effect is described below using a male figure as the original focus target and a female figure as the new focus target.

[0121] In one possible implementation, the large window of the preview interface includes both male and female figures, while the small window displays the male figure as the focus target. When the electronic device 100 detects a click on the tracking box for the female figure, the preview in the small window can switch from focusing on the male figure to a panoramic view, and then back to focusing on the female figure. For example, if the small window initially displays the male figure centered, after the user clicks on the tracking box for the female figure, the cropping ratio between the small and large window previews increases. The small window preview can include more content from the large window preview, which can be represented by the male figure and his background gradually shrinking until the small window can simultaneously display a panoramic view of both the male and female figures. Subsequently, the small window centers and enlarges the female figure within the panoramic view. Optionally, the panoramic view can be a proportionally scaled-down preview of the large window, or it can be an image cropped from the shared area of ​​the male and female figures within the large window preview.

[0122] In another possible implementation, the large window of the preview interface includes both male and female figures, while the small window displays the male figure as the focus target. When the electronic device 100 detects a click on the tracking frame targeting the female figure, the focus point in the small window's preview gradually shifts from the male figure to the female figure. For example, if the small window initially displays the male figure in the center, after the user clicks on the female figure's tracking frame, the cropping ratio between the small and large window's preview remains unchanged, but the small window's preview will be cropped closer to the female figure according to the original cropping ratio. For instance, if the female figure is to the left of the male figure, during the switching of focus targets, the electronic device 100 shifts the male figure and his background in the small window to the right until the female figure is centered in the small window.

[0123] In this way, when the electronic device 100 switches the focus target, the transition from the original focus target to the new focus target in the small window is smoother, improving the user's recording experience.

[0124] In the preview mode of the main character mode, after a small window appears in the preview interface of the electronic device 100, the user can adjust the style of the small window according to their needs. The following section combines... Figure 6A , Figure 6B and Figure 6CThis section explains how to adjust the small window.

[0125] For example, Figure 6A A schematic diagram of an interface for adjusting the size of a small window, provided in an embodiment of this application, is shown. Figure 6A As shown in Figure a, the small window may include a functional area for a small window close button 306 and a second landscape / portrait screen switch button 305, and a non-functional area for displaying a preview of the focus-tracking object. The user can press any position in the non-functional area of ​​the small window with two fingers and adjust the distance between the two fingers. When the electronic device 100 detects the above pressing operation, it enters... Figure 6A The interface shown in b. Users can control the size of the small window by adjusting the distance between their two fingers. When the electronic device 100 detects a change in the distance between the two fingers, it can adjust the size of the small window accordingly; for example, the small window shrinks when the distance between the two fingers decreases, and increases when the distance increases. After the user releases their two fingers, the size of the small window remains the same as when the fingers were released. The preview interface is shown below. Figure 6A As shown in c.

[0126] It should be noted that in some embodiments, in the preview mode of the main character mode, when the electronic device 100 adjusts the size of the small window, the preview image of the small window can be enlarged or reduced relative to the original preview image of the small window by a certain proportion. For example, during the enlargement of the small window, the content of the preview image displayed in the small window remains unchanged; only the original preview image is enlarged proportionally according to the distance adjusted by two fingers. The characters and background in the original preview image are enlarged accordingly, resulting in an enlarged preview image, such as... Figure 6A shown.

[0127] In some embodiments, when the electronic device 100 adjusts the size of the small window, the preview image of the small window is enlarged and reduced relative to the preview image of the large window. For example, during the enlargement of the small window, the content displayed in the preview image of the small window increases. The electronic device 100 can add content to the preview image of the small window based on the distance adjusted by two fingers. For instance, the electronic device 100 can crop more content from the preview interface of the large window and preview it in the small window, resulting in the character in the preview image of the small window being the same size as the character in the original preview image, but with more background content.

[0128] In this embodiment, the size of the small window can be customized. Users can adjust the size of the small window to an appropriate size, allowing them to view the preview of the tracked object more clearly and improving the user's recording experience.

[0129] It is understandable that, due to the small size of the small window, the contact area of ​​the fingers is too large when operating with two fingers, which may lead to accidental triggering of buttons in the small window. Optionally, this application embodiment provides a method to prevent accidental button triggering during the adjustment of the small window, wherein, when the electronic device 100 detects a click or long press operation in the non-functional area of ​​the small window, the small window close button and the second landscape / portrait switching button in the small window are hidden. When the electronic device 100 detects that there is no touch operation in the small window, the hidden buttons will be displayed again.

[0130] In the preview mode of the main character mode, to enhance the user's recording experience, the electronic device 100 also provides a function to adjust the display position of the small window. For example, Figure 6B This demonstrates a method for adjusting the display position of a small window, such as... Figure 6B As shown:

[0131] Figure 6B In the interface shown in Figure a, the small window is positioned at the lower left of the large window. When the electronic device 100 detects that the user clicks and drags the small window, it enters the following state: Figure 6B The interface shown in b. Figure 6B In interface b, the display position of the small window can follow the user's finger movement. For example, when the user long-presses the small window and drags it upwards, the small window moves upwards accordingly. After the user drags the small window to the preset position and releases their finger, the electronic device 100 detects the user's release operation, stops moving the small window, and the small window remains at the position where the finger was released. The electronic device 100 then enters the following state: Figure 6B The interface shown in c. Figure 6B In the C interface, the small window has been moved to the upper left.

[0132] It should be noted that in one possible implementation, after the display position of the small window is adjusted, as the person in the lens of the electronic device 100 moves or the lens of the electronic device 100 moves, the small window may obscure the image or face in the preview image of the large window, affecting the user's preview experience. This application embodiment can automatically or by default adjust the display position of the small window. For example, the electronic device 100 can detect the display position of the face in the large window and the display position of the small window. When the electronic device 100 detects that the area where the small window and the face are located overlaps, the electronic device 100 can automatically adjust the display position of the small window until the two areas no longer overlap. Alternatively, the electronic device 100 can detect the position of the face and the position of the rectangular border of the small window. When the electronic device 100 detects that the distance between the border and the face area is less than a certain threshold, the electronic device 100 will stop moving the small window towards the face by default, so that the small window cannot move to an area where a face is displayed.

[0133] In another possible implementation, after the display position of the small window is adjusted, some of the small windows may extend beyond the edge of the large window, preventing the user from previewing the complete small window and affecting the user's preview experience. This application embodiment can automatically or by default adjust the display position of the small window. For example, the electronic device 100 can detect whether the small window is completely displayed on the screen. If some of the small window cannot be completely displayed, the electronic device 100 will automatically move the small window in the opposite direction of the last adjustment of its display position until the small window is completely displayed. The electronic device 100 can also automatically adjust the display position of the small window based on other principles, which is not limited in this application embodiment. Alternatively, the electronic device 100 can detect the boundary line positions of the rectangular borders of the large window and the small window. When either boundary line of the two rectangular borders coincides, the electronic device 100 will stop moving the small window by default, so that the small window cannot be moved beyond the edge of the large window.

[0134] Optionally, in the preview mode of the protagonist mode, the electronic device 100 can also hide the buttons in the small window to prevent accidental button triggering while adjusting the position of the small window, which will not be elaborated here.

[0135] In this embodiment, the position of the small window can be customized and adjusted. Users can adjust the position of the small window in real time to reduce the obstruction of the large window preview and improve the user's recording experience.

[0136] This application provides a method for adjusting a small window using gestures, which allows for flexible and quick adjustment of the small window's style. This application is not limited to the above method; the electronic device 100 can also adjust the small window in other ways. For example, the electronic device 100 can adjust the small window's dimensions using buttons with zoom in / out functions. The electronic device 100 can set a small window adjustment menu, allowing manual input of parameters such as length / width to adjust the small window's dimensions. The electronic device 100 can also use movement buttons to move the small window up, down, left, or right to adjust its position. This application does not limit the method for adjusting the small window.

[0137] In one possible implementation, the small window in the preview mode of the protagonist mode in this application embodiment can also have a memory mechanism, which will be discussed below. Figure 6C The memory mechanism of the protagonist mode will be explained.

[0138] For example, the main character mode provides the function of adjusting the display position of the small window, allowing users to move the position of the small window according to their own shooting habits, such as... Figure 6C As shown in Figure a, the small window is adjusted to the upper left position of the preview interface. When the electronic device 100 detects that the user clicks the small window close button 306, the electronic device 100 enters the following state: Figure 6C The interface shown in b. Figure 6C In interface b, the small window closes. The large window displays no selected tracking object, and the tracking box 311 reverts to its unselected state. When the electronic device 100 receives a user's click on the tracking box 311, it enters the following... Figure 6C The interface shown in c. Figure 6C In the C interface, the small window is reopened. At this time, the style of the small window is the same as... Figure 6C The small windows in 'a' have the same style, for example... Figure 6C The interface of a and Figure 6C In the C interface, the size, position, and screen orientation of the small windows are all the same.

[0139] This application embodiment can also set a time limit for the memory mechanism. If the electronic device 100 does not display the small window for a period of time within the time limit, or if the electronic device 100 does not use the recording function for a period of time within the time limit, and if the electronic device 100 triggers the display of the small window again, the style of the small window will retain the style of the small window before it was closed last time. If the electronic device 100 does not display the small window for a period of time exceeding the time limit, or if the electronic device 100 does not use the recording function for a period of time exceeding the time limit, and if the electronic device 100 triggers the small window again, the small window will restore the default style.

[0140] In the preview mode of the main character mode, the default style of the small window can be either the style displayed when the main character mode is first entered and the small window is opened that day, or the system default style. The time limit can be set to 15 minutes (min). For example, Figure 6C The small window in interface b closes. 15 minutes later, the user clicks the tracking box 311 for the male figure, and the electronic device 100 enters... Figure 6C The interface shown in Figure d. Figure 6C The style of the small window in the d interface can be the default style, such as a large window for landscape preview and a small window for portrait preview, with the small window appearing in the lower left corner of the preview interface.

[0141] Understandably, a default style for a small window can be like... Figure 6C As shown in the d interface, the preview orientation of the large window is inconsistent with that of the small window. For example, when the large window is in landscape mode, the small window is in portrait mode; conversely, when the large window is in portrait mode, the small window is in landscape mode. In another default setting for the small window, the preview orientation of the large window is consistent with that of the small window. For example, when the large window is in landscape mode, the small window is also in landscape mode; and vice versa. This application embodiment does not limit the default setting style of the small window.

[0142] This application provides a small window memory mechanism in the preview mode of the main character mode. The small window can be reopened within a certain time limit, and the style of the small window can be retained from the last time the user adjusted the small window style. At the same time, the electronic device 100 can automatically adjust to the style before the small window was accidentally closed when the small window is accidentally closed, reducing the time the user spends adjusting the small window style and improving the user's recording experience.

[0143] In this embodiment, the electronic device 100 can independently set a memory mechanism for the small window, so that when the small window is opened for the Kth time within a certain period of time, the style of the small window is consistent with the style of the small window opened for the (K-1)th time. Furthermore, in this embodiment, the electronic device 100 can also establish a binding relationship between the style of the small window and the focus tracking object.

[0144] For example, in some embodiments, the electronic device 100 may set up a memory mechanism based on facial recognition technology. When the electronic device 100 selects a focus target and adjusts the style of the small window, the electronic device 100 can establish a binding relationship between the facial features of the focus target and the adjustment data of the small window, and save the data in the electronic device 100. When the electronic device 100 recognizes the focus target again, the electronic device 100 automatically recalls the adjustment data bound to the focus target, and the style of the current small window can be consistent with the style of the small window when the focus target was last selected.

[0145] In some embodiments, when the electronic device 100 detects the focusing object for the Kth time, it adjusts the style of the small window. The electronic device 100 can save the adjustment data of the Kth time and establish a binding relationship between the new adjustment data and the focusing object. Optionally, the adjustment data of the Kth time can replace the adjustment data of the (K-1)th time, and the electronic device 100 only retains the adjustment data corresponding to the most recent focusing object. Optionally, the electronic device 100 can save the adjustment data of the Kth time and establish a new binding relationship between the new adjustment data and the focusing object, and the electronic device 100 can retain the binding relationship established based on the adjustment data of the (K-1)th time. When the electronic device 100 selects the focusing object for the (K+1)th time, the electronic device 100 can provide the adjustment data of the small window from the Kth and (K-1)th times for the user to choose from.

[0146] Optionally, when the electronic device 100 selects a focus target but does not adjust the style of the small window, the electronic device 100 can store the facial features of the focus target and the data of the default style of the small window and establish a binding relationship. When the electronic device 100 recognizes the focus target again, the small window can display the default style.

[0147] This application provides a memory mechanism for the focus tracking object and the small window in the preview mode of the main character mode. When the electronic device 100 recognizes the selected focus tracking object, the electronic device 100 can automatically recall the adjustment data of the small window when the focus tracking object was selected last time according to the binding relationship, so that the style of the small window is consistent with the style of the small window when the focus tracking object was selected last time, thereby reducing the adjustment time of the small window and improving the user's recording experience.

[0148] The above embodiments represent application scenarios where users normally preview using the main character mode. In some embodiments, the electronic device 100 may not be able to identify the focus-tracking object in preview mode. The following will combine... Figure 7A , Figure 7B and Figure 7C This section explains several scenarios where the focus object is lost in the large window during preview mode in protagonist mode.

[0149] For example, Figure 7A This illustration shows a schematic diagram of the interface that can be displayed in the small window when the electronic device 100 cannot detect the object being tracked in the preview mode of the main character mode according to an embodiment of this application.

[0150] In one possible implementation, the position of the lens of the electronic device 100 remains unchanged, but the object being tracked moves out of the lens's field of view, resulting in the object being lost in the large window. In this case, it can be done as follows: Figure 7A As shown in a and b, in Figure 7A In the interface of device 'a', the large preview window can include male and female figures, as well as background elements such as road signs and trees. At this point, the male figure moves from its original position, and the electronic device 100 can no longer detect the object being tracked. The electronic device 100 then enters a similar state... Figure 7A The interface shown in b. Figure 7A In the b interface, the view in the small window can be a cropped view of the current view in the large window. For example, if the lens of the electronic device 100 is stationary, the male figure (i.e., the object being tracked) in the preview view of the large window disappears, while the background, such as the female figure, trees, and road signs, remains unchanged. When the lens of the electronic device 100 is stationary, the preview view in the small window can display a preview of the background where the object being tracked was before it disappeared.

[0151] In another possible implementation, the person does not move, and the lens of the electronic device 100 deviates from the object being tracked, resulting in the object being lost in the large window. In this case, it could be as follows: Figure 7A As shown in a and c Figure 7A The C interface is relative to Figure 7AAs shown in the interface (a), the lens of electronic device 100 shifts to the left. In the preview window, the trees move from the left to the right, the person moves out of the shooting range, and the subject is lost from focus. It's understandable that before the subject is lost, the small window displays the image of the subject at the corresponding position in the large window (the focus position is on the right side of the large window preview, and the small window displays the image at that position); after the subject is lost, the small window still displays the image at that corresponding position in the large window (the focus position remains on the right side of the large window preview, and the small window displays the image at that position). For example, before the subject is lost, such as... Figure 7A As shown in Figure a, the small window displays the image of the male figure on the right side of the large window. After the focus target is lost, as... Figure 7A As shown in c, the scene where the male character is located is currently a tree, and a preview of the tree is displayed in a small window of the electronic device 100.

[0152] Understandably, in the scenarios described above where the movement of a person or the camera of electronic device 100 causes the object being tracked in the large window to be lost, when the large window of electronic device 100 selects the object to be tracked, the small window can display the image of the object being tracked. Before the object is lost, the image displayed in the small window corresponds to the image displayed in the large window; after the object is lost, the image displayed in the small window still corresponds to the image displayed in the large window, and the corresponding position remains consistent with before the loss. At this time, the large window is in a dynamic preview state, and the small window also dynamically displays the image at the corresponding position in the large window.

[0153] In another possible implementation, the electronic device 100 switches the landscape / portrait mode of the small window in preview mode. Taking the switch from landscape to portrait preview as an example, when the electronic device 100 switches from landscape to portrait, the data used for cropping the small window in the original landscape preview mode may be lost, or the data in the landscape preview may not match the data in the portrait preview. This causes the small window to be unable to obtain the focus position in the large window after the switch, resulting in the small window losing the focus target. This scenario will be discussed in detail later. Figure 8A and Figure 8B A detailed explanation will not be provided here.

[0154] In another possible implementation, in the preview mode of the main character mode, after the focus object is lost, a small window on the electronic device 100 can display a static image of the last frame before the focus object was lost. For example, such as... Figure 7A As shown in d, the object being tracked is lost in the large window's view. The electronic device 100 can retain the last frame that can identify the male figure, and the small window can not dynamically track the preview of the large window, but only display the static image of the last frame.

[0155] In another possible implementation, if the focus object is lost in the preview mode of the main character mode, a small window can be displayed. Figure 7A The interfaces shown in b, c, and d are displayed in small windows. Figure 7A Taking the C interface as an example, when the object being tracked is lost, the electronic device 100 enters a state such as Figure 7A The interface shown in e. Figure 7A In the e-interface, the electronic device 100 adds a mask to the small window. For example, the mask can be a grayscale layer floating above the preview screen of the small window. The brightness of the small window under the mask is lower than that of the large window, which can alert the user that the preview of the current small window is abnormal.

[0156] Understandably, when the object being tracked is lost, the image displayed in the small window by the electronic device 100 can be... Figure 7A Any of the interfaces shown in b, c, and d can also be Figure 7A The combination of any interface and mask state shown in b, c and d is not limited in this embodiment.

[0157] This application provides implementation methods for displaying a small window interface after several tracking objects are lost in the preview mode of the main character mode. It is understood that the small window may also have other display interface implementation methods, which will not be listed here. The following uses one of the above implementation methods as an example to illustrate the processing process of the electronic device 100 after the tracking object in the large window is lost in the preview mode. In the following scenario, a male character can be selected as the tracking object.

[0158] Scenario 1, an example, Figure 7B In the interface shown in Figure a, when the male figure in the large window is lost as the focus target, the male figure is not displayed in either the large or small window, and the small window is in a masked state. For a certain period of time, the small window of the electronic device 100 can maintain the display of the preview image; for example, the time can be 5 seconds. If the electronic device 100 re-identifies the male figure within 5 seconds, the electronic device 100 enters the following state: Figure 7B As shown in the interface (b), the small window can automatically track and focus on the male figure and restore the preview image of the male figure. The electronic device 100 cancels the overlay state of the small window. If the electronic device 100 fails to recognize the male figure within 5 seconds, the electronic device 100 will then... Figure 7B Enter the A interface Figure 7B The C interface, Figure 7B In the C interface, the small window is closed, and the electronic device 100 retains the preview screen of the large window.

[0159] Scenario 2: For example, after electronic device 100 loses the male figure (i.e., the focus target) in the large window and fails to recognize the male figure within 5 seconds, the terminal enters... Figure 7C The A interface, Figure 7CThe small window in interface a is closed. If the electronic device 100 re-identifies the male figure as the focus target when the small window is closed, it will proceed to the following... Figure 7C The interface shown in c. Figure 7C In the c interface, the small window redisplays the preview of the object being tracked.

[0160] A kind of Figure 7C Enter the A interface Figure 7C In the implementation of the C interface, when the small window is not open, the electronic device 100 identifies multiple focusable objects (the focusable objects may or may not include the focusable objects that were lost before being lost). The preview screen of the large window includes multiple focusable objects and their corresponding tracking boxes, such as... Figure 7C As shown in b. When the electronic device 100 detects that the user clicks on the tracking box 311, it enters the following... Figure 7C The interface shown in c. Figure 7C In the C interface, a small window allows the user to track the selected object. During this process, the user can click the tracking box again to select the object to track or change the object to track.

[0161] In another implementation, when the electronic device 100 recognizes the object that was previously in focus, the small window can automatically restore and display the object. For example, such as... Figure 7C a to Figure 7C As shown in Figure c, the electronic device 100 identifies multiple focusable objects within a certain period after the small window is closed. Based on facial recognition technology, the electronic device 100 can determine whether the current multiple focusable objects include the object that was lost from focus. If the object was included, the small window can automatically preview the focus on that object. If not, the electronic device 100 can perform the method described above of clicking the tracking box to reselect the focusable object.

[0162] In the preview mode of the main character mode, if the focus target in the large window is lost, and the electronic device 100 detects the user clicking "start recording" within 5 seconds of the focus target being lost, the large window will begin recording video without the focus target. Since the small window also lacks a focus target, it's impossible to trim the video from the large window based on the focus target, nor can a focus-tracked video be generated. At this time, the small window can perform actions such as... Figure 7D The method shown.

[0163] Scenario 3, an example, Figure 7D In interface A, the male figure in the large window disappears within 5 seconds. At this time, the small window is overlaid, and the male figure in the small window also disappears. The user clicks the "Start Recording" button 303 in the large window, and the electronic device 100 responds to this operation, entering the following... Figure 7D The interface shown in b. Figure 7DIn interface b, the large window of electronic device 100 begins recording video. The recording time is displayed in the large window; for example, the recording time may include, but is not limited to, a combination of dots and time. A small window displays a preview of the overlay state. Understandably, the small window in overlay state can still dynamically display the corresponding position of the image in the large window, but the small window is only in preview mode and cannot record or save the video.

[0164] Optionally, in some embodiments, in Figure 7D The interface of A is switched to Figure 7D When the screen is in the "b" interface, the first landscape / portrait switching button 304 and the second landscape / portrait switching button 305 can display icons. In some embodiments, after recording starts in a large window, the landscape / portrait switching buttons can be hidden, such as... Figure 7D As shown in b.

[0165] When the small window is in a masked state, if the electronic device 100 detects the object being tracked within 5 seconds of the large window starting recording, the small window can remove the mask and begin recording; otherwise, the small window closes. This can be understood as the small window maintaining its masked state for 5 seconds. If the electronic device 100 finds the object being tracked within 5 seconds, the small window begins recording video focusing on that object. If the electronic device 100 fails to find the object being tracked within 5 seconds, the small window disappears.

[0166] For example, Figure 7D In interface b, the large window has no focus-tracking object and is recording video, while the small window has no focus-tracking object and is in a masked state. The electronic device 100 detects a male figure 3 seconds after starting recording in the large window, and then enters... Figure 7D The C interface. Figure 7D In the c interface, the mask state of the small window disappears, and the small window automatically starts recording for the tracking object. At this time, the recording time of the large window (3s) is inconsistent with the recording time of the small window (1s). The recording time difference can be the time difference between when the electronic device detects the user clicking the record button and when it re-identifies the tracking object.

[0167] For example, Figure 7D In interface b, the large window has no focus-tracking object and is recording video, while the small window has no focus-tracking object and is in a masked state. If the electronic device 100 fails to re-identify the focus-tracking object within 5 seconds of losing focus, the electronic device 100 enters... Figure 7D The d interface. Figure 7D In the d interface, the small window is closed, and the electronic device retains only the recording screen of the large window.

[0168] The above scenario can be understood as the masked state displayed in the small window after the subject is lost, and this masked state can be maintained for 5 seconds. If the time from when the subject is lost to when it is found is within 5 seconds, the small window will automatically start recording the subject. If the time from when the subject is lost to when it is found exceeds 5 seconds or the subject is not found, the small window will close.

[0169] Understandably, if recording begins in the large window and the small window closes due to loss of focus for more than 5 seconds, and the electronic device 100 re-identifies the focus object, the user can reselect the focus object based on its tracking frame. After the electronic device 100 responds to this operation, the small window automatically reappears and resumes recording. This scenario is similar to... Figure 9B The scenarios shown are similar, so they will not be described in detail here.

[0170] In the preview mode of the main character mode, the electronic device 100 may lose focus on the large window due to human factors such as camera movement or character movement. The electronic device 100 may also lose focus due to data loss or mismatch in the small window when switching between portrait and landscape modes. This application embodiment provides a method for handling the loss of focus on the small window when switching between portrait and landscape modes.

[0171] For example, in a scenario where switching between portrait and landscape modes in a small window causes the object to be lost in focus, this application provides the following embodiment: Figure 8A and Figure 8B The processing method shown.

[0172] Figure 8A In interface A, a small window is displayed in landscape mode, and the small window includes the object to be tracked. When electronic device 100 receives a click operation on the landscape / portrait switching button (for example, ...), Figure 8A On interface A, the user clicks the second screen orientation switch button 305, and the electronic device 100 can enter... Figure 8A The interface shown in b. Due to the switching between portrait and landscape modes in the small window, the focus target in the small window of electronic device 100 is lost. Figure 8A In the B interface, the preview screen in the small window has no focus tracking object and is in a masked state; the tracking box 311 reverts to its unselected style. It's understandable that due to data loss or inapplicability in the small window, it cannot accurately track the focus position, but it remains in preview mode, displaying a cropped view of any location within the larger window in real time. For example, Figure 8A In interface b, the focus position of the small window changes from the position of the male character to the position of the tree. When the electronic device 100 receives the user's click on the tracking box 311, it enters... Figure 8A The C interface. The electronic device 100 reselects the focus target. Figure 8AIn the C interface, the focus target appears in the preview window. The focus target selected by the electronic device 100 can be the focus target before it was lost, or it can be other objects; there are no restrictions here.

[0173] Optionally, the electronic device 100 can also resume tracking the object in the small window when it detects the object in focus again. For example, when the small window of the electronic device 100 switches from landscape to portrait mode, the object in focus may be lost. When the electronic device 100 recognizes the object in focus again, the object in focus is automatically displayed in the small window and the overlay disappears. For example, before the small window of the electronic device 100 switches between landscape and portrait modes, the object in focus is a male figure, such as... Figure 8A As shown in Figure a; during the switching between landscape and portrait modes in the small window, the male character disappears, and the small window appears as a mask, as shown in Figure a. Figure 8A As shown in b; after the small window is switched, the electronic device 100 recognizes the male figure again and automatically resumes tracking the male figure in the small window, as shown in b. Figure 8A As shown in c.

[0174] Understandably, when switching between portrait and landscape modes in the small window, the electronic device 100 needs to recalculate the cropping position of the small window relative to the large window. The electronic device 100 can reacquire the focus position data and crop the image at that focus position when it receives a user's click action on the object being tracked. This reduces the likelihood of the object being lost during screen rotation in the small window.

[0175] Optionally, if the focus target is lost due to screen orientation switching in the small window, the preview in the small window will not include the focus target and will be in a masked state, such as... Figure 8B As shown in a. After a period of time (e.g., 5 seconds after the tracked object is lost), if the electronic device 100 does not receive a user's click operation on the tracking frame, the electronic device 100 may enter as follows: Figure 8B The interface shown in b. Figure 8B The B-screen interface includes a large preview window, while the small window is closed.

[0176] Optionally, if the focus target is lost due to screen orientation switching in the small window, the focus target will not be included in the preview of the small window. Figure 8B As shown in c. When the electronic device 100 detects a click operation on the start recording button 303 in the large window during a period when the tracking object is lost in the small window (e.g., within 5 seconds after the tracking object is lost), the electronic device 100 can enter... Figure 8B The d interface. Figure 8B The d interface displays the recording screen and recording time in a large window, while the small window disappears. Subsequent electronic devices 100 can also receive clicks on the tracking frame to start recording in the small window; details will not be elaborated here.

[0177] This application provides several scenarios where the focus target is lost in the preview mode of the main character mode. Based on the methods of the above embodiments, the electronic device 100 can select the corresponding processing method when the focus target is lost, which improves the flexibility and accuracy of the electronic device 100 in recording using the main character mode, thereby improving the user's recording experience.

[0178] The above embodiments have described the preview mode of the main character mode. The recording mode of the main character mode will now be described with reference to the accompanying drawings. In the recording mode of the main character mode, the electronic device 100 can start a small window to record video of the focused object and save the video.

[0179] For example, firstly, combined with Figure 9A , Figure 9B and Figure 9C This section describes a scenario where an electronic device 100 initiates a small window recording.

[0180] In a scenario where recording begins in a small window, videos in both the large and small windows can start recording simultaneously. Figure 9A In interface A, the large window previews the subject being tracked (e.g., a male person), while the small window displays a preview of the subject being tracked. When electronic device 100 detects a click on the start recording button 303 in the large window, electronic device 100 enters... Figure 9A The interface shown in b is as follows. The electronic device 100 simultaneously initiates recording in both the large window and the small window. The small window can display the subject being tracked in real-time within the large window. Simultaneously, the electronic device 100 can acquire raw audio signals in real-time using multiple microphones (e.g., two microphones, three microphones, etc.). These raw audio signals include: sounds emitted by multiple subjects in the large window recording (e.g., the male and female figures shown in the illustration), ambient noise, etc. Figure 9A In the b interface, a small window displays the recording screen and recording time. For example, the small window also displays a "Stop Recording" button 315 for recording mode, while the "Start Recording" button 303 in the large window is converted into a "Pause Recording" button 313 and a "Stop Recording" button 314 for recording mode. The large and small windows can each display their respective recording times, which can be consistent across the large and small windows. To improve the recording interface and reduce obstruction of the tracking object, the display position of the recording time in this embodiment can be as follows: Figure 9A As shown in b, the recording time can also be set at other locations that do not affect the recording.

[0181] Optionally, in some embodiments, when the electronic device 100 enters recording mode from preview mode, the first landscape / portrait switching button, the second landscape / portrait switching button, the zoom button, and the small window close button may disappear, such as... Figure 9AFigure b. Some embodiments may also retain these buttons, and this application embodiment does not limit this.

[0182] In another scenario where recording is initiated in a small window, the videos in the large window and the small window can be recorded sequentially. Figure 9B In interface A, the preview screen in the large window does not include the object to be tracked, and the electronic device 100 does not open the small window. When the electronic device 100 detects a click operation on the start recording button 303 in the large window, the electronic device 100 enters the following state: Figure 9B The interface shown in b. Figure 9B In interface b, electronic device 100 starts recording in a large window, which displays the recording time. After a period of time, electronic device 100 detects the object being tracked, and the recording footage in the large window is displayed as shown. Figure 9B As shown in c. Figure 9B The interface includes a male character and a tracking box 311 corresponding to the male character. When the electronic device 100 receives a click operation from the user on the tracking box 311 of the male character at the 7th second of the recording time, the electronic device 100 starts recording in a small window and enters the following... Figure 9B The interface shown in d. Figure 9B In the d interface, the small window can display the recording screen of the object being tracked, the recording time, and the small window recording button, etc. The small window can record the object being tracked in the large window in real time.

[0183] In another scenario where recording is initiated in a small window, the videos in the large window and the small window can be recorded sequentially. Figure 9C In interface A, the preview screen in the large window includes the object to be tracked. Because the electronic device 100 did not select the object to be tracked, the small window did not open. In response to the user's click on the start recording button 303, the electronic device 100 starts recording in the large window and enters the following... Figure 9C The interface shown in b. Figure 9C In interface b, the large window displays the recording screen and recording time, while the electronic device 100 does not open a small window. During the recording process in the large window, when the electronic device 100 detects a click operation by the user selecting the tracking box 311, the electronic device 100 displays... Figure 9C The interface shown in c. Figure 9C In the c interface, the electronic device 100 can maintain recording in a large window and start recording in a small window.

[0184] The electronic device 100 can start video recording in a small window based on the above scenarios and obtain multiple video streams. It should be noted that the small window can display the image of the object being tracked in the large window, but the video recorded in the small window and the video recorded in the large window are multiple independent videos, and are not a picture-in-picture composite video in which a small window is nested within a large window recording.

[0185] It should be noted that if electronic device 100 does not enable small window recording, it will receive one video stream recorded in the large window. If electronic device 100 enables small window recording, it will receive one video stream recorded in the large window and multiple video streams recorded in the small windows. For example, during large window recording, electronic device 100 can enable small window recording multiple times. When it detects a click on the "End Small Window Recording" button, it can end the small window recording and receive one video stream. When the small window recording resumes, electronic device 100 will receive a new video stream. The number of videos received in the small window is related to the number of times the small window recording is enabled.

[0186] In the recording mode of the main character mode, after a small window appears in the recording interface of the electronic device 100, the user can adjust the style of the small window according to their needs. The following is a combination of... Figures 10A-10C This section explains how to adjust the small window.

[0187] For example, Figure 10A This diagram illustrates a scenario where the recording is stopped using the "End Recording in Small Window" button in the main character mode.

[0188] The recording mode window includes a "Stop Recording in Small Window" button 315. The electronic device 100 can stop recording in the small window at any time during the recording process based on the user's operation of the "Stop Recording in Small Window" button 315. For example, Figure 10A In interface A, when electronic device 100 receives a click operation from the user on the button 315 to end recording in the small window after recording in the large window for 4 seconds, electronic device 100 can enter the following... Figure 10A The interface shown in b. Figure 10A In interface b, the small window closes, the large window displays no selected focus object, and the tracking box 311 reverts to its unselected state. At this point, recording in the small window has ended, and the electronic device 100 can save the video recorded in the small window. After a period of time, such as... Figure 10A As shown in the interface c, when the electronic device 100 receives a click operation on the tracking box 311 during the 7th second of recording in the large window, the electronic device 100 can enter as follows: Figure 10A The interface shown in d. Figure 10A The interface can include a small window, which can be used to start recording video. Figure 10A Videos recorded in d can be compared with Figure 10A The videos recorded at the end of segment 'a' are not the same video. For example, when the small window recording ends at 4 seconds, the electronic device 100 saves the first 4-second video. At 7 seconds, the electronic device 100 selects the focus target, and the small window begins recording the second video.

[0189] It is understandable that the "End Small Window Recording" button 315 in the small window can control the recording of the small window, but whether or not the small window is recorded does not affect the recording of the large window. That is, when the electronic device 100 performs actions such as... Figure 10A During the recording process, the large window remains open for recording.

[0190] Furthermore, in the recording mode of the main character mode, the electronic device 100 can also be set with a memory mechanism. If the recording of the small window is restarted within 15 minutes, the current style of the small window will be consistent with the style of the previous small window. For example... Figure 10A a and Figure 10A The position, size, and screen orientation of the small window in d are the same. For the memory mechanism, please refer to [reference needed]. Figure 6C The description of that will not be repeated here.

[0191] In the recording mode of the main character mode, when the electronic device 100 starts recording in a small window, the user can also adjust the size of the small window. For example, the user can adjust the size of the small window using two fingers. The electronic device 100 can detect the position of the user's two fingers tapping the display screen and control the enlargement and reduction of the small window based on the distance between the two tap positions. The process of adjusting the size of the small window can be as follows: Figure 10B From a to c. For the process of adjusting the small window here, please refer to... Figure 6A The description is omitted here. It is understandable that adjusting the small window will not affect recording in either the large or small window.

[0192] It should be noted that in one possible implementation, in the recording mode of the main character mode, when the electronic device 100 adjusts the size of the small window, the recording screen of the small window can be correspondingly enlarged and reduced. For example, during the enlargement of the small window, the recording screen itself remains unchanged, only enlarged, allowing the user to clearly view the recording process. In this case, adjusting the small window during recording will not affect the recorded video; that is, during playback of the video, there will be no discontinuous images with sudden enlargement or reduction. This method improves the viewing experience of the recording process without affecting the smoothness of video playback.

[0193] In another possible implementation, when the electronic device 100 adjusts the size of the small window, the preview image of the small window is enlarged and shrunk relative to the large window. For example, during the enlargement of the small window, the preview image changes, and the amount of recording content in the small window increases. That is, during the playback of the video in the small window, the amount of content in the frame increases, and the object being tracked shrinks accordingly. In this method, the user can adjust the display ratio of the object being tracked in the video within the small window based on its enlargement and shrinkage, resulting in a better display of the object and thus improving the user's recording experience.

[0194] In the recording mode of the main character mode, the electronic device 100 can also adjust the display position of the small window.

[0195] For example, a user can adjust the display position of a small window by long-pressing and dragging. The electronic device 100 can detect the user's click and drag operation, and the small window can adjust its position in the large window according to the movement of the user's finger. The adjustment process of the small window's position can be as follows: Figure 10C From a to c. For the process of adjusting the small window here, please refer to... Figure 6B The description is omitted here. It is understandable that adjusting the small window will not affect recording in either the large or small window.

[0196] Optionally, in the recording mode of the main character mode, the electronic device 100 can also hide the buttons in the small window when the user adjusts its size and position. For example, when the electronic device 100 detects a finger touching the display screen, it will hide the buttons. Figure 10C The recording ends in the small window (315). This prevents users from accidentally triggering the button and ending the recording in the small window.

[0197] This application also provides a method for switching the focus tracking object in the recording mode of protagonist mode, such as... Figure 11 As shown. For example, Figure 11 In the A interface, during the recording scene in protagonist mode, the focus target in the large window is the male character, and the tracking frame 311 corresponding to the male character is highlighted. The recording screen of the male character is displayed in the small window. When the electronic device 100 detects that the user clicks on the tracking frame 312 corresponding to the female character, it enters... Figure 11 The b interface. The electronic device 100 changes the focus target from a male character to a female character. Figure 11 In the B interface, the focus target in the large window is the female figure, and the tracking frame 312 corresponding to the female figure is highlighted. The recording footage of the female figure is displayed in the small window.

[0198] Optionally, when switching focus targets in recording mode, the electronic device 100 can also set dynamic effects to improve the smoothness of the recorded footage during focus target switching. See [link to dynamic effects section] for details. Figure 5 The description of the embodiments in this application will not be repeated here.

[0199] This application provides several scenarios for flexibly adjusting the small window recording screen to improve the user experience during the recording process.

[0200] The recording method provided in this application embodiment allows the electronic device 100 to provide a pause recording function in the main character mode recording mode.

[0201] For example, such as Figure 12As shown in the interface (a), the electronic device 100 is in the process of recording. The large window may include a pause recording button 313, an end recording button 314, and the recording time of the large window. The small window may include an end small window recording button 315 and the recording time of the small window. When the electronic device 100 receives a click operation on the pause recording button 313, the electronic device 100 enters the interface shown in Figure a. Figure 12 The interface shown in b. Figure 12 In interface b, both the large and small windows pause recording simultaneously. The pause recording button 313 is changed to a resume recording button 1301. The resume recording button 1301 instructs the electronic device 100 to resume recording the video from its current paused state. In both the large and small windows, the recording time is the pause time of the electronic device 100, which can be represented as a combination of "||" and time. When the electronic device 100 receives a click operation on the resume recording button 1301, both the large and small windows of the electronic device 100 simultaneously resume recording, as shown below. Figure 12 As shown in c.

[0202] It should be noted that during the pause recording period, the large window of the electronic device 100 can display the image captured by the camera in real time, and the small window can display the image of the object being tracked in real time, but the electronic device 100 will not save the images displayed in the large window and the small window.

[0203] It's understandable that the video paused in the large window is the same as the video before the pause, and the video paused in the small window is also the same as the video before the pause. For example, if a user clicks the pause recording button at 4 seconds, the electronic device 100 responds to the click and pauses video recording for 4 seconds. After a period of time, when the electronic device 100 receives a click on the resume recording button, it begins recording the 5th second of the video, based on the 4-second video.

[0204] In the recording mode of the main character mode, the electronic device 100 may lose focus during the recording process. The following is a summary... Figure 13A and Figure 13B This section explains how to handle the loss of focus on the object during recording.

[0205] For example, such as Figure 13A As shown in the interface (a), both the large and small windows of the electronic device 100 are recording. The small window contains the object being tracked. When the electronic device 100... Figure 13AWhen the tracking object in the large window cannot be identified as shown in b, the tracking object in the small window is also lost, and the small window continues recording and remains in a masked state. In some embodiments, after the tracking object is lost, the small window can continue recording the scene at the position where the tracking object was before it was lost. The recorded scene can be an empty shot excluding the tracking object, and the recording time can be 5 seconds. If the electronic device 100 re-identifies the tracking object within 5 seconds after it is lost, the electronic device 100 then enters the following state: Figure 13A As shown in Figure c, the small window maintains the state before the tracked object was lost. In this scenario, the loss of the tracked object does not affect the tracked video in the small window, and the small window continues recording. When the tracked object is found and recording resumes, the electronic device 100 can edit the video before the tracked object was lost, the video of the empty shot, and the video after the tracked object was found in the small window. In one possible implementation, the electronic device 100 can delete the empty shot and splice the video before the tracked object was lost with the video after the tracked object was found to create a single video. In another possible implementation, the electronic device 100 can blur, soften, or add a mask to the empty shot video to reduce its impact on the overall video's continuity and improve the user experience when viewing the recorded video later.

[0206] If the electronic device 100 fails to detect the target object within 5 seconds after it is lost, the electronic device 100 will then... Figure 13A Enter the interface shown in b as follows Figure 13A The interface shown in d. Figure 13A In the d interface, electronic device 100 continues recording in the large window while pausing recording in the small window, which may still be in a masked state. For example, if electronic device 100 fails to detect the tracking object at 6 seconds and does not recognize the tracking object within the following 5 seconds, then electronic device 100 will pause recording in the small window at 11 seconds, while recording in the large window will not be affected. In this scenario, if electronic device 100 fails to find the tracking object or does not end recording in the small window, the small window will remain paused and masked. Alternatively, electronic device 100 can also end recording in the small window, no longer displaying the small window and its contents in the large window recording interface.

[0207] It should be noted that when recording is paused in the small window of the electronic device 100, the small window can continue to track the image in the relative position of the image in the large window, but the electronic device 100 will not record the small window. Alternatively, when recording is paused in the small window, the electronic device 100 can display the static image of the last frame during recording in the small window, without tracking the image in the relative position of the image in the large window.

[0208] When electronic device 100 fails to recover the tracked object within 5 seconds after it is lost, and recording is paused in the small window, the recording interface of electronic device 100 will appear as follows: Figure 13AAs shown in d. During the paused recording in the small window, if the electronic device 100 detects the object being tracked, the small window can automatically resume recording, entering as shown in... Figure 13A The interface shown in e and f. For example, electronic device 100 pauses small window recording at 11s. At 15s, electronic device 100 recognizes the tracking object and resumes small window recording. At this time, the small window can continue recording the 12th second of video based on the 11s video.

[0209] When the electronic device 100 encounters a loss of focus in small window recording mode, it can perform the above process. In another possible scenario, the electronic device 100 may encounter a loss of focus when recording is paused in a large window, such as... Figure 13B As shown:

[0210] For example, Figure 13B Interface A can be the interface entered by the electronic device 100 when it detects a click on the pause recording button in the large window. Both the large and small windows are simultaneously in a paused recording state. When recording is paused, the electronic device 100 can be in preview mode, meaning that both the large and small windows can continuously track the preview image of the subject being focused on, but recording is not performed. During the paused recording and preview phase, the electronic device 100 may fail to detect the subject being focused on, such as... Figure 13B As shown in b. When the electronic device 100 does not recognize the object being tracked, Figure 13B In the interface b, a small window automatically displays a mask state. The mask state can prompt the user that the focus object is lost. The electronic device 100 needs to find the focus object or reselect the focus object to resume recording.

[0211] Alternatively, in one possible implementation, Figure 13B In interface b, the small window is in a paused recording state and is covered by a mask. If the electronic device 100 fails to find the focus target or does not end / resume recording, the mask state of the small window will remain.

[0212] This application provides several scenarios where the focus target is lost during recording in the main character mode. The electronic device 100 can perform corresponding processing methods for the scenario to reduce the impact of the focus target loss on the recording process and improve the user's recording experience.

[0213] Users can choose to exit protagonist mode and revert to the regular recording mode when they do not need to use protagonist mode.

[0214] The following is combined Figure 14 The scenarios for exiting the protagonist mode provided in the embodiments of this application will be described, such as... Figure 14 As shown:

[0215] For example, when electronic device 100 receives a request for... Figure 14When the "End Recording" button 314 is clicked on interface A, the electronic device 100 can simultaneously end the recording of the large window and the recording of the small window and enter... Figure 14 The interface is described in section b. When the electronic device 100 finishes recording, it can simultaneously save the video recorded in the large window and the video recorded in the small window. The electronic device 100 can save the two videos to the same path or to different paths. For example, the video in the large window and the video in the small window can be saved to a folder in the album, or the video in the large window can be saved to a regular path, and the video in the small window can be saved to the folder of the main character mode in the album. This application embodiment does not limit the saving path of the two videos. Among them, the video in the large window saved by the electronic device 100 can be called a panoramic video file, and the video in the small window saved by the electronic device 100 can be called a close-up video file. The panoramic video file includes a panoramic video frame and a target audio signal 1, and the close-up video file includes a close-up video frame and a target audio signal 2. In this example of the user interface, the panoramic video frame is the recording frame displayed in the large window from the start to the end of the recording period of the electronic device 100. The target audio signal 1 is: all the sound signals in the environment where the main character is located, obtained by processing the original audio signal from the start to the end of the recording period. The close-up video frame is the recording frame displayed in the small window from the start to the end of the recording period. The target audio signal 2 is a sound signal in which the protagonist's voice is clearer and more prominent than the voices of other sound sources. The electronic device 100 can encapsulate the panoramic video image and the target audio signal 1 into a panoramic video file, and encapsulate the close-up video image and the target audio signal 2 into a close-up video file, and then save the panoramic video file and the close-up video file to local storage.

[0216] Figure 14 In interface b, both the large and small windows have finished recording and returned to preview mode. When a click operation is received on the exit protagonist mode button 307, the electronic device 100 enters... Figure 14 The interface shown in Figure c. Electronic device 100 resumes normal recording mode. Of course, electronic device 100 can also... Figure 14 After detecting a click on the "End Recording" button 314 in interface A, the system immediately exits the main character mode and displays... Figure 14 The interface shown in c. Alternatively, the user can also trigger the electronic device 100 to exit the main character mode through gestures or other methods; this embodiment of the application does not impose any restrictions on this.

[0217] The following describes the module interaction flow in the audio processing method provided in the embodiments of this application.

[0218] Please refer to Figure 15 , Figure 15 An exemplary diagram illustrating the module interaction flow of an audio processing method is provided.

[0219] like Figure 15 As shown, the application layer of electronic device 100 may include: a main character mode module, a camera management module, a camera focus tracking module, and an audio focus tracking module. The application framework layer may include: an audio recording (AudioRecorder) module and an audio management (AudioManager) module. The hardware abstract layer (HAL) includes: a camera HAL layer and an audio HAL layer. The camera HAL layer includes a main character recognition module, and the audio HAL layer includes an audio tracking (AudioTrack) module. The hardware layer includes a microphone.

[0220] like Figure 15 As shown, the module interaction flow of this audio processing method may specifically include:

[0221] S501, The protagonist mode module receives input 1 applied to the protagonist mode control.

[0222] For example, taking the user interface shown above as an example, input 1 applied to the protagonist mode control is equivalent to a touch operation (e.g., a click) applied to the protagonist mode button 404, which is the protagonist mode control. In response to input 1, the electronic device 100 enters protagonist mode, and the protagonist mode module can perform the following step S502.

[0223] S502. In response to input 1, the protagonist mode module starts the corresponding function of protagonist mode in the camera management module.

[0224] Specifically, the camera management module includes a device management module, a session management module, and a surface management module. When the electronic device 100 receives input 1 applied to the main mode control through the main mode module, the electronic device 100 can respond to input 1 and enter main mode. For a description of main mode, please refer to the aforementioned user interface section; it will not be repeated here. The main mode module can activate the corresponding functions in the camera management module, such as: calling the camera corresponding to main mode for shooting through the device management module, managing the session functions corresponding to main mode through the session management module, and controlling the display screen to show the display interface corresponding to main mode through the surface management module.

[0225] S503, the main character mode module loads the camera tracking focus module.

[0226] Specifically, the main character mode module can load the program instructions of the camera tracking module into a location in the electronic device 100 that the processor can directly read and write (e.g., random access memory), so that the processor can execute the program instructions of the camera tracking module.

[0227] S504, the camera tracking focus module sends the audio transmission path configuration command to the audio tracking focus module and initializes the audio tracking focus module.

[0228] S505, the audio tracking module sends the configuration information of the audio transmission path to the audio recording module based on the configuration command of the audio transmission path.

[0229] Specifically, the configuration information for the audio transmission path may include one or more of the following parameters: number of channels (channelCount), sample size per audio sample point (bytePerSample), audio sampling rate, audio frame size, total buffer size, and number of frames in the buffer. Detailed examples of these parameters and the relationships between them will be explained in subsequent embodiments and will not be repeated here.

[0230] S506 The audio recording module verifies the number of channels in the configuration information based on the audio policy management information.

[0231] The electronic device 100 may store audio policy management information (i.e., audiopolicy) in a designated area. This audio policy management information may be information preset and saved to the electronic device 100 at the factory. This audio policy management information may include, but is not limited to, the following: information about the audio input ports (e.g., rear microphone, bottom microphone, etc.) and audio output ports (e.g., earpiece, speaker, Bluetooth headset) of the electronic device 100, which can be used by the electronic device 100 to manage the audio input / output hardware devices; maximum and minimum volume adjustment values, which can be used by the electronic device 100 to adjust the audio volume; in this embodiment, the audio policy management information also includes an identifier for the number of configurable channels of the electronic device 100, which can be used to indicate the number of channels that the electronic device 100 can configure. The specific form, usage, and description of this identifier will be described in detail in subsequent embodiments and will not be repeated here.

[0232] Specifically, when the configuration information of the audio transmission path includes a channel number value M1 (for example, M1 can be 2, 4, or 6), the audio recording module can determine whether the electronic device 100 can be configured with an audio transmission path of M1 channels based on whether the audio policy management information includes the identifier corresponding to M1 (also known as the first identifier). If the audio policy management information includes the identifier corresponding to M1, the electronic device 100 can be configured with M1 channels, i.e., the verification is passed and subsequent steps can be executed; if the audio policy management information does not include the identifier corresponding to M1, the electronic device 100 cannot be configured with M1 channels, i.e., the verification is not passed. The specified area mentioned above can be an original design manufacturer (ODM) partition or other storage locations within the electronic device 100; this application does not impose any restrictions on this.

[0233] For example, the number of audio channels and their corresponding identifiers can be shown in List 1 below:

[0234] Table 1

[0235]

[0236]

[0237] As shown in Table 1, the identifier for 2 channels can be 0x04|0x08. That is, if the Audio policy management information includes the identifier "0x04|0x08", it means that the hardware capabilities and software attributes of electronic device 100 can support 2-channel configuration; otherwise, electronic device 100 cannot be configured with 2 channels. The identifier for 4 channels can be 0x04|0x08|0x10|0x20. If the Audio policy management information includes the identifier "0x04|0x08|0x10|0x20", it means that the hardware capabilities and software attributes of electronic device 100 can support 4-channel configuration; otherwise, electronic device 100 cannot be configured with 4 channels. The symbol "|" represents the bitwise OR operator in the prior art.

[0238] It should be noted that Table 1 is only used as an example to explain this application. The identifiers corresponding to the number of channels may also be different from those in Table 1, and this application does not impose any restrictions on this.

[0239] In this embodiment, the steps are specifically explained using the configuration of 4 channels as an example. The audio recording module obtains the configuration information of the audio transmission path from the audio tracking module. The channel number value M1 in this configuration information can be 4, indicating that a 4-channel audio transmission path needs to be configured. Then, the audio recording module can query whether the Audio policy management information in a specified area (e.g., ODM partition) includes the identifier corresponding to 4 as shown in Table 1. If the Audio policy management information includes the identifier "0x04|0x08|0x10|0x20", then the hardware capabilities and software attributes on the electronic device 100 can support the configuration of 4 channels, that is, the verification is passed and the subsequent steps can be executed. If the Audio policy management information does not include the identifier "0x04|0x08|0x10|0x20", then the electronic device 100 cannot be configured with 4 channels, that is, the verification is not passed.

[0240] S507. After the audio recording module passes the configuration information verification, the audio recording module can call the audio tracking module to configure the audio transmission path.

[0241] S508, the audio tracking module can configure the audio transmission path based on the configuration information of the audio transmission path. The audio transmission path may include a first number of channels.

[0242] The first quantity is the same as the channel number M1 in the configuration information. The first quantity is related to the number of windows p and the number of channels q corresponding to each window, and its value is the product of p and q. For example, in this embodiment, the interface has two windows, namely the aforementioned large window and the aforementioned small window, each corresponding to two channels; then the first quantity is 2 x 2 = 4, and the audio transmission path includes 4 channels. In some examples, if the interface has three windows, each corresponding to two channels, then the first quantity is 3 x 2 = 6, and the audio transmission path includes 6 channels. This application does not limit the specific value of the first quantity. Preferably, the first quantity can be 4, that is, the audio tracking module can be configured to include an audio transmission path with 4 channels, and the channel number M1 in the above configuration information is 4.

[0243] Specifically, the audio transmission path can be used to carry a buffer containing audio data. That is, when configuring an audio transmission path including a first number of channels, the audio tracking module can divide each frame in the buffer into a first number of regions. In some examples, when configuring an audio transmission path with a first number of channels, the audio tracking module can configure a single buffer including the first number of regions. In other examples, when configuring an audio transmission path with a first number of channels, the audio tracking module can configure a dual buffer including the first number of regions. This application does not limit this. Subsequent embodiments will use a single buffer as an example to illustrate the implementation process of this solution.

[0244] For example, if a 4-channel audio transmission path is configured, the configuration information of the audio transmission path may include the following parameters:

[0245] channelCount (number of audio channels) = 4;

[0246] bytePerSample(audio sample size) = 2;

[0247] frameSize (size of each frame in the buffer) = channelCount * bytePerSample = 8;

[0248] totalBufferSize (total buffer size) = 15360;

[0249] frameCount(frame number)=totalBufferSize / frameSize=1920

[0250] In other words, based on the above configuration information, the audio tracking module can be configured with a 4-channel buffer, which includes 1920 frames. Each frame includes 4 regions, such as region 1, region 2, region 3, and region 4. Each region can carry the data information of one audio sample point on the corresponding channel's audio data, and its size is 2 bytes (that is, the sampling size of the audio sample point in the configuration information is 2 bytes). For example, region 1 in the first frame can carry the data information of the first audio sample point on the channel 1 audio data; region 2 in the first frame can carry the data information of the first audio sample point on the channel 2 audio data; region 3 in the first frame can carry the data information of the first audio sample point on the channel 3 audio data; region 4 in the first frame can carry the data information of the first audio sample point on the channel 4 audio data, and so on. Among them, regions 1 and 2 in each frame can be used to carry the audio data of target audio signal 1, and regions 3 and 4 in each frame can be used to carry the audio data of target audio signal 2. The size of each frame is the number of regions multiplied by the sampling size of the audio sample point, which is 4*2=8 bytes in this configuration. The total buffer size (totalBufferSize) can be determined based on the chip's capabilities, and is typically 15360. Audio sampling points are samples taken from the audio signal at any point in time during the duration of the audio signal, and include the audio data information at that point in time.

[0251] S509, the audio recording module sends the configuration results reported by the audio tracking module to the audio focus tracking module.

[0252] Specifically, if the audio recording module passes the configuration information verification and calls the audio tracking module to complete the audio transmission path configuration based on the aforementioned configuration information, the audio recording module can send a configuration success message reported by the audio tracking module to the audio tracking module. If the audio recording module fails the configuration information verification, it cannot call the audio tracking module to configure the audio transmission path, and the audio recording module can send a configuration failure message to the audio tracking module.

[0253] S510, The protagonist mode module receives input 2 from the user selecting the shooting object 1 as the protagonist.

[0254] Specifically, the protagonist mode module can control the display screen to show and identify one or more shooting objects (e.g., male and female figures in the preview screen shown in the aforementioned user interface example) in the large window preview, and display tracking icons associated with the shooting objects when the protagonist is not identified (e.g., tracking box 311 for the male figure and tracking box 312 for the female figure when the protagonist is not identified, as shown in the aforementioned user interface example). These one or more shooting objects may include shooting object 1. The protagonist mode module can receive input 2 from the user selecting shooting object 1 as the protagonist. For example, this input 2 of selecting shooting object 1 as the protagonist can be... Figure 5 In step a, the user clicks on the tracking box 311 corresponding to the male character. In response to this action, the male character is identified as the main subject. This male character is the aforementioned subject 1.

[0255] S511, in response to input 2, the main character mode module calls the camera tracking module.

[0256] S512, the camera focus tracking module calls the subject recognition module.

[0257] Specifically, the camera tracking module can call the subject recognition module to start a specified algorithm in the subject recognition module, so as to calculate the subject's coordinate information (e.g., the subject's face coordinate information, the subject's body coordinate information, etc.). The specified algorithm can be a re-identification (ReID) algorithm or other algorithms, and this application does not limit it.

[0258] S513, The protagonist recognition module obtains the tracking data information of the protagonist in the image based on the specified algorithm.

[0259] Specifically, the preview images displayed in the large and small windows on the electronic device 100 are images captured by the camera on the electronic device 100. The tracking data information of the protagonist in the image may include: the protagonist's face coordinates, the protagonist's body coordinates, and the protagonist's center coordinates. Among them, the protagonist's center coordinates can be calculated based on the protagonist's face coordinates and body coordinates.

[0260] S514, The main character recognition module sends the tracking data information of the main character in the image to the camera tracking module.

[0261] S515, the camera tracking module obtains the face coordinates of the main character from the tracking data information of the main character in the image.

[0262] S516 The camera tracking module sends the protagonist's face coordinates to the audio tracking module through the audio management (AudioManager) module every specified period (e.g., every 500ms).

[0263] Specifically, since the position of the main character in each frame may change during video recording, steps S513-S515 can be executed at specified intervals (e.g., every 500ms) to obtain the main character's face coordinate information. Then, the camera tracking module can send the main character's face coordinate information to the audio tracking module through the audio management module at specified intervals, so that the audio tracking module can promptly know that the main character's face coordinate information has changed, in order to execute subsequent steps.

[0264] In some examples, the main character may change based on user actions. In such cases, the process by which the audio tracking module obtains the face coordinates of the new main character can still refer to steps S513-S516. For instance, the electronic device 100 can display shooting object 1 and shooting object 2 in a large preview window. When the electronic device 100 responds to the user's selection of shooting object 1 as the main character, the audio tracking module executes steps S513-S516 through the aforementioned modules to obtain the face coordinates of shooting object 1. Then, the electronic device 100 receives the user's selection of shooting object 2 as the main character; at this point, the main character changes from shooting object 1 to shooting object 2. The audio tracking module can then execute steps S513-S516 through the aforementioned modules to obtain the face coordinates of shooting object 2.

[0265] S517, The protagonist mode module receives input 3 from the user applied to the recording control.

[0266] For example, taking the aforementioned user interface example, the user's input 3 to the recording control is... Figure 9A In step a, the user performs a touch operation (e.g., click) on the start recording button 303.

[0267] S518, in response to input 3, the main character mode module sends a microphone start command to the audio recording module.

[0268] Specifically, in response to input 3, the protagonist mode module can begin recording video through various modules and hardware. The camera can be used to capture images to form the visuals in the video file, and the microphone can capture sound signals to form the audio in the video file.

[0269] S519, The audio recording module sends a microphone start command to the audio tracking module.

[0270] The S520 audio tracking module sends a microphone activation command to the microphone.

[0271] S521, The microphone acquires audio signal 1 based on the microphone start command. This audio signal 1 is the aforementioned original audio signal.

[0272] Specifically, audio signal 1 may include mono sound signals collected in real time by multiple microphones (e.g., 2, 3, etc.). That is, audio signal 1 is an audio signal collected in real time by the microphones on electronic device 100. Preferably, audio signal 1 may be collected by 3 microphones, then audio signal 1 may include: sound signal 1 collected by microphone 1, sound signal 2 collected by microphone 2, and sound signal 3 collected by microphone 3. All of the above sound signals 1, 2, and 3 are mono sound signals.

[0273] For example, such as Figure 16 As shown, the preferred three microphones can be the top microphone (which can be referred to as the top microphone), the bottom microphone (which can be referred to as the bottom microphone), and the microphone on the same side as the rear camera of the electronic device 100 (which can be referred to as the back microphone). The back microphone can capture a mono audio signal 1, meaning that each frame of audio signal 1 contains only mono data; the top microphone can capture a mono audio signal 2, meaning that each frame of audio signal 2 contains only mono data; and the bottom microphone can capture a mono audio signal 3, meaning that each frame of audio signal 3 contains only mono data. Audio signal 1 can include the aforementioned audio signal 1, audio signal 2, and audio signal 3.

[0274] In some examples, audio signal 1 may also be acquired using two microphones. Audio signal 1 could then include sound signal 1 acquired by microphone 1 and sound signal 2 acquired by microphone 2. Both sound signal 1 and sound signal 2 are mono audio signals. This application does not limit the number of microphones used to acquire audio signal 1.

[0275] S522, the microphone sends the acquired audio signal 1 to the audio tracking module.

[0276] S523, The audio tracking module obtains target audio signal 1 and target audio signal 2 based on audio signal 1 and the face coordinate information of the protagonist.

[0277] Target audio signal 1 and target audio signal 2 are both stereo audio signals. Target audio signal 1 is the audio in the aforementioned panoramic video file, and target audio signal 2 is the audio in the aforementioned close-up video file. For a detailed explanation, please refer to the description above, which will not be repeated here.

[0278] S524, The audio tracking module sends target audio signal 1 and target audio signal 2 to the audio tracking module through the audio recording module based on the buffer in the audio transmission path.

[0279] Specifically, in the aforementioned steps, the audio tracking module has been configured with a buffer including a first number of regions. At this point, the audio tracking module can fill the buffer with target audio signal 1 and target audio signal 2. Target audio signal 1 and target audio signal 2 can occupy different regions in the buffer, as detailed in the preceding description.

[0280] S525, the audio tracking module extracts target audio signal 1 and target audio signal 2 based on the buffer in the audio transmission path.

[0281] Specifically, the target audio signal 1 extracted in this step can be combined with the panoramic video footage to form a panoramic video file. Similarly, the target audio signal 2 extracted in this step can be combined with the close-up video footage to form a close-up video file.

[0282] It should be noted that during the period from the start to the end of video recording, the electronic device 100 can process the audio stream captured by the microphone based on the above-described module interaction process. When the electronic device 100 receives a user's end-recording operation (e.g., a click operation on the end-recording button 314 as shown in the aforementioned user interface example), the electronic device 100 stops executing the above-described module interaction process. The electronic device 100 saves the target audio signal 1 and the panoramic video image as a panoramic video file, and saves the target audio signal 2 and the close-up video image as a close-up video file.

[0283] Next, the specific process of an audio processing method provided in this application will be introduced.

[0284] Please refer to Figure 17 , Figure 17 An exemplary schematic diagram of a specific audio processing method is shown.

[0285] like Figure 17 As shown, the specific process may include the following:

[0286] S601, the electronic device 100 displays a preview interface. This preview interface may include a preview screen, a protagonist mode control, and a recording control. The preview screen is an image captured in real-time by a camera, and includes one or more subjects being filmed.

[0287] Specifically, the aforementioned preview interface can be referred to as the first interface, and the aforementioned preview screen can be referred to as the first screen, which is an image captured in real time by the electronic device through the camera. The one or more shooting objects include shooting object 1 (also referred to as the first shooting object) and shooting object 2 (also referred to as the second shooting object).

[0288] For example, taking the aforementioned user interface example, the electronic device 100 can display as follows: Figure 4A The preview interface shown in C is shown below. This preview interface includes the preview screen, the protagonist mode button 404 (i.e., the protagonist mode control), and... Figure 3 The "Start Recording" button 303 shown in Figure a (i.e., the recording control) Figure 4A (Not marked in the text). The image displayed in the large window is the preview screen, which includes the male and female figures in the example.

[0289] S602, Electronic device 100 receives input 1 applied to the main character mode control.

[0290] For details on this step, please refer to step S501 above, which will not be repeated here.

[0291] In some examples, in response to input 1 applied to the protagonist mode control, the electronic device 100 may display a first tracking mark (e.g., the tracking box for a male character in the aforementioned user interface example) in the area of ​​the subject 1 and a second tracking mark (e.g., the tracking box for a female character in the aforementioned user interface example) in the area of ​​the subject 2; wherein the first and second tracking marks are used to mark the subjects identified by the electronic device 100 based on the preview screen.

[0292] Understandably, the protagonist mode control can be called the first shooting mode control, and inputting 1 can be called the first operation.

[0293] S603, in response to input 1, electronic device 100 activates main mode and obtains configuration information of audio transmission path.

[0294] For details regarding the configuration information of the audio transmission path, please refer to step S505, which will not be repeated here.

[0295] Understandably, the protagonist mode can be called the first shooting mode.

[0296] S604, Electronic device 100 verifies the configuration information of the audio transmission path based on the audio policy management information.

[0297] For a detailed explanation of this step, please refer to step S506 above, which will not be repeated here.

[0298] S605. After the above configuration information passes verification, the electronic device 100 configures the audio transmission path based on the configuration information. The audio transmission path includes a first number of channels.

[0299] For a detailed explanation of this step, please refer to steps S507-S509 above, which will not be repeated here.

[0300] It is understandable that when the first number is 4, the audio transmission path includes 4 channels, namely: channel 1 (which can be called the first channel), channel 2 (which can be called the second channel), channel 3 (which can be called the third channel) and channel 4 (which can be called the fourth channel).

[0301] S606, Electronic device 100 receives input 2 from the user selecting subject 1 as the main subject.

[0302] For a detailed explanation of this step, please refer to step S510 above, which will not be repeated here.

[0303] Understandably, input 2 can be referred to as the first input.

[0304] S607, in response to input 2, the electronic device 100 obtains the face coordinate information of the protagonist.

[0305] For a detailed explanation of this step, please refer to steps S511-S516 above, which will not be repeated here.

[0306] S608, Electronic device 100 receives input 3 from the user acting on the recording control.

[0307] For details on this step, please refer to step S517 above, which will not be repeated here.

[0308] S609. In response to input 3, electronic device 100 acquires audio signal 1 in real time through multiple microphones. Audio signal 1 includes sound signals acquired in real time by the multiple microphones respectively.

[0309] For a detailed explanation of this step, please refer to steps S518-S522 above, which will not be repeated here.

[0310] S610, electronic device 100 displays panoramic video images and close-up video images.

[0311] The panoramic video footage described here refers to the recording screen displayed in the large window of the aforementioned user interface example during recording. The close-up video footage refers to the recording screen displayed in the small window of the aforementioned user interface example during recording. For details on the panoramic and close-up video footage, please refer to the preceding description.

[0312] Specifically, the electronic device can display a second interface when recording begins (such as the one mentioned above). Figure 9A In section b), the second interface may include a panoramic video frame (which may be referred to here as the first video frame) and a close-up video frame (which may be referred to here as the second video frame). The panoramic video frame described here is the recording frame displayed in the large window in the aforementioned user interface example during recording, which is an image captured in real time by the electronic device 100 through the camera during video recording. The close-up video frame is the recording frame displayed in the small window in the aforementioned user interface example during recording, which is a video frame cropped from the position of the main character in the panoramic video frame during video recording.

[0313] In some examples, electronic device 100 can double-click the selected subject to make it the main subject and trigger video recording. Electronic device 100 displays a second interface and captures audio signals in real time via a microphone. This double-click selection is the first input. That is to say, there are no restrictions on how electronic device 100 triggers video recording and displays the second interface, or on how it captures audio signals in real time via a microphone.

[0314] S611, the electronic device 100 obtains target audio signal 1 and target audio signal 2 based on the protagonist's face coordinate information and audio signal 1. Target audio signal 1 and target audio signal 2 are different.

[0315] For a detailed explanation of this step, please refer to steps S523-S525 above, which will not be repeated here.

[0316] S612, Electronic device 100 receives input 4 from the user to end recording control.

[0317] For example, using the aforementioned user interface example, during video recording, the electronic device 100 may display something like... Figure 14 The end recording button 314 (also known as the end recording control) is shown in Figure a. The electronic device 100 can receive a click operation (i.e., input 4) on the end recording button 314.

[0318] Understandably, input 4 can be referred to as the second input.

[0319] S613, in response to input 4, electronic device 100 saves the panoramic video image and target audio signal 1 as a panoramic video file, and saves the close-up video image and target audio signal 2 as a close-up video file.

[0320] Specifically, target audio signal 1 and target audio signal 2 are transmitted within electronic device 100 based on the aforementioned configured audio transmission path. For explanations of panoramic video images, close-up video images, panoramic video files, close-up video files, target audio signal 1, and target audio signal 2, please refer to the foregoing descriptions; they will not be repeated here.

[0321] It is understandable that target audio signal 1 can be referred to as the first target audio signal, target audio signal 2 can be referred to as the second target audio signal, panoramic video file can be referred to as the first video file, and close-up video file can be referred to as the second video file.

[0322] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0323] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0324] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An audio processing method, the method is applied to an electronic device, the electronic device comprises a display screen, a camera and a microphone, characterized in that, The method includes: The electronic device displays a first interface, which includes a first screen and a first shooting mode control. The first screen is an image captured in real time by the electronic device through the camera, and the first screen includes a first shooting object. In response to a first operation applied to the first shooting mode control, the electronic device enters a first shooting mode and configures an audio transmission path, wherein the audio transmission path includes a first number of channels, the first number of channels including a first channel, a second channel, a third channel and a fourth channel; After the electronic device receives the first input that the first subject to be photographed is selected as the main subject, the electronic device starts recording and displays a second interface, and collects audio signals in real time through the microphone. The second interface includes a first window and a second window. The first window includes a first video frame captured in real time by the electronic device through the camera, and the second window includes a second video frame cropped based on the position of the first subject in the first video frame. The electronic device obtains a first target audio signal from the audio signal based on the audio data of the first channel and the audio data of the second channel; The electronic device obtains a second target audio signal from the audio signal based on the audio data of the third channel and the audio data of the fourth channel; The electronic device receives a second input; In response to the second input, the electronic device obtains a first video file based on the first video frame and the first target audio signal; the electronic device obtains a second video file based on the second video frame and the second target audio signal.

2. The method of claim 1, wherein, In response to a first operation applied to the first shooting mode control, the electronic device enters a first shooting mode and configures an audio transmission path, specifically including: In response to the first operation, the electronic device acquires configuration information of the audio transmission path; The electronic device verifies the number of audio channels in the configuration information based on audio policy management information; Once the configuration information passes verification, the electronic device configures the audio transmission path based on the configuration information.

3. The method according to claim 2, characterized in that, The configuration information includes one or more of the following: Number of channels, sample size of audio sampling points, total size of buffer blocks, and number of frames in the buffer.

4. The method according to claim 3, characterized in that, The number of audio channels in the configuration information is a first value; The electronic device verifies the configuration information based on audio policy management information, specifically including: Based on the first value, the electronic device determines whether the audio policy management information includes the first identifier corresponding to the first value; When the electronic device determines that the audio policy management information includes the first value, the electronic device determines that the configuration information passes the verification. When the electronic device determines that the first value is not included in the audio policy management information, the electronic device determines that the configuration information has failed the verification.

5. The method according to claim 4, characterized in that, Once the configuration information passes verification, the electronic device configures the audio transmission path based on the configuration information, specifically including: When the electronic device determines that the audio policy management information includes the first identifier, the electronic device configures the audio transmission path based on the configuration information; wherein, the audio transmission path includes a first number of channels, and the first number is the same as the first value.

6. The method according to claim 1, characterized in that, In response to a first operation performed on the first shooting mode control, the method further includes: The electronic device displays a first tracking mark in the area of ​​the first photographed object and a second tracking mark in the area of ​​the second photographed object; wherein the first tracking mark and the second tracking mark are used to mark the photographed object identified by the electronic device based on the first image.

7. The method according to claim 6, characterized in that, After the electronic device receives the first input that the first subject to be photographed is selected as the main subject, the color and / or shape of the first tracking mark changes.

8. The method according to claim 1, characterized in that, The first quantity is 4.

9. An electronic device, characterized in that, include: One or more processors, one or more memories, and a display screen; the one or more memories are coupled to one or more processors, the one or more memories being used to store computer program code, the computer program code including computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-8.

11. A chip or chip system, characterized in that, It includes a processing circuit and an interface circuit, wherein the interface circuit is used to receive code instructions and transmit them to the processing circuit, and the processing circuit is used to execute the code instructions to perform the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-8.

Images