Audio processing method and electronic device
By determining the position of the face or mouth in a video picture, the audio processing method enhances sound pickup within a specific range, addressing poor audio quality issues in electronic devices and improving recording effectiveness.
Patent Information
- Application Number
- JP2023513516
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-08-26
- Filing Date
- 2021-07-26
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2041-07-26
AI Technical Summary
Existing electronic devices face challenges in achieving ideal sound reception during videography due to movement of the photographed person or external noise, leading to poor audio quality, and the addition of external sound reception devices increases complexity and cost, while current audio enhancement methods require high processing power and hardware performance.
An audio processing method that determines the position of the face or mouth in a video picture to enhance sound pickup within a specific range, simplifying the audio processing algorithm and improving audio quality by emphasizing audio signals from the target area and reducing external noise interference.
The method enhances audio quality by focusing on the sound collection range of the primary speaker, reducing external noise interference, and improving the overall audio recording effect without increasing hardware complexity or cost.
Smart Images

Figure 0007583914000009 
Figure 0007583914000010 
Figure 0007583914000011
Abstract
Description
[Technical field]
[0001] This application claims priority to Chinese Patent Application No. 202010868463.5, entitled “AUDIO PROCESSING METHOD AND ELECTRONIC DEVICE”, filed with the State Intellectual Property Office of the People's Republic of China on August 26, 2020, the entirety of which is incorporated herein by reference.
[0002] The present application relates to the field of electronic technology, and in particular to audio processing methods and electronic devices. [Background technology]
[0003] Electronic devices such as mobile phones or tablet computers are widely used in the field of videography, such as short video shooting and webcasting. In the video shooting process, the sound receiving effect is usually not ideal due to reasons such as the movement of the person being shot or external noise, resulting in poor sound quality.
[0004] In order to improve the sound receiving effect, an external sound receiving device usually needs to be added based on the sound receiving using an electronic device. This leads to higher shooting difficulty and higher cost for users. In addition, a voice enhancement method is further proposed, and in the video shooting process, an audio algorithm is used to process the audio file collected by the electronic device to remove noise. However, since the shooting environment is relatively complex, the requirements for the processing ability of the audio algorithm are relatively strict. In addition, the complex audio processing process also increases the requirements for the hardware performance of the electronic device. Summary of the Invention [Means for solving the problem]
[0005] According to the audio processing method and electronic device provided in the present application, the position of the face or mouth of the person emitting the sound in a video picture is determined, and the area in which sound pickup needs to be improved is determined based on the position of the face or mouth of the person emitting the sound to implement directional sound enhancement, thereby not only simplifying the audio processing algorithm but also improving the audio quality.
[0006] In order to achieve the aforementioned objectives, the following technical solutions are used in this application.
[0007] According to a first aspect, the application provides an audio processing method. The method is applied to an electronic device, and may include the steps of: detecting a first operation of opening a camera application; displaying a shooting preview interface in response to the first operation; detecting a second operation of starting a video recording; collecting a video picture and a first audio in response to the second operation and displaying the shooting interface, the shooting interface including a preview interface of the video picture; recognizing a target image in the video picture, the target image being a first face image and / or a first mouth image, the first face image being a face image of a pronunciation object in the video image, and the first mouth image being a mouth image of a pronunciation object in the video image; then determining a first sound collection range corresponding to the pronunciation object based on the target image; and obtaining a second audio corresponding to the video picture based on the first sound collection range and the first audio, the volume of the second audio within the first sound collection range being greater than the volume outside the first sound collection range.
[0008] The method in this embodiment of the present application may be applied to a scenario in which a user instruction is received to directly launch a camera application, or a scenario in which a user launches another third-party application (e.g., a short video application, a live broadcast application, or a video calling application) to call and launch a camera. The first operation or the second operation includes, for example, a touch operation, a keystroke operation, an air gesture operation, or a voice operation.
[0009] Optionally, after the step of displaying the shooting preview interface in response to the first operation, the method further includes the steps of detecting a sixth operation to enable a voice enhancement mode, and enabling the voice enhancement mode in response to the sixth operation.
[0010] In some embodiments, after a switch to the video recording function is detected, the user is first asked whether to enable the voice enhancement mode. After the user decides to enable the voice enhancement mode, the voice enhancement mode is enabled. Alternatively, the voice enhancement mode is automatically enabled after a switch to the video recording function is detected. In some other embodiments, after a switch to the video recording function is detected, a video recording preview interface is first displayed, and then, after an operation of providing a shooting instruction is detected by the user, the voice enhancement mode is enabled based on a user instruction or the voice enhancement mode is automatically enabled.
[0011] After the voice enhancement mode is enabled, the electronic device needs to process the collected first audio, recognize the audio of the pronunciation object, and enhance the audio to obtain a better voice recording effect. The first audio is, for example, the collected initial audio signal, and the second audio is the audio obtained after the voice enhancement process is performed.
[0012] Optionally, the first face image or the first mouth image is recognized by using a face image recognition algorithm. For example, in the process of recording a video picture, by using a face image recognition algorithm, it is determined whether the collected video picture contains a face image. If a face image is included, the face image included in the collected video picture is recognized, and whether the face image is making a sound is determined based on the change status of the face feature data, such as the five sensory organ data and the face contour data, of the face image in a preset period. The criterion for determining that the face image is making a sound includes determining that the face image is currently making a sound. Alternatively, if the face image is determined to be making a sound again within a preset period after it is determined that the face image is making a sound for the first time, it is determined that the face image is making a sound. It may be understood that the human vocal organ is the human mouth. Also, when the data of the sound mouth is acquired, the data of the first mouth image may be preferentially determined, and the first sound collection range may be determined thereafter based on the data of the first mouth image. It should be noted that if a person in a video picture is making a sound but cannot be recognized, the image corresponding to the person making the sound is not the target image, i.e., the target image is the image corresponding to the recognized speaking face and / or mouth.
[0013] In this way, by recognizing the sound target image in the video picture, a first sound collection range in which sound collection should be emphasized is determined. Then, a second audio is obtained based on the collected initial audio signal and the first sound collection range. Therefore, the volume of the second audio within the first sound collection range is larger than that outside the first sound collection range. That is, the sound of the person who made the sound is made louder, thereby improving the audio recording effect.
[0014] In a possible implementation form, the step of determining a first sound pickup range corresponding to the sound-producing object based on the target image includes the steps of obtaining a first feature value based on the target image, the first feature value including one or more of a front / back attribute parameter, an area ratio, or location information, where the front / back attribute parameter is used to indicate whether the video picture is a video picture taken by a front camera or a video picture taken by a rear camera, the area ratio is used to indicate a ratio of the area of the target image to the area of the video picture, and the location information is used to indicate a location of the target image within the video picture, and then determining a first sound pickup range corresponding to the sound-producing object based on the first feature value.
[0015] The first feature value is used to describe a relative positional relationship between the face of a real person corresponding to the first face image and the electronic device, or the first feature value is used to describe a relative positional relationship between the mouth of a real person corresponding to the first mouth image and the electronic device. Thus, the electronic device can determine a first sound collection range based on the first feature value. For example, when a real person corresponding to the first face image is located directly in front of the electronic device, that is, when the first face image is located at the center position of the captured video picture, the first sound collection range is a directly in front sound collection range of the electronic device. Then, after acquiring an initial audio signal including audio signals in various directions, the electronic device can acquire audio corresponding to the first face image based on the initial audio signal and the first sound collection range.
[0016] In some embodiments, the first feature value may change in the recording process of the video picture. Therefore, the first sound pickup range also changes accordingly. Therefore, for the audio in the recorded video, the audio recorded by the electronic device includes at least an audio of a first duration and an audio of a second duration. The audio of the first duration is the audio corresponding to the first sound pickup range, and the audio of the second duration is the audio corresponding to the changed sound pickup range. That is, the electronic device can dynamically determine the sound pickup range based on the change of the pronunciation face or the pronunciation mouth in the video picture, and then record the audio based on the sound pickup range. After the operation of giving a recording stop instruction by the user is finally detected, the audio of the formed video picture may include multiple audios of different durations or the same duration recorded based on the time sequence and the changing sound pickup range.
[0017] In this way, based on the change of the sound pickup range, the electronic device can always focus on improving the audio recording quality of the part where the voice enhancement needs to be performed, thereby ensuring the audio recording effect.In addition, when the user plays the video file, the user can be presented with a playback experience that is consistent with the dynamic changes, such as the sound range changes, of the video content.
[0018] In a possible implementation form, the step of determining a first sound pickup range corresponding to the sound-producing object based on the first feature value includes a step of determining that the first sound pickup range is a front camera side sound pickup range when the video picture is a front video picture, and a step of determining that the first sound pickup range is a rear camera side sound pickup range when the video picture is a rear video picture.
[0019] For example, it is assumed that the sound pickup range of the electronic device includes a 180° front sound pickup range and a 180° rear sound pickup range. In this case, if the video picture is determined to be a front video picture, the 180° front sound pickup range is used as the first sound pickup range. If the video picture is determined to be a rear video picture, the 180° rear sound pickup range is used as the first sound pickup range. Furthermore, in the process of recording the video picture, in response to the user's operation of switching between the front camera and the rear camera, the first sound pickup range is also switched between the front and rear to ensure that the first sound pickup range is the sound pickup range corresponding to the sound-producing object in the video picture.
[0020] In a possible implementation, determining a first sound pickup range corresponding to the sound-producing object based on the first feature value includes determining the first sound pickup range based on an area ratio and a sound pickup range of the first audio.
[0021] The first audio pickup range is, for example, a panoramic audio pickup range. In the video recording process, the electronic device collects initial audio signals in different directions by using microphones, that is, obtains the initial audio signals within the panoramic audio pickup range.
[0022] Specifically, in the process of a user taking a video picture using a mobile phone, the person of interest of the user is usually located at the center position of the video picture, that is, the first face image or the first mouth image is located at the center position of the viewfinder frame. Different regions of the first face image or the first mouth image correspond to different sound pickup ranges, and the size such as the radius, diameter or area of the first sound pickup range can be described by using an area ratio.
[0023] For example, assume that X is used to represent the area of the first face image or the area of the first mouth image, Y is used to represent the area of the video picture displayed in the viewfinder frame, and N represents the pickup range corresponding to the framing range. In this case, the area ratio is X / Y, and the first pickup range is N*X / Y. That is, the ratio of the first pickup range to the panoramic pickup range is directly proportional to the area ratio.
[0024] In a possible implementation, determining a first sound pickup range corresponding to the sound object based on the first feature value includes determining a position of the first sound pickup range in the sound pickup range of the first audio based on position information.
[0025] In some scenarios, the sounding object is not located at the center position of the video picture. In this case, the position of the image (i.e., the target image) corresponding to the sounding object in the video picture can be obtained based on the position information. It can be understood that there is a correspondence between the position of the target image in the video picture and the position of the first sound collection range in the panoramic sound collection range.
[0026] In a possible implementation, the position information includes a first offset of the center point of the target image relative to a first reference point, the first reference point being a center point or a focus point of the video picture. Determining the position of the first audio pickup range in the first audio pickup range based on the position information includes determining a second offset of the center point of the first audio pickup range relative to the center point of the first audio pickup range based on the first offset, where the second offset is directly proportional to the first offset, and then determining the position of the first audio pickup range in the first audio pickup range based on the second offset.
[0027] The offset includes, for example, an offset direction, an offset angle, and / or an offset distance. The offset direction means that the center point of the first face image or the first mouth image has a left offset, a right offset, an upper offset, a lower offset, an upper left offset, a right upper offset, a lower left offset, a lower right offset, etc., relative to the first reference point. The offset angle is an angle at which the upper left offset, the right upper offset, the lower left offset, or the lower right offset exists. The offset distance is a distance of the left offset, the right offset, the upper offset, the lower offset, the offset distance at a specific offset angle, etc.
[0028] For example, the coordinate system is constructed by using the first reference point as the origin, the direction parallel to the bottom edge of the mobile phone (or the bottom edge of the current viewfinder frame) as the x-axis, and the direction perpendicular to the x-axis as the y-axis, and the current coordinate system is parallel to the display of the mobile phone. The offset direction, offset angle, and offset distance of the center point of the first face image or the first mouth image relative to the first reference point are defined by using the constructed coordinate system. For example, if the position information of the target image is the lower left of the center point of the viewfinder frame, the first sound pickup range is within the panoramic sound pickup range, and the center point of the first sound pickup range is the lower left of the center point of the panoramic sound pickup range.
[0029] In a possible implementation, the centre point of the video picture is the centre point of the viewfinder frame, or the centre point of the video picture is the centre point of the display.
[0030] In some scenarios, the center point of the viewfinder frame is used as the first reference point, i.e., the center point of the viewfinder frame is used to represent the center point of the video picture. It can be understood that based on the display format of the video picture, the first reference point may alternatively be represented in another format. For example, the center point of the entire screen of the display of a mobile phone is used to represent the center point of the video picture, i.e., as the first reference point.
[0031] In a possible implementation, the step of obtaining a second audio corresponding to the video picture based on the first pickup range and the first audio includes a step of emphasizing audio signals that are within the first pickup range and within the first audio, and / or a step of attenuating audio signals that are outside the first pickup range and within the first audio, to obtain the second audio.
[0032] For example, the first audio includes audio signals from different directions. After the first pickup range corresponding to the sound-producing object is determined, the audio signals in the first pickup range are enhanced to improve the audio quality in the recorded video. Optionally, the audio signals outside the pickup range are further attenuated to reduce the interference of external noise and to emphasize the sound emitted by the sound-producing object in the audio.
[0033] In a possible implementation, the electronic device includes one or more microphones, and the one or more microphones are configured to collect the first audio. When the pickup range of a first microphone of the one or more microphones includes a part or all of the first pickup range, acquiring the second audio corresponding to the video picture based on the first pickup range and the first audio includes performing at least one of the following steps to acquire the second audio: enhancing an audio signal that is within the first pickup range and within the pickup range of the first microphone, attenuating an audio signal that is outside the first pickup range and within the pickup range of the first microphone, and attenuating an audio signal of a microphone other than the first microphone of the one or more microphones.
[0034] For example, microphone 1 and microphone 2 are configured on a mobile phone. When the first sound collection range is within the sound collection range of microphone 1, after obtaining an initial audio signal using microphone 1 and microphone 2, the mobile phone may enhance the audio signal that is within the initial audio signal and within the first sound collection range and collected by microphone 1, attenuate the audio signal that is within the initial audio signal and outside the first sound collection range and collected by microphone 1, and attenuate the audio signal that is collected by microphone 2 to obtain audio corresponding to the first face image or the first mouth image. In another example, microphone 1 and microphone 2 are configured on a mobile phone. The first sound collection range includes sound collection range 1, which is the sound collection range of microphone 1, and sound collection range 2, which is the sound collection range of microphone 2. That is, the first sound collection range is the union of sound collection range 1 and sound collection range 2. In this case, after obtaining the initial audio signal by using microphone 1 and microphone 2, the mobile phone can enhance the audio signal that is within the sound collection range 1 of microphone 1 and is in the initial audio signal and the audio signal that is within the sound collection range 2 of microphone 2 and is in the initial audio signal, and weaken the remaining audio signals in the initial audio signal, so as to obtain audio corresponding to the first face image or the first mouth image. It can be understood that the sound collection range 1 and the sound collection range 2 may be completely overlapped or partially overlapped.
[0035] In a possible implementation, the electronic device includes at least two microphones, and the at least two microphones are configured to collect a first audio. The step of obtaining a second audio corresponding to the video picture based on the first pickup range and the first audio includes disabling the second microphone if the pickup range of the second microphone of the at least two microphones does not include the first pickup range, and the audio collected by a microphone other than the second microphone of the at least two microphones is the second audio.
[0036] For example, microphone 1 and microphone 2 are configured on a mobile phone. A first pickup range is within the pickup range of microphone 1 and outside the pickup range of microphone 2. In this case, the mobile phone disables microphone 2, processes the audio signal collected by microphone 1, and uses the processed audio signal as the audio corresponding to the video picture. That is, the audio corresponding to the first face image or the first mouth image is the audio collected by microphone 1.
[0037] In a possible implementation, when the second microphone is disabled, the method further includes a step of emphasizing audio signals that are within the first pickup range and within the pickup range of a microphone other than the second microphone of the at least two microphones, and / or a step of attenuating audio signals that are outside the first pickup range and within the pickup range of a microphone other than the second microphone of the at least two microphones.
[0038] For example, microphone 1 and microphone 2 are configured on a mobile phone. A first pickup range is within the pickup range of microphone 1 and outside the pickup range of microphone 2. In this case, the mobile phone disables microphone 2, enhances audio signals within the first pickup range in the audio signals collected by microphone 1, and attenuates audio signals outside the first pickup range in the audio signals collected by microphone 1, in order to obtain audio corresponding to the first face image or the first mouth image.
[0039] In a possible implementation, there are one or more first face images and one or more first mouth images. image exists.
[0040] There may be one or more people making sounds in the video picture. Therefore, there are one or more first face images and one or more first mouth images. If some people are making sounds in the currently captured video picture, but the mobile phone cannot recognize that the people are making sounds, it can be understood that the face images or mouth images of the unrecognized people making sounds are not classified as the first face images or first mouth images.
[0041] In some embodiments, when there are multiple first face images or multiple first mouth images, in the process of determining the first feature value, the first feature value needs to be determined based on the multiple first face images or multiple first mouth images. For example, in the process of determining the area ratio, the ratio of the area of the multiple first face images to the area of the video picture is used as the area ratio of the target image. In another example, in the process of determining the position information, the offset of the center point of the placeholder frame in which the multiple first face images are located relative to the center point of the video picture is used as the position information of the target image. The placeholder frame in which the multiple first face images are located is used to represent the minimum selection frame that includes the multiple face images.
[0042] In a possible implementation, after the step of collecting the video picture and the first audio and displaying the shooting interface in response to the second operation, the method further includes the steps of detecting a third operation to stop shooting, and in response to the third operation, stopping recording and generating a recorded video, the recorded video including the video picture and the second audio, and detecting a fourth operation to play the recorded video, and in response to the fourth operation, displaying a video playback interface and playing the video picture and the second audio.
[0043] In some embodiments, in the process of recording video pictures, the electronic device determines a first sound pickup range based on the pronunciation face image or the pronunciation mouth image, and then records audio based on the first sound pickup range. After that, the recorded audio needs to be stored. A user can play the video pictures and audio of the stored video.
[0044] It should be noted that when the scenario of recording the video picture is a real-time communication scenario such as live broadcast or video call, the method for recording audio in the process of recording the video picture can be referred to the above-mentioned method. However, after the user's operation of stopping the shooting is detected, that is, after the operation of stopping the communication is detected, there is no need to generate the recorded video, and the communication is stopped as it is. It can be understood that in some real-time communication scenarios, the user can alternatively choose to store the recorded video. In response to the user's operation, the electronic device determines whether to store the recorded video in the real-time communication scenario.
[0045] In a possible implementation, the recorded video further includes a third audio, the third audio being audio determined based on a second pickup range. The second pickup range is a pickup range determined based on the first pickup range and different from the first pickup range. The video playback interface includes a first control and a second control, the first control corresponding to the second audio and the second control corresponding to the third audio.
[0046] In some embodiments, since there may be an error between the first sound pickup range determined by the electronic device based on the first feature value and the display range of the first face image or the first mouth image, the electronic device may determine one or more reference first sound pickup ranges near the first sound pickup range. The electronic device obtains one audio based on the first sound pickup range, and obtains at least one audio based on the reference first sound pickup range. The electronic device may further use the panoramic audio as the one audio. In this case, the electronic device may obtain multiple audios corresponding to the first face image or the first mouth image based on the first sound pickup range. The one audio may be understood as one audio file.
[0047] Optionally, the video recording function may include a single-channel video recording function and a multi-channel video recording function. The single-channel video recording function means that the electronic device displays one viewfinder frame in the shooting process to record one video picture. The multi-channel video recording function means that the electronic device displays at least two viewfinder frames in the shooting process, and each viewfinder frame is used for one video picture. In the process of using the multi-channel video recording function, each video picture and the corresponding audio collection manner can be referred to the implementation form of the single-channel video recording function.
[0048] In this manner, the electronic device can switch to play audio corresponding to different pickup ranges to provide the user with multiple audio playback options, thereby implementing audio adjustment functionality and improving the user's audio playback experience.
[0049] In a possible implementation, the method further includes a step of playing the video picture and the second audio in response to a fourth operation, the fourth operation including an operation of manipulating the player control or an operation of the first control, a step of detecting a fifth operation of manipulating the second control, and a step of playing the video picture and the third audio in response to the fifth operation.
[0050] In another possible implementation, during video playback, the electronic device can display a video playback interface without first playing audio. After detecting a user instruction, the electronic device plays the audio as instructed by the user.
[0051] In a possible implementation, the method further includes deleting the secondary audio or the tertiary audio in response to an operation to delete the secondary audio or the tertiary audio.
[0052] In this way, in the video playing process, the audio that the user does not want to remember can be deleted based on the user requirements, thereby improving the user's usage experience.
[0053] According to a second aspect, the present application provides an electronic device. The electronic device includes a processor, a memory, a microphone, a camera, and a display. The memory, the microphone, the camera, and the display are coupled to the processor. The memory is configured to store computer program code, the computer program code including computer instructions. When the processor reads the computer instructions from the memory, the electronic device is enabled to perform the steps of detecting a first operation of opening a camera application; in response to the first operation, displaying a shooting preview interface; detecting a second operation of starting a video recording; in response to the second operation, collecting a video picture and a first audio and displaying the shooting interface, where the shooting interface includes a video picture preview interface; recognizing a target image in the video picture, where the target image is a first face image and / or a first mouth image, where the first face image is a face image of a pronunciation object in the video image, and the first mouth image is a mouth image of the pronunciation object in the video image; determining a first sound pickup range corresponding to the pronunciation object based on the target image; and obtaining a second audio corresponding to the video picture based on the first sound pickup range and the first audio, where a volume of the second audio within the first sound pickup range is greater than a volume outside the first sound pickup range.
[0054] In a possible implementation form, the step of determining a first sound pickup range corresponding to the sound-producing object based on the target image includes the steps of obtaining a first feature value based on the target image, where the first feature value includes one or more of a front / back attribute parameter, an area ratio, or location information, where the front / back attribute parameter is used to indicate whether the video picture is a video picture taken by a front camera or a video picture taken by a rear camera, the area ratio is used to indicate a ratio of the area of the target image to the area of the video picture, and the location information is used to indicate the location of the target image within the video picture; and determining a first sound pickup range corresponding to the sound-producing object based on the first feature value.
[0055] In a possible implementation form, the step of determining a first sound pickup range corresponding to the sound-producing object based on the first feature value includes a step of determining that the first sound pickup range is a front camera side sound pickup range when the video picture is a front video picture, and a step of determining that the first sound pickup range is a rear camera side sound pickup range when the video picture is a rear video picture.
[0056] In a possible implementation, determining a first sound pickup range corresponding to the sound-producing object based on the first feature value includes determining the first sound pickup range based on an area ratio and a sound pickup range of the first audio.
[0057] In a possible implementation, determining a first sound pickup range corresponding to the sound object based on the first feature value includes determining a position of the first sound pickup range in the sound pickup range of the first audio based on position information.
[0058] In a possible implementation, the position information includes a first offset of the center point of the target image relative to a first reference point, the first reference point being a center point of the video picture or a focal point of focus. Determining the position of the first audio pickup range in the first audio pickup range based on the position information includes determining a second offset of the center point of the first audio pickup range relative to the center point of the first audio pickup range based on the first offset, where the second offset is directly proportional to the first offset, and determining the position of the first audio pickup range in the first audio pickup range based on the second offset.
[0059] In a possible implementation, the centre point of the video picture is the centre point of the viewfinder frame, or the centre point of the video picture is the centre point of the display.
[0060] In a possible implementation, the step of obtaining a second audio corresponding to the video picture based on the first pickup range and the first audio includes a step of emphasizing audio signals that are within the first pickup range and within the first audio, and / or a step of attenuating audio signals that are outside the first pickup range and within the first audio, to obtain the second audio.
[0061] In a possible implementation, the electronic device includes one or more microphones, and the one or more microphones are configured to collect the first audio. When the pickup range of a first microphone of the one or more microphones includes a part or all of the first pickup range, acquiring the second audio corresponding to the video picture based on the first pickup range and the first audio includes performing at least one of the following steps to acquire the second audio: enhancing an audio signal that is within the first pickup range and within the pickup range of the first microphone, attenuating an audio signal that is outside the first pickup range and within the pickup range of the first microphone, and attenuating an audio signal of a microphone other than the first microphone of the one or more microphones.
[0062] In a possible implementation, the electronic device includes at least two microphones, and the at least two microphones are configured to collect a first audio. The step of obtaining a second audio corresponding to the video picture based on the first pickup range and the first audio includes disabling the second microphone if the pickup range of the second microphone of the at least two microphones does not include the first pickup range, and the audio collected by a microphone other than the second microphone of the at least two microphones is the second audio.
[0063] In a possible implementation, when the second microphone is disabled and when the processor reads the computer instructions from the memory, the electronic device is further enabled to perform a step of emphasizing audio signals that are within the first pickup range and within the pickup range of a microphone other than the second microphone of the at least two microphones, and / or a step of attenuating audio signals that are outside the first pickup range and within the pickup range of a microphone other than the second microphone of the at least two microphones.
[0064] In a possible implementation, there are one or more first face images and one or more first mouth images. image exists.
[0065] In a possible implementation, when the processor reads the computer instructions from the memory, the electronic device is enabled to perform the steps of detecting a third operation to stop filming, and in response to the third operation, stopping recording and generating a recorded video, the recorded video including the video picture and the second audio, and detecting a fourth operation to play the recorded video, and in response to the fourth operation, displaying a video playback interface and playing the video picture and the second audio.
[0066] In a possible implementation, the recorded video further includes a third audio, the third audio being audio determined based on a second pickup range. The second pickup range is a pickup range determined based on the first pickup range and different from the first pickup range. The video playback interface includes a first control and a second control, the first control corresponding to the second audio and the second control corresponding to the third audio.
[0067] In a possible implementation, when the processor reads the computer instructions from the memory, the electronic device is further enabled to perform the steps of: playing the video picture and the second audio in response to a fourth operation, the fourth operation including an operation of manipulating the player control or an operation of the first control; detecting a fifth operation of manipulating the second control; and playing the video picture and the third audio in response to the fifth operation.
[0068] In a possible implementation, when the processor reads the computer instructions from the memory, the electronic device is further enabled to perform a step of deleting the second audio or the third audio in response to an operation of deleting the second audio or the third audio.
[0069] In a possible implementation, when the processor reads the computer instructions from the memory, the electronic device is further enabled to perform the steps of detecting a sixth action that enables the voice enhancement mode and enabling the voice enhancement mode in response to the sixth action.
[0070] In addition, for the technical effects of the electronic device according to the second aspect, please refer to the technical effects of the audio processing method according to the first aspect, which will not be described in detail here.
[0071] According to a third aspect, the present application provides an electronic device. The electronic device has a function for implementing the audio processing method according to the first aspect or any one of the possible implementation forms of the first aspect. The function may be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the function.
[0072] According to a fourth aspect, the present application provides a computer-readable storage medium comprising computer instructions which, when executed on an electronic device, enable the electronic device to perform an audio processing method according to the first aspect or any one of the possible implementations of the first aspect.
[0073] According to a fifth aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to perform an audio processing method according to the first aspect or any one of the possible implementations of the first aspect.
[0074] According to a sixth aspect, there is provided a circuit system including a processing circuit configured to perform an audio processing method according to the first aspect or any one of the possible implementations of the first aspect.
[0075] According to a seventh aspect, an embodiment of the present application provides a chip system including at least one processor and at least one interface circuit. The at least one interface circuit is configured to perform a transceiver function and send instructions to the at least one processor. When the at least one processor executes the instructions, the at least one processor performs an audio processing method according to the first aspect or any one of the possible implementations of the first aspect. [Brief description of the drawings]
[0076] [Figure 1] FIG. 1 is a schematic diagram of a structure of an electronic device according to an embodiment of the present application. [Figure 2A] FIG. 2 is a schematic layout diagram of a camera according to an embodiment of the present application. [Figure 2B(a)] FIG. 2 is a schematic layout diagram of a microphone according to an embodiment of the present invention. [Figure 2B(b)] FIG. 2 is a schematic layout diagram of a microphone according to an embodiment of the present invention. [Figure 2B(c)] FIG. 2 is a schematic layout diagram of a microphone according to an embodiment of the present invention. [Figure 2B(d)] FIG. 2 is a schematic layout diagram of a microphone according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a schematic block diagram of the software structure of an electronic device according to an embodiment of the present application. [Figure 4(a)] 1 is a schematic diagram 1 of a group of interfaces according to an embodiment of the present application; [Figure 4(b)] 1 is a schematic diagram 1 of a group of interfaces according to an embodiment of the present application; [Figure 4(c)] 1 is a schematic diagram 1 of a group of interfaces according to an embodiment of the present application; [Figure 4(d)] 1 is a schematic diagram 1 of a group of interfaces according to an embodiment of the present application; [Diagram 5] 1 is a schematic diagram of a sound pickup range according to an embodiment of the present application. [Figure 6] 1 is a schematic flowchart 1 of an audio processing method according to an embodiment of the present application. [Figure 7] 1 is a schematic diagram of an interface according to an embodiment of the present application; [Figure 8] 2 is a schematic diagram 2 of a group of interfaces according to an embodiment of the present application. [Figure 9] 2 is a schematic diagram of a sound pickup range according to an embodiment of the present application. [Figure 10(a)] 3 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 10(b)] 3 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 10(c)] 3 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 11(a)] 4 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 11(b)] 4 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 11(c)] 4 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 12(a)] 5 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 12(b)] 5 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 12(c)] 5 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 13] FIG. 2 is a schematic diagram of a coordinate system according to an embodiment of the present application. [Figure 14] FIG. 2 is a schematic diagram of an offset angle according to an embodiment of the present application. [Figure 15] FIG. 2 is a schematic diagram of an offset distance according to an embodiment of the present application. [Figure 16A(a)] 1 is a schematic diagram 1 of a first sound collection range according to an embodiment of the present application. [Figure 16A(b)] 1 is a schematic diagram 1 of a first sound collection range according to an embodiment of the present application. [Figure 16B(a)] FIG. 2 is a schematic diagram of a first sound collection range according to an embodiment of the present application. [Figure 16B(b)] FIG. 2 is a schematic diagram of a first sound collection range according to an embodiment of the present application. [Figure 16B(c)] FIG. 2 is a schematic diagram of a first sound collection range according to an embodiment of the present application. [Figure 16C] FIG. 3 is a schematic diagram of a first sound collection range according to an embodiment of the present application. [Figure 17] 2 is a schematic diagram of an interface according to an embodiment of the present application. [Figure 18(a)] 6 is a schematic diagram 6 of a group of interfaces according to an embodiment of the present application. [Figure 18(b)] 6 is a schematic diagram 6 of a group of interfaces according to an embodiment of the present application. [Figure 18(c)] 6 is a schematic diagram 6 of a group of interfaces according to an embodiment of the present application. [Figure 18(d)] 6 is a schematic diagram 6 of a group of interfaces according to an embodiment of the present application. [Figure 19(a)] 7 is a schematic diagram 7 of a group of interfaces according to an embodiment of the present application. [Figure 19(b)] 7 is a schematic diagram 7 of a group of interfaces according to an embodiment of the present application. [Figure 19(c)] 7 is a schematic diagram 7 of a group of interfaces according to an embodiment of the present application. [Figure 20(a)] 8 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 20(b)] 8 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 20(c)]8 is a schematic diagram of a group of interfaces according to an embodiment of the present application. [Figure 21A] 2 is a schematic flowchart 2 of an audio processing method according to an embodiment of the present application. [Figure 21B] 2 is a schematic flowchart 2 of an audio processing method according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0077] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS With reference to the accompanying drawings, the following describes in detail the audio processing method and electronic device provided in the embodiments of the present application.
[0078] The audio processing method provided in the embodiments of the present application may be applied to an electronic device. For example, the electronic device may be specifically a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an artificial intelligence device, a dedicated camera (e.g., a single-lens reflex camera or a card camera), etc. The specific type of the electronic device is not limited in the embodiments of the present application.
[0079] 1 shows a schematic diagram of the structure of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, a subscriber identification module (SIM) card interface 195, and the like.
[0080] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or may be integrated into one or more processors.
[0081] The controller may be the brain and command center of the electronic device 100. The controller may generate operation control signals based on instruction operation codes and time series signals to complete the control of instruction fetching and instruction execution.
[0082] A memory may further be disposed in the processor 110 and configured to store instructions and data. In some embodiments, the memory in the processor 110 is a cache. The memory may store instructions or data that have been used or are used periodically by the processor 110. When the processor 110 needs to use the instructions or data again, the processor can directly recall the instructions or data from the memory. This avoids repeated accesses, reduces the latency of the processor 110, and improves system efficiency.
[0083] In some embodiments of the present application, the processor 110 performs image recognition on multiple frames of images in the collected video pictures to obtain face image data and / or mouth image data included in the frames of images. By comparing the changes in face image data and / or mouth image data in the frames of images, such as the changes in the distance between the upper lip and the lower lip and the changes in the contours of the face, information such as the position and ratio of the pronunciation face and / or the pronunciation mouth in the frames of images (i.e., in the video picture) is determined. Also, based on the information such as the position and ratio of the pronunciation face and / or the pronunciation mouth in the video picture, the sound pickup range to be emphasized is determined. That is, the position area of the sound of the person who made the sound in the panoramic audio is determined. The audio signal of the sound pickup range is emphasized to improve the audio quality of the recorded video. Optionally, the audio signal outside the sound pickup range is further weakened to reduce the interference of external noise.
[0084] The charging management module 140 is configured to receive a charging input from a charger.
[0085] The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the display 194, the camera 193, etc.
[0086] The wireless communication function of the electronic device 100 may be implemented by using antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, a modem processor, a baseband processor, and the like.
[0087] The mobile communication module 150 may be applied to the electronic device 100 to provide wireless communication solutions including 2G / 3G / 4G / 5G, etc. The wireless communication module 160 may be applied to the electronic device 100 to provide wireless communication solutions including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), etc.
[0088] The electronic device 100 can implement display functionality via a GPU, a display 194, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display 194 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render images. The processor 110 can include one or more GPUs that execute program instructions to generate or modify display information.
[0089] The display 194 is configured to display images, videos, etc. The display 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.
[0090] In some embodiments, the display 194 may display a capture preview interface, a video record preview interface, a capture interface in video record mode, and may further display a video playback interface during video playback, and the like.
[0091] The electronic device 100 can implement a photographing function using an ISP, a camera 193, a video codec, a GPU, a display 194, an application processor, and the like.
[0092] The ISP is configured to process data fed back by the camera 193. For example, during shooting, the shutter is pressed and light is sent to the camera's photosensitive elements through the lens. The light signal is converted into an electrical signal, and the camera's photosensitive elements send the electrical signal to the ISP for processing and convert the electrical signal into a visible image. The ISP can further perform algorithm optimization on noise, brightness, and skin tone of the image. The ISP can further optimize parameters such as exposure and color temperature for the shooting scenario. In some embodiments, the ISP may be located in the camera 193. For example, in this embodiment of the present application, the ISP can control the photosensitive elements to perform exposure and shooting based on the shooting parameters.
[0093] The camera 193 is configured to capture still images or videos. An optical image of an object is generated through a lens and projected onto a photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) photoelectric transistor. The photosensitive element converts the optical signal into an electrical signal, which is then sent to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, e.g., RGB or YUV.
[0094] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1. The cameras 193 may be located at an edge region of the electronic device, may be under-display cameras, or may be elevation cameras. The cameras 193 may include rear cameras and may further include front cameras. The specific location and form of the cameras 193 are not limited in this embodiment of the application.
[0095] For example, see Figure 2A for the layout of cameras on electronic device 100. The front side of electronic device 100 is the plane on which display 194 is located. As shown in Figure 2A (a), camera 1931 is located on the front of electronic device 100 and is a front camera. As shown in Figure 2A (b), camera 1932 is located on the back of electronic device 100 and is a rear camera.
[0096] Optionally, the solution in the embodiment of the present application may be applied to an electronic device 100 with a foldable screen (i.e., the display 194 can be folded) with multiple displays. In FIG. 2A (c), an electronic device 100 with a foldable screen is shown. In response to a user's operation, as shown in FIG. 2A (d), the display is folded inward (or outward) along a folding edge so that the display forms at least two screens (e.g., screen A and screen B). As shown in FIG. 2A (e), there is a display (e.g., screen C) on the outside of the folded display. In the unfolded scenario of the electronic device 100 shown in FIG. 2A (c), if a camera is placed on the electronic device 100 on the surface on which screen C is located, the camera on screen C is on the back of the electronic device 100 and can be considered as a rear camera. In the folded scenario of electronic device 100 shown in FIG. 2A(e), the camera on screen C becomes the front side of electronic device 100 and can be considered a front camera. That is, the front camera and rear camera in this application do not limit the nature of the cameras, but are merely descriptions of the positional relationship.
[0097] Thus, the electronic device 100 may determine whether the camera is a front camera or a rear camera based on the position of the used camera on the electronic device 100, and then determine the direction to collect the sound. For example, if the electronic device 100 is currently collecting images by using a rear camera located on the rear of the electronic device 100, the electronic device 100 should focus on collecting sounds at the rear of the electronic device 100. In another example, if the electronic device 100 is currently collecting images by using a front camera located on the front of the electronic device 100, the electronic device 100 should focus on collecting sounds in front of the electronic device 100. In this way, it is ensured that the collected sounds can match the collected images.
[0098] The digital signal processor is configured to process digital signals, and may process other digital signals in addition to the digital image signal. For example, when the electronic device 100 selects a frequency, the digital signal processor is configured to perform a Fourier transform on the frequency energy.
[0099] A video codec is configured to compress or decompress digital video. Electronic device 100 may support one or more types of video codecs. Thus, electronic device 100 can play or record video in multiple coding formats, e.g., moving picture experts group (MPEG)-1, MPEG-2, MPEG-3, and MPEG-4.
[0100] The NPU is a neural-network (NN) computing processor. The NPU can quickly process input information by referring to the structure of biological neural networks, such as the transmission mode between human brain neurons, and further perform self-learning continuously. Applications such as intelligent cognition, face recognition, speech recognition, and text understanding of the electronic device 100 can be implemented through the NPU.
[0101] In some embodiments, the NPU recognizes whether the image collected by the camera 193 includes a face image and / or a mouth image by using image recognition technology. In addition, the NPU may further determine the pronunciation face or the pronunciation mouth in the face image and / or the mouth image based on the data of the face image and / or the mouth image to determine the sound pickup range in which the directional audio recording needs to be performed.
[0102] The external memory interface 120 can be used to connect to an external storage card, such as a Micro SD card, to expand the storage capabilities of the electronic device 100. The external storage card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. The external storage card stores files such as music and videos.
[0103] The internal memory 121 may be configured to store computer-executable program code, which includes instructions. The processor 110 executes the instructions stored in the internal memory 121 and / or in a memory located in the processor to perform various functions, applications, and data processing of the electronic device 100.
[0104] The electronic device 100 can implement audio functions, such as music playback and recording, via an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, an application processor, and the like.
[0105] The audio module 170 is configured to convert digital audio data to an analog audio electrical signal output, and is further configured to convert an analog audio electrical signal input to digital audio data. The audio module 170 may include an analog-to-digital converter and a digital-to-analog converter. For example, the audio module 170 is configured to convert an analog audio electrical signal output by the microphone 170C to digital audio data. The audio module 170 may be further configured to encode and decode the audio data. In some embodiments, the audio module 170 may be disposed in the processor 110, or some functional modules of the audio module 170 are disposed in the processor 110.
[0106] The speaker 170A, also referred to as a "loudspeaker," is configured to convert audio electrical signals into sound signals. The electronic device 100 can use the speaker 170A to listen to music or answer hands-free calls.
[0107] Receiver 170B, also referred to as an "earpiece," is configured to convert electrical audio signals into sound signals. When a call is taken or audio information is listened to using electronic device 100, receiver 170B can be held close to a person's ear to hear the sound.
[0108] The microphone 170C is also referred to as a "mike" or a "mic" and is configured to convert a sound signal into an electrical signal. When making a call or transmitting speech information, a user can bring the user's mouth close to the microphone 170C to emit a sound and input the sound signal into the microphone 170C. The microphone 170C may be built into the electronic device 100 or may be externally attached to the electronic device 100.
[0109] In some embodiments, the electronic device 100 may include one or more microphones 170C. Each microphone or multiple microphones may cooperate to implement a function of collecting sound signals in different directions and converting the collected sound signals into analog audio electrical signals, and may further implement a noise reduction, sound source recognition, or directional voice recording function.
[0110] For example, as shown in Figure 2B(a)-Figure 2B(d), two schematic layout diagrams of multiple microphones on the electronic device 100 and the corresponding sound pickup ranges of the microphones are provided as examples. As shown in Figure 2B(a), when the electronic device 100 is placed in the position shown in the figure, the front side of the electronic device 100 is the plane where the display 194 is located, the microphone 21 is located at the top of the electronic device 100 (the side where the earpiece and camera are usually located), the microphone 22 is located at the right side of the electronic device 100, and the microphone 23 is located at the bottom of the electronic device 100 (part of the bottom of the electronic device 100 shown in Figure 2B(a) is not visible from the current angle, and the position of the microphone 23 is shown generally by using a dotted line).
[0111] Please note that the terms "upper", "lower", "left" and "right" described in the following embodiments are based on the directions shown in Figures 2B(a) to 2B(d). Details will not be described later.
[0112] In the schematic diagram of the sound collection range shown in FIG. 2B(b), the sound collection range corresponding to the microphone 21 includes a front upper sound collection range and a rear upper sound collection range, the sound collection range corresponding to the microphone 22 includes a front middle sound collection range and a rear middle sound collection range, and the sound collection range corresponding to the microphone 23 includes a front lower sound collection range and a rear lower sound collection range. By combining the microphones 21 to 23, sound signals in various directions around the electronic device 100 can be collected. The front camera may correspond to the front sound collection range, and the rear camera may correspond to the rear sound collection range. Therefore, when the electronic device 100 records a video using the front camera, the sound collection range is determined to be the front sound collection range. In addition, it is more accurately determined that the sound collection range is a range included in the front sound collection range based on the position of the sound-making face or sound-making mouth in the video picture. A specific method will be described in detail below.
[0113] It can be understood that the electronic device 100 may alternatively include a greater number of microphones. As shown in Fig. 2B(c), the electronic device 100 includes six microphones. Microphone 24 is located at the top of the electronic device 100, microphone 25 is located at the left side of the electronic device 100, microphone 26 is located at the bottom of the electronic device 100, and microphones 27-29 are located at the right side of the electronic device 100. The left side of the electronic device 100 shown in Fig. 2B(c) is not visible from the current angle, and the positions of microphones 25 and 26 are shown diagrammatically by dotted lines. 2B(d), the pickup range corresponding to microphone 24 includes an upper front pickup range, the pickup range corresponding to microphone 25 includes a middle front pickup range, the pickup range corresponding to microphone 26 includes a lower front pickup range, the pickup range corresponding to microphone 27 includes an upper rear pickup range, the pickup range corresponding to microphone 28 includes a middle rear pickup range, and the pickup range corresponding to microphone 29 includes a lower rear pickup range. By combining microphones 24 to 29, sound signals from various directions around electronic device 100 can be collected.
[0114] As shown in FIG. 2B(b) and FIG. 2B(d), the pickup ranges of the microphones of the electronic device 100 collecting audio signals overlap partially, i.e., in the shaded areas of FIG. 2B(b) and FIG. 2B(d). In the audio recording process, the audio signals in the overlapping areas need to be fused. For the same direction, the sound signal collected by a microphone may have better sound quality (e.g., higher signal-to-noise ratio and lower spike and glitch noise), and the sound signal collected by another microphone may have worse sound quality. In this case, the audio data with better sound quality in the corresponding direction is selected for fusion processing, and the audio with better effect is recorded and generated based on the processed audio data. In addition, when the pickup ranges of the multiple microphones include a pickup range corresponding to the speaking face or the speaking mouth, the audio data collected by the multiple microphones may be fused to obtain the audio corresponding to the speaking face or the speaking mouth.
[0115] In some embodiments, microphone 170C may be a directional microphone that may collect sound signals in a particular direction. Microphone 170C may alternatively be an omnidirectional microphone that collects sound signals in various directions, or may collect sound signals within a particular range based on the location of microphone 170C on electronic device 100.
[0116] In some other embodiments, the microphone 170C may be rotated, and the electronic device 100 may adjust the sound collection direction by rotating the microphone. Also, the electronic device 100 may configure one microphone 170C for a sound collection range corresponding to the speaking face or the sound mouth, and may collect sounds in various directions by rotating the microphone. When the electronic device 100 is configured with a plurality of microphones 170C, audio signals in the corresponding sound collection range may be collected using a combination of different microphones 170C. For example, it is not necessary to use all the microphones 170C of the electronic device 100 for sound collection, and some of the microphones 170C may be used for sound collection. In another example, audio signals collected by some of the microphones 170C are emphasized, and audio signals collected by some of the microphones 170C are weakened.
[0117] The number of microphones 170C is not particularly limited in this embodiment of the present application.
[0118] The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, an optical proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.
[0119] The distance sensor 180F is configured to measure distance. The electronic device 100 can measure distance by infrared or laser. In some embodiments, in a photography scenario, the electronic device 100 may measure distance by using the distance sensor 180F to implement rapid focusing.
[0120] The touch sensor 180K may be referred to as a touch panel. The touch sensor 180K may be disposed on a display 194, and the touch sensor 180K and the display 194 form a touch screen, also referred to as a "touch screen." The touch sensor 180K is configured to detect a touch operation performed on or near the touch sensor.
[0121] For example, in this embodiment of the present application, the electronic device 100 may detect an action of a user providing a start video recording instruction and / or a stop video recording instruction by using the touch sensor 180K.
[0122] It may be understood that the structure shown in this embodiment of the present application does not constitute a particular limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figures, combine some components, divide some components, or have a different component arrangement. The components shown in the figures may be implemented using hardware, software, or a combination of software and hardware.
[0123] The software system of the electronic device 100 can use a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In this embodiment of the present invention, the Android system of the layered architecture is used as an example to illustrate the software structure of the electronic device 100.
[0124] FIG. 3 is a block diagram illustrating the software architecture of electronic device 100 according to one embodiment of the present invention.
[0125] In layered architecture, software is divided into multiple layers, each layer has a clear role and task. The layers communicate with each other through software interfaces. In some embodiments, the operating system of an electronic device (e.g., Android system) is divided into four layers, which are, from bottom to top, kernel layer, hardware abstraction layer (HAL), application framework layer, and application layer, respectively.
[0126] The kernel layer is a layer between the hardware and the software, and includes at least a camera driver, an audio driver, a display driver, and a sensor driver.
[0127] In some embodiments, for example, in a video recording application scenario, the touch sensor 180K transmits the received touch operation to the upper layer camera application by using a sensor driver in the kernel layer. After the camera application recognizes that the touch operation is an operation to start video recording, the camera application calls the camera 193 to record a video picture by using the camera driver, and calls the microphone 170C to record audio by using the audio driver. In the above process, a corresponding hardware interrupt is sent to the kernel layer, and the kernel layer can process the corresponding operation into an original input event (for example, the touch operation includes information such as the touch coordinates and timestamp of the touch operation). The original input event is stored in the kernel layer.
[0128] A hardware abstraction layer (HAL) is located between the kernel layer and the application framework layer, and is configured to define an interface implemented by the application driver hardware and convert values implemented by the driver hardware into a software implementation programming language. For example, to call a camera service system, a value of a camera driver is recognized, the value is converted into a software programming language, and the software programming language is uploaded to the application framework layer.
[0129] In some embodiments, the HAL may upload raw data obtained after facial image recognition is performed on the video pictures collected by the camera 193 to the application framework layer for further processing. The raw data obtained after facial image recognition may include, for example, facial image data and / or mouth image data. The facial image data may include an amount of pronunciation facial images, position information of the pronunciation facial images in the video pictures, etc. The mouth image data may include an amount of pronunciation mouth images, position information of the pronunciation mouth images in the video pictures, etc.
[0130] For example, the priority of face image data and mouth image data is set in advance. The human vocal organ is the human mouth, and the sound collection range can be determined more accurately by using the sound collection data. For this reason, the priority of the mouth image data is set higher than the priority of the face image data. For example, the HAL can determine the sound collection face image data and the sound collection mouth image data based on the collected video pictures, and upload the sound collection mouth data as raw data based on the priority. Then, the audio processing system determines the sound collection range corresponding to the sound collection mouth image based on the sound collection mouth image data and the correspondence between the video pictures and the panoramic audio. In another example, the HAL determines only the sound collection face image data based on the collected video pictures, and uploads the sound collection face image data as raw data to determine the sound collection range corresponding to the sound collection face image. In yet another example, the HAL determines only the sound collection mouth image data based on the video pictures, and uploads the sound collection mouth image data as raw data to determine the sound collection range corresponding to the sound collection mouth image.
[0131] The application framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. The application framework layer obtains original input events from the kernel layer through the HAL and identifies controls corresponding to the input events. The application framework layer includes some predefined functions.
[0132] As shown in FIG. 3, the application framework layer may include a camera service system, an audio processing system, a view system, a telephone manager, a resource manager, a notification manager, a window manager, etc.
[0133] The camera service system is used to provide a camera application and invoke the camera application to collect images based on the original events input by the kernel layer.
[0134] The audio processing system is used to manage audio data and process the audio data by using different audio algorithms. For example, the audio processing system cooperates with the camera service system to process the audio signals collected in the video recording process. For example, the audio processing system determines a pickup range based on face image data, and enhances the audio signals within the pickup range and attenuates the audio signals outside the pickup range.
[0135] In some embodiments, the camera application calls a camera service system at the application framework layer to start the camera application, then calls the kernel layer to launch the camera driver and capture video using the camera 193. The audio processing system is called and the kernel layer is used to launch the audio driver, collect sound signals using the microphone 170C, generate analog audio electrical signals, generate digital audio data from the analog audio electrical signals using the audio module 170, and generate audio based on the digital audio data.
[0136] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be configured to build applications. A display interface can include one or more views. For example, a display interface that includes a notification icon for a message may include a text display view and a picture display view.
[0137] The telephone manager is configured to provide communication functions of the electronic device 100, such as managing the call status (including answering, rejecting, etc.).
[0138] The resource manager provides various resources such as localized strings, icons, pictures, layout files, video files, etc. to an application.
[0139] The notification manager may be configured to allow applications to display notification information in the status bar and to send notification type messages. The displayed information may disappear automatically without user interaction after a short pause. For example, the notification manager may be configured to notify of download completion, provide message notifications, etc. The notification manager may alternatively be notifications that appear in the top status bar of the system in the form of graphs or scrollbar text, notifications of applications running in the background, or notifications that appear on the screen in the form of dialog windows. For example, text information may be displayed in the status bar, an announcement may be made, the electronic device may vibrate, an indicator light may flash, etc.
[0140] A window manager is configured to manage window programs. The window manager may get the size of the display, determine if there is a status bar, perform screen locking, take screenshots, etc.
[0141] The application layer may include a set of application packages.
[0142] As shown in FIG. 3, the application package may include applications such as camera, video, phone, WLAN, music, messaging, Bluetooth, maps, calendar, gallery, and navigation.
[0143] The application layer and the application framework layer run on a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is configured to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0144] The audio processing method provided in the embodiments of the present application is described below by using an example in which the electronic device is a mobile phone having the structure shown in FIG. 1 and FIG.
[0145] In some embodiments, the methods in the embodiments of the present application may be applied to a scenario in which a user instruction is received to directly launch a camera application (hereinafter sometimes referred to as camera for short), or may be applied to a scenario in which a user launches another third-party application (e.g., a short video application, a live broadcast application, or a video calling application) to call and launch the camera.
[0146] In the following, we use the scenario where the camera is directly activated as an illustrative example.
[0147] Optionally, the user may instruct the mobile phone to start the camera and display the shooting preview interface by a method such as a touch operation, a keystroke operation, an air gesture operation, or a voice operation. For example, in the main interface 401 shown in FIG. 4(a), the mobile phone starts the camera in response to a user's operation of tapping the camera icon 41, and displays the shooting preview interface 402 shown in FIG. 4(b). Alternatively, the mobile phone starts the camera in response to a user's voice instruction operation to open the camera, and displays the shooting preview interface 402 shown in FIG. 4(b). The control 421 is used to set the shooting function of the mobile phone, for example, delay shooting. The control 422 is used to enable or disable the filter function. The control 423 is used to enable or disable the flash function.
[0148] On the shooting preview interface, the camera can switch between different functions in response to a user tapping on different function controls. For example, as shown in FIG. 4(b), controls 431-434 are used to switch between functions that can be implemented by the camera. For example, control 432 is currently selected and initiates a shooting function. In another example, in response to a user tapping on control 431, the camera switches to a portrait shooting function. Alternatively, in response to a user tapping on control 433, the camera switches to a video recording function. Alternatively, in response to a user tapping on control 434, more functions that the camera can switch to are displayed, such as panoramic shooting.
[0149] In the following, an explanation is provided by using an example in which a mobile phone initiates a video recording function to record video pictures and audio.
[0150] Generally, after the mobile phone launches the camera, the shooting function is launched by default. After an operation to switch the function is detected, for example, after an operation to tap the video record control is detected, the video record function is started and the video record preview interface is displayed. For example, after launching the camera, the mobile phone displays the shooting preview interface 402 shown in FIG. 4(b) by default. After detecting an operation to tap the control 433 by the user, the mobile phone starts the video record function and displays the video record preview interface 403 shown in FIG. 4(c). Alternatively, in other examples, the mobile phone may alternatively start the video record function by default after starting the camera. For example, after launching the camera, the mobile phone directly displays the video record preview interface 403 shown in FIG. 4(c). That is, after detecting an operation to open the camera application by the user, the mobile phone may start the video record function. In still other examples, the mobile phone starts the video record function by detecting an air gesture, a voice instruction operation, etc. For example, if the mobile phone receives a voice command from the user "open the camera and perform video recording", the mobile phone directly starts the video record function of the camera and displays the video record preview interface. In yet another example, in another possible implementation, after activating the camera, the mobile phone defaults to the function last applied before the camera was last closed, for example, the portrait shooting function.The mobile phone then starts the video recording function of the camera by detecting an operation to start the video recording function, and displays a video recording preview interface.
[0151] In some embodiments, after the mobile phone detects a switch to the video recording function, the user is first asked whether to enable the voice enhancement mode. After the user decides to enable the voice enhancement mode, the voice enhancement mode is enabled. Alternatively, the voice enhancement mode is automatically enabled after the mobile phone detects a switch to the video recording function. In some other embodiments, after the mobile phone detects a switch to the video recording function, the video recording preview interface is first displayed, and then, after the user detects an operation of providing a shooting instruction, the voice enhancement mode is enabled based on a user instruction or the voice enhancement mode is automatically enabled.
[0152] For example, in response to a tapping operation on the video record control 433 as shown in Fig. 4(b), the mobile phone displays the video record preview interface 403 shown in Fig. 4(c) and displays a prompt box 44 on the video record preview interface 403 to prompt the user whether to enable the voice enhancement mode. If the mobile phone detects the user tapping "Yes", the mobile phone enables the voice enhancement mode and displays the shooting interface 404 shown in Fig. 4(d). Alternatively, after switching from the shooting preview interface 402 to the video record function, the mobile phone directly enables the voice enhancement mode and displays the shooting interface 404 shown in Fig. 4(d).
[0153] In another example, after switching to the video recording function, the mobile phone only displays the video recording preview interface 403 shown in Fig. 4(c). Then, in response to the user tapping the capture control 45, the mobile phone further displays the prompt box 44 and determines whether to enable the voice enhancement mode based on the user's selection. Alternatively, after detecting the user tapping the capture control 45 on the video recording preview interface 403, the mobile phone directly enables the voice enhancement mode and displays the capture interface 404 shown in Fig. 4(d).
[0154] In some other embodiments, the mobile phone enables or disables the voice enhancement mode after detecting a user's operation to enable or disable the voice enhancement mode on the video recording preview interface or in the process of recording a video picture. The operation to enable the voice enhancement mode may include, for example, tapping a preset control or a voice operation.
[0155] For example, on the video recording preview interface 403 shown in FIG. 4(c), the mobile phone can enable or disable the voice enhancement mode by detecting an operation performed by a user on the control 46. For example, the current display status of the control 46 indicates that the mobile phone does not currently enable the voice enhancement mode, and the mobile phone enables the voice enhancement mode after detecting a user tapping operation on the control 46. Before the start of filming or during the filming process, the mobile phone can enable or disable the voice enhancement mode by detecting a user tapping operation on the control 46.
[0156] After the voice enhancement mode is enabled, the mobile phone may start recording video pictures after detecting a user's operation to give a shooting instruction, and may perform processing such as video encoding on the collected video pictures to generate and store a video file.
[0157] For example, in response to a user tapping the capture control 45 on the video recording preview interface 403 shown in FIG. 4(c), the mobile phone displays the capture interface 404 shown in FIG. 4(d) and begins recording a video picture.
[0158] The audio enhancement mode is used to enhance the audio collection of some specific objects in the video picture of the captured video, thereby improving the audio recording effect. For example, when a user performs video recording by using a camera in an interview process, the user needs to focus on collecting the voice of the interviewed person. The operation of giving a shooting instruction by the user may include multiple operation methods, such as, for example, an operation of tapping a shooting control and an operation of voice instruction.
[0159] For example, as shown in FIG. 5(a), a large circle 501 is used to represent the maximum range (which may be described as a panoramic sound collection range) that all current microphones of a mobile phone can perform sound collection, and a small circle 502 is used to represent a sound collection range corresponding to a person of interest to the user (usually a person making a sound). Also, for example, as shown in FIG. 5(b), the sound collection range of the person of interest to the user (i.e., sound collection range 1) is within the panoramic sound collection range. In this embodiment of the present application, the sound collection range where the audio recording needs to be enhanced may be determined based on the position information of the image of the person of interest to the user in the recorded video picture. That is, the audio recording effect in the sound collection range 1 shown in FIG. 5(b) is improved. Thus, in the recorded audio, the impact of other noises in the panoramic audio on the sound made by the person of interest to the user is reduced.
[0160] In some embodiments, the face image recognized by the mobile phone and making a sound may be described as the first face image, and the mouth image making a sound may be described as the first mouth image, or may be described as a pronunciation face image or a pronunciation mouth image. There may be one or more people making a sound in the video picture. Therefore, there may be one or more first face images and one or more first mouth images. It may be understood that if some people are making a sound in the currently taken video picture, but the mobile phone cannot recognize that the people are making a sound, the face image or mouth image of the unrecognized person making a sound is not classified as the first face image or the first mouth image.
[0161] Therefore, after the mobile phone enables the voice enhancement mode and starts recording video pictures, the mobile phone needs to recognize the first face image or the first mouth image, and based on the first face image or the first mouth image, determine the first sound pickup range in which the voice recording effect needs to be enhanced to obtain a better voice recording effect.
[0162] For example, after determining the first pickup range, the mobile phone calls a microphone corresponding to the first pickup range to emphasize an audio signal within the first pickup range. In some scenarios, the mobile phone includes one or more microphones, and the one or more microphones are configured to collect a first audio (i.e., an initial audio signal). When the pickup range of a first microphone of the one or more microphones includes a part or all of the first pickup range, an audio signal within the first pickup range and within the pickup range of the first microphone is emphasized, an audio signal outside the first pickup range and within the pickup range of the first microphone is attenuated, and / or an audio signal of a microphone other than the first microphone of the one or more microphones is attenuated to obtain a second audio (i.e., an audio corresponding to the first face image or the first mouth image). In some other scenarios, the mobile phone includes at least two microphones, and the at least two microphones are configured to collect the first audio. When the pickup range of the second microphone of the at least two microphones does not include the first pickup range, the second microphone is disabled and the sound collected by the microphone other than the second microphone of the at least two microphones is the second audio. Alternatively, when the second microphone is disabled, audio signals that are within the first pickup range and within the pickup range of the microphone other than the second microphone of the at least two microphones are emphasized and / or audio signals that are outside the first pickup range and within the pickup range of the microphone other than the second microphone of the at least two microphones are attenuated.
[0163] For example, microphone 1 and microphone 2 are configured on a mobile phone. When the first sound collection range is within the sound collection range of microphone 1, after acquiring the initial audio signal using microphone 1 and microphone 2, the mobile phone may enhance the audio signal that is within the initial audio signal and within the first sound collection range and collected by microphone 1, attenuate the audio signal that is within the initial audio signal and outside the first sound collection range and collected by microphone 1, and attenuate the audio signal that is collected by microphone 2 to acquire the audio corresponding to the first face image or the first mouth image. Alternatively, the mobile phone may disable microphone 2, enhance the audio signal within the first sound collection range in the audio signal collected by microphone 1, and attenuate the audio signal outside the first sound collection range in the audio signal collected by microphone 1 to acquire the audio corresponding to the first face image or the first mouth image. In another example, microphone 1 and microphone 2 are configured on a mobile phone. The first sound collection range includes sound collection range 1, which is the sound collection range of microphone 1, and sound collection range 2, which is the sound collection range of microphone 2. That is, the first sound collection range is the union of sound collection range 1 and sound collection range 2. In this case, after obtaining the initial audio signal by using microphone 1 and microphone 2, the mobile phone can emphasize the audio signal that is in the sound collection range 1 of microphone 1 and is in the initial audio signal and the audio signal that is in the sound collection range 2 of microphone 2 and is in the initial audio signal, and weaken the remaining audio signal in the initial audio signal, so as to obtain the audio corresponding to the first face image or the first mouth image. It can be understood that the sound collection range 1 and the sound collection range 2 may be completely overlapped or partially overlapped.
[0164] For example, a shooting interface 404 is shown in FIG. 4(d), which includes a viewfinder frame 48 used to display a video picture. The sound pickup range corresponding to the viewfinder frame 48 is the maximum sound pickup range of the currently recorded video picture. In the currently recorded video picture, the mobile phone recognizes a first face image 47, and it is assumed that the first face image is located at the center position of the viewfinder frame 48. In this case, the mobile phone determines that the first sound pickup range is the center position of the maximum sound pickup range. The mobile phone emphasizes the audio signal of the first sound pickup range. Optionally, a prompt box 49 is displayed on the shooting interface 404, which is used to prompt the user that the sound recording effect of the middle position is currently enhanced. The prompt box 49 may be displayed continuously in the shooting process, and the displayed content changes with the change of the first sound pickup range. The prompt box 49 is automatically hidden after the shooting is stopped. Alternatively, the prompt box 49 is displayed only for a preset period of time and disappears automatically after the preset period to avoid obstructing the video picture displayed in the viewfinder frame 48 .
[0165] It can be found that in the process of recording audio, the mobile phone can obtain the audio corresponding to the pronunciation face or the pronunciation mouth by emphasizing the audio signal within the first sound collection range, thereby enhancing the sound receiving effect for the pronunciation face or the pronunciation mouth, thereby reducing the interference of external noise. Furthermore, based on emphasizing the audio signal within the first sound collection range, the audio signal outside the first sound collection range can be further weakened to obtain a better voice recording effect. Or, only the audio signal outside the first sound collection range is weakened to reduce the interference of external noise.
[0166] Fig. 6 is a schematic flowchart of an audio processing method according to an embodiment of the present application. Hereinafter, steps S601 to S604 shown in Fig. 6 will be used to explain in detail the process of the mobile phone recognizing the first face image or the first mouth image, determining the first sound collection range in which voice enhancement is required, and acquiring the audio corresponding to the first sound collection range, which are described in Fig. 4(a) to Fig. 4(d).
[0167] S601: The mobile phone recognizes a first face image or a first mouth image.
[0168] Optionally, the mobile phone may recognize the first face image or the first mouth image by using a face image recognition algorithm. For example, in the process of recording a video picture by the mobile phone, the face image recognition algorithm is used to determine whether the collected video picture contains a face image. If a face image is included, the face image included in the collected video picture is recognized, and whether the face image is making a sound is determined based on the change status of the face feature data, such as the five sense organ data and face contour data, of the face image in a preset period. The criterion for determining that the face image is making a sound includes the mobile phone determining that the face image is currently making a sound. Alternatively, if the mobile phone determines that the face image is making a sound again within a preset period after the mobile phone determines that the face image is making a sound for the first time, the mobile phone determines that the face image is making a sound. It may be understood that the human vocal organ is the human mouth. Also, if the data of the vocal mouth is acquired, the data of the first mouth image may be preferentially determined, and the first sound collection range may then be determined based on the data of the first mouth image.
[0169] For example, on the interface 701 shown in FIG. 7, the mobile phone may collect a face image 71, and by using a face image recognition algorithm, recognize face feature key points corresponding to the face image 71 (e.g., circular feature points displayed on the face image 71 to determine whether the face image 71 is making a sound), and obtain face data and / or mouth data. For example, the face feature points may include an upper lip feature point and a lower lip feature point, and the distance between the upper lip and the lower lip may be obtained in real time based on the upper lip feature point and the lower lip feature point. Thus, a distance threshold between the upper lip and the lower lip of the face image is preset. If the number of times that the mobile phone detects that the distance between the upper lip and the lower lip of the face image exceeds the distance threshold exceeds a preset number of times in a preset period after the mobile phone first detects that the distance between the upper lip and the lower lip of the face image exceeds the distance threshold, the mobile phone determines that the current face image is making a sound.
[0170] Also, the facial feature points may further include facial contour feature points. In this case, the mobile phone can obtain data of jaw changes, data of facial muscle changes, etc. based on the facial contour feature points to determine whether the facial image is making a sound. For example, if the number of times that the data generated by the jaw moving up and down exceeds a preset threshold in a preset period, it is determined that the current facial image is making a sound. Of course, the mobile phone may alternatively determine the pronunciation face or the pronunciation mouth based on changes in other data corresponding to the mouth, such as laryngeal prominence change data. In addition, the mobile phone can further implement more accurate recognition of the first facial image or the first mouth image with reference to the facial data and the mouth data.
[0171] Please note that for the face image recognition algorithm, please refer to the face image recognition algorithm in the prior art. The face recognition algorithm and the calculation process of the face recognition algorithm are not described in detail in this embodiment of the present application.
[0172] There are one or more first face images. In a scenario where there are multiple first face images, i.e., multiple face images make sounds simultaneously or multiple face images make sounds consecutively within a first preset period, the mobile phone can filter out face images with relatively small face image areas or face images at the edge of the video picture, and do not consider the face images as first face images. Generally, in the process of recording a video picture, a user points the camera at a person of interest to the user. Therefore, the face images of interest to the user should be face images with relatively large areas or face images displayed at or near the center of the video picture. That is, the sound pickup range of interest to the user is generally the sound within the picture range of interest to the user, and the voice enhancement needs to be performed within the picture range. The first preset period may be a preconfigured short time range. For example, the mobile phone determines that user A is making a sound, starts timing when user A stops making a sound, and detects that user B starts making a sound in the first preset period. Furthermore, the mobile phone detects that user A starts making a sound again in a first preset period after user B stops making a sound. That is, in the video recording process, if user B makes a sound immediately after user A makes a sound, or if user A and user B make sounds alternately, the face images corresponding to user A and user B may be determined as the first face image. Therefore, it may be avoided to frequently determine the sound pickup range corresponding to the first face image in a short time range, thereby reducing the amount of data processing and improving efficiency.
[0173] In this case, after recognizing the multiple pronunciation face images, the mobile phone determines the face image with the largest area or the face image closest to the center of the video picture, and determines the face image and the pronunciation face image whose area difference with the face image is less than a preset threshold as the first face image, or determines the face image and the pronunciation face image within a preset range near the face image as the first face image, and determines the first sound collection range based on the first face image. Similarly, the scenario in which the mobile phone determines the multiple first mouth images is the same as the scenario in which the mobile phone determines the multiple first face images, and will not be described in detail. The center point of the video picture includes, for example, the center point of the viewfinder frame and the center point of the display screen of the mobile phone.
[0174] S602: The mobile phone obtains a first feature value corresponding to a first face image or a first mouth image.
[0175] S603: The mobile phone determines a first sound collection range based on the first characteristic value.
[0176] The first feature value is used to describe a relative positional relationship between the face of a real person corresponding to the first face image and the mobile phone, or the first feature value is used to describe a relative positional relationship between the mouth of a real person corresponding to the first mouth image and the mobile phone. Therefore, the mobile phone can determine a first sound collection range based on the first feature value. For example, when a real person corresponding to the first face image is located directly in front of the mobile phone, that is, when the first face image is located at a center position of the captured video picture, the first sound collection range is a directly in front sound collection range of the mobile phone. Subsequently, after acquiring an initial audio signal including audio signals in various directions, the mobile phone can acquire audio corresponding to the first face image based on the initial audio signal and the first sound collection range. The first feature value includes one or more of a front / back attribute parameter, an area ratio, or position information. The front / back attribute parameter, an area ratio, and position information are parameters determined by the mobile phone based on the first face image or the first mouth image. Please refer to the following description for the meaning of the parameters.
[0177] The following describes a specific method for determining the first sound collection range by a mobile phone when the first feature value includes different parameters.
[0178] Solution 1: The first feature value includes a front / back attribute parameter of the first face image, or the first feature value includes a front / back attribute parameter corresponding to the first mouth image.
[0179] The "front / back attribute parameter" is used to indicate whether the video picture including the first face image or the first mouth image is a video picture taken by a front camera (for ease of explanation, also referred to as a front video picture in this specification) or a video picture taken by a rear camera (for ease of explanation, also referred to as a rear video picture in this specification). The front / back attribute parameter can be used to determine whether the first sound pickup range is in a 180-degree front range or a 180-degree rear range of the mobile phone. For example, as shown in FIG. 2B(b), the sound pickup range corresponding to the front video picture may include the range represented by ellipse 204, ellipse 205, and ellipse 206, and the sound pickup range corresponding to the rear video picture may include the range represented by ellipse 201, ellipse 202, and ellipse 203.
[0180] For example, the video picture displayed in the viewfinder frame of the mobile phone may be switched between the picture collected by the front camera and the picture collected by the rear camera. In the photographing interface 801 shown in Fig. 8(a), the mobile phone is in the voice enhancement mode and determines that the pronunciation face image 81 exists. If the mobile phone determines that the video picture in which the pronunciation face image 81 is located is the video picture collected by the front camera, that is, if the first feature value is the front attribute parameter, the mobile phone determines that the first sound pickup range is within the front range of 180°, and displays a prompt box 82 to prompt the user that the front voice recording effect is now enhanced.
[0181] Further, the photographing interface 801 further includes a front / back switching control 83 used to switch between the front camera and the rear camera. For example, the mobile phone can switch from the front camera to the rear camera in response to the user tapping the front / back switching control 83. Thus, the video picture displayed by the mobile phone is switched from the video picture collected by the front camera displayed on the photographing interface 801 shown in FIG. 8(a) to the video picture collected by the rear camera displayed on the photographing interface 802 shown in FIG. 8(b). If the mobile phone recognizes the pronunciation face image 84 in the current video picture, the mobile phone determines that the first feature value is rear attribute parameter information, and determines that the first sound pickup range is within the 180° rear range of the mobile phone. The mobile phone displays a prompt box 85 to prompt the user that the rear voice recording effect is currently enhanced.
[0182] As shown in Fig. 2B(b), the sound collection range corresponding to the rear video picture is the range represented by ellipse 201, ellipse 202, and ellipse 203, and the sound collection range corresponding to the front video picture is the range represented by ellipse 204, ellipse 205, and ellipse 206. For example, if the mobile phone determines that the first face image corresponds to the rear video picture based on the first feature value, the mobile phone determines that the first sound collection range is the range represented by ellipse 201, ellipse 202, and ellipse 203. Alternatively, referring to Fig. 2B(d), if the mobile phone determines that the first face image corresponds to the rear video picture based on the first feature value, the mobile phone determines that the first sound collection range is the sound collection range corresponding to microphone 27, microphone 28, and microphone 29.
[0183] Solution 2: The first feature value includes an area ratio corresponding to the first face image, or the first feature value includes an area ratio corresponding to the first mouth image.
[0184] "Area ratio" is used to indicate the ratio of the area of the first face image or the first mouth image to the area of the video picture. The area ratio is used to measure the radial range (or diameter range) over which the microphone collects audio.
[0185] Specifically, in the process of a user using a mobile phone to take a video picture, the person of the user's interest is usually located at the center position of the video picture, that is, the first face image or the first mouth image is located at the center position of the viewfinder frame. Different regions of the first face image or the first mouth image correspond to different sound pickup ranges. For example, as shown in FIG. 9, it is assumed that the mobile phone determines two first face images in different periods, which are first face image 1 and first face image 2, respectively. The areas of the two face images are different, and the area of the first face image 1 is larger than the area of the first face image 2. In this case, as shown in FIG. 9, the sound pickup range determined based on the first face image 1 is the sound pickup range 1, and the sound pickup range determined based on the first face image 2 is the sound pickup range 2. The sound pickup range 1 is larger than the sound pickup range 2.
[0186] In some embodiments, X is used to represent the area of the first face image or the area of the first mouth image, Y is used to represent the area of the video picture displayed within the viewfinder frame, and N represents the sound pickup range corresponding to the framing range, as shown in Table 1 below.
[0187] [Table 1]
[0188] In some embodiments, the area ratio is used to indicate the ratio of the area of the first facial image to the area of the video picture displayed in the viewfinder frame. There may be one or more first facial images. Thus, the area of the first facial image is the area of one facial image or the sum of the areas of multiple facial images. The sum of the areas of multiple facial images can be represented by using the area of the placeholder frame in which the multiple facial images are located, i.e., the area of the smallest selected frame that contains the multiple facial images.
[0189] For example, in the interface 1001 shown in Fig. 10(a), the number of the first face image is 1. In the process of performing face image recognition by a mobile phone, a dotted box 101 for framing the face area of the first face image 11 is determined based on the position of the feature point of the upper part of the forehead, the position of the feature point of the lower part of the chin, and the position of the feature points on the extreme edge of the left and right faces not including the ears in the face feature points of the face image 11, and the image area within the framed range is the area of the first face image. Image In the process of determining the area, only the face area is calculated, and the influence of ears, hats, jewelry, neck, etc. is excluded. The area of the video picture displayed in the viewfinder frame is the image area within the framed range of the dotted box 102. Therefore, the mobile phone may determine the area ratio based on the area ratio corresponding to the recognized dotted box 101 and the dotted box 102. Subsequently, for the method for determining the area of the first face image, please refer to the current method for determining the area of the first face image. Details will not be described again.
[0190] As another example, on the interface 1002 shown in FIG. 10(b), two face images are displayed on the interface 1002, and both of the two face images are recognized by the mobile phone as the first face image to be pronounced. The area of the right face image 12 is the image area within the framed range of the dotted box 103, and the area of the left face image 13 is the image area within the framed range of the dotted box 104. In this case, the area of the first face image is the image area within the framed range of the dotted box 105, i.e., the area of the smallest selection frame that includes all face images (e.g., the total frame area determined based on the edge limit values of the area selection frames of all face images). The dotted box 105 is used to represent a placeholder frame in which the face images 12 and 13 are located. The final determined area of the first face image includes the image areas corresponding to the two face images. The area of the video picture displayed in the viewfinder frame is the image area within the framed range of the dotted box 106. Therefore, the mobile phone may determine the area ratio based on the area ratio corresponding to the recognized dotted box 105 and dotted box 106 .
[0191] In a scenario where multiple faces emit sounds, on the interface 1003 shown in FIG. 10(c), when two people in the video picture are both emitting sounds, the mobile phone determines that the face image 14 on the right side has the largest area. The mobile phone may filter out some of the sounding face images that the user is not interested in by using a preset threshold. For example, the preset threshold is less than 20% of the largest face image area. For example, on the interface 1003, the mobile phone may filter out face images 15 that are on the left side and whose area is less than 20% of the area of the face image 14 on the right side. In this case, the first face image includes the face image 14 on the right side. In another example, the preset threshold is that the distance from the face image with the largest area is more than 35% of the length or width of the video picture displayed in the viewfinder frame. For example, on the interface 1003, the mobile phone may filter out face images 15 that are on the left side and whose distance from the face image 14 on the right side is more than 35% of the length of the video picture displayed in the viewfinder frame. In this case, the first face image includes the right face image 14 .
[0192] In some other embodiments, the area ratio is used to indicate the ratio of the area of the first mouth image to the area of the video picture displayed in the viewfinder frame. There may be one or more first mouth images. Thus, the area of the first mouth image is the area of one mouth image or the sum of the areas corresponding to the multiple mouth images. The sum of the areas of the multiple mouth images may be represented by using the area of the placeholder frame in which the multiple mouth images are located, i.e., the area of the smallest selected frame that contains the multiple mouth images.
[0193] For example, in the interface 1101 shown in FIG. 11(a), the number of the first mouth image is 1. In the process of performing face image recognition by a mobile phone, a dotted box 111 for framing the area of the first mouth image 16 is set to the feature points of the mouth image in the facial feature points. top end , lower endThe area of the video picture displayed in the viewfinder frame is the image area within the framed range of the dotted box 112. Therefore, the mobile phone may determine the area ratio based on the area ratio corresponding to the recognized dotted box 111 and the dotted box 112. Next, the method for determining the area of the first mouth image will be described in detail below. mouth Please refer to the current method for determining the region of an image, the details will not be explained again.
[0194] As another example, in the interface 1102 shown in FIG. 11(b), two mouth images are displayed on the interface 1102, and both of the two mouth images are The first The area of the first mouth image 17 on the right side is the image area within the framed range of the dotted box 113, and the area of the first mouth image 18 on the left side is the image area within the framed range of the dotted box 114. In this case, the area of the first mouth image is the image area within the framed range of the dotted box 115, i.e., the area of the smallest selected frame that includes all the mouth images (e.g., the total frame area determined based on the edge limits of the area selected frames of all the mouth images). The dotted box 115 is used to represent a placeholder frame in which the first mouth image 17 and the first mouth image 18 are located. The final determined area of the first mouth image includes the image areas corresponding to the two mouth images. The area of the video picture displayed in the viewfinder frame is the image area within the framed range of the dotted box 116. Therefore, the mobile phone may determine the area ratio based on the area ratio corresponding to the recognized dotted box 115 and the dotted box 116.
[0195] Similarly, in a scenario where multiple faces emit sounds, on the interface 1103 shown in FIG. 11(c), if two people in the current video picture are both emitting sounds, the mobile phone determines that the mouth image on the right side has the largest area. The mobile phone may filter out some of the speaking mouth images that the user is not interested in by using a preset threshold. For example, the preset threshold is less than 20% of the largest mouth image area. In another example, the preset threshold is that the distance from the mouth image with the largest area is more than 35% of the length or width of the video picture displayed in the viewfinder frame. In the interface 1103 shown in FIG. 11(c), the speaking mouth image on the left side is filtered out, and the first mouth image includes only the first mouth image on the right side that emits sounds, and the first mouth image on the right side is filtered out. image A radius of the first sound collection range is determined based on the area of the first sound collection range.
[0196] For example, in a scenario in which the region of the first face image is determined, it is assumed that a mobile phone collects video pictures by using a rear camera. The sound collection range determined by the mobile phone based on the first feature value of the first face image shown in Fig. 10(a) may be the sound collection range 2 shown in Fig. 9. The sound collection range determined by the mobile phone based on the first feature value of the first face image shown in Fig. 10(b) may be the sound collection range 1 shown in Fig. 9.
[0197] It should be noted that in the process of determining the area of the first face image and the area of the first mouth image, both the first face image and the first mouth image are converted into a rectangle, and then the area of the rectangle is used as the area of the corresponding first face image or the area of the corresponding first mouth image. It can be understood that an irregular geometric shape can alternatively be used to correspond to the first face image and the first mouth image in order to more accurately determine the corresponding area. The rectangle in this embodiment of the present application is only an example for explanation. This is not particularly limited in this embodiment of the present application.
[0198] It should be noted that in the process of determining the area ratio of the first face image and the area ratio of the first mouth image, the area of the viewfinder frame is used as the area of the video picture. It can be understood that if the mobile phone is a full-screen mobile phone, the area of the display of the mobile phone can be used as the area of the video picture. In addition, other areas and areas of other shapes may be used as the area of the video picture. The area of the viewfinder frame in this embodiment of the present application is only an example for explanation. It is not particularly limited in this embodiment of the present application.
[0199] Solution 3: The first feature value includes position information corresponding to the first face image, or the first feature value includes position information corresponding to the first mouth image.
[0200] "Position information" is used to indicate the position of the first face image or the first mouth image in the video picture. The position information includes an offset of the center point of the first face image relative to a first reference point, such as an offset direction, an offset angle, and / or an offset distance. Alternatively, the position information includes an offset of the center point of the first mouth image relative to a first reference point. The first reference point is the center point or the focal point of the video picture. The offset direction means that the center point of the first face image or the first mouth image has a left offset, a right offset, an upper offset, a lower offset, an upper left offset, a right upper offset, a lower left offset, a lower right offset, etc., relative to the first reference point. The offset angle is an angle at which the upper left offset, the right upper offset, the lower left offset, or the lower right offset exists. The offset distance is, for example, the distance of the left offset, the right offset, the upper offset, the lower offset, or the distance of the offset at a particular offset angle.
[0201] In some embodiments, the coordinates of the center point of the first face image may be determined based on the limit positions of the feature points in various directions of the first face image. For example, in the process of determining the area of the first face image, the coordinates of the center point of the first face image are determined based on the position of the feature point of the upper part of the forehead, the position of the feature point of the lower part of the chin, and the position of the feature points on the extreme edges of the left and right faces not including the ears in the facial feature points of the first face image. Similarly, among the feature points of the mouth image in the facial feature points of the face image, top end , lower end Based on the positions of the feature points at the left and right ends, the coordinates of the center point of the first mouth image are determined.
[0202] Then, the first reference point is preset and may include, for example, the center point of the video picture displayed in the viewfinder frame (or may be described as the center point of framing), or the focal point in the framing range. The coordinate system is constructed by using the first reference point as the origin, the direction parallel to the bottom edge of the mobile phone (or the bottom edge of the current viewfinder frame) as the x-axis, and the direction perpendicular to the x-axis as the y-axis, and the current coordinate system is parallel to the display of the mobile phone. The offset direction, offset angle, and offset distance of the center point of the first face image or the first mouth image relative to the first reference point are defined by using the constructed coordinate system. For example, as shown in FIG. 13(a), the case of the coordinate system when the mobile phone is displayed in portrait orientation is shown, and the x-axis is parallel to the bottom side (i.e., the short side) of the mobile phone. As shown in FIG. 13(b), the case of the coordinate system when the mobile phone is displayed in landscape orientation is shown, and the x-axis is parallel to the side side (i.e., the long side) of the mobile phone. The coordinate of the intersection of the x-axis and the y-axis, i.e., the origin, is (0,0). The positive direction of the x-axis is to the right, and the positive direction of the y-axis is up. It can be seen that after the mobile phone switches between vertical display and horizontal display, the directions of the x-axis and y-axis of the coordinate system change, and the offset direction, offset angle, and offset distance of the center point of the first face image or the first mouth image relative to the first reference point change accordingly.
[0203] For example, on the interface 1201 shown in FIG. 12(a), the number of the first face image is 1, the center point of the first face image is a position corresponding to the mark 121, and the center point of the video picture displayed in the viewfinder frame is a position corresponding to the mark 122. The position of the center point of the viewfinder frame is determined based on the limit coordinates of the top, bottom, left and right ends of the viewfinder frame. The mobile phone determines the position information of the first face image based on the positional relationship between the mark 121 and the mark 122. For example, in a scenario displayed on the interface 1201, the position information of the first face image is the bottom left of the center point of the viewfinder frame. Alternatively, on the interface 1202 shown in FIG. 12(b), the number of the first face image is 1, the center point of the first mouth image is a position corresponding to the mark 123, and the center point of the video picture displayed in the viewfinder frame is a position corresponding to the mark 124. The mobile phone determines the position information of the first face image based on the positional relationship between the mark 123 and the mark 124. mouth Determine the position information of the images. For example, in the scenario displayed on the interface 1202, the position information of the first mouth image is the bottom left of the center point of the viewfinder frame.
[0204] In some embodiments, when there are multiple first face images, the center point of the first face image is the center point within the image range that includes the multiple face images. For example, in the scenario shown in FIG. 10(b), the center point of the first face image is the geometric center point within the framed range of the dotted box 105. As another example, in the scenario shown in FIG. 11(b), the center point of the first mouth image is the geometric center point within the framed range of the dotted box 115. Similarly, the center point of the video picture displayed in the viewfinder frame is also the geometric center point of the viewfinder frame.
[0205] It should be noted that in the process of determining the center point of the first face image and the center point of the first mouth image, both the first face image and the first mouth image are converted into a rectangle, and then the center point of the rectangle is used as the center point of the corresponding first face image or the center point of the corresponding first mouth image. It can be understood that an irregular geometric shape can alternatively be used to correspond to the first face image and the first mouth image in order to more accurately determine the corresponding center point. The rectangle in this embodiment of the present application is only an example for explanation. This is not particularly limited in this embodiment of the present application.
[0206] In addition, in the process of determining the position information corresponding to the first face image or the first mouth image, in some scenarios, the center point of the viewfinder frame is used as the first reference point, i.e., the center point of the viewfinder frame is used to represent the center point of the video picture. It can be understood that the first reference point may alternatively be represented in another format based on the display format of the video picture. For example, the center point of the entire screen of the display of the mobile phone is used to represent the center point of the video picture, i.e., as the first reference point. In this embodiment of the present application, the center point of the viewfinder frame is used as the first reference point only as an example for explanation. This is not particularly limited in this embodiment of the present application.
[0207] In some scenarios, in the process of recording a video picture, a user may not place an object of interest at a central position within a framing range, but select a relatively interesting object through a focus method. The mobile phone can obtain the user's intention by detecting the focus position of the focus and determine the object in which the user is interested. The focus position of the focus may alternatively be a focus position obtained by the mobile phone through autofocus. For example, the mobile phone automatically recognizes an image and determines the corresponding focus position after autofocus.
[0208] For example, in the interface 1203 shown in FIG. 12(c), in the current scenario, the number of first face images is 2, and the center point of the first face image is a position corresponding to the mark 125. The mobile phone detects an operation of the user tapping the screen, obtains the focus position of the focus, and displays the dotted box 126. The range surrounded by the dotted box 126 is the focus range determined by the mobile phone based on the user's intention. In this case, the central focus of the focus range is a position corresponding to the mark 127. The mobile phone determines the position information of the first face image based on the positional relationship between the mark 125 and the mark 127. For example, the position information of the first face image is the upper left of the focus center.
[0209] In a possible implementation form, the mobile phone can determine a relative positional relationship between the first facial image or the first mouth image and the first reference point based on the coordinates of the center point of the first facial image or the coordinates of the center point of the first mouth image and the coordinates of the first reference point, and then determine an offset direction of the first facial image or the first mouth image in the video picture displayed in the viewfinder frame.
[0210] For example, referring to the coordinate system shown in FIG. 13(a) or (b), it is assumed that the coordinates of the center point of the first face image or the coordinates of the center point of the first mouth image are (X1, Y1), the coordinates of the first reference point are (X2, Y2), and the first reference point is set at the origin (0, 0) of the coordinate system. Please refer to Table 2 below for the relative positional relationship between the first face image or the first mouth image and the first reference point. For example, X1< X2 If X1=X2 and Y1=Y2, it indicates that the first face image or the first mouth image is located to the left of the first reference point, i.e., the offset direction is left. As another example, if X1=X2 and Y1=Y2, it indicates that both the left-right offset and the up-down offset of the center point of the first face image or the first mouth image from the first reference point are zero. Or the first mouth image The center point of the first criterion Dots and They match and the offset direction is no offset.
[0211] [Table 2]
[0212] In another possible implementation form, the mobile phone may determine an offset angle (the included angle θ between the x-axis and the connecting line between the coordinates (X1, Y1) of the center point of the first face image or the center point of the first mouth image and the first reference point (X2, Y2) as shown in FIG. 14) of the first face image in the video picture displayed in the viewfinder frame based on the coordinates of the center point of the first face image or the coordinates of the center point of the first mouth image and the coordinates of the first reference point. For example, as shown in FIG. 14, a great circle 141 is used to represent the maximum sound collection range corresponding to the viewfinder frame of the mobile phone, and the coordinates of the center point of the viewfinder frame are set to (0, 0), that is, the center point of the viewfinder frame is set to the first reference point. The maximum sound collection range is divided into four quadrants, for example, a first quadrant 142, a second quadrant 143, a third quadrant 144, and a fourth quadrant 145. The offset angle is assumed to be θ, and the mobile phone may determine the offset angle θ based on the value of the included angle between the x-axis and the connecting line between (X1, Y1) and (X2, Y2) in each quadrant, where 0<θ<90°. Alternatively, the mobile phone determines the offset angle θ based on all quadrants, where 0<θ<360°. For example, in FIG. 14, the first face image is displayed in the second quadrant 143 of the viewfinder frame, and tan θ=|Y2-Y1| / |X2-X1|, so that the mobile phone can obtain the offset angle θ of the first face image in the video picture displayed in the viewfinder frame.
[0213] In yet another possible implementation form, the mobile phone can determine an offset distance of the first face image in the video picture displayed in the viewfinder frame based on the coordinates of the center point of the first face image or the coordinates of the center point of the first mouth image and the coordinates of the first reference point. The mobile phone can determine whether the sound pickup range corresponding to the first face image exceeds the sound pickup range corresponding to the framing range based on the offset distance and the radius of the sound pickup range corresponding to the first face image, and then determine the first sound pickup range.
[0214] For example, as shown in Fig. 15(a), a large circle 151 is the maximum sound collection range corresponding to the viewfinder frame, and has a radius of R. The first reference point is the center point of the video picture displayed in the viewfinder frame, i.e., the center point of the maximum sound collection range, and the coordinates of the first reference point are (X2, Y2). The coordinates of the center point of the first face image are (X1, Y1). Area ratio Parameter Feelings The radius of the small circle 152 determined by the mobile phone based on the information is r. The mobile phone determines the offset distance based on the Pythagorean theorem.
number
[0215] It should be noted that in the process of determining the center point of the first face image or the first mouth image, both the first face image and the first mouth image are converted into a rectangle, and then the geometric center point of the rectangle is used as the center point of the corresponding first face image or the corresponding first mouth image. It can be understood that an irregular geometric shape can alternatively be used to correspond to the first face image and the first mouth image in order to more accurately determine the position of the corresponding center point. The rectangle in this embodiment of the present application is only an example for explanation. This is not particularly limited in this embodiment of the present application.
[0216] In some embodiments, the mobile phone may determine the first pickup range by using any one of the above solutions 1 to 3. Alternatively, the mobile phone may determine the first pickup range by combining a plurality of solutions in the above solutions 1 to 3. Alternatively, the mobile phone may determine the first pickup range by combining one or more parameters in the above solutions 1 to 3 with another parameter. Also, the mobile phone may determine the first pickup range using another parameter.
[0217] For example, the following describes a method for determining a first sound collection range by a mobile phone by combining the above-mentioned solutions 1 to 3.
[0218] For example, as shown in Fig. 16A(a), it is assumed that the user currently selects to record a video picture by using the rear camera. In this case, the mobile phone determines that the video picture corresponding to the first face image is a rear video picture based on the front / back attribute parameter of the video picture corresponding to the first face image. As shown in Fig. 16A(b), the first sound pickup range is the range of 180° behind the mobile phone, that is, the range represented by ellipse 161, ellipse 162, and ellipse 163.
[0219] Then, the mobile phone can further determine the first sound collection range based on the position information corresponding to the first face image. For example, as shown in FIG. 16B(a), the first face image is a left face image, and the center point 164 of the first face image is located at the upper left of the center point 165 of the viewfinder frame. The mobile phone determines, based on the position information, that the offset direction is the upper left, and the center point of the first sound collection range is located at the upper left of the center point of the rear sound collection range. For example, for the first sound collection range, please refer to the left side of the range represented by the ellipse 161 and the ellipse 162 shown in FIG. 16B(b). As shown in FIG. 16B(c), the large circle 166 is the largest sound collection range corresponding to the rear video picture, and the sound collection range is divided into left and right along the central dotted line, so that the corresponding left and right sound collection ranges can be determined. For example, for the first sound collection range on the upper left of the back surface, please refer to the range represented by the left semi-ellipse 1611 and the left semi-ellipse 1621 shown in FIG. 16B(c).
[0220] Based on this, it is assumed that the position information further includes an offset angle and an offset distance. For example, the offset angle is greater than 45 degrees, and the offset distance is greater than 1 / 2 the radius of the video picture displayed in the viewfinder frame. That is, the first face image is located above the center position of the video picture displayed in the viewfinder frame and is relatively far from the center position. As shown in (a) of FIG. 16C, the first face image is a left face image, and the offset distance of the center point 166 of the first face image from the center point 167 of the viewfinder frame is relatively large. In this case, the auxiliary effect of the mid-range sound pickup range on the audio corresponding to the first face image is relatively small. For the first sound pickup range, please refer to the range represented by the ellipse 161 shown in (b) of FIG. 16C. Also, the first face image may be the range represented by the left half ellipse 1611 shown in (c) of FIG. 16B.
[0221] For example, as shown in Table 3 below, in the multi-microphone scenario shown in FIG. 2B(d), an example of a sound pickup range determined by a mobile phone based on the front / back attribute parameters of a video picture corresponding to a first face image and the position information corresponding to the first face image is described. Alternatively, an example of a sound pickup range determined by a mobile phone based on the front / back attribute parameters of a video picture corresponding to a first mouth image and the position information corresponding to the first mouth image is described.
[0222] [Table 3]
[0223] Finally, the mobile phone can determine a final first sound collection range based on the area ratio corresponding to the first face image. The mobile phone can determine a radius of the first sound collection range corresponding to the first face image using the area ratio and the sound collection range corresponding to the framing range.
[0224] For example, in a process of determining a first sound collection range using a method in which the above solutions 1 to 3 are combined, for example, the first sound collection range is surrounded by a circle 152 shown in FIG. 15(a). The radius of the circle 152 can be used to correspondingly represent the radius range of the first sound collection range. In this case, the first sound collection range can be represented using a range represented by a left semi-ellipse 1611 shown in FIG. 16B(c). As another example, in the scenario shown in FIG. 15(b), the radius of the first sound collection range is finally determined to be the distance between the center point of the first face image and the edge of the maximum sound collection range. In this case, the first sound collection range can be represented using a range represented by a left semi-ellipse 1611 and a left semi-ellipse 1612 shown in FIG. 16B(c).
[0225] It should be noted that in the process in which the mobile phone determines the first sound collection range by referring to the multiple solutions in the above solutions 1 to 3, the order in which the parameters are determined is not limited, and the mobile phone may determine the parameters in another order different from the order in the above example. For example, the parameters are determined simultaneously.
[0226] A first sound pickup range corresponding to the first face image or the first mouth image may be determined by using the above-mentioned solution, and then audio may be obtained by using the first sound pickup range, thereby improving the audio quality.
[0227] S604: The mobile phone acquires audio based on the first pickup range.
[0228] The mobile phone can use a single microphone or multiple microphones to collect ambient sound signals in various directions, i.e., collect panoramic sound signals. After pre-processing the panoramic sound signals collected by the multiple microphones, the mobile phone can obtain initial audio data, where the initial audio data includes sound information in various directions. Then, the mobile phone may record audio corresponding to the first face image based on the initial audio data and the first sound pickup range.
[0229] Optionally, after determining the first sound pickup range corresponding to the first face image or the first mouth image, the mobile phone may emphasize sounds that are within the first sound pickup range and are in the initial audio data, and suppress (or attenuate) sounds that are outside the first sound pickup range and are in the initial audio data. The processed audio data is then recorded to obtain audio corresponding to the first face image or the first mouth image.
[0230] In this way, the audio corresponding to the first face image or the first mouth image is a recording of the sound of the first sound collection range, and the first sound collection range is a sound collection range determined based on the first feature value corresponding to the first face image or the first mouth image. Therefore, the sound of the first sound collection range is a sound corresponding to the pronunciation face or the pronunciation mouth that the user is interested in. That is, the interference of noise in the recorded video picture with the sound emitted from the pronunciation face or the pronunciation mouth is reduced.
[0231] In addition, directional audio enhancement is performed based on the first sound collection range, so that in a complex shooting environment, only a part of the audio signal can be enhanced by using an audio algorithm, which can simplify the audio processing algorithm, improve processing efficiency, and reduce the requirements for the hardware computing performance of the mobile phone.
[0232] In some other scenarios, there may be an error between the first sound pickup range determined by the mobile phone based on the first feature value and the display range of the first face image or the first mouth image, so the mobile phone may determine one or more reference first sound pickup ranges near the first sound pickup range. The mobile phone obtains one audio based on the first sound pickup range, and obtains at least one audio based on the reference first sound pickup range. The mobile phone may further use a panoramic audio as the one audio. In this case, the mobile phone may obtain multiple audios corresponding to the first face image or the first mouth image based on the first sound pickup range. The one audio may be understood as one audio file.
[0233] In a possible implementation, the mobile phone can determine one or more corresponding reference first sound pickup ranges based on an area ratio corresponding to the first face image or the first mouth image. The first sound pickup range and the reference first sound pickup range are ratio Parameter Feelings It is assumed that the first sound pickup range and the reference first sound pickup range are determined based on the information. For example, based on Table 1, as shown in Table 4 below, the mobile phone may determine the first sound pickup range and the reference first sound pickup range based on the rules in Table 4 below. In Table 4 below, the first sound pickup range is a recommended value, and the reference first sound pickup range includes emphasis value 1, emphasis value 2, and emphasis value 3.
[0234] [Table 4]
[0235] In another possible implementation form, the mobile phone may determine the audio corresponding to the first pickup range and the reference first pickup range based on different audio processing methods. For example, based on the procedure of determining the first pickup range, the audio corresponding to the first pickup range is the audio determined by using the Dolby sound effect algorithm, and the audio corresponding to the reference first pickup range is the audio determined based on the Histen sound effect algorithm. As shown in Table 5 below, algorithm 1 to algorithm 4 are different audio algorithms, and the audio corresponding to the first pickup range and the reference first pickup range are determined based on different audio algorithms. The first pickup range is a recommended value, and the reference first pickup range includes emphasis value 1, emphasis value 2, and emphasis value 3.
[0236] [Table 5]
[0237] In yet another possible implementation form, the mobile phone may refer to the area parameter ratio information corresponding to the first face image or the first mouth image and the audio algorithm to obtain the audio corresponding to the first sound pickup range and the reference first sound pickup range. As shown in Table 6 below, the first sound pickup range is a recommended value, and the reference first sound pickup range includes an emphasis value 1, an emphasis value 2 and an emphasis value 3.
[0238] [Table 6]
[0239] It can be understood that the mobile phone may alternatively determine the reference first sound pickup range by using another method, which is not particularly limited in this embodiment of the present application.
[0240] In addition, the mobile phone may process the initial audio data to emphasize sounds within the reference first sound pickup range and suppress sounds outside the reference first sound pickup range, and further record the processed audio data to obtain one or more audio corresponding to the first face image or the first mouth image.
[0241] In this way, the mobile phone can record and obtain the first feature value corresponding to the first face image or the first mouth image and the multiple audios matching the picture of the first face image or the first mouth image based on the first sound pickup range and the reference first sound pickup range, for subsequent selection and playback by the user. Each audio data corresponding to the first face image or the first mouth image may be stored as one audio file, and the first face image may correspond to multiple audio files.
[0242] When the mobile phone records multiple audios corresponding to the first face image or the first mouth image based on the first sound pickup range and the reference first sound pickup range, the multiple audios provide the user with more audios of different sound pickup ranges, making it more likely to match the sound corresponding to the first face image or first mouth image that the user is interested in, and providing greater selectivity for the audio played by the user.
[0243] In some embodiments, the mobile phone may alternatively record the audio corresponding to the first face image or the first mouth image based on the first sound pickup range or the reference first sound pickup range selected by the user. For example, in the interface 1701 shown in FIG. 17, if the mobile phone detects the user tapping the recommended value selection control 171, in the process of recording the video picture, the audio corresponding to the first face image or the first mouth image is recorded based on the first sound pickup range and the initial audio data. Similarly, if the mobile phone detects the user tapping the emphasis value 1 selection control, in the process of recording the video picture, the audio corresponding to the first face image or the first mouth image is recorded based on the reference first sound pickup range and the initial audio data corresponding to the emphasis value 1. If the mobile phone detects the user tapping the non-processing selection control 172, in the process of recording the video picture, the audio signals of different directions are fused based on the initial audio data to obtain a panoramic audio. That is, the audio corresponding to the non-processed selection control 172 is panoramic audio, which may be understood as the audio obtained when the mobile phone is in a non-speech enhancement mode. For methods for determining the recommended values, enhancement value 1, enhancement value 2, and enhancement value 3 on the interface 1701, please refer to Tables 4 to 6, and details will not be described in this specification.
[0244] In some embodiments, before officially recording a video picture, the user may experience the recording effect corresponding to different sound pickup ranges to determine the sound pickup range to be selected in the process of finally recording the video picture. The mobile phone can only store the corresponding audio file based on the user's selection, so that the storage space of the mobile phone can be saved when the user's requirements are guaranteed to be met.
[0245] In some other scenarios, in the process of recording a video picture by a mobile phone, the first sound pickup range may change to a second sound pickup range. For example, in the process of recording a video picture, the mobile phone detects an operation by a user to give an instruction to switch between the front camera and the rear camera. The sound pickup range before switching is the first sound pickup range, and the sound pickup range after switching is the second sound pickup range. Therefore, for the audio in the recorded video, the audio recorded by the mobile phone includes at least an audio of a first duration and an audio of a second duration. The audio of the first duration is the audio corresponding to the first sound pickup range, and the audio of the second duration is the audio corresponding to the second sound pickup range. That is, the mobile phone can dynamically determine the sound pickup range based on the change of the pronunciation face or the pronunciation mouth in the video picture, and then record the audio based on the sound pickup range. After the operation of giving an instruction to stop recording by the user is finally detected, the audio of the formed video picture may include multiple audios of different durations or the same duration recorded based on the time sequence and the changing sound pickup range.
[0246] In this way, based on the change of the sound pickup range, the mobile phone can always focus on improving the audio recording quality of the part where the voice enhancement needs to be performed, thereby ensuring the audio recording effect.In addition, when the user plays the video file, the user can be presented with a playback experience that is consistent with the dynamic changes such as the sound range changes of the video content.
[0247] In a possible implementation, in the process of recording a video picture by a mobile phone, the first feature value corresponding to the first face image or the first mouth image changes, resulting in a change in the sound pickup range. For example, it is assumed that the front / back attribute parameter of the video picture changes, resulting in a change in the first sound pickup range. The interface 1801 shown in FIG. 18(a) displays a front video picture. When the duration of the recording is 00:15, the mobile phone detects an operation of tapping the front / back switching control 181 by the user, switches to the rear camera for shooting, and displays the interface 1802 shown in FIG. 18(b). In this case, the first feature value corresponding to the first face image or the first mouth image changes around the duration of 00:15, and the audio of the recorded audio with a duration of 00:00 to 00:15 is the audio corresponding to the first sound pickup range, and the audio after the duration of 00:15 is the audio corresponding to the second sound pickup range. Alternatively, the mobile phone detects that the focus position of the focus selected by the user has changed. In this case, the position information corresponding to the first face image or the first mouth image changes, and as a result, the first sound collection range changes.
[0248] Alternatively, the picture range and picture size of the video picture in the viewfinder frame may change with the change of the zoom ratio (i.e., Zoom value). The zoom ratio may be a preset zoom ratio, a zoom ratio last used before the camera is closed, a zoom ratio pre-instructed by the user, etc. In addition, the zoom ratio corresponding to the viewfinder frame may also be changed based on the user's instruction. In this case, in the scenario, the framing range changes as the zoom ratio changes. This changes the area of the first face image or the area of the first mouth image, and further changes the area ratio corresponding to the first face image or the first mouth image. That is, changing the zoom ratio changes the sound pickup range. In this way, in the subsequent video playback process, the recorded audio can dynamically change with the change of the video content display area, etc., thereby improving the user's playback experience.
[0249] For example, if other parameters are the same, when the zoom ratio is increased to twice the original value, the sound pickup range can be reduced to 1 / 3 of the original value, and when the zoom ratio is increased to three times the original value, the sound pickup range can be reduced to 1 / 6 of the original value. Therefore, the mobile phone can determine the sound pickup range corresponding to the framing range and the sound pickup range corresponding to the area ratio of the first face image or the area ratio of the first mouth image based on the zoom ratio. As shown in Table 7 below, X is used to represent the area of the first face image or the area of the first mouth image, and Y is used to represent the area of the video picture displayed in the viewfinder frame. When the Zoom value changes, the values of X and Y also change. The corresponding sound pickup range also changes.
[0250] [Table 7]
[0251] It should be noted that the sound pickup range may alternatively not be changed when the zoom ratio is changed. For example, in the recording process, after the zoom ratio is changed, the first face image does not change, indicating that the content that the user is interested in does not change. For example, user A interviews user B and shoots the interview process of user B by using a mobile phone. The mobile phone determines that the first face image in the video picture is the face image of user B. The mobile phone detects that the zoom ratio has increased, but in this case, the first face image in the video picture is still the face image of user B. In this case, the mobile phone does not need to obtain the first sound pickup range again, and the amount of calculation and power consumption can be reduced. Alternatively, if the mobile phone detects multiple operations that change the zoom ratio in a preset time range, the sound pickup range may not be changed. For example, the preset period is 2 seconds. After detecting an operation that changes the zoom ratio for the first time, the mobile phone does not need to recalculate the sound pickup range first. If the mobile phone does not detect an operation that changes the zoom ratio in 2 seconds, the mobile phone recalculates the sound pickup range. If the mobile phone detects another operation to change the zoom ratio within the two seconds, the mobile phone does not need to recalculate the sound pickup range, but instead uses the time node where the operation to change the zoom ratio was detected this time as the starting point and monitors whether an operation to change the zoom ratio is detected within the next two second period.
[0252] In a possible implementation, in the process of recording a video picture by a mobile phone, if the first face image or the first mouth image changes, the first sound pickup range changes. For example, the above-mentioned scenario of switching between the front camera and the rear camera may be understood as the change of the first face image or the first mouth image. Or, the pronunciation face image or the pronunciation mouth image changes, resulting in the change of the first face image or the first mouth image. For example, on the interface 1802 shown in FIG. 18(b), it is assumed that at the duration from 00:16 to 00:20, the mobile phone determines that the first face image is one of the two face images included in the video picture. At the duration from 00:21 to 00:30, the mobile phone recognizes that the first face image is the face image 182 on the right side of the video picture. Alternatively, if the captured picture moves and the currently recorded video picture does not contain the previously recognized first face image or first mouth image, the first sound pickup range needs to be re-recognized by using the above-mentioned method.
[0253] In a possible implementation form, the second sound collection range is determined in response to an operation of changing the selection of the first sound collection range or the reference first sound collection range by the user. For example, on the interface 1803 shown in FIG. 18(c), before the duration of 00:30, the mobile phone records a video picture using the first sound collection range corresponding to the recommended value, and detects an operation of tapping the emphasis value 2 selection control 183 by the user at 00:30. In response to this operation, the mobile phone determines the second sound collection range as the sound collection range corresponding to the emphasis value 2, and displays the interface 1804 shown in FIG. 18(d). After the duration of 00:30, the mobile phone acquires audio using the sound collection range corresponding to the emphasis value 2.
[0254] In some embodiments of the present application, before generating an audio file for each audio, the mobile phone can perform multiple types of sound effect processing for each audio, so that the recorded audio achieves higher audio quality and better audio processing effects. For example, the sound effect processing may include Dolby sound effects, Histen sound effects, sound retrieval system (SRS) sound effects, bass enhanced engine (BBE) sound effects, dynamic bass enhanced engine (DBEE) sound effects, etc.
[0255] It should be noted that in order to prevent frequent changes of the first sound collection range caused by frequent changes of the first feature value due to jitter of the mobile phone, the mobile phone may set a preset time threshold, and the mobile phone will not change the first sound collection range for changes within the preset time threshold. For example, based on the setting, if the first feature value changes twice consecutively in one second, the mobile phone will consider that the current change of the first feature value is caused by jitter of the mobile phone, and will not change the corresponding first sound collection range.
[0256] Optionally, in the process of using the aforementioned method to collect audio signals by using a microphone, the mobile phone can collect audio signals, and process the audio signals based on the first sound collection range to obtain audio corresponding to the first face image or the first mouth image. Finally, the final audio is generated directly after the video recording is finished. Alternatively, the mobile phone may collect audio signals first, and after the video recording is completed, process the audio signals based on the first sound collection range to obtain audio corresponding to the first face image or the first mouth image. Alternatively, the mobile phone calls the corresponding microphone to collect audio signals within the first sound collection range based on the first sound collection range, and processes the audio signals to obtain audio corresponding to the first face image or the first mouth image.
[0257] Optionally, the video recording function may include a single-channel video recording function and a multi-channel video recording function. The single-channel video recording function means that the mobile phone displays one viewfinder frame in the shooting process to record one video picture. The multi-channel video recording function means that the mobile phone displays at least two viewfinder frames in the shooting process, and each viewfinder frame is used for one video picture. In the process of using the multi-channel video recording function, each video picture and the corresponding audio collection manner can be referred to the implementation form of the single-channel video recording function. In the above method in which the first sound collection range is determined based on the first face image and the first mouth image, and the audio recording is performed based on the first sound collection range, an example in which the shooting interface includes one viewfinder frame is used for explanation. In addition, the process corresponding to the multi-channel video recording function including two or more viewfinder frames is similar to this process, and will not be described in detail.
[0258] In the process of recording a video picture, the mobile phone determines a first sound pickup range based on the pronunciation face image or the pronunciation mouth image, and then records audio based on the first sound pickup range. After that, the recorded audio needs to be stored. The user can play the video picture and audio of the stored video. It should be noted that if the scenario of recording a video picture is a real-time communication scenario such as a live broadcast or a video call, the method for recording audio in the process of recording a video picture can be referred to the above-mentioned method. However, after the operation of the user to stop shooting is detected, that is, after the operation of stopping communication is detected, there is no need to generate a recorded video, and the communication is stopped as it is. It can be understood that in some real-time communication scenarios, the user may alternatively choose to store the recorded video. In response to the user's operation, the mobile phone determines whether to store the recorded video in the real-time communication scenario.
[0259] The following describes a scenario in which a mobile phone stores recorded video and plays back stored recorded video.
[0260] Optionally, after detecting the user's operation of giving an instruction to stop recording, the mobile phone stops recording the video picture and audio, and generates a recorded video. The user's operation of giving an instruction to stop recording may be an operation of tapping the control 45 displayed on the video recording preview interface 403 shown in Fig. 4(c), an operation of the user instructing to stop recording by voice, an air gesture operation, or other operations. This is not particularly limited in this embodiment of the present application.
[0261] Optionally, after detecting the user's operation to give an instruction to stop shooting, the mobile phone generates a recorded video and returns to the video recording preview interface or the shooting preview interface. The recorded video may include video pictures and audio. For example, for the thumbnail of the recorded video generated by the mobile phone, please refer to the thumbnail 191 displayed on the interface 1901 shown in FIG. 19(a) or the thumbnail 192 displayed on the interface 1902 shown in FIG. 19(b).
[0262] In a possible implementation, the mobile phone may prompt the user that the recorded video has multiple audios. For example, the thumbnail of the recorded video or the detailed information of the recorded video may include prompt information used to indicate the multiple audios. For example, the prompt information may be a mark 193 of multiple loudspeakers displayed on the interface 1902 shown in FIG. 19(b), another form of mark, text information, etc. Each audio may correspond to the audio collected in the first sound collection range and the reference first sound collection range, respectively.
[0263] In a possible implementation form, in response to an operation by the user to give an instruction to stop shooting, the mobile phone displays an interface 1903 shown in FIG. 19(c) to prompt the user to store the required audio of the video file. The video file currently includes audio 194-197, which respectively correspond to audio files recorded in different sound collection ranges, or audio files recorded in the same sound collection range and obtained after processing by using different audio algorithms. For example, corresponding to the method shown in Tables 4-6 above, audio 194-197 are audio corresponding to the recommended value, emphasis value 1, emphasis value 2, and emphasis value 3, respectively. In response to an operation by the user to give a playback instruction, the mobile phone can play the video file and the corresponding audio. For example, when the mobile phone detects an operation by the user to instruct playback of audio 194, the mobile phone plays the video file and audio 194. After watching the video file, the user may choose to store the audio file that the user considers to have a better audio effect. In response to the user's selection, the audio that the user needs to store is determined, thereby improving the user's usage experience and avoiding the problem of excessive storage space being occupied due to excessive audio storage. On the interface 1903 shown in Fig. 19(c), the user selects to store the audio 194 and audio 197 of the current video file. In response to the user tapping the save control 198, the mobile phone stores the video file and displays the interface 1902 shown in Fig. 19(b). The number of loudspeakers in the loudspeaker mark 193 may correspond to the number of audios currently included in the video file.
[0264] Optionally, after detecting an operation of the user instructing to play the recorded video, the mobile phone plays the video picture and audio of the recorded video. The operation of the user instructing to play the recorded video may be the user tapping the thumbnail 191 on the video recording preview interface shown in Fig. 19(a). Alternatively, the operation of the user instructing to play the recorded video may be the user tapping the thumbnail 192 in the gallery shown in Fig. 19(b).
[0265] In a possible implementation, after detecting an operation by a user to instruct to play the recorded video, the mobile phone plays the recorded video based on the video picture and audio recorded in the video recording process. During the video playback, the mobile phone can display a video playback interface, and the video playback interface can include the recorded video picture. In addition, by default, the mobile phone may play the audio corresponding to the first sound pickup range, and then switch to play other audio based on the user's instruction. Alternatively, if the user selects a specific sound pickup range in the recording process, the mobile phone automatically plays the audio corresponding to the sound pickup range selected by the user.
[0266] For example, during video playback, the video playback interface may include multiple audio switching controls, each corresponding to an audio. After detecting a user tapping an audio switching control, the mobile phone plays the audio corresponding to the audio switching control.
[0267] For example, during video playback, the mobile phone may display a video playback interface 2001 shown in FIG. 20(a), where the video playback interface 2001 displays a video picture. The video playback interface 2001 further displays audio switching controls 201-205. As shown in FIG. 20(a), if the mobile phone currently selects the audio switching control 201 or selects a recommended value by default, the mobile phone plays audio corresponding to a first sound pickup range. If the mobile phone detects a user's tapping operation of the audio switching control 203, the mobile phone may play audio corresponding to a reference first sound pickup range corresponding to the audio switching control 203.
[0268] As another example, in response to a user's operation, the mobile phone may delete a portion of the audio corresponding to a video file. In the interface 2002 shown in FIG. 20(b), the mobile phone detects an operation of the user touching and holding the audio switching control 205, and displays a delete prompt box. If the user confirms the deletion, the mobile phone deletes the audio corresponding to the audio switching control 205, and displays the interface 2003 shown in FIG. 20(c). On the interface 2003, the audio corresponding to the audio confirmed for deletion by the user is displayed. switching The control 205 is no longer displayed. In this way, in the video playing process, the audio that the user does not want to remember can be deleted based on the user requirements, thereby improving the user's usage experience.
[0269] In another possible implementation, during video playback, the mobile phone can display a video playback interface without first playing audio, and after detecting a user's instruction, the mobile phone plays the audio as instructed by the user.
[0270] In the solution described in the above embodiment, during video playback, the mobile phone can play audio corresponding to the first face image or the first mouth image, so that the interference caused by noise on the sound emitted by the pronunciation face or the pronunciation mouth in the played audio is reduced. In addition, the played audio is matched in real time with the face image that the user is interested in, thereby improving the audio experience of the user.
[0271] In addition, the mobile phone can switch to play audio corresponding to different pickup ranges to provide the user with multiple audio playback options, thereby implementing an audio adjustment function and improving the user's audio playback experience.
[0272] In addition, the mobile phone can play audio corresponding to the first face image or first mouth image and the first feature value that change in real time, so that the audio matches the changing video picture in real time, thereby improving the user's audio experience.
[0273] 21A and 21B are schematic flowcharts of an audio processing method according to an embodiment of the present application. The audio processing method may be applied to the electronic device 100 shown in FIG.
[0274] In some embodiments, after detecting the user's operation of instructing to open the camera, the electronic device starts the camera and displays a shooting preview interface. Then, after detecting the user's operation of instructing to shoot, the electronic device starts to collect video pictures and first audio (i.e., initial audio signal).
[0275] It should be noted that the image collected by the camera of the electronic device is an initial video image, and after the initial video image is processed, a video picture is obtained that can be displayed on the display. The step of processing the initial video image is performed by a processor. In Figures 21A and 21B, the video picture collected by the camera is only an example for explanation.
[0276] The electronic device enables the voice enhancement mode in response to the user's operation before or after detecting the user's operation of giving a shooting instruction, or enables the voice enhancement mode after detecting the user's operation of giving a shooting instruction.
[0277] In some embodiments, the first audio is an audio signal collected in different directions by one or more microphones of the electronic device. The speech-enhanced audio can then be obtained based on the first audio.
[0278] For example, an example in which a processor includes a GPU, an NPU, and an AP is used for explanation. It can be understood that the steps performed by the GPU, the NPU, and the AP in this specification can be alternatively performed by another processing unit in the processor. This is not limited in this embodiment of the present application.
[0279] In some embodiments, the NPU in the processor recognizes whether the video picture includes a face image and / or a mouth image by using image recognition technology. In addition, the NPU may further determine the pronunciation face or the pronunciation mouth in the face image and / or the mouth image based on the data of the face image and / or the mouth image to determine the sound pickup range in which the directional voice recording needs to be performed.
[0280] By using the target image, a first feature value of the target image can be determined, and then a first sound pickup range is determined based on the first feature value. The first feature value includes one or more of a front / back attribute parameter, an area ratio, or location information. The front / back attribute parameter is used to indicate whether the video picture is a video picture taken by a front camera or a video picture taken by a rear camera. The area ratio is used to indicate the ratio of the area of the target image to the area of the video picture. The location information is used to indicate the location of the target image in the video picture.
[0281] In some scenarios, the first feature value includes a front / back attribute parameter corresponding to the target image. That is, the AP in the processor determines whether the video picture in which the current target image is located is a front video picture or a back video picture. If the video picture is a front video picture, the first sound collection range is the sound collection range of the front camera side. If the video picture is a back video picture, the first sound collection range is the sound collection range of the back camera side.
[0282] In some other scenarios, the first feature value includes an area ratio corresponding to the target image. The "area ratio" is used to indicate the ratio (e.g., expressed as X / Y) of the area of the first face image or the first mouth image to the area of the video picture. For example, the electronic device determines the first feature value based on the ratio of the area of the first face image to the area of the viewfinder frame.
[0283] Specifically, the area ratio is used to measure the size of the first sound collection range corresponding to the first face image or the first mouth image, for example, the radius range or diameter range of the first sound collection range. Thus, the AP can determine the radius range of the first sound collection range based on the area ratio of the first face image. Alternatively, the AP may determine the radius range of the first sound collection range based on the area ratio of the first mouth image. Then, the AP may determine the first sound collection range (for example, expressed as N*X / Y) based on the area ratio and the first audio sound collection range. For example, the area of the target image / area of the video picture=the first sound collection range / the first audio sound collection range.
[0284] In some other embodiments, the first feature value includes position information corresponding to the target image. The AP determines a position of the first sound collection range corresponding to the target image in the first audio sound collection range based on the position information of the target image in the video picture. Specifically, the AP determines a first offset of the center point of the target image relative to a first reference point, the first reference point being the center point or the focus point of the video picture. Then, the AP determines a second offset of the center point of the first sound collection range relative to the center point of the first audio sound collection range based on the first offset, the second offset being directly proportional to the first offset, so that the first sound collection range is obtained.
[0285] The first offset or the second offset includes an offset angle and / or an offset distance. For example, a coordinate system is constructed by using the first reference point as the origin, a direction parallel to the bottom edge of the electronic device (or the bottom edge of the current viewfinder frame) as the x-axis, and a direction perpendicular to the x-axis as the y-axis. The first reference point is used as the coordinate origin of the coordinate system, and the coordinate system is parallel to the display of the electronic device. If the first offset is 45 degrees to the top left, the second offset is 45 degrees to the top left. In this case, the first sound pickup range is within the sound pickup range of the first audio, and the center point of the first sound pickup range is 45 degrees to the top left of the center point of the sound pickup range of the first audio.
[0286] For example, the offset of the center of the target image relative to the reference point includes an offset angle θ1 and an offset distance L1. Surrounded by The offset of the first pickup range includes an offset angle θ2 and an offset distance L2, where θ1=θ2 and L1 / L2=constant.
[0287] It may be understood that the AP may determine the first pickup range by using one or any combination of the front / rear attribute parameters, the area ratio, and the location information.
[0288] In some embodiments, after determining the first pickup range, the AP in the processor enhances audio signals within the first pickup range and / or attenuates audio signals outside the first pickup range by using the first audio collected by the one or more microphones to obtain audio corresponding to the first face image or the first mouth image, i.e., to obtain the second audio.
[0289] In some embodiments, the AP can call a microphone corresponding to the first pickup range to emphasize the audio signal within the first pickup range so that the volume within the first pickup range is louder than the volume outside the first pickup range.
[0290] For example, the electronic device includes one or more microphones, the one or more microphones configured to collect the first audio, and when a pickup range of a first microphone of the one or more microphones includes a part or all of the first pickup range, the second audio is obtained by at least one of: enhancing an audio signal within the pickup range of the first microphone within the first pickup range, attenuating an audio signal within the pickup range of the first microphone outside the first pickup range, or attenuating an audio signal of a microphone other than the first microphone of the one or more microphones.
[0291] As another example, the electronic device includes at least two microphones, and the at least two microphones are configured to collect a first audio. When the pickup range of the second microphone of the at least two microphones does not include the first pickup range, the second microphone is disabled, and the audio collected by the microphone other than the second microphone of the at least two microphones is the audio corresponding to the first face image or the first mouth image. Alternatively, when the second microphone is disabled, an audio signal that is within the first pickup range and within the pickup range of the microphone other than the second microphone of the at least two microphones is emphasized, and / or an audio signal that is outside the first pickup range and within the pickup range of the microphone other than the second microphone of the at least two microphones is weakened.
[0292] In some embodiments, after obtaining the second audio, the AP in the processor obtains a recorded video by using the obtained video picture. After detecting an operation to give an instruction to stop shooting, the recorded video including the second audio and the video picture is obtained.
[0293] In some embodiments, the recorded video may include multiple audio files, and each audio file includes one audio. For example, since there may be an error between the first sound pickup range determined by the electronic device based on the first feature value and the display range of the first face image or the first mouth image, the electronic device may determine one or more reference first sound pickup ranges near the first sound pickup range. The electronic device obtains one audio based on the first sound pickup range, and obtains at least one audio based on the reference first sound pickup range. The electronic device may further use the panoramic audio as one audio. In this case, the electronic device may obtain multiple audios corresponding to the first face image or the first mouth image based on the first sound pickup range. One audio may be understood as one audio file.
[0294] In this way, multiple types of audio experiences can be provided to the user. In addition, the user can choose to delete some audio and store the audio that the user considers optimal based on personal audiovisual experience, thereby improving the user's usage experience and reducing memory storage pressure.
[0295] An embodiment of the present application further provides an electronic device including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are configured to store computer program code, the computer program code including computer instructions. When the one or more processors execute the computer instructions, the electronic device is enabled to perform the relevant method steps described above to implement the audio processing method in the aforementioned embodiment.
[0296] An embodiment of the present application further provides a chip system including a processor. The processor is coupled to a memory, and the memory is configured to store a program or instruction. When the program or instruction is executed by the processor, the chip system is enabled to implement the method in any one of the above-mentioned method embodiments.
[0297] Optionally, there may be one or more processors in the chip system. The processor may be implemented by using hardware or by using software. When the processor is implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When the processor is implemented by using software, the processor may be a general-purpose processor and is implemented by reading software code stored in a memory.
[0298] Optionally, there may be one or more memories in the chip system. The memory may be integrated with the processor or may be located separately from the processor. This is not limited in this application. For example, the memory may be a non-transitory processor, such as a read-only memory ROM. The memory and the processor may be integrated in the same chip or may be located separately on different chips. The type of memory and the manner of arranging the memory and the processor are not particularly limited in this application.
[0299] For example, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), or a programmable logic device (PLD) or another integrated chip.
[0300] It should be understood that the steps in the above-described method embodiments can be implemented by using integrated logic circuitry in a hardware processor or by using instructions in the form of software. The steps of the methods disclosed with reference to the embodiments of the present application may be directly executed by a hardware processor, or may be executed by a combination of hardware and software modules in a processor.
[0301] An embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions, which, when executed on a terminal device, enable the terminal device to perform the aforementioned associated method steps to implement the audio processing method in the aforementioned embodiment.
[0302] An embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to perform the relevant steps described above to implement the audio processing method in the aforementioned embodiment.
[0303] In addition, an embodiment of the present application further provides an apparatus. The apparatus may specifically be a component or module, and the apparatus may include a connected processor and a memory. The memory is configured to store computer-executable instructions. When the apparatus operates, the processor can execute the computer-executable instructions stored in the memory, so that the apparatus executes the audio processing method in the above-mentioned method embodiment.
[0304] The terminal device, computer readable storage medium, computer program product, or chip provided in the embodiments of the present application are all used to execute the corresponding method provided above. Therefore, for the beneficial effects that can be achieved by the terminal device, computer readable storage medium, computer program product, or chip, please refer to the beneficial effects in the corresponding method provided above. Details will not be described here.
[0305] It can be understood that to implement the aforementioned functions, the electronic device includes corresponding hardware and / or software modules for performing each function. The algorithm steps of the examples described with reference to the embodiments disclosed herein can be implemented in the present application in the form of hardware or a combination of hardware and computer software. Whether the functions are performed by hardware or by hardware driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the functions described for each specific application, but the implementation form should not be considered to go beyond the scope of the present application.
[0306] In an embodiment, the division of the functional modules may be performed on the electronic device based on the above-mentioned method example. For example, each functional module may be obtained through a division corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module may be implemented in the form of hardware. It should be noted that in the embodiment of the present application, the division into modules is an example, and is merely a logical functional division, and may be other divisions in the actual implementation form.
[0307] Based on the above description of the implementation, those skilled in the art can clearly understand that for the purpose of convenient and concise description, the above division into functional modules is used only as an example for description. During actual application, the above functions can be allocated to different functional modules for implementation based on requirements, in other words, the internal structure of the device is divided into different functional modules to implement all or part of the above-described functions. For the specific operation processes of the above-described system, device, and unit, please refer to the corresponding processes in the above-described method embodiments, and the details will not be described again here.
[0308] In the embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the described embodiment of a terminal device is merely an example. For example, the division into modules and units is merely a logical division of functions, and may be other divisions in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. Indirect couplings or communication connections between modules or units may be implemented in electronic, mechanical, or other forms.
[0309] The units described as separate parts may or may not be physically separate, and the parts shown as units may or may not be physical units, and may be located in one location or distributed over multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.
[0310] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, and each of the units may exist physically alone, or two or more units may be integrated into one unit. The integrated units may be implemented in the form of hardware or in the form of a software functional unit.
[0311] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application in essence, or the part contributing to the prior art, or all or part of the technical solution may be implemented in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for instructing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes any medium that can store program code, such as a flash memory, a removable hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.
[0312] The above description is merely a specific implementation form of the present application, and is not intended to limit the protection scope of the present application. Any variation or replacement within the technical scope disclosed in this application shall be included within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims. [Explanation of symbols]
[0313] 1 Antenna 2 Antennas 12 Right face image 13 Left face image 14 Right face image 15 Face images 16 First Mouth Image 17 First mouth image on the right 18 First mouth image on the left 21 Microphone 22 Microphone 23 Microphone 24 Microphones 25 Microphones 26 Microphones 27 Microphone 28 Microphones 29 Microphone 41 Camera Icon 44 Prompt Box 45 Shooting Control 46 Control 47 First face image 48 Viewfinder Frame 49 Prompt Box 71 Face Image 81 pronunciation face images 82 Prompt Box 83 Front / Rear Switching Control 84 pronunciation face images 85 Prompt Box 100 Electronic Devices 101 Dotted Box 102 Dotted Box 103 Dotted Box 104 Dotted Box 105 Dotted Box 106 Dotted Box 110 Processor 111 Dotted Box 112 Dotted Box 113 Dotted Box 114 Dotted Box 115 Dotted Box 116 Dotted Box 120 External Memory Interface 121 Internal Memory 121 marks 122 marks 123 marks 124 marks 125 marks 126 Dotted Box 127 marks 130 USB interface 140 Charging Management Module 141 Great Circle 141 Power Management Module 142 First Quadrant 142 Battery 143 Second Quadrant 144 Third Quadrant 145 Fourth Quadrant 150 Mobile communication module 152 Small Circle 160 Wireless Communication Module 161 Oval 162 Oval 163 Oval 164 center point 165 center point 166 Great Circle 170 Audio Module 170A Speaker 170B Receiver 170C Microphone 170D Headset Jack 171 Recommended Value Selection Control 172 Non-treated selection control 180 Sensor Module 180A PRESSURE SENSOR 180B Gyro sensor 180C Barometric Pressure Sensor 180D Magnetic Sensor 180E Acceleration Sensor 180F Distance Sensor 180G Optical Proximity Sensor 180H Fingerprint Sensor 180J Temperature Sensor 180K touch sensor 180L Ambient Light Sensor 180M Bone Conduction Sensor 181 Front / Rear Switching Control 182 Right Face Image 183 Emphasis value 2 selection control 190 Button 191 Motor 191 thumbnails 192 Indicator 192 thumbnails 193 marks 193 Camera 194 Display 195 SIM card interface 195 Audio 196 Audio 197 Audio 198 Save Control 201 Oval 201 Audio Switching Control 202 Oval 202 Audio Switching Control 203 Oval 203 Audio Switching Control 204 Oval 204 Audio Switching Control 205 Oval 205 Audio Switching Control 206 Oval 401 Main Interface 402 Shooting preview interface 403 Video recording preview interface 404 Shooting interface 421 Control 422 Control 423 Control 431 Control 432 Control 433 Control 434 Control 501 Great Circle 502 Small Circle 701 Interface 801 Shooting interface 802 Shooting interface 1001 Interface 1002 Interface 1003 Interface 1101 Interface 1102 Interface 1103 Interface 1201 Interface 1202 Interface 1203 Interface 1611 Left half ellipse 1621 Left half ellipse 1701 Interface 1801 Interface 1802 Interface 1803 Interface 1804 Interface 1901 Interface 1902 Interface 1903 Interface 1931 Camera 1932 Camera 2001 Video Playback Interface 2002 Interface 2003 Interface
Claims
1. 1. An audio processing method, the method being applied in an electronic device, the method comprising the steps of: detecting a first operation to open a camera application; displaying a shooting preview interface in response to the first operation; detecting a second action to initiate video recording; In response to the second operation, collecting a video picture and a first audio, and displaying a photographing interface, the photographing interface including a preview interface of the video picture; recognizing a target image in the video picture, the target image being a first face image and / or a first mouth image, the first face image being a face image of a sounding object in the video picture, and the first mouth image being a mouth image of the sounding object in the video picture; determining a first sound collection range corresponding to the sound producing object based on the target image; obtaining a second audio corresponding to the video picture based on the first sound collection range and the first audio, wherein a volume of the second audio within the first sound collection range is greater than a volume of the second audio outside the first sound collection range; Including, The step of determining a first sound collection range corresponding to the sound-producing object based on the target image includes: obtaining a first feature value based on the target image, the first feature value including a front / back attribute parameter, the front / back attribute parameter being used to indicate whether the video picture is a video picture taken by a front camera or a video picture taken by a rear camera; determining the first sound collection range corresponding to the sound-producing object based on the first feature value; Including, The step of determining the first sound collection range corresponding to the sound-producing object based on the first feature value includes: determining, when the video picture is a frontal video picture, that the first sound collection range is a frontal camera side sound collection range; determining, when the video picture is a rear video picture, that the first sound collection range is a rear camera side sound collection range; 13. An audio processing method comprising:
2. The method described in claim 1, wherein the first feature value further includes one or more of an area ratio or position information, the area ratio being used to indicate a ratio of the area of the target image to the area of the video picture, and the position information being used to indicate a position of the target image within the video picture.
3. The step of determining the first sound collection range corresponding to the sound-producing object based on the first feature value includes: determining the first sound collection range based on the area ratio and the first audio sound collection range; 3. The method of claim 2, comprising:
4. The step of determining the first sound collection range corresponding to the sound-producing object based on the first feature value includes: determining a position of the first sound pickup range in the sound pickup range of the first audio based on the position information; 3. The method of claim 2, comprising:
5. the position information includes a first offset of a center point of the target image relative to a first reference point, the first reference point being a center point or a focus point of the video picture; The step of determining a position of the first sound collection range in the sound collection range of the first audio based on the position information includes: determining a second offset of a center point of the first pickup range relative to a center point of the pickup range of the first audio based on the first offset, the second offset being directly proportional to the first offset; determining the location of the first pickup range in the pickup range of the first audio based on the second offset; 5. The method of claim 4, comprising:
6. The method of claim 5 , wherein the center point of the video picture is a center point of a viewfinder frame or the center point of the video picture is a center point of a display.
7. The step of acquiring a second audio corresponding to the video picture based on the first sound pickup range and the first audio includes: Emphasizing an audio signal that is within the first pickup range and within the first audio and / or attenuating an audio signal that is outside the first pickup range and within the first audio to obtain the second audio.
2. The method of claim 1, comprising:
8. the electronic device comprises one or more microphones, the one or more microphones configured to collect the first audio; The step of acquiring a second audio corresponding to the video picture based on the first sound pickup range and the first audio includes:
8. The method of claim 7, comprising, when a pickup range of a first microphone of the one or more microphones includes a part or all of the first pickup range, performing at least one of the following steps to obtain the second audio: emphasizing an audio signal that is within the first pickup range and within the pickup range of the first microphone; attenuating an audio signal that is outside the first pickup range and within the pickup range of the first microphone; and attenuating audio signals of microphones other than the first microphone of the one or more microphones.
9. the electronic device comprises at least two microphones, the at least two microphones configured to collect the first audio; The step of acquiring a second audio corresponding to the video picture based on the first sound pickup range and the first audio includes: disabling a second microphone of the at least two microphones if the pickup range of the second microphone does not include the first pickup range, and audio collected by a microphone other than the second microphone of the at least two microphones is the second audio.
8. The method of claim 7, comprising:
10. When the second microphone is disabled, the method further comprises:
10. The method of claim 9, further comprising: emphasizing audio signals that are within the first pickup range and within a pickup range of a microphone other than the second microphone of the at least two microphones; and / or attenuating audio signals that are outside the first pickup range and within the pickup range of a microphone other than the second microphone of the at least two microphones.
11. The method of claim 2 , wherein there are one or more first face images and there are one or more first mouth images.
12. After the step of collecting a video picture and a first audio and displaying a shooting interface in response to the second operation, the method further comprises: detecting a third operation to stop shooting; in response to the third operation, stopping recording and generating a recorded video, the recorded video including the video picture and the second audio; detecting a fourth operation of playing the recorded video; in response to the fourth operation, displaying a video playback interface and playing the video picture and the second audio; The method of claim 1, further comprising:
13. 13. The method of claim 12, wherein the recorded video further includes a third audio, the third audio being determined based on a second pickup range, the second pickup range being determined based on the first pickup range and different from the first pickup range, and the video playback interface includes a first control and a second control, the first control corresponding to the second audio and the second control corresponding to the third audio.
14. The method comprises: playing the video picture and the second audio in response to the fourth operation, the fourth operation including an operation of operating a player control or an operation of operating the first control; detecting a fifth operation of operating the second control; playing the video picture and the third audio in response to the fifth operation; 14. The method of claim 13, further comprising:
15. The method comprises: In response to an operation of deleting the second audio or the third audio, deleting the second audio or the third audio.
15. The method of claim 13 or 14, further comprising:
16. After the step of displaying a shooting preview interface in response to the first operation, the method further comprises: detecting a sixth operation of enabling a voice enhancement mode; enabling the voice enhancement mode in response to the sixth operation; The method of claim 1, further comprising:
17. 11. An electronic device comprising a processor, a memory, a microphone, a camera, and a display, wherein the memory, the microphone, the camera, and the display are coupled to the processor, and the memory is configured to store computer program code, the computer program code including computer instructions, and when the processor reads the computer instructions from the memory, the electronic device is enabled to perform the audio processing method of claim 1.
18. 13. A computer-readable storage medium having instructions stored therein that, when executed on an electronic device, enable the electronic device to perform the audio processing method of claim 1.
19. A computer program comprising instructions, when said computer program is run on an electronic device, said electronic device being enabled to perform the audio processing method according to claim 1.
Citation Information
Patent Citations
Camera, reproducing device, and reproducing method
JP2010093603A
Imaging apparatus, directivity control method, and program therefor
JP2011061461A
Recording device
JP2011223491A
Imaging device
JP2013179466A
