Sound processing method and related device
By adding a second microphone to the electronic device to reduce the noise to the sound collected by the first microphone, the problem of electronic devices collecting more noises during the recording process is solved, and a better recording experience is achieved.
Patent Information
- Application Number
- CN202311375964.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-20
- Publication Date
- 2025-04-29
AI Technical Summary
Electronic devices will collect more noises during the recording process, affecting the recording experience.
By adding a second microphone to the electronic device, the collected sound reduces the noise collected by the first microphone and reduces noise.
It effectively reduces noise during the recording process and improves the recording experience.
Smart Images
Figure CN119906931A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal technology, and in particular to a sound processing method and related devices. Background Art
[0002] The electronic device is installed with a camera application, and the camera application can support a video recording function. When the electronic device records a video, the electronic device can collect images through the camera and collect sounds through a microphone (MIC).
[0003] Currently, electronic devices will collect a lot of noise during the video recording process, affecting the video recording experience. Summary of the invention
[0004] The present application provides a sound processing method and related devices, which are applied to the field of terminal technology. In the present application, a MIC is added, and the sound collected during video recording is subjected to noise reduction processing through the sound collected by the MIC, which is beneficial to reducing the noise during the video recording process.
[0005] In the first aspect, the present application proposes a sound processing method, which is applied to an electronic device, wherein the electronic device includes a first microphone MIC and a second MIC, wherein the first MIC is deployed at the decoration DECO of the camera of the electronic device, and the second MIC is deployed in an area outside the DECO of the camera, and the collection object of the first MIC is the same as the collection object of the second MIC; the method includes: obtaining a first sound collected by the first MIC and a second sound collected by the second MIC; using the second sound to perform noise reduction processing on the first sound to obtain a third sound. In this way, the first MIC can collect noise generated by camera zoom or shaking, while the second MIC is not easy to collect noise generated by camera zoom or shaking, and using the second sound to perform noise reduction processing on the first sound is conducive to reducing the noise included in the first sound.
[0006] In a possible implementation, the second MIC is also deployed in a circuit board without a board vibration capacitor. This helps reduce the probability that the second MIC collects noise caused by the board vibration capacitor. In addition, the second MIC is not likely to collect noise caused by the board vibration capacitor, and the second sound is used to perform noise reduction processing on the first sound, which helps reduce the noise included in the first sound.
[0007] In a possible implementation, the orientation of the sound receiving port of the first MIC is the same as the orientation of the sound receiving port of the second MIC, so that the sound collected by the first MIC is the same as the sound collected by the second MIC.
[0008] In a possible implementation, obtaining a first sound collected by the first MIC and a second sound collected by the second MIC includes: in response to an operation for triggering the start of video recording, obtaining the first sound collected by the first MIC and the second sound collected by the second MIC; the method further includes: storing the third sound in the video obtained by the video recording. In this way, it is helpful to reduce the noise recorded in the video recording.
[0009] In a possible implementation, the third sound is a common sound between the second sound and the first sound. This helps to reduce the impact of noise in the second sound on the noise reduction process and helps to better remove the noise in the first sound.
[0010] In a possible implementation, using the second sound to perform noise reduction processing on the first sound to obtain the third sound includes: inputting the second sound and the first sound into a first model to obtain the third sound, and the first model is used to filter out sounds with the same pitch, frequency, timbre, amplitude or loudness from the second sound and the first sound to obtain the third sound. In this way, it is helpful to obtain the common sound between the second sound and the first sound.
[0011] In a possible implementation, storing the third sound in the video obtained by recording includes: if the magnification of the video is less than or equal to 1 when the first sound and the second sound are obtained, storing the third sound in the video obtained by recording. This is conducive to reducing the noise recorded in the video when the magnification of the video is less than or equal to 1.
[0012] In a possible implementation, the method further includes: if the magnification of the video is greater than 1 when acquiring the first sound and the second sound, then storing a fourth sound in the video obtained by the video recording, the fourth sound being a sound within a first angle of the third sound, the first angle corresponding to the magnification, and being used to indicate the range of sounds recorded under the magnification. In this way, when the magnification of the video recording is greater than 1, the sound recorded by the video recording is reduced, which is conducive to matching the sound recorded by the video recording with the captured image.
[0013] In a possible implementation, the electronic device further includes a third MIC and a fourth MIC, the third MIC and the fourth MIC are arranged on both sides of the first MIC, the third MIC and the fourth MIC collect different objects, the third MIC and the first MIC collect different objects, and the fourth MIC and the first MIC collect different objects; the fourth sound is obtained by processing the third sound through the sound collected by the third MIC and the sound collected by the fourth MIC, and the sound collected by the third MIC and the sound collected by the fourth MIC are used to suppress the sound outside the first angle in the third sound. In this way, it is conducive to obtaining the fourth sound.
[0014] In one possible implementation, when acquiring the first sound and the second sound, if the amplification factor is greater than 1, the second sound is used to perform noise reduction processing on the first sound to obtain a third sound, including: inputting the first sound into the first filter to obtain the sound output by the first filter; when the error between the sound output by the first filter and the second sound is less than or equal to the target error, the third sound is obtained. In some implementations, the first filter may be an adaptive filter. In this way, when the difference between the sound output by the first filter and the second sound is less than or equal to the target error, it can be considered that the sound output by the first filter is close to the second sound, the noise is removed, and the third sound can be obtained.
[0015] In a possible implementation, when the error between the sound output by the first filter and the second sound is less than or equal to the target error, a third sound is obtained, including: when the error between the sound output by the first filter and the second sound is greater than the target error, the first filter is updated using the least mean square algorithm (LMS) to obtain an updated first filter; the first sound is input to the updated first filter to obtain the sound output by the updated first filter; when the error between the sound output by the updated first filter and the second sound is less than the target error, the third sound is obtained. If the first error is greater than the target error, it can be considered that the difference between the sound output by the first filter and the second sound is large, indicating that the current first filter cannot effectively remove the noise in the first sound. In this way, the first filter can be updated to better remove the noise.
[0016] In one possible implementation, the second MIC is also deployed in an area outside the first area, and the first area is an area where the vibration of the first motor can be collected, and the first motor is used to drive the electronic device to vibrate. When the vibration mode of the electronic device is turned on, the first motor drives the electronic device to vibrate when the electronic device detects an incoming call or notification message. In this case, the second MIC can be deployed in an area outside the first area, which is conducive to making it difficult for the second MIC to collect the sound of the motor vibration, so that the second MIC includes less noise.
[0017] In a second aspect, the present application provides an electronic device, including a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the method described in any possible implementation of the first aspect.
[0018] In a third aspect, the present application provides an electronic device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The electronic device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.
[0019] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method described in any possible implementation manner of the first aspect is implemented.
[0020] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the method described in any possible implementation manner of the first aspect.
[0021] In a sixth aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.
[0022] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).
[0023] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;
[0025] Figure 2 is a schematic diagram of deploying a microphone in an electronic device provided by an embodiment of the present application;
[0026] Figure 3 It is a software structure block diagram of an electronic device to which the embodiments of the present application are applicable;
[0027] Figure 4 is a schematic flow chart of a sound processing method provided in an embodiment of the present application;
[0028] Figure 5 is a schematic diagram of a video recording provided by an embodiment of the present application;
[0029] Figure 6 is a schematic flow chart of a sound processing method provided in an embodiment of the present application;
[0030] Figure 7 is a schematic block diagram of a sound processing provided by an embodiment of the present application;
[0031] Figure 8 is a schematic flow chart of another sound processing method provided in an embodiment of the present application;
[0032] Fig. 9 is a schematic diagram of a sound processing provided by an embodiment of the present application;
[0033] Fig.10 It is a schematic block diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION
[0034] In order to clearly describe the technical solution of the embodiment of the present application, the following description is first made:
[0035] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first MIC and the second MIC are only used to distinguish different MICs, and do not limit their order. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.
[0036] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0037] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.
[0038] It should be noted that the "at..." in the embodiment of the present application can be the instant when a certain situation occurs, or can be a period of time before or after a certain situation occurs, and the embodiment of the present application does not specifically limit this. In addition, the display interface provided in the embodiment of the present application is only an example, and the display interface can also include more or less content.
[0039] Currently, electronic devices will collect a lot of noise during the video recording process, affecting the video recording experience.
[0040] Electronic devices may pick up noise in the following situations:
[0041] In the first case, the MIC for collecting sound (hereinafter referred to as the first MIC) is deployed at the decoration DECO of the camera. As the camera device becomes larger and larger, the sound when the camera is zoomed or shaken is also getting louder and louder. This sound can be easily captured by the first MIC, causing the electronic device to collect more noise during the recording process, affecting the recording experience. Among them, camera zoom refers to the object being photographed appearing closer or farther. The motor deployed near the camera will drive the camera to zoom to collect an enlarged image or a reduced image. It can be understood that the sound of the camera zoom refers to the sound of the motor working when the camera zooms. The camera shakes because the camera can move within a small range. When the electronic device shakes, it will cause the camera to shake.
[0042] In the second case, the camera and the first MIC of the electronic device are deployed on the main board of the electronic device. A board vibration capacitor is also deployed on the main board. If the board vibration capacitor works during the process of the first MIC collecting sound, it will cause the main board to vibrate. When the first MIC is close to the board vibration capacitor, the sound of the main board vibration will be recorded together, causing the electronic device to record more noise during the recording process, affecting the recording experience.
[0043] In view of this, the embodiments of the present application provide a sound processing method and a related device. For situation one, the noise caused by camera zoom or shaking in the sound collected by the first MIC can be reduced, which is beneficial to reducing the noise during the recording process; for situation two, the noise caused by the operation of the plate vibration capacitor in the sound collected by the first MIC can be reduced, which is beneficial to reducing the noise during the recording process.
[0044] Exemplarily, for situation one, a first MIC and a second MIC are deployed in the electronic device, and the collection object of the second MIC is the same as that of the first MIC, so that both the second MIC and the third MIC collect the sound required for recording. At the same time, the second MIC is deployed in an area outside the DECO of the camera, so that the second MIC is not easy to collect the sound of camera zooming or shaking. By performing noise reduction processing on the sound collected by the first MIC by the second MIC, the noise caused by the zooming or shaking of the camera in the sound collected by the first MIC can be reduced, which is beneficial to reduce the noise during the recording process.
[0045] For situation two, the electronic device is deployed with a first MIC and a second MIC, and the collection object of the second MIC is the same as that of the first MIC, so that both the second MIC and the second MIC collect the sound required for recording. At the same time, the second MIC can be deployed in a circuit board in the electronic device where no board vibration capacitor is deployed, so that the second MIC is not easy to collect noise caused by the vibration of the board vibration capacitor. By performing noise reduction processing on the sound collected by the first MIC by the second MIC, the noise caused by the operation of the board vibration capacitor in the sound collected by the first MIC can be reduced, which is beneficial to reduce the noise during the recording process.
[0046] The method provided in the embodiment of the present application can be applied to electronic devices, such as mobile phones or tablets. For ease of understanding, the hardware structure of the electronic device in the embodiment of the present application is introduced below.
[0047] For example, Figure 1 The hardware structure diagram of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 1As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, and a display screen 194, etc.
[0048] Optionally, the above-mentioned sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0049] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0050] The microphone 170C may include a first MIC, a second MIC, a third MIC and a fourth MIC. The first MIC is deployed at the DECO of the camera, the second MIC is deployed in an area outside the DECO of the camera, the third MIC and the fourth MIC may be deployed on both sides of the first MIC, and the objects collected by the third MIC and the fourth MIC are different. The collection object of the third MIC is different from that of the first MIC, and the collection object of the fourth MIC is different from that of the first MIC. The collection object is used to indicate the collection target, or in other words, the collection object is used to indicate the collection direction. The different collection objects of the two MICs are used to indicate that the collection directions of the two MICs are different, so that the two MICs collect different sounds; the same collection objects of the two MICs are used to indicate that the collection directions of the two MICs are the same, so that the two MICs collect the same sound.
[0051] For example, Figure 2 A schematic diagram of deploying a microphone in an electronic device is shown. Figure 2As shown in a, the electronic device is a mobile phone, which is placed vertically. The first MIC can be deployed at the DECO of the rear camera of the mobile phone, the second MIC can be deployed on the back of the mobile phone and deployed in an area other than the DECO of the camera, the third MIC can be deployed on the top of the mobile phone, and the fourth MIC can be deployed on the bottom of the mobile phone, for example, deployed next to the charging port of the mobile phone. The collection objects of these four MICs can be as follows Figure 2 As shown in b. Figure 2 In b, the sound receiving direction of the third MIC can be opposite to that of the fourth MIC, but both are perpendicular to the direction directly behind the mobile phone screen. The sound receiving direction of the first MIC can be the same as that of the second MIC, both are the direction directly behind the mobile phone screen. The sound receiving direction of the first MIC is the same as the direction of the camera shooting picture.
[0052] If the sound receiving direction of the third MIC is opposite to that of the fourth MIC, the collection object of the third MIC is different from that of the fourth MIC; if the sound receiving direction of the first MIC is the same as that of the second MIC, the collection object of the first MIC is the same as that of the second MIC. In other words, the sound collected by the first MIC and the sound collected by the second MIC both include the sound directly behind the screen, such as the far-end sound. The first MIC is close to the camera and can also collect interference sounds caused by camera zoom or shaking. The second MIC is deployed in an area outside the DECO of the camera and is far away from the camera, so it is not easy to collect the sound of camera zoom or shaking.
[0053] In some implementations, the electronic device may include multiple circuit boards, and the second MIC may be deployed in a circuit board in the electronic device where no board shock capacitor is deployed, so that the second MIC is not likely to collect noise caused by the operation of the board shock capacitor.
[0054] The processor 110 can obtain the sound collected by the first MIC and the sound collected by the second MIC during recording, and perform noise reduction processing on the sound collected by the first MIC through the sound collected by the second MIC, which can reduce the noise in the sound collected by the first MIC, thereby helping to reduce the noise during recording.
[0055] In some implementations, if the magnification of the video recording is less than or equal to 1 when acquiring the sound collected by the first MIC and the sound collected by the second MIC, the processor 110 may perform noise reduction processing on the sound collected by the first MIC through the second MIC, and store the noise-reduced sound in the video recorded. If the magnification of the video recording is greater than 1 when acquiring the sound collected by the first MIC and the sound collected by the second MIC, the processor 110 may simultaneously acquire the sound collected by the third MIC and the sound collected by the fourth MIC, perform noise reduction processing on the sound collected by the first MIC through the sound collected by the second MIC, locate the noise-reduced sound through the sound collected by the third MIC and the sound collected by the fourth MIC, obtain the sound within the first angle, and store the sound in the video recorded.
[0056] The motor 191 may include one or more motors. For example, the motor 191 may include two motors, one motor is used to drive the camera to zoom, and the other motor is used to drive the electronic device to vibrate.
[0057] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The layered architecture can adopt an Android system, an Apple (IOS) system, or other operating systems, which are not limited in the embodiments of the present application. The following takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device.
[0058] Figure 3 The following is a software structure diagram of an electronic device applicable to an embodiment of the present application. The layered architecture divides the software system of an electronic device into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into four layers, from top to bottom: application layer (applications), application framework layer (application framework), hardware abstraction layer (hardware abstraction layer, HAL), kernel layer (kernel) and hardware layer. The application layer can include a series of application packages, and the application layer runs applications by calling the application programming interface (application programming interface, API) provided by the application framework layer. Figure 3 As shown, the application package may include applications such as music, phone, and video.
[0059] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 3As shown, the application framework layer may include a noise reduction processing module, a view system, a content provider, and a phone manager. The noise reduction processing module is used to obtain the sound collected by one or more microphones, and to reduce the noise of the sound collected by the microphone based on the demand. For example, the noise reduction processing module can execute the sound processing method provided in the embodiment of the present application. For example, in the video recording scene, the instruction to start recording of the camera application in the application layer is received, and the sound collected by the microphone is obtained from the microphone in the hardware layer through the HAL layer and the kernel layer, and the sound collected by the microphone is denoised to obtain the denoised sound, and the denoised sound is transmitted to the camera application. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. In the embodiment of the present application, the view system may include controls for opening the video recording and controls for closing the video recording. The view system can be used to build an application. The display interface may be composed of one or more views. For example, a display interface including a text notification icon may include a view for displaying text and a view for displaying pictures, as well as an interface for making a call. The content provider is used to store and obtain data and make these data accessible to the application. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The phone manager is used to provide communication functions of the electronic device, such as the management of call status (including connected, hung up, etc.).
[0060] The purpose of the HAL layer is to abstract the hardware and provide a unified interface for querying hardware devices for upper-layer applications, or to provide data storage services for upper-layer applications. Figure 3 As shown, the HAL layer may include a display driver module and a sensor hardware abstraction (sensor hidl).
[0061] The kernel layer is the layer between hardware and software. Figure 3 As shown, the kernel layer may include one or more of the following: sensor driver, display driver, etc. In the embodiment of the present application, the sensor driver includes a sensor subsystem framework and a sensor subsystem driver, which are located in the coprocessor (SCP).
[0062] The hardware layer may include hardware such as microphones, cameras, and displays.
[0063] It should be understood that in some embodiments, the layers implementing the same function may be called by other names, or the layers that can implement the functions of multiple layers may be regarded as one layer, or the layers that can implement the functions of multiple layers may be divided into multiple layers. The embodiments of the present application are not limited to this.
[0064] Combined with the above Figures 1 to 3 The electronic device according to the embodiment of the present application has been introduced, and the method provided by the embodiment of the present application is introduced below.
[0065] The embodiment of the present application is applicable to any scenario where sound is collected by the first MIC, wherein the first MIC is deployed at the DECO of the camera. For example, if the electronic device uses the first MIC to collect sound in scenarios such as video recording, audio recording, video calls, or making or receiving calls, the electronic device can use the method provided in the embodiment of the present application to perform noise reduction processing on the sound collected by the first MIC. The embodiment of the present application uses the video recording scenario to illustrate the method provided in the embodiment of the present application.
[0066] Figure 4 The schematic flow chart of a sound processing method provided by an embodiment of the present application is shown. The method can be executed by an electronic device, and the hardware architecture diagram of the electronic device can be as described above. Figure 1 As shown, the software architecture diagram of the electronic device can be as described above Figure 3 The electronic device includes a first MIC and a second MIC, the first MIC is deployed at the DECO of the camera of the electronic device, and the second MIC is deployed in an area outside the DECO of the camera, and the first MIC and the second MIC have the same collection object. Figure 4 As shown, the method comprises the following steps:
[0067] S401 : In response to an operation for triggering the start of video recording, a first sound collected by a first MIC and a second sound collected by a second MIC are obtained.
[0068] The first MIC is deployed at the DECO of the camera, and can collect the sound when the camera zooms or shakes, while the second MIC is deployed in an area outside the DECO of the camera, and it is not easy to collect the sound of the camera zooming or shaking. At the same time, the collection object of the first MIC is the same as the collection object of the second MIC, which means that the collection directions of the first MIC and the second MIC are the same, and both can collect the sounds required for recording. In this way, the difference between the second sound collected by the second MIC and the first sound collected by the first MIC includes: there is no sound of the camera zooming or shaking, or the sound of the camera zooming or shaking is smaller.
[0069] In some implementations, the orientation of the sound receiving port of the first MIC may be the same as the orientation of the sound receiving port of the second MIC, so that the first MIC and the second MIC collect the same object. Figure 2b in the figure is not described here. It should be noted that the orientation of the sound receiving port of the first MIC is the same as the orientation of the sound receiving port of the second MIC, which may mean that the orientation of the sound receiving port of the first MIC is exactly the same as the orientation of the sound receiving port of the second MIC, or that there is a certain deviation between the orientation of the sound receiving port of the first MIC and the orientation of the sound receiving port of the second MIC, or that there is a certain angle difference between the orientation of the sound receiving port of the first MIC and the orientation of the sound receiving port of the second MIC, and the embodiments of the present application do not limit this.
[0070] In some implementations, in order to prevent the second MIC from easily collecting the sound of camera zooming or shaking, the second MIC can be farther away from the camera. For example, the distance between the second MIC and the camera can be greater than a preset value, which can be measured by R&D personnel after multiple tests.
[0071] For example, in the above Figure 2 In the example shown in a, the first MIC is deployed at the DECO of the camera, and can collect the interference sound caused by the camera zooming or shaking. The second MIC is deployed outside the DECO of the camera and is far away from the first MIC, so it is not easy to collect the sound of the camera zooming or shaking.
[0072] S402: Use the second sound to perform noise reduction processing on the first sound to obtain a third sound.
[0073] The second sound is used to perform noise reduction processing on the first sound, so that the third sound does not include the sound of camera zooming or shaking, or the sound of camera zooming or shaking in the third sound is smaller.
[0074] S403: Store the third sound in the video obtained by recording.
[0075] When the video recording ends, the third sound included in the obtained video may not include the sound of the camera zooming or shaking, or the sound of the camera zooming or shaking is relatively small, which is helpful to reduce the noise recorded in the video recording.
[0076] The video recording function of the electronic device generally supports the user to change the magnification of the video recording. When the magnification of the video recording is less than or equal to 1, the electronic device can adjust the focal length of the camera according to the magnification to collect images, and can collect sounds through the first MIC and the second MIC. Figure 4 The method shown processes the sounds collected by the two MICs, and stores the processed sounds and collected images in a video obtained by recording.
[0077] When the magnification of the video is greater than 1, the electronic device can adjust the focal length of the camera according to the magnification to capture the image, and can capture the sound through the first MIC and the second MIC. Figure 4 The method shown processes the sounds collected by two MICs to obtain processed sounds, and stores the sounds within a first angle in the processed sounds and the collected images in the video obtained by recording. Among them, there is a one-to-one correspondence between the first angle and the magnification factor, and / or a one-to-many relationship, which is not limited in the embodiments of the present application. For example, when the magnification factor is 1, the first angle is 60 degrees, when the magnification factor is 1.5, the first angle is 45 degrees, and when the magnification factor is greater than or equal to 3, the first angle is 30 degrees. If the electronic device amplifies the volume of the sound within the first angle, obtains the sound with the amplified volume, and stores the sound with the amplified volume in the video obtained by recording, the effect of image amplification and sound amplification can be achieved in the video.
[0078] For example, Figure 5 A schematic diagram of a video recording is shown. Figure 5 As shown, the electronic device is a mobile phone. Figure 5 As shown in a, the magnification of the video is 1, the magnification of the image captured by the mobile phone video is 1, and the range of the recorded sound can be It can be 60 degrees; Figure 5 As shown in b, the magnification of the video is 1.5, the magnification of the image captured by the mobile phone video is 1.5, the sound range recorded can be β, and β can be 45 degrees; Figure 5 a to Figure 5 In b, the user adjusts the magnification of the video from 1 to 1.5, and the electronic device can magnify the image captured by the video and expand the sound range of the video from It is reduced to β, and the sound within the recorded β range is amplified to achieve the effect of image amplification and sound amplification.
[0079] It is understandable that if the magnification of the video is greater than 1 when acquiring the first sound and the second sound, the electronic device also needs to process the third sound to obtain the fourth sound, and the fourth sound is the sound within the first angle of the third sound. Figure 2 As shown in a, the electronic device can obtain the sound collected by the third MIC and the sound collected by the fourth MIC at the same time when obtaining the first sound and the second sound, and use the sound collected by the third MIC and the sound collected by the fourth MIC to locate the third sound, so as to suppress the sound outside the first angle in the third sound and obtain the sound within the first angle in the third sound.
[0080] When recording, if the magnification of the video is less than or equal to 1, the electronic device can denoise the sound collected by the first MIC, and does not need to retain the sound within a certain angle. At this time, there is no need to have a specific direction for the sound, or in other words, it is not necessary to specify the sound to be collected in a certain direction. At this time, the sound collected by the first MIC and the sound collected by the second MIC can both match the captured picture, so the electronic device can collect the sound through the second MIC when the magnification of the video is less than or equal to 1, and store the collected sound in the video obtained by the video. In this way, the second MIC is not easy to collect the sound of the camera zooming or shaking, and there is less noise, which is conducive to reducing the noise collected by the video.
[0081] When recording a video, if the magnification of the video is greater than 1, the electronic device can denoise the sound collected by the first MIC and retain the sound within a certain angle. At this time, the sound can be specifically directed, or in other words, the sound of a certain direction can be specified to be recorded. At this time, the sound collected by the first MIC at this direction can be better matched with the captured image than the sound collected by the second MIC at this direction. Therefore, the electronic device can make the sound collected by the first MIC close to the sound collected by the second MIC when the magnification of the video is greater than 1, so as to reduce the impact of the sound of camera zooming or shaking on the video.
[0082] Exemplarily Figure 6 A schematic flow chart of a sound processing method provided in an embodiment of the present application is shown. The method can be executed by an electronic device. The software architecture of the electronic device can be as described above. Figure 3 As shown in the above Figure 3 In the architecture shown, the method can be implemented by a noise reduction processing module of a terminal device. Figure 6 As shown, the method comprises the following steps:
[0083] S601: If the magnification of the video recording is greater than 1 when acquiring the first sound and the second sound, the first sound is input into an adaptive filter to obtain a sound output by the adaptive filter.
[0084] The adaptive filter is a filter with adjustable parameters, that is, the parameters in the adaptive filter can be changed according to the actual application. The unchanged parameters in the adaptive filter can become the initial parameters, which is just an example and is not limited in the embodiments of the present application.
[0085] S602: Process the sound output by the adaptive filter and the second sound through a computing unit to obtain a first error.
[0086] The first error is used to represent the difference between the sound output by the adaptive filter and the second sound. In some implementations, the operation unit can be a difference operation. The electronic device can obtain the difference between the sound output by the adaptive filter and the second sound through the operation unit to obtain the first error.
[0087] S603: If the first error is greater than the target error, use LMS to update the adaptive filter to obtain an updated adaptive filter.
[0088] The target error is used to represent the acceptable difference between the sound output by the adaptive filter and the second sound. In other words, when the difference between the sound output by the adaptive filter and the second sound is less than or equal to the target error, the sound output by the adaptive filter can be considered to be close to the second sound. The target error is preset and can be calibrated by R&D personnel after multiple tests. The embodiments of the present application do not specifically limit this. Among them, the target error can also be called an expected value, which is not limited in the embodiments of the present application.
[0089] If the first error is greater than the target error, it can be considered that the difference between the sound output by the adaptive filter and the second sound is large, indicating that the current adaptive filter cannot effectively remove the noise in the first sound, so the adaptive filter can be updated. Using LMS to update the adaptive filter is a common processing method, but the embodiments of the present application are not limited to this. Using LMS to update the adaptive filter means updating the parameters of the adaptive filter through LMS. In other words, the initial parameters in the adaptive filter are changed through LMS to obtain an updated adaptive filter.
[0090] Optionally, if the first error is less than or equal to the target error, a third sound may be obtained.
[0091] S604: Input the first sound into the updated adaptive filter to obtain the sound output by the updated adaptive filter.
[0092] S605 , processing the sound output by the updated adaptive filter and the second sound through a computing unit to obtain a second error.
[0093] When the difference between the sound output by the updated adaptive filter and the second sound is less than or equal to the target error, it can be considered that the sound output by the adaptive filter is close to the second sound, which can indicate that the updated adaptive filter can better remove the noise in the first sound, so there is no need to update the adaptive filter again. It is understandable that if the difference between the sound output by the updated adaptive filter and the second sound is still greater than the target error, the electronic device needs to update the adaptive filter again and verify whether the adaptive filter needs to be updated again, i.e. repeat the above S603 and S604.
[0094] S606: If the first error is less than or equal to the target error, a third sound is obtained, where the third sound is the sound output by the updated adaptive filter.
[0095] The difference between the sound output by the updated adaptive filter and the second sound is less than or equal to the target error, the sound output by the adaptive filter is close to the second sound, and the electronic device can store the sound output by the updated adaptive filter in the video obtained by recording.
[0096] For example, Figure 7 A schematic block diagram of sound processing is shown. Figure 7 As shown, the first sound is represented by x(n), the second sound is represented by d(n), the input of the adaptive filter is x(n), the output of the adaptive filter is represented by y(n), the input of the operation unit is y(n) and d(n), the output of the operation unit is represented by e(n), the target error is represented by u, when e(n)≤u, it can be said that y(n) is close to d(n), it can be considered that the noise in the first sound is removed, and the sound after the noise is removed is y(n). When e(n)>u, it can be said that the difference between y(n) and d(n) is large, it can be considered that the noise in the first sound is not removed, and the adaptive filter is updated by the LMS algorithm.
[0097] The sound processing method provided in the embodiment of the present application uses an adaptive filter with variable parameters to reduce noise on the first sound. Compared with an adaptive filter with fixed parameters, the method is more applicable to different scenarios and has greater flexibility.
[0098] In the above Figure 6 In the method shown, if the sound collected by the second MIC includes less noise, the sound collected by the first MIC can be better denoised.
[0099] The embodiments of the present application provide a variety of methods for reducing noise in the sound collected by the second MIC.
[0100] In one possible implementation, the electronic device may include a main board and a small board, wherein a board-vibration capacitor and a camera are disposed on the main board, and no board-vibration capacitor is disposed on the small board. If the first MIC is disposed on the main board of the electronic device and at the DECO of the camera, the second MIC may be disposed on the small board. In this way, it is helpful to reduce the probability that the second MIC collects noise caused by the board-vibration capacitor. Among them, the main board and the small board are different circuit boards. In this case, the sound collected by the second MIC is used to perform noise reduction processing on the sound collected by the first MIC to obtain a third sound, and the noise caused by the board-vibration capacitor in the third sound is small or absent. If in a video recording scene, the third sound included in the obtained video has less noise, which is helpful to reduce the noise recorded in the video.
[0101] In another possible implementation, the electronic device also includes a motor for driving the electronic device to vibrate. The second MIC can be deployed in an area outside the DECO of the camera and in an area outside the first area, wherein the first area refers to an area where the motor vibration can be collected. For example, when the vibration mode of the electronic device is turned on, when the electronic device detects an incoming call or notification message, the motor drives the electronic device to vibrate and generate vibration, and the second MIC is not easy to collect the sound of the motor vibration. In this way, it is conducive to including less noise in the sound collected by the second MIC.
[0102] If the second sound collected by the second MIC includes other abnormal sounds, it may affect the noise reduction processing of the first sound. Therefore, the embodiment of the present application provides another sound processing method, which is conducive to reducing the impact of other abnormal sounds included in the second sound on the noise reduction processing of the first sound. This method can be applied to scenes where the magnification of the video is less than or equal to 1, and can also be applied to scenes where the magnification of the video is greater than 1.
[0103] The following description is based on a scenario where the magnification of the video is less than or equal to 1.
[0104] For example, Figure 8 FIG. 1 is a schematic flow chart of a sound processing method provided in an embodiment of the present application. Figure 8 As shown, the method comprises the following steps:
[0105] S801 . In response to an operation for triggering the start of video recording, a first sound collected by a first MIC and a second sound collected by a second MIC are obtained.
[0106] This step may refer to the above-mentioned S401 and will not be described in detail here.
[0107] S802: Amplify the volume of the first sound to obtain the amplified volume of the first sound, and amplify the volume of the second sound to obtain the amplified volume of the second sound.
[0108] The embodiment of the present application does not limit the amplification factor of the volume. Amplifying the volume of the first sound and the volume of the second sound is helpful to reduce the probability of failing to process the sound due to the volume being too low. The amplification factor of the volume of the first sound can be the same as the amplification factor of the volume of the second sound, so as to facilitate the subsequent screening of the common sound between the two sounds.
[0109] S803. Input the amplified second sound and the amplified first sound into the first model to obtain a third sound. The first model is used to select sounds having the same pitch, frequency, timbre, amplitude or loudness from the amplified second sound and the amplified first sound to obtain the third sound.
[0110] The third sound is a common sound between the amplified second sound and the amplified first sound, wherein the common sound refers to a sound having at least one factor of the same pitch, frequency, timbre, amplitude or loudness in the two sounds.
[0111] The second sound after the volume is amplified includes other abnormal sounds and far-end sounds generated near the deployment location. The first sound after the volume is amplified includes interference sounds and far-end sounds caused by camera zooming or shaking. If these two sounds include the same far-end sound, the common sound between the two sounds is the far-end sound, that is, the sound required to be collected for video recording.
[0112] Obtaining the third sound through the first model is merely an example, and the embodiments of the present application do not limit this.
[0113] For example, Fig. 9 A schematic diagram of sound processing is shown. Data A is used to represent the second sound after the volume is amplified, and data B is used to represent the first sound after the volume is amplified. The second sound after the volume is amplified includes other abnormal sounds and far-end sounds generated near the deployment position, and the first sound after the volume is amplified includes interference sounds and far-end sounds caused by camera zooming or shaking. The intersection of data A and data B is the far-end sound. The electronic device can separate the far-end sound from data A and data B to obtain the sound required for recording.
[0114] S804: If the magnification of the video recording is less than or equal to 1 when the first sound and the second sound are obtained, the third sound is stored in the video obtained by the recording.
[0115] When the video recording ends, the third sound included in the obtained video does not include the sound of the camera zooming or shaking, or the sound of the camera zooming or shaking is relatively small, which is helpful to reduce the noise recorded in the video recording.
[0116] It is understandable that if the magnification of the video is greater than 1 when acquiring the first sound and the second sound, the electronic device can store a fourth sound in the video obtained by the video recording, and the fourth sound is the sound within the first angle of the third sound.
[0117] In some implementations, the electronic device can input the first sound, the second sound, the third sound, and the first angle into the second model to obtain a fourth sound. The second model is used to suppress the sound in the third sound except for the first angle through the first sound and the second sound to obtain the sound in the third sound that is within the first angle.
[0118] In the above Figure 8 In the method shown, S802 is optional, and the electronic device may store the common sound between the first sound and the second sound as the third sound in the video obtained by recording.
[0119] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.
[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0121] The sound processing method of the embodiment of the present application has been described above, and the device for executing the above sound processing method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above sound processing method.
[0122] In order to achieve the above functions, the device for implementing the sound processing method includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the method steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0123] The embodiment of the present application can divide the functional modules of the device for implementing the sound processing method according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0124] like Fig.10 A schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip 100 includes one or more (including two) processors 1001 , a communication line 1002 , a communication interface 1003 and a memory 1004 .
[0125] In some implementations, the memory 1004 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.
[0126] The method performed by the electronic device described in the above embodiment of the present application can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps performed by the electronic device in the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1001. The above-mentioned processor 1001 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1001 can implement or execute the methods, steps and logic block diagrams performed by the disclosed electronic device in the embodiment of the present application.
[0127] The steps performed by the electronic device disclosed in the embodiments of the present application can be directly embodied as being performed by a hardware decoding processor, or being performed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 1004, and the processor 1001 reads the information in the memory 1004, and completes the steps performed by the above electronic device in combination with its hardware.
[0128] The processor 1001 , the memory 1004 , and the communication interface 1003 may communicate with each other via the communication line 1002 .
[0129] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.
[0130] The present application embodiment also provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium may include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.
[0131] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.
[0132] As a possible design, the computer readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connection line may also be appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while optical discs reproduce data optically using lasers.
[0133] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
Claims
1. A sound processing method, characterized in that: Applied to an electronic device, the electronic device comprises a first microphone MIC and a second MIC, the first MIC is deployed at a decoration DECO of a camera of the electronic device, the second MIC is deployed in an area outside the DECO of the camera, and the collection object of the first MIC is the same as the collection object of the second MIC; The method comprises: Acquire a first sound collected by the first MIC and a second sound collected by the second MIC; The first sound is subjected to noise reduction processing using the second sound to obtain a third sound.
2. The method according to claim 1, characterized in that The second MIC is also disposed in a circuit board in the electronic device where no board shock capacitor is disposed.
3. The method according to claim 1 or 2, characterized in that: The orientation of the sound receiving port of the first MIC is the same as the orientation of the sound receiving port of the second MIC.
4. The method according to any one of claims 1 to 3, characterized in that The acquiring the first sound collected by the first MIC and the second sound collected by the second MIC includes: In response to an operation for triggering the start of video recording, acquiring the first sound and the second sound; The method further comprises: The third sound is stored in the video obtained by recording.
5. The method according to claim 4, characterized in that The third sound is a common sound between the second sound and the first sound.
6. The method according to claim 5, characterized in that The step of using the second sound to perform noise reduction processing on the first sound to obtain a third sound includes: The second sound and the first sound are input into a first model to obtain the third sound, and the first model is used to filter out sounds with the same pitch, frequency, timbre, amplitude or loudness from the second sound and the first sound to obtain the third sound.
7. The method according to any one of claims 4 to 6, characterized in that: Storing the third sound in the video obtained by recording includes: If the magnification of the video recording is less than or equal to 1 when the first sound and the second sound are acquired, the third sound is stored in the video obtained by the recording.
8. The method according to claim 4, characterized in that The method further comprises: If the magnification of the video recording is greater than 1 when acquiring the first sound and the second sound, the fourth sound is stored in the video obtained by the recording. The fourth sound is the sound within a first angle of the third sound. The first angle corresponds to the magnification and is used to indicate the sound range recorded at the magnification.
9. The method according to claim 8, characterized in that The electronic device further includes a third MIC and a fourth MIC, wherein the third MIC and the fourth MIC are deployed on both sides of the first MIC, the objects collected by the third MIC and the fourth MIC are different, the objects collected by the third MIC and the first MIC are different, and the objects collected by the fourth MIC and the first MIC are different; The fourth sound is obtained by processing the third sound through the sound collected by the third MIC and the sound collected by the fourth MIC, and the sound collected by the third MIC and the sound collected by the fourth MIC are used to suppress the sound outside the first angle in the third sound.
10. The method according to claim 8 or 9, characterized in that: When acquiring the first sound and the second sound, if the amplification factor is greater than 1, performing noise reduction processing on the first sound using the second sound to obtain a third sound includes: Inputting the first sound into a first filter to obtain a sound output by the first filter; When the error between the sound output by the first filter and the second sound is less than or equal to the target error, the third sound is obtained.
11. The method according to claim 10, characterized in that When the error between the sound output by the first filter and the second sound is less than or equal to the target error, obtaining the third sound comprises: When the error between the sound output by the first filter and the second sound is greater than the target error, updating the first filter using a least mean square algorithm LMS to obtain an updated first filter; Inputting the first sound into the updated first filter to obtain the sound output by the updated first filter; When the error between the sound output by the updated first filter and the second sound is smaller than the target error, the third sound is obtained.
12. The method according to claim 10 or 11, characterized in that: The second MIC is also deployed in an area outside the first area, where the first area is an area where the vibration of the first motor can be collected, and the first motor is used to drive the electronic device to vibrate.
13. An electronic device, characterized in that: include: A processor, a memory, a first MIC and a second MIC, wherein the first MIC is deployed at a DECO of a camera of the electronic device, and the second MIC is deployed in an area outside the DECO of the camera, and the collection object of the first MIC is the same as the collection object of the second MIC; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 12.
14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.
15. A computer program product, characterized in that The invention comprises a computer program which, when being executed, causes a computer to execute the method according to any one of claims 1 to 12.