Method and device for processing audio signal

Abstract

The present application provides a method and device for processing audio signals. The method includes: obtaining an audio signal from the user's environment; reducing the pickup frequency when the duration of the audio signal is greater than or equal to a time threshold, and when the audio signal meets one or both of the following two conditions; condition one is that the volume of the audio signal is less than or equal to a volume threshold, and condition two is that the frequency of the audio signal is within a healthy frequency band. The above method can be executed by a smart watch. If the ambient sound meets one or both of the above two conditions, the smart watch can prompt the user to reduce the pickup frequency, thereby reducing the power consumption of ambient sound detection without affecting the user's health.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a method and device for processing audio signals. Background Art

[0002] Ambient sound refers to the collective sound produced by all sound sources in a user's environment. For example, when a user is standing on the street, the clatter of car tires, the chatter of pedestrians, and the sound of store advertisements all constitute the ambient sound. Ambient sound has a significant impact on user health. Prolonged exposure to high-volume ambient sound can cause irreversible damage to the hearing system.

[0003] One approach to protecting user health is to use a smartwatch to monitor ambient sound volume. When the volume exceeds a critical threshold, the user is notified of the current ambient sound volume, allowing them to take protective measures. To ensure user health, ambient sound monitoring needs to be enabled for extended periods of time. However, the battery life of smartwatches is limited, so reducing the power consumption of this monitoring function is a pressing issue. Summary of the Invention

[0004] This application provides a method for processing audio signals. By detecting and analyzing the audio signals in the user's environment, the method changes the pickup frequency of wearable devices such as smartwatches based on the duration, volume, and frequency of the audio signals, thereby effectively reducing the power consumption of wearable devices such as smartwatches when monitoring ambient sound. Furthermore, this application also provides a method for evaluating ambient sound based on the user's physiological information, so that the user can choose a comfortable environment, as well as an interactive hearing improvement solution based on the identification of the sound source of ambient sound.

[0005] In a first aspect, a method for processing an audio signal is provided, comprising: obtaining an audio signal from a user's environment; reducing a pickup frequency when a duration of the audio signal is greater than or equal to a time threshold, and when the audio signal satisfies one or both of the following two conditions; and the volume of the audio signal is less than or equal to a volume threshold, and the frequency of the audio signal is within a healthy frequency band.

[0006] The above method can be executed by a smartwatch. When the volume of the ambient sound is less than or equal to the volume threshold for a period of time, it indicates that the volume of the ambient sound is likely to be less than the volume threshold in the future. When the frequency of the ambient sound is within a healthy frequency range for a period of time, it indicates that the frequency of the ambient sound is likely to be within a healthy frequency range in the future. Therefore, if the ambient sound meets one or both of the above two conditions, the smartwatch can prompt the user to reduce the pickup frequency, thereby reducing the power consumption of ambient sound detection without affecting the user's health.

[0007] Optionally, reducing the sound pickup frequency includes: prompting the user to reduce the sound pickup frequency; receiving control information input by the user, the control information being used to reduce the sound pickup frequency; and reducing the sound pickup frequency according to the control information.

[0008] When the pickup frequency needs to be reduced, the smart watch can prompt the user through text or sound to choose whether to reduce the pickup frequency; when the user chooses to reduce the pickup frequency, the smart watch reduces the pickup frequency based on the user's choice, which can meet the user's personalized needs.

[0009] Optionally, the method further includes: displaying a characteristic graph showing the volume and / or frequency of the audio signal changing over time.

[0010] The smart watch can display detection records of different time periods. For example, it can display the daily ambient sound frequency value or volume value, the weekly ambient sound frequency value or volume value, and the monthly ambient sound frequency value or volume value, so that users can take targeted hearing prevention measures.

[0011] Optionally, it also includes: obtaining real-time physiological information of the user, the acquisition time of the real-time physiological information is the same as the acquisition time of the audio signal; determining a first comfort level corresponding to the audio signal based on a difference between the real-time physiological information and the preset physiological information, wherein the first comfort level is negatively correlated with the difference.

[0012] The above-mentioned real-time physiological information can be heart rate, blood oxygen content, or blood pressure. Taking the real-time physiological information as heart rate as an example, when the difference between the heart rate value collected by the smartwatch in real time and the preset heart rate value is large, it means that the user's heart rate value deviates significantly from the healthy heart rate value, and it can be determined that the user's comfort level is poor, that is, the first comfort level corresponding to the audio signal is poor; when the difference between the heart rate value collected by the smartwatch in real time and the preset heart rate value is small, it means that the user's heart rate value is close to the healthy heart rate value, and it can be determined that the user's comfort level is good, that is, the first comfort level corresponding to the audio signal is good. Therefore, the application of the above solution can prompt the user to change the environment in time when no serious physiological reaction occurs, thereby maintaining the user's health.

[0013] Optionally, it also includes: obtaining first indication information input by the user, the first indication information is used to indicate a second comfort level of the audio signal; training a comfort algorithm according to the second comfort level to make the first comfort level the same as the second comfort level, wherein the comfort algorithm is used to determine the first comfort level.

[0014] Different users have different tolerance levels for ambient sounds. Smart watches can prompt users to evaluate the comfort level of the current ambient sounds, determine the preset physiological information corresponding to different comfort levels based on the user's evaluation, and make decisions that are more in line with the user's individual characteristics.

[0015] Optionally, the method also includes: playing multiple sounds, the multiple sounds having different volumes and frequencies; obtaining second indication information input by the user, the second indication information being used to indicate a third comfort level corresponding to the multiple sounds; training the comfort algorithm based on the third comfort level, the comfort algorithm being used to determine the first comfort level.

[0016] The above embodiment uses actively played sounds to determine the user's perception of different sounds, i.e., to determine the third comfort level. Compared to using environmental sounds to train the comfort algorithm, this embodiment can provide a richer variety of sounds, allowing the user to actively and quickly train the comfort algorithm, rapidly improving its performance.

[0017] Optionally, the method further includes: when the frequency band of the audio signal is within a harmful frequency band, prompting the user to perform protective processing.

[0018] Audio signals that are too low or too high in frequency can also have a negative impact on the user's health. Smart watches can detect the frequency band of audio signals so that they can prompt users to take protective measures when receiving ambient sounds in harmful frequency bands.

[0019] Optionally, it also includes: when the audio signal is a mixed signal of multiple sound sources, prompting the user to select a sound source to be processed from the multiple sound sources; obtaining third indication information input by the user, the third indication information is used to indicate the sound source to be processed; obtaining fourth indication information input by the user, the fourth indication information is used to indicate a processing method for the sound source to be processed, the processing method including noise reduction processing or enhancement processing; performing noise reduction processing or enhancement processing on the audio signal corresponding to the sound source to be processed according to the third indication information and the fourth indication information.

[0020] The audio signal acquired by a smartwatch is typically a mixed signal from multiple sound sources. The smartwatch can determine the multiple sound sources corresponding to the mixed signal through a neural network or other analysis method, and prompt the user to select the sound source to be processed. If the user needs to obtain a specific sound source from the multiple sound sources, the third and fourth indication information can be used to indicate that the sound of the specific sound source needs to be enhanced. Alternatively, the third and fourth indication information can be used to indicate that the sound emitted by sound sources other than the specific sound source needs to be reduced. Therefore, this embodiment can process the audio signal based on the user's needs to enhance the user experience.

[0021] Optionally, the method further includes: when the sound source to be processed does not match the processing method, prompting the user to confirm whether the sound source to be processed and the processing method are correct.

[0022] The sound source to be processed is, for example, the speech sound in a live meeting. Usually, the user's processing method for the speech sound is enhancement processing. The smart watch or mobile phone can determine that the speech sound and the enhancement processing match. When the user selects speech sound and noise reduction processing, the smart watch or mobile phone can determine that the sound source to be processed does not match the processing method, and prompt the user to confirm whether the selected sound source to be processed and processing method are correct, thereby avoiding the user's wrong choice.

[0023] Optionally, obtaining the audio signal of the user's environment includes: obtaining the audio signal through multiple Internet of Things devices.

[0024] The aforementioned multiple IoT devices can be Bluetooth speakers, mobile phones, smart TVs, and cameras. Obtaining audio signals from multiple IoT devices can predict the audio environment around the user and remind the user to take preventive measures as soon as possible.

[0025] In a second aspect, another method for processing audio signals is provided, including: obtaining an audio signal of a user's environment; obtaining real-time physiological information of the user, wherein the acquisition time of the real-time physiological information is the same as the acquisition time of the audio signal; determining a first comfort level corresponding to the audio signal based on a difference between preset physiological information and the real-time physiological information, wherein the first comfort level is negatively correlated with the difference.

[0026] Optionally, the method further includes: obtaining first indication information input by the user, the first indication information being used to indicate a second comfort level corresponding to the audio signal; and training a comfort level algorithm based on the second comfort level, the comfort level algorithm being used to determine the first comfort level.

[0027] Optionally, it also includes: playing multiple sounds, the multiple sounds having different volumes and frequencies; obtaining second indication information input by the user, the second indication information being used to indicate a third comfort level corresponding to the multiple sounds; training the comfort level algorithm according to the noise threshold and the harmful frequency band, the comfort level algorithm being used to determine the first comfort level.

[0028] Optionally, it also includes: when the duration of the audio signal is greater than or equal to a time threshold, and when the audio signal meets one or both of the following two conditions, reducing the pickup frequency; the volume of the audio signal is less than or equal to a volume threshold, and the frequency of the audio signal is within a healthy frequency band.

[0029] Optionally, the method further includes: when the volume of the audio signal is greater than the volume threshold, increasing the sound pickup frequency.

[0030] Optionally, the method further includes: when the frequency band of the audio signal is within a harmful frequency band, prompting the user to perform protective processing.

[0031] Optionally, it also includes: when the audio signal is a mixed signal of multiple sound sources, prompting the user to select a sound source to be processed from the multiple sound sources; obtaining third indication information input by the user, the third indication information is used to indicate the sound source to be processed; obtaining fourth indication information input by the user, the fourth indication information is used to indicate a processing method for the sound source to be processed, the processing method including noise reduction processing or enhancement processing; performing noise reduction processing and / or enhancement processing on the audio signal corresponding to the sound source to be processed according to the third indication information and the fourth indication information.

[0032] Optionally, the method further includes: when the sound source to be processed does not match the processing method, prompting the user to confirm whether the sound source to be processed and the processing method are correct.

[0033] Optionally, obtaining the audio signal of the user's environment includes: obtaining the audio signal through multiple Internet of Things devices.

[0034] In a third aspect, the present application provides an apparatus for processing audio signals, comprising a unit for executing the method described in the first aspect. The apparatus may be a terminal device or a chip within the terminal device. The apparatus may include an input unit and a processing unit. The processing unit may be a processor, and the input unit may be a communication interface. The terminal device may also include a storage unit, which may be a memory. The storage unit is configured to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the terminal device to execute the method described in the first aspect.

[0035] In a fourth aspect, the present application provides an apparatus for processing audio signals, comprising a unit for executing the method described in the second aspect. The apparatus may be a terminal device or a chip within the terminal device. The apparatus may include an input unit and a processing unit. The processing unit may be a processor, and the input unit may be a communication interface. The terminal device may also include a storage unit, which may be a memory. The storage unit is configured to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the terminal device to execute the method described in the second aspect.

[0036] In a fifth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the method described in the first aspect.

[0037] In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the method described in the second aspect.

[0038] In a seventh aspect, the present application provides a computer program product, which includes: a computer program code, which, when executed by a processor, enables the processor to execute the method described in the first aspect.

[0039] In an eighth aspect, the present application provides a computer program product, which includes: a computer program code, which, when executed by a processor, enables the processor to execute the method described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic diagram of a hardware system of an apparatus for processing audio signals provided by the present application;

[0041] Figure 2 is a schematic diagram of a software system of a device for processing audio signals provided by the present application;

[0042] Figure 3 This is a schematic diagram of an application scenario for processing audio signals provided by the present application;

[0043] Figure 4 This is a schematic diagram of a method for monitoring environmental sound provided by the present application;

[0044] Figure 5 This is a schematic diagram of another application scenario for processing audio signals provided by the present application;

[0045] Figure 6 is a schematic diagram of a method for processing an audio signal provided by the present application;

[0046] Figure 7 This is a schematic diagram of a mobile phone interface for turning on the energy-saving sound pickup function provided by this application;

[0047] Figure 8 This is a schematic diagram of a mobile phone interface for a user to set a threshold value provided by the present application;

[0048] Figure 9 This is a schematic diagram of a mobile phone interface provided by the present application that prompts the user to take protective measures;

[0049] Figure 10 is a schematic diagram of another method for processing audio signals provided by the present application;

[0050] Figure 11This is a schematic diagram of a mobile phone interface for predicting comfort provided by this application;

[0051] Figure 12 This is a schematic diagram of a mobile phone interface for improving the user's hearing environment provided by the present application;

[0052] Figure 13 This is a schematic diagram of a mobile phone interface for prompting a user to confirm a method for processing an audio signal provided by the present application;

[0053] Figure 14 It is a structural diagram of another device for processing audio signals provided by the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0055] Figure 1 The figure shows a hardware structure of a device applicable to the present application, which is used to obtain an audio signal of a user's environment.

[0056] like Figure 1 As shown, the device 100 can be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc. The embodiment of the present application does not impose any limitation on the specific type of the device 100.

[0057] The device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a photoplethysmograph (PPG) sensor 180M, etc.

[0058] It should be noted that Figure 1 The structure shown does not constitute a specific limitation on the device 100. In other embodiments of the present application, the device 100 may include Figure 1 More or fewer components than those shown, or apparatus 100 may include Figure 1 Combinations of some of the components shown, or alternatively, the apparatus 100 may include Figure 1 Subassemblies of some of the components shown. Figure 1 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0059] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). The different processing units may be independent devices or integrated devices.

[0060] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0061] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0062] In some embodiments, the processor 110 may include one or more interfaces. For example, the processor 110 may include at least one of the following interfaces: an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and a USB interface.

[0063] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the device 100.

[0064] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0065] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0066] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0067] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the device 100.

[0068] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal interface or as a data signal interface. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, and sensor module 180. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, or a MIPI interface.

[0069] The USB interface 130 is an interface that complies with USB standards, such as a mini USB interface, a micro USB interface, or a USB Type-C interface. The USB interface 130 can be used to connect a charger to charge the device 100, transfer data between the device 100 and peripheral devices, and connect headphones to play audio. The USB interface 130 can also be used to connect other devices 100, such as AR devices.

[0070] Figure 1 The connection relationship between the modules shown is only for illustrative purposes and does not limit the connection relationship between the modules of the apparatus 100. Optionally, the modules of the apparatus 100 may also adopt a combination of the multiple connection modes in the above embodiments.

[0071] The charging management module 140 is used to receive power from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive current from the wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive electromagnetic waves through the wireless charging coil of the device 100 (the current path is shown as a dotted line). While the charging management module 140 is charging the battery 142, it can also provide power to the device 100 through the power management module 141.

[0072] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (e.g., leakage, impedance). Optionally, the power management module 141 can be set in the processor 110, or the power management module 141 and the charging management module 140 can be set in the same device.

[0073] The wireless communication function of the device 100 can be implemented by components such as the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

[0074] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0075] The mobile communication module 150 can provide a wireless communication solution applied to the device 100, such as at least one of the following solutions: second generation (2 th generation, 2G) mobile communication solutions, third generation (3 th generation, 3G) mobile communication solutions, fourth generation (4 th generation, 5G) mobile communication solutions, the fifth generation (5 th generation, 5G) mobile communication solutions. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and perform processing such as filtering and amplifying the received electromagnetic waves, and then transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor, and the amplified signal is converted into an electromagnetic wave and radiated out through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0076] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (e.g., speaker 170A, receiver 170B) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0077] Similar to the mobile communication module 150, the wireless communication module 160 can also provide wireless communication solutions applied to the device 100, such as at least one of the following solutions: wireless local area networks (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR). The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signal, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate and amplify it, and the signal is converted into electromagnetic waves and radiated through the antenna 2.

[0078] In some embodiments, antenna 1 of device 100 is coupled to mobile communication module 150 , and antenna 2 of device 100 is coupled to wireless communication module 160 .

[0079] Device 100 can implement display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0080] Display screen 194 can be used to display images or videos. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0081] The device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0082] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0083] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard red, green, blue (RGB), YUV or other format. In some embodiments, the device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0084] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0085] Video codecs are used to compress or decompress digital video. Device 100 may support one or more video codecs. This allows device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0086] The NPU is a processor that draws inspiration from biological neural network architecture, such as the transmission patterns between neurons in the human brain, to rapidly process input information and continuously learn. The NPU enables intelligent cognitive functions of device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0087] External memory interface 120 can be used to connect an external memory card, such as a secure digital (SD) card, to expand the storage capacity of device 100. The external memory card communicates with processor 110 via external memory interface 120 to implement data storage. For example, files such as music and videos can be stored on the external memory card.

[0088] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, and applications required for at least one function (for example, a sound playback function and an image playback function). The data storage area can store data created during the use of the device 100 (for example, audio data and a phone book). In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as: at least one disk storage device, a flash memory device, and a universal flash storage (UFS). The processor 110 executes various processing methods of the device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0089] The device 100 can implement audio functions, such as music playback and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0090] The audio module 170 is used to convert digital audio information into analog audio signal output, and can also be used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 or some functional modules of the audio module 170 can be provided in the processor 110.

[0091] The speaker 170A, also known as a horn, is used to convert an audio electrical signal into a sound signal. The device 100 can listen to music or make hands-free calls through the speaker 170A.

[0092] The receiver 170B, also known as an earpiece, is used to convert audio electrical signals into sound signals. When a user uses the device 100 to answer a call or voice message, the user can listen to the voice by placing the receiver 170B close to the ear.

[0093] Microphone 170C, also known as a microphone or a microphone, is used to convert sound signals into electrical signals. When a user makes a call or sends a voice message, the sound signal can be input into microphone 170C by speaking close to microphone 170C. The device 100 can be provided with at least one microphone 170C. In other embodiments, the device 100 can be provided with two microphones 170C to achieve a noise reduction function. In other embodiments, the device 100 can also be provided with three, four or more microphones 170C to achieve functions such as identifying the source of sound and directional recording. The processor 110 can process the electrical signal output by the microphone 170C. For example, the audio module 170 and the wireless communication module 160 can be coupled through a PCM interface. After the microphone 170C converts the ambient sound into an electrical signal (such as a PCM signal), the electrical signal is transmitted to the processor 110 through the PCM interface; the processor 110 performs volume analysis and frequency analysis on the electrical signal to determine the volume and frequency of the ambient sound.

[0094] The headphone jack 170D is used to connect a wired headphone and can be a USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0095] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, including resistive, inductive, and capacitive pressure sensors. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes, and device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to display screen 194, device 100 detects the touch operation based on pressure sensor 180A. Device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed; when a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0096] The gyroscope sensor 180B can be used to determine the motion posture of the device 100. In some embodiments, the angular velocity of the device 100 around three axes (i.e., the x-axis, the y-axis, and the z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and somatosensory games.

[0097] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the device 100 calculates altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0098] Magnetic sensor 180D includes a Hall effect sensor. Device 100 can use magnetic sensor 180D to detect the opening and closing of a flip case. In some embodiments, when device 100 is a flip phone, device 100 can detect the opening and closing of the flip cover based on magnetic sensor 180D. Device 100 can configure features such as automatic unlocking of the flip cover based on the detected opening and closing status of the case or flip cover.

[0099] Accelerometer 180E can detect the magnitude of acceleration of device 100 in various directions (typically the x-, y-, and z-axes). When device 100 is stationary, it can detect the magnitude and direction of gravity. Accelerometer 180E can also be used to identify the posture of device 100, serving as an input parameter for applications such as landscape / portrait switching and pedometers.

[0100] The distance sensor 180F is used to measure distance. The device 100 can measure distance using infrared or laser. In some embodiments, such as in a photography scenario, the device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0101] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The device 100 emits infrared light outward through the LED. The device 100 uses a photodiode to detect infrared reflected light from nearby objects. When reflected light is detected, the device 100 can determine that there is an object nearby. When no reflected light is detected, the device 100 can determine that there is no object nearby. The device 100 can use the proximity light sensor 180G to detect whether the user holds the device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used for automatic unlocking and automatic screen locking in holster mode or pocket mode.

[0102] Ambient light sensor 180L is used to sense ambient light brightness. Device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether device 100 is in a pocket to prevent accidental touches.

[0103] The fingerprint sensor 180H is used to collect fingerprints. The device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0104] Temperature sensor 180J is used to detect temperature. In some embodiments, device 100 uses the temperature detected by temperature sensor 180J to implement a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, device 100 reduces the performance of a processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, device 100 heats battery 142 to prevent abnormal shutdown of device 100 due to low temperature. In other embodiments, when the temperature is below yet another threshold, device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0105] The touch sensor 180K is also referred to as a touch-sensitive device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a touch screen. The touch sensor 180K is configured to detect touch operations applied thereto or in the vicinity thereof. The touch sensor 180K can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the device 100, in a location different from the display screen 194.

[0106] The PPG sensor 180M can measure a user's heart rate and blood oxygen level using photoelectric technology. In some embodiments, the PPG sensor 180M emits a beam of green light. After penetrating the skin and entering living tissue, some photons are absorbed, while others are reflected and captured by a photon receiver. Photon absorption by tissues like muscle, bone, and veins remains essentially constant, while arterial absorption varies due to vascular contraction and dilation. Consequently, the photon receiver receives two optical signals: a direct current (DC) signal reflected from tissues like muscle, bone, and veins, and an alternating current (AC) signal reflected from arteries. The PPG sensor 180M converts the optical signal into an electrical signal and inputs it into the processor 110. The processor 110 can then determine the user's heart rate based on the electrical signal converted from the AC optical signal. The processor 110 can also determine the user's blood oxygen level based on the electrical signal output by the PPG sensor 180M.

[0107] The buttons 190 include a power button and a volume button. The buttons 190 can be mechanical buttons or touch buttons. The device 100 can receive button input signals and implement functions related to the case input signals.

[0108] Motor 191 can generate vibration. Motor 191 can be used for incoming call reminders or for touch feedback. Motor 191 can produce different vibration feedback effects for touch operations acting on different applications. Motor 191 can also produce different vibration feedback effects for touch operations acting on different areas of the display screen 194. Different application scenarios (for example, time reminders, receiving messages, alarm clocks, and games) can correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0109] The indicator 192 may be an indicator light, which may be used to indicate the charging status and power level change, and may also be used to indicate messages, missed calls, and notifications.

[0110] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into the SIM card interface 195 to achieve contact with the device 100, and can also be pulled out from the SIM card interface 195 to achieve separation from the device 100. The device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. Multiple cards can be inserted into the same SIM card interface 195 at the same time, and the types of the multiple cards can be the same or different. The SIM card interface 195 is also compatible with external memory cards. The device 100 interacts with the network through the SIM card to achieve functions such as calls and data communications. In some embodiments, the device 100 uses an embedded SIM (eSIM) card, and the eSIM card can be embedded in the device 100 and cannot be separated from the device 100.

[0111] The hardware system of device 100 is described in detail above. The following describes the software system of device 100. The software system can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. In the embodiments of this application, the layered architecture is used as an example to exemplify the software system of device 100.

[0112] like Figure 2 As shown, a software system using a layered architecture is divided into several layers, each with clear roles and divisions of labor. Layers communicate with each other via software interfaces. In some embodiments, the software system can be divided into four layers: from top to bottom, the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0113] The application layer may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

[0114] The application framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. The application framework layer may include some predefined functions.

[0115] For example, the application framework layer includes the window manager, content provider, view system, telephony manager, resource manager, and notification manager.

[0116] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, and take screenshots.

[0117] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, and phone books.

[0118] The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon can include a view for displaying text and a view for displaying images.

[0119] The phone manager is used to provide communication functions of the device 100, such as management of call status (connected or disconnected).

[0120] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, and video files.

[0121] The Notification Manager allows applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the Notification Manager is used to notify downloads and message reminders. The Notification Manager can also manage notifications that appear in the status bar at the top of the system in the form of icons or scrolling text, such as notifications from applications running in the background. The Notification Manager can also manage notifications that appear on the screen in the form of dialog windows, such as prompting text messages in the status bar, emitting alert sounds, vibrating electronic devices, and flashing indicator lights.

[0122] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0123] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0124] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine performs functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0125] The system library may include multiple functional modules, such as a surface manager, a media library, a 3D graphics processing library (such as OpenGL ES), and a 2D graphics engine (such as SGL).

[0126] The surface manager is used to manage the display subsystem and provide the fusion of 2D layers and 3D layers for multiple applications.

[0127] The media library supports playback and recording of multiple audio and video formats, as well as still image files. The media library supports a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0128] The 3D graphics processing library can be used to implement 3D graphics drawing, image rendering, compositing and layer processing.

[0129] A 2D graphics engine is a drawing engine for 2D drawings.

[0130] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0131] The following describes the workflow of the software system and hardware system of the device 100 in conjunction with a sound pickup scenario.

[0132] When a user performs a touch operation on touch sensor 180K, a corresponding hardware interrupt is sent to the kernel layer, which processes the touch operation into a raw input event. This raw input event includes, for example, information such as the touch coordinates and the timestamp of the touch operation. The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the raw input event. If the touch operation is a single click, and the control corresponding to the single click is the control of the microphone application icon, the control calls the application framework layer's interface to start the microphone application, which then calls the kernel layer to start the microphone driver and obtain the audio signal through microphone 170C.

[0133] The following describes the method for processing audio signals provided in this application by taking the device 100 as a smart watch as an example.

[0134] Figure 3 This is a schematic diagram of an application scenario for processing audio signals provided by this application.

[0135] Smartwatches, as sound pickup devices, capture audio signals (i.e., ambient sound) from the user's surroundings. This audio signal is typically a mixture of multiple sound sources. For example, when a user is standing on the street, the rustling of car tires, the conversations of pedestrians, and the sounds of store advertisements all constitute the ambient sound.

[0136] Ambient sound has a significant impact on user health. Prolonged exposure to high-volume ambient sound can cause irreversible damage to the hearing system. Furthermore, the frequency range of ambient sound also has a significant impact on user health. Long-term exposure to low-frequency ambient sound (e.g., below 20Hz) can lead to illnesses such as neurasthenia.

[0137] The smartwatch can monitor ambient sound and prompt the user to take protective measures when the volume of the ambient sound is greater than or equal to a volume threshold, and / or when the frequency band of the ambient sound is in a harmful frequency band.

[0138] Figure 4 A schematic diagram of a method for monitoring environmental sound provided by the present application is shown.

[0139] The smartwatch's microphone picks up ambient sound and converts it into a pulse code modulation (PCM) signal. This PCM signal then undergoes volume analysis and frequency analysis to produce a monitoring result containing both volume and frequency band values.

[0140] During volume analysis, the volume value can be based on Determine, where L p is the volume value, p ref is the maximum value of the total amplitude of the sound, p rms The amplitude value of the current sound.

[0141] During the frequency analysis process, the PCM signal is subjected to a fast Fourier transform (FFT) to obtain a spectrum signal, and then the frequency distribution of the spectrum signal can be determined to obtain the frequency band value of the ambient sound.

[0142] This application does not limit the specific methods of volume analysis and frequency analysis of audio signals.

[0143] When the frequency band of the ambient sound is in a harmful frequency band, the smart watch can remind the user that the current ambient sound is harmful to health through vibration, flashing, tone or text, so that the user can leave the current environment. When the volume of the ambient sound is too loud, the smart watch can be connected via Bluetooth (such as Figure 3 ) indicates that the headset turns on the noise reduction function to protect the user's health. This application does not limit the communication method between the smart watch and the headset.

[0144] Smartwatches can also collaborate with other electronic devices to process audio signals. For example, when a smartwatch's processing power is insufficient or its battery is low, it can transmit the PCM signal to a mobile phone, which then analyzes the volume and frequency of the PCM signal and, based on the results, controls the headphones to perform noise reduction and other operations.

[0145] In addition to picking up sound through smart watches, sound can also be picked up through electronic devices such as smart bracelets, smart TVs, laptops, tablets, etc. The audio signal processing systems of these electronic devices are as follows: Figure 5As shown, smart watches, smart TVs, laptops and tablets can be called IoT devices. IoT devices can communicate with audio signal processing devices (such as mobile phones) via WLAN or other communication methods, transmit the collected audio signals to the processing device, and the processing device analyzes the audio signals to determine the processing method for the audio signals. After determining the processing method for the audio signal, the processing device can interact with the headphones and control the headphones to perform corresponding processing (such as noise reduction or playing prompt sounds). Obtaining audio signals through multiple electronic devices can predict the audio environment around the user and remind the user to take preventive measures as soon as possible.

[0146] To ensure the health of users, the ambient sound monitoring function needs to be turned on for a long time. However, some electronic devices have limited battery life and need to reduce the power consumption of ambient sound monitoring. The following describes the method for processing audio signals provided by this application.

[0147] like Figure 6 As shown, the method 600 includes:

[0148] S610: Acquire an audio signal of the user's environment.

[0149] S620: The duration of the audio signal is greater than or equal to a time threshold, and when the audio signal satisfies one or both of the following two conditions, the pickup frequency is reduced: the volume of the audio signal is less than or equal to a volume threshold, and the frequency of the audio signal is within a healthy frequency band.

[0150] Method 600 can be executed by a smartwatch. When the volume of ambient sound is less than or equal to a volume threshold for a period of time, it indicates that the volume of the ambient sound is likely to be less than the volume threshold in the future. When the frequency of the ambient sound is within a healthy frequency range for a period of time, it indicates that the frequency of the ambient sound is likely to be within a healthy frequency range in the future. Therefore, if the ambient sound meets one or both of the above two conditions, the smartwatch can prompt the user to reduce the pickup frequency, thereby reducing the power consumption of ambient sound detection without affecting the user's health.

[0151] For example, if the smartwatch detects that the volume of the ambient sound is less than or equal to the volume threshold within 30 minutes, and the frequency of the ambient sound is in a healthy frequency band, the smartwatch or mobile phone can prompt the user to turn on the energy-saving sound pickup function. Figure 7 After the user chooses to turn on the energy-saving mode, the mobile phone sends a command to the smart watch to turn on the energy-saving mode, and the smart watch reduces the sound pickup frequency to save power.

[0152] The volume threshold can be set by the user according to his or her own feeling, or by the smartwatch based on the healthy volume threshold given by the national health department, for example, it can be set to 80dB; similarly, the range of the healthy frequency band can be set by the user according to his or her own feeling, or by the smartwatch based on the healthy frequency band range given by the national health department, for example, it can be set to 20-20000Hz. The interface diagram of the user setting the volume threshold and frequency band threshold is shown as follows Figure 8 As shown, there is one threshold for the volume setting section and two thresholds for the frequency band setting section. This is because a quiet environment is beneficial to the user's health, so there is no need to set a minimum alarm threshold for the volume. This application does not limit the setting method of the volume threshold and the healthy frequency band.

[0153] Optionally, when the volume of the audio signal is greater than a volume threshold, the smart watch may increase the pickup frequency.

[0154] When the volume of the ambient sound is greater than the volume threshold for a period of time, it means that the volume of the ambient sound is likely to be higher than the volume threshold in the future. In this case, the smart watch can prompt the user to increase the sound pickup frequency so as to remind the user to take protective measures in time.

[0155] For example, if the smartwatch detects that the volume of the ambient sound is greater than 80dB, and the frequency of the ambient sound is less than 20Hz, which is within the harmful frequency band, the smartwatch can remind the user to pay attention to hearing protection through the mobile phone. Figure 9 The phone can alert the user by vibrating, flashing, sounding, or writing text.

[0156] In addition, the mobile phone can also display the detection records of different time periods, such as Figure 9 As shown, the daily ambient sound frequency value or volume value can be displayed, the weekly ambient sound frequency value or volume value can be displayed, and the monthly ambient sound frequency value or volume value can be displayed, so that users can take targeted hearing prevention measures.

[0157] It should be noted that the above reduction or increase of the pickup frequency can also be completed without the smart watch prompting the user, that is, it is automatically completed by the smart watch without the user's perception, thereby improving the user experience.

[0158] The above describes in detail the method for monitoring harmful environmental sounds provided by this application. In some scenarios, although the environmental sounds are not harmful to health, users may still feel uncomfortable. This application also provides a method for evaluating environmental sounds based on user physiological information, so that users can choose a comfortable environment.

[0159] like Figure 10 As shown, the method 1000 includes:

[0160] S1010: Acquire an audio signal of the user's environment.

[0161] S1020: Acquire real-time physiological information of the user, where the acquisition time of the real-time physiological information is the same as the acquisition time of the audio signal.

[0162] S1030: Determine a first comfort level corresponding to the audio signal according to a difference between preset physiological information and the real-time physiological information, wherein the first comfort level is negatively correlated with the difference.

[0163] The real-time physiological information can be heart rate, blood oxygen content, or blood pressure values ​​monitored by the smartwatch. The preset physiological information can be user-set heart rate, blood oxygen content, or blood pressure values, or healthy heart rate, blood oxygen content, or blood pressure values ​​provided by national health authorities. This application does not limit the specific content of the real-time physiological information and the preset physiological information.

[0164] The first comfort level is negatively correlated with the difference, which means that when the difference between the real-time physiological information and the preset physiological information increases, the first comfort level decreases; when the difference between the real-time physiological information and the preset physiological information decreases, the first comfort level increases.

[0165] Taking real-time physiological information as heart rate, for example, if the difference between the heart rate value collected by the smartwatch in real time and the preset heart rate value is large, it indicates that the user's heart rate value deviates significantly from the healthy heart rate value, and it can be determined that the user's comfort level is poor, that is, the first comfort level corresponding to the audio signal is poor. If the difference between the heart rate value collected by the smartwatch in real time and the preset heart rate value is small, it indicates that the user's heart rate value deviates closely from the healthy heart rate value, and it can be determined that the user's comfort level is good, that is, the first comfort level corresponding to the audio signal is good. Therefore, the above solution can prompt the user to change the environment in a timely manner before experiencing serious physiological reactions, thereby maintaining the user's health.

[0166] Different users have different tolerances for ambient sound. When users A and B are in the same environment, their physiological profiles may differ significantly. Therefore, a smartwatch can prompt the user to rate their comfort level with ambient sound and, based on their evaluation, determine preset physiological profiles corresponding to different levels of comfort, enabling decisions more tailored to individual user characteristics.

[0167] For example, a mobile phone can display Figure 11The interface shown prompts the user to rate the comfort level of the current ambient sound; the user can select one of the following based on their current experience: comfortable, average, tolerable, or unbearable. These four comfort levels represent the second comfort level of the audio signal, and the user input represents the first indication. The phone trains the comfort algorithm based on the second comfort level to ensure that the first and second comfort levels are the same. After training is complete, the phone can either stop displaying the user input field and directly provide the predicted result (one of comfortable, average, tolerable, or unbearable). Alternatively, the user input field can be displayed while providing the predicted result to further improve prediction accuracy.

[0168] The above comfort algorithm can be an artificial intelligence model such as a neural network model, a decision tree or a random forest, and this application does not limit this.

[0169] The above example is an implementation of using environmental sound to train a comfort algorithm. In some scenarios, the content of the environmental sound is relatively simple. For example, the environmental sound lacks low-frequency or high-volume sounds. Using environmental sound to train the comfort algorithm may not achieve the desired prediction results. In this case, the mobile phone can control the earphones to actively play low-frequency or high-volume sounds and obtain second indication information input by the user. The second indication information indicates the comfort level of the currently playing sound. This comfort level is the third comfort level and can be one of comfortable, average, tolerable, and unbearable. The mobile phone can train the comfort algorithm based on the third comfort level so that the comfort algorithm's prediction results are consistent with the third comfort level.

[0170] After the comfort algorithm is trained, the phone can prompt the user to take measures to protect their hearing based on the prediction results of the comfort algorithm. For example, when the prediction result is comfortable, the phone reminds the user that the sound comfort level of the environment is ideal and no measures are needed; when the score is average, the phone reminds the user that the sound comfort level of the environment is average and there are potential risks to hearing and physical health, so please take timely protection; when the score is tolerable, the phone reminds the user that the sound level of the environment is bad, please pay attention to your hearing and physical health and stay away from noise sources; when the score is unbearable, please pay attention to your hearing and physical health and quickly stay away from noise sources.

[0171] In addition to monitoring environmental sounds that are harmful to health, smart watches or mobile phones can also improve the user's listening environment when the environmental sounds are harmless.

[0172] Ambient sound is typically a mixed signal from multiple sound sources. After detecting the ambient sound, the smartwatch can use a neural network or other analysis method to determine the multiple sound sources corresponding to the mixed signal and prompt the user to select the sound source to be processed. If the user needs to obtain a specific sound source from the multiple sound sources, the third indication information can be used to indicate that the sound of the specific sound source needs to be enhanced. Alternatively, the third indication information can be used to indicate that the sound emitted by sound sources other than the specific sound source needs to be reduced.

[0173] For example, after the smart watch detects the ambient sound, it can display the following information on the phone: Figure 12 The interface shown prompts the user to select a sound source to be processed from the multiple sound sources. The user can select noise reduction processing and / or enhancement processing according to the current situation.

[0174] If the user is taking a break, he can click on the options corresponding to "background sound" 121, "speech sound" 122 and "whistle sound" 123, that is, select all the sound sources to be processed, and click on the option corresponding to "noise reduction" 125 to perform noise reduction on all the sound sources to be processed. After the mobile phone obtains the information input by the user, it sends a noise reduction instruction to the headset, instructing the headset to perform noise reduction on the audio signals of all sound sources. If the user is participating in an on-site meeting, he can first click on the option corresponding to "background sound" 121, and then click on the option corresponding to "noise reduction" 125. The mobile phone can control the headset to perform noise reduction on the sound emitted by the background sound source in order to obtain a clear speech sound; in addition, the user can also click on the option corresponding to "speech sound" 122, and then click on the option corresponding to "enhancement" 124. The mobile phone can control the headset to enhance the speech sound of the on-site meeting.

[0175] Therefore, this embodiment can process the audio signal based on the needs of the user to enhance the user experience.

[0176] The phone can also predict user actions based on the content of the collected ambient sounds. For example, if the audio signal received from a smartwatch includes speech, the phone can determine that the user is in a conversation and predict that the user wishes to reduce "background noise" and / or "whistle noise" and enhance "speech noise." If the sound source and processing method selected by the user match the phone's prediction, the phone will process the sound according to the user's instructions. If the sound source and processing method selected by the user differ from the phone's prediction, the phone will prompt the user to confirm the selection.

[0177] For example, the user Figure 12 The user performs a touch operation on the interface shown in the figure, which triggers the generation of the third indication information and the fourth indication information. If the third indication information indicates that the sound source to be processed is "speech sound" 122, and the fourth indication information indicates that the selected processing mode is "enhancement" 124, the mobile phone can directly enhance the speech sound of the on-site meeting according to the user's instructions. If the third indication information indicates that the sound source to be processed is "speech sound" 122, and the fourth indication information indicates that the selected processing mode is "noise reduction" 125, the mobile phone can display the following Figure 13The interface shown prompts the user to confirm the selection result again; if the user selects "Yes", the mobile phone can perform noise reduction processing on the speech sound of the on-site meeting according to the user's needs; if the user selects "No", the user is guided to reselect the sound source and processing mode to be processed, thereby avoiding user's incorrect operation.

[0178] In the above example, users can Figure 12 Select one or more sound sources on the interface shown, and select the corresponding processing method for the one or more sound sources. The phone can fill the option box in front of the selected object with color to indicate that the option is selected. If the user only selects the processing method but does not select the sound source to be processed, the phone can Figure 12 The interface shown prompts the user to select a sound source to be processed through a dialog box.

[0179] In some specific scenarios, for example, the user has a hearing impairment or the ambient noise is too complex, and the enhancement or noise reduction effect of the headphones is difficult to achieve the ideal effect, then the mobile phone can prompt the user to move to the location of the sound source of the "speech sound" 122 so that the user can obtain the required information in a timely manner.

[0180] The above describes in detail an example of the method for processing audio signals provided by the present application. It is understandable that, in order to implement the above functions, the corresponding device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0181] The present application can divide the device for processing audio signals into functional units based on the above method examples. For example, each function can be divided into individual functional units, or two or more functions can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units. It should be noted that the division of units in this application is schematic and is only a logical functional division. In actual implementation, other division methods may be used.

[0182] Figure 14 A schematic structural diagram of a device for processing audio signals provided by the present application is shown. Figure 14 The dotted line in the figure indicates that the unit or module is optional. The apparatus 1400 can be used to implement the method described in the above method embodiment. The apparatus 1400 can be a terminal device or a chip.

[0183] Device 1400 includes one or more processors 1401, which can support device 1400 in implementing the method in the method embodiment. Processor 1401 can be a general-purpose processor or a dedicated processor. For example, processor 1401 can be a central processing unit (CPU). The CPU can be used to control device 1400 and execute software programs to implement the function of processing audio signals.

[0184] The processor 1401 may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components. This application does not limit the specific type of processor.

[0185] The apparatus 1400 may further include a communication module 1405 for implementing input (reception) and / or output (transmission) of signals.

[0186] For example, the communication module 1405 may be a transceiver of a terminal device, through which the terminal device sends or receives wireless signals; or, the communication module 1405 may be a communication interface of a chip, through which the chip sends or receives wired signals (such as noise reduction instructions).

[0187] The device 1400 may include one or more memories 1402, on which a program 1404 is stored. The program 1404 can be executed by the processor 1401 to generate instructions 1403, so that the processor 1401 performs the method described in the above method embodiment according to the instructions 1403. Optionally, data (such as various thresholds in the method embodiment) can also be stored in the memory 1402. Optionally, the processor 1401 can also read data stored in the memory 1402. The data can be stored at the same storage address as the program 1404, or the data can be stored at a different storage address from the program 1404.

[0188] The processor 1401 and the memory 1402 may be provided separately or integrated together, for example, integrated on a system on chip (SOC).

[0189] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or a software-based instruction in the processor 1401. The specific manner in which the device 1400 executes the method for processing audio signals and the beneficial effects produced can be found in the relevant description in the method embodiment.

[0190] The present application also provides a computer program product, which, when executed by the processor 1401, implements the method described in any method embodiment of the present application.

[0191] The computer program product may be stored in the memory 1402 , for example, a program 1404 . The program 1404 is converted into an executable target file that can be executed by the processor 1401 after undergoing processes such as preprocessing, compilation, assembly, and linking.

[0192] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method described in any method embodiment of the present application. The computer program can be a high-level language program or an executable target program.

[0193] The computer-readable storage medium is, for example, memory 1402. Memory 1402 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0194] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0195] In the several embodiments provided in this application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not executed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0196] In the various embodiments of the present application, the size of the serial number does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0197] In addition, the term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0198] In short, the above description is only a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the principles of this application should be included in the scope of protection of this application.

Claims

1. A method for processing an audio signal, characterized in that: include: Obtaining audio signals from the user's environment; When the duration of the audio signal is greater than or equal to a time threshold, and when the audio signal satisfies one or both of the following two conditions, reducing the pickup frequency; The volume of the audio signal is less than or equal to a volume threshold, and the frequency of the audio signal is within a healthy frequency band.

2. The method according to claim 1, characterized in that The reducing the pickup frequency comprises: Prompting the user to reduce the pickup frequency; receiving control information input by the user, wherein the control information is used to reduce the sound pickup frequency; The sound pickup frequency is reduced according to the control information.

3. The method according to claim 1 or 2, characterized in that Also includes: A characteristic graph showing changes in volume and / or frequency of the audio signal over time is displayed.

4. The method according to claim 1 or 2, characterized in that Also includes: Acquiring real-time physiological information of the user, where the acquisition time of the real-time physiological information is the same as the acquisition time of the audio signal; A first comfort level corresponding to the audio signal is determined according to a difference between preset physiological information and the real-time physiological information, wherein the first comfort level is negatively correlated with the difference.

5. The method according to claim 4, characterized in that Also includes: Acquire first indication information input by the user, where the first indication information is used to indicate a second comfort level corresponding to the audio signal; A comfort algorithm is trained based on the second comfort level, and the comfort algorithm is used to determine the first comfort level.

6. The method according to claim 4, characterized in that Also includes: Playing a plurality of sounds, the plurality of sounds having different volumes and frequencies; obtaining second indication information input by the user, where the second indication information is used to indicate a third comfort level corresponding to the multiple sounds; A comfort algorithm is trained based on the third comfort level, and the comfort algorithm is used to determine the first comfort level.

7. The method according to claim 1 or 2, characterized in that Also includes: When the frequency band of the audio signal is within a harmful frequency band, the user is prompted to perform protective processing.

8. The method according to claim 1 or 2, characterized in that Also includes: When the audio signal is a mixed signal of multiple sound sources, prompting the user to select a sound source to be processed from the multiple sound sources; Acquiring third indication information input by the user, where the third indication information is used to indicate the sound source to be processed; Acquiring fourth indication information input by the user, where the fourth indication information is used to indicate a processing method for the sound source to be processed, where the processing method includes noise reduction processing or enhancement processing; Noise reduction processing or enhancement processing is performed on the audio signal corresponding to the sound source to be processed according to the third indication information and the fourth indication information.

9. The method according to claim 8, characterized in that Also includes: When the sound source to be processed does not match the processing method, the user is prompted to confirm whether the sound source to be processed and the processing method are correct.

10. The method according to claim 1 or 2, characterized in that The obtaining of the audio signal of the user's environment includes: The audio signal is obtained through multiple IoT devices.

11. A device for processing an audio signal, characterized in that: The apparatus comprises a processor and a memory, the processor and the memory being coupled, the memory being used to store a computer program, and when the computer program is executed by the processor, the apparatus causes the apparatus to perform the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 10.

13. A chip, characterized in that: The method comprises a processor, and when the processor executes instructions, the processor performs the method according to any one of claims 1 to 10.

Images