Electronic device and audio noise reduction method and medium thereof

Abstract

The embodiments of the present application provide an audio noise reduction method and its electronic device and medium, which relate to speech recognition technology in the field of artificial intelligence (AI). The audio noise reduction method of the present application includes: after a first electronic device collects various modal data (for example, event sensing data, environmental images and environmental sounds), the first electronic device calculates the single-mode scene probability of each modal data indicating that the first electronic device is in a first scene; at the same time, the first electronic device determines a plurality of weights corresponding to each modal data, and based on the plurality of weights, fuses the different single-mode scene probabilities corresponding to each modal data to obtain a fused scene probability indicating that the first electronic device is in the first scene. When the fused scene probability meets the recognition condition of the first scene, the first electronic device determines that the audio noise reduction mode is the noise reduction mode corresponding to the first scene.

Description

[0001] This application claims priority to Chinese patent application number "202010759885.9" filed on July 31, 2020, and application name "Electronic device and its audio noise reduction method and medium", the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to speech recognition technology in the field of artificial intelligence, and more particularly to an electronic device and its audio noise reduction method and medium. Background Art

[0003] The current active noise reduction technology of headphones can only perform scene recognition based on single-modal data (for example, ambient sound) and then set the noise reduction mode for the headphones. However, electronic devices connected to headphones are equipped with multiple sensors, which can collect various modal data of the current environment (for example, event sensing data, environmental images and ambient sounds) through sensors and built-in applications. There is an urgent need for a technology that can fuse and calculate various modal data, identify the scene of the current environment, and then set the noise reduction mode for the headphones.

[0004] At the same time, existing fusion calculation technologies for modal data assign fixed weights to each modal data point, or manually adjust weights, without using artificial intelligence technology to adjust weights. This over-reliance on expert knowledge during fusion calculations fails to consider modal data bias and dynamic changes. As a result, fusion calculation results are often inaccurate. Summary of the Invention

[0005] Embodiments of the present application provide an electronic device and an audio noise reduction method and medium thereof to improve the flexibility and accuracy of audio noise reduction.

[0006] A first aspect of the present application provides an audio noise reduction method for an electronic device, comprising: obtaining event sensing data of the electronic device; if the scene type of the electronic device cannot be determined based on the event sensing data, obtaining audio and scene images currently collected by the electronic device, and determining the scene type of the electronic device based on audio features of the obtained audio and image features of the scene image; and determining an audio noise reduction mode of the electronic device based on the determined scene type.

[0007] In a possible implementation of the first aspect, when the scene type in which the electronic device is located is determined according to the event sensing data, an audio noise reduction mode of the electronic device is determined according to the determined scene type.

[0008] In this method, the electronic device can be any terminal device, such as a mobile phone. After a user opens a subway app on their phone and scans a QR code to enter a subway station, the phone monitors the app to obtain event-sensing data. Based on this event-sensing data, the phone determines that the phone is currently in a subway scene. After obtaining the audio noise reduction mode corresponding to the subway scene, the phone configures the noise-cancelling headphones with that audio noise reduction mode to achieve audio noise reduction.

[0009] After a user swipes their bus card to enter a subway station, the phone cannot capture event-sensing data. Instead, it uses its own camera and the noise-canceling headphones' microphone to collect the subway scene's noise and image. The scene noise here refers to audio. The phone extracts audio and image features from the scene noise and image, respectively. Based on these features, the phone determines it is currently in a subway scene and configures the noise-canceling headphones with an audio noise reduction mode appropriate for the subway scenario to achieve audio noise reduction.

[0010] In a possible implementation of the first aspect, the event sensing data is a result of the electronic device detecting that an application of the electronic device is running.

[0011] In a possible implementation of the first aspect above, when the result of running the application is that the shared bicycle application of the electronic device runs the scan code to ride, it is determined that the scene type of the electronic device is an outdoor riding scene.

[0012] In a possible implementation of the first aspect, the event sensing data is an operating result of a sensor of the electronic device.

[0013] In a possible implementation of the first aspect, when the sensor is an air pressure sensor, when the operating result of the air pressure sensor is a low pressure environment, it is determined that the scene type of the electronic device is a low pressure scene.

[0014] In this method, in addition to monitoring the applications installed on the mobile phone, the mobile phone can also obtain event sensing data by monitoring the operating results of its own sensors.

[0015] In a possible implementation of the first aspect, the audio features of the audio are obtained in the following manner:

[0016] Separating a first frequency band portion and a second frequency band portion from the audio using a linear predictor, wherein the frequency band included in the first frequency band portion has a lower frequency than the frequency band included in the second frequency band portion;

[0017] At least one of the first audio feature extracted from the first frequency band portion and the second audio feature extracted from the second frequency band portion is used as the audio feature of the audio.

[0018] In a possible implementation of the first aspect, the first audio feature and the second audio feature are fused to obtain the audio feature.

[0019] In this method, audio recognition is performed simultaneously using the first audio feature and the second audio feature through fusion. For example, when the first audio feature and the second audio feature are MFCC feature parameters and time-frequency feature parameters, respectively, the two can be linearly fused to form a feature vector. Alternatively, the two can be normalized and then linearly fused, or they can be weighted and then linearly fused to form a feature vector. The eigenvalue corresponding to the eigenvector is calculated. If the difference between the calculated eigenvalue and the eigenvalue corresponding to the second audio type is greater than the second matching threshold, the type of the audio is the second audio type.

[0020] In a possible implementation of the first aspect, the first audio feature is extracted from the first frequency band using an audio feature extraction algorithm that simulates the perception capability of the human cochlea.

[0021] The audio feature extraction algorithm that simulates the perception ability of the human cochlea is a Mel-frequency cepstral coefficient MFCC extraction method, and the first audio feature is a Mel-frequency cepstral coefficient MFCC.

[0022] Mel-frequency cepstral coefficients (MFCCs) are extracted from the vocal tract signals in the audio, so that the extracted MFCC feature parameters are free from interference from high-frequency harmonics and can better describe the vocal tract characteristics of the sound-emitting object of the audio.

[0023] In a possible implementation of the first aspect, the second audio feature is extracted from the second frequency band portion through wavelet transform, wherein the second audio feature is a time-frequency feature obtained through wavelet transform.

[0024] By extracting time-frequency feature parameters of the sound source signal separated from the audio by the linear predictor through multi-scale wavelet transform, the sound source characteristics of the sound-emitting object of the audio can be effectively characterized.

[0025] In a possible implementation of the first aspect, the first frequency band partially characterizes characteristics of a sound channel of a sound-emitting object that emits audio, and the second frequency band partially characterizes characteristics of a sound source of the sound-emitting object.

[0026] In a possible implementation of the first aspect, separating the first frequency band portion and the second frequency band portion from the audio using a linear predictor includes:

[0027] A first frequency band portion is separated from the audio through a linear predictor, and the remaining portion of the audio after the first frequency band portion is separated is used as a second frequency band portion.

[0028] In a possible implementation of the first aspect, the scene image is converted into a conversion image in an RGB color space format.

[0029] In a possible implementation of the first aspect, image features are extracted from the converted image using an image recognition neural network model.

[0030] In a possible implementation of the first aspect, the audio and scene images are framed according to a preset framing threshold so that the dimensions of the audio features and the image features are the same.

[0031] In a possible implementation of the first aspect, determining the scene type in which the electronic device is located based on the acquired audio features of the audio and image features of the scene image includes:

[0032] fusing the audio features and the image features to obtain a fused audio feature, matching the fused audio feature with a scene feature corresponding to the scene type, and determining the scene type in which the electronic device is located when a matching degree is greater than a matching degree threshold;

[0033] When the scene type in which the electronic device is located is determined according to the fused audio features, the audio noise reduction mode of the electronic device is determined according to the determined scene type.

[0034] For example, a user wearing noise-canceling headphones takes the subway. The user enters the subway by swiping a bus card. At this time, the mobile phone cannot obtain the scene through event sensing data. Therefore, the noise-canceling headphones use the microphone to collect audio in the subway. When the audio intensity exceeds the preset sound intensity threshold in the noise-canceling headphones, the noise-canceling headphones extract MFCC feature parameters from the audio. At the same time, the mobile phone's camera captures the scene image of the subway and extracts image features from the scene image. Finally, based on the audio and image features, the scene type is identified, that is, the subway scene, and the noise-canceling headphones are configured to perform noise reduction according to the noise reduction mode corresponding to the subway scene.

[0035] A second aspect of the present application provides an electronic device, characterized by comprising:

[0036] The processor is coupled to a memory, and the memory stores program instructions. When the program instructions stored in the memory are executed by the processor, the electronic device executes any one of the methods provided in the first aspect.

[0037] The third aspect of the present application provides a computer-readable medium, characterized in that the computer-readable medium stores instructions, which, when executed on a computer, enable the computer to execute any one of the methods provided in the first aspect.

[0038] A fourth aspect of the present application provides an audio noise reduction method for a first electronic device, characterized by comprising:

[0039] The first electronic device acquires data of multiple modalities for scene recognition;

[0040] The first electronic device calculates, respectively, data of each modality indicating a single-mode scenario probability that the first electronic device is in the first scenario;

[0041] The first electronic device determines multiple weights corresponding to the modal data, and based on the multiple weights, fuses different single-mode scene probabilities corresponding to the modal data to obtain a fused scene probability indicating that the first electronic device is in the first scene, wherein different weights are used for fusing data of the same modality if the data falls within different preset data ranges;

[0042] When the fused scene probability satisfies the recognition condition of the first scene, the first electronic device determines that the audio noise reduction mode is the noise reduction mode corresponding to the first scene.

[0043] In this method, the first electronic device can be a mobile phone. In addition to the audio and image data in the first aspect mentioned above, the first electronic device can also perform scene recognition by collecting various modal data (for example, event sensing data, environmental images and environmental sounds). After the collection is completed, the various modal data are fused through the trained model, and the scene recognition result is obtained based on the fusion result.

[0044] In a possible implementation of the fourth aspect, the data of the multiple modalities includes at least one of the following:

[0045] The running results of the application running on the first electronic device, the running results of the sensor on the first electronic device, the detection data of the sensor on the first electronic device, the image data collected by the first electronic device, the audio data collected by the first electronic device, the WiFi signal received by the first electronic device, and the base station signal of the base station connected to the first electronic device for communication.

[0046] In a possible implementation of the fourth aspect, the application program operation result includes a shared bicycle application program executing a scan code to ride, and a subway travel application program executing a scan code to enter a subway station; and

[0047] The operating result of the sensor on the first electronic device includes, when the sensor is a heart rate sensor, an operating result of the heart rate sensor indicating a sleep state; and

[0048] The detection data of the sensor on the first electronic device includes at least one of light data collected by the light sensor, acceleration of the first electronic device collected by the acceleration sensor, and air pressure data collected by the air pressure sensor.

[0049] In a possible implementation of the fourth aspect above, the multiple modalities include a first modality, and the multiple preset data ranges corresponding to the first modality include a first preset data range and a second preset data range, wherein when calculating the fusion scene probability of the first scene, the single-mode scene probability of the first modal data belonging to the first preset data range is greater than the single-mode scene probability of the first modal data belonging to the second preset data range, and the weight corresponding to the first preset data range is greater than the weight corresponding to the second preset data range.

[0050] In this method, taking the modal data as illumination data as an example, the modal data can be divided into intervals, for example, the modal data can be divided into a first preset data range and a second preset data range, and the single-mode scene probability of the first preset data range and the second preset data range can be obtained respectively.

[0051] In a possible implementation of the fourth aspect above, the first electronic device fuses different single-mode scene probabilities corresponding to each modal data based on a neural network model to obtain a fused scene probability indicating that the first electronic device is in the first scene.

[0052] In a possible implementation of the fourth aspect, a weight mapping sub-model for determining the weights of each modal data in the neural network model is trained in the following manner:

[0053] Obtaining a plurality of sample data belonging to the first modality and expected single-mode scene probabilities corresponding to the plurality of sample data respectively;

[0054] Arrange the plurality of sample data belonging to the first mode in descending order according to the value;

[0055] Dividing the sorted sample data into a plurality of preset data ranges according to a predetermined number, wherein the plurality of preset data ranges include a first preset data range and a second preset data range;

[0056] Calculate the average or median value of the preset range of each data, and input the average or median value into the weight mapping sub-model;

[0057] The coefficients of the weight mapping sub-model are adjusted by comparing the output of the weight mapping sub-model with the expected unimodal scenario probability.

[0058] In this method, taking illumination data as an example, the illumination data can be arranged in ascending or descending order to reduce the dispersion of the illumination data. Furthermore, the illumination data can be equally divided into a first preset data range (k1) and a second preset data range (k2) based on the amount of illumination data, thereby preventing excessive concentration of illumination data. Weights w1 and w2 corresponding to the first preset data range (k1) and the second preset data range (k2) can be trained, respectively.

[0059] In one possible implementation of the fourth aspect, before determining multiple weights corresponding to each modal data and fusing the different single-mode scene probabilities corresponding to each modal data based on the multiple weights, the single-mode scene probabilities are screened to remove single-mode scene probabilities whose values ​​fall within the median range of the single-mode scene probabilities.

[0060] In a possible implementation of the fourth aspect, the method further includes:

[0061] The first electronic device sends a noise reduction mode to the second electronic device.

[0062] In a possible implementation of the fourth aspect, the first electronic device is a mobile phone, and the second electronic device is a headset.

[0063] In a possible implementation of the fourth aspect, the first electronic device is capable of acquiring data in M ​​modalities through the first mode and acquiring data in N modalities through the second mode, where M and N are positive integers and M is less than or equal to N; and

[0064] In the first mode, the number of devices enabled when the first electronic device obtains M modal data is less than the number of devices enabled when the first electronic device obtains N modal data in the second mode.

[0065] In this method, when a mobile phone identifies an outdoor scene, the mobile phone can collect light data through a light sensor in a low-power mode (first mode). When the light data meets the accuracy threshold, the mobile phone starts a high-power collection mode (second mode). In addition to collecting modal data corresponding to scene sounds and scene images through a microphone and camera, the mobile phone can also collect acceleration data through an accelerometer and air pressure data through a pressure sensor to identify outdoor scenes.

[0066] A fifth aspect of the present application provides an audio noise reduction method for a first electronic device, characterized by comprising:

[0067] The first electronic device continuously collects first data of the first scene in a low-power collection manner;

[0068] When the first data meets a preset first accuracy threshold, the first electronic device collects second data of multiple modalities for identifying the first scene in a high-power consumption collection manner;

[0069] The first electronic device calculates respectively the second data of the multiple modes to indicate the single-mode scenario probability that the first electronic device is in the first scenario;

[0070] The first electronic device determines a plurality of weights corresponding to the second data, and based on the plurality of weights, fuses different single-mode scene probabilities corresponding to the second data of the plurality of modalities to obtain a fused scene probability indicating that the first electronic device is in the first scene;

[0071] When the fused scene probability satisfies the recognition condition of the first scene, the first electronic device determines that the audio noise reduction mode is the noise reduction mode corresponding to the first scene.

[0072] A sixth aspect of the present application provides an electronic device, characterized by comprising:

[0073] a memory storing instructions; and

[0074] At least one processor is configured to access a memory and to execute instructions on the memory to perform operations, the operations comprising:

[0075] Acquire data from multiple modalities for scene recognition;

[0076] Calculating the single-mode scenario probability that the data of each mode indicates that the electronic device is in the first scenario;

[0077] Determining multiple weights corresponding to each modal data, and fusing different single-mode scene probabilities corresponding to each modal data based on the multiple weights to obtain a fused scene probability indicating that the electronic device is in a first scene, wherein different weights are used for fusing data of the same modality if the data falls within different preset data ranges;

[0078] When the fusion scene probability satisfies the recognition condition of the first scene, the audio noise reduction mode is determined to be the noise reduction mode corresponding to the first scene.

[0079] The seventh aspect of the present application is a computer-readable medium, characterized in that instructions are stored on the computer-readable medium, and when the instructions are executed on a computer, the computer executes the audio noise reduction method of the fourth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 According to some embodiments of the present application, a scenario of audio noise reduction provided by the present application is shown;

[0081] Figure 2a According to some embodiments of the present application, a hardware structure diagram of an audio noise reduction system provided by the present application is shown;

[0082] Figure 2b According to some embodiments of the present application, another hardware structure diagram of the audio noise reduction system provided by the present application is shown;

[0083] Figure 3aAccording to some embodiments of the present application, a process of audio noise reduction is shown;

[0084] Figure 3b According to some embodiments of the present application, a process of obtaining a scene probability vector corresponding to the scene noise from the scene noise is shown;

[0085] Figure 3c According to some embodiments of the present application, a process of obtaining a scene probability vector corresponding to a scene image from a scene image is shown;

[0086] Figure 3d According to some embodiments of the present application, a process of fusing a scene probability vector of scene noise and a scene probability vector of a scene image to calculate the probability of a fused scene is shown;

[0087] Figure 4 According to some embodiments of the present application, another scenario of audio noise reduction provided by the present application is shown;

[0088] Figure 5 According to some embodiments of the present application, a schematic structural diagram of a scene recognition device is shown;

[0089] Figure 6 According to some embodiments of the present application, a schematic diagram of a multimodal model for fusion computing is shown;

[0090] Figure 7 According to some embodiments of the present application, a flowchart of training a weight mapping sub-model in a multimodal model is shown;

[0091] Figure 8 According to some embodiments of the present application, a schematic diagram of using illumination data to train a weight mapping sub-model in a multimodal model is shown;

[0092] Figure 9 According to some embodiments of the present application, a flow chart of performing fusion calculation using a multimodal model is shown;

[0093] Figure 10 According to some embodiments of the present application, a schematic diagram of obtaining weights of each modality data through a multimodal model is shown;

[0094] Figure 11 According to some embodiments of the present application, another audio noise reduction process is shown;

[0095] Figure 12 According to some embodiments of the present application, a module structure diagram of an electronic device is shown;

[0096] Figure 13According to some embodiments of the present application, a schematic structural diagram of a terminal device is shown. DETAILED DESCRIPTION

[0097] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0098] Figure 1 According to some embodiments of the present application, an application scenario diagram of an audio noise reduction method is shown. Figure 1 As shown, when the user wears the headset 200 and the headset 200 is in communication with the terminal device 100, the terminal device 100 identifies the current scene by obtaining relevant data of the user's environment. For example, in some embodiments, the terminal device 100 first obtains event sensing data (such as sensed outdoor cycling events, subway ride events, sleep events, etc.) by monitoring the running results of its own installed application (Application, APP) or internal sensors, and identifies the current scene based on the event sensing data. If the scene is not identified through the event sensing data, the terminal device 100 collects the scene noise and scene image of the current environment through the headset 200 or its own microphone, and its own camera respectively. The terminal device 100 sends the above-mentioned scene noise and scene image to the server 300. After the server 300 identifies the current scene based on the scene noise and scene image, it obtains the noise reduction mode corresponding to the scene according to the current scene and returns it to the mobile phone 100. Then, the mobile phone 100 sets the noise reduction mode for the headset 200 to achieve the noise reduction effect. In some embodiments of the present application, the environmental noise and environmental image can be extracted for the current environment where the user is located, respectively, and the corresponding probability vectors are obtained based on the environmental noise and the environmental image. After the probability vectors of the two are fused and calculated, the scene corresponding to the current environment is identified. Through the above implementation, the scene corresponding to the current environment can be more accurately identified, and the noise reduction mode corresponding to the scene can be used for noise reduction. For another example, in some other embodiments of the present application, the server 300 can also obtain the single-mode scene probability of a certain scene represented by each single-mode model data based on each modal data including the above-mentioned event sensing data, environmental noise and environmental image, and then identify the scene corresponding to the current environment after the single-mode scene probability is fused and calculated by the multimodal fusion model.

[0099] The noise reduction mode in this application refers to a sound mode set by the terminal device 100 for the headset 200, through which a sound signal is generated to combat noise to eliminate it.

[0100] The above-mentioned scene recognition based on scene noise and scene image includes: collecting scene noise and collecting scene images, extracting scene noise features and scene image features from the scene noise and scene image, respectively, inputting the scene noise features and scene image features into a noise neural network model and an image neural network model, obtaining a scene probability vector for the scene noise and a scene probability vector for the scene image, fusing the two scene probability vectors through an algorithm to obtain a fused scene probability vector, and determining the scene in which the user is located based on the fused scene probability vector. Here, the scene probability vector for the scene noise and the scene probability vector for the scene image refer to the probability of the corresponding scene represented by the scene noise and the scene image in the form of a vector; the fused scene probability vector refers to the probability of the corresponding scene obtained by fusing the scene probability vector for the scene noise and the scene image. In the noise neural network model and the image neural network model, at least one multidimensional scene probability vector can be obtained based on the input scene noise and scene image features. Each dimension in the scene probability vector represents a scene probability.

[0101] The following combination Figures 2a to 4 The audio noise reduction method provided by some embodiments of the present application is described in detail.

[0102] Example 1

[0103] Figure 2a An audio noise reduction system 10 according to an embodiment of the present application is shown. The system 10 includes: a terminal device 100, a headset 200 and a server 300.

[0104] It is understood that the terminal device 100 may be able to use various electronic devices of the headset, for example, including but not limited to laptop computers, desktop computers, tablet computers, mobile phones, servers, wearable devices, head-mounted displays, mobile email devices, portable game consoles, portable music players, reader devices, or other electronic devices capable of accessing the Internet. In some embodiments, the embodiments of the present application may also be applied to wearable devices worn by users. For example, smart watches, bracelets, jewelry (for example, devices made into decorative items such as earrings, bracelets, etc.) or glasses, etc., or as part of a watch, bracelet, jewelry or glasses, etc.

[0105] The earphone 200 may include, but is not limited to, various analog earphones or digital earphones with a microphone 201 and a speaker 202. It may be a wired earphone or a wireless earphone.

[0106] The server 300 may be a hardware server or embedded in a virtualized environment. For example, according to some embodiments of the present application, the server 300 may be a virtual machine executed on a hardware server that includes one or more other virtual machines. According to some embodiments of the present application, the server 300 may interact with the terminal device 100 via a network, for example, sending data to the terminal device 100 and / or receiving data from the terminal device 100.

[0107] The terminal device 100 can identify the current scene by acquiring event sensing data. For example, if the terminal device 100 detects that the user has swiped a card to enter the station using the installed subway travel APP, the terminal device 100 determines that the user is now in the subway scene. For another example, if the terminal device 100 obtains the user's movement speed close to the running speed over a period of time through the sensor 103 based on the Global Positioning System (GPS), the terminal device 100 determines that the user is now in an outdoor running scene. In some embodiments, when the terminal device 100 cannot directly obtain data related to the current scene by monitoring the running results of its own installed applications or internal sensors, the terminal device 100 can also collect the scene noise and scene image of the current environment through the microphone 201 of the headset 200 or its own microphone 101 and its own camera 102. The scene noise and scene image are sent to the server 300. The server 300 extracts features from the scene noise and scene image respectively, inputs the features into the scene search model, obtains the scene probability vectors corresponding to the scene noise and scene image respectively through the scene search model, and obtains the fused scene probability vector after the scene probability vector is subjected to the fusion algorithm. The current scene is judged based on the fused scene probability vector.

[0108] After the server 300 determines the current scene, it sends the current scene to the terminal device 100. The terminal device 100 obtains and sets the noise reduction mode corresponding to the scene on the headset 200 to achieve the noise reduction effect. The terminal device 100 can obtain the noise reduction mode corresponding to the scene from the server 300. In some embodiments, the noise reduction mode corresponding to the scene can also be stored in the memory of the terminal device 100.

[0109] In some embodiments, the server 300 may first train a noise neural network model for identifying scene noise and an image neural network model for identifying scene graphics. Specifically, the terminal device 100 collects the scene noise and scene image of the current environment, and then sends the scene noise and scene image to the server 300. The server 300 extracts scene noise features from the scene noise and scene image features from the scene image, and then inputs the noise neural network model and the image neural network model respectively to obtain a scene probability vector for the scene noise and a scene probability vector for the scene image. After the two scene probability vectors are fused by an algorithm, a fused scene probability vector is obtained, and the current scene is determined based on the fused scene probability vector.

[0110] In some embodiments, the server 300 may also save, collect, and update various scenes and the noise reduction modes corresponding to the scenes. In some embodiments, the server 300 collects various scenes and the noise reduction modes corresponding to the scenes from the various terminal devices 100 communicating with it or from the developers of the various noise reduction modes. For example, when the server 200 detects that the scenes and the noise reduction modes corresponding to the scenes saved by the terminal device 100 communicating with it are different in number or type from those saved in the server 300, the server 300 may obtain the scenes and the noise reduction modes corresponding to the scenes from the terminal device 100. For another example, the developer of the noise reduction mode may regularly update the newly created scenes and the noise reduction modes corresponding to the scenes to the server 300.

[0111] In addition, in some embodiments, the server 300 may synchronize scenes for each terminal device 100 based on a request from the terminal device 100 or by periodic push notifications. The server 300 may also synchronize the noise reduction mode corresponding to the scene for each existing scene on each terminal device 100. For example, if a developer updates a scene and the noise reduction mode corresponding to the scene and reports it to the server 300, the server 300 may push these updated scenes and the noise reduction mode corresponding to the scene to each terminal device 100. Alternatively, the server 300 may periodically detect the scene and the noise reduction mode corresponding to the scene being used by the terminal device 100. After the detection is complete, the server 300 may send the latest scene and the noise reduction mode corresponding to the scene to the terminal device 100. After the terminal device 100 obtains the latest scene and the noise reduction mode corresponding to the scene, it may update and save them in the terminal device 100's memory and use the obtained scene and the noise reduction mode corresponding to the scene to perform noise reduction.

[0112] Of course, the above system 10 is only one specific embodiment of the present application. In some embodiments, such as Figure 2bAs shown, the mobile phone 100 may also not interact with the server 300, but only use the processor 104 to execute the noise and image neural network model and scene fusion algorithm set in its own memory 105 to identify the current scene by itself, and obtain the noise reduction mode corresponding to the current scene stored in its own memory through the current scene.

[0113] The following uses a mobile phone 100 as an example of a terminal device 100. Figure 3a -d to Figure 4 Let's explain the audio noise reduction method in detail.

[0114] Figure 3a It shows a process in which the mobile phone 100 recognizes the scene in which it is located through the earphone 200 and implements noise reduction based on the scene.

[0115] exist Figure 3a In the illustrated embodiment, the scene and the noise reduction mode corresponding to the scene can be stored in the memory of the mobile phone 100, or can be obtained and stored by the mobile phone 100 from the server 300, or can be stored in the server 300 and queried by the mobile phone 100 from the server 300. Here, the scene and the noise reduction mode corresponding to the scene are stored in the memory of the mobile phone 100 in the form of a correspondence between specific scene information and its noise reduction mode data. The above correspondence can also be stored in the form of a table, a data file, or a database.

[0116] like Figure 3a As shown, the technical solution for achieving audio noise reduction includes:

[0117] S301: After the user wears the earphone 200, the mobile phone 100 detects that the earphone 200 is connected and the earphone 200 is in working state. When it is detected that the earphone 200 is connected to the mobile phone 100 and is in working state, the process proceeds to S302.

[0118] For example, the headset 200 is connected to the mobile phone 100 via a USB interface of the mobile phone 100, where the USB interface may be a micro universal serial bus (USB) Type C interface. In other embodiments of the present application, the headset 200 may also establish a communication connection with the mobile phone 100 via Bluetooth or Wi-Fi, etc., which is not limited here. For example, after the user turns on the mobile phone 100, the user can enter the music app by clicking on the music app icon. When the mobile phone 100 detects that the music app starts playing music, it confirms that the headset 200 is in working state and proceeds to S302.

[0119] S302: The mobile phone 100 obtains event sensing data by monitoring the running results of the APP installed on itself or the internal sensors, and obtains the scene according to the event sensing data. The scene can be obtained from the memory of the mobile phone 100 or directly queried from the server 300.

[0120] If the scene is determined and the noise reduction mode is found, the process proceeds to S303: the mobile phone 100 performs noise reduction by setting the noise reduction mode. If no noise reduction mode is found or the mobile phone 100 does not detect the scene, the process proceeds to S304 and S305.

[0121] For example, a user enters a subway station by scanning a QR code through the subway travel app installed on the mobile phone 100. At this time, the mobile phone 100 detects that the user clicks to open the subway travel app and executes the command to scan the QR code to enter the station through the subway travel app. It confirms that the current event sensing data is opening the subway boarding QR code and scanning the QR code to enter the subway station. At the same time, the scene corresponding to the event sensing data obtained by the mobile phone 100 is the subway scene.

[0122] For another example, a user scans a shared bike code through a shared bike app installed on mobile phone 100. At this point, mobile phone 100 detects that the user clicks to open the shared bike app and executes the command to scan the code to open the shared bike through the shared bike app. It then confirms that the current event sensing data is opening the shared bike app and that the scene corresponding to the event sensing data obtained by mobile phone 100 is an outdoor cycling scene.

[0123] In another embodiment, the mobile phone 100 obtains the user's heart rate changes in real time through the heart rate sensor of the smart bracelet to which it communicates. When the heart rate change information is consistent with the heart rate information of the sleeping state, the mobile phone 100 confirms that the current scene is a sleeping scene.

[0124] Other situations also include that when the mobile phone 100 detects through its built-in air pressure sensor that the user is currently in a low-pressure environment, the mobile phone 100 confirms that the current scene is a low-pressure scene.

[0125] The relationship between various event sensing data and corresponding output scenarios is shown in Table 1.

[0126] Event sensing data Scenario Open the shared bicycle app or mini program to scan the QR code and ride Outdoor cycling scene Open the subway and scan the QR code to enter the subway station Subway scene Can watches / bracelets respond to sleep events Sleep scene Pressure sensor senses low air pressure Low pressure scenario

[0127] Subsequently, the mobile phone 100 can search for a noise reduction mode corresponding to the scene from the noise reduction modes corresponding to the scene stored in its own memory. If the scene and the noise reduction modes corresponding to the scene are stored in a table, the mobile phone 100 can search for the noise reduction mode corresponding to the scene. It is understood that the mobile phone 100 can also search for a noise reduction mode from the noise reduction modes corresponding to the scene stored on the server 300.

[0128] As mentioned above, the noise reduction mode corresponding to the scene can also be stored on the server 300. Usually, some scenes and noise reduction modes corresponding to the scenes are pre-configured in the mobile phone 100 or in the server 300. These scenes can cover most environments in daily life, so that the mobile phone 100 can switch the noise reduction mode in time according to the pre-configured noise reduction mode. However, in some embodiments, the event sensing data and the scene corresponding to the event sensing data are not pre-configured in the mobile phone 100 or the server 300. The mobile phone 100 can collect scene noise and scene images, identify the scene corresponding to the scene noise and scene image, and finally obtain the noise reduction mode corresponding to the scene, and then achieve the noise reduction effect by setting the noise reduction mode for the headset 200.

[0129] The situation mentioned above where the mobile phone 100 fails to detect the scene may be as follows: Figure 4 The user shown enters the subway station by swiping the bus card, and the mobile phone 100 cannot obtain the scene through event sensing data. At this time, the mobile phone 100 collects scene noise and scene images through the microphone 201 of the headset 200 or the microphone 101 and camera 102 of the mobile phone 100.

[0130] S304: Mobile phone 100 collects the current scene noise and obtains a scene probability vector corresponding to the scene noise by performing feature extraction and scene recognition. The scene probability vector corresponding to the scene noise can be in the form of an N-dimensional vector, where each dimension of the scene probability vector represents the probability of a specific scene. This will be described in detail below.

[0131] S305: Mobile phone 100 captures the current scene image and obtains a scene probability vector for the scene image by performing feature extraction and scene image recognition. The scene probability vector corresponding to the scene image is in the same format as the scene probability vector corresponding to the scene noise described above. This will be described in detail below.

[0132] S306: The mobile phone 100 fuses the scene probability vector of the scene noise and the scene probability vector of the scene image to calculate the probability of the fused scene, and obtains the current scene corresponding to the probability of the fused scene. The scene probability vectors of the scene noise and the scene image can be calculated using a fusion algorithm that transposes and multiplies the matrix to obtain the probability value of the fused scene. The above fusion algorithm can make the probability value more accurate by fusing the probability matrices of the scene noise and the scene image after framing within the same period of time.

[0133] After obtaining the probability of the fused scene, the phone 100 proceeds to S307, where it obtains the noise reduction mode corresponding to the scene and sets it on the headset 200. Simultaneously, the phone 100 proceeds to S308, where it monitors changes in the scene or noise in real time and adjusts the noise reduction mode accordingly. The technical solutions for S307 and S308 are described in detail below.

[0134] Regarding the above step 304, in some embodiments, the method of calculating the scene probability vector corresponding to the scene noise is as follows: Figure 3b Specifically including:

[0135] S304a: The mobile phone 100 determines whether the current scene noise exceeds the preset sound intensity threshold. If the current scene noise exceeds the preset sound intensity threshold, the process proceeds to S304b; otherwise, the process proceeds to S304c.

[0136] The mobile phone 100 can collect the current scene noise through the microphone 201 of the connected headset 200 and convert the collected noise signal into a digital signal.

[0137] In some embodiments, the mobile phone 100 can also collect the current scene noise through a noise collection device including a microphone. The noise collection device can be a part of the mobile phone 100, or an independent device, which sends the collected noise signal to the mobile phone 100 by communicating with the mobile phone 100. For example, the noise collection device can be a pickup, a recording microphone, etc.

[0138] While the mobile phone 100 collects the current scene noise through the microphone 201 of the connected headset 200, the mobile phone 100 can perform a sound intensity level detection on the collected scene noise. If the sound intensity of the scene noise does not exceed a preset first sound intensity threshold, it can be considered that the current environment is similar to an indoor environment with less noise. Therefore, the scene probability vector of the scene noise is determined to be the scene probability vector of the scene noise corresponding to an indoor scene. The mobile phone 100 can obtain an indoor noise reduction mode corresponding to the indoor scene and configure the headset 200 to perform noise reduction. The method for detecting the sound intensity level of the mobile phone 100 includes: the mobile phone 100 can receive a voltage signal of the scene noise through the microphone of the headset 100, amplify the signal through an amplifier connected to the microphone, then convert the amplified signal into a digital signal through an analog-to-digital converter (ADC), and then search the decibel value corresponding to the value of the digital signal from the decibel value comparison table. The decibel value comparison table can be stored in the memory of the mobile phone 100. For example, when the mobile phone 100 detects that the decibel value of the scene noise of the current scene is 30dB and the first sound intensity threshold is 35dB, the mobile phone 100 can directly confirm that the scene probability vector of the scene noise is the scene probability vector of the scene noise corresponding to the indoor scene.

[0139] If the sound intensity level of the current scene noise exceeds the first sound intensity threshold, the mobile phone 100 collects the current scene noise and obtains a scene probability vector corresponding to the scene noise by performing feature extraction and scene recognition on the scene noise.

[0140] S304c: Determine that the current scene is an indoor scene.

[0141] S304b: The mobile phone 100 obtains speech features of the scene noise.

[0142] Mobile phone 100 extracts features from the noise signal of the scene noise to obtain speech features of the scene noise, including: filtering the noise through a filter in mobile phone 100 to obtain a spectrum of each frame of the noise signal in the scene noise; mapping the frequency information and intensity information indicated by the spectrum of each frame of the signal into a two-dimensional vector; and arranging multiple two-dimensional vectors in chronological order to obtain speech features of the scene noise. In the case of multiple speech features, the speech features of the scene noise are a group of speech feature vectors of the scene noise. The filter here is a device for simulating the nonlinear characteristics of the human cochlear perception ability. The filter can be a filter that conforms to the Mel characteristic, which is not limited in the present embodiment.

[0143] In an embodiment of the present application, the mobile phone 100 can collect scene noise within a collection time period through the microphone of the headset 200. For example, when the collection time period is 2000ms, the mobile phone 100 can collect 2000ms of scene noise through the microphone of the headset 200, and then perform frame processing on the 2000ms long scene noise, that is, divide the scene noise into equal parts based on a framing threshold. For example, when the framing threshold is 100, the 2000ms long scene noise can be divided into 100 frames, each frame of 20ms. Subsequently, for each frame of the scene noise, a speech feature extraction algorithm is used to generate a feature vector describing the speech of the scene noise, and then the feature vectors are arranged in chronological order to obtain a feature vector T1. The speech features may be Linear Predictive Coding (LPC) features, Linear Predictive Cepstral Coefficients (LPCC) features, Mel-Scale Frequency Cepstral Coefficients (MFCC) features, or Linear Prediction Based Mel Frequency Cepstral Coefficients (LBPMFCC) features of the speech in scene noise. Each feature vector generated in T1 may be a two-dimensional vector (h1, h2), where h1 and h2 may be values ​​representing certain features of the speech in scene noise, such as the frequency and intensity of the speech in scene noise.

[0144] S304d: The mobile phone 100 inputs the speech features of the scene noise into the noise neural network model to calculate the scene probability vector corresponding to the scene noise.

[0145] In the embodiment of the present application, the extracted scene noise feature vector T1 is input into the trained noise neural network model, and the model is used to calculate the scene probability vector T corresponding to the feature vector T1. A The noise neural network model here can be the model trained in S304d. The scene probability vector T A It can be represented by a six-dimensional vector, T A =(A noisy A plane A train A bus A metro A indoor ), where A noisy Indicates the probability of a noisy human voice scene, A plane represents the probability of the airplane scene, A trainrepresents the probability of the train scene, A bus represents the probability of the bus scene, A metro represents the probability of the subway scene, A indoor represents the probability of indoor scenes. It can be understood that A noisy -A indoor The probability of the corresponding scenario can be represented by a numerical value, such as T A =(0.4 0 0.1 0.9 0.3 1).

[0146] In some embodiments, the noise neural network model of the scene noise can be a trained neural network model, and the mobile phone 100 can directly input the speech features of the scene noise into the model to obtain the corresponding scene probability vector. The noise neural network here can be a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), and a time recursive neural network (LSTM). In the case where the noise neural network is a convolutional neural network, multiple output nodes can be configured for the neural network, each output node matches a category of a scene corresponding to the noise, and which output node corresponds to which category of scene can be pre-set. The output value of each output node is used to represent the probability of the scene corresponding to the output node. In an embodiment of the present application, the noise neural network model can include output nodes for six scenes: noisy human voice scene, airplane scene, train scene, bus scene, subway scene, and indoor scene. The training process of the noise neural network model may include: inputting the speech features of the scene noise corresponding to the subway scene in the above six scenes into the noise neural network model, and then comparing the model training results (i.e., the scene probability vector corresponding to the scene noise collected in the subway scene) with the data representing the subway scene, calculating the error (i.e., the difference between the two), and updating the model weights according to the error. When the model finally outputs data representing the subway scene, the model training is considered to be completed. It is understandable that the speech features of the scene noise in other scenes can also be input to train the model, so that in the training of a large number of sample scenes, by continuously adjusting the weights, when the output error reaches a very small value (for example, meeting a predetermined error threshold), the neural network model is considered to have converged and the noise neural network model has been trained.

[0147] In other embodiments of the present application, the mobile phone 100 may also separate the low-frequency portion (characteristics of the sound channel of the sound-emitting object emitting the scene noise) and the high-frequency harmonic portion (characteristics of the sound source of the sound-emitting object emitting the scene noise) in the scene noise through a linear prediction method, and then perform feature extraction on the separated low-frequency portion and high-frequency harmonic portion using corresponding feature extraction algorithms, respectively, to obtain low-frequency audio features corresponding to the low-frequency portion of the audio (hereinafter referred to as the sound channel signal) and high-frequency audio features corresponding to the high-frequency harmonic portion (hereinafter referred to as the sound source signal). Finally, the low-frequency audio features of the sound channel signal and the high-frequency audio features of the sound source signal are fused to obtain the speech features of the scene noise.

[0148] For example, using the mobile phone 100's extraction of characteristic parameters for scene noise in a subway scene as an example, in some embodiments, the mobile phone 100 uses an 1-order linear predictor to separate the channel signal from the scene noise collected in the subway scene. By calculating the difference between the scene noise in the subway scene and the channel signal, the sound source signal in the scene noise in the subway scene can be obtained.

[0149] Mobile phone 100 then performs feature extraction on the channel signal and sound source signal separated from the scene noise. For example, using the feature extraction method described in S304c, MFCC extraction is performed on the channel signal separated from the subway scene to obtain the MFCC feature vector corresponding to the channel signal. Mobile phone 100 also uses a multi-scale wavelet transform to extract the time-frequency feature vector of the sound source signal separated from the subway scene.

[0150] It is understood that the vocal tract features of the above-mentioned vocal tract signals can also be extracted using other audio feature extraction algorithms that simulate the perception capabilities of the human cochlea, such as the Linear Prediction Cepstrum Coefficient (LPCC) extraction algorithm. In addition to wavelet transform, other algorithms can also be used to extract time-frequency feature vectors from the sound source signal, without limitation here, for example, the pitch period extraction method.

[0151] Finally, the mobile phone 100 fuses the MFCC feature vector extracted from the channel signal and the time-frequency feature vector extracted from the sound source signal to obtain the speech features of the scene noise. For example, in some embodiments, the MFCC feature vector and the time-frequency feature vector can be linearly fused to form a feature vector, or the two can be normalized and then linearly fused, or the two can be weighted and then linearly fused. In other embodiments, the two can also be nonlinearly fused, for example, by multiplying the two. In the specific implementation process, the fusion rules can be pre-set as needed, and this solution does not limit this.

[0152] In other embodiments, after separating the channel signal and the sound source signal from the scene noise in the subway scene, the mobile phone 100 may perform MFCC extraction on only the channel signal to obtain the MFCC feature vector corresponding to the channel signal. This MFCC feature vector is used as the speech feature of the scene noise in the subway scene. Alternatively, a multi-scale wavelet transform may be performed on only the sound source signal to extract the time-frequency feature vector, and the time-frequency feature vector of the sound source signal is used as the speech feature of the scene noise in the subway scene.

[0153] Regarding the above step S305, in some embodiments, the process of obtaining the scene probability vector of the scene image by performing feature extraction and scene image recognition on the scene image is as follows: Figure 3c Specifically including:

[0154] S305a: The mobile phone 100 obtains the current scene image.

[0155] Mobile phone 100 captures the current scene image through its own camera. In some embodiments, mobile phone 100 may also capture the current scene image through an image capture device including a camera. This image capture device may be part of mobile phone 100 or an independent device that communicates with mobile phone 100 to send the captured scene image to mobile phone 100. For example, this image capture device may be smart glasses with a camera, an external camera, etc.

[0156] S305b: The mobile phone 100 obtains image features of the scene image.

[0157] The process of the mobile phone 100 obtaining the image features of the scene image includes: the image features can be represented by a pixel matrix. Taking the RGB color space (grayscale value) standard as an example, a scene image can be represented as a pixel matrix of three color channels, such as an R channel pixel matrix, a G channel pixel matrix and a B channel pixel matrix. Here, each value in the R channel pixel matrix represents the component value of the corresponding pixel point in the R channel, and the same applies to the G channel pixel matrix and the B channel pixel matrix. The grayscale value and image size of each frame of the scene image are mapped into a two-dimensional vector; the image features of the scene image are arranged in chronological order to obtain the image features of the scene image. In the case of image features of multiple scene images, the image features of the scene image are the image feature vector group of the scene image.

[0158] In an embodiment of the present application, the mobile phone 100 can capture a 2000ms long video of a scene image through a camera based on the acquisition duration and frame threshold as described in S304 above, and at the same time, divide the video of the scene image into 100 frames, each frame of 20ms. Afterwards, for each frame of the scene image, the residual network Resnet-34 algorithm can be used to generate a feature vector T2 of the scene image in chronological order. Here, each feature vector in T2 can be a two-dimensional vector (h3, h4), and h3 and h4 can be values ​​representing certain features of the scene image, such as h3 can be a grayscale value, and h4 can be a numerical value of the image size.

[0159] S305c: The mobile phone 100 inputs the image features of the scene image into the image neural network model to calculate the scene probability vector corresponding to the image features.

[0160] In the embodiment of the present application, the extracted feature vector T2 is input into an image neural network model, and the scene probability vector T corresponding to the feature vector T2 is calculated using the neural network model. V . T V =(V noisy V plane V train V bus V metro V in的oor ), where V noisy represents the probability of a noisy human voice scene, V plane represents the probability of the airplane scene, V train represents the probability of the train scene, V bus represents the probability of the bus scene, V metro represents the probability of the subway scene, V in的oor represents the probability of an indoor scene.

[0161] The output structure of the image neural network model is the same as that of the noise neural network model, and can also include output nodes for six scenarios: noisy human voice scenes, airplane scenes, train scenes, bus scenes, subway scenes, and indoor scenes. The process of iteratively training the image neural network model based on the image features of the scene images is similar to the iterative training process of the noise neural network model. For example, the image features of the scene image corresponding to the subway scene in the six aforementioned scenes are input into the image neural network model. The model's training results are then compared with the data representing the subway scene to calculate the error, and the model weights are updated based on this error. Model training is considered complete when the model finally outputs data representing the subway scene.

[0162] Regarding the above step S306, in some embodiments, the mobile phone 100 fuses the scene probability vector of the scene noise and the scene probability vector of the scene image to obtain the probability of the fused scene, and the process of obtaining the current scene corresponding to the probability of the fused scene is as follows: Figure 3d As shown, specifically including:

[0163] After the mobile phone 100 identifies the scene noise and the scene image and obtains the scene probability vectors corresponding to the scene noise and the scene image, the mobile phone 100 performs a fusion calculation on the scene probability vectors of the scene noise and the scene image to obtain the current scene.

[0164] S306a: Obtain a probability matrix corresponding to the scene probability vector of the scene noise and the scene probability vector of the scene image.

[0165] The scene noise and scene image here are acquired through the same acquisition time and according to the same frame threshold, and the scene probability vector T of the scene noise is obtained. A The scene probability vector T corresponding to the scene image V The dimensions of the matrix of the composed vectors are also the same. In the embodiment of the present application, for example, each vector T generated by the noise neural network model and the image neural network model is A and T V They are all six-dimensional. When the frame threshold is 100, that is, N=100, T A and T V The size of the matrix of the composed vectors is [100,6]. A and T V The matrix of the composed vectors is fused and calculated to obtain the scene fusion probability vector.

[0166] Probability matrix for noisy scenes:

[0167]

[0168] Probability matrix of scene image:

[0169]

[0170] S306b: Obtain a probability vector group corresponding to the probability matrix of the scene noise and the scene image.

[0171] The fusion calculation algorithm of the scene fusion probability vector is as follows: the column vectors of the probability matrices of the noise scene and the image scene are extracted respectively, and the probability vector groups of the noise scene and the image scene can be obtained.

[0172] The probability vector group of scene noise can be expressed as:

[0173]

[0174] The probability vector group of the scene image can be expressed as:

[0175]

[0176] in: and It is another way to represent the probability vector group of scene noise and scene image.

[0177]

[0178]

[0179] X∈{noisy, plane, train, bus, metro, indoor}

[0180] Here, X represents six specific scenarios: noisy human voice scene, airplane scene, train scene, bus scene, subway scene, and indoor scene. The value range of X corresponds to the output nodes of the neural network model for the noise scene and image scene mentioned above.

[0181] S306c: Transpose the probability vector group of the scene noise and then perform a dot multiplication with the probability vector group of the scene image to calculate a fused scene probability vector.

[0182] Transpose each vector of the scene noise probability vector group, then multiply it by the scene image probability vector group, and perform fusion calculation to obtain the final fusion scene probability vector F X .

[0183]

[0184] Among them F noisy Indicates the probability of noisy human voice scene in the fusion scene, F plane represents the probability of the airplane scene, F train represents the probability of the train scene, F bus represents the probability of the bus scene, F metro represents the probability of the subway scene, F in的oor represents the probability of an indoor scene.

[0185] The probability of a specific scene in each fusion scene can be calculated using the following formula:

[0186]

[0187] X∈{noisy, plane, train, bus, metro, indoor}

[0188] It can be understood that in the case of X = noisy, the probability vector group of scene noise is Transpose to get In the probability vector group of the scene image Perform dot multiplication to obtain the probability of a noisy human voice scene in the fusion scene. Through dot multiplication calculation, the scene noise and the probability of a noisy human voice scene corresponding to each frame in the scene image can be multiplied together, and then the multiplication results of each frame are added together to obtain the probability of a noisy human voice scene in the fusion scene.

[0189] S306d: Determine the output scene according to the fused scene probability vector, and obtain a noise reduction mode corresponding to the output scene.

[0190] After the fusion scene probability vector is calculated, the output scene can be determined based on the probability value of each specific scene in the vector. The determination method is: if the probability of a specific scene in the fusion scene probability vector exceeds the probability threshold, the one with the largest probability exceeding the threshold is taken as the output scene; if no scene probability exceeds the probability threshold, the output scene is another scene. For example: in F X =(0 1 2 4 1 3), if the probability threshold is 3, then F X It indicates that the current scene of the mobile phone 100 is a bus scene, as shown in Table 2.

[0191] Table 2

[0192]

[0193]

[0194] It is understood that in addition to the scenarios in Table 2, in some embodiments of the present application, custom scenarios may also be added, and these custom scenarios may be shown in Table 3. In the case where the output scenario cannot be determined or the noise reduction mode corresponding to the output scenario does not exist, the default noise reduction mode corresponding to other scenarios may be selected to set the earphone 200 to achieve the noise reduction function.

[0195] Table 3

[0196] Outdoor cycling scene Outdoor cycling noise reduction mode Sleeping scene Sleep noise reduction mode Air pressure scene Low pressure noise reduction mode

[0197] S307: The mobile phone 100 obtains the noise reduction mode corresponding to the identified scene, inputs the noise reduction mode and the speech features of the scene noise into a noise reduction neural network model, calculates the noise reduction features corresponding to the speech features of the scene noise, and inputs the noise reduction features into the microphone 201 of the headset 200 for noise reduction.

[0198] In an embodiment of the present application, the noise reduction neural network here can be a deep neural network. The mobile phone 100 can generate speech features describing the scene noise based on each frame of scene noise collected in S304d through a speech feature extraction algorithm to obtain a feature value P1. Each feature value in the generated P1 can be a two-dimensional array (p1, p2), and p1 and p2 can be values ​​representing certain features of the speech describing the scene noise, such as the phase and amplitude of the speech of the scene noise. At the same time, the noise reduction neural network determines the noise reduction mode corresponding to each scene as a weight factor a, and performs a weighted operation on the feature value P1 of the scene noise according to the weight factor a to obtain a noise reduction feature H1. Here, the noise reduction feature H1 includes the phase and amplitude of the speech of the scene noise after the weighted operation. After that, the noise reduction feature H1 is input into the microphone 201 of the headset 200, and a sound with the opposite phase and the same amplitude as the noise reduction feature H1 is output for noise reduction.

[0199] The noise reduction features corresponding to each scene noise can also be determined by iteratively training a noise reduction neural network. The noise reduction neural network can be a three-layer neural network, with the first layer serving as the input layer, the second layer serving as the hidden layer, and the third layer serving as the output layer. After the noise reduction neural network model is constructed, the speech features of each scene noise and the noise reduction mode corresponding to each scene can be used as inputs to iteratively train the noise reduction neural network to construct the model. The model can have two input nodes, one for the speech features of the scene noise and the noise reduction mode corresponding to each scene, with the noise reduction mode serving as a weighting factor. The model can have one output node, each corresponding to a noise reduction feature of a scene noise. The output value of the output node, the noise reduction feature, the input noise reduction mode, and the calculated speech features of the scene noise can be represented by the following function: H1 = a·P1. The weight factor a here will take different values ​​according to the change of the noise reduction mode corresponding to the scene. In order to prevent the adverse effects caused by completely canceling out the scene noise in some scenes, for example, in the case of outdoor cycling, if the scene noise is greatly or completely canceled out, it will bring unsafe risks to the user. In the embodiment of the present application, when the noise reduction mode is the subway noise reduction mode, the weight factor a can be 1, and when the noise reduction mode is the outdoor cycling noise reduction mode, the weight factor a can be 0.5. In the case of P1 = (3, 5), when the noise reduction mode is the subway noise reduction mode, the sound output by the microphone 201 of the headset 200 is P1 = 1 · (-3, 5) = (-3, 5); when the noise reduction mode is the outdoor cycling noise reduction mode, the sound output by the microphone 201 of the headset 200 is P1 = 0.5 · (-3, 5) = (-1.5, 2.5).

[0200] S308: When the scene or noise in which the mobile phone 100 is located changes, the mobile phone 100 updates the noise reduction mode or turns off the noise reduction mode in real time.

[0201] In an embodiment of the present application, the mobile phone 100 has set the noise reduction mode for the headset 200. When the scene in which the mobile phone 100 is located changes, the mobile phone 100 can also update the noise reduction mode in real time according to the change of the scene. For example, in some embodiments, after the mobile phone 100 detects that the user uses the installed subway travel APP to swipe the card to enter the station, it confirms that the mobile phone 100 is in the subway scene. After the mobile phone 100 sets the subway noise reduction mode for the headset 200, the mobile phone 100 detects that the user uses the installed subway travel APP to perform the card swiping operation to exit the station. After that, the user uses the shared bicycle APP installed on the mobile phone 100 to scan the code and ride. The mobile phone 100 determines that it is now in an outdoor riding scene and sets the outdoor riding noise reduction mode corresponding to the outdoor riding scene for the headset 200. For another example, if the scene in which the phone 100 is located remains unchanged, but the scene noise within the scene changes, as described above, the phone 100 sets the earphone 200 to subway noise reduction mode. Simultaneously, the phone 100 detects the sound intensity of the scene noise in the current scene in real time. If the sound intensity exceeds a second sound intensity threshold and the phone 100 does not detect that the user has used the installed subway travel app to swipe their card to exit the station, the phone 100 can re-identify the scene noise and fuse the scene noise with the scene image to obtain the scene corresponding to the updated scene noise and scene image, as well as the noise reduction mode for that scene, and then re-set the noise reduction mode for the earphone 200. The method by which the phone 100 detects the sound intensity of the scene noise in the current scene can be the same as the method in S304. For example, if the phone 100 detects that the decibel value of the scene noise in the current scene is 80 dB and the second sound intensity threshold is 70 dB, the phone 100 can re-identify the scene noise and fuse the scene noise recognition result with the current scene image. It is understood that at this point, the scene image has not changed.

[0202] In some embodiments, when the user takes off and turns off the earphones 200, the noise reduction mode set for the earphones 200 is turned off.

[0203] Figure 5A scene recognition device 500 for identifying scene noise and scene image operations according to an embodiment of the present application is shown. The scene recognition device 500 includes a sound intensity detection module 501, a sound feature extraction module 502, a scene noise recognition module 503, an image feature extraction module 504, a scene image recognition module 505, and a fusion scene recognition module 506. It will be appreciated that the scene recognition device 500 can be located in the server 300 or in the mobile phone 100, or some modules of the scene recognition device 500 can be located in the server 300 and others in the mobile phone 100.

[0204] Sound intensity detection module 501 stores a first sound intensity threshold and a second sound intensity threshold. When the mobile phone 100 detects the sound intensity level of the scene noise, if it does not exceed the first sound intensity threshold, the scene probability vector of the scene noise can be directly determined to be an indoor scene probability vector. When the mobile phone 100 has set the headset 200 to noise reduction mode, the mobile phone 100 also detects the sound intensity of the scene noise in the current scene in real time. If the sound intensity exceeds the second sound intensity threshold, the mobile phone 100 can re-identify the scene noise.

[0205] The sound feature extraction module 502 is used to extract linear prediction Mel-frequency cepstral coefficient features from the scene noise and compose the speech features of the scene noise in chronological order.

[0206] The scene noise recognition module 503 may be pre-configured with a trained noise neural network model. After the speech features of the scene noise are input into the model, the scene probability vector of the scene noise is obtained.

[0207] The image feature extraction module 504 is used to extract the grayscale value and image size features of the scene image and compose the features of the scene image in chronological order.

[0208] The scene image recognition module 505 may be pre-configured with a trained image neural network model. After the features of the scene image are input into the model, the scene probability vector of the scene image is obtained.

[0209] The fusion scene recognition module 506 is configured to fuse the scene noise and the scene probability vector of the scene image to calculate the probability of the fusion scene, and obtain the current scene corresponding to the probability of the fusion scene.

[0210] Example 2

[0211] In the noise reduction scheme described in the above embodiment, the scene can be identified by obtaining event sensing data through various sensors and APP operation results on the mobile phone 100, and when the event sensing data cannot identify the scene, the environmental image and environmental sound are collected for scene recognition. In this recognition process, it is necessary to fuse the recognition results of the environmental image and environmental sound to obtain the final scene. The fusion of environmental image and environmental sound is calculated by matrix vectors. It can be understood that in other embodiments of the present application, for this scene recognition involving various modal data (for example, event sensing data, environmental image and environmental sound), the fusion of each modal data can be performed through a trained model, and the scene recognition result is obtained based on the fusion result.

[0212] For example, the measurement data of various modalities may include the aforementioned event sensing data, environmental images, and environmental sounds, as well as illumination data collected by illumination sensors, Wi-Fi signals, base station signals, etc. When the model fuses the data of each modality, the data of each modality has a corresponding weight. For example, for a multimodal model that includes illumination data, Wi-Fi signals, base station signals, environmental images, and environmental sounds, when fusing the scenes represented by these data, the probability and weight of each modality indicating that the current scene is an indoor scene are shown in Table 4 below:

[0213] Lighting data Wi-Fi signal Base station signal Environment Image Ambient Sound Internal scene probability 0.8 0.7 0.7 0.4 0.8 value 0.25 0.3 0.2 0 0.25

[0214] Therefore, the multimodal model can conclude that the probability that the scene currently occupied by the phone 100 and the headset 200 is indoors is: 0.8*0.25+0.7*0.3+0.7*0.2+0.4*0+0.8*0.25=0.75. If the scene threshold for indoor scenes is 0.6, it can be determined that the scene currently occupied by the phone 100 and the headset 200 is indoors. The noise reduction mode corresponding to the identified indoor scene is then used to reduce noise on the headset 200.

[0215] In the embodiments of the present application, the weights of the above-mentioned modal data are adjusted in real time according to the probability interval of the probability of the scene represented by the measured modal data, and are not fixed weights or empirical values. For example, when the above-mentioned illumination data indicates that the probability of an indoor scene is 0.8, its weight is 0.25, and when it indicates that the probability of an indoor scene is 0.5, the illumination data cannot determine which scene it is, and its contribution to scene recognition is small or no contribution, then its weight becomes smaller than when the probability of the indoor scene is 0.8, such as becoming 0.1 or 0. Compared with the scheme in the prior art where the weight is fixed or the weight is an empirical value, the weights that can be adjusted with the scene probability disclosed in the present application have better robustness, and thus the scene recognition accuracy of the multimodal model is higher.

[0216] The following is based on Figure 1 The noise reduction system shown, combined with Figures 6 to 8 , introducing the training process of the weight mapping sub-model and scene recognition in the multimodal model.

[0217] It can be understood that, unlike the above embodiment, Figure 1 In the noise reduction system shown, the server 300 can train a unimodal model for extracting unimodal scene probabilities from unimodal data and a multimodal model for fusing the unimodal scene probabilities. The trained unimodal and multimodal models are then ported to the mobile phone 100 for scene recognition.

[0218] Among them, the unimodal scene probability refers to the probability that the mobile phone 100 or the headset 200 is in a certain scene determined based on the data of a certain modality. For example, in Table 4 above, after the illumination data is input into the unimodal model that analyzes the illumination data, the probability that the current scene is indoor is 0.8, which is the probability value of the unimodal scene. The multimodal model is used to further fuse the 5 unimodal scene probabilities obtained from the 5 modal data in Table 4 above, and finally outputs a probability value of 0.75, that is, outputs the fused scene probability, which indicates that the probability that the current scene is indoor is 0.75. In an embodiment of the present application, when the multimodal model fuses multiple unimodal scene probabilities, it obtains the weight of each modal data. As mentioned above, for data of the same modality, the weights can be different depending on the data range. Therefore, the multimodal model determines the weight of the modal data according to the size of the data. These weights can be obtained in advance based on the training of the unimodal model. The training of the multimodal model (including the weight mapping submodel in the multimodal model) and the unimodal model will be introduced in detail below.

[0219] Figure 7 According to an embodiment of the present application, a training process of a weight mapping sub-model in a multimodal model is shown. Before the server 300 trains the method for obtaining weights for the weight mapping sub-model, as shown in FIG. Figure 6 As shown, first, server 300 obtains various raw data collected by mobile phone 100, such as ambient sound, ambient image, lighting data, Wi-Fi signal, base station data, and mobile speed. Then, each of these raw data is input into the corresponding unimodal model to obtain unimodal scene probabilities. Finally, the unimodal scene probabilities are used to train the weight mapping submodel in the multimodal model. It will be understood that in the multimodal model, the weight mapping submodel is used to determine the weights of each unimodal scene probability output by each unimodal scene model.

[0220] like Figure 7 As shown in Figure 2, the training process of the weight mapping sub-model in the multimodal model includes:

[0221] S701: The server 300 obtains raw data for training.

[0222] It is understood that the raw data here can be data collected by the mobile phone 100 or obtained from other means. For example, the event sensing data, environmental images and environmental sounds mentioned above can also include light data collected by the light sensor and Wi-Fi signals and base station signals collected by the communication module of the mobile phone 100.

[0223] S702: The server 300 inputs the acquired raw data into the corresponding unimodal model to obtain the corresponding unimodal scene probability.

[0224] It is understood that for ease of understanding and description, this example uses the input of data from one modality into a unimodal model. However, it is understood that in actual training, multimodal data can be simultaneously input into multiple unimodal models to obtain the unimodal scenario probabilities corresponding to each modality data.

[0225] For example, taking illumination data as an example, the server 200 can communicate with the mobile phone 100, access the illumination sensor of the mobile phone 100, and obtain the illumination data of the current scene through the photosensors contained in the illumination sensor. Then, the server 200 inputs the illumination data into a unimodal model of illumination data (hereinafter referred to as the illumination model), assuming that the unimodal scene probability of the illumination model represents the probability that the current scene is indoors. Specifically, Figure 8 As shown, if 100 illumination data x1, x2, ..., x100 are input into the illumination model, 100 single-mode scene probabilities are output. The numerical value of the single-mode scene probability is used to represent the probability of belonging to a certain scene (such as an indoor scene). The larger the value, the higher the probability of belonging to the scene, and vice versa. For example, the single-mode scene probability corresponding to 100 illumination data can be [0.98, ...0.11, ..., 0.34]. In some embodiments, other numbers of illumination data can also be used, for example: 1000.

[0226] It is understandable that the above-mentioned illumination data may also be collected in advance by the mobile phone 100 and uploaded and stored in the storage area of ​​the server 200 .

[0227] In embodiments of the present application, a single modal model, such as an illumination model, may be a neural network model obtained through training. In some embodiments, the training method may include:

[0228] The server 100 can input the illumination data into the illumination model for training. For example, the illumination data can be that when the user is walking outdoors, the mobile phone 100 collects the illumination data of the current scene through the illumination sensor. Then, the mobile phone 100 sends the illumination data to the server 200, and the server 200 inputs it into the illumination model, and then compares the output of the model (that is, the probability of the single-mode scene corresponding to the illumination data collected in the outdoor scene) with the expected single-mode scene probability corresponding to the illumination data representing the outdoor scene, and calculates the error (that is, the difference between the two), and then adjusts the parameters of the illumination model according to the error, for example, by adjusting the parameters of the illumination model through a loss function, until the error between the single-mode scene probability output by the illumination model and the expected single-mode scene probability of the input illumination data is less than a predetermined threshold or meets the requirements, the model training is considered to be completed.

[0229] It is understandable that the single-mode scenario probability of other modal data can also be obtained here. For example, 100 WIFI signals are input into Figure 6 In the Wi-Fi signal model shown, the single-mode scenario probabilities of 100 Wi-Fi signals are output.

[0230] S703: The server 200 pre-processes the single-mode scenario probability.

[0231] In order to avoid the single-mode scenario probabilities used for training being too discrete, the single-mode scenario probabilities can be arranged in descending order in an embodiment of the present application. The server 200 first arranges the single-mode scenario probabilities in descending order according to their numerical values; then the single-mode scenario probabilities are evenly divided into multiple intervals according to their number. The reason why the server 200 uses the number of single-mode scenario probabilities rather than the numerical value of the single-mode scenario probabilities to divide the intervals equally is to prevent data concentration, resulting in some intervals having no values. Using the number of single-mode scenario probabilities to divide the intervals equally can also reduce the overfitting problem and reduce the sensitivity of the model output to small difference inputs.

[0232] Specifically, in some embodiments, the process of preprocessing the single-mode scene probability includes:

[0233] a) The server 200 arranges the single-mode scenario in descending order according to its numerical value.

[0234] For example, server 200 arranges the 100 illumination data items' single-mode scene probabilities in descending order and relabels them as p1, p2, ..., p100, so that the single-mode scene probabilities are arranged linearly, for example, [0.98, ...0.34, ...,0.11]. In other words, the single-mode scene probabilities are arranged from high to low according to their numerical values.

[0235] In another embodiment of the present application, the numerical values ​​of the single-mode scene probabilities may also be arranged in ascending order, so that the numerical values ​​of the single-mode scene probabilities are arranged from low to high.

[0236] b) The server 200 divides the probability of single-mode scenarios into multiple intervals according to their number

[0237] For example, server 200 divides the 100 descending single-mode scenario probabilities into k intervals according to their number. For example, when k is 5, the intervals include k1, k2, ..., k5, where each interval includes 20 single-mode scenario probabilities. It is understood that in some embodiments, other positive integers may be used, such as k=10.

[0238] S704: The server 200 inputs the pre-processed single-mode scene probability into the multi-modal model to train the weight mapping sub-model.

[0239] For example, the server 200 uses the single-mode scene probability of the illumination data in the above five intervals as the input value k, and the expected single-mode scene probability corresponding to the single-mode scene probability of 100 illumination data as the result y, and the prediction function of the weight mapping sub-model is: The weight w is trained. In the above prediction function, w is a weight expressed in the form of a multidimensional vector, and its dimension can correspond one-to-one to the interval of the single-mode scenario probability used for training. For example, when the input value k is divided into 5 intervals, the w obtained after training can be [w1,w2,w3,w4,w5]. T represents the transposition of w, and b is the parameter of the weight mapping sub-model to be trained. T k+b represents the linear regression based on the input value k, that is, the input value k is used to predict its corresponding result through the parameter b. Since the output value of linear regression can range from (-∞, ∞+), that is, from negative infinity to positive infinity, and the weight value corresponding to the probability of a single-mode scene must range from (0, 1), therefore, through The final output result can be between (0, 1).

[0240] After the server 200 completes the training of the weight w and parameter b of the weight mapping sub-model using the unimodal scene probability of the illumination data in steps a and b, when the server 200 inputs the unimodal scene probability corresponding to the illumination data into the weight mapping sub-model of the trained multimodal model, the weight corresponding to the unimodal scene probability of the illumination data can be obtained.

[0241] The weight mapping sub-model first outputs weights corresponding to the unimodal scene probabilities output by different unimodal models. For example, taking the unimodal scene probability of the illumination model as an example, after inputting it into the weight mapping sub-model, the corresponding weight W_illumination can be obtained. Similarly, for the unimodal scene probability of the Wi-Fi signal model, the weight mapping sub-model outputs the corresponding weight W_wifi.

[0242] Secondly, for the same unimodal model, the output unimodal scene probabilities vary. After inputting these probabilities into the weight mapping sub-model, the resulting weights will also differ. For example, taking the unimodal scene probabilities of the illumination model as an example, if the unimodal scene probabilities are [0.98 and 0.11], inputting these [0.98 and 0.11] into the weight mapping sub-model yields W_illumination1 = 0.8 and W_illumination2 = 0.2.

[0243] At the same time, for the unimodal scene probability of the same unimodal model, according to step b above, when training the weight mapping sub-model, the unimodal scene probability will be divided into multiple intervals and then input into the weight mapping sub-model for training. During the training process, the corresponding weights for the unimodal scene probabilities within the same interval can be the same. After the model training is completed, after the unimodal scene probabilities of multiple unimodal models are input into the weight mapping sub-model, if the weight mapping sub-model determines that they belong to the same interval, the output weights can be the same. For example, taking the unimodal scene probability of the illumination model as an example, the unimodal scene probabilities are [0.98 and 0.96]. After inputting the above [0.98 and 0.96] into the weight mapping sub-model, the output W_illumination1 = W_illumination2 = 0.8 can be output.

[0244] Furthermore, when identifying different scenes, the unimodal scene probabilities calculated after inputting the same unimodal model input data into the unimodal model can be different; and the corresponding weights can also be different after inputting the unimodal scene probabilities into the weight mapping submodel of the multimodal model. For example, for the same illumination data, when performing outdoor scene recognition, the unimodal scene probability obtained after inputting it into the illumination model can be 0.91. Therefore, after inputting 0.91 into the weight mapping submodel of the multimodal model, the corresponding weight can be W_illumination = 0.8. However, when performing indoor scene recognition, the unimodal scene probability obtained after inputting it into the illumination model can be 0.21. Therefore, the weight obtained by inputting it into the weight mapping submodel of the multimodal model can be W_illumination = 0.11.

[0245] It can be understood that the above values ​​are exemplary. When the above model is used for scene recognition, the single-mode scene probability and weight may vary depending on the scene.

[0246] Although illumination data is used as an example in S704, the weight mapping sub-model of the corresponding multimodal model can be trained for the unimodal scene probabilities of other unimodal models. In other words, the unimodal scene probabilities corresponding to each modal data can be used as training data to train the weight mapping sub-model. After training, the multimodal model can receive the unimodal scene probabilities corresponding to multiple modal data as input data, obtain the corresponding weights for the unimodal scene probabilities of each modal data, and then calculate the fused scene probability through fusion weighted average calculation to identify the scene.

[0247] For example, in another embodiment of the present application, the data used for training the multimodal model is environmental sounds and environmental images collected from the mobile phone 100 through a microphone and a camera.

[0248] In passing Figure 7 After converting the ambient sound and ambient image into unimodal scene probabilities in S701 to S703, when training the weight mapping sub-model of the multimodal model in S704, the mobile phone 100 can also use the event sensing data described in Table 1, such as the event sensing data corresponding to scanning a subway QR code to enter a subway station, to train the multimodal model's weight acquisition. This allows the multimodal model to adjust the weights of the unimodal scene probabilities corresponding to the ambient sound and ambient image based on the event sensing data.

[0249] For example, when using the multimodal model for scene recognition, after the mobile phone 100 inputs the ambient sound, ambient image, and event sensing data into the multimodal model, if the multimodal model can determine from the event sensing data that the current scene is a subway scene, the multimodal model can increase the weights of the single-modal scene probabilities corresponding to the ambient sound and ambient image. This makes the fused scene probabilities corresponding to the ambient sound and ambient image, after fusion calculation, more accurate.

[0250] The weight mapping sub-model in the above-mentioned multimodal model can be implemented using a logistic regression (LR) classification model. In other embodiments of the present invention, the weight mapping sub-model can also use other classification models, such as a gradient boosting decision tree (GBDT) classification model.

[0251] It is understood that in addition to using illumination data to train the weight mapping sub-model of the multimodal model, you can also use mobile speed, Wi-Fi signals, base station signals, environmental images, environmental sounds, etc. The training process is the same as that for illumination data and will not be repeated here.

[0252] After training the multimodal model and the unimodal model including the weight mapping sub-model on the server 200, an Android project can be established to read and parse the model through the model reading interface in the aforementioned project, and then compile and generate an APK (Android application package) file, which is installed in the mobile phone 100 to complete the model transplantation.

[0253] Continue to refer to Figure 6 Combined with Figure 9 and Figure 10 Taking the user's movement speed, lighting data, Wi-Fi signal, base station signal, environmental image, and environmental sound collected by mobile phone 100 as an example, the process of scene recognition is introduced using the above-trained multimodal model. Specifically, the process of scene recognition includes:

[0254] S901: The mobile phone 100 collects data through its own sensors or applications.

[0255] In addition to illumination data, the mobile phone 100 can also collect horizontal acceleration through its own acceleration sensor and measure the user's movement speed through acceleration. Similarly, the mobile phone 100 can also obtain the user's movement speed through the GPS module.

[0256] S902: The mobile phone 100 inputs the collected data into the unimodal model respectively to obtain the corresponding unimodal scene probability.

[0257] Next, using the user's movement speed as an example, mobile phone 100 inputs the movement speed collected in S901 into the movement speed model to obtain the corresponding unimodal scene probability. The movement speed model here is also a unimodal model, and its unimodal scene probability is the same as the illumination model, also a unimodal scene probability. The unimodal scene probability ranges between 0 and 1 and represents the probability of a scene. A larger value indicates a higher probability of belonging to that scene, and vice versa.

[0258] The above-mentioned moving speed model can also be a neural network model obtained through training, and its training method can be the same as the training method of the above-mentioned illumination model.

[0259] S903: The mobile phone 100 inputs the unimodal scene probability into the multimodal model, and the multimodal model performs robustness enhancement processing on the unimodal scene probability.

[0260] It can be understood that robustness enhancement processing refers to screening the single-mode scene probability before the multimodal model performs fusion calculation on the single-mode scene probability, and removing the single-mode scene probability whose values ​​are near the median of the range of single-mode scene probabilities. After removing the above-mentioned single-mode scene probabilities, the influence of the above-mentioned single-mode scene probabilities on the fusion calculation of the multimodal model can be reduced, and the accuracy of the fusion calculation can be increased. Taking the single-mode scene probability of illumination data as an example, when the user is in a critical state such as the inside and outside of the building entrance, the single-mode scene probability of the illumination model will be close to the median range of the single-mode scene probability, for example, between 0.4 and 0.6. Therefore, when the single-mode scene probability output by the illumination model is between 0.4 and 0.6, the single-mode scene probability is not used for fusion calculation.

[0261] S904: The multimodal model performs weighted averaging on the fusion of the single-mode scene probabilities and their corresponding weights to obtain the fusion scene probabilities corresponding to the respective single-mode scene probabilities, thereby identifying the scene.

[0262] The trained multimodal model can obtain its corresponding weight according to the actual input single-mode scene probability, and the corresponding weight is different for different values ​​of the single-mode scene probability.

[0263] Taking the illumination model's unimodal scene probability as an example, when this probability is input into the multimodal model, its corresponding weight, W_illumination, is obtained. This W_illumination serves as the weight for the weighted average fusion of the illumination model's unimodal scene probability. For different illumination model unimodal scene probabilities, probability 1 and probability 2, their corresponding weights can be W_illumination1 and W_illumination2.

[0264] For mobile speed, Wi-Fi signal, base station signal, environmental sound and environmental image, after inputting them into the multimodal model, the corresponding W_speed, W_wifi, W_base station, W_sound and W_image can also be obtained.

[0265] The multimodal model then combines the single-mode scene probabilities and their corresponding weights for illumination data, mobile speed, base station signals, ambient sound, and ambient image data using the following weighted formula: Single-mode scene probability for mobile speed * W_Speed ​​+ Single-mode scene probability for illumination data * W_Light + Single-mode scene probability for Wi-Fi signals * W_Wifi + Single-mode scene probability for base stations * W_Basestation + Single-mode scene probability for ambient sound * W_Sound + Single-mode scene probability for ambient image * W_Image, to obtain the fused scene probability. The probability and weight of each single-mode scene probability indicating that the current scene is indoor are shown in Table 5 below. Different single-mode scene probabilities adjust their corresponding weights accordingly, resulting in different fused scene probabilities.

[0266] Lighting data Movement Speed Wi-Fi signal Base station signal Environment Image Ambient Sound Internal scene probability 0.8 0.9 0.7 0.7 0.4 0.8 value 0.20 0.05 0.3 0.2 0 0.25 Internal scene probability 0.9 0.6 0.3 0.6 0.2 0.3 value 0.50 0.10 0.05 0.2 0.05 0.1

[0267] When the scene threshold for indoor scenes is 0.6, for the first set of data, the fusion scene probability is 0.755, and the mobile phone 100 confirms that the current scene is an indoor scene. For the second set of data, the fusion scene probability is 0.91, and the mobile phone 100 confirms that the current scene is an indoor scene.

[0268] It is understood that the mobile phone 100 uses the above-mentioned light data, moving speed, WIFI signal, base station signal, environmental image, and environmental sound. Figure 6 The mobile phone 100 can also collect, for example, air pressure data and user biometric data. It can then combine the aforementioned lighting data, movement speed, Wi-Fi signal, base station signal, ambient image, and ambient sound to identify whether the current environment falls within the scenarios described in Tables 2 and 3.

[0269] The following describes an audio noise reduction scenario in another embodiment of the present application. The scenario includes a system 10 composed of a mobile phone 100, a headset 200, and a server 300. Compared with the mobile phone 100 continuously collecting environmental images and environmental sounds through a camera and a microphone, the difference is that the mobile phone 100 can continuously collect data of the current environment through a low-power collection method. When the result of the data collected through the low-power collection method meets the accuracy threshold, the mobile phone 100 starts a high-power collection method, obtains data of the user's current environment through the high-power collection method, and then fuses and calculates the data to identify the current scene. Finally, the headset 200 uses the noise reduction mode corresponding to the current scene to perform noise reduction. The noise reduction mode in this application refers to the terminal device 100 setting a sound mode for the headset 200, and generating a sound signal to combat noise through the sound mode to eliminate it.

[0270] For example, after the mobile phone 100 is connected to the headset 200, the mobile phone 100 can continuously collect the current Wi-Fi signal strength through the Wi-Fi module. If the current Wi-Fi signal strength cannot be detected or the current Wi-Fi signal strength is less than the signal strength threshold, the mobile phone 100 determines that the user is currently outdoors. At this time, while continuously detecting the Wi-Fi signal strength, the mobile phone 100 uses the acceleration sensor to obtain the user's movement speed and the light sensor to obtain outdoor light data, and then integrates and calculates the above collected data to identify the current scene.

[0271] The server 300 can synchronize the multimodal model used for fusion calculations with each mobile phone 100 based on a request from the mobile phone 100 or by periodic push notifications. The server 300 can also synchronize the multimodal model with an existing multimodal model on each mobile phone 100. For example, if a developer updates a multimodal model and reports it to the server 300, the server 300 can push the updated multimodal model to each mobile phone 100. Alternatively, the server 300 can periodically detect the multimodal model currently in use by the mobile phone 100. Once the detection is complete, the server 300 can send the latest multimodal model to the mobile phone 100. After obtaining the latest multimodal model, the mobile phone 100 can update and save it in the mobile phone 100's memory.

[0272] Of course, the above-mentioned system 10 is only a specific implementation of the present application. In some implementations, the server 300 can also regularly train its own stored multimodal model and push the trained multimodal model to the terminal device 100; the terminal device 100 can also regularly update its own multimodal model from the server 300.

[0273] Below through Figure 11 and Figure 12 , through mobile phone 100 Figure 6 The model described in the embodiment is used to perform scene recognition and set the noise reduction mode for the earphone 200 to describe an audio noise reduction method in detail. In this audio noise reduction method, the mobile phone 100 continuously collects data of the current environment through a low-power collection method. When the collected data meets the accuracy threshold, the mobile phone 100 collects data of the current environment through a high-power collection method. Figure 6 After the model described in recognizes the scene, the noise reduction mode is set for the earphone 200 to perform noise reduction.

[0274] like Figure 11 As shown, the technical solution for achieving audio noise reduction includes:

[0275] S1101: After the user wears the earphone 200, the mobile phone 100 detects that the earphone 200 is connected and the earphone 200 is in working state. When the earphone 200 is detected to be connected to the mobile phone 100 and in working state, the process proceeds to S1102. Figure 3a Same as S301 in.

[0276] S1102: The mobile phone 100 continuously collects data of the current environment in a low-power collection manner.

[0277] Reference Figure 12Here, the mode in which the mobile phone 100 continuously executes the low-power collection mode can be that the mobile phone 100 continuously executes the low-power collection mode. The mobile phone 100 can also set the low-power collection mode to execute once every time period, for example, the mobile phone 100 sets the low-power collection mode to execute once every 1 second until the mobile phone 100 is turned off.

[0278] For example, a user enters a subway station by scanning a code on a subway travel APP installed on the mobile phone 100. At this time, the mobile phone 100 can continuously detect whether the user clicks to open the subway travel APP through its low-power collection method and executes the instruction to scan the code to enter the station through the subway travel APP.

[0279] For another example, the mobile phone 100 continuously detects the sound intensity level of the ambient sound through the microphone of the headset 200 or its own microphone. The process of the sound intensity level detection here can be the same as that described in S304a, and will not be repeated here.

[0280] S1103: When the result of the data collected by the low-power collection method meets the preset accuracy threshold, the mobile phone 100 executes S1104, otherwise the mobile phone 100 returns to S1102 and continues to collect data by the low-power collection method.

[0281] For example, when a user is on the subway and the mobile phone 100 collects ambient sound using a low-power collection method such as sound intensity level detection, the process for determining whether the sound intensity level detection result meets the preset sound intensity threshold can be the same as described in S304a. However, the difference is that if the current ambient sound does not exceed the preset sound intensity threshold, the mobile phone 100 returns to S603 and resumes the low-power collection method. If it exceeds the preset sound intensity threshold, the process proceeds to S605 and the mobile phone 100 performs the high-power collection method.

[0282] For another example, when the low-power collection method is to detect whether the subway travel APP of the mobile phone 100 executes the code scanning instruction to enter the subway station, the above-mentioned accuracy threshold can be whether the subway travel APP of the mobile phone 100 executes the code scanning instruction to enter the station.

[0283] For another example, the mobile phone 100 may obtain the Wi-Fi signal strength from the collected Wi-Fi signal and compare the Wi-Fi signal strength with a preset Wi-Fi signal strength threshold. The unit of Wi-Fi signal strength is dBm (decibel relative to one milliwatt). Wi-Fi signal strength is represented by a negative number; the closer to 0, the better the signal. For example, if the strength of signal A is -40dBm and the strength of signal B is -60dBm, then signal A is stronger than signal B. The memory of the mobile phone 100 may store a preset Wi-Fi signal strength threshold, such as (-75dBm to -50dBm). If the Wi-Fi signal strength obtained by the mobile phone 100 is -20dBm, the mobile phone 100 determines that the Wi-Fi signal in the current scene is excellent, and can preliminarily determine that the user is indoors. Next, the mobile phone 100 may enter S1105 to obtain data about the current environment through a high-power acquisition method.

[0284] S1104: The mobile phone 100 collects data of the current environment through a high-power collection method.

[0285] For example, the process of the mobile phone 100 collecting scene sounds and scene images through the microphone and camera is the same as the process of obtaining scene sounds and scene images in Figure 3, and will not be repeated here.

[0286] For example, to determine user speed, ambient light data, and base station information, after the phone 100 has initially confirmed that the user is indoors through a low-power acquisition method such as collecting Wi-Fi signals, it can access its own accelerometer to detect horizontal acceleration and measure the user's movement speed. Similarly, the phone 100 can access its own light sensor and, through the photosensor contained in the light sensor, obtain light data for the current scene. The phone 100 can also access base stations through its own antenna to obtain base station information.

[0287] S1105: The mobile phone 100 fuses and calculates the data of the current environment collected by the high-power collection method to obtain a fusion scene probability.

[0288] Regarding the technical solution of S1105, Figure 9 Same as described.

[0289] S1106: The mobile phone 100 determines whether it can identify the corresponding scene based on the obtained fusion scene probability. If so, the mobile phone 100 proceeds to S1107 to obtain the noise reduction mode corresponding to the scene and set the noise reduction mode on the headset 200. After S907 is completed, the mobile phone 100 returns to S1103 and continues to execute the low-power collection method. If the mobile phone 100 cannot determine the scene corresponding to the obtained fusion scene probability, the mobile phone 100 similarly returns to S1103 and continues to execute the low-power collection method.

[0290] S1107: The mobile phone 100 obtains the noise reduction mode corresponding to the scene and sets the noise reduction mode for the headset 200. S907 here is the same as S307 above and will not be repeated here.

[0291] For example, after the mobile phone 100 determines that the current scene is a subway scene and sets the noise reduction mode for the headset 200, the mobile phone 100 still continues to collect data through a low-power collection method. At this time, if the user continues to take the subway in the next hour, the mobile phone 100 will not execute the high-power collection method for an hour. The high-power collection method will not be executed again until the result of the collection changes through the low-power collection method of the mobile phone 100. Compared with the mobile phone 100 continuously executing the high-power collection method within the above-mentioned one hour, for example, the mobile phone 100 executes the high-power collection method once every 30 seconds, the power consumption of the mobile phone 100 can be reduced, and at the same time, the accuracy of the mobile phone 100's fusion scene recognition can be guaranteed.

[0292] After the phone 100 determines that the current scene is indoors and sets the headset 200 to noise reduction mode, the phone 100 continues to collect data using the low-power collection method. At this time, if the user remains indoors for the next hour, the phone 100 will not execute the high-power collection method. At this time, the phone 100 will not execute the high-power collection method until the results obtained by the low-power collection method change, such as when the phone 100 detects that the current light data exceeds the light data threshold through ambient light collection, at which point the high-power collection method will be executed again.

[0293] In another embodiment, after the mobile phone 100 determines that the current scene is an indoor scene and sets the noise reduction mode for the earphone 200, the mobile phone 100 continues to collect data through a low-power collection method. At the same time, the mobile phone 100 can also execute a high-power collection method according to an execution interval (such as once every 30 seconds) to collect data for scene recognition. For example, when the user is in an outdoor exercise state (such as running), at this time, in addition to executing a low-power collection method to collect data (such as WIFI signals), the mobile phone 100 also executes a high-power collection method every 30 seconds to collect data (such as sound, image) for fusion scene recognition. So that when the user 100 is running, the mobile phone 100 can set the noise reduction mode of the earphone 200 according to changes in the user's surrounding environment.

[0294] Figure 13 A schematic structural diagram of a terminal device 100 according to an embodiment of the present application is shown.

[0295] The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 1100, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0296] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0297] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0298] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0299] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0300] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0301] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt a different interface connection method from the above embodiments, or a combination of multiple interface connection methods.

[0302] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0303] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0304] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0305] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0306] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0307] The wireless communication module 160 can provide wireless communication solutions applied to the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0308] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0309] The terminal device 100 implements display functions through a GPU, a display screen 194, and an application processor. The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0310] The terminal device 100 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor, etc. In some embodiments, the camera 193 can be the camera 102 in the embodiment of the present application, which is used to collect scene images of the current environment.

[0311] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0312] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor. In some embodiments, the internal memory 121 can be used to store the noise neural network model and the image neural network model trainer in the embodiment of the present application, as well as the noise reduction mode corresponding to the scene.

[0313] The terminal device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0314] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0315] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0316] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the user can hear the voice by placing the receiver 170B close to the ear.

[0317] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, and realization of directional recording functions, etc. In some embodiments, the microphone 170C can be the microphone 101 in the embodiment of the present application, which is used to collect scene noise in the current environment where the terminal device 100 is located.

[0318] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0319] In some embodiments, the sensor module 180 may be the sensor 103 in the embodiment of the present application, and is used to determine the scene in which the terminal device 100 is located by sensing event sensing data.

[0320] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0321] The touch sensor 180K is also called a "touch control device." The touch sensor 180K can be provided on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be provided on the surface of the terminal device 100, at a location different from that of the display screen 194.

[0322] The buttons 1100 include a power button, a volume button, etc. The buttons 1100 may be mechanical buttons or touch buttons. The terminal device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the terminal device 100.

[0323] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0324] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0325] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the terminal device 100 by inserting or removing it from the SIM card interface 195. The terminal device 100 can support one or N SIM card interfaces, where N is a positive integer greater than one. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0326] It should be understood that although the terms "first," "second," and the like may be used herein to describe various features, these features should not be limited by these terms. These terms are used merely to distinguish and should not be understood as indicating or implying relative importance. For example, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature, without departing from the scope of the exemplary embodiments.

[0327] Furthermore, various operations will be described as multiple, separate operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be construed to imply that the operations must be performed in order of description, and many of the operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can terminate when the described operations are completed, but can also have additional operations not included in the figures. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0328] References in the specification to "one embodiment," "an embodiment," "an illustrative embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or property, but that every embodiment may or may not necessarily include the particular feature, structure, or property. Furthermore, these phrases are not necessarily referring to the same embodiment. Furthermore, while particular features may be described in conjunction with a specific embodiment, the knowledge of those skilled in the art may affect how those features may be combined with other embodiments, whether or not those embodiments are explicitly described.

[0329] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A / B" means "A or B." The phrase "A and / or B" means "(A), (B), or (A and B)."

[0330] As used herein, the term "module" may refer to, be part of, or include: memory (shared, dedicated, or group) for running one or more software or firmware programs, application-specific integrated circuits (ASICs), electronic circuits and / or processors (shared, dedicated, or group), combinational logic circuits, and / or other suitable components that provide the functionality.

[0331] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order is not required. Rather, in some embodiments, these features may be illustrated in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular drawing does not mean that all embodiments need to include such features. In some embodiments, these features may not be included, or they may be combined with other features.

[0332] The embodiments of the present application are described in detail above with reference to the accompanying drawings. However, the application of the technical solution of the present application is not limited to the various applications mentioned in the embodiments of the present patent. Various structures and variations can be easily implemented with reference to the technical solution of the present application to achieve the various beneficial effects mentioned herein. Various changes made within the knowledge of ordinary technicians in this field without departing from the purpose of the present application should fall within the scope of coverage of the patent application.

Claims

1. An audio noise reduction method for a first electronic device, characterized in that: include: The first electronic device obtains data of multiple modalities for identifying a current environmental scene, wherein the current environmental scene includes an outdoor riding scene, a subway scene, a sleeping scene, and a low-pressure scene, and the data of the multiple modalities includes multiple of the following: an operation result of an application running on the first electronic device, an operation result of a sensor on the first electronic device, detection data of the sensor on the first electronic device, a Wi-Fi signal received by the first electronic device, and a base station signal of a base station communicatively connected to the first electronic device; the operation result of the application includes a shared bicycle application running a scan code to ride, and a subway travel application running a scan code to enter a subway station; The operating result of the sensor on the first electronic device includes, when the sensor is a heart rate sensor, that the operating result of the heart rate sensor is a sleep state; The detection data of the sensor on the first electronic device includes at least one of light data collected by the light sensor and air pressure data collected by the air pressure sensor; The first electronic device calculates, for each modality, a single-mode scenario probability indicating that the first electronic device is in a first scenario; the first electronic device determines a plurality of weights corresponding to the data of each modality, and based on the plurality of weights, fuses the different single-mode scenario probabilities corresponding to the data of each modality to obtain a fused scenario probability indicating that the first electronic device is in the first scenario, wherein data of the same modality belongs to different preset data ranges, and the weights used in the fusion are different; When the fusion scene probability satisfies the identification condition of the first scene, the first electronic device determines that the audio noise reduction mode corresponding to the second electronic device is the noise reduction mode corresponding to the first scene, and the second electronic device is a headset; Before determining multiple weights corresponding to each modal data and fusing different single-mode scene probabilities corresponding to each modal data based on the multiple weights, the single-mode scene probabilities are screened to remove the single-mode scene probabilities whose values ​​fall within the median range of the single-mode scene probabilities.

2. The audio noise reduction method according to claim 1, wherein: The detection data of the sensor on the first electronic device also includes the acceleration of the first electronic device collected by the acceleration sensor.

3. The audio noise reduction method according to claim 1, wherein: The multiple modalities include a first modality, and the multiple preset data ranges corresponding to the first modality include a first preset data range and a second preset data range, wherein when calculating the fusion scene probability of the first scene, the single-mode scene probability of the first modal data belonging to the first preset data range is greater than the single-mode scene probability of the first modal data belonging to the second preset data range, and the weight corresponding to the first preset data range is greater than the weight corresponding to the second preset data range.

4. The audio noise reduction method according to claim 3, wherein: The first electronic device fuses the different single-mode scene probabilities of the modal data based on a neural network model to obtain a fused scene probability indicating that the first electronic device is in the first scene.

5. The audio noise reduction method according to claim 4, characterized in that: The weight mapping sub-model for determining the weights of each modality data in the neural network model is trained in the following manner: Acquire a plurality of sample data belonging to the first modality and expected single-mode scene probabilities corresponding to the plurality of sample data respectively; Arrange the plurality of sample data belonging to the first modality in descending order according to numerical values; Dividing the sorted sample data into a plurality of preset data ranges according to a predetermined number, wherein the plurality of preset data ranges include the first preset data range and the second preset data range; Calculate the average or median value of each data within a preset range, and input the average or median value into the weight mapping sub-model; The coefficients of the weight mapping sub-model are adjusted by comparing the output of the weight mapping sub-model with the expected single-mode scene probability.

6. The audio noise reduction method according to claim 1, wherein: Also includes: The first electronic device sends a noise reduction mode to the second electronic device.

7. The audio noise reduction method according to claim 1, wherein: The first electronic device is a mobile phone.

8. The audio noise reduction method according to claim 1, wherein: The first electronic device is capable of acquiring data in M ​​modalities through a first mode and acquiring data in N modalities through a second mode, where M and N are positive integers and M is less than or equal to N; and In the first mode, the number of devices enabled by the first electronic device for obtaining the M modal data is less than the number of devices enabled by the first electronic device for obtaining the N modal data in the second mode.

9. An audio noise reduction method for a first electronic device, characterized in that: include: The first electronic device continuously collects first data of a current environmental scene in a low-power collection manner, where the current environmental scene includes an outdoor cycling scene, a subway scene, a sleeping scene, and a low-pressure scene; When the first data meets a preset first accuracy threshold, the first electronic device collects second data of multiple modalities for identifying the current environmental scene in a high-power collection manner, where the second data of the multiple modalities include multiple types of the following: an operation result of an application running on the first electronic device, an operation result of a sensor on the first electronic device, detection data of the sensor on the first electronic device, a Wi-Fi signal received by the first electronic device, and a base station signal of a base station communicatively connected to the first electronic device; the operation results of the application include a shared bicycle application running a scan code to ride, and a subway travel application running a scan code to enter a subway station; The operating result of the sensor on the first electronic device includes, when the sensor is a heart rate sensor, that the operating result of the heart rate sensor is a sleep state; The detection data of the sensor on the first electronic device includes at least one of light data collected by the light sensor and air pressure data collected by the air pressure sensor; The first electronic device calculates, respectively, a single-mode scenario probability that the second data of the plurality of modalities indicates that the first electronic device is in a first scenario; The first electronic device determines a plurality of weights corresponding to the second data, and based on the plurality of weights, fuses different single-mode scene probabilities corresponding to the second data of multiple modalities to obtain a fused scene probability indicating that the first electronic device is in the first scene; When the fusion scene probability satisfies the identification condition of the first scene, the first electronic device determines that the audio noise reduction mode corresponding to the second electronic device is the noise reduction mode corresponding to the first scene, and the second electronic device is a headset; Before determining multiple weights corresponding to each modal data and fusing different single-mode scene probabilities corresponding to each modal data based on the multiple weights, the single-mode scene probabilities are screened to remove the single-mode scene probabilities whose values ​​fall within the median range of the single-mode scene probabilities.

10. An electronic device, characterized in that: include: a memory storing instructions; as well as At least one processor is configured to access the memory and to execute instructions on the memory to perform the audio noise reduction method according to any one of claims 1 to 8.

11. A computer-readable medium, characterized in that The computer-readable medium stores instructions, which, when executed on a computer, enable the computer to perform the audio noise reduction method according to any one of claims 1 to 8.

Images