Scene recognition method and related apparatus

Abstract

The application provides a scene recognition method and related device, applied to the terminal technical field. The method comprises: acquiring audio information collected by a microphone of a terminal device; inputting the audio information into a first model to obtain a first output result; and recognizing a scene represented by the first output result; wherein the first model is trained by a first source model, a second source model, a third source model, a fourth source model and a fifth source model, so that the first model can extract features for scene recognition. In this way, the first model can extract features for scene recognition in the audio information, recognize the scene of collecting the audio information based on the features for scene recognition, which is conducive to reducing the influence of features in the audio information due to different collection devices on scene recognition, and further conducive to improving the robustness of scene recognition.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a scene recognition method and related apparatus. Background Technology

[0002] The terminal device can collect audio information from its environment, input this information into a scene recognition model to determine the scene in which the device is located, and then determine whether to process the audio being played or the audio being collected based on the scene. For example, the terminal device, through the scene recognition model, identifies its environment as a subway scene based on the collected audio information. In this subway scene, if the terminal device is in call mode, it can perform noise reduction processing on the collected sound and amplify the volume of the audio being played.

[0003] Currently, scene recognition models have poor robustness, resulting in different recognition accuracy rates for the same scene recognition model on different terminal devices. Summary of the Invention

[0004] This application provides a scene recognition method and related apparatus, which are applied in the field of terminal technology. The scene recognition model in this application can extract features for scene recognition from audio information and identify the scene of the collected audio information based on the features for scene recognition. This helps to reduce the influence of features generated by different acquisition devices in the audio information on scene recognition, thereby improving the robustness of the scene recognition model.

[0005] In a first aspect, this application proposes a scene recognition method, comprising: acquiring audio information collected by a microphone of a terminal device; inputting the audio information into a first model to obtain a first output result; recognizing a scene represented by the first output result, wherein the scene includes scenes distinguished by noise in the audio and / or the number of sounds included in the audio; wherein the first model is trained by a first source model, a second source model, a third source model, a fourth source model, and a fifth source model, wherein the inputs of the first source model and the second source model are both audio samples, the output of the first source model is the input of the third source model, and the output of the second source model is the input of the third source model. The input to the fourth source model, the output of the first source model, and the output of the second source model also serve as the input to the fifth source model; the outputs of the third and fourth source models are both the device types for acquiring the audio samples, with the third source model outputting incorrect device types and the fourth source model outputting correct device types; the fifth source model is used to superimpose and reconstruct the outputs of the first and second source models, and converges when the difference between the superimposed and reconstructed result and the audio sample is less than a first preset value; when both the fourth and fifth source models converge, the training of the first source model ends, and the first model is obtained.

[0006] The scene recognition method provided in this application allows the fourth source model to constrain the second source model when the fourth source model converges, which helps the second source model extract more accurate features for device type identification. When the fifth model converges, and the second source model can extract more accurate features for device type identification, it helps constrain the first source model to extract features for scene recognition, thus obtaining the first model. In use, the first model can extract features for scene recognition from audio information. Based on these features, the scene represented by the features can be identified, which helps to identify scenes on different terminal devices and improves the robustness of scene recognition.

[0007] In one possible implementation, audio samples are represented by x. i The first source model is denoted by G1, and its output is denoted by G1(x). i The third source model is represented by D1, and its input is G1(x). i The output of the third source model is (D1[G1(x)). i )]) j It means that (D1[G1(x)) i )]) j Used to represent the collected audio sample x i Let be the predicted probability of the j-th device, and let y be the ground truth value of the audio sample. i,j It means that y i,jUsed to represent the collected audio sample x i The device is the true probability of the j-th device, where j is an integer greater than or equal to 1. The loss function of the first source model is -L1, and the loss function of the third source model is L1. L1 and (D1[G1(x)) are the true probabilities of the j-th device. i )]) j and y i,j In this way, the loss function of the first source model is negatively correlated with the loss function of the third source model. The output of the first source model is the input of the third source model. The features generated by the first source model are used to confuse the third source model, preventing it from outputting the correct device category. This is beneficial for training the first source model to extract features for scene recognition and for training the third source model to prevent it from outputting the correct device category.

[0008] In one possible implementation, L1 satisfies the following formula In this way, the loss function of the third source model is the cross-entropy loss function, which is beneficial to improving the convergence speed of the third source model.

[0009] In one possible implementation, audio samples are represented by x. i The second source model is denoted by G2, and its output is denoted by G2(x). i The fourth source model is represented by D2, and its input is G2(x). i The output of the fourth source model is (D2[G2(x)). i )]) j It means that (D2[G2(x)) i )]) j Used to represent the collected audio sample x i Let be the predicted probability of the j-th device, and let y be the ground truth value of the audio sample. i,j It means that y i,j Used to represent the collected audio sample x i The device is the true probability of the j-th device, where j is an integer greater than or equal to 1. The loss function of the second source model and the loss function of the fourth source model are both L2. L2 and (D2[G2(x)) are the true probabilities of the j-th device. i )]) j and y i,j This is relevant. In this way, the loss function of the second source model is the same as that of the fourth source model, and the output of the second source model is the input of the fourth source model. This is beneficial for training the second source model to extract features for device identification and for training the fourth source model to output the correct device category.

[0010] In one possible implementation, L2 satisfies the following formula In this way, the loss function of the second source model and the loss function of the fourth source model are both cross-entropy loss functions, which is beneficial to improving the convergence speed of the second source model and the fourth source model.

[0011] In one possible implementation, audio samples are represented by x. i The first source model is denoted by G1, and its output is denoted by G1(x). i The second source model is denoted by G2, and its output is denoted by G2(x). i This indicates that the fifth source model relates to G1(x) i ) and G2(x i The result of the superimposed reconstruction is obtained by overlay reconstruction x. i ′, x i ′ and x i The difference is x i ′-x i The loss function of the fifth source model is L. R L R With x i ′-x i The squares of x are positively correlated. Thus, x i ′ and x i The smaller the difference, the greater the sum of the features extracted by G1 and G2. i All features are beneficial for not losing x. i Features of [the text].

[0012] In one possible implementation, L R Satisfying the following formula L R =||x i ′-x i || 2 .

[0013] In one possible implementation, before inputting the audio information into the first model and obtaining the first output result, the method further includes: fixing the parameters of the first source model, the second source model, the fourth source model, and the fifth source model, training the third source model, and obtaining the second model when the third source model converges; fixing the parameters of the second model, and training the first source model, the second source model, the fourth source model, and the fifth source model; wherein the loss function used to train the first source model, the second source model, the fourth source model, and the fifth source model is L. G The loss function of the first source model is -L1, the loss functions of the second and fourth source models are both L2, and the loss function of the fifth source model is L... R L G Positively correlated with -L1, positively correlated with the product of the hyperparameters of L2 and the second source model, and positively correlated with L RThe product of the hyperparameters of the fifth source model is positively correlated. Therefore, training the third source model first, followed by the other models, improves training speed compared to training all models together. Furthermore, training one model first before training the others improves the compatibility of the other models, ensuring they all reach their optimal state. Optionally, the processing device can perform the model training steps to obtain the trained first source model, i.e., the first model. This first model can be transmitted to the terminal device, which can then input audio information into it to obtain the first output result. This helps conserve the computing power of the terminal device.

[0014] In one possible implementation, L G Satisfying the following formula: L G = -L1 + λ1*L2 + λ2*L R Where λ1 is the hyperparameter of the second source model and λ2 is the hyperparameter of the fifth source model. In this way, multiple models are trained together. 1, λ1, and λ2 represent the degree of attention paid to the loss function of each model. The loss functions of each model are summed, and it is assumed that the loss function is an accumulative process during joint training. Training each model using the summed loss function is beneficial for achieving the best overall performance through training multiple models together.

[0015] In one possible implementation, the scene represented by the first output is identified by a third model, which is jointly trained with the first model. During the training of the third model, some parameters in the first model and / or some parameters in the third model are adjusted until the third model converges and the output of the first model ensures that the accuracy of the scene identified by the third model on different devices meets the preset requirements. In this way, the third model is jointly trained with the first model, which helps to ensure compatibility between the third and first models and allows them to reach optimal performance simultaneously. The output of the first model ensures that the accuracy of the scene identified by the third model on different devices meets the preset requirements, which helps to reduce the probability that the training of the third model may result in inaccurate extraction of features for scene recognition.

[0016] In one possible implementation, during the training of the third model, some parameters in the first model and some parameters in the third model are adjusted. The loss function used to train both the first and third models is L. scene Audio information is represented by x. i "" indicates that the first model is denoted by G1′, and the output of the first model is denoted by G1′(x). i ") indicates that the third model is represented by D scene This indicates that the input to the third model is G1′(x). i The output of the third model is (D). scene [G1′(x i")]) k It means that (D) scene [G1′(x i ")]) k Used to represent the collected audio information x i "The predicted probability of the scene at that time is the k-th scene, and the true value of the audio information is represented by z." i,j It means that z i,k Used to represent the collected audio information x i "The scenario in which you are located is the true probability of the k-th scenario, where k is an integer greater than or equal to 1, L" scene With (D) scene [G1′(x i ")]) k and z i,k This is relevant. In this way, the loss function used to train the first model is the same as the loss function used to train the third model, which helps to optimize the performance of both models simultaneously.

[0017] In one possible implementation, L scene The following formula is satisfied: In this way, the loss function of the first model and the loss function of the third model are both cross-entropy loss functions, which is beneficial to improving the convergence speed of the first model and the third model.

[0018] In one possible implementation, the method of acquiring audio information collected by the microphone of the terminal device includes: acquiring audio information in response to a call being connected; after identifying the scenario represented by the first output result, the method further includes: if the scenario indicates that the number of sounds included in the audio information is greater than a second preset value, then amplifying the volume of the sound to be played by the terminal device and playing the amplified sound. This amplifying the sound to be played by the terminal device and playing the amplified sound in scenarios with a large number of sounds helps the user hear the sound played by the terminal device clearly.

[0019] In a second aspect, this application provides a terminal device including a processor and a memory, wherein the memory is used to store code instructions and the processor is used to execute the code instructions to perform the method described in any possible implementation of the first aspect.

[0020] Thirdly, this application provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the methods described in any possible implementation of the first aspect.

[0021] Fourthly, this application provides a computer program product comprising a computer program that, when run, causes a computer to perform the method described in any possible implementation of the first aspect.

[0022] Fifthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0023] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0024] It should be understood that the second to fifth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application;

[0026] Figure 2 This is a software structure block diagram of a terminal device according to an embodiment of this application;

[0027] Figure 3 A schematic diagram of a model network architecture provided in an embodiment of this application;

[0028] Figure 4 A schematic diagram illustrating a model training method provided in an embodiment of this application;

[0029] Figure 5 A schematic flowchart illustrating a scene recognition method provided in an embodiment of this application;

[0030] Figure 6 This is a schematic block diagram of a chip provided in an embodiment of this application. Detailed Implementation

[0031] To facilitate a clear description of the technical solutions in the embodiments of this application, the following explanation is provided first:

[0032] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with essentially the same function and purpose. For example, "first model" and "second model" are used only to distinguish different models and do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0033] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0034] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0035] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or it can refer to a period of time before or after the occurrence of a certain situation. The embodiments of this application do not impose a specific limitation on this. In addition, the display interface provided in the embodiments of this application is only an example, and the display interface may include more or less content.

[0036] The terminal device can collect audio information from its environment, input this information into a scene recognition model to determine the scene in which the device is located, and then determine whether to process the audio being played or the audio being collected based on the scene. For example, the terminal device, through the scene recognition model, identifies its environment as a subway scene based on the collected audio information. In this subway scene, if the terminal device is in call mode, it can perform noise reduction processing on the collected sound and amplify the volume of the audio being played.

[0037] Currently, scene recognition models have poor robustness, resulting in different recognition accuracy rates for the same scene recognition model on different terminal devices.

[0038] The robustness of scene recognition models is poor because audio information is greatly affected by the channel through which it is collected. Different terminal devices have different microphone layouts and / or models, resulting in different channels for collecting audio information. This affects the audio features in the audio information collected by different terminal devices, causing the scene recognition model to be unable to accurately extract the audio features. Consequently, the scene recognition model has different recognition accuracy rates on different terminal devices.

[0039] If, in order to improve the robustness of the scene recognition model, a scene recognition model is trained separately for different terminal devices, then a large amount of data needs to be collected for each terminal device, and a scene recognition model suitable for that device needs to be trained based on this data. Due to the diversity of terminal devices, a lot of manpower and time are required, resulting in high model training costs.

[0040] In view of this, embodiments of this application provide a scene recognition method and related apparatus, which can extract features from audio information for scene recognition, identify the scene of the collected audio information based on the features for scene recognition, reduce the influence of features in the audio information generated by different acquisition devices on scene recognition, and thus help improve the robustness of the scene recognition model.

[0041] The scene recognition method provided in this application can be applied to any terminal device, such as mobile phones, tablets, foldable screens, watches, or headphones. The training method for the scene recognition model involved in this application can be performed by a processing device with any processing capability, such as a desktop computer or server. The scene recognition model trained using the model training method provided in this application can be deployed on any type of terminal device.

[0042] To better understand the embodiments of this application, the hardware structure of the terminal device will be introduced first.

[0043] Figure 1 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application is shown. Figure 1As shown, the terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0044] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the terminal device. In other embodiments of this application, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0045] The terminal device can implement audio functions, such as music playback, recording, and calls, through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, and an application processor. The audio module 170 converts digital audio information into analog audio signals for output, and also converts analog audio input into digital audio signals. The speaker 170A, also called a "loudspeaker," converts audio electrical signals into sound signals. The terminal device can listen to music or make hands-free calls through the speaker 170A. The receiver 170B, also called a "handpiece," converts audio electrical signals into sound signals. When the terminal device answers a phone call or voice message, the receiver 170B can be brought close to the user's ear to hear the voice. The microphone 170C, also called a "microphone" or "voice transducer," converts sound signals into electrical signals.

[0046] For example, the terminal device can collect audio information through the microphone 170C and identify the audio information using the scene recognition model provided in this application embodiment to obtain the scene in which the audio information was collected. If the scene is a subway scene and the terminal device is in call mode, the terminal device can amplify the volume of the audio data to be played and play the amplified sound through the speaker 170A or the receiver 170B.

[0047] In addition to the aforementioned components, the device also runs an operating system, such as iOS, Android, or Windows. Applications can be installed and run on this operating system.

[0048] The software system of terminal devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc., which will not be elaborated here.

[0049] This application uses the layered architecture of the Android system as an example to illustrate the software structure of a terminal device.

[0050] Figure 2 This is a software structure block diagram of a terminal device according to an embodiment of this application. For example... Figure 2 As shown, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer (HAL), kernel layer, and hardware layer.

[0051] The application layer can include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the framework layer. For example... Figure 2 As shown, the application package can include applications such as phone, music, and voice assistant. The phone application can trigger the microphone, as described above. Figure 1 The microphone 170 shown collects audio information and can trigger a speaker or receiver, for example... Figure 1 The speaker 170A or receiver 170B shown plays sound. Music applications can trigger the speaker, for example... Figure 1 The speaker 170A shown is playing music. The voice assistant application can trigger the microphone, for example... Figure 1 The microphone 170 shown collects audio information and can trigger a speaker or receiver, for example... Figure 1The speaker 170A or receiver 170B shown plays sound.

[0052] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0053] like Figure 2 As shown, the application framework layer can include a window manager service and a display manager service, among others. The window manager manages window applications. It can obtain the screen size, determine the presence of a status bar, lock the screen, allow screen touch, drag the screen, and capture the screen. The display manager manages the display lifecycle, determining how to control its logical display based on the currently connected physical display device, and sending notifications to the system and applications when the state changes. The application framework layer can transmit audio data triggered by applications in the application layer to the hardware abstraction layer. For example, when a user triggers a music application to play music, the application framework layer can obtain the audio data from the music application in the application layer and transmit that audio data to the hardware abstraction layer.

[0054] The purpose of the HAL layer is to abstract hardware, providing a unified interface for upper-layer applications to query hardware devices, or to provide data storage services for upper-layer applications. For example... Figure 2 As shown, this HAL layer can include a display driver module and a sensor hardware abstraction layer (sensor hidl). The display driver module controls the display driver to show images, interfaces, and notification information on the screen. The sensor hardware abstraction layer can control the sensors and monitor event change notifications from the sensor drivers. The hardware abstraction layer can obtain audio data from the application framework layer and transmit the audio data to the kernel layer.

[0055] The kernel layer is the layer between hardware and software. For example... Figure 2 As shown, the kernel layer may include one or more of the following: sensor drivers and display drivers, etc. The kernel layer can obtain audio data from the hardware abstraction layer and transmit audio data to the kernel layer.

[0056] The hardware layer may include hardware such as a microphone, a digital signal processing (DSP) chip, and a display screen. The DSP chip deploys a scene recognition model, which can obtain audio information captured by the microphone, identify the scene in which the terminal device is located based on the audio information, and determine whether to process the audio data obtained from the kernel layer, and / or whether to process the audio information captured by the microphone.

[0057] In some embodiments, layers that perform the same function may be called by other names, or layers that can perform the functions of multiple layers may be considered as one layer, or layers that can perform the functions of multiple layers may be divided into multiple layers. This application does not impose any limitations on these embodiments.

[0058] The above combination Figure 1 and Figure 2 The hardware structure and software architecture of the terminal device have been introduced. The following section will combine... Figure 3 The model network architecture of this application embodiment is described.

[0059] Figure 3 A schematic diagram of a model network architecture provided in an embodiment of this application is shown. Figure 3 As shown, the model network architecture includes a device-independent feature generator G1, a device-related feature generator G2, a device discriminator D1, a device discriminator D2, and a decoder D. e and scene classifier D scene .

[0060] The device-independent feature generator G1 takes audio information collected from multiple terminal devices as input. G1 extracts device-independent features, which represent features in the audio information used for scene recognition. The features extracted by G1 confuse the device discriminator D1, preventing it from correctly classifying the device. In one example, the loss function for G1 can be the cross-entropy loss function; training of G1 ends when the cross-entropy loss function is minimized.

[0061] The input to device discriminator D1 is the output of device-independent feature generator G1. The output of device discriminator D1 can be multiple probabilities, each corresponding to a specific terminal device. Each probability represents the probability that the device collecting the audio information is the correct device. Device discriminator D1 is used to identify the features output by device-independent feature generator G1 as the correct device collecting the audio information. Therefore, the training objective of device discriminator D1 is to maximize the probability of correctly identifying the device. In one example, the loss function for device discriminator D1 can be the cross-entropy loss function. Training of device discriminator D1 ends when the cross-entropy loss function is minimized.

[0062] The input to the device-related feature generator G2 is audio information collected from multiple terminal devices. G2 extracts device-related features, which represent features in the audio information used for device identification. The features extracted by G2 enable the device discriminator D2 to correctly classify the devices. In one example, the loss function of G2 can be the cross-entropy loss function; training of G2 ends when the cross-entropy loss function is minimized.

[0063] The input to device discriminator D2 is the output of device-related feature generator G2. The output of device discriminator D2 can be multiple probabilities, each corresponding to a specific terminal device. Each probability represents the probability of correctly identifying the device collecting audio information as its corresponding device. Device discriminator D2 uses the features output by device-related feature generator G2 to determine if it is the correct device collecting audio information; therefore, the training objective of device discriminator D2 is to maximize the probability of correctly identifying the device. In one example, the loss function of device discriminator D2 can be the cross-entropy loss function. Training of device discriminator D2 ends when the cross-entropy loss function is minimized.

[0064] Decoder D e The inputs include the outputs of the device-independent feature generator G1 and the outputs of the device-dependent feature generator G2. Decoder D e Used to reconstruct the input information. For decoder D... e The training objective is to reduce the gap between the reconstructed information and the audio information. Here, audio information refers to the input of the device-independent feature generator G1, or more specifically, the input of the device-related feature generator G2. Since the features output by the device-related feature generator G2 include features used for device identification, a smaller gap between the reconstructed information and the audio information helps ensure that the features output by the device-independent feature generator G1 retain more of the original audio information, i.e., the features used for scene identification, thus preventing the loss of key acoustic features.

[0065] Scene classifier D scene The input is the output of the device-independent feature generator G1, and the scene classifier D. scene The output of the scene classifier D is the probability of correctly identifying the scene in which the audio information is located. scene Used to identify the output of the device-independent feature generator G1 as the scene in which the audio information was acquired.

[0066] The device-independent feature generator G1 and device discriminator D1 can be constructed into one GAN network, and the device-related feature generator G2 and device discriminator D2 can be constructed into another GAN network. The device-independent feature generator G1 and scene classifier D... scene This can be constructed as a scene recognition model. The embodiments in this application are based on a dual GAN ​​network, using decoder D... e As a constraint, training the device-independent feature generator G1 in the scene recognition model helps the device-independent feature generator G1 to more accurately extract the features in the audio information that are truly related to the essence of the scene, i.e. the features used for scene recognition. This reduces the impact of features generated by different acquisition devices on scene recognition and helps improve the robustness of the scene recognition model.

[0067] The scene recognition model includes a device-independent feature generator G1 and a scene classifier D. scene In this embodiment, the device-independent feature generator G1 can be trained first to extract features from audio information for scene recognition. Then, the trained device-independent feature generator G1 is combined with the scene classifier D. scene Joint optimization, or combined training, is used to improve the compatibility of the two models and enable both to achieve better results simultaneously.

[0068] First, we introduce the method for training the device-independent feature generator G1. We then train the aforementioned device-independent feature generator G1, device-related feature generator G2, device discriminator D1, device discriminator D2, and decoder D. e In this case, the cross-entropy loss function can be used for all loss functions. If the cross-entropy loss function is used for all loss functions, the loss function of the device-independent feature generator G1 can be represented by -L1, which satisfies the formula: Where, x i G1(x) represents the i-th audio information among audio information collected by multiple terminal devices. i () represents the output of the device-independent feature generator G1, (D1[G1(x)) i )]) j This represents the output of device discriminator D1, (D1[G1(x i )]) j y is used to represent the predicted probability that the device collecting the i-th audio information is the j-th device. i,j Let y represent the truth value of the i-th audio message. i,j This indicates that the i-th audio information x is being collected. i The probability that a device is the j-th device is denoted by , where j is an integer greater than or equal to 1. The loss function of the device discriminator D1 is negative of the loss function of the device-independent feature generator G1. The loss function of the device discriminator D1 can be represented by L1, which satisfies the formula:

[0069] The loss function of the device-related feature generator G2 can be represented by L2, and L2 satisfies the following formula Where, x i G2(x) represents the i-th audio information among audio information collected by multiple terminal devices. i () represents the output of device-related feature generator G2, (D2[G2(x)) i )]) j This represents the output of device discriminator D2, (D2[G2(x i )]) j y is used to represent the predicted probability that the device collecting the i-th audio information is the j-th device. i,j Let y represent the truth value of the i-th audio message. i,j This indicates that the i-th audio information x is being collected. i The device is the true probability of the j-th device, where j is an integer greater than or equal to 1. The loss function of the device discriminator D2 is the same as the loss function of the device-related feature generator G2, both being L2. The formula for L2 is as above, and will not be repeated here.

[0070] Decoder D e The loss function can be L R It means, L R The formula L that is satisfied R =||x i ′-x i || 2 , where x i Let x represent the i-th audio information among the audio information collected by multiple terminal devices. i ′ represents decoder D e The output.

[0071] It should be noted that the formulas for the loss functions described above are all based on the i-th audio information as an example. The audio information collected by multiple terminal devices includes multiple such audio information. Based on the above formulas, multiple loss function values ​​can be obtained. During the training process, the average of these multiple loss functions can be used as the total loss function to train the model, or the sum of some of these multiple loss functions can be used as the total loss function to train the model.

[0072] For example, the sample data contains audio information collected by five different types of terminal devices, totaling 10,000 audio messages. These five different types of terminal devices can be terminal device 1, terminal device 2, terminal device 3, terminal device 4, and terminal device 5. If the input to the device-independent feature generator G1 is the first audio message among these 10,000 audio messages, and this audio message was collected by terminal device 1, then the truth value corresponding to this audio message is represented by y.1,j It means that, among them, y 1,1 =1, y 1,2 =0, y 1,3 =0, y 1,4 =0, y 1,5 =0. If the output of device discriminator D1 is (D1[G1(x1)]) j It means that, among them,

[0073] This is used to represent the probability that the device collecting the audio information is terminal device 1. This is used to represent the probability that the device collecting the audio information is terminal device 2. This is used to represent the probability that the device collecting the audio information is terminal device 3. This is used to represent the probability that the device collecting the audio information is terminal device 4. The probability that the device collecting the audio information is terminal device 5 is used to represent the cross-entropy loss function of the device discriminator D1. Similarly, each of these 10,000 audio messages corresponds to a loss function value, resulting in 10,000 loss function values. The processing device can use the average of these 10,000 loss function values ​​as the total loss function value. Training ends when this total loss function value is less than a preset value. Alternatively, the total loss function value can be the sum of several loss function values ​​among these 10,000. Training also ends when this total loss function value is less than a preset value.

[0074] For example, the processing device can sum five loss function values ​​out of 10,000 as the total loss function value. Training ends when this total loss function value is less than a preset value. (Using decoder D...) e Let's take an example to illustrate. Decoder D e The inputs are the outputs of the device-independent feature generator G1 and the outputs of the device-dependent feature generator G2, and the decoder D. e The output is x i If the inputs to both the device-independent feature generator G1 and the device-dependent feature generator G2 can be 10,000 audio information entries, then the decoder D... e The output contains 10,000 values, and the processing device is based on decoder D. e The output and the truth value of the audio information can be obtained from 10,000 L values. R Furthermore, the sum of five random loss functions can be selected as the total loss function value, which can be expressed as: Where m is an integer from 1 to 5. When the total loss function value is less than the preset value, the decoder D... e Training is over.

[0075] Based on the above Figure 3 The model network architecture shown can be used to train the device-independent feature generator G1 using various training methods.

[0076] In a first possible implementation, the processing device can handle the device-independent feature generator G1, the device-related feature generator G2, the device discriminator D1, the device discriminator D2, and the decoder D. e Each model is trained individually to complete the training of the device-independent feature generator G1. This individual training of each model helps to achieve optimal performance for each model.

[0077] In a second possible implementation, the processing device can first train the device discriminator D1 separately, and then train the device-independent feature generator G1, the device-related feature generator G2, the device discriminator D2, and the decoder D. e The device-independent feature generator G1 can be trained together. Alternatively, the processing device can first train the device discriminator D2 separately, and then train the device-independent feature generator G1, the device-related feature generator G2, the device discriminator D1, and the decoder D2 together. e Perform training together to complete the training of the device-independent feature generator G1.

[0078] In this way, training the device discriminator D1 or D2 first, and then training the other models, can improve the training speed compared to training all models together. In addition, training one model first and then training the other models together can improve the compatibility of the other models and enable all other models to reach their optimal state.

[0079] In a third possible implementation, the processing device can first train the device discriminator D1 and the device discriminator D2 together, and then train the device-independent feature generator G1, the device-related feature generator G2, and the decoder D. e Training is performed together to complete the training of the device-independent feature generator G1. Thus, by constraining the device-independent feature generator G1 through the device discriminator D1 and the device-related feature generator G2 through the device discriminator D2, it is beneficial to more quickly train the device-independent feature generator G1, the device-related feature generator G2, and the decoder D... e Convergence can help improve training speed.

[0080] It should be noted that the embodiments in this application are not limited to using these three possible implementation methods, based on the above. Figure 3 The model network architecture shown, and the methods for training the device-independent feature generator G1, are all within the protection scope of the embodiments of this application.

[0081] To better understand the embodiments of this application, this application will describe the training method in detail using the second possible implementation described above as an example.

[0082] For example, Figure 4 A schematic diagram of a model training method provided in an embodiment of this application is shown. This method can be applied to processing devices. Figure 4 As shown, the method includes the following steps:

[0083] S401, Fixed equipment-independent feature generator G1, Equipment discriminator D2, Equipment-related feature generator G2, and decoder D e The parameters are used to train the device discriminator D1. When the loss function L1 is less than the threshold a, the trained device discriminator D1 is obtained.

[0084] The input to device discriminator D1 is the output of device-independent feature generator G1. The input to device-independent feature generator G1 is audio information. The ground truth value of the audio information represents the true probability that the device collecting the audio information is one of the various devices. The output of device discriminator D1 represents the predicted probability that the device collecting the audio information is one of the various devices. The processing device inputs the ground truth value of the audio information and the output of device discriminator D1 into a formula satisfied by L1. If L1 is less than a threshold a, the trained device discriminator D1 is obtained. If L1 is greater than or equal to the threshold a, the processing device updates the parameters of device discriminator D1 to make L1 less than the threshold a. The threshold a is a preset small value, such as 0.01, 0.05, etc.

[0085] S402. Fix the parameters of the trained device discriminator D1, and modify the device-independent feature generator G1, device discriminator D2, device-related feature generator G2, and decoder D. e During training, when the loss function L R When the value is less than the threshold b, the trained device-independent feature generator G1, the trained device discriminator D2, the trained device-related feature generator G2, and the trained decoder D are obtained. e Among them, L G Satisfying the formula: L G = -L1 + λ1*L2 + λ2*L R λ1 represents the hyperparameters of the device discriminator D2 and the device-related feature generator G2, and λ2 represents the decoder D. e Hyperparameters.

[0086] The inputs to the device-independent feature generator G1 and the device-dependent feature generator G2 are the same, both being audio information. The input to the device discriminator D1 is the output of the device-independent feature generator G1, and the input to the device discriminator D2 is the output of the device-dependent feature generator G2. The decoder D... eThe inputs are the outputs of the device-independent feature generator G1 and the device-dependent feature generator G2. The true value of the audio information can be the true probability that the device acquiring the audio information is any of the various devices. The outputs of both the device discriminator D1 and the device discriminator D2 can be the predicted probabilities that the device acquiring the audio information is any of the various devices. The processing device can input the true value of the audio information and the output of the device discriminator D1 into the formula satisfied by -L1 to obtain -L1, and the processing device can input the true value of the audio information and the output of the device discriminator D2 into the formula satisfied by L2 to obtain L2. The processing device can then input the audio information and the decoder D... e Output input to L R The formula that satisfies L is obtained. R and L1, L2 and L R Enter to L G The formula that satisfies L is obtained. G If L G When the value is less than the threshold b, the trained device-independent feature generator G1, the trained device discriminator D2, the trained device-related feature generator G2, and the trained decoder D are obtained. e If L G If the value is greater than or equal to the threshold b, then the processing device updates the device-independent feature generator G1, the device discriminator D2, the device-related feature generator G2, and the decoder D. e The parameters of L R Less than the threshold b. λ1 is a hyperparameter of the device discriminator D2 and the device-related feature generator G2, used to balance L2, and λ2 is the decoder D. e The hyperparameters used to balance L R The device discriminator D2 and the device-related feature generator G2 have the same loss function and hyperparameters, so when training them together, a single model's loss function and hyperparameters can be used.

[0087] The model training method provided in this application first trains the device discriminator D1, and then trains the device-independent feature generator G1, the device discriminator D2, the device-related feature generator G2, and the decoder D. e Compared to training all models together, this method improves training speed. Furthermore, by training the device discriminator D1 first, and then training the device-independent feature generator G1, device discriminator D2, device-related feature generator G2, and decoder D together, the training speed can be improved. e This is beneficial for improving the device-independent feature generator G1, device discriminator D2, device-related feature generator G2, and decoder D. e To ensure compatibility, implement the device-independent feature generator G1, device discriminator D2, device-related feature generator G2, and decoder D. e Both can reach the optimal state.

[0088] Optionally, when the processing device executes S401 and S402 above, it can record the loss function calculated by each model due to parameter updates. After executing S402, the processing device can determine whether the change in the value of the loss function calculated by each model due to parameter updates is within a preset fluctuation range, or whether the value of the loss function calculated by each model due to parameter updates is less than or equal to a preset threshold. If the change in the value of the loss function of one or more models is not within the preset fluctuation range, or if the value of the loss function of one or more models is greater than the preset threshold, it can be said that the value of the parameter of one or more models is inaccurate. Then, the hyperparameters corresponding to one or more models can be increased or decreased, for example, increasing λ1 and decreasing λ2, and S401 and S402 above can be re-executed based on the increased or decreased hyperparameters.

[0089] After the processing device executes S401 and S402 as described above, it can determine whether the number of times S401 and S402 are executed is greater than the preset number. If the number of times S401 and S402 are executed is greater than or equal to the preset number, the training ends. If the number of times S401 and S402 are executed is less than the preset number, S401 and S402 are executed again until the number of times S401 and S402 are executed is greater than or equal to the preset number.

[0090] The above describes the method for training the device-independent feature generator G1. The following describes how to train the device-independent feature generator G1 and the scene classifier D. scene A method of training together. Wherein, the scene classifier D... scene It can be used as an existing classifier for identifying the scene in which audio information is located.

[0091] Training scene classifier D scene In this case, the cross-entropy loss function can be used. If the cross-entropy loss function is used, then the scene classifier D... scene The loss function can be L scene It means, L scene Satisfying the formula: Where, x i " represents the i-th audio information among audio information collected from multiple terminal devices, and G1′ represents the trained device-independent feature generator G1, G1′(x i ") represents the output of G1', (D scene [G1′(x i ")]) k Represents the scene classifier D scene The output, (D) scene [G1′(x i ")]) k Used to represent the acquisition of the i-th audio information xi The current scene is the predicted probability of the k-th scene, z. i,j z represents the truth value of the i-th audio message. i,j This indicates that the i-th audio information x is being collected. i The scenario in which the user is located is the true probability of the k-th scenario, where k is an integer greater than or equal to 1.

[0092] G1′ and D scene The samples trained together can be the same as those trained by the device-independent feature generator G1, i.e., x i "With x i The samples can be the same, or they can be different from those of the feature generator G1, which is independent of the training equipment described above. This application does not limit this in its embodiments. If x i "With x i The same principle applies, which allows for fewer changes to the parameters in G1′ during training, and enables G1′ to extract features for scene recognition more accurately.

[0093] This application embodiment focuses on G1′ and D. scene When training together, it can be fine-tuned, that is, only G1′ and / or D can be changed during training. scene The parameters of a few layers or the parameters of the last layer of the network can help make the two models compatible without changing the functions that were previously present.

[0094] This application provides various methods for handling G1′ and D. scene Methods for training together.

[0095] The first possible implementation is to fix the parameters of G1′ and adjust D. scene During training, when the loss function L scene Upon convergence, we obtain the trained D. scene Therefore, only D is updated. scene The parameters are not updated during training of G1′, which helps G1′ to extract features for scene recognition more accurately.

[0096] The second possible implementation involves G1′ and D. scene During training, when the loss function L scene Upon convergence, we obtain the trained G1′ and the trained D. scene And by testing the training samples, the trained G1′ and trained D scene The training process is repeated after testing. If the accuracy of scene recognition based on audio information collected from different devices meets the requirements, the training ends. If the accuracy does not meet the requirements, the training is repeated. This updates the parameters of G1′ and D. sceneThe parameters are designed to make the two models compatible, enabling them to reach the optimal state simultaneously. They also help reduce the probability of decreasing the accuracy of features extracted for scene recognition due to updating the parameters of G1′.

[0097] The third possible implementation is to fix D. scene The parameters are used to train G1′, and the loss function L is used. scene Upon convergence, the trained G1′ is obtained, and G1′ and D are compared using test samples. scene The training process is repeated after testing. If the accuracy of scene recognition based on audio information collected from different devices meets the requirements, the training ends. If the accuracy does not meet the requirements, the training continues. In this way, only the parameters of G1′ are updated, and D is not updated. scene The parameters are designed to make the two models compatible, enabling them to reach the optimal state simultaneously. They also help reduce the probability of decreasing the accuracy of features extracted for scene recognition due to updating the parameters of G1′.

[0098] This application embodiment relates to G1′ and D scene After training, the trained model can be called a scene recognition model. This scene recognition model includes a first model and a third model. The first model is the trained G1′, which is used to extract features from the audio information for scene recognition. The third model is the trained D... scene This system can identify the scene in which audio information is collected based on features used for scene recognition. The scene recognition model can be deployed on terminal devices. When the terminal device is in a call or voice assistant mode, the collected audio information is input into the scene recognition model. The model outputs the scene in which the terminal device is located and, based on the scene, determines whether to perform noise reduction processing on the collected audio information and whether to amplify the volume of the played sound.

[0099] The following section will introduce how to apply the model.

[0100] Figure 5 A schematic flowchart illustrating a scene recognition method provided in an embodiment of this application is shown. This method can be applied to terminal devices. Figure 5 As shown, the method may include the following steps:

[0101] S501. Obtain audio information collected by the microphone of the terminal device.

[0102] The hardware structure of the terminal device can be as described above. Figure 1 As shown above, in the above Figure 1 In the architecture shown, the microphone can be a microphone 170C. The terminal device can acquire the audio information captured by the microphone through the microphone.

[0103] S502. Input the audio information into the first model to obtain the first output result.

[0104] The first model is used to extract features from the audio information for subsequent scene recognition. The first output represents the output of the first model based on the audio information. The first model can be G1′ trained in the above embodiment.

[0105] S503. Identify the scene represented by the first output result, including scenes distinguished by noise in the audio and / or the number of sounds included in the audio.

[0106] The scenarios can include subway, bus, street, and home environments, which can be distinguished by noise and / or the number of sounds. The noise levels can differ across scenarios; these noises can also be understood as standard sound events in different scenarios. For example, a standard sound event on a street is a car horn, while a standard sound event on a subway is the sound of a light rail train. This application does not limit the specific implementation method for identifying the scenario represented by the first output result.

[0107] The first model in this embodiment can be trained using a first source model, a second source model, a third source model, a fourth source model, and a fifth source model. The inputs to both the first and second source models are audio samples. The output of the first source model is the input of the third source model, and the output of the second source model is the input of the fourth source model. The outputs of both the first and second source models are also the inputs of the fifth source model. The outputs of the third and fourth source models are the device types used to collect the audio samples, with the third source model outputting incorrect device types and the fourth source model outputting correct device types. The fifth source model is used to superimpose and reconstruct the outputs of the first and second source models, and converges when the difference between the superimposed and reconstructed result and the audio sample is less than a first preset value. When both the fourth and fifth source models converge, the training of the first source model ends, and the first model is obtained.

[0108] The audio sample can be x in the above embodiments. i The first source model can be the device-independent feature generator G1 in the above embodiments, the second source model can be the device-related feature generator G2, the third source model can be the device discriminator D1, the fourth source model can be the device discriminator D2, and the fifth source model can be the decoder D. e The first model can be used to extract features for scene recognition. The specific architecture can be found above. Figure 3 This will not be elaborated upon here.

[0109] The scene recognition method provided in this application embodiment can constrain the second source model when the fourth source model converges, which is beneficial for the second source model to extract more accurate features for device type recognition. When the fifth model converges, and the second source model can extract more accurate features for device type recognition, it is beneficial for constraining the first source model to extract features for scene recognition, thus obtaining the first model. In use, the first model can extract features for scene recognition from audio information. Based on these features, the scene represented by the features can be recognized, which is beneficial for recognizing scenes on different terminal devices and improving the robustness of scene recognition.

[0110] Optionally, audio samples are represented by x. i The first source model is denoted by G1, and its output is denoted by G1(x). i The third source model is represented by D1, and its input is G1(x). i The output of the third source model is (D1[G1(x)). i )]) j It means that (D1[G1(x)) i )]) j Used to represent the collected audio sample x i Let be the predicted probability of the j-th device, and let y be the ground truth value of the audio sample. i,j It means that y i,j Used to represent the collected audio sample x i The device is the true probability of the j-th device, where j is an integer greater than or equal to 1. The loss function of the first source model is -L1, and the loss function of the third source model is L1. L1 and (D1[G1(x)) are the true probabilities of the j-th device. i )]) j and y i,j In this way, the loss function of the first source model is negatively correlated with the loss function of the third source model. The output of the first source model is the input of the third source model. The features generated by the first source model are used to confuse the third source model, preventing it from outputting the correct device category. This is beneficial for training the first source model to extract features for scene recognition and for training the third source model to prevent it from outputting the correct device category.

[0111] Optionally, L1 satisfies the following formula In this way, the loss function of the third source model is the cross-entropy loss function, which is beneficial to improving the convergence speed of the third source model.

[0112] Optionally, audio samples are represented by x. i The second source model is denoted by G2, and its output is denoted by G2(x). i The fourth source model is represented by D2, and its input is G2(x). iThe output of the fourth source model is (D2[G2(x)). i )]) j It means that (D2[G2(x)) i )]) j Used to represent the collected audio sample x i Let be the predicted probability of the j-th device, and let y be the ground truth value of the audio sample. i,j It means that y i,j Used to represent the collected audio sample x i The device is the true probability of the j-th device, where j is an integer greater than or equal to 1. The loss function of the second source model and the loss function of the fourth source model are both L2. L2 and (D2[G2(x)) are the true probabilities of the j-th device. i )]) j and y i,j This is relevant. In this way, the loss function of the second source model is the same as that of the fourth source model, and the output of the second source model is the input of the fourth source model. This is beneficial for training the second source model to extract features for device identification and for training the fourth source model to output the correct device category.

[0113] Optionally, L2 satisfies the following formula In this way, the loss function of the second source model and the loss function of the fourth source model are both cross-entropy loss functions, which is beneficial to improving the convergence speed of the second source model and the fourth source model.

[0114] Optionally, audio samples are represented by x. i The first source model is denoted by G1, and its output is denoted by G1(x). i The second source model is denoted by G2, and its output is denoted by G2(x). i This indicates that the fifth source model relates to G1(x) i ) and G2(x i The result of the superimposed reconstruction is obtained by overlay reconstruction x. i ′, x i ′ and x i The difference is x i ′-x i The loss function of the fifth source model is L. R L R With x i ′-x i The squares of x are positively correlated. Thus, x i ′ and x i The smaller the difference, the greater the sum of the features extracted by G1 and G2. i All features are beneficial for not losing x. i Features of [the text].

[0115] Optionally, L RSatisfying the following formula L R =||x i ′-x i || 2 .

[0116] Optionally, before inputting the audio information into the first model to obtain the first output result, the method further includes: fixing the parameters of the first source model, the second source model, the fourth source model, and the fifth source model, training the third source model, and obtaining the second model when the third source model converges; fixing the parameters of the second model, and training the first source model, the second source model, the fourth source model, and the fifth source model; wherein the loss function used to train the first source model, the second source model, the fourth source model, and the fifth source model is L. G The loss function of the first source model is -L1, the loss functions of the second and fourth source models are both L2, and the loss function of the fifth source model is L... R L G Positively correlated with -L1, positively correlated with the product of the hyperparameters of L2 and the second source model, and positively correlated with L R And the product of the hyperparameters of the fifth source model is positively correlated. For a detailed training process, please refer to the above. Figure 4 This will not be elaborated upon here. In this way, training the third source model first, and then training the other models, can improve the training speed compared to training all models together. In addition, training one model first and then training the other models together is beneficial to improving the compatibility of the other models and ensuring that all other models can reach their optimal state.

[0117] Optionally, L G Satisfying the following formula: L G = -L1 + λ1*L2 + λ2*L R Where λ1 is the hyperparameter of the second source model and λ2 is the hyperparameter of the fifth source model. In this way, multiple models are trained together. 1, λ1, and λ2 represent the degree of attention paid to the loss function of each model. The loss functions of each model are summed, and it is assumed that the loss function is an accumulative process during joint training. Training each model using the summed loss function is beneficial for achieving the best overall performance through training multiple models together.

[0118] Optionally, the scene represented by the first output result is identified by a third model. The third model is jointly trained with the first model. During the training of the third model, some parameters in the first model and / or some parameters in the third model are adjusted until the third model converges and the output of the first model ensures that the accuracy of the scene identified by the third model on different devices meets the preset requirements. The third model is the D model trained in the above embodiment. sceneIn this way, the third model is trained in conjunction with the first model, which helps to make the third model compatible with the first model and achieve optimal performance simultaneously. The output of the first model ensures that the accuracy of the scene recognition by the third model on different devices meets the preset requirements, which helps to reduce the probability that the features for scene recognition cannot be accurately extracted due to the training of the third model.

[0119] Optionally, during the training of the third model, some parameters in the first model and some parameters in the third model are adjusted, with the loss function used to train both the first and third models being L. scene Audio information is represented by x. i "" indicates that the first model is denoted by G1′, and the output of the first model is denoted by G1′(x). i ") indicates that the third model is represented by D scene This indicates that the input to the third model is G1′(x). i The output of the third model is (D). scene [G1′(x i ")]) k It means that (D) scene [G1′(x i ")]) k Used to represent the collected audio information x i "The predicted probability of the scene at that time is the k-th scene, and the true value of the audio information is represented by z." i,j It means that z i,k Used to represent the collected audio information x i "The scenario in which you are located is the true probability of the k-th scenario, where k is an integer greater than or equal to 1, L" scene With (D) scene [G1′(x i ")]) k and z i,k This is relevant. In this way, the loss function used to train the first model is the same as the loss function used to train the third model, which helps to optimize the performance of both models simultaneously.

[0120] Optionally, L scene The following formula is satisfied: In this way, the loss function of the first model and the loss function of the third model are both cross-entropy loss functions, which is beneficial to improving the convergence speed of the first model and the third model.

[0121] Optionally, the method of acquiring audio information collected by the microphone of the terminal device includes: acquiring audio information in response to a call being connected; after identifying the scenario represented by the first output result, the method further includes: if the scenario indicates that the number of sounds included in the audio information is greater than a second preset value, then amplifying the volume of the sound to be played by the terminal device and playing the amplified sound. The second preset value can be a positive integer value, such as 3 or 4. In this way, amplifying the sound to be played by the terminal device and playing the amplified sound in scenarios with a large number of sounds helps the user hear the sound played by the terminal device clearly. In one example, the terminal device can also perform noise reduction processing on the audio information and transmit the noise-reduced audio information to the other end of the call to reduce the influence of other sounds on the audio information.

[0122] Optionally, the method of acquiring audio information collected by the microphone of the terminal device includes: acquiring audio information in response to the user's operation of activating the voice assistant; after recognizing the scene represented by the first output result, the method further includes: if the scene indicates that the noise in the audio information is greater than a third preset value, then performing noise reduction processing on the audio information and amplifying the volume of the sound to be played by the terminal device. The third preset value can be a threshold; exceeding this threshold indicates that there is a lot or a large amount of noise in the audio information. In order for the terminal device to accurately analyze the acquired audio information, noise reduction processing can be performed on the audio information; and in order for the user to hear the sound clearly, the volume of the sound to be played by the terminal device can be amplified.

[0123] The methods of the embodiments of this application have been described above. The apparatus for performing the above methods, provided in the embodiments of this application, is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the embodiments of this application can perform the steps in the above methods.

[0124] To achieve the above functions, the apparatus for implementing the scene recognition method includes hardware structures and / or software modules corresponding to each function. Those skilled in the art will readily recognize that, based on the method steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0125] This application embodiment can divide the device for implementing the scene recognition method into functional modules based on the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0126] like Figure 6 This is a schematic diagram of a chip structure provided in an embodiment of this application. The chip 60 includes one or more processors 601, communication lines 602, communication interfaces 603, and memory 604.

[0127] In some implementations, memory 604 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0128] The methods described in the embodiments of this application can be applied to, or implemented by, processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 601 or by instructions in software form. Processor 601 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. Processor 601 can implement or execute the various processing-related methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0129] The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). This storage medium is located in memory 604, and processor 601 reads information from memory 604 and, in conjunction with its hardware, completes the steps of the above method.

[0130] The processor 601, memory 604 and communication interface 603 can communicate with each other through communication line 602.

[0131] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0132] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disk, hard disk, or magnetic tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0133] This application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. The computer-readable medium may include computer storage media and communication media, and may also include any medium capable of transferring a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0134] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may also include disk storage or other disk storage devices. Furthermore, any connecting cable may also be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers.

[0135] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

Claims

1. A scene recognition method, characterized in that, include: Acquire audio information captured by the microphone of the terminal device; The audio information is input into the first model to obtain the first output result; Identify the scene represented by the first output result, the scene including scenes distinguished by noise in the audio and / or the number of sounds included in the audio; The first model is a trained first source model. The training process of the first source model is related to the training processes of the second, third, fourth, and fifth source models. The inputs of the first and second source models are audio samples. The output of the first source model is the input of the third source model, and the output of the second source model is the input of the fourth source model. The outputs of the first and second source models are also the inputs of the fifth source model. The outputs of the third and fourth source models are the device types that collected the audio samples. The third source model is used to output incorrect device types, and the fourth source model is used to output correct device types. The fifth source model is used to superimpose and reconstruct the outputs of the first and second source models. It converges when the difference between the superimposed and reconstructed result and the audio sample is less than a first preset value. When both the fourth and fifth source models converge, the training of the first source model ends, and the first model is obtained.

2. The method according to claim 1, characterized in that, The audio samples are used This indicates that the first source model uses This indicates that the output of the first source model is used... This indicates that the third source model uses... This indicates that the input to the third source model is the... The output of the third source model is used It means that the Used to indicate the acquisition of the audio samples The device is the predicted probability of the j-th device, and the ground truth value of the audio sample is... It means that the Used to indicate the acquisition of the audio samples The device is the true probability of the j-th device, where j is an integer greater than or equal to 1, and the loss function of the first source model is - The loss function of the third source model is The With the and stated Related.

3. The method according to claim 2, characterized in that, The Satisfy the following formula .

4. The method according to any one of claims 1 to 3, characterized in that, The audio samples are used This indicates that the second source model uses This indicates that the output of the second source model is used... This indicates that the fourth source model uses... This indicates that the input to the fourth source model is the... The output of the fourth source model is used It means that the Used to indicate the acquisition of the audio samples The device is the predicted probability of the j-th device, and the ground truth value of the audio sample is... It means that the Used to indicate the acquisition of the audio samples The device is the true probability of the j-th device, where j is an integer greater than or equal to 1. The loss function of the second source model and the loss function of the fourth source model are both... The With the and stated Related.

5. The method according to claim 4, characterized in that, The Satisfy the following formula .

6. The method according to any one of claims 1 to 3, characterized in that, The audio samples are used This indicates that the first source model uses This indicates that the output of the first source model is used... This indicates that the second source model uses This indicates that the output of the second source model is used... This indicates that the fifth source model relates to the... and stated Superimposed reconstruction yields the result after superimposed reconstruction. The and The difference is The loss function of the fifth source model is The With the The squares are positively correlated.

7. The method according to claim 6, characterized in that, The Satisfy the following formula .

8. The method according to any one of claims 1 to 3, characterized in that, Before inputting the audio information into the first model to obtain the first output result, the method further includes: The parameters of the first source model, the second source model, the fourth source model, and the fifth source model are fixed, and the third source model is trained. When the third source model converges, the second model is obtained. With the parameters of the second model fixed, the first source model, the second source model, the fourth source model, and the fifth source model are trained; wherein, the loss function used to train the first source model, the second source model, the fourth source model, and the fifth source model is: The loss function of the first source model is -L1, the loss functions of the second and fourth source models are both L2, and the loss function of the fifth source model is... The Positively correlated with -L1, positively correlated with the product of L2 and the hyperparameters of the second source model, and positively correlated with the And the product of the hyperparameters of the fifth source model is positively correlated.

9. The method according to claim 8, characterized in that, The Satisfy the following formula: ,in, These are the hyperparameters of the second source model. These are the hyperparameters of the fifth source model.

10. The method according to any one of claims 1 to 3, characterized in that, The scene represented by the first output result is identified by the third model, which is trained in conjunction with the first model. During the training of the third model, some parameters in the first model and / or some parameters in the third model are adjusted until the third model converges and the output of the first model makes the accuracy of the scene identified by the third model on different devices meet the preset requirements.

11. The method according to claim 10, characterized in that, During the training of the third model, some parameters in the first model and some parameters in the third model are adjusted. The loss function used to train the first model and the loss function used to train the third model are both... ; The audio information is used This indicates that the first model uses This indicates that the output of the first model is used This indicates that the third model uses... This indicates that the input to the third model is... The output of the third model is used It means that the Used to indicate the acquisition of the audio information The predicted probability of the scene at that time is the k-th scene, and the truth value of the audio information is used as... It means that the Used to indicate the acquisition of the audio information The scenario in which the user is located is the true probability of the k-th scenario, where k is an integer greater than or equal to 1. With the and stated Related.

12. The method according to claim 11, characterized in that, The The following formula is satisfied: .

13. The method according to any one of claims 1 to 3, characterized in that, The acquisition of audio information collected by the microphone of the terminal device includes: In response to the operation of connecting a call, the audio information is acquired; After identifying the scene represented by the first output result, the method further includes: If the scenario indicates that the number of sounds included in the audio information is greater than a second preset value, then the volume of the sound to be played by the terminal device is amplified, and the amplified sound is played.

14. A terminal device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the terminal device to perform the method as described in any one of claims 1-13.

15. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-13.

16. A computer program product, characterized in that, Includes a computer program that, when run, causes a computer to perform the method as described in any one of claims 1-13.

Images