Scene recognition method and related device

Abstract

The invention provides a scene recognition method and a related device, and is applied to the technical field of terminals. The method comprises the following steps: acquiring audio information collected by a microphone of terminal equipment; inputting the audio information into a first model to obtain a first output result; identifying a scene represented by the first output result; wherein the first model is obtained by training a first source model, a second source model, a third source model, a fourth source model and a fifth source model, so that the first model can extract features for scene recognition. Thus, the first model can extract the features used for scene recognition in the audio information, the scene for collecting the audio information is recognized based on the features used for scene recognition, the influence of the features generated due to different collection devices in the audio information on scene recognition is reduced, and then the robustness of scene recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a scene recognition method and related devices. Background Art

[0002] The terminal device can collect audio information of the environment in which the terminal device is located, input the audio information into the scene recognition model, obtain the scene in which the terminal device is located, and determine whether to process the played audio or the collected audio based on the scene in which the terminal device is located. For example, the terminal device recognizes that the environment in which the terminal device is located is a subway scene based on the collected audio information through the scene recognition model. In the subway scene, when the terminal device is in call mode, the terminal device can perform noise reduction processing on the collected sound and amplify the volume of the played audio.

[0003] Currently, the robustness of scene recognition models is relatively poor, resulting in different recognition accuracy rates of the same scene recognition model in different terminal devices. Summary of the invention

[0004] The present application provides a scene recognition method and related devices, which are applied to the field of terminal technology. The scene recognition model in the present application can extract features used for scene recognition in audio information, and identify the scene of the collected audio information based on the features used for scene recognition, which is beneficial to reducing the impact of features generated by different collection devices in the audio information on scene recognition, and further helps to improve the robustness of the scene recognition model.

[0005] In a first aspect, the present application proposes a scene recognition method, comprising: obtaining audio information collected by a microphone of a terminal device; inputting the audio information into a first model to obtain a first output result; identifying the scene represented by the first output result, the scene including scenes distinguished by noise in the audio and / or the number of sounds included in the audio; wherein the first model is obtained by training a first source model, a second source model, a third source model, a fourth source model and a fifth source model, wherein the inputs of the first source model and the second source model are both audio samples, the output of the first source model is the input of the third source model, and the output of the second source model is the input of the third source model. The input of the fourth source model, the output of the first source model and the output of the second source model are also the input of the fifth source model; the output results of the third source model and the fourth source model are both the device type for collecting the audio sample, and the third source model is used to output the wrong device type, and the fourth source model is used to output the correct device type; the fifth source model is used to superimpose and reconstruct the output of the first source model and the output of the second source model, and converge when the difference between the superimposed and reconstructed result and the audio sample is less than a first preset value; when the fourth source model and the fifth source model both converge, the training of the first source model is completed, and the first model is obtained.

[0006] The scene recognition method provided by the present application is that when the fourth source model converges, the fourth source model can constrain the second source model, which is conducive to the second source model to extract more accurate features for device type identification. When the fifth model converges and the second source model can extract more accurate features for device type identification, it is conducive to constraining the first source model to extract features for scene recognition to obtain the first model. During use, the first model can extract features for scene recognition in audio information, and identify scenes represented by features based on these features, which is conducive to identifying scenes on different terminal devices and improving the robustness of scene recognition.

[0007] In one possible implementation, the audio samples are represented by x i The first source model is represented by G1, and the output of the first source model is represented by G1(x i ), the third source model is represented by D1, and the input of the third source model is G1(x i ), the output of the third source model is (D1[G1(x i )]) j It means that (D1[G1(x i )]) j Used to represent the acquisition of audio samples x i The device is the predicted probability of the jth device, and the true value of the audio sample is y i,j Indicates that y i,jUsed to represent the acquisition of audio samples x i The device is the true probability of the jth device, j is an integer greater than or equal to 1, the loss function of the first source model is -L1, and the loss function of the third source model is L1, L1 and (D1[G1(x i )]) j and i,j In this way, the loss function of the first source model is negatively correlated with the loss function of the third source model, the output of the first source model is the input of the third source model, and the features generated by the first source model are used to confuse the third source model so that it cannot output the correct device category, which is conducive to training the first source model to extract features for scene recognition and training the third source model so that it cannot output the correct device category.

[0008] In one possible implementation, L1 satisfies the following formula In this way, the loss function of the third source model is a cross entropy loss function, which is conducive to improving the convergence speed of the third source model.

[0009] In one possible implementation, the audio samples are represented by x i The second source model is represented by G2, and the output of the second source model is represented by G2(x i ), the fourth source model is represented by D2, and the input of the fourth source model is G2(x i ), the output of the fourth source model is (D2[G2(x i )]) j It means that (D2[G2(x i )]) j Used to represent the acquisition of audio samples x i The device is the predicted probability of the jth device, and the true value of the audio sample is y i,j Indicates that y i,j Used to represent the acquisition of audio samples x i The device is the true probability of the jth device, j is an integer greater than or equal to 1, the loss function of the second source model and the loss function of the fourth source model are both L2, L2 and (D2[G2(x i )]) j and i,j In this way, the loss function of the second source model is the same as the loss function of the fourth source model, and the output of the second source model is the input of the fourth source model, which is conducive to training the second source model to extract features for device identification and training the fourth source model to output the correct device category.

[0010] In one possible implementation, L2 satisfies the following formula In this way, the loss function of the second source model and the loss function of the fourth source model are both cross entropy loss functions, which is conducive to improving the convergence speed of the second source model and the fourth source model.

[0011] In one possible implementation, the audio samples are represented by x i The first source model is represented by G1, and the output of the first source model is represented by G1(x i ), the second source model is represented by G2, and the output of the second source model is represented by G2(x i ) indicates that the fifth source model has a i ) and G2(x i ) to obtain the result x after superposition reconstruction i ′,x i ′ and x i The difference is x i ′-x i , the loss function of the fifth source model is L R , L R With x i ′-x i The square of x is positively correlated. i ′ and x i The smaller the difference is, the sum of the features extracted by G1 and G2 is x. i All the characteristics of x are beneficial to not lose i Features in .

[0012] In one possible implementation, L R Satisfies the following formula L R =||x i ′-x i || 2 .

[0013] In a possible implementation, before the audio information is input into the first model and the first output result is obtained, the method further includes: fixing the parameters of the first source model, the second source model, the fourth source model and the fifth source model, training the third source model, and obtaining the second model when the third source model converges; fixing the parameters of the second model, training the first source model, the second source model, the fourth source model and the fifth source model; wherein the loss function used for training the first source model, the second source model, the fourth source model and the fifth source model is L G , the loss function of the first source model is -L1, the loss functions of the second and fourth source models are both L2, and the loss function of the fifth source model is L R , L G Positively correlated with -L1, positively correlated with L2 and the product of the hyperparameters of the second source model, and positively correlated with L RAnd the product of the hyperparameters of the fifth source model is positively correlated. In this way, training the third source model first and then training the other models can improve the training speed compared to training all the models together. In addition, training one of them first and then training the other models together is conducive to improving the compatibility of other models and achieving the optimal state of other models. Optionally, the processing device can execute the step of training the model to obtain the trained first source model, i.e., the first model, and the first model can be transmitted to the terminal device. The terminal device can input the audio information into the first model to obtain the first output result. In this way, it is conducive to saving the computing power of the terminal device.

[0014] In one possible implementation, L G Satisfies the following formula: L G =-L1+λ1*L2+λ2*L R , where λ1 is the hyperparameter of the second source model, and λ2 is the hyperparameter of the fifth source model. In this way, multiple models are trained together, and 1, λ1, and λ2 represent the attention paid to the loss function of each model. The loss functions of each model are summed up. It is considered that when training together, the loss function is an accumulation process. Training each model through the summed loss function is conducive to training multiple models together to achieve the best overall effect.

[0015] In one possible implementation, the scene represented by the first output result is recognized by a third model, and the third model is trained in conjunction with the first model. During the training of the third model, some parameters in the first model and / or some parameters in the third model are adjusted until the third model converges and the output of the first model makes the accuracy of the scenes recognized by the third model in different devices meet the preset requirements. In this way, the third model is trained in conjunction with the first model, which is conducive to making the third model compatible with the first model and achieving the optimality at the same time. The output of the first model makes the accuracy of the scenes recognized by the third model in different devices meet the preset requirements, which is conducive to reducing the probability of not being able to accurately extract features for scene recognition due to training the third model.

[0016] In a possible implementation, in the process of training the third model, some parameters in the first model and some parameters in the third model are adjusted, and the loss function used for training the first model and the loss function used for training the third model are both L scene ; Audio information uses x i ″, the first model is represented by G1′, and the output of the first model is represented by G1′(x i ″) indicates that the third model is represented by D scene Indicates that the input of the third model is G1′(x i ″), the output of the third model is (D scene [G1′(x i″)]) k It means that (D scene [G1′(x i ″)]) k Used to indicate the collection of audio information x i The scene at the time of ″ is the predicted probability of the kth scene, and the true value of the audio information is z i,j Indicates that z i,k Used to indicate the collection of audio information x i ″ is the true probability of the scene in which the kth scene is located, k is an integer greater than or equal to 1, L scene and (D scene [G1′(x i ″)]) k and z i,k In this way, the loss function used to train the first model is the same as the loss function used to train the third model, which is conducive to optimizing the performance of the two models at the same time.

[0017] In one possible implementation, L scene Satisfies the following formula: In this way, the loss function of the first model and the loss function of the third model are both cross entropy loss functions, which is conducive to improving the convergence speed of the first model and the third model.

[0018] In a possible implementation, the method of obtaining the audio information collected by the microphone of the terminal device includes: obtaining the audio information in response to the operation of answering the phone; after identifying the scene represented by the first output result, the method further includes: if the scene is used to represent that the number of sounds included in the audio information is greater than a second preset value, amplifying the volume of the sound to be played by the terminal device, and playing the amplified sound. In this way, amplifying the sound to be played by the terminal device in a scene with a large number of sounds, and playing the amplified sound, is conducive to enabling the user to clearly hear the sound played by the terminal device.

[0019] In a second aspect, the present application provides a terminal device, including a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the method described in any possible implementation of the first aspect.

[0020] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method described in any possible implementation manner of the first aspect is implemented.

[0021] In a fourth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the method described in any possible implementation manner of the first aspect.

[0022] In a fifth aspect, the present application provides a chip or a chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0023] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0024] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0026] Figure 2 A software structure block diagram of a terminal device according to an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a model network architecture provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a model training method provided in an embodiment of the present application;

[0029] Figure 5 A schematic flow chart of a scene recognition method provided in an embodiment of the present application;

[0030] Figure 6 A schematic block diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to clearly describe the technical solution of the embodiment of the present application, the following description is first made:

[0032] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first model and the second model are only used to distinguish different models, and do not limit their order. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0033] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0034] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0035] It should be noted that the "at..." in the embodiment of the present application can be the instant when a certain situation occurs, or a period of time before or after a certain situation occurs, and the embodiment of the present application does not specifically limit this. In addition, the display interface provided in the embodiment of the present application is only an example, and the display interface can also include more or less content.

[0036] The terminal device can collect audio information of the environment in which the terminal device is located, input the audio information into the scene recognition model, obtain the scene in which the terminal device is located, and determine whether to process the played audio or the collected audio based on the scene in which the terminal device is located. For example, the terminal device recognizes that the environment in which the terminal device is located is a subway scene based on the collected audio information through the scene recognition model. In the subway scene, when the terminal device is in call mode, the terminal device can perform noise reduction processing on the collected sound and amplify the volume of the played audio.

[0037] Currently, the robustness of scene recognition models is relatively poor, resulting in different recognition accuracy rates of the same scene recognition model in different terminal devices.

[0038] The reason why the scene recognition model is less robust is that the audio information is greatly affected by the channel through which the audio information is collected. Different terminal devices have different microphone layouts and / or models, and the channels through which the audio information is collected are different, which will affect the audio features in the audio information collected by different terminal devices. This causes the scene recognition model to be unable to accurately extract the audio features in the audio information, which in turn causes different recognition accuracy rates of the scene recognition model in different terminal devices.

[0039] If the scene recognition model is trained separately for different terminal devices in order to improve the robustness of the scene recognition model, it is necessary to collect a large amount of data for each terminal device and train a scene recognition model suitable for this device based on these data. Due to the diversity of terminal devices, a large amount of manpower and time are required, resulting in high model training costs.

[0040] In view of this, the embodiments of the present application provide a scene recognition method and related devices, which can extract features used for scene recognition in audio information, identify the scene of the collected audio information based on the features used for scene recognition, reduce the impact of features in the audio information generated by different collection devices on scene recognition, and thus help improve the robustness of the scene recognition model.

[0041] The scene recognition method provided in the embodiment of the present application can be applied to any terminal device, for example, a mobile phone, a tablet, a folding screen, a watch or a headset. The training method of the scene recognition model involved in the embodiment of the present application can be performed by a processing device with any processing function, for example, a desktop computer or a server. The scene recognition model trained by the model training method of the scene recognition model provided in the embodiment of the present application can be deployed in any model of terminal device.

[0042] In order to better understand the embodiments of the present application, the hardware structure of the terminal device is first introduced.

[0043] Figure 1 FIG. 1 is a schematic diagram showing the hardware structure of a terminal device provided in an embodiment of the present application. Figure 1As shown, the terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0044] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal device. In other embodiments of the present application, the terminal device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0045] The terminal device can implement audio functions such as music playing, recording and calling through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, and an application processor. Among them, the audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The speaker 170A, also known as a "speaker", is used to convert audio electrical signals into sound signals. The terminal device can listen to music or listen to hands-free calls through the speaker 170A. The receiver 170B, also known as a "handset", is used to convert audio electrical signals into sound signals. When the terminal device answers a call or voice message, the voice can be answered by placing the receiver 170B close to the human ear. The microphone 170C, also known as a "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0046] Exemplarily, the terminal device can collect audio information through microphone 170C, and identify the audio information through the scene recognition model provided in the embodiment of the present application to obtain the scene in which the audio information is collected. If the scene is a subway scene and the terminal device is in call mode, the terminal device can amplify the volume of the audio data to be played, and play the amplified sound through the speaker 170A or the receiver 170B.

[0047] In addition, on top of the above components, the device also runs an operating system, such as an iOS operating system, an Android operating system, or a Windows operating system, etc. Applications can be installed and run on the operating system.

[0048] The software system of the terminal device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture, etc., which will not be elaborated here.

[0049] The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the terminal device.

[0050] Figure 2 : is a software structure block diagram of a terminal device in an embodiment of the present application. Figure 2 As shown, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the hardware abstraction layer (HAL), the kernel layer, and the hardware layer.

[0051] The application layer can include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the framework layer. Figure 2 As shown, the application package may include applications such as phone, music, and voice assistant. The phone application may trigger the microphone, such as the above Figure 1 The microphone 170 shown collects audio information and can trigger a speaker or a receiver, for example Figure 1 The speaker 170A or the receiver 170B shown plays the sound. The music application can trigger the speaker, for example Figure 1 The speaker 170A shown plays music. The voice assistant application can trigger the microphone, for example Figure 1 The microphone 170 shown collects audio information and can trigger a speaker or a receiver, for example Figure 1The speaker 170A or the receiver 170B shown plays sound.

[0052] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0053] like Figure 2 As shown, the application framework layer may include a window manager (window manager service) and a display manager (display manager service), etc. Among them, the window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, touch the screen, drag the screen, capture the screen, etc. The display manager is used to manage the life cycle of the display. It determines how to control its logical display according to the currently connected physical display device, and sends notifications to the system and applications when the state changes, etc. The application framework layer can transmit audio data triggered by the application of the application layer to the hardware abstraction layer. For example, when a user triggers a music application to play music, the application framework layer can obtain audio data from the music application of the application layer and transmit the audio data to the hardware abstraction layer.

[0054] The purpose of the HAL layer is to abstract the hardware and provide a unified interface for querying hardware devices for upper-layer applications, or to provide data storage services for upper-layer applications. Figure 2 As shown, the HAL layer may include a display driver module and a sensor hardware abstraction (sensor hidl). The display driver module is used to control the display driver to display pictures, interfaces, notification information, etc. on the display screen. The sensor hardware abstraction can control the sensor and monitor the event change notification of the sensor driver. The hardware abstraction layer can obtain audio data from the application framework layer and transmit the audio data to the kernel layer.

[0055] The kernel layer is the layer between hardware and software. Figure 2 As shown, the kernel layer may include one or more of the following: sensor driver and display driver, etc. The kernel layer may obtain audio data from the hardware abstraction layer and transmit the audio data to the kernel layer.

[0056] The hardware layer may include: microphone, digital signal processing (DSP) chip, display screen and other hardware. Among them, the DSP chip deploys a scene recognition model, which can obtain the audio information collected by the microphone, identify the scene of the terminal device based on the audio information, and determine whether to process the audio data obtained from the kernel layer based on the scene, and / or determine whether to process the audio information collected by the microphone.

[0057] In some embodiments, the layers implementing the same function may be called by other names, or the layers that can implement the functions of multiple layers may be regarded as one layer, or the layers that can implement the functions of multiple layers may be divided into multiple layers. This embodiment of the application does not limit this.

[0058] Combined with the above Figure 1 and Figure 2 This article introduces the hardware structure and software architecture of the terminal equipment. Figure 3 Introduce the model network architecture of the embodiment of the present application.

[0059] Figure 3 A schematic diagram of a model network architecture provided by an embodiment of the present application is shown. Figure 3 As shown in the figure, the model network architecture includes a device-independent feature generator G1, a device-related feature generator G2, a device discriminator D1, a device discriminator D2, and a decoder D e And the scene classifier D scene .

[0060] The input of the device-independent feature generator G1 is the audio information collected by multiple terminal devices. The device-independent feature generator G1 is used to extract device-independent features, wherein the device-independent features are used to represent features in the audio information for scene recognition. The features extracted by the device-independent feature generator G1 are used to confuse the device discriminator D1 and prevent it from correctly classifying the device. In one example, the loss function of the device-independent feature generator G1 can be a cross-entropy loss function. When the cross-entropy loss function is minimized, the training of the device-independent feature generator G1 ends.

[0061] The input of the device discriminator D1 is the output of the device-independent feature generator G1. The output of the device discriminator D1 can be multiple probabilities. These multiple probabilities correspond one-to-one to multiple terminal devices. Each probability in the multiple probabilities is used to represent the probability that the device that collects audio information is the device to which it corresponds. The device discriminator D1 is used to judge the features output by the device-independent feature generator G1 as the correct device for collecting audio information. Therefore, the training goal of the device discriminator D1 is to maximize the probability of identifying the correct device. In one example, the loss function of the device discriminator D1 can be a cross-entropy loss function. When the cross-entropy loss function is minimized, the training of the device discriminator D1 ends.

[0062] The input of the device-related feature generator G2 is the audio information collected by multiple terminal devices. The device-related feature generator G2 is used to extract device-related features, wherein the device-related features are used to represent the features in the audio information used for device identification. The features extracted by the device-related feature generator G2 are used to enable the device discriminator D2 to perform correct device classification. In an example, the loss function of the device-related feature generator G2 can be a cross-entropy loss function. When the cross-entropy loss function is minimized, the training of the device-related feature generator G2 ends.

[0063] The input of the device discriminator D2 is the output of the device-related feature generator G2. The output of the device discriminator D2 can be multiple probabilities, which correspond one-to-one to multiple terminal devices. Each probability in the multiple probabilities is used to represent the probability of identifying the device that collects audio information as the device to which it corresponds. The device discriminator D2 is used to judge the features output by the device-related feature generator G2 as the correct device for collecting audio information, so the training goal of the device discriminator D2 is to maximize the probability of identifying the correct device. In one example, the loss function of the device discriminator D2 can be a cross-entropy loss function. When the cross-entropy loss function is minimized, the training of the device discriminator D2 ends.

[0064] Decoder D e The input of D includes the output of the device-independent feature generator G1 and the output of the device-dependent feature generator G2. e Used to reconstruct the input information. e The training goal is to reduce the gap between the reconstructed information and the audio information. The audio information refers to the input of the device-independent feature generator G1, or the audio information refers to the input of the device-related feature generator G2. Since the features output by the device-related feature generator G2 include features for device identification, when the gap between the reconstructed information and the audio information is small, it is beneficial to ensure that the features output by the device-independent feature generator G1 retain more original audio information, that is, features used for scene identification, and it is beneficial to not lose key acoustic features.

[0065] Scene Classifier D scene The input is the output of the device-independent feature generator G1 and the scene classifier D scene The output of is the probability of identifying the correct scene of the audio information. Scene classifier D scene Used to identify the output of the device-independent feature generator G1 as the scene in which the audio information is collected.

[0066] The above-mentioned device-independent feature generator G1 and device discriminator D1 can be constructed as a GAN network, the device-related feature generator G2 and device discriminator D2 can be constructed as another GAN network, and the device-independent feature generator G1 and scene classifier D scene It can be constructed as a scene recognition model. The embodiment of the present application is based on a dual GAN ​​network with a decoder D e As a constraint condition, training the device-independent feature generator G1 in the scene recognition model is conducive to enabling the device-independent feature generator G1 to more accurately extract features in the audio information that are truly relevant to the essence of the scene, that is, features used for scene recognition, and reduce the impact of features generated by different acquisition devices on scene recognition, which is conducive to improving the robustness of the scene recognition model.

[0067] The scene recognition model includes a device-independent feature generator G1 and a scene classifier D scene In the embodiment of the present application, the device-independent feature generator G1 can be trained first to have features for extracting scene recognition features from audio information, and then the trained device-independent feature generator G1 and the scene classifier D scene Joint optimization, i.e. combined training, is performed to improve the compatibility of the two models so that both can achieve better results at the same time.

[0068] First, a method for training the device-independent feature generator G1 is introduced. The device-independent feature generator G1, the device-dependent feature generator G2, the device discriminator D1, the device discriminator D2 and the decoder D e When , the loss function can use the cross entropy loss function. If the cross entropy loss function is used, the loss function of the device-independent feature generator G1 can be expressed as -L1, which satisfies the formula: Among them, x i represents the i-th audio information among the audio information collected by multiple terminal devices, G1(x i ) represents the output of the device-independent feature generator G1, (D1[G1(x i )]) j represents the output of device discriminator D1, (D1[G1(x i )]) j It is used to indicate the predicted probability that the device collecting the i-th audio information is the j-th device, y i,j Represents the true value of the i-th audio information, y i,j Indicates the collection of the i-th audio information x i The device is the true probability of the jth device, where j is an integer greater than or equal to 1. The loss function of the device discriminator D1 and the loss function of the device-independent feature generator G1 are negative numbers of each other. The loss function of the device discriminator D1 can be expressed by L1, which satisfies the formula:

[0069] The loss function of the device-related feature generator G2 can be expressed by L2, which satisfies the following formula Among them, x i represents the i-th audio information among the audio information collected by multiple terminal devices, G2(x i ) represents the output of the device-related feature generator G2, (D2[G2(x i )]) j represents the output of device discriminator D2, (D2[G2(x i )]) j It is used to indicate the predicted probability that the device collecting the i-th audio information is the j-th device, y i,j Represents the true value of the i-th audio information, y i,j Indicates the collection of the i-th audio information x i The device is the true probability of the jth device, and j is an integer greater than or equal to 1. The loss function of the device discriminator D2 is the same as the loss function of the device-related feature generator G2, both of which are L2. The formula satisfied by L2 is as above and will not be repeated here.

[0070] Decoder D e The loss function can be expressed as L R Indicates that L R The formula L that satisfies R =||x i ′-x i || 2 , where x i represents the i-th audio information among the audio information collected by multiple terminal devices, x i ′ represents the decoder D e Output.

[0071] It should be noted that the formulas satisfied by the above loss functions are all explained using the i-th audio information as an example. The audio information collected by multiple terminal devices includes multiple such audio information. Based on the above formula, multiple loss function values ​​can be obtained. During the training process, the average value of these multiple loss functions can be used as the total loss function to train the model, or the sum of several of these multiple loss functions can be used as the total loss function to train the model.

[0072] For example, there are 10,000 audio information collected by 5 different types of terminal devices in the sample data. The 5 different types of terminal devices can be terminal device 1, terminal device 2, terminal device 3, terminal device 4 and terminal device 5. If the input of the device-independent feature generator G1 is the first audio information among the 10,000 audio information, and the audio information is collected by terminal device 1, then the true value corresponding to the audio information is y1,j Indicates that, y 1,1 =1,y 1,2 =0,y 1,3 =0,y 1,4 =0,y 1,5 = 0. If the output of device discriminator D1 is (D1[G1(x1)]) j It indicates that,

[0073] It is used to indicate the probability that the device collecting the audio information is terminal device 1. It is used to indicate the probability that the device collecting the audio information is terminal device 2, It is used to indicate the probability that the device collecting the audio information is terminal device 3, It is used to indicate the probability that the device collecting the audio information is the terminal device 4, It is used to indicate the probability that the device collecting the audio information is the terminal device 5. The cross entropy loss function of the device discriminator D1 can be expressed as By analogy, each of the 10,000 audio information corresponds to a loss function value, so there will be 10,000 loss function values. The processing device can use the average of the 10,000 loss function values ​​as the total loss function value. When the total loss function value is less than the preset value, the training ends. Alternatively, the sum of several loss functions in the 10,000 loss function values ​​can be used as the total loss function value. When the total loss function value is less than the preset value, the training ends.

[0074] For example, the processing device may take the sum of 5 loss functions out of 10,000 loss function values ​​as the total loss function value, and when the total loss function value is less than a preset value, the training ends. e Decoder D e The input of the decoder D is the output of the device-independent feature generator G1 and the output of the device-dependent feature generator G2. e The output is x i ′, the input of the device-independent feature generator G1 and the input of the device-dependent feature generator G2 can both be 10,000 audio information, then the decoder D e There are 10,000 outputs, and the processing device is based on the decoder D e The output and the true value of the audio information can be obtained by 10,000 L R , and the sum of 5 loss functions can be randomly selected as the total loss function value. The total loss function value can be expressed as Wherein, m is an integer from 1 to 5. When the total loss function value is less than the preset value, the decoder D e The training is over.

[0075] Based on the above Figure 3 In the model network architecture shown, the processing device may include multiple training methods to train the device-independent feature generator G1.

[0076] In a first possible implementation, the processing device may process the device-independent feature generator G1, the device-dependent feature generator G2, the device discriminator D1, the device discriminator D2, and the decoder D e Perform separate training to complete the training of the device-independent feature generator G1. In this way, each model is trained separately, which is conducive to achieving the optimal performance of each model.

[0077] In a second possible implementation, the processing device may first train the device discriminator D1 separately, and then train the device-independent feature generator G1, the device-dependent feature generator G2, the device discriminator D2, and the decoder D e Alternatively, the processing device may first train the device discriminator D2 separately, and then train the device-independent feature generator G1, the device-dependent feature generator G2, the device discriminator D1, and the decoder D e Train together to complete the training of the device-independent feature generator G1.

[0078] In this way, training the device discriminator D1 or the device discriminator D2 first and then training other models can improve the training speed compared to training all models together. In addition, training one of them first and then training other models together is conducive to improving the compatibility of other models and ensuring that other models can reach the optimal state.

[0079] In a third possible implementation, the processing device may first train the device discriminator D1 and the device discriminator D2 together, and then train the device-independent feature generator G1, the device-dependent feature generator G2, and the decoder D e The device-independent feature generator G1 is trained together to complete the training of the device-independent feature generator G1. In this way, the device-independent feature generator G1 is constrained by the device discriminator D1, and the device-dependent feature generator G2 is constrained by the device discriminator D2, which is conducive to faster training of the device-independent feature generator G1, the device-dependent feature generator G2 and the decoder D e Convergence is beneficial to improve training speed.

[0080] It should be noted that the embodiments of the present application are not limited to the use of these three possible implementation methods. Figure 3 The model network architecture shown and the manner in which the device-independent feature generator G1 can be trained are all within the protection scope of the embodiments of the present application.

[0081] In order to better understand the embodiments of the present application, the present application takes the above-mentioned second possible implementation method as an example to explain the training method in detail.

[0082] For example, Figure 4 A schematic diagram of a model training method provided in an embodiment of the present application is shown. The method can be applied to a processing device. Figure 4 As shown, the method comprises the following steps:

[0083] S401, fixed device-independent feature generator G1, device discriminator D2, device-dependent feature generator G2 and decoder D e The parameters of the device discriminator D1 are trained. When the loss function L1 is less than the threshold a, the trained device discriminator D1 is obtained.

[0084] The input of the device discriminator D1 is the output of the device-independent feature generator G1. The input of the device-independent feature generator G1 is audio information. The true value of the audio information can be the true probability that the device collecting the audio information is each device. The output of the device discriminator D1 can be the predicted probability that the device collecting the audio information is each device. The processing device inputs the true value of the audio information and the output of the device discriminator D1 into the formula satisfied by L1. If L1 is less than the threshold a, the trained device discriminator D1 is obtained. If L1 is greater than or equal to the threshold a, the processing device updates the parameters of the device discriminator D1 so that L1 is less than the threshold a. The threshold a is a preset smaller value, for example, 0.01, 0.05, etc.

[0085] S402, fix the parameters of the trained device discriminator D1, and perform the following operations on the device-independent feature generator G1, the device discriminator D2, the device-dependent feature generator G2, and the decoder D e For training, when the loss function L R When the value is less than the threshold value b, the trained device-independent feature generator G1, the trained device discriminator D2, the trained device-related feature generator G2 and the trained decoder D e Among them, L G Satisfy the formula: L G =-L1+λ1*L2+λ2*L R , λ1 is the hyperparameter of the device discriminator D2 and the device-related feature generator G2, λ2 is the decoder D e Hyperparameters of .

[0086] The input of the device-independent feature generator G1 is the same as the input of the device-dependent feature generator G2, both of which are audio information. The input of the device discriminator D1 is the output of the device-independent feature generator G1, and the input of the device discriminator D2 is the output of the device-dependent feature generator G2. eThe input of is the output of the device-independent feature generator G1 and the output of the device-dependent feature generator G2. The true value of the audio information can be the true probability that the device collecting the audio information is each device. The output of the device discriminator D1 and the output of the device discriminator D2 can both be the predicted probability that the device collecting the audio information is each device. The processing device can input the true value of the audio information and the output of the device discriminator D1 into the formula satisfied by -L1 to obtain -L1. The processing device can input the true value of the audio information and the output of the device discriminator D2 into the formula satisfied by L2 to obtain L2. The processing device can compare the audio information with the decoder D e Output input to L R The formula satisfied is L R , and -L1, L2 and L R Input to L G The formula satisfied is L G If L G When it is less than the threshold value b, the trained device-independent feature generator G1, the trained device discriminator D2, the trained device-related feature generator G2 and the trained decoder D e If L G If the value is greater than or equal to the threshold value b, the processing device updates the device-independent feature generator G1, the device discriminator D2, the device-related feature generator G2, and the decoder D e The parameters to make L R is less than the threshold b. λ1 is the hyperparameter of the device discriminator D2 and the device-related feature generator G2, which is used to balance L2, and λ2 is the decoder D e is a hyperparameter used to balance L R The device discriminator D2 and the device-related feature generator G2 have the same loss function and hyperparameters. When training together, the loss function and hyperparameters of one model can be used.

[0087] The model training method provided in the embodiment of the present application first trains the device discriminator D1, then trains the device-independent feature generator G1, the device discriminator D2, the device-related feature generator G2 and the decoder D e Compared with training all models together, the training speed can be improved. In addition, the device discriminator D1 is trained first, and then the device-independent feature generator G1, the device discriminator D2, the device-related feature generator G2 and the decoder D are trained together. e , which is beneficial to improve the device-independent feature generator G1, device discriminator D2, device-related feature generator G2 and decoder D e compatibility, realizing the device-independent feature generator G1, the device discriminator D2, the device-related feature generator G2 and the decoder D e Can reach the optimal state.

[0088] Optionally, when the processing device executes the above S401 and S402, it can record the loss function calculated by each model due to parameter update. After executing S402, the processing device can determine whether the change in the value of the loss function calculated by each model due to parameter update is within the preset fluctuation, or whether the value of the loss function calculated by each model due to parameter update is less than or equal to a preset threshold; if the change in the value of the loss function of one or more models is not within the preset fluctuation, or the value of the loss function of one or more models is greater than the preset threshold, it can be explained that the value of the parameter of this one or more models is inaccurate, and the hyperparameters corresponding to this one or more models can be increased or decreased, for example, increasing λ1 and decreasing λ2, and the above S401 and S402 can be re-executed based on the increased or decreased hyperparameters.

[0089] After the processing device executes the above S401 and S402, it can determine whether the number of executions of S401 and S402 is greater than the preset number. If the number of executions of S401 and S402 is greater than or equal to the preset number, the training ends; if the number of executions of S401 and S402 is less than the preset number, S401 and S402 are executed again until the number of executions of S401 and S402 is greater than or equal to the preset number.

[0090] The above describes the method of training the device-independent feature generator G1. The following describes the method of training the device-independent feature generator G1 and the scene classifier D. scene The scene classifier D scene It can be an existing classifier for identifying the scene in which the audio information is located.

[0091] Training scene classifier D scene When , the loss function can use the cross entropy loss function. If the cross entropy loss function is used, the scene classifier D scene The loss function can be expressed as L scene Indicates that L scene Satisfy the formula: Among them, x i ″ represents the i-th audio information among the audio information collected by multiple terminal devices, G1′ represents the trained device-independent feature generator G1, G1′(x i ″) represents the output of G1′, (D scene [G1′(x i ″)]) k Denotes the scene classifier D scene The output (D scene [G1′(x i ″)]) k Used to indicate the collection of the i-th audio information xi The scene at the time is the predicted probability of the kth scene, z i,j represents the true value of the i-th audio information, z i,j Indicates the collection of the i-th audio information x i The scene at that time is the true probability of the kth scene, where k is an integer greater than or equal to 1.

[0092] G1′ and D scene The samples trained together can be the same as the samples used to train the device-independent feature generator G1, that is, x i ″ and x i The same may also be different from the samples of the training device-independent feature generator G1, which is not limited in the present embodiment. i ″ and x i Similarly, it is beneficial to change the parameters in G1′ less during the training process, which is beneficial for G1′ to more accurately extract features for scene recognition.

[0093] In the embodiment of the present application, G1′ and D scene When trained together, they can be fine-tuned, i.e., only G1′ and / or D are changed during training. scene Several parameters of the last layer of the network are adjusted to make the two models compatible without changing the previous functions.

[0094] The present application provides a variety of G1' and D scene Ways to train together.

[0095] The first possible implementation method is to fix the parameters of G1′ and scene For training, when the loss function L scene When converged, we get the trained D scene In this way, only D is updated scene The parameters of G1′ are not updated, which is beneficial for G1′ to more accurately extract features for scene recognition.

[0096] The second possible implementation is to scene For training, when the loss function L scene When converged, we get the trained G1′ and trained D scene , and the trained G1′ and the trained D are compared by testing samples scene Test, if the accuracy of scene recognition for audio information collected by different devices meets the requirements, the training ends; if the accuracy of scene recognition for audio information collected by different devices does not meet the requirements, repeat the training. In this way, update the parameters of G1′ and D sceneThe parameters of G1′ are conducive to making the two models compatible and achieving the optimality at the same time, and are also conducive to reducing the probability of reducing the accuracy of extracting features for scene recognition due to updating the parameters of G1′.

[0097] The third possible implementation method is to fix D scene Parameters, train G1′, when the loss function L scene When convergence occurs, the trained G1′ is obtained, and the test samples are used to compare G1′ and D scene Test, if the accuracy of scene recognition for audio information collected by different devices meets the requirements, the training ends, if the accuracy of scene recognition for audio information collected by different devices does not meet the requirements, repeat the training. In this way, only the parameters of G1′ are updated, and D is not updated. scene The parameters of G1′ are conducive to making the two models compatible and achieving the optimality at the same time, and are also conducive to reducing the probability of reducing the accuracy of extracting features for scene recognition due to updating the parameters of G1′.

[0098] In the present embodiment, G1′ and D scene After the training is completed, the trained model can be called a scene recognition model. The scene recognition model includes a first model and a third model. The first model is the trained G1′, which is used to extract features for scene recognition in audio information, and the third model is the trained D scene , the scene in which the audio information is collected can be identified based on the features used for scene recognition. The scene recognition model can be deployed in the terminal device. When the terminal device is in a call state or a voice assistant state, the collected audio information is input into the scene recognition model. The scene recognition model outputs the scene in which the terminal device is located, and based on the scene in which the terminal device is located, determines whether to perform noise reduction processing on the collected audio information and whether to perform volume amplification processing on the played sound.

[0099] The following is an introduction to how to apply the model.

[0100] Figure 5 FIG. 1 is a schematic flow chart of a scene recognition method provided by an embodiment of the present application. The method can be applied to a terminal device. Figure 5 As shown, the method may include the following steps:

[0101] S501. Acquire audio information collected by a microphone of a terminal device.

[0102] The hardware structure of the terminal device can be as described above Figure 1 As shown in the above Figure 1 In the illustrated architecture, the microphone may be microphone 170C. The terminal device may obtain audio information collected by the microphone through the microphone.

[0103] S502: Input the audio information into the first model to obtain a first output result.

[0104] The first model is used to extract features from the audio information for subsequent scene recognition. The first output result is used to represent the output of the first model based on the audio information. The first model may be G1′ after training in the above embodiment.

[0105] S503: Identify a scene represented by the first output result, where the scene includes a scene distinguished by noise in the audio and / or the number of sounds included in the audio.

[0106] The scene may be a subway, a bus, a road, or a home, and these scenes may be distinguished by noise and / or the number of sounds. The noises in different scenes may be different, and these noises may also be understood as standard sound events in different scenes. For example, a standard sound event on the road may include a whistle, and a standard sound event on the subway may include a light rail sound. The embodiment of the present application does not limit the specific implementation method of identifying the scene represented by the first output result.

[0107] The first model in the embodiment of the present application can be obtained by training the first source model, the second source model, the third source model, the fourth source model and the fifth source model. The inputs of the first source model and the second source model are both audio samples, the output of the first source model is the input of the third source model, the output of the second source model is the input of the fourth source model, and the output of the first source model and the output of the second source model are also the input of the fifth source model; the output results of the third source model and the fourth source model are both the device type for collecting audio samples, and the third source model is used to output the wrong device type, and the fourth source model is used to output the correct device type; the fifth source model is used to superimpose and reconstruct the output of the first source model and the output of the second source model, and converges when the difference between the superimposed and reconstructed result and the audio sample is less than a first preset value; when the fourth source model and the fifth source model converge, the training of the first source model ends and the first model is obtained.

[0108] The audio sample may be x in the above embodiment. i The first source model may be the device-independent feature generator G1 in the above embodiment, the second source model may be the device-dependent feature generator G2, the third source model may be the device discriminator D1, the fourth source model may be the device discriminator D2, and the fifth source model may be the decoder D e The first model can be used to extract features for scene recognition. The specific architecture can be referred to above Figure 3 , I will not go into details here.

[0109] The scene recognition method provided in the embodiment of the present application is that when the fourth source model converges, the fourth source model can constrain the second source model, which is beneficial for the second source model to extract more accurate features for device type identification. When the fifth model converges and the second source model can extract more accurate features for device type identification, it is beneficial to constrain the first source model to extract features for scene recognition to obtain the first model. During use, the first model can extract features from audio information for scene recognition, and the scenes represented by the features are identified based on these features, which is beneficial for identifying the scenes on different terminal devices and improving the robustness of scene recognition.

[0110] Optionally, audio samples are represented by x i The first source model is represented by G1, and the output of the first source model is represented by G1(x i ), the third source model is represented by D1, and the input of the third source model is G1(x i ), the output of the third source model is (D1[G1(x i )]) j It means that (D1[G1(x i )]) j Used to represent the acquisition of audio samples x i The device is the predicted probability of the jth device, and the true value of the audio sample is y i,j Indicates that y i,j Used to represent the acquisition of audio samples x i The device is the true probability of the jth device, j is an integer greater than or equal to 1, the loss function of the first source model is -L1, and the loss function of the third source model is L1, L1 and (D1[G1(x i )]) j and i,j In this way, the loss function of the first source model is negatively correlated with the loss function of the third source model, the output of the first source model is the input of the third source model, and the features generated by the first source model are used to confuse the third source model so that it cannot output the correct device category, which is conducive to training the first source model to extract features for scene recognition and training the third source model so that it cannot output the correct device category.

[0111] Optionally, L1 satisfies the following formula In this way, the loss function of the third source model is a cross entropy loss function, which is conducive to improving the convergence speed of the third source model.

[0112] Optionally, audio samples are represented by x i The second source model is represented by G2, and the output of the second source model is represented by G2(x i ), the fourth source model is represented by D2, and the input of the fourth source model is G2(x i), the output of the fourth source model is (D2[G2(x i )]) j It means that (D2[G2(x i )]) j Used to represent the acquisition of audio samples x i The device is the predicted probability of the jth device, and the true value of the audio sample is y i,j Indicates that y i,j Used to represent the acquisition of audio samples x i The device is the true probability of the jth device, j is an integer greater than or equal to 1, the loss function of the second source model and the loss function of the fourth source model are both L2, L2 and (D2[G2(x i )]) j and i,j In this way, the loss function of the second source model is the same as the loss function of the fourth source model, and the output of the second source model is the input of the fourth source model, which is conducive to training the second source model to extract features for device identification and training the fourth source model to output the correct device category.

[0113] Optionally, L2 satisfies the following formula In this way, the loss function of the second source model and the loss function of the fourth source model are both cross entropy loss functions, which is conducive to improving the convergence speed of the second source model and the fourth source model.

[0114] Optionally, audio samples are represented by x i The first source model is represented by G1, and the output of the first source model is represented by G1(x i ), the second source model is represented by G2, and the output of the second source model is represented by G2(x i ) indicates that the fifth source model has a i ) and G2(x i ) to obtain the result x after superposition reconstruction i ′,x i ′ and x i The difference is x i ′-x i , the loss function of the fifth source model is L R , L R With x i ′-x i The square of x is positively correlated. i ′ and x i The smaller the difference is, the sum of the features extracted by G1 and G2 is x. i All the characteristics of x are beneficial to not lose i Features in .

[0115] Optionally, L RSatisfies the following formula L R =||x i ′-x i || 2 .

[0116] Optionally, before the audio information is input into the first model and the first output result is obtained, the method further includes: fixing the parameters of the first source model, the second source model, the fourth source model and the fifth source model, training the third source model, and obtaining the second model when the third source model converges; fixing the parameters of the second model, training the first source model, the second source model, the fourth source model and the fifth source model; wherein the loss function used for training the first source model, the second source model, the fourth source model and the fifth source model is L G , the loss function of the first source model is -L1, the loss functions of the second and fourth source models are both L2, and the loss function of the fifth source model is L R , L G Positively correlated with -L1, positively correlated with L2 and the product of the hyperparameters of the second source model, and positively correlated with L R And the product of the hyperparameters of the fifth source model is positively correlated. The detailed training process can be referred to above Figure 4 In this way, training the third source model first and then training other models can improve the training speed compared to training all models together. In addition, training one of them first and then training other models together is conducive to improving the compatibility of other models and achieving the optimal state of other models.

[0117] Optionally, L G Satisfies the following formula: L G =-L1+λ1*L2+λ2*L R , where λ1 is the hyperparameter of the second source model, and λ2 is the hyperparameter of the fifth source model. In this way, multiple models are trained together, and 1, λ1, and λ2 represent the attention paid to the loss function of each model. The loss functions of each model are summed up. It is considered that when training together, the loss function is an accumulation process. Training each model through the summed loss function is conducive to training multiple models together to achieve the best overall effect.

[0118] Optionally, the scene represented by the first output result is identified by a third model, and the third model is trained in conjunction with the first model. During the training of the third model, some parameters in the first model and / or some parameters in the third model are adjusted until the third model converges and the output of the first model makes the accuracy of the scenes identified by the third model in different devices meet the preset requirements. The third model is the D trained in the above embodiment. sceneIn this way, the third model is trained jointly with the first model, which is conducive to making the third model compatible with the first model and achieving the optimality at the same time. The output of the first model makes the accuracy of the scenes recognized by the third model in different devices meet the preset requirements, which is conducive to reducing the probability of not being able to accurately extract features for scene recognition due to training the third model.

[0119] Optionally, in the process of training the third model, some parameters in the first model and some parameters in the third model are adjusted, and the loss function used for training the first model and the loss function used for training the third model are both L scene ; Audio information uses x i ″, the first model is represented by G1′, and the output of the first model is represented by G1′(x i ″) indicates that the third model is represented by D scene Indicates that the input of the third model is G1′(x i ″), the output of the third model is (D scene [G1′(x i ″)]) k It means that (D scene [G1′(x i ″)]) k Used to indicate the collection of audio information x i The scene at the time of ″ is the predicted probability of the kth scene, and the true value of the audio information is z i,j Indicates that z i,k Used to indicate the collection of audio information x i ″ is the true probability of the scene in which the kth scene is located, k is an integer greater than or equal to 1, L scene and (D scene [G1′(x i ″)]) k and z i,k In this way, the loss function used to train the first model is the same as the loss function used to train the third model, which is conducive to optimizing the performance of the two models at the same time.

[0120] Optionally, L scene Satisfies the following formula: In this way, the loss function of the first model and the loss function of the third model are both cross entropy loss functions, which is conducive to improving the convergence speed of the first model and the third model.

[0121] Optionally, the above-mentioned acquisition of audio information collected by the microphone of the terminal device includes: in response to the operation of answering the phone, acquiring audio information; after identifying the scene represented by the first output result, the method also includes: if the scene is used to represent that the number of sounds included in the audio information is greater than a second preset value, then the volume of the sound to be played by the terminal device is amplified, and the sound with the amplified volume is played. The second preset value can be a positive integer value, such as 3 or 4. In this way, amplifying the sound to be played by the terminal device in a scene with a large number of sounds and playing the sound with the amplified volume is conducive to enabling the user to hear the sound played by the terminal device clearly. In one example, the terminal device can also perform noise reduction processing on the audio information, and transmit the audio information after noise reduction processing to the other end of the call to reduce the impact of other sounds on the audio information.

[0122] Optionally, the above-mentioned acquisition of audio information collected by the microphone of the terminal device includes: in response to the user's operation of invoking the voice assistant, acquiring the audio information; after identifying the scene represented by the first output result, the method also includes: if the scene is used to indicate that the noise in the audio information is greater than a third preset value, the audio information is subjected to noise reduction processing, and the volume of the sound to be played by the terminal device is amplified. The third preset value can be a threshold. Exceeding this threshold can indicate that there is more or greater noise in the audio information. In order to enable the terminal device to accurately analyze the acquired audio information, the audio information can be subjected to noise reduction processing. In order to enable the user to hear the sound clearly, the volume of the sound to be played by the terminal device can also be amplified.

[0123] The method of the embodiment of the present application has been described above, and the device for executing the above method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above method.

[0124] In order to achieve the above functions, the device for implementing the above scene recognition method includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the method steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0125] The embodiment of the present application can divide the functional modules of the device for implementing the scene recognition method according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0126] like Figure 6 A schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip 60 includes one or more (including two) processors 601 , a communication line 602 , a communication interface 603 and a memory 604 .

[0127] In some implementations, the memory 604 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0128] The method described in the above embodiment of the present application can be applied to the processor 601, or implemented by the processor 601. The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware or software instructions in the processor 601. The above processor 601 can be a general processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 601 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiment of the present application.

[0129] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 604, and the processor 601 reads the information in the memory 604 and completes the steps of the above method in combination with its hardware.

[0130] The processor 601 , the memory 604 , and the communication interface 603 may communicate with each other via the communication line 602 .

[0131] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0132] The present application embodiment also provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium may include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0133] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0134] As a possible design, the computer readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connection line may also be appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while optical discs reproduce data optically using lasers.

[0135] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. A scene recognition method, characterized in that: include: Get the audio information collected by the microphone of the terminal device; Inputting the audio information into a first model to obtain a first output result; Identifying a scene represented by the first output result, the scene comprising a scene distinguished by noise in the audio and / or the number of sounds included in the audio; Among them, the first model is obtained by training with a first source model, a second source model, a third source model, a fourth source model and a fifth source model, wherein the inputs of the first source model and the second source model are both audio samples, the output of the first source model is the input of the third source model, the output of the second source model is the input of the fourth source model, and the output of the first source model and the output of the second source model are also the inputs of the fifth source model; the output results of the third source model and the fourth source model are both the device type for collecting the audio sample, and the third source model is used to output the wrong device type, and the fourth source model is used to output the correct device type; the fifth source model is used to superimpose and reconstruct the output of the first source model and the output of the second source model, and converges when the difference between the superimposed and reconstructed result and the audio sample is less than a first preset value; when the fourth source model and the fifth source model both converge, the training of the first source model is completed to obtain the first model.

2. The method according to claim 1, characterized in that The audio samples are represented by x i The first source model is represented by G1, and the output of the first source model is represented by G1(x i ), the third source model is represented by D1, and the input of the third source model is G1(x i ), the output of the third source model is (D1[G1(x i )]) j It means that (D1[G1(x i )]) j Used to indicate the acquisition of the audio sample x i The device is the predicted probability of the jth device, and the true value of the audio sample is y i,j Indicates that the y i,j Used to indicate the acquisition of the audio sample x i The device is the true probability of the j-th device, j is an integer greater than or equal to 1, the loss function of the first source model is -L1, the loss function of the third source model is L1, and the L1 is the same as the (D1[G1(x i )]) j and the i,j Related.

3. The method according to claim 2, characterized in that The L1 satisfies the following formula 4. The method according to any one of claims 1 to 3, characterized in that The audio samples are represented by x i The second source model is represented by G2, and the output of the second source model is represented by G2(x i ), the fourth source model is represented by D2, and the input of the fourth source model is G2(x i ), the output of the fourth source model is (D2[G2(x i )]) j It means that (D2[G2(x i )]) j Used to indicate the acquisition of the audio sample x i The device is the predicted probability of the jth device, and the true value of the audio sample is y i,j Indicates that the y i,j Used to indicate the acquisition of the audio sample x i The device is the true probability of the j-th device, j is an integer greater than or equal to 1, the loss function of the second source model and the loss function of the fourth source model are both L2, and the L2 is i )]) j and the i,j Related.

5. The method according to claim 4, characterized in that The L2 satisfies the following formula 6. The method according to any one of claims 1 to 5, characterized in that The audio samples are represented by x i The first source model is represented by G1, and the output of the first source model is represented by G1(x i ), the second source model is represented by G2, and the output of the second source model is represented by G2(x i ) indicates that the fifth source model has a significant impact on the G1(x i ) and G2(x i ) to obtain the result x after superposition reconstruction i ′, the x i ′ and x i The difference is x i ′-x i , the loss function of the fifth source model is L R , the L R With the x i ′-x i The square of is positively correlated.

7. The method according to claim 6, characterized in that The L R Satisfies the following formula L R =||x i ′-x i || 2 .

8. The method according to any one of claims 1 to 7, characterized in that Before inputting the audio information into the first model to obtain the first output result, the method further includes: Fixing the parameters of the first source model, the second source model, the fourth source model and the fifth source model, training the third source model, and obtaining the second model when the third source model converges; The parameters of the second model are fixed, and the first source model, the second source model, the fourth source model and the fifth source model are trained; wherein the loss function used for training the first source model, the second source model, the fourth source model and the fifth source model is L G The loss function of the first source model is -L1, the loss functions of the second source model and the fourth source model are both L2, and the loss function of the fifth source model is L R , the L G is positively correlated with the -L1, is positively correlated with the product of the L2 and the hyperparameter of the second source model, and is positively correlated with the L R and the product of the hyperparameters of the fifth source model.

9. The method according to claim 8, characterized in that The L G Satisfies the following formula: L G =-L1+λ1*L2+λ2*L R , where λ1 is the hyperparameter of the second source model, and λ2 is the hyperparameter of the fifth source model.

10. The method according to any one of claims 1 to 9, characterized in that The scene represented by the first output result is recognized by a third model, and the third model is trained in conjunction with the first model. During the training of the third model, some parameters in the first model and / or some parameters in the third model are adjusted until the third model converges and the output of the first model makes the accuracy of the scenes recognized by the third model in different devices meet preset requirements.

11. The method according to claim 10, characterized in that In the process of training the third model, some parameters in the first model and some parameters in the third model are adjusted, and the loss function used in training the first model and the loss function used in training the third model are both L scene ; The audio information is expressed by x i ″, the first model is represented by G1′, and the output of the first model is represented by G1′(x i ″), the third model is represented by D scene Indicates that the input of the third model is G1′(x i ″), the output of the third model is (D scene [G1′(x i ″)]) k Indicates that (D scene [G1′(x i ″)]) k Used to indicate the collection of the audio information x i The scene at the time of ″ is the predicted probability of the kth scene, and the true value of the audio information is z i,j Indicates that zi,k Used to indicate the collection of the audio information x i ″ is the true probability of the kth scene, k is an integer greater than or equal to 1, and the L scene With the above (D scene [G1′(x i ″)]) k and the z i,k Related.

12. The method according to claim 11, characterized in that The L scene Satisfies the following formula:

13. The method according to any one of claims 1 to 9, characterized in that The step of obtaining audio information collected by a microphone of a terminal device includes: In response to an operation of answering the call, acquiring the audio information; After identifying the scene represented by the first output result, the method further includes: If the scenario is used to indicate that the number of sounds included in the audio information is greater than a second preset value, the volume of the sound to be played by the terminal device is amplified, and the sound with the amplified volume is played.

14. A terminal device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. A computer program product, characterized in that The invention comprises a computer program which, when being executed, causes a computer to execute the method according to any one of claims 1 to 13.

Images