A language recognition method, device, storage medium and equipment

Abstract

The application discloses a language recognition method and device, a storage medium and equipment. The method comprises the following steps: firstly, acquiring target speech and extracting acoustic features thereof; then, inputting the acoustic features of the target speech into a pre-constructed language recognition model to obtain a language recognition result of the target speech, wherein a backbone network in the language recognition model is obtained by jointly training speech data of multiple channels from different languages and a first target function; and an attention layer and a linear fully connected layer are obtained by jointly training speech data of different languages with balanced sampling and a second target function based on the backbone network. It can be seen that the language recognition model is constructed by using speech data of multiple channels from different languages and a target function, and then the acoustic features of the target speech are input into the recognition model, which not only ensures the recognition accuracy of speech of each language, but also avoids the influence of channel interference on the recognition result, thereby improving the language recognition effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, and in particular to a language recognition method and device, a storage medium and equipment. BACKGROUND

[0002] With the continuous breakthrough of artificial intelligence technology and the increasing popularity of various intelligent terminal devices, human-computer interaction appears more and more frequently in people's daily work and life. As the next generation of human-computer interaction, voice interaction can bring great convenience to people's life, and one of the most important things is the technology of language recognition based on voice.

[0003] The traditional language recognition method is a TV (Total variability) system. Although this recognition method has good language recognition effect when the long-time voice (such as voice with effective duration longer than 30s) and the language data channel source are single, it can reach the available level. However, in the actual application environment, the source of language data is often wide, and it may be necessary to process data from multiple channel sources at the same time. In addition, the distribution of each language data is often unbalanced, and in the extreme case, the proportion of some languages is very small. At this time, if the TV system is still used for language recognition, the recognition effect will be significantly reduced. Therefore, how to improve the language recognition effect under the condition of wide language data sources and unbalanced language distribution is a technical problem to be solved at present. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a language recognition method, device, storage medium and equipment, which can effectively improve the accuracy of the recognition result when performing language recognition.

[0005] The embodiments of the present application provide a language recognition method, comprising:

[0006] obtaining a target voice to be recognized;

[0007] extracting the acoustic features of the target voice;

[0008] inputting the acoustic features of the target voice into a pre-constructed language recognition model to obtain a language recognition result of the target voice;

[0009] The language recognition model comprises a backbone network, an attention layer and a linear fully connected layer; the backbone network is trained by using voice data from multiple channel sources of different languages and a first target function; and the attention layer and the linear fully connected layer are trained by using voice data sampled in a class-balanced manner of different languages and a second target function based on the backbone network.

[0010] In a possible implementation manner, the backbone network is determined in the following manner:

[0011] obtaining sample speeches of multi-channel sources of different languages;

[0012] extracting acoustic features of the sample speeches;

[0013] training a first initial language identification model by using acoustic features of the sample speeches carrying language labels and channel labels, the first target loss function, and language labels and channel labels corresponding to the sample speeches carrying the language labels and the channel labels, generating a first language identification model, and taking a backbone network in the first language identification model as a backbone network constituting the language identification model;

[0014] The first initial language identification model comprises an initial backbone network, a first linear fully connected layer, and a channel classification layer.

[0015] In a possible implementation, the first target function comprises a cross-entropy function, a domain loss function, and an optimized maximum mean difference loss function; the cross-entropy function is used for the first linear fully connected layer, and is used for distinguishing language types of each sample speech; the domain loss function is used for the channel classification layer, and is used for optimizing each sample speech in a direction in which channel information cannot be distinguished; and the optimized maximum mean difference loss function is used for the initial backbone network, and is used for making language features of different channel sample speeches extracted by the initial backbone network tend to be in the same distribution.

[0016] In a possible implementation, before the acoustic features of the sample speeches are extracted, the method further comprises:

[0017] filtering noise and / or silence in the sample speeches to obtain preprocessed sample speeches;

[0018] The acoustic features of the sample speeches are extracted by:

[0019] extracting acoustic features from the preprocessed sample speeches.

[0020] In a possible implementation, the attention layer and the linear fully connected layer are determined as follows:

[0021] obtaining sample speeches of different languages and class-balanced sampling;

[0022] extracting acoustic features of the sample speeches;

[0023] training a second initial language identification model by using acoustic features of the sample speeches, the second target loss function, and language labels corresponding to the sample speeches, and generating a language identification model comprising a backbone network, an attention layer, and a linear fully connected layer;

[0024] The second initial language recognition model comprises a backbone network, an attention layer, and a second linear fully connected layer in the first language recognition model.

[0025] In a possible implementation manner, the second target function comprises a cross-entropy function and a triple loss function; the cross-entropy function is used for distinguishing the language types of the sample voices; and the triple loss function is used for shortening the distances between the sample voices of the same language and lengthening the distances between the sample voices of different languages.

[0026] In a possible implementation manner, the acoustic feature of the target voice is a shift differential cepstrum feature.

[0027] In a possible implementation manner, the inputting of the acoustic feature of the target voice into the pre-constructed language recognition model to obtain a language recognition result of the target voice comprises:

[0028] inputting the acoustic feature of the target voice into a backbone network of the language recognition model to obtain a language representation vector of the target voice;

[0029] inputting the language representation vector of the target voice into an attention layer of the language recognition model for weighted processing to obtain a weighted language representation vector;

[0030] inputting the weighted language representation vector into a linear fully connected layer of the language recognition model to obtain a language classification probability distribution, and taking a language type corresponding to a maximum probability value in the language classification probability distribution as the language recognition result of the target voice.

[0031] Embodiments of the present application further provide a language recognition device, comprising:

[0032] a first obtaining unit configured to obtain a target voice to be recognized;

[0033] a first extracting unit configured to extract an acoustic feature of the target voice;

[0034] a recognition unit configured to input the acoustic feature of the target voice into a pre-constructed language recognition model to obtain a language recognition result of the target voice;

[0035] The language recognition model comprises a backbone network, an attention layer, and a linear fully connected layer; the backbone network is trained by using voice data of multiple channels of different languages and a first target function; and the attention layer and the linear fully connected layer are trained by using voice data of class-balanced sampling of different languages and a second target function based on the backbone network.

[0036] In a possible implementation manner, the device further comprises:

[0037] A second obtaining unit is configured to obtain sample speeches of multi-channel sources in different languages;

[0038] A second extracting unit is configured to extract acoustic features of the sample speeches;

[0039] A first training unit is configured to train a first initial language recognition model by using acoustic features of sample speeches carrying language labels and channel labels, the first target loss function, and language labels and channel labels corresponding to the sample speeches carrying the language labels and the channel labels, to generate a first language recognition model, and to use a backbone network in the first language recognition model as a backbone network of the language recognition model.

[0040] The first initial language recognition model includes an initial backbone network, a first linear fully connected layer, and a channel classification layer.

[0041] In a possible implementation, the first target function includes a cross-entropy function, a domain loss function, and an optimized maximum mean difference loss function; the cross-entropy function is used for the first linear fully connected layer, and is configured to distinguish language types of each sample speech; the domain loss function is used for the channel classification layer, and is configured to optimize each sample speech in a direction in which channel information cannot be distinguished; and the optimized maximum mean difference loss function is used for the initial backbone network, and is configured to make language features of different channel sample speeches extracted by the initial backbone network tend to be in the same distribution.

[0042] In a possible implementation, the apparatus further includes:

[0043] A filtering unit is configured to filter out noise and / or silence in the sample speeches, to obtain preprocessed sample speeches.

[0044] The second extracting unit is specifically configured to:

[0045] extract acoustic features from the preprocessed sample speeches.

[0046] In a possible implementation, the apparatus further includes:

[0047] A third obtaining unit is configured to obtain sample speeches of class-balanced sampling in different languages;

[0048] A third extracting unit is configured to extract acoustic features of the sample speeches;

[0049] a second training unit, configured to train a second initial language recognition model by using the acoustic features of the sample speech, the second target loss function, and the language labels corresponding to the sample speech, to generate a language recognition model comprising a backbone network, an attention layer, and a second linear fully connected layer;

[0050] The second initial language recognition model comprises the backbone network, the attention layer, and the second linear fully connected layer in the first language recognition model.

[0051] In a possible implementation, the second target function comprises a cross-entropy function and a triplet loss function; the cross-entropy function is used for distinguishing the language types of the sample speeches and is applied to the second linear fully connected layer; and the triplet loss function is used for narrowing the distances between the sample speeches of the same language and widening the distances between the sample speeches of different languages.

[0052] In a possible implementation, the acoustic features of the target speech are shift differential cepstrum features.

[0053] In a possible implementation, the recognition unit comprises:

[0054] a first input sub-unit, configured to input the acoustic features of the target speech into the backbone network of the language recognition model, to obtain a language representation vector of the target speech;

[0055] a second input sub-unit, configured to input the language representation vector of the target speech into the attention layer of the language recognition model for weighted processing, to obtain a weighted language representation vector;

[0056] a recognition sub-unit, configured to input the weighted language representation vector into the linear fully connected layer of the language recognition model, to obtain a language classification probability distribution, and take a language type corresponding to a maximum probability value in the language classification probability distribution as a language recognition result of the target speech.

[0057] Embodiments of the present application further provide a language recognition device, comprising: a processor, a memory, and a system bus;

[0058] The processor and the memory are connected through the system bus;

[0059] The memory is configured to store one or more programs, the one or more programs comprising instructions which, when executed by the processor, cause the processor to perform any one of the implementation manners of the above language recognition method.

[0060] The embodiment of the present application further provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions run on a terminal device, the terminal device executes any one of the above language recognition methods.

[0061] The embodiment of the present application further provides a computer program product, which, when running on a terminal device, causes the terminal device to execute any one of the above language recognition methods.

[0062] The embodiment of the present application provides a language recognition method, device, storage medium and equipment. First, a target voice to be recognized is acquired, and then acoustic features of the target voice are extracted. Next, the acoustic features of the target voice are input into a pre-constructed language recognition model to obtain a language recognition result of the target voice, wherein the language recognition model comprises a backbone network, an attention layer and a linear fully connected layer. The backbone network is obtained by jointly training a plurality of channel source voice data of different languages and a first target function. The attention layer and the linear fully connected layer are obtained by jointly training class-balanced sampled voice data of different languages and a second target function based on the backbone network. It can be seen that the language recognition model is constructed by using the plurality of channel source voice data of different languages and the first target function and the second target function, and then the acoustic features of the target voice are input into the recognition model, which not only ensures the recognition accuracy of the target voice of each language, but also avoids the influence of channel interference on the language recognition result, thereby improving the language recognition effect of the target voice of different languages from each channel under the condition that the language data sources are wide and the language distribution is unbalanced. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0064] Figure 1 It is a recognition flowchart of the existing TV language recognition system;

[0065] Figure 2 It is a flowchart of a language recognition method provided by the embodiment of the present application;

[0066] Figure 3 It is a structure diagram of a language recognition model provided by the embodiment of the present application;

[0067] Figure 4A flowchart for constructing a language recognition model is provided for an embodiment of the present application.

[0068] Figure 5 A composition diagram of a language recognition device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0069] Language recognition can also be referred to as language identification, which refers to a process of automatically determining a language category to which a speech segment belongs by a machine. The research on language recognition can be traced back to the 1970s. After decades of development, language recognition technology has shown great application value and development potential, and is widely used in international conferences, emergency rescue and other situations and tasks.

[0070] The commonly used language recognition technology is a TV system, as shown in Figure 1 The recognition principle is based on factor analysis. Each sentence of speech is first mapped into a low-dimensional vector in the factor space, and then the similarity between vectors is used to determine the language to which the speech belongs. From Figure 1 It can be seen that the TV language recognition system mainly includes two stages of model training and recognition, and the specific implementation process is as follows:

[0071] I. Model training stage:

[0072] 1. Preprocess all training data and extract shifted delta cepstral (SDC) features.

[0073] 2. Use the EM algorithm to iterate all SDC feature data obtained in step 1 to obtain a universal background model UBM and a factor orthogonal space T.

[0074] 3. Load the UBM and T obtained in step 2 to process all SDC feature data obtained in step 1. Map SDC feature data with different time lengths into equal-dimensional discriminant vectors i-vector.

[0075] 4. Input the i-vector obtained in step 3 according to the language category information labeled by the business expert, and train the language LDA linear transformation space.

[0076] 5. Use the LDA space trained in step 4 to project all discriminant vectors i-vector obtained in step 3 to obtain reduced-dimensional discriminant vectors i-vector.

[0077] 6. According to the labeled language category information, use the class center of the discriminant vector i-vector processed by step 5 as each language model, and each language model is a low-dimensional vector with the same dimension as the reduced i-vector.

[0078] II. Model identification phase:

[0079] (1) Preprocess the voice data to be identified, and extract SDC features.

[0080] (2) Load the UBM and T obtained in step 2 of the model training phase, and calculate the i-vector of the SDC feature to be identified obtained in step (1).

[0081] (3) Load the LDA space of step 4 of the model training phase, and project the i-vecotr of step (2) in the LDA space to obtain the reduced i-vector vector.

[0082] (4) Language score calculation: Calculate the cosine distance between the i-vector obtained in step (3) and each language model in step 6 of the model training phase, and determine the language category according to the size of the cosine distance.

[0083] It should be noted that from the above TV language recognition system, the training phase has two important steps, one is the training of UBM and T, and the other is the LDA training. Among them, the corresponding model is obtained after the training of UBM and T, which is used to map SDC features of different lengths to i-vectors of equal dimensions. It is used to deal with the language recognition scene where a single data contains only a single language, the same channel, the effective time is long, and the distribution of each language is balanced. Through practical test, the recognition effect is good.

[0084] However, in actual application environment, it often involves data from multiple channel sources and language distribution imbalance scenes, such as language data can come from telephone channel, instant messaging channel, satellite channel, etc. Due to the wide range of channel sources, some languages may account for a very small proportion. At this time, if TV system is still used for language recognition, different channel sources of data may lead to significant decline in recognition effect due to differences in coding method, acquisition equipment, recording distance and other factors. In addition, the imbalance of language data distribution will also lead to the bias of the trained model to large class data, and the recognition effect of small class data is poor.

[0085] Specifically, when the training data comes from multiple channels and the language distribution is imbalanced, the i-vector extracted by UBM and T will inevitably contain channel interference information and be biased towards large class data. Affected by channel interference and representation distribution, the language recognition effect will inevitably decline. The subsequent LDA training uses the original labeled i-vector to train the dimension reduction matrix, so that the distance between data of the same language is as small as possible, and the distance between data of different languages is as large as possible. Affected by channel interference and data distribution imbalance, the obtained dimension reduction matrix is also affected by channel interference and will shift to the distribution direction of the majority class data. The final recognition result is not accurate enough.

[0086] Therefore, the current language recognition method has poor language recognition effect under the condition that the source of language data is wide and the language distribution is unbalanced.

[0087] To solve the above-mentioned defects, the present application provides a language recognition method, which first acquires a target voice to be recognized, and then extracts the acoustic features of the target voice. Next, the acoustic features of the target voice can be input into a pre-constructed language recognition model to obtain a language recognition result of the target voice, wherein the language recognition model comprises a backbone network, an attention layer and a linear fully connected layer. The backbone network is obtained by jointly training the multi-channel source voice data of different languages and a first target function. The attention layer and the linear fully connected layer are obtained by jointly training the class-balanced sampled voice data of different languages and a second target function based on the backbone network. It can be seen that the embodiments of the present application first construct a language recognition model by using the multi-channel source voice data of different languages and the first target function and the second target function, and then input the acoustic features of the target voice into the recognition model, which not only ensures the recognition accuracy of the target voice of each language, but also avoids the influence of channel interference on the language recognition result, so as to improve the language recognition effect of the target voice of different languages from each channel under the condition that the source of language data is wide and the language distribution is unbalanced.

[0088] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0089] First embodiment

[0090] Referring to Figure 2 A flowchart of a language recognition method provided by the present embodiment is shown in the figure, which comprises the following steps:

[0091] S201: Acquire a target voice to be recognized.

[0092] In the present embodiment, any voice that needs to be recognized as a language is defined as a target voice. It should be noted that the present embodiment does not limit the language type of the target voice, for example, the target voice can be a Chinese voice or an English voice, etc. Meanwhile, the present embodiment also does not limit the length of the target voice, for example, the target voice can be a sentence or a paragraph, etc.

[0093] It can be understood that the target voice can be obtained by recording or the like according to actual needs, for example, the telephone call voice in people's daily life or the recording of instant messaging software can be used as the target voice, and the target voice is used to process the target voice by using the scheme provided in the embodiment to identify the language category to which the target voice belongs.

[0094] S202: Extracting the acoustic feature of the target voice.

[0095] In the embodiment, after obtaining the target voice to be identified by step S201, in order to accurately identify the language category to which the target voice belongs, the acoustic feature of the target voice needs to be extracted by using a feature extraction method, and the acoustic feature is used as a recognition basis to effectively identify the language category to which the target voice belongs by subsequent step S203.

[0096] Specifically, when extracting the acoustic feature of the target voice, first, the target voice needs to be frame-processed to obtain a corresponding voice frame sequence, and then the frame-processed voice frame sequence is pre-emphasized; and then the acoustic feature of each voice frame is extracted in sequence, wherein the acoustic feature refers to feature data for representing the acoustic information of the corresponding voice frame, such as SDC feature, Mel-scale Frequency Cepstral Coefficients (MFCC) feature or Perceptual Linear Predictive (PLP) feature.

[0097] It should be noted that the embodiment of the present application does not limit the extraction method of the acoustic feature of the target voice, nor the specific extraction process, and appropriate extraction method can be selected according to actual situation, and corresponding feature extraction operation is performed. For the convenience of understanding, the acoustic feature of the target voice will be taken as SDC feature in the subsequent embodiment.

[0098] S203: Inputting the acoustic feature of the target voice into a pre-constructed language recognition model to obtain a language recognition result of the target voice, wherein the language recognition model comprises a backbone network, an attention layer and a linear fully connected layer; the backbone network is obtained by jointly training the voice data of multiple channels of different languages and a first target function; the attention layer and the linear fully connected layer are obtained by jointly training the class-balanced sampled voice data of different languages and a second target function based on the backbone network.

[0099] In this embodiment, after obtaining the acoustic features of the target speech through step S202, in order to effectively improve the accuracy of the recognition result of the target speech, the acoustic features of the target speech can be further input into a pre-constructed language recognition model, so as to obtain the language recognition result of the target speech.

[0100] It should be noted that, in order to improve the accuracy of the speech recognition result, the language recognition model is pre-constructed, as shown in Figure 3 , which includes a backbone network (such as Resnet backbone) in Figure 3 , an attention layer, and a linear fully connected layer. The backbone network is trained by using the multi-channel source speech data of different languages and the first target function. The attention layer and the linear fully connected layer are trained based on the backbone network, by using the class-balanced sampled speech data of different languages and the second target function. In this way, by inputting the acoustic features of the target speech into the recognition model, the recognition accuracy of the target speech of each language can be ensured, and the influence of channel interference on the language recognition result can be avoided, so that the language recognition effect of the target speech of different languages from each channel can be improved under the condition that the source of language data is wide and the language distribution is unbalanced.

[0101] Specifically, one optional implementation manner is that the specific implementation process of "inputting the acoustic features of the target speech into the pre-constructed language recognition model to obtain the language recognition result of the target speech" in step S203 can include steps S2031-S2033:

[0102] Step S2031: inputting the acoustic features of the target speech into the backbone network of the language recognition model to obtain the language representation vector of the target speech.

[0103] In this implementation manner, after extracting the acoustic features (such as SDC features) of the target speech through step S202, as shown in Figure 3 , the SDC features of the target speech can be further input into the backbone network (Resnet backbone) of the language recognition model to obtain the language representation vector of the target speech, and the language representation vector can be made to remove channel interference as much as possible for performing the subsequent step S2032.

[0104] Step S2032: inputting the language representation vector of the target speech into the attention layer of the language recognition model for weighting processing to obtain the weighted language representation vector.

[0105] In this implementation manner, after inputting the acoustic features of the target speech into the backbone network of the language recognition model through step S2031 to obtain the language representation vector of the target speech, as shown in Figure 3As shown, the language type vector of the target voice can be further input into the attention layer of the language recognition model for weighting processing to achieve feature enhancement, so as to obtain a weighted language type vector, which is used to perform the subsequent step S2033.

[0106] Step S2033: inputting the weighted language type vector into the linear fully connected layer of the language recognition model to obtain a language classification probability distribution, and taking the language type corresponding to the maximum probability value as the language recognition result of the target voice.

[0107] In the present implementation, after obtaining the weighted language type vector through step S2032, as shown in Figure 3 the weighted language type vector can be further input into the linear fully connected layer of the language recognition model for classification recognition to output a classification probability distribution (i.e. a group of vectors) representing the language type to which the target voice belongs. The value of each dimension in the vector distribution can be a numerical value in the interval [0, 1], and the value of each dimension represents the probability value of the target voice belonging to each preset language classification type. At this time, the language classification type corresponding to the maximum probability value can be taken as the language type to which the target voice belongs.

[0108] For example, assuming that there are 7 preset language classification types, i.e. "Chinese, English, Korean, Japanese, German, Thai, and French", and the output vector of the model is s = [0.90, 0.01, 0.02, 0.02, 0.01, 0.01, 0.03], it can be seen that the value 0.90 of the first dimension is the highest, and therefore the language classification type corresponding to this dimension is the language classification type to which the target voice belongs, i.e. the language type to which the target voice belongs is "Chinese".

[0109] Next, the present embodiment will introduce the determination process of the network structure of the "backbone network", "attention layer" and "linear fully connected layer" in the language recognition model. As shown in Figure 4 the flowchart for constructing the language recognition model provided by the present embodiment, which includes a first training stage and a second training stage. The structure and parameter composition of the "backbone network" in the language recognition model can be trained through the first training stage, and the structure and parameter composition of the "attention layer" and "linear fully connected layer" in the language recognition model can be trained through the second training stage.

[0110] It should be noted that, in order to improve the recognition accuracy of the language recognition model, the basic model used by the present application when constructing the model is a deep residual network (Deep residual network, referred to as ResNet), and the training method adopts a decoupled training method, which is divided into two stages, respectively training the BackBone main skeleton part (i.e. the backbone network) of the ResNet network, and the attention layer and the linear fully connected layer.

[0111] The method decouples the training process into two stages: in the first stage of training, the speech of multiple channel sources carrying the full amount of preset language labels and without language labels is used as sample speech, and a first objective function composed of a cross-entropy function, a domain loss function and an optimized maximum mean difference loss function is used to train the network backbone part, so as to train a Resnet backbone capable of extracting a channel-robust language representation vector as the "backbone network" in the language recognition model. In the second stage of training, the network backbone part Resnet backbone obtained in the first stage is fixed, and the speech of different languages is obtained as sample speech in a class-balanced sampling manner, so that each batch of sample speech data is uniformly distributed in each language data to reduce the dominance of large class data on the classification layer. In the training process, an attention mechanism is used to further improve the classification ability of the language representation vector, and a second objective function composed of a cross-entropy function and a triple loss function is used to train the linear classification layer, so as to train the "attention layer" and "linear fully connected layer" in the language recognition model.

[0112] In the first stage of training, the determination process of the "backbone network" can include the following steps A1-A3:

[0113] Step A1: Obtain sample speech of multiple channel sources of different languages.

[0114] In this embodiment, in order to construct the language recognition model and determine the network structure of the "backbone network", a large amount of preparation work needs to be done in advance. First, a large amount of speech data of each language (such as Chinese, English, German, French, etc.) needs to be collected. For example, the speech can be picked up by a microphone array, and the pickup device can be a tablet computer, or a smart hardware device such as a smart sound box, a television and an air conditioner, etc. Typically, at least 1 hour of speech data needs to be collected for each language, and the channel source of the speech can not be required. These speech data can cover multiple application scenarios (such as vehicle-mounted, home, etc.). In addition, a large amount of speech data without language labels of multiple channel sources needs to be collected, typically 500 hours or more for each channel. Then each piece of collected speech data can be used as sample speech, and at the same time, the language classification corresponding to the sample speech is manually labeled in advance for subsequent step A2.

[0115] Step A2: Extract the acoustic features of the sample speech.

[0116] In this embodiment, in order to construct the language recognition model, the network structure of the "backbone network" is determined. After obtaining a large number of sample speeches of different languages from multiple channels in step A1, the sample speeches need to be further preprocessed to filter out invalid speech data such as noise and / or silence in the sample speeches, so as to obtain preprocessed sample speeches. Then, the target speech pair is replaced with the preprocessed sample speeches by using a method similar to that of extracting the acoustic features of the target speech in step S202, that is, the acoustic features (such as SDC features) of each preprocessed sample speech are extracted. For details, please refer to the introduction of step S202, which will not be repeated here, to perform the subsequent step A3.

[0117] Step A3: Select the acoustic features of the sample speech carrying the language label and the channel label, the first target loss function, and the language label and the channel label corresponding to the sample speech carrying the language label and the channel label to train the first initial language recognition model, generate the first language recognition model, and use the backbone network in the first language recognition model as the backbone network constituting the language recognition model.

[0118] It should be noted that in the training process of the first stage, the first initial language recognition model used in the present application includes an initial backbone network (Resnet backbone), a first linear fully connected layer, and a channel classification layer, as shown in Figure 4 The initial backbone network (Resnet backbone) is used to extract the language representation vector of the sample speech; the first linear fully connected layer is used to classify the sample speech by language; and the channel classification layer is used to classify the sample speech by channel category.

[0119] Specifically, in the embodiment, after the acoustic features of the sample speech are extracted by step A2, the acoustic features of the sample speech carrying the language label and the channel label can be further selected, and then the first initial language recognition model is trained by using the first target loss function composed of the cross-entropy function (CELoss), the domain loss function (DomainLoss) and the optimized maximum mean difference loss function (AMMDLoss), and the language label and the channel label corresponding to the sample speech carrying the language label and the channel label. During the training process, the model parameters of the recognition model can be continuously updated according to the change of the first target loss function value, until the first target loss function value meets the requirement, such as reaching the minimum value and having a small change (basically unchanged), or reaching the preset maximum iteration number (such as 100 times), then the updating of the model parameters is stopped, the training of the first language recognition model is completed, and the backbone network therein is taken as the backbone network constituting the language recognition model.

[0120] The cross-entropy function (CELoss) in the first target function acts on the first linear fully connected layer of the first initial language recognition model, and is used for distinguishing the language types of the sample speeches. Its gradient descent direction is towards the direction of correct classification of the language of the sample speeches, so that the features extracted by the Resnet BackBone can be towards the direction of correct classification of the language. It can be understood that the sample speech data for training the CELoss is required to be randomly sampled, so that the language representation ability is stronger and more suitable for specific actual use scenarios.

[0121] The domain loss function (DomainLoss) in the first target function acts on the channel classification layer of the first initial language recognition model, and is used for optimizing the directions in which the sample speeches cannot be distinguished from the channel information. It is responsible for classifying the channel sources of the sample speech data. Its gradient descent direction is towards the direction of correct classification of the channel of the sample speeches. In the training, the loss generated by the channel classification layer is reversed, so that the Resnet BackBone parameters can be towards the direction of incorrect classification of the channel, so as to form an opposition, and then the language representation vector finally extracted by the Resnet BackBone can be optimized towards the direction in which the channel information cannot be distinguished.

[0122] The optimized maximum mean difference loss function (AMMDLoss) in the first objective function acts on the initial backbone network of the first initial language identification model, which is an improvement of the maximum mean difference loss function (MMDLoss). The MMDLoss can only act on two different channel data, so that the feature distribution extracted by the network tends to be the same, while the AMMDLoss can act on more than two channel data, so that the distribution of the features of multiple channel data extracted by the network is optimized in the direction of tending to the same feature distribution. Therefore, in the training process of the present application, the AMMDLoss is used to make the language features of the sample speech of different channels extracted by the initial backbone network tend to the same distribution.

[0123] In this way, in the training process of the first stage, the first objective function composed of the CELoss, the DomainLoss and the AMMDLoss is jointly trained, which can effectively update the parameters of the network, so that the language representation extracted by the Resnet BackBone is optimized in the direction of resisting multi-channel interference. The specific calculation formula of the first objective function is as follows:

[0124]

[0125]

[0126]

[0127] In formula (1), Loss 1 represents the first objective function. represents the nth sample speech data with a language label (with or without a channel label) input, which belongs to the ith preset language category; represents the corresponding language label; represents the corresponding language label; represents the sample speech with a channel label and without a language label input represents the corresponding channel label; σ() represents the softmax function; g1() represents the mapping function of the first linear fully connected layer; f() represents the mapping function of the Resnet BackBone; N represents the size of a batch of input sample speech data; g2() represents the classification function of the channel classification layer. Since the DomainLoss corresponding to g2() is in the direction of making the channel category correct, in the joint training, the DomainLoss is negated, so that the features extracted by the Resnet BackBone are optimized in the direction of being unable to distinguish the channel information. In this way, the adversarial training is formed, so that when the final model converges, the language representation vector obtained can remove the channel information as much as possible, avoiding the influence of channel interference on the language identification result.

[0128] In the AMMDLoss, average batch l represents the mean of the language feature vector extracted by the Resnet BackBone after the input sample voice data of the lth channel with channel labels and without language labels, and the specific calculation formula is shown in the above formula (2). N l The total number of all sample voice data of the lth channel category with channel labels and without language labels is represented. The average batch all represents the mean of the S channel data feature distribution with channel labels and without language labels, and the specific calculation formula is shown in the above formula (3). It should be noted that the constraint of the AMMDLoss can make the features of different channel data extracted by the Resnet BackBone tend to the same distribution, thereby reducing the influence of channel interference on the language recognition result. In addition, α and β in the above formula (1) are both adjustment parameters set in advance, and the specific values can be set according to actual conditions and experience values, and the values of the embodiments of the present application are not limited. For example, the values of the two can be 0.01 and 1 respectively, so that the cross entropy function (CELoss), the domain loss function (DomainLoss) and the optimized maximum mean difference loss function (AMMDLoss) can all be in a reasonable range.

[0129] In this way, the "backbone network" of the language recognition model is trained by using sample voice data of different language multi-channel sources, and a reasonable de-channel related loss function is set, so that the language representation vector extracted by the backbone network can remove the channel interference as much as possible.

[0130] In a possible implementation manner of the embodiments of the present application, in the second stage training, the determination process of the "attention layer" and the "linear fully connected layer" of the language recognition model can include the following steps B1-B3:

[0131] Step B1: Obtain class-balanced sampled sample voice of different languages.

[0132] In the embodiment, in order to construct the language recognition model, the network structure of the "attention layer" and the "linear full connection layer" is determined, and a large amount of preparation work needs to be performed in advance. First, a large amount of voice data of various languages (such as Chinese, English, German, French, etc.) needs to be collected. For example, the voice can be picked up through a microphone array, and the pickup device can be a tablet computer or a smart hardware device such as a smart sound box, a television, and an air conditioner. Generally, at least 1 hour of voice data needs to be collected for each language, and the channel source of the voice can not be required. The voice data can cover various application scenarios (such as vehicle-mounted and home). Then, K1 language categories are randomly extracted from the various languages. Next, K2 pieces of voice data are randomly extracted from each category of the K1 language categories as sample voice, and the language categories corresponding to the sample voice are labeled by artificial labeling in advance, so as to perform the subsequent step B2. K1 and K2 are both preset positive integers, and the product of K1 and K2 is N2.

[0133] Step B2: Extracting the acoustic features of the sample voice.

[0134] In the embodiment, in order to construct the language recognition model, the network structure of the "attention layer" and the "linear full connection layer" is determined, and after a large amount of sample voice of balanced sampling of different languages is obtained through step A1, the sample voice needs to be further preprocessed to filter out invalid voice data such as noise and / or silence in the sample voice, to obtain preprocessed sample voice. Then, the target voice is replaced with the preprocessed sample voice by using a method similar to that of extracting the acoustic features of the target voice in step S202, that is, the acoustic features (such as SDC features) of each piece of preprocessed sample voice are extracted. For details, please refer to the introduction of step S202, which will not be repeated here, to perform the subsequent step B3.

[0135] Step B3: Training the second initial language recognition model by using the acoustic features of the sample voice, the second target loss function, and the language labels corresponding to the sample voice, to generate a language recognition model including the backbone network, the attention layer, and the linear full connection layer.

[0136] It should be noted that in the training process of the second stage, the second initial language recognition model used by the present application includes the backbone network (Resnetbackbone), the attention layer, and the second linear full connection layer that have been trained (i.e., no parameter update) in the first language recognition model, as shown in Figure 4The backbone network (Resnet backbone) is used to extract a channel-robust language representation vector of the sample voice; the attention layer is used to perform feature enhancement on the channel-robust language representation vector output by the Resnet BackBone, and input the enhanced language representation vector into the second linear fully connected layer for language classification; and the second linear fully connected layer is used to classify the sample voice into a language category.

[0137] The specific calculation formula of the attention layer for performing feature enhancement on the channel-robust language representation vector output by the Resnet BackBone through the attention mechanism is as follows:

[0138] a t = tanh(Wf(x) t ) (4)

[0139]

[0140]

[0141] wherein W represents an initialized weight, which can be set according to actual conditions and experience values, and the embodiments of the present application do not make any limitation; f(x) represents the channel-robust language representation vector output by the Resnet BackBone, and specifically is a concatenation of multiple vectors, i.e., f(x) = (f(x)1,…f(x) t ,…f(x) T ), T represents a specific channel number, and w t represents the weight of f(x) t , wherein the contribution of each single vector to language classification is different in size. If the vectors are simply concatenated into an embeding to perform language classification, the classification result is not accurate enough. Therefore, the vectors can be processed through the attention layer before classification and recognition, and the different f(x) t are weighted by w t , and then the weighted language representation vector f(x) final is input into the second linear fully connected layer for language classification, so as to improve the classification and recognition effect of the language.

[0142] Specifically, in the present embodiment, the acoustic features of the sample voice are extracted through step B2 and processed through the Resnet BackBone and the attention layer, and then the processed language representation vector f(x) finalThe second initial language recognition model is trained using a second objective loss function composed of cross-entropy function (CELoss) and triplet loss function, as well as language labels corresponding to the sample speech. During training, the model parameters can be continuously updated according to the changes in the value of the second objective loss function until the value of the second objective loss function meets the requirements, such as reaching the minimum value and the change is very small (basically unchanged), or reaching the preset maximum number of iterations (such as 100 times). At this point, the update of the model parameters is stopped, and the training of the language recognition model is completed, resulting in a language recognition model including a backbone network, attention layer, and linear fully connected layer.

[0143] In the second objective function, the cross-entropy function (CELoss) operates on the second fully connected layer of the second initial language recognition model to distinguish the language type of each sample speech. Its gradient descent direction is towards the correct language classification of the sample speech. The triplet loss function in the second objective function is used to narrow the distance between sample speech of the same language and to widen the distance between sample speech of different languages.

[0144] Thus, in the second stage of training, by jointly training the second objective function composed of CELows and TripletLoss, the network parameters can be effectively updated, making the language recognition results output by the linear fully connected layer more accurate. This avoids the dominance of large-class data in the training of the classification network and improves the recognition effect of small-class languages. The specific calculation formula of the second objective function is as follows:

[0145]

[0146]

[0147] In formula (7), Loss 2 represents the second objective function; This indicates that the nth input sample speech data belongs to the ith preset language category; express Corresponding language labels; σ() represents the softmax function; g3() represents the mapping function of the third linear fully connected layer; f() final N2 represents the mapping function that weights the language representation vectors output by the ResNet Backbone after passing through the attention layer; N2 represents the size of the batch of class-balanced sample speech data from different languages; d + () indicates that the current language representation vector f(x) is used. n ) finalThe average of the sum of distances of all positive example language representations of the same language as the anchor language after being anchored; d - () represents the current language representation vector f(x n ) final The average of the sum of distances of all negative example language representations of different languages after being anchored. The distance between two language representations is calculated as shown in formula (8) above. ε represents a margin parameter value for controlling the degree of dispersion between positive examples and negative examples. The specific value can be set according to actual conditions and experience values, and the present embodiment does not limit this. For example, ε can be set to 0.2.

[0148] In this way, by decoupling the overall training process into two stages, the problem of single-stage model training biasing towards large-class data or small-class overfitting is avoided. In the second stage of training, the attention mechanism is used, which also improves the language recognition effect. At the same time, using class-balanced sampling to obtain sample data for training also avoids the dominance of large-class data on the training of the classification network, thereby improving the recognition effect of small-class languages.

[0149] To sum up, the language recognition method provided in the embodiment first obtains a target voice to be recognized, and then extracts acoustic features of the target voice. Then, the acoustic features of the target voice can be input into a pre-constructed language recognition model to obtain a language recognition result of the target voice, wherein the language recognition model includes a backbone network, an attention layer, and a linear fully connected layer. The backbone network is trained by using multi-channel source voice data of different languages and a first target function. The attention layer and the linear fully connected layer are trained by using class-balanced sampled voice data of different languages and a second target function based on the backbone network. As can be seen, the language recognition model is constructed by using multi-channel source voice data of different languages and the first target function and the second target function, and then the acoustic features of the target voice are input into the recognition model, which not only ensures the recognition accuracy of each language target voice, but also avoids the influence of channel interference on the language recognition result, thereby improving the language recognition effect of different language target voices from each channel under the condition that the source of language data is wide and the language distribution is unbalanced.

[0150] Second embodiment

[0151] The embodiment will introduce a language recognition device. For related content, please refer to the above method embodiment.

[0152] Referring to Figure 5 A composition schematic diagram of a language recognition device provided in the embodiment is shown in FIG. 5. The device 500 includes:

[0153] The first obtaining unit 501 is configured to obtain a target voice to be recognized.

[0154] The first extracting unit 502 is configured to extract an acoustic feature of the target voice.

[0155] The recognition unit 503 is configured to input the acoustic feature of the target voice to a pre-constructed language recognition model to obtain a language recognition result of the target voice.

[0156] The language recognition model comprises a backbone network, an attention layer, and a linear fully connected layer; the backbone network is obtained by jointly training a plurality of channel source voice data of different languages and a first target function; and the attention layer and the linear fully connected layer are obtained by jointly training class-balanced sampled voice data of different languages and a second target function based on the backbone network.

[0157] In an implementation form of the embodiment, the device further comprises:

[0158] The second obtaining unit is configured to obtain sample voice of a plurality of channel sources of different languages.

[0159] The second extracting unit is configured to extract an acoustic feature of the sample voice.

[0160] The first training unit is configured to train a first initial language recognition model by selecting the acoustic feature of the sample voice carrying a language label and a channel label, the first target loss function, and the language label and the channel label corresponding to the sample voice carrying the language label and the channel label, to generate a first language recognition model, and to use a backbone network in the first language recognition model as the backbone network constituting the language recognition model.

[0161] The first initial language recognition model comprises an initial backbone network, a first linear fully connected layer, and a channel classification layer.

[0162] In an implementation form of the embodiment, the first target function comprises a cross-entropy function, a domain loss function, and an optimized maximum mean difference loss function; the cross-entropy function is applied to the first linear fully connected layer and is used to distinguish the language types of the sample voices; the domain loss function is applied to the channel classification layer and is used to optimize the sample voices in a direction in which the channel information cannot be distinguished; and the optimized maximum mean difference loss function is applied to the initial backbone network and is used to make the language features of the sample voices of different channels extracted by the initial backbone network tend to be in the same distribution.

[0163] In an implementation form of the embodiment, the device further comprises:

[0164] a filtering unit, configured to filter noise and / or silence in the sample voice to obtain preprocessed sample voice;

[0165] The second extraction unit is specifically configured to:

[0166] extract acoustic features from the preprocessed sample voice.

[0167] In an implementation form of the apparatus, the apparatus further includes:

[0168] a third acquisition unit, configured to acquire sample voice of class-balanced sampling of different languages;

[0169] a third extraction unit, configured to extract acoustic features of the sample voice.

[0170] a second training unit, configured to train a second initial language recognition model by using the acoustic features of the sample voice, the second target loss function, and a language label corresponding to the sample voice, to generate a language recognition model including a backbone network, an attention layer, and a linear fully connected layer.

[0171] The second initial language recognition model includes the backbone network, the attention layer, and a second linear fully connected layer in the first language recognition model.

[0172] In an implementation form of the apparatus, the second target function includes a cross-entropy function and a triplet loss function; the cross-entropy function is applied to the second linear fully connected layer, and is configured to distinguish language types of the sample voice; and the triplet loss function is configured to narrow distances of sample voice of the same language and widen distances of sample voice of different languages.

[0173] In an implementation form of the apparatus, the acoustic features of the target voice are shift differential cepstrum features.

[0174] In an implementation form of the apparatus, the recognition unit 503 includes:

[0175] a first input subunit, configured to input the acoustic features of the target voice into the backbone network of the language recognition model to obtain a language representation vector of the target voice;

[0176] a second input subunit, configured to input the language representation vector of the target voice into the attention layer of the language recognition model for weighted processing to obtain a weighted language representation vector;

[0177] a recognition subunit, configured to input the weighted language representation vector into the linear fully connected layer of the language recognition model to obtain a language classification probability distribution, and take a language type corresponding to a maximum probability value in the language classification probability distribution as a language recognition result of the target voice.

[0178] Further, the embodiment of the present application further provides a language recognition device, comprising: a processor, a memory, a system bus;

[0179] The processor and the memory are connected through the system bus;

[0180] The memory is used for storing one or more programs, the one or more programs comprising instructions, which, when executed by the processor, cause the processor to perform any of the implementation methods of the above language recognition method.

[0181] Further, the embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores instructions, when the instructions run on a terminal device, cause the terminal device to perform any of the implementation methods of the above language recognition method.

[0182] Further, the embodiment of the present application further provides a computer program product, when the computer program product runs on a terminal device, causes the terminal device to perform any of the implementation methods of the above language recognition method.

[0183] From the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software and necessary universal hardware platforms. Based on such understanding, the technical solutions of the present application can be embodied in the form of software products, which can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) execute the methods described in the various embodiments or some parts of the embodiments of the present application.

[0184] It should be noted that the embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0185] It is also to be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" can include a combination of two or more components, and the term "an element" can include comparable reference to a plurality of elements. Additionally, the term "or" as used herein means any one member of a logical disjunction (i.e., it is equivalent to "or" and "or else") and not a logical exclusion. Also, the terms "comprise," "comprising," "include," "including," and the like mean "including but not limited to." Furthermore, the description herein uses terms such as first and second to denote different elements, but the applicants do not intend these terms to denote an ordering, because some of these elements can, in fact, precede other ones in some of the implementations.

[0186] The above description of disclosed embodiments provides enabling concepts for practicing or using the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A language identification method characterized by, The method comprises the following steps: acquiring a target voice to be recognized; extracting acoustic features of the target voice; inputting the acoustic features of the target voice into a pre-constructed language recognition model to obtain a language recognition result of the target voice; wherein the language recognition model comprises a backbone network, an attention layer and a linear fully connected layer; the backbone network is obtained by jointly training a plurality of channel source voice data of different languages and a first target function; the attention layer and the linear fully connected layer are obtained by jointly training class-balanced sampled voice data of different languages and a second target function based on the backbone network; the determination method of the backbone network is as follows: acquiring sample voices of different languages and multiple channels; extracting acoustic features of the sample voices; selecting the acoustic features of the sample voices carrying language labels and channel labels, a first target loss function, and the language labels and channel labels corresponding to the sample voices to train a first initial language recognition model to generate a first language recognition model, and taking the backbone network in the first language recognition model as the backbone network constituting the language recognition model; wherein the first initial language recognition model comprises an initial backbone network, a first linear fully connected layer and a channel classification layer; the first target function comprises a cross-entropy function, a domain loss function and an optimized maximum mean difference loss function; the cross-entropy function is applied to the first linear fully connected layer to distinguish the language types of the sample voices; the domain loss function is applied to the channel classification layer to optimize the sample voices in a direction in which the channel information cannot be distinguished; and the optimized maximum mean difference loss function is applied to the initial backbone network to make the language features of the sample voices of different channels extracted by the initial backbone network tend to be in the same distribution.

2. The method of claim 1, wherein, Before the step of extracting the acoustic features of the sample voices, the method further comprises: filtering out noise and / or silence in the sample voices to obtain pre-processed sample voices; the step of extracting the acoustic features of the sample voices comprises: extracting acoustic features from the pre-processed sample voices.

3. The method according to any of claims 1-2, characterized in that, The determination method of the attention layer and the linear fully connected layer is as follows: acquiring class-balanced sampled sample voices of different languages; extracting acoustic features of the sample voices; training a second initial language recognition model by using the acoustic features of the sample voices, the second target function and the language labels corresponding to the sample voices to generate a language recognition model comprising a backbone network, an attention layer and a linear fully connected layer; wherein the second initial language recognition model comprises the backbone network, the attention layer and a second linear fully connected layer in the first language recognition model.

4. The method of claim 3, wherein, The second target function comprises a cross-entropy function and a triplet loss function; the cross-entropy function is applied to the second linear fully connected layer to distinguish the language types of the sample voices; and the triplet loss function is used to shorten the distance between sample voices of the same language and lengthen the distance between sample voices of different languages.

5. The method of claim 1, wherein, The acoustic features of the target voice are shift differential cepstrum features.

6. The method of claim 1, wherein, The method comprises the following steps: inputting the acoustic features of the target speech into a backbone network of a pre-constructed language recognition model to obtain language representation vectors of the target speech; inputting the language representation vectors of the target speech into an attention layer of the language recognition model for weighted processing to obtain weighted language representation vectors; inputting the weighted language representation vectors into a linear fully connected layer of the language recognition model to obtain a language classification probability distribution, and taking a language type corresponding to a maximum probability value in the language classification probability distribution as a language recognition result of the target speech.

7. A language identification apparatus characterized by comprising: The method comprises the following steps: a first obtaining unit is configured to obtain a target speech to be recognized; a first extracting unit is configured to extract acoustic features of the target speech; a recognition unit is configured to input the acoustic features of the target speech into a pre-constructed language recognition model to obtain a language recognition result of the target speech. The language recognition model comprises a backbone network, an attention layer and a linear fully connected layer; the backbone network is obtained by jointly training multi-channel source speech data of different languages and a first target function; the attention layer and the linear fully connected layer are obtained by jointly training class-balanced sampled speech data of different languages and a second target function based on the backbone network; The apparatus further comprises: a second obtaining unit is configured to obtain sample speech of different languages and from different channels; a second extracting unit is configured to extract acoustic features of the sample speech; a first training unit is configured to train a first initial language recognition model by selecting acoustic features of sample speech carrying language labels and channel labels, a first target loss function, and language labels and channel labels corresponding to the sample speech, to generate a first language recognition model, and use a backbone network in the first language recognition model as a backbone network of a language recognition model; The first initial language recognition model comprises an initial backbone network, a first linear fully connected layer and a channel classification layer; the first target function comprises a cross-entropy function, a domain loss function and an optimized maximum mean difference loss function; the cross-entropy function is applied to the first linear fully connected layer to distinguish language types of each sample speech; the domain loss function is applied to the channel classification layer to optimize each sample speech in a direction in which channel information cannot be distinguished; and the optimized maximum mean difference loss function is applied to the initial backbone network to make language features of different channel sample speech extracted by the initial backbone network tend to be in the same distribution.

8. A language identification device, characterized by The apparatus comprises: a processor, a memory and a system bus; the processor and the memory are connected through the system bus; the memory is configured to store one or more programs, the one or more programs comprising instructions which, when executed by the processor, cause the processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, when the instructions run on a terminal device, cause the terminal device to perform the method of any one of claims 1-6.

Images