Intelligent device control method and apparatus, electronic device, and storage medium

Abstract

The application provides a kind of intelligent device control method, device, electronic equipment and storage medium, the method includes: obtaining to be identified voice;To be identified voice includes the voice for controlling intelligent device;According to to be identified voice, determine input information, input information is input into pinyin sequence prediction model, obtain the pinyin sequence of to be identified voice that pinyin sequence prediction model output is predicted;According to pinyin sequence, obtain the target character sequence of to be identified voice under target language type;According to target character sequence, control intelligent device;Wherein, pinyin sequence prediction model is based on sample voice and sample voice sample label, and is obtained by training to multiple task models;Sample label includes pinyin sequence label, tone sequence label, tone label and vocabulary label.The application realizes through voice interaction to realize intelligent device control, avoid direct contact intelligent device button, can effectively inhibit the spread of bacteria, guarantee user health.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and in particular to an intelligent device control method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, smart devices are widely used in daily life, such as elevators, which play an extremely important role not only in carrying people but also in carrying goods.

[0003] In existing technology, passengers control elevators by clicking elevator buttons. However, in elevators, which are used very frequently and by a diverse and densely populated environment, multiple passengers directly touch the elevator buttons to control them, resulting in a high concentration of bacteria on the buttons and seriously affecting passenger health.

[0004] Therefore, how to provide a smart device control technology that avoids direct contact with smart device buttons and enables efficient and safe control of smart devices is an important issue that the industry urgently needs to address. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and storage medium for controlling smart devices, in order to solve the problem that in the prior art, passengers directly touch the buttons of smart devices to control them, which seriously affects the health of passengers.

[0006] This invention provides a method for controlling intelligent devices, comprising:

[0007] Acquire the speech to be recognized; the speech to be recognized includes voice used to control smart devices;

[0008] Based on the speech to be recognized, input information is determined, and the input information is input into the pinyin sequence prediction model to obtain the pinyin sequence of the speech to be recognized output by the pinyin sequence prediction model.

[0009] Based on the pinyin sequence, obtain the target text sequence of the speech to be recognized under the target language type;

[0010] The intelligent device is controlled according to the target text sequence; wherein, the pinyin sequence prediction model is obtained by training a multi-task model based on sample speech and sample labels of the sample speech; the sample labels include pinyin sequence labels, tone sequence labels, chord labels and vocabulary labels.

[0011] According to a smart device control method provided by the present invention, the multi-task model includes a shared feature extraction module and a prediction module, wherein the prediction module includes a pinyin sequence prediction branch module, a tone sequence prediction branch module and a syllable prediction branch module;

[0012] The pinyin sequence prediction model was trained based on the following steps:

[0013] Based on the sample speech, the sample input information is determined, and the sample input information is input into the shared feature extraction module to obtain the sample features of the sample speech output by the shared feature extraction module;

[0014] The sample features are input into the prediction module to obtain the sample pinyin sequence of the sample speech output by the pinyin sequence prediction branch module, the sample tone sequence of the sample speech output by the tone sequence prediction branch module, and the sample chord prediction result and sample word prediction result of the sample speech output by the chord word prediction branch module.

[0015] The parameters of the multi-task model are iteratively trained based on the pinyin sequence labels, the tone sequence labels, the chord labels, the vocabulary labels, the sample pinyin sequence, the sample tone sequence, the sample chord prediction results, and the sample vocabulary prediction results.

[0016] The pinyin sequence prediction model is constructed based on the shared feature extraction module and the pinyin sequence prediction branch module in the trained multi-task model.

[0017] According to a smart device control method provided by the present invention, the step of iteratively training the parameters of the multi-task model based on the pinyin sequence label, the tone sequence label, the chord label, the vocabulary label, and the sample pinyin sequence, the sample tone sequence, the sample chord prediction result, and the sample vocabulary prediction result includes:

[0018] Based on the pinyin sequence label and the sample pinyin sequence, obtain the first loss function;

[0019] Based on the tone sequence labels and the sample tone sequences, a second loss function is obtained;

[0020] A third loss function is obtained based on the phonology tags, the vocabulary tags, the sample phonology prediction results, and the sample vocabulary prediction results;

[0021] The parameters of the multi-task model are iteratively trained based on the first loss function, the second loss function, and the third loss function.

[0022] According to a smart device control method provided by the present invention, controlling the smart device based on the target text sequence includes:

[0023] If it is determined that the target text sequence contains command text, keywords are extracted from the target text sequence to obtain a keyword set;

[0024] Perform a validity verification on the keyword set;

[0025] If the keyword set passes the legality verification, a smart device control command is generated based on the keyword set.

[0026] The intelligent device is controlled according to the intelligent device control instructions.

[0027] According to a smart device control method provided by the present invention, the method further includes:

[0028] Based on the set of keywords, determine the response information corresponding to the speech to be recognized;

[0029] The response information is then broadcast according to the language type corresponding to the speech to be recognized.

[0030] According to a smart device control method provided by the present invention, the step of obtaining the target text sequence of the speech to be recognized under the target language type based on the pinyin sequence includes:

[0031] Based on the first dictionary database, generate a candidate set of text sequences corresponding to the pinyin sequence;

[0032] Based on the multilingual model, the score of each candidate text sequence in the candidate text sequence set is calculated;

[0033] The target text sequence is obtained based on the candidate text sequence corresponding to the highest score.

[0034] According to a smart device control method provided by the present invention, obtaining the target text sequence based on the candidate text sequence corresponding to the highest score includes:

[0035] The candidate text sequence corresponding to the highest score is segmented into multiple words;

[0036] Based on the second dictionary database, the language types corresponding to the multiple words are converted to the target language type;

[0037] Based on the conversion result, the target text sequence is obtained.

[0038] According to a smart device control method provided by the present invention, determining input information based on the speech to be recognized includes:

[0039] The speech to be recognized is preprocessed, including noise reduction and / or feature extraction of Mel frequency cepstral coefficients;

[0040] The input information is determined based on the preprocessed speech to be recognized.

[0041] The present invention also provides a smart device control device, comprising:

[0042] A voice acquisition unit is used to acquire the voice to be recognized; the voice to be recognized includes voice used to control smart devices;

[0043] The first recognition unit is used to determine input information based on the speech to be recognized, input the input information into the pinyin sequence prediction model, and obtain the pinyin sequence of the speech to be recognized output by the pinyin sequence prediction model.

[0044] The second recognition unit is used to obtain the target text sequence of the speech to be recognized under the target language type based on the pinyin sequence;

[0045] A device control unit is used to control the smart device according to the target text sequence;

[0046] The pinyin sequence prediction model is obtained by training a multi-task model based on sample speech and sample labels of the sample speech; the sample labels include pinyin sequence labels, tone sequence labels, chord labels and vocabulary labels.

[0047] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the intelligent device control method described above.

[0048] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent device control method as described above.

[0049] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the intelligent device control method as described above.

[0050] The intelligent device control method, apparatus, electronic device, and storage medium provided by this invention predict the pinyin sequence of the speech to be recognized for controlling the intelligent device, and convert the pinyin sequence with tone into a target text sequence under a common language type, so as to automatically, efficiently, and safely control the intelligent device according to the target text sequence, avoid direct contact with the intelligent device buttons, effectively inhibit the spread of bacteria, and protect user health. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is one of the flowcharts illustrating the intelligent device control method provided by the present invention;

[0053] Figure 2 This is the second flowchart illustrating the intelligent device control method provided by the present invention;

[0054] Figure 3 This is a schematic diagram of the multi-task model in the intelligent device control method provided by the present invention;

[0055] Figure 4 This is a schematic diagram of the structure of the convolutional layer in the intelligent device control method provided by the present invention;

[0056] Figure 5 This is the third flowchart illustrating the intelligent device control method provided by the present invention;

[0057] Figure 6 This is the fourth flowchart illustrating the intelligent device control method provided by the present invention;

[0058] Figure 7 This is the fifth flowchart illustrating the intelligent device control method provided by the present invention;

[0059] Figure 8 This is the sixth flowchart illustrating the intelligent device control method provided by the present invention;

[0060] Figure 9 This is a schematic diagram of the structure of the intelligent device control device provided by the present invention;

[0061] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0063] In existing technologies, for smart devices such as elevators, which are used frequently, have a complex user base, and a high population density, elevators are mainly controlled by passengers manually clicking elevator buttons. This results in various bacteria remaining on the elevator buttons, which seriously affects the health of passengers and is also inefficient.

[0064] Controlling smart devices via voice interaction, compared to the traditional method of clicking elevator buttons, avoids the risk of virus infection from passengers directly touching the buttons, offering advantages such as hygiene, safety, and efficiency. Therefore, how to control smart devices based on voice has become a research hotspot in this field.

[0065] To address the above problems, embodiments of the present invention provide a method for controlling intelligent devices. Figure 1 This is a flowchart illustrating the intelligent device control method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0066] Step 101: Obtain the voice to be recognized; the voice to be recognized includes the voice used to control the smart device; it should be noted that the smart device can be an elevator, smart access control and smart air conditioner, or any other device that can be used for voice interaction control. This embodiment does not specifically limit this.

[0067] Optionally, the speech to be recognized includes speech that needs to be recognized, such as speech made by passengers in an elevator that needs to control the elevator; the language type of the speech to be recognized can be a dialect or Mandarin, and this embodiment does not specifically limit it.

[0068] The speech to be recognized can be obtained through a speech acquisition device, which can be a smartphone, tablet computer or smart appliance with speech acquisition function, or a TV or air conditioner. After the speech to be recognized is acquired by the microphone array, the speech acquisition device can also amplify and / or reduce noise, etc., depending on the actual application scenario.

[0069] Step 102: Based on the speech to be recognized, determine the input information and input the input information into the pinyin sequence prediction model to obtain the pinyin sequence of the speech to be recognized output by the pinyin sequence prediction model; wherein, the pinyin sequence prediction model is obtained by training a multi-task model based on sample speech and sample labels of the sample speech, and the sample labels include pinyin sequence labels, tone sequence labels, chord labels and vocabulary labels;

[0070] The pinyin sequence prediction model can be built and generated based on one or more neural network models, such as one or more of convolutional neural networks (CNN), recurrent neural networks (RNN), and fully connected networks. This embodiment does not make specific limitations on this.

[0071] The Pinyin sequence prediction model is used to extract features from input information and predict them to automatically output Pinyin sequences with tones.

[0072] In this multi-task model, the number of tasks is adapted to the number of sample labels. When the sample labels include, but are not limited to, pinyin sequence labels, tone sequence labels, chord labels, and vocabulary labels, the multi-task model includes a main task and supplementary tasks. The main task is pinyin sequence prediction; the supplementary tasks include, but are not limited to, tone sequence prediction and chord-specific vocabulary prediction. Through shallow parameter sharing, the multiple tasks in the multi-task model complement, learn from, and share with each other, promoting the training of the pinyin sequence prediction model. This results in a pinyin sequence prediction model with better generalization performance and higher prediction accuracy.

[0073] Optionally, before performing step 102, a pinyin sequence prediction model can be pre-trained, specifically including the following steps:

[0074] First, a multi-task model is created. This multi-task model can include a shared feature extraction module and a prediction module. The prediction module includes a pinyin sequence prediction branch module, a tone sequence prediction branch module, and a syllable prediction branch module.

[0075] The shared feature extraction module can be a module prepared for feature extraction after parameter initialization, or it can be a pre-trained module with feature extraction function; similarly, each module in the prediction module can be a module prepared for prediction after parameter initialization, or it can be a pre-trained prediction model with prediction function. This embodiment of the invention does not make specific limitations on this.

[0076] In addition, it is necessary to construct a sample corpus to obtain sample speech from the sample corpus, and to manually annotate and verify the sample speech to obtain sample tags, including but not limited to pinyin sequence tags, tone sequence tags, chord tags and vocabulary tags.

[0077] It should be noted that different language types have different linguistic characteristics. When labeling the tags of sample speech, it is necessary to analyze the linguistic characteristics of the language type corresponding to the sample speech, including but not limited to initial consonant characteristics, vowel characteristics, tone characteristics, contracted syllable characteristics, and inherent vocabulary characteristics, so as to label corresponding pinyin sequence tags, tone sequence tags, contracted syllable tags and vocabulary tags for the sample speech according to the linguistic characteristics.

[0078] Taking Henan dialect as an example, Henan dialect has three more initial consonants and two more vowels than Mandarin; in terms of tones, it is consistent with Mandarin, having four tones: level tone, rising tone, falling-rising tone, and falling tone, and the tone values of these four tones are respectively: 24 for level tone, 42 for rising tone, 55 for falling-rising tone, and 31 for falling tone; in terms of contracted syllables, some disyllabic words in Henan dialect are contracted into monosyllabic words. For example, the word meaning "inside" is pronounced [li tou] before contraction and [liu] after contraction, and a large amount of corpus collection is required for similar contracted syllables; in terms of inherent vocabulary, for example, the word "中", pronounced "zhong", can be equivalently translated as "okay", "OK", and "all right", and a large amount of corpus collection is required for similar inherent vocabulary.

[0079] After obtaining the sample speech and the sample tags of the sample speech, iterative training can be performed on the multi-task model; for example, two-level iterative training is performed on the multi-task model, including internal iterative training and external iterative training; for each internal iteration, for any task in the multi-task model, the prediction branch module corresponding to the task and the shared feature extraction module are iteratively trained according to the sample speech corresponding to the task and the sample tag corresponding to the task. Based on the shared feature extraction module obtained in this training, the prediction branch modules corresponding to other tasks and the shared feature extraction module continue to be iteratively trained until the prediction branch models corresponding to all tasks are all trained, thereby completing one internal iterative training; then, the above internal iterative training is executed iteratively until the multi-task model converges under the main task, so as to complete the external iterative training.

[0080] For another example, overall iterative training is performed on the multi-task model; specifically, the sample input of the multi-task model is determined according to the sample speech, the shared feature extraction module performs feature extraction on the sample speech to obtain sample features of the sample speech, and then the prediction branch models corresponding to each task predict and output the prediction results of the sample speech under each task. After obtaining the prediction results of the sample speech under each task, the loss function can be calculated based on the prediction results of the sample speech under each task and the sample tags under each task, and overall training of the multi-task model is performed based on the loss function.

[0081] The trained multi-task model is thus obtained, so that the shared feature extraction module and the pinyin sequence prediction branch module in the trained multi-task model are used to construct a pinyin sequence prediction model.

[0082] After obtaining the Pinyin sequence prediction model, the input information of the Pinyin sequence prediction model can be determined based on the speech to be recognized. Specifically, the speech to be recognized can be used directly as the input information; or, the speech to be recognized can be preprocessed, such as noise reduction and multi-dimensional feature extraction, and then the input information can be determined based on the preprocessed speech to be recognized. This embodiment does not specifically limit this.

[0083] Then, based on the pinyin sequence prediction model, the input information is used to predict the pinyin sequence, thereby automatically outputting a pinyin sequence with tones.

[0084] Step 103: Based on the pinyin sequence, obtain the target text sequence of the speech to be recognized in the target language type;

[0085] The target speech type is a common language type, such as Mandarin.

[0086] Optionally, after obtaining the pinyin sequence, it can be directly mapped to form a target text sequence under the target language type; alternatively, it can first determine whether the language type of the speech to be recognized is consistent with the target language type. If they are consistent, the pinyin sequence can be directly converted into a target text sequence; if they are inconsistent, the pinyin sequence can be first converted into an initial text sequence corresponding to the language type of the speech to be recognized, and then the initial text sequence can be mapped to form a target text sequence under the target language type. This embodiment does not specifically limit the method of obtaining the target text sequence.

[0087] Step 104: Control the smart device according to the target text sequence.

[0088] Optionally, after obtaining the target text sequence, text related to the control parameters of the smart device can be extracted from the target text sequence to convert the text related to the control parameters of the smart device into smart device control instructions, so as to control the smart device according to the smart device control instructions.

[0089] Understandably, compared to the existing technology that uses manual clicking of smart device buttons to control smart devices, this embodiment uses voice interaction to automatically control smart devices, avoiding the risk of bacteria generation from touching smart device buttons. It has the advantages of being hygienic, safe, and efficient. Moreover, it supports various language types and can be applied to various scenarios. For some elderly users who are not accustomed to Mandarin, the smart device supports dialect functions, which makes it easier for users to accurately control smart devices, such as selecting floors in elevators, and can effectively improve the user experience.

[0090] The intelligent device control method provided in this embodiment predicts the pinyin sequence of the speech to be recognized for controlling the intelligent device, and converts the pinyin sequence with tone into a target text sequence under a common language type. Based on the target text sequence, the intelligent device can be automatically, efficiently and safely controlled, avoiding direct contact with the intelligent device buttons, effectively inhibiting the spread of bacteria and protecting user health.

[0091] In some embodiments, the multi-task model includes a shared feature extraction module and a prediction module, wherein the prediction module includes a pinyin sequence prediction branch module, a tone sequence prediction branch module, and a syllable prediction branch module;

[0092] like Figure 2 The diagram shown illustrates the training process of the Pinyin sequence prediction model, which includes:

[0093] Step 201: Based on the sample speech, determine the sample input information, input the sample input information into the shared feature extraction module, and obtain the sample features of the sample speech output by the shared feature extraction module;

[0094] Step 202: Input the sample features into the prediction module to obtain the sample pinyin sequence of the sample speech output by the pinyin sequence prediction branch module, the sample tone sequence of the sample speech output by the tone sequence prediction branch module, and the sample chord prediction result and sample word prediction result of the sample speech output by the chord word prediction branch module.

[0095] Step 203: Iteratively train the parameters of the multi-task model based on the pinyin sequence label, the tone sequence label, the chord label, the vocabulary label, the sample pinyin sequence, the sample tone sequence, the sample chord prediction result, and the sample vocabulary prediction result;

[0096] Step 204: Construct the pinyin sequence prediction model based on the shared feature extraction module and the pinyin sequence prediction branch module in the trained multi-task model.

[0097] like Figure 3 The diagram shown is a structural schematic of the multi-task model. The multi-task model includes a shared feature extraction module and a pinyin sequence prediction branch module, a tone sequence prediction branch module, and a chord word prediction branch module.

[0098] The feature extraction module is constructed based on multiple CNN layers and multiple RNN layers, the specific number of which can be set according to actual needs. For example, it is formed by stacking two CNN layers and two RNN layers. The feature extraction module is a shared layer for multiple prediction tasks in a multi-task model, enabling multiple tasks to complement, learn from, share, and promote each other, thereby improving the prediction accuracy of the main task (i.e., the pinyin sequence prediction task).

[0099] like Figure 4 The diagram shows the structure of each CNN layer, which includes convolutional sub-layers, batch normalization sub-layers, activation function sub-layers, and pooling sub-layers. The pooling sub-layers can be max pooling.

[0100] Each prediction branch module, including the pinyin sequence prediction branch module, the tone sequence prediction branch module, and the syllable prediction branch module, can be constructed based on multiple fully connected layers and loss function layers. The specific number of fully connected layers can be set according to actual needs. For example, the pinyin sequence prediction branch module is formed by stacking two fully connected layers and a CTC (Connectionist Temporal Classification) loss function layer. CTC can automatically align the temporal frame sequence of the sample speech with the corresponding transcribed text sequence during model training, thereby obtaining a pinyin sequence with tone.

[0101] Preferably, after determining the initial structure of the multi-task model and the sample speech and sample labels, training can begin on the multi-task model. Specifically, noise reduction and / or Mel-frequency cepstral coefficient feature extraction can be performed on the sample speech to determine the sample input information of the multi-task model based on the preprocessed sample speech. Then, the shared feature extraction module in the multi-task model processes the sample input information through two CNN layers of convolution operations, followed by two RNN layers, to obtain the sample features of the sample speech. These sample features are then input into the prediction module. The pinyin sequence prediction branch module in the prediction model processes the sample features through two fully connected layers, followed by activation functions, to predict the sample pinyin sequence; the tone sequence prediction branch module in the prediction model processes the sample features through two fully connected layers, followed by activation functions, to predict the sample tone sequence; and the chord prediction branch module in the prediction model processes the sample features through two fully connected layers, followed by activation functions, to predict the sample chord prediction result and the sample word prediction result.

[0102] Then, the loss function layer of each prediction branch module calculates and obtains the loss function of the multi-task model based on the prediction results of its own prediction output and the corresponding sample labels. Based on the loss function, backpropagation training is performed to update the parameters of the multi-task model until the model of the prediction branch module corresponding to the main task (i.e., the pinyin sequence prediction branch module) meets the training termination condition. The training termination condition includes reaching the maximum number of training iterations or the model performance meeting the optimal performance requirements. This embodiment does not specifically limit this.

[0103] One way to obtain the loss function is to calculate the loss function for each task separately, and then directly add the loss functions for multiple tasks or add them together with weights to obtain the loss function for the multi-task model.

[0104] After obtaining the trained multi-task model, a pinyin sequence prediction model can be constructed based on the shared feature extraction module and the pinyin sequence prediction branch module in the trained multi-task model.

[0105] In this embodiment, during the process of the pinyin sequence prediction model, the tone sequence prediction task and the chord word prediction task are combined as auxiliary tasks. The main task of pinyin sequence prediction is trained collaboratively through mutual learning, mutual promotion, mutual supplementation and mutual sharing among multiple tasks. This makes the trained pinyin sequence prediction model perform better, has stronger semantic rationality, and further improves the accuracy of intelligent device control.

[0106] In some embodiments, step 203, which iteratively trains the parameters of the multi-task model, further includes:

[0107] Based on the pinyin sequence label and the sample pinyin sequence, obtain the first loss function;

[0108] Based on the tone sequence labels and the sample tone sequences, a second loss function is obtained;

[0109] A third loss function is obtained based on the phonology tags, the vocabulary tags, the sample phonology prediction results, and the sample vocabulary prediction results;

[0110] The parameters of the multi-task model are iteratively trained based on the first loss function, the second loss function, and the third loss function.

[0111] Optionally, after obtaining the sample pinyin sequence, sample tone sequence, sample chord prediction result and sample word prediction result, the first loss function can be calculated based on the difference between the pinyin sequence label and the sample pinyin sequence.

[0112] The second loss function is calculated based on the difference between the pitch sequence label and the sample pitch sequence;

[0113] The third loss function is calculated based on the differences between the phonology labels and the sample phonology prediction results, as well as the differences between the vocabulary labels and the sample vocabulary prediction results. For example, the third loss function can be obtained by weighted fusion of the differences between the two.

[0114] Finally, the first, second, and third loss functions are fused to obtain the loss function of the multi-task model. Then, the parameters of the multi-task model are iteratively updated based on the loss function of the multi-task model to obtain a multi-task model that can accurately predict pinyin sequences.

[0115] In this embodiment, by combining the loss functions corresponding to multiple tasks, a multi-task model can be trained collaboratively, which can quickly and accurately obtain a pinyin sequence prediction model with good prediction performance, thereby improving the accuracy of intelligent device control and user experience.

[0116] In some embodiments, controlling the smart device according to the target text sequence includes:

[0117] If it is determined that the target text sequence contains command text, keywords are extracted from the target text sequence to obtain a keyword set;

[0118] Perform a validity verification on the keyword set;

[0119] If the keyword set passes the legality verification, a smart device control command is generated based on the keyword set.

[0120] The intelligent device is controlled according to the intelligent device control instructions.

[0121] like Figure 5 As shown, taking an elevator as an example, step 104, which involves controlling the intelligent device, further includes:

[0122] Step 1041, obtain the target text sequence;

[0123] Step 1042: Determine whether the target text sequence contains command text; if it is determined that the target text sequence contains command text, proceed to step 1043; otherwise, proceed to step 1046.

[0124] Step 1043: Extract keywords from the target text sequence to obtain a keyword set.

[0125] The keyword set includes control parameters and / or numerical parameters. Control parameters are used to control smart devices, such as keywords like "go," "up," and "down" for elevators. Numerical parameters represent the data effect that the user wants to achieve, such as keywords like "floor number" and "12" for elevators.

[0126] Step 1044: If the keyword set is successfully extracted, further determine the legality of each parameter in the keyword set. If the legality verification is passed, proceed to step 1045; otherwise, proceed to step 1046.

[0127] Step 1045: Generate elevator control instructions based on the keyword set, and control the elevator according to the elevator control instructions;

[0128] Step 1046: Exit elevator control.

[0129] In this embodiment, the voice to be recognized for controlling the smart device can be automatically recognized, and the legality verification and smart device control instructions can be automatically generated based on the recognized target text sequence. This enables contactless control of the smart device, avoids the spread of bacteria, effectively protects the user's health, and improves the safety, efficiency, and convenience of smart device control.

[0130] In some embodiments, the smart device control method further includes:

[0131] Based on the set of keywords, determine the response information corresponding to the speech to be recognized;

[0132] The response information is then broadcast according to the language type corresponding to the speech to be recognized.

[0133] The response information is divided into first response information or second response information, depending on the parameter type in the keyword set.

[0134] Optionally, taking an elevator as an example of a smart device, during elevator control, if the keyword set contains numeric parameters, then when the elevator control command triggers the corresponding floor control operation, the numeric parameters in the keyword set are configured in the prompt template in the prompt resource library to generate the first response information corresponding to the voice to be recognized, and the first response information is broadcast in the language type corresponding to the voice to be recognized. The first response information is used to indicate that the floor control operation has been triggered. For example, if the numeric parameter is 10, and the control operation for the 10th floor is triggered based on the elevator control command, then the response information "The elevator to the 10th floor has been pressed" is broadcast in the language type (such as a dialect) corresponding to the voice to be recognized. It should be noted that the prompt resource library has pre-configured prompt templates, and the prompt resource library supports multiple language types.

[0135] If the keyword set does not contain numeric parameters, a second response is generated and broadcast in the language type corresponding to the speech to be recognized. The first response is used to prompt the passenger to select a floor. For example, the response "Please tell me the floor you want to go to" is broadcast in the language type (such as a dialect) corresponding to the speech to be recognized.

[0136] In this embodiment, the system can adaptively respond to the user's speech based on the language type corresponding to the speech to be recognized, according to the keyword set, so as to achieve good voice interaction and further improve the user experience.

[0137] In some embodiments, obtaining the target text sequence of the speech to be recognized under the target language type based on the pinyin sequence includes:

[0138] Based on the first dictionary database, generate a candidate set of text sequences corresponding to the pinyin sequence;

[0139] Based on the multilingual model, the score of each candidate text sequence in the candidate text sequence set is calculated;

[0140] The target text sequence is obtained based on the candidate text sequence corresponding to the highest score.

[0141] The first dictionary database is a dictionary containing the mapping relationship between pinyin and characters (i.e., the pinyin-Chinese character dictionary database), and the pinyin and characters in this dictionary are of the same language type.

[0142] Optionally, after obtaining the pinyin sequence with tones, it can be input into a language model. The language model maps the pinyin sequence with tones to obtain the target text sequence in the target language type. The language model is constructed based on a statistical multivariate language model, such as an N-gram language model. This language model is used to analyze the corpus and dictionary, statistically calculating the candidate text sequence with the highest probability of belonging to the pinyin sequence—that is, the text sequence with the highest matching degree with the pinyin sequence—to obtain the target text sequence.

[0143] like Figure 6 As shown, step 103, obtaining the target text sequence, includes the following steps:

[0144] Step 1031: Input the target text sequence into the language model, and the language model loads the first dictionary database to obtain all legal candidate text sequences corresponding to the pinyin sequence with tone marks, forming a candidate text sequence set.

[0145] Step 1032: Based on the corpus of the N-gram language model, calculate the score of each candidate text sequence in the candidate text sequence set, that is, the output probability of each candidate text sequence in the N-gram language model.

[0146] Step 1033: Select the candidate text sequence with the highest score from the candidate text sequence set as the output, and obtain the target text sequence from the candidate text sequence with the highest score.

[0147] The N-gram language model is trained on a database of text sequences from multiple language types.

[0148] The theoretical basis of the N-gram language model is that, assuming a candidate word sequence S, it consists of a series of words w1, w2, ..., w T Given a sequence of words (T is the length of the word sequence), the probability P(S) of a given text S is the probability of the first word appearing multiplied by the probability of the second word appearing given the first word appearing, then multiplied by the probability of the third word appearing given the first two words appearing, and so on until the last word. The probability of each word appearing is related to all the preceding words.

[0149] The probability P(S) of the candidate text sequence S can then be obtained as follows:

[0150] P(S) = P(w1,w2,…,w n )=

[0151] P(w1)*P(w2|w1)*P(w3|w2)…P(w n |w n-1 );

[0152] In this embodiment, by using an N-gram language model and a dictionary, the target text sequence can be obtained efficiently and accurately, so as to achieve contactless and precise control of smart devices through voice interaction.

[0153] In some embodiments, step 1033, which involves obtaining the target text sequence based on the candidate text sequence corresponding to the highest score, further includes:

[0154] The candidate text sequence corresponding to the highest score is segmented into multiple words;

[0155] Based on the second dictionary database, the language types corresponding to the multiple words are converted to the target language type;

[0156] Based on the conversion result, the target text sequence is obtained.

[0157] The second dictionary database contains a mapping relationship between the language type (such as dialect) to which the speech to be identified belongs and the target language type (such as Mandarin), such as a dialect-Mandarin dictionary database.

[0158] Optionally, the candidate text sequence corresponding to the highest score can be segmented into words to obtain multiple words in the candidate text sequence corresponding to the highest score.

[0159] Furthermore, by loading a second dictionary, the language types corresponding to multiple words are converted to the target language type based on the mapping relationship between the language type of the speech to be recognized and the target language type in the second dictionary. This converts the candidate text sequence corresponding to the highest score into the target text sequence under the target language type, and enables automatic and accurate intelligent device control based on the target text sequence.

[0160] In some embodiments, determining the input information based on the speech to be recognized includes:

[0161] The speech to be recognized is preprocessed, including noise reduction and / or feature extraction of Mel frequency cepstral coefficients;

[0162] The input information is determined based on the preprocessed speech to be recognized.

[0163] Optionally, during the speech acquisition process, the speech acquisition device may collect a large amount of noise signals due to environmental influences, severely affecting the speech recognition accuracy. Therefore, in this embodiment, before inputting the speech to be recognized into the pinyin sequence prediction model, a speech denoising algorithm is first used to process the speech to be recognized, thereby eliminating noise signals in the speech. The speech denoising algorithm includes filtering denoising and denoising algorithms based on various neural network models, etc., which are not specifically limited in this embodiment.

[0164] In addition, feature extraction of Mel frequency cepstral coefficients can be performed on the audio information in the speech to be recognized. By performing a linear transformation on the logarithmic energy spectrum of the nonlinear Mel scale in the audio information, relevant features that can more accurately represent speech can be obtained. This allows the input information in the input pinyin sequence prediction model to more accurately represent speech information, thereby helping the subsequent speech recognition process, improving speech recognition accuracy, and thus improving the control accuracy of intelligent devices.

[0165] like Figure 7 The diagram shown illustrates the feature extraction process for Mel frequency cepstral coefficients, which mainly includes the following steps:

[0166] Step 701: Perform frame segmentation, pre-emphasis, and windowing on the speech to be recognized;

[0167] Since the speech to be recognized can be stored on the hard drive as a raw audio file, and it is a non-fixed-length speech sequence, it first needs to be divided into multiple small segments of fixed length using a certain method, i.e., frame segmentation. Next, pre-emphasis processing is required to strengthen the high-frequency components of each frame of the speech signal, thereby improving its high-frequency resolution. Finally, windowing is performed to eliminate signal discontinuities that occur at the edges of each frame's short-time signal.

[0168] For frame segmentation, the duration of each audio frame needs to be set according to the rapid changes in the speech being recognized, such as 10 to 30 milliseconds. This ensures that each frame has enough periods and the changes are not too drastic, making it more suitable for Fourier transform, which is used to analyze stationary signals. Furthermore, due to the different sampling rates of digital audio, the dimensions of each frame vector obtained from frame segmentation are also different. To avoid information loss due to time window boundaries, there needs to be some overlap between adjacent frames when offsetting the time window for each frame taken from the signal. Typically, the time window offset is half the frame length. To avoid excessive changes in characteristics between frames, the time window length and offset need to be appropriately selected, such as a time window length of 25 milliseconds and a time window offset of 10 milliseconds.

[0169] For pre-emphasis operation, the main function of pre-emphasis is to enhance the low-energy portions of the signal, resulting in a more evenly distributed speech signal across frequency bands. Generally, the energy of the acquired speech to be recognized is weaker in the high-frequency band than in the low-frequency band; therefore, it is necessary to enhance the high-frequency energy. To enhance the high-frequency energy, a digital filter is used to improve the spectrum, and the calculation formula is as follows:

[0170] H(z) = 1 - αz -1 ;

[0171]

[0172] Where α is the preemphasis coefficient, and its value range can be set according to actual needs, such as between 0.9 and 1.0; H(z) is the transfer function of the digital filter, and z is the parameter of the digital filter, which is the independent variable; S(n) is the audio signal of the nth frame after pre-emphasis, S(n) is the audio signal of the nth frame before pre-emphasis processing, and N is the total number of frames of the speech to be recognized.

[0173] When performing windowing operations on speech signals, the frequency band around zero points widens during frame processing, causing some energy loss. Therefore, a window function is introduced to truncate the speech signal. However, this process can lead to spectral leakage due to truncation. Adding a Hamming window can eliminate the signal discontinuity at the edges of each frame's short-time signal. The expression for the Hamming window is:

[0174]

[0175] (0≤n≤N, n=0,1,2,…,N;

[0176] Where W(n) is the window signal of the nth frame audio signal.

[0177] Multiplying the window signal of the nth frame audio signal with the nth frame audio signal yields the windowed nth frame audio signal.

[0178] Step 702, Fourier Transform: After the series of processing steps described above, the resulting audio signals are still time-domain signals, and the amount of speech information that can be directly obtained from the time-domain information is relatively small. Therefore, for further feature extraction, it is necessary to use Fast Fourier Transform to convert the time-domain signal of each audio signal to the corresponding frequency-domain signal of each audio signal.

[0179]

[0180] Where P(n) is the frequency domain signal of the nth frame audio signal, S(n) is the nth frame audio signal, k is the kth sampling point in the nth frame audio signal, and L is the total number of sampling points.

[0181] Step 703, Mel-spectrum filtering: A set of triangular bandpass filters is used to filter the Fourier transform of the speech to be recognized. The specific calculation formula is as follows:

[0182]

[0183] Among them, H m [k] is the frequency response function of the m-th Mel-linear frequency filter, f m-1Let m be the frequency of the m-th Mel-linear frequency filter.

[0184] Next, calculate H. m The logarithmic energy E of [k] m The formula is:

[0185]

[0186] Step 704, Discrete Cosine Transform; After the above processing, the N-dimensional eigenvector values ​​have a certain correlation between signal values ​​of different orders. In order to remove this correlation and map the signal back to a lower-dimensional space, a discrete cosine transform needs to be performed. The calculation formula is as follows:

[0187]

[0188] Among them, C i Let be the parameters of the i-th order Mel-linear frequency filter.

[0189] Step 705: Calculate dynamic features. The Mel-frequency cepstral coefficient algorithm described above only yields static features, while dynamic features require the difference between the static and static features. Combining dynamic and static features can effectively improve speech recognition performance. The formula for calculating the difference parameters is as follows:

[0190]

[0191] Where, d t C is the first-order difference value of the audio signal in frame t. t t is the value of the t-th cepstral coefficient, Q is the maximum order of the cepstral coefficient, and K is the time difference between the frame audio signals participating in the first-order difference.

[0192] In this embodiment, by performing noise reduction processing on the speech to be recognized and feature extraction processing on the Mel frequency cepstral coefficients, the input information in the input pinyin sequence prediction model can more accurately represent the speech information, thereby helping to improve the speech recognition accuracy and the control accuracy of intelligent devices.

[0193] like Figure 8 The diagram shown illustrates the complete flow of the intelligent device control method provided in this embodiment, specifically including the following steps:

[0194] Step 801: Construct a sample corpus, obtain sample speech from the sample corpus, and manually annotate the sample speech to obtain sample labels for the sample speech;

[0195] Step 802: Denoise the sample speech and extract the features of the Mel frequency cepstral coefficients to obtain sample features. Based on the sample features and sample labels, train the multi-task model to obtain the pinyin sequence prediction model.

[0196] Step 803: Acquire the speech to be recognized;

[0197] Step 804: Perform noise reduction processing and feature extraction of Mel frequency cepstral coefficients on the speech to be recognized to obtain the input information of the Pinyin sequence prediction model;

[0198] Step 805: Input the input information into the Pinyin sequence prediction model trained in step 802 to obtain the Pinyin sequence with tone of the speech to be recognized;

[0199] Step 806: Input the pinyin sequence with tones into the language model;

[0200] Step 807: Load the first dictionary and the second dictionary, as well as the N-gram language model, into the pinyin sequence with tone to obtain the target text sequence of the speech to be recognized in the target language type.

[0201] Step 808: Control the smart device based on the target text sequence;

[0202] Step 809: Announce the response information of the speech to be recognized via voice.

[0203] In summary, this embodiment provides a smart device control method with the following advantages:

[0204] Firstly, it supports multiple language types, including various dialects and Mandarin, improving the accuracy, flexibility, and applicability of smart device control;

[0205] Secondly, multi-task learning technology is introduced into the training process of the Pinyin sequence prediction model. Multiple learning tasks are created for the unique tones, chord words, and inherent vocabulary of dialects compared to Mandarin. These multiple tasks can complement, learn from, share with, and promote each other by sharing a shallow layer, thereby improving the prediction performance of the Pinyin sequence prediction model and further enhancing the accuracy of intelligent device control.

[0206] Third, the data processing flow has been optimized. The speech to be recognized is sequentially processed through speech feature extraction to obtain multi-dimensional feature data, then through a pinyin sequence prediction model to obtain a pinyin sequence, and finally through a language model to obtain a text sequence, thereby improving the speech recognition accuracy in this scenario and further enhancing the accuracy of intelligent device control.

[0207] Fourth, it supports language interaction in various languages, such as dialects, which effectively improves the user experience.

[0208] The intelligent device control device provided by the present invention is described below. The intelligent device control device described below and the intelligent device control method described above can be referred to in correspondence.

[0209] like Figure 9 As shown, this embodiment provides a smart device control device, which includes:

[0210] The voice acquisition unit 901 is used to acquire the voice to be recognized; the voice to be recognized includes voice used to control smart devices;

[0211] The first recognition unit 902 is used to determine input information based on the speech to be recognized, input the input information into the pinyin sequence prediction model, and obtain the pinyin sequence of the speech to be recognized output by the pinyin sequence prediction model;

[0212] The second recognition unit 903 is used to obtain the target text sequence of the speech to be recognized under the target language type based on the pinyin sequence;

[0213] The device control unit 904 is used to control the smart device according to the target text sequence;

[0214] The pinyin sequence prediction model is obtained by training a multi-task model based on sample speech and sample labels of the sample speech; the sample labels include pinyin sequence labels, tone sequence labels, chord labels and vocabulary labels.

[0215] The intelligent device control device provided in this embodiment predicts the pinyin sequence of the speech to be recognized for controlling the intelligent device, and converts the pinyin sequence with tone into a target text sequence under a common language type. Based on the target text sequence, the intelligent device can be automatically, efficiently and safely controlled, avoiding direct contact with the intelligent device buttons, effectively inhibiting the spread of bacteria and protecting user health.

[0216] In some embodiments, the multi-task model includes a shared feature extraction module and a prediction module, wherein the prediction module includes a pinyin sequence prediction branch module, a tone sequence prediction branch module, and a syllable prediction branch module;

[0217] The device also includes a training unit for:

[0218] Based on the sample speech, the sample input information is determined, and the sample input information is input into the shared feature extraction module to obtain the sample features of the sample speech output by the shared feature extraction module;

[0219] The sample features are input into the prediction module to obtain the sample pinyin sequence of the sample speech output by the pinyin sequence prediction branch module, the sample tone sequence of the sample speech output by the tone sequence prediction branch module, and the sample chord prediction result and sample word prediction result of the sample speech output by the chord word prediction branch module.

[0220] The parameters of the multi-task model are iteratively trained based on the pinyin sequence labels, the tone sequence labels, the chord labels, the vocabulary labels, the sample pinyin sequence, the sample tone sequence, the sample chord prediction results, and the sample vocabulary prediction results.

[0221] The pinyin sequence prediction model is constructed based on the shared feature extraction module and the pinyin sequence prediction branch module in the trained multi-task model.

[0222] In some embodiments, the training unit is further configured to:

[0223] Based on the pinyin sequence label and the sample pinyin sequence, obtain the first loss function;

[0224] Based on the tone sequence labels and the sample tone sequences, a second loss function is obtained;

[0225] A third loss function is obtained based on the phonology tags, the vocabulary tags, the sample phonology prediction results, and the sample vocabulary prediction results;

[0226] The parameters of the multi-task model are iteratively trained based on the first loss function, the second loss function, and the third loss function.

[0227] In some embodiments, the device control unit 904 is specifically used for:

[0228] If it is determined that the target text sequence contains command text, keywords are extracted from the target text sequence to obtain a keyword set;

[0229] Perform a validity verification on the keyword set;

[0230] If the keyword set passes the legality verification, a smart device control command is generated based on the keyword set.

[0231] The intelligent device is controlled according to the intelligent device control instructions.

[0232] In some embodiments, the device further includes a voice broadcasting unit, specifically used for:

[0233] Based on the set of keywords, determine the response information corresponding to the speech to be recognized;

[0234] The response information is then broadcast according to the language type corresponding to the speech to be recognized.

[0235] In some embodiments, the second identification unit 903 is specifically used for:

[0236] Based on the first dictionary database, generate a candidate set of text sequences corresponding to the pinyin sequence;

[0237] Based on the multilingual model, the score of each candidate text sequence in the candidate text sequence set is calculated;

[0238] The target text sequence is obtained based on the candidate text sequence corresponding to the highest score.

[0239] In some embodiments, the second identification unit 903 is further configured to:

[0240] The candidate text sequence corresponding to the highest score is segmented into multiple words;

[0241] Based on the second dictionary database, the language types corresponding to the multiple words are converted to the target language type;

[0242] Based on the conversion result, the target text sequence is obtained.

[0243] In some embodiments, the first identification unit 902 is specifically used for:

[0244] The speech to be recognized is preprocessed, including noise reduction and / or feature extraction of Mel frequency cepstral coefficients;

[0245] The input information is determined based on the preprocessed speech to be recognized.

[0246] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10As shown, the electronic device may include: a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004. The processor 1001 can call logical instructions in the memory 1003 to execute a smart device control method, which includes: acquiring speech to be recognized; the speech to be recognized includes speech used to control the smart device; determining input information based on the speech to be recognized, inputting the input information into a pinyin sequence prediction model to obtain a pinyin sequence of the speech to be recognized output by the pinyin sequence prediction model; acquiring a target text sequence of the speech to be recognized in a target language type based on the pinyin sequence; and controlling the smart device based on the target text sequence; wherein the pinyin sequence prediction model is obtained by training a multi-task model based on sample speech and sample labels of the sample speech; the sample labels include pinyin sequence labels, tone sequence labels, harmony labels, and vocabulary labels.

[0247] Furthermore, the logical instructions in the aforementioned memory 1003 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0248] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the intelligent device control method provided by the above methods. The method includes: acquiring speech to be recognized; the speech to be recognized includes speech for controlling an intelligent device; determining input information based on the speech to be recognized, inputting the input information into a pinyin sequence prediction model to obtain a pinyin sequence of the speech to be recognized output by the pinyin sequence prediction model; acquiring a target text sequence of the speech to be recognized in a target language type based on the pinyin sequence; and controlling the intelligent device based on the target text sequence. The pinyin sequence prediction model is obtained by training a multi-task model based on sample speech and sample labels of the sample speech; the sample labels include pinyin sequence labels, tone sequence labels, chord labels, and vocabulary labels.

[0249] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the intelligent device control method provided by the above methods. The method includes: acquiring speech to be recognized; the speech to be recognized includes speech for controlling an intelligent device; determining input information based on the speech to be recognized, inputting the input information into a pinyin sequence prediction model to obtain a pinyin sequence of the speech to be recognized output by the pinyin sequence prediction model; acquiring a target text sequence of the speech to be recognized in a target language type based on the pinyin sequence; and controlling the intelligent device based on the target text sequence; wherein the pinyin sequence prediction model is obtained by training a multi-task model based on sample speech and sample labels of the sample speech; the sample labels include pinyin sequence labels, tone sequence labels, chord labels, and vocabulary labels.

[0250] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0251] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0252] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling an intelligent device, characterized in that, include: Acquire the speech to be recognized; the speech to be recognized includes voice used to control smart devices; Based on the speech to be recognized, input information is determined, and the input information is input into the pinyin sequence prediction model to obtain the pinyin sequence of the speech to be recognized output by the pinyin sequence prediction model. Based on the pinyin sequence, obtain the target text sequence of the speech to be recognized under the target language type; The intelligent device is controlled according to the target text sequence; wherein, the training steps of the pinyin sequence prediction model include: predicting the prediction results of sample speech under each task based on the prediction branch model corresponding to each task in the multi-task model, calculating the loss function based on the prediction results of sample speech under each task and the sample labels under each task, and training the multi-task model as a whole based on the loss function; the multi-task model includes a pinyin sequence prediction task, a tone sequence prediction task, and a chord inherent vocabulary prediction task, and the sample labels include pinyin sequence labels, tone sequence labels, chord labels, and vocabulary labels.

2. The intelligent device control method according to claim 1, characterized in that, The multi-task model includes a shared feature extraction module and a prediction module. The prediction module includes a pinyin sequence prediction branch module, a tone sequence prediction branch module, and a syllable prediction branch module. The pinyin sequence prediction model was trained based on the following steps: Based on the sample speech, the sample input information is determined, and the sample input information is input into the shared feature extraction module to obtain the sample features of the sample speech output by the shared feature extraction module; The sample features are input into the prediction module to obtain the sample pinyin sequence of the sample speech output by the pinyin sequence prediction branch module, the sample tone sequence of the sample speech output by the tone sequence prediction branch module, and the sample chord prediction result and sample word prediction result of the sample speech output by the chord word prediction branch module. The parameters of the multi-task model are iteratively trained based on the pinyin sequence labels, the tone sequence labels, the chord labels, the vocabulary labels, the sample pinyin sequence, the sample tone sequence, the sample chord prediction results, and the sample vocabulary prediction results. The pinyin sequence prediction model is constructed based on the shared feature extraction module and the pinyin sequence prediction branch module in the trained multi-task model.

3. The intelligent device control method according to claim 2, characterized in that, The step of iteratively training the parameters of the multi-task model based on the pinyin sequence labels, the tone sequence labels, the chord labels, the vocabulary labels, the sample pinyin sequence, the sample tone sequence, the sample chord prediction results, and the sample vocabulary prediction results includes: Based on the pinyin sequence label and the sample pinyin sequence, obtain the first loss function; Based on the tone sequence labels and the sample tone sequences, a second loss function is obtained; A third loss function is obtained based on the phonology tags, the vocabulary tags, the sample phonology prediction results, and the sample vocabulary prediction results; The parameters of the multi-task model are iteratively trained based on the first loss function, the second loss function, and the third loss function.

4. The intelligent device control method according to any one of claims 1-3, characterized in that, The step of controlling the smart device based on the target text sequence includes: If it is determined that the target text sequence contains command text, keywords are extracted from the target text sequence to obtain a keyword set; Perform a validity verification on the keyword set; If the keyword set passes the legality verification, a smart device control command is generated based on the keyword set. The intelligent device is controlled according to the intelligent device control instructions.

5. The intelligent device control method according to claim 4, characterized in that, The method further includes: Based on the set of keywords, determine the response information corresponding to the speech to be recognized; The response information is then broadcast according to the language type corresponding to the speech to be recognized.

6. The intelligent device control method according to any one of claims 1-3, characterized in that, The step of obtaining the target text sequence of the speech to be recognized under the target language type based on the pinyin sequence includes: Based on the first dictionary database, generate a candidate set of text sequences corresponding to the pinyin sequence; Based on the multilingual model, the score of each candidate text sequence in the candidate text sequence set is calculated; The target text sequence is obtained based on the candidate text sequence corresponding to the highest score.

7. The intelligent device control method according to claim 6, characterized in that, The step of obtaining the target text sequence based on the candidate text sequence corresponding to the highest score includes: The candidate text sequence corresponding to the highest score is segmented into multiple words; Based on the second dictionary database, the language types corresponding to the multiple words are converted to the target language type; Based on the conversion result, the target text sequence is obtained.

8. The intelligent device control method according to any one of claims 1-3, characterized in that, The step of determining input information based on the speech to be recognized includes: The speech to be recognized is preprocessed, including noise reduction and / or feature extraction of Mel frequency cepstral coefficients; The input information is determined based on the preprocessed speech to be recognized.

9. A smart device control device, characterized in that, include: A voice acquisition unit is used to acquire the voice to be recognized; the voice to be recognized includes voice used to control smart devices; The first recognition unit is used to determine input information based on the speech to be recognized, input the input information into the pinyin sequence prediction model, and obtain the pinyin sequence of the speech to be recognized output by the pinyin sequence prediction model. The second recognition unit is used to obtain the target text sequence of the speech to be recognized under the target language type based on the pinyin sequence; A device control unit is used to control the smart device according to the target text sequence; The training steps of the pinyin sequence prediction model include: predicting the prediction results of sample speech under each task based on the prediction branch model corresponding to each task in the multi-task model; calculating the loss function based on the prediction results of sample speech under each task and the sample labels under each task; and training the multi-task model as a whole based on the loss function. The multi-task model includes a pinyin sequence prediction task, a tone sequence prediction task, and a chord vocabulary prediction task. The sample labels include pinyin sequence labels, tone sequence labels, chord labels, and vocabulary labels.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the intelligent device control method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the intelligent device control method as described in any one of claims 1 to 8.

Images