Voice control method and system for intelligent products based on Internet of Things

By enhancing the historical voice of smart products and optimizing model, combining multi-microphone arrays and noise constraint technology, the problem of inaccurate voice control of smart products is solved, more efficient speech recognition and user intention understanding are achieved, and smart products are improved speech control accuracy.

CN118675504BActive Publication Date: 2024-12-03CHENGDU LECHAOREN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411155845.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-12-03
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The voice control of smart products is not accurate enough and is easily affected by factors such as noise and regional accents, resulting in bias in speech recognition.

Method used

By acquiring the user's historical voice for enhanced processing, joint optimization based on the acoustic model and language model, multi-microphone array is constructed, speech direction is evaluated, noise constraints are performed, acoustic features are extracted, speech recognition model is used to convert it into speech text, and text is performed Analyze to identify user intent and generate control instructions.

Benefits of technology

It improves the accuracy of voice control, enhances the robustness and generalization capabilities of the model under real-world conditions, and achieves precise control of smart products.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118675504B_ABST
    Figure CN118675504B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of voice control, and discloses a voice control method and system for realizing intelligent products based on the Internet of Things, wherein the method comprises: jointly optimizing a preset acoustic model and a language model to obtain a voice recognition model of the product to be controlled; identifying product features of the product to be controlled, constructing a multi-microphone array of the product to be controlled, and collecting target voice of the product to be controlled; evaluating the voice direction of the target voice, constructing a voice beam of the multi-microphone array, performing noise constraints on the target voice, and obtaining a constrained target voice; extracting the acoustic features of the constrained target voice, and converting the constrained target voice into voice text using a voice recognition model; parsing the voice text to obtain a parsed voice text, identifying the user intention of the voice user, constructing the control instructions of the voice user, and executing voice control of the product to be controlled. The present invention can improve the accuracy of voice control of intelligent products.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of voice control, and in particular to a method and system for realizing voice control of intelligent products based on the Internet of Things. Background Art

[0002] Voice control of smart products refers to the ability to control smart devices using voice recognition technology and natural language processing technology. Users interact with smart products through voice commands without physical contact or using traditional input devices such as keyboards or touch screens. This interaction method allows users to communicate with devices in a more natural and intuitive way, improving convenience and user experience.

[0003] At present, the voice control of smart products mainly collects voice through the device microphone and converts the voice into control commands. This method is easily affected by factors such as noise and regional accents, resulting in certain deviations in voice recognition, making the voice control of smart products inaccurate. Summary of the invention

[0004] The present invention provides a method and system for realizing voice control of intelligent products based on the Internet of Things, the main purpose of which is to improve the accuracy of voice control of intelligent products.

[0005] To achieve the above object, the present invention provides a method for realizing voice control of smart products based on the Internet of Things, comprising:

[0006] Acquire historical speech of a voice user corresponding to the product to be controlled, enhance the historical speech, and jointly optimize a preset acoustic model and language model based on the enhanced historical speech to obtain a speech recognition model of the product to be controlled;

[0007] Identify product features of the product to be controlled, construct a multi-microphone array for the product to be controlled based on the product features, and collect target voice of the product to be controlled based on the multi-microphone array;

[0008] Evaluate the speech direction of the target speech, construct a speech beam of the multi-microphone array based on the speech direction, and perform noise constraint on the target speech based on the speech beam to obtain a constrained target speech;

[0009] Extracting acoustic features of the constrained target speech, and based on the acoustic features, converting the constrained target speech into speech text using the speech recognition model;

[0010] The voice text is parsed to obtain parsed voice text, the user intent of the voice user is identified based on the parsed voice text, control instructions of the voice user are constructed based on the user intent, and voice control of the product to be controlled is performed based on the control instructions.

[0011] Optionally, the performing enhancement processing on the historical speech to obtain enhanced historical speech includes:

[0012] identifying a speech enhancement factor of the historical speech;

[0013] determining an audio processing component of the historical speech;

[0014] Based on the audio processing component and the speech enhancement factor, writing a speech enhancement script for the historical speech;

[0015] Based on the speech enhancement script, the historical speech is enhanced to obtain the enhanced historical speech.

[0016] Optionally, based on the enhanced historical speech, jointly optimizing a preset acoustic model and a language model to obtain a speech recognition model of the product to be controlled includes:

[0017] Pre-training and labeling the enhanced historical speech to obtain the labeled enhanced historical speech;

[0018] Based on the annotated enhanced historical speech, the acoustic model and the language model are synchronously trained to obtain a training acoustic model and a training language model;

[0019] Calculating acoustic model performance and language model performance of the training acoustic model and the training language model;

[0020] When the acoustic model performance and the language model performance meet the preset acoustic model standard performance and language model standard performance, the trained acoustic model and the trained language model are fused in parallel to obtain the speech recognition model of the product to be controlled.

[0021] Optionally, constructing a multi-microphone array of the product to be controlled based on the product features includes:

[0022] Determining the microphone type of the product to be controlled;

[0023] Analyzing the microphone array geometry of the product to be controlled based on the microphone type and the product characteristics;

[0024] Based on the microphone array geometry, configuring an array processor of the product to be controlled;

[0025] Based on the array processor, a multi-microphone array of the product to be controlled is constructed.

[0026] Optionally, the evaluating the speech direction of the target speech includes:

[0027] Extracting target speech features of the target speech;

[0028] Based on the target speech features, constructing a signal covariance matrix and a noise covariance matrix of the target speech;

[0029] Performing eigenvalue decomposition on the signal covariance matrix and the noise covariance matrix to obtain signal matrix eigenvalues, signal matrix eigenvectors, noise matrix eigenvalues, and noise matrix eigenvectors;

[0030] The speech direction of the target speech is evaluated based on the signal matrix eigenvalues, the signal matrix eigenvectors, the noise matrix eigenvalues, and the noise matrix eigenvectors.

[0031] Optionally, the evaluating the speech direction of the target speech based on the signal matrix eigenvalue, the signal matrix eigenvector, the noise matrix eigenvalue, and the noise matrix eigenvector comprises:

[0032] Constructing a signal subspace of the target speech based on the signal matrix eigenvalues ​​and the signal matrix eigenvectors;

[0033] Constructing a noise subspace of the target speech based on the noise matrix eigenvalues ​​and the noise matrix eigenvectors;

[0034] The orthogonal coefficients of the signal subspace and the noise subspace are calculated using the following formula:

[0035]

[0036] in, represents the orthogonal coefficients of the signal subspace and the noise subspace, represents the noise subspace The noise matrix eigenvectors, represents transpose, represents the noise subspace The noise matrix eigenvalues ​​corresponding to the noise matrix eigenvectors are Represents the signal subspace signal matrix eigenvectors, Represents the signal subspace The signal matrix eigenvalues ​​corresponding to the signal matrix eigenvectors are represents the number of eigenvectors of the noise matrix, represents the number of eigenvectors of the signal matrix;

[0037] Based on the orthogonal coefficient, the speech direction of the target speech is estimated.

[0038] Optionally, constructing a speech beam of the multi-microphone array based on the speech direction includes:

[0039] Based on the speech direction, the weight matrix of the multi-microphone array is calculated using the following formula:

[0040]

[0041] in, represents the weight matrix of the multi-microphone array, Represents the signal covariance matrix of the target speech corresponding to the speech direction, Represents the noise covariance matrix of the target speech corresponding to the speech direction, represents the inverse of the signal covariance matrix multiplied by the 2-norm of the noise covariance matrix;

[0042] The weight matrix is ​​applied to corresponding microphones of the multi-microphone array to obtain a speech beam of the multi-microphone array.

[0043] Optionally, the converting the constrained target speech into speech text by using the speech recognition model based on the acoustic features includes:

[0044] identifying a feature vector of the acoustic feature;

[0045] Constructing an eigenvector matrix of the eigenvector;

[0046] Based on the feature vector matrix, using the speech recognition model to calculate the word sequence probability of the constrained target speech;

[0047] Based on the word sequence probability, the constrained target speech is analyzed and converted into speech text.

[0048] Optionally, parsing the voice text to obtain the parsed voice text includes:

[0049] Segmenting the speech text to obtain segmented text;

[0050] Marking the part of speech of the segmented text;

[0051] Identifying a segmented text entity of the segmented text;

[0052] Analyzing the segmentation dependency relationship of the segmentation text based on the segmentation part of speech and the segmentation text entity;

[0053] Based on the word segmentation parts of speech, the word segmentation text entities and the word segmentation dependencies, a parsed speech text of the speech text is constructed.

[0054] In order to solve the above problems, the present invention also provides a voice control system for smart products based on the Internet of Things, the system comprising:

[0055] A speech recognition model building module is used to obtain the historical speech of the voice user corresponding to the product to be controlled, enhance the historical speech, and jointly optimize the preset acoustic model and language model based on the enhanced historical speech to obtain the speech recognition model of the product to be controlled;

[0056] A target voice acquisition module, used for identifying product features of the product to be controlled, constructing a multi-microphone array of the product to be controlled based on the product features, and acquiring a target voice of the product to be controlled based on the multi-microphone array;

[0057] A target speech denoising module is used to evaluate the speech direction of the target speech, construct a speech beam of the multi-microphone array based on the speech direction, and perform noise constraints on the target speech based on the speech beam to obtain a constrained target speech;

[0058] A speech-to-text module, used for extracting acoustic features of the constrained target speech, and based on the acoustic features, converting the constrained target speech into speech text using the speech recognition model;

[0059] The product voice control module is used to parse the voice text to obtain the parsed voice text, identify the user intention of the voice user based on the parsed voice text, construct the control instruction of the voice user based on the user intention, and perform voice control on the product to be controlled based on the control instruction.

[0060] The embodiment of the present invention enhances the historical speech to obtain enhanced historical speech, which can greatly increase the diversity of training data and improve the robustness and generalization ability of the model under real-world conditions; further, the embodiment of the present invention jointly optimizes the preset acoustic model and language model based on the enhanced historical speech to obtain the speech recognition model of the product to be controlled, which can make full use of the advantages of the acoustic model and the language model and improve the overall performance of the speech recognition system; the embodiment of the present invention constructs a multi-microphone array of the product to be controlled based on the product characteristics, which can more efficiently collect speech, and the embodiment of the present invention can provide data for subsequent beamforming and noise suppression by evaluating the speech direction of the target speech. Based on; further, the embodiment of the present invention performs noise constraints on the target speech based on the speech beam, and the constrained target speech can obtain speech with higher quality. The embodiment of the present invention uses the speech recognition model to convert the constrained target speech into speech text based on the acoustic features to achieve accurate speech recognition. The embodiment of the present invention parses the speech text to obtain the parsed speech text, which can provide data basis for the subsequent user intent analysis. Finally, the embodiment of the present invention recognizes the user intent of the voice user based on the parsed speech text, which can accurately understand the user intent and based on the control instruction, executes voice control of the product to be controlled to achieve accurate control of the product to be controlled. Therefore, the voice control method and system for intelligent products based on the Internet of Things proposed in the present invention can improve the accuracy of voice control of intelligent products. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A flow chart of a method for implementing voice control of smart products based on the Internet of Things provided by one embodiment of the present invention;

[0062] Figure 2 A functional module diagram of a voice control system for smart products based on the Internet of Things provided by an embodiment of the present invention;

[0063] Figure 3 A schematic diagram of the structure of an electronic device for implementing a voice control system for smart products based on the Internet of Things provided by an embodiment of the present invention;

[0064] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0065] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0066] The embodiment of the present application provides a method for voice control of intelligent products based on the Internet of Things. The execution subject of the method for voice control of intelligent products based on the Internet of Things includes but is not limited to at least one of the electronic devices such as the server, the terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for voice control of intelligent products based on the Internet of Things can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0067] Reference Figure 1 FIG. 1 is a flow chart of a method for implementing voice control of smart products based on the Internet of Things according to an embodiment of the present invention. In this embodiment, the method for implementing voice control of smart products based on the Internet of Things includes:

[0068] S1. Obtain the historical speech of the voice user corresponding to the product to be controlled, enhance the historical speech, and jointly optimize the preset acoustic model and language model based on the enhanced historical speech to obtain the speech recognition model of the product to be controlled.

[0069] In the embodiment of the present invention, the historical voice refers to the voice data generated by a specific user when using the voice control system.

[0070] Optionally, the embodiment of the present invention can greatly increase the diversity of training data and improve the robustness and generalization ability of the model under real-world conditions by enhancing the historical speech to obtain enhanced historical speech. The enhanced historical speech refers to speech with enhanced speech diversity.

[0071] As an embodiment of the present invention, the method of enhancing the historical speech to obtain enhanced historical speech includes: identifying a speech enhancement factor of the historical speech; determining an audio processing component of the historical speech; writing a speech enhancement script for the historical speech based on the audio processing component and the speech enhancement factor; and enhancing the historical speech based on the speech enhancement script to obtain the enhanced historical speech.

[0072] Among them, the speech enhancement factor refers to a technology that can enhance the historical speech, such as time stretching, frequency perturbation, adding background noise and other technologies; the audio processing component includes: Python's librosa, soundfile, scipy and other components; the speech enhancement script refers to a code that can enhance and store speech by implementing time stretching, frequency perturbation, adding background noise and other technologies.

[0073] Furthermore, the embodiment of the present invention jointly optimizes the preset acoustic model and language model based on the enhanced historical speech to obtain the speech recognition model of the product to be controlled, which can make full use of the advantages of the acoustic model and the language model and improve the overall performance of the speech recognition system. The speech recognition model refers to the model after optimizing and fusing the acoustic model and the language model.

[0074] As an embodiment of the present invention, based on the enhanced historical speech, the preset acoustic model and language model are jointly optimized to obtain the speech recognition model of the product to be controlled, including: pre-training and labeling the enhanced historical speech to obtain labeled enhanced historical speech; based on the labeled enhanced historical speech, synchronously training the acoustic model and the language model to obtain a trained acoustic model and a trained language model; calculating the acoustic model performance and language model performance of the trained acoustic model and the trained language model; when the acoustic model performance and the language model performance meet the preset acoustic model standard performance and language model standard performance, parallelly fusing the trained acoustic model and the trained language model to obtain the speech recognition model of the product to be controlled.

[0075] Among them, the annotated enhanced historical speech refers to the speech data after the enhanced historical speech is pre-processed by word segmentation, part-of-speech tagging, etc., the trained acoustic model and the trained language model refer to the acoustic and language models after performance training using the annotated enhanced historical speech, the acoustic model performance and the language model performance refer to the evaluation of the performance of the acoustic model and the language model in a specific task, and the acoustic model standard performance and the language model standard performance refer to indicators for evaluating the performance of the trained acoustic model and the trained language model.

[0076] In detail, the acoustic model and the language model are trained synchronously; the trained acoustic model and the trained language model are obtained mainly by optimizing the model parameters through the annotated enhanced historical speech until the training is stopped when the requirements are met. The model parameters include: the structure of the neural network, the learning rate, the size of the n-gram and other parameters.

[0077] S2. Identify product features of the product to be controlled, construct a multi-microphone array for the product to be controlled based on the product features, and collect target voice of the product to be controlled based on the multi-microphone array.

[0078] In the embodiment of the present invention, the product feature refers to the characteristic attribute of the product to be controlled, such as product shape, product type, product volume and other features.

[0079] Optionally, in the embodiment of the present invention, based on the product features, a multi-microphone array of the product to be controlled is constructed to more efficiently collect voice, wherein the multi-microphone array refers to an array composed of multiple microphones.

[0080] Optionally, as an embodiment of the present invention, constructing a multi-microphone array of the product to be controlled based on the product characteristics includes: determining the microphone type of the product to be controlled; analyzing the microphone array geometry of the product to be controlled based on the microphone type and the product characteristics; configuring an array processor of the product to be controlled based on the microphone array geometry; and constructing the multi-microphone array of the product to be controlled based on the array processor.

[0081] Among them, the microphone type refers to selecting a suitable microphone type according to requirements such as environmental noise, signal frequency range and spatial resolution, such as capacitor, electret, dynamic coil and other types; the microphone array geometry refers to selecting a suitable microphone array geometry according to the size and shape of the product, such as linear array, ring array, spherical array and other structures; the array processor refers to a microphone processor with functions such as signal synchronization, beamforming, noise suppression, and echo cancellation.

[0082] In the embodiment of the present invention, the target speech refers to the speech collected by the multi-microphone array.

[0083] S3. Evaluate the speech direction of the target speech, construct a speech beam of the multi-microphone array based on the speech direction, and perform noise constraint on the target speech based on the speech beam to obtain a constrained target speech.

[0084] The embodiment of the present invention can provide a data basis for later beamforming and noise suppression by evaluating the speech direction of the target speech, wherein the speech direction refers to the direction in which the target speech is received.

[0085] As an embodiment of the present invention, the evaluation of the speech direction of the target speech includes: extracting the target speech features of the target speech; constructing the signal covariance matrix and the noise covariance matrix of the target speech based on the target speech features; performing eigenvalue decomposition on the signal covariance matrix and the noise covariance matrix to obtain signal matrix eigenvalues, signal matrix eigenvectors, noise matrix eigenvalues ​​and noise matrix eigenvectors; based on the signal matrix eigenvalues, the signal matrix eigenvectors, the noise matrix eigenvalues ​​and the noise matrix eigenvectors, evaluating the speech direction of the target speech.

[0086] Among them, the signal covariance matrix and refer to matrices representing signal energy distribution, and the noise covariance matrix refers to a matrix of noise energy distribution. In detail, the construction of the covariance matrix is ​​calculated by constructing a feature matrix (where each row represents the feature of a frame and each column represents the feature of a microphone) through the target speech feature. The signal matrix eigenvalue, the signal matrix eigenvector, the noise matrix eigenvalue and the noise matrix eigenvector refer to the numerical values ​​and corresponding vectors obtained by eigendecomposition of the covariance matrix. The eigenvalue can help us understand the energy distribution and characteristics of the signal and noise, and the eigenvector can help us understand and separate the different parts of the signal and noise.

[0087] Further, in an optional embodiment of the present invention, the speech direction of the target speech is evaluated based on the signal matrix eigenvalues, the signal matrix eigenvectors, the noise matrix eigenvalues ​​and the noise matrix eigenvectors, including: constructing a signal subspace of the target speech based on the signal matrix eigenvalues ​​and the signal matrix eigenvectors; constructing a noise subspace of the target speech based on the noise matrix eigenvalues ​​and the noise matrix eigenvectors; and calculating the orthogonal coefficients of the signal subspace and the noise subspace using the following formula:

[0088]

[0089] in, represents the orthogonal coefficients of the signal subspace and the noise subspace, represents the noise subspace The noise matrix eigenvectors, represents transpose, represents the noise subspace The noise matrix eigenvalues ​​corresponding to the noise matrix eigenvectors are Represents the signal subspace signal matrix eigenvectors, Represents the signal subspace The signal matrix eigenvalues ​​corresponding to the signal matrix eigenvectors are represents the number of eigenvectors of the noise matrix, represents the number of eigenvectors of the signal matrix;

[0090] Based on the orthogonal coefficient, the speech direction of the target speech is estimated.

[0091] The signal subspace refers to a feature space composed of actual speech signals in a multi-microphone array system, the noise subspace refers to a feature space composed of actual speech noise in a multi-microphone array system, and the orthogonal coefficient refers to the relationship between the noise subspace and the signal subspace. The closer the orthogonality is to zero, the better the orthogonality of the noise subspace and the signal subspace. The better the orthogonality, the less interference the noise subspace has on the signal subspace, so that the speech direction can be estimated more accurately. The transpose is a linear algebra operation used to change the arrangement of the matrix.

[0092] Optionally, in the embodiment of the present invention, based on the speech direction, constructing the speech beam of the multi-microphone array can achieve real-time enhancement of the target signal and real-time suppression of noise. Wherein, the speech beam refers to a sound signal enhancement area pointing to a specific direction.

[0093] Optionally, as an embodiment of the present invention, constructing the speech beam of the multi-microphone array based on the speech direction includes: calculating the weight matrix of the multi-microphone array based on the speech direction using the following formula:

[0094]

[0095] in, represents the weight matrix of the multi-microphone array, Represents the signal covariance matrix of the target speech corresponding to the speech direction, Represents the noise covariance matrix of the target speech corresponding to the speech direction, represents the inverse of the signal covariance matrix multiplied by the 2-norm of the noise covariance matrix;

[0096] The weight matrix is ​​applied to corresponding microphones of the multi-microphone array to obtain a speech beam of the multi-microphone array.

[0097] The weight matrix refers to a weighted coefficient matrix used to construct beamforming, which determines the contribution of each microphone signal, thereby enhancing the target signal and suppressing the interference signal.

[0098] Furthermore, the embodiment of the present invention performs noise constraints on the target speech based on the speech beam, and obtains constrained target speech to obtain speech with higher quality. In detail, performing noise constraints on the target speech based on the speech beam means that the target speech signal is enhanced in the beam direction and suppressed in other directions.

[0099] S4. Extracting acoustic features of the constrained target speech, and based on the acoustic features, converting the constrained target speech into speech text using the speech recognition model.

[0100] In an embodiment of the present invention, the acoustic features refer to parameters extracted from speech signals that can represent essential features of speech, and the acoustic features include: Mel-frequency cepstral coefficients, perceptual linear prediction (PLP) features, linear prediction coefficients (LPC), short-time energy, and short-time average zero-crossing rate.

[0101] Optionally, the embodiment of the present invention can realize accurate speech recognition by converting the constrained target speech into speech text based on the acoustic features using the speech recognition model, wherein the speech text refers to the text form into which the received target speech is converted.

[0102] Optionally, as an embodiment of the present invention, the method of converting the constrained target speech into speech text based on the acoustic features using the speech recognition model includes: identifying the feature vector of the acoustic features; constructing a feature vector matrix of the feature vector; based on the feature vector matrix, calculating the word sequence probability of the constrained target speech using the speech recognition model; and based on the word sequence probability, analyzing the conversion of the constrained target speech into speech text.

[0103] Among them, the feature vector refers to a feature vector formed by the feature extraction result of each frame. For example, if the MFCC feature extraction method is used and 13 MFCC coefficients are extracted, then the feature vector of each frame will be 13-dimensional, containing 13 MFCC coefficients. The feature vector matrix refers to a feature matrix organized by the feature vectors of multiple frames. Each row of the feature matrix represents a feature vector of a frame, and each column represents a feature. After the word sequence probability model processes the input speech signal, it predicts the probability of each possible word sequence appearing.

[0104] Furthermore, in an optional embodiment of the present invention, the analysis of the constrained target speech converted into speech text based on the word sequence probability can be generated by finding the most likely word sequence in the probability distribution using a decoding algorithm (such as a Viterbi algorithm, a beam search algorithm, etc.).

[0105] S5. Parse the voice text to obtain a parsed voice text, identify the user intent of the voice user based on the parsed voice text, construct a control instruction of the voice user based on the user intent, and perform voice control on the product to be controlled based on the control instruction.

[0106] The embodiment of the present invention parses the voice text to obtain the parsed voice text, which can provide data basis for the later user intention analysis. The parsed voice text refers to the text set after the voice text is parsed.

[0107] As an embodiment of the present invention, the parsing of the speech text to obtain the parsed speech text includes: segmenting the speech text to obtain the segmented text; marking the segmented word parts of speech of the segmented text; identifying the segmented word text entities of the segmented text; analyzing the segmented word dependency relationship of the segmented text based on the segmented word parts of speech and the segmented word text entities; and constructing the parsed speech text of the speech text based on the segmented word parts of speech, the segmented word text entities and the segmented word dependency relationship.

[0108] Among them, the segmented text refers to the set of words after the speech text is segmented, the segmented part of speech refers to the part of speech tagging of each word after segmentation to determine the word nature of each word (such as noun, verb, adjective, etc.), the segmented text entity refers to the named entity in the text, such as names of people, places, organization names and other entities, and the segmented dependency relationship refers to the dependency relationship between words, such as subject-predicate relationship, modification relationship and other relationships.

[0109] Furthermore, the embodiment of the present invention can accurately understand the user's intention by identifying the user's intention of the voice user based on the parsed voice text. The user intention refers to the expectation or purpose expressed by the user through voice or other means. In detail, the user intention of the voice user can be identified by mapping the input text to a predefined intent category through a preset intent classifier, and combining a preset entity extractor (Entity Extractor) to extract key information such as named entities, values, dates, and times from the parsed voice text.

[0110] Optionally, in an embodiment of the present invention, the control instruction refers to a standardized command that can be understood and executed by the device, such as an instruction to turn on / off the device, adjust parameters, etc.

[0111] Finally, the embodiment of the present invention performs voice control on the product to be controlled based on the control instruction to achieve precise control of the product to be controlled.

[0112] The embodiment of the present invention enhances the historical speech to obtain enhanced historical speech, which can greatly increase the diversity of training data and improve the robustness and generalization ability of the model under real-world conditions; further, the embodiment of the present invention jointly optimizes the preset acoustic model and language model based on the enhanced historical speech to obtain the speech recognition model of the product to be controlled, which can make full use of the advantages of the acoustic model and the language model and improve the overall performance of the speech recognition system; the embodiment of the present invention constructs a multi-microphone array of the product to be controlled based on the product characteristics, which can more efficiently collect speech, and the embodiment of the present invention can provide data for subsequent beamforming and noise suppression by evaluating the speech direction of the target speech. Based on; further, the embodiment of the present invention performs noise constraints on the target speech based on the speech beam, and the constrained target speech can obtain speech with higher quality. The embodiment of the present invention uses the speech recognition model to convert the constrained target speech into speech text based on the acoustic features to achieve accurate speech recognition. The embodiment of the present invention parses the speech text to obtain the parsed speech text, which can provide data basis for the subsequent user intent analysis. Finally, the embodiment of the present invention recognizes the user intent of the voice user based on the parsed speech text, which can accurately understand the user intent and based on the control instruction, executes voice control of the product to be controlled to achieve accurate control of the product to be controlled. Therefore, the voice control method for smart products based on the Internet of Things proposed in the present invention can improve the accuracy of voice control of smart products.

[0113] like Figure 2 , which is a functional module diagram of a voice control system for smart products based on the Internet of Things provided by an embodiment of the present invention.

[0114] The voice control system 200 for realizing intelligent products based on the Internet of Things of the present invention can be installed in an electronic device. According to the functions implemented, the voice control system 200 for realizing intelligent products based on the Internet of Things can include a voice recognition model building module 201, a target voice acquisition module 202, a target voice denoising module 203, a voice conversion text module 204 and a product voice control module 205. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0115] In this embodiment, the functions of each module / unit are as follows:

[0116] The speech recognition model building module 201 is used to obtain the historical speech of the voice user corresponding to the product to be controlled, enhance the historical speech, and jointly optimize the preset acoustic model and language model based on the enhanced historical speech to obtain the speech recognition model of the product to be controlled;

[0117] The target voice acquisition module 202 is used to identify the product characteristics of the product to be controlled, build a multi-microphone array of the product to be controlled based on the product characteristics, and acquire the target voice of the product to be controlled based on the multi-microphone array;

[0118] The target speech denoising module 203 is used to evaluate the speech direction of the target speech, construct the speech beam of the multi-microphone array based on the speech direction, and perform noise constraint on the target speech based on the speech beam to obtain the constrained target speech;

[0119] The speech-to-text module 204 is used to extract the acoustic features of the constrained target speech, and based on the acoustic features, convert the constrained target speech into speech text using the speech recognition model;

[0120] The product voice control module 205 is used to parse the voice text to obtain the parsed voice text, identify the user intention of the voice user based on the parsed voice text, construct the control instruction of the voice user based on the user intention, and perform voice control on the product to be controlled based on the control instruction.

[0121] In detail, the modules described in the voice control system 200 for realizing smart products based on the Internet of Things described in the embodiment of the present invention adopt the same technical means as the voice control method for realizing smart products based on the Internet of Things described in the accompanying drawings when used, and can produce the same technical effects, which will not be repeated here.

[0122] An embodiment of the present invention provides an electronic device for implementing a voice control method for smart products based on the Internet of Things.

[0123] See also Figure 3 As shown, the electronic device may include a processor 30, a memory 31, a communication bus 32 and a communication interface 33, and may also include a computer program stored in the memory 31 and executable on the processor 30, such as a voice control method program for implementing smart products based on the Internet of Things.

[0124] In some embodiments, the processor may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips, etc. The processor is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device, and executes or executes the programs or modules stored in the memory (for example, executing the voice control program for smart products based on the Internet of Things, etc.), and calls the data stored in the memory to execute various functions of the electronic device and process data.

[0125] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory may also include both an internal storage unit of the electronic device and an external storage device. The memory can be used not only to store application software and various types of data installed in the electronic device, such as the code of the voice control program for realizing smart products based on the Internet of Things, but also to temporarily store data that has been output or is to be output.

[0126] The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory and at least one processor, etc.

[0127] The communication interface is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0128] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor through a power management system, so that the power management system can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0129] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0130] The voice control program for realizing smart products based on the Internet of Things stored in the memory of the electronic device is a combination of multiple instructions. When running in the processor, the following can be achieved:

[0131] Acquire historical speech of a voice user corresponding to the product to be controlled, enhance the historical speech, and jointly optimize a preset acoustic model and language model based on the enhanced historical speech to obtain a speech recognition model of the product to be controlled;

[0132] Identify product features of the product to be controlled, construct a multi-microphone array for the product to be controlled based on the product features, and collect target voice of the product to be controlled based on the multi-microphone array;

[0133] Evaluate the speech direction of the target speech, construct a speech beam of the multi-microphone array based on the speech direction, and perform noise constraint on the target speech based on the speech beam to obtain a constrained target speech;

[0134] Extracting acoustic features of the constrained target speech, and based on the acoustic features, converting the constrained target speech into speech text using the speech recognition model;

[0135] The voice text is parsed to obtain parsed voice text, the user intention of the voice user is identified based on the parsed voice text, control instructions of the voice user are constructed based on the user intention, and voice control of the product to be controlled is performed based on the control instructions.

[0136] Specifically, the specific implementation method of the processor for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0137] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or system that can carry the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0138] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0139] Acquire historical speech of a voice user corresponding to the product to be controlled, enhance the historical speech, and jointly optimize a preset acoustic model and language model based on the enhanced historical speech to obtain a speech recognition model of the product to be controlled;

[0140] Identify product features of the product to be controlled, construct a multi-microphone array for the product to be controlled based on the product features, and collect target voice of the product to be controlled based on the multi-microphone array;

[0141] Evaluate the speech direction of the target speech, construct a speech beam of the multi-microphone array based on the speech direction, and perform noise constraint on the target speech based on the speech beam to obtain a constrained target speech;

[0142] Extracting acoustic features of the constrained target speech, and based on the acoustic features, converting the constrained target speech into speech text using the speech recognition model;

[0143] The voice text is parsed to obtain parsed voice text, the user intention of the voice user is identified based on the parsed voice text, control instructions of the voice user are constructed based on the user intention, and voice control of the product to be controlled is performed based on the control instructions.

[0144] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0145] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0147] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0148] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0149] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0150] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim can also be implemented by one unit or system through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for realizing voice control of intelligent products based on the Internet of Things, characterized in that: The method comprises: Acquire historical speech of a voice user corresponding to the product to be controlled, enhance the historical speech, and jointly optimize a preset acoustic model and language model based on the enhanced historical speech to obtain a speech recognition model of the product to be controlled; Identify product features of the product to be controlled, construct a multi-microphone array for the product to be controlled based on the product features, and collect target voice of the product to be controlled based on the multi-microphone array; Evaluate the speech direction of the target speech, construct a speech beam of the multi-microphone array based on the speech direction, and perform noise constraint on the target speech based on the speech beam to obtain a constrained target speech; Extracting acoustic features of the constrained target speech, and based on the acoustic features, converting the constrained target speech into speech text using the speech recognition model; Parsing the voice text to obtain parsed voice text, identifying the user intent of the voice user based on the parsed voice text, constructing a control instruction of the voice user based on the user intent, and executing voice control of the product to be controlled based on the control instruction; The performing enhancement processing on the historical speech to obtain enhanced historical speech includes: identifying a speech enhancement factor of the historical speech; determining an audio processing component of the historical speech; Based on the audio processing component and the speech enhancement factor, writing a speech enhancement script for the historical speech; Based on the speech enhancement script, the historical speech is enhanced to obtain the enhanced historical speech; The method of jointly optimizing a preset acoustic model and a language model based on the enhanced historical speech to obtain a speech recognition model of the product to be controlled includes: Pre-training and labeling the enhanced historical speech to obtain the labeled enhanced historical speech; Based on the annotated enhanced historical speech, the acoustic model and the language model are synchronously trained to obtain a training acoustic model and a training language model; Calculating acoustic model performance and language model performance of the training acoustic model and the training language model; When the acoustic model performance and the language model performance meet the preset acoustic model standard performance and language model standard performance, the trained acoustic model and the trained language model are fused in parallel to obtain a speech recognition model of the product to be controlled; The step of constructing a multi-microphone array of the product to be controlled based on the product characteristics includes: Determining the microphone type of the product to be controlled; Analyzing the microphone array geometry of the product to be controlled based on the microphone type and the product characteristics; Based on the microphone array geometry, configuring an array processor of the product to be controlled; Based on the array processor, construct a multi-microphone array of the product to be controlled; The product features refer to the characteristic attributes of the product to be controlled, including product shape, product type, and product volume; the microphone type refers to selecting a suitable microphone type according to environmental noise, signal frequency range, and spatial resolution, including condenser, electret, and dynamic; the microphone array geometry refers to selecting a suitable microphone array geometry according to the size and shape of the product, including linear array, ring array, and spherical array; the array processor refers to a microphone processor with signal synchronization, beamforming, noise suppression, and echo cancellation functions; The evaluating the speech direction of the target speech comprises: Extracting target speech features of the target speech; Based on the target speech features, constructing a signal covariance matrix and a noise covariance matrix of the target speech; Performing eigenvalue decomposition on the signal covariance matrix and the noise covariance matrix to obtain signal matrix eigenvalues, signal matrix eigenvectors, noise matrix eigenvalues, and noise matrix eigenvectors; Based on the signal matrix eigenvalues, the signal matrix eigenvectors, the noise matrix eigenvalues, and the noise matrix eigenvectors, evaluating the speech direction of the target speech; The step of evaluating the speech direction of the target speech based on the signal matrix eigenvalue, the signal matrix eigenvector, the noise matrix eigenvalue, and the noise matrix eigenvector comprises: Constructing a signal subspace of the target speech based on the signal matrix eigenvalues ​​and the signal matrix eigenvectors; Constructing a noise subspace of the target speech based on the noise matrix eigenvalues ​​and the noise matrix eigenvectors; The orthogonal coefficients of the signal subspace and the noise subspace are calculated using the following formula: in, represents the orthogonal coefficients of the signal subspace and the noise subspace, represents the noise subspace The noise matrix eigenvectors, represents transpose, represents the noise subspace The noise matrix eigenvalues ​​corresponding to the noise matrix eigenvectors are Represents the signal subspace signal matrix eigenvectors, Represents the signal subspace The signal matrix eigenvalues ​​corresponding to the signal matrix eigenvectors are represents the number of eigenvectors of the noise matrix, represents the number of eigenvectors of the signal matrix; Based on the orthogonal coefficient, evaluating the speech direction of the target speech; The constructing the speech beam of the multi-microphone array based on the speech direction includes: Based on the speech direction, the weight matrix of the multi-microphone array is calculated using the following formula: in, represents the weight matrix of the multi-microphone array, Represents the signal covariance matrix of the target speech corresponding to the speech direction, Represents the noise covariance matrix of the target speech corresponding to the speech direction, represents the inverse of the signal covariance matrix multiplied by the 2-norm of the noise covariance matrix; Applying the weight matrix to corresponding microphones of the multi-microphone array to obtain a speech beam of the multi-microphone array; The converting the constrained target speech into speech text by using the speech recognition model based on the acoustic features includes: identifying a feature vector of the acoustic feature; Constructing an eigenvector matrix of the eigenvector; Based on the feature vector matrix, using the speech recognition model to calculate the word sequence probability of the constrained target speech; Based on the word sequence probability, analyzing the constrained target speech and converting it into speech text; The step of parsing the speech text to obtain the parsed speech text includes: Segmenting the speech text to obtain segmented text; Marking the part of speech of the segmented text; Identifying a segmented text entity of the segmented text; Analyzing the segmentation dependency relationship of the segmentation text based on the segmentation part of speech and the segmentation text entity; Based on the word segmentation parts of speech, the word segmentation text entities and the word segmentation dependencies, a parsed speech text of the speech text is constructed.

2. A voice control system for intelligent products based on the Internet of Things, characterized in that: The system is used to execute the voice control method for realizing intelligent products based on the Internet of Things as claimed in claim 1, and the system comprises: A speech recognition model building module is used to obtain the historical speech of the voice user corresponding to the product to be controlled, enhance the historical speech, and jointly optimize the preset acoustic model and language model based on the enhanced historical speech to obtain the speech recognition model of the product to be controlled; A target voice acquisition module, used for identifying product features of the product to be controlled, constructing a multi-microphone array of the product to be controlled based on the product features, and acquiring a target voice of the product to be controlled based on the multi-microphone array; A target speech denoising module is used to evaluate the speech direction of the target speech, construct a speech beam of the multi-microphone array based on the speech direction, and perform noise constraints on the target speech based on the speech beam to obtain a constrained target speech; A speech-to-text module, used for extracting acoustic features of the constrained target speech, and based on the acoustic features, converting the constrained target speech into speech text using the speech recognition model; A product voice control module, used to parse the voice text to obtain a parsed voice text, identify the user intention of the voice user based on the parsed voice text, construct a control instruction of the voice user based on the user intention, and perform voice control on the product to be controlled based on the control instruction; The performing enhancement processing on the historical speech to obtain enhanced historical speech includes: identifying a speech enhancement factor of the historical speech; determining an audio processing component of the historical speech; Based on the audio processing component and the speech enhancement factor, writing a speech enhancement script for the historical speech; Based on the speech enhancement script, the historical speech is enhanced to obtain the enhanced historical speech; The method of jointly optimizing a preset acoustic model and a language model based on the enhanced historical speech to obtain a speech recognition model of the product to be controlled includes: Pre-training and labeling the enhanced historical speech to obtain the labeled enhanced historical speech; Based on the annotated enhanced historical speech, the acoustic model and the language model are synchronously trained to obtain a training acoustic model and a training language model; Calculating acoustic model performance and language model performance of the training acoustic model and the training language model; When the acoustic model performance and the language model performance meet the preset acoustic model standard performance and language model standard performance, the trained acoustic model and the trained language model are fused in parallel to obtain a speech recognition model of the product to be controlled; The step of constructing a multi-microphone array of the product to be controlled based on the product characteristics includes: Determining the microphone type of the product to be controlled; Analyzing the microphone array geometry of the product to be controlled based on the microphone type and the product characteristics; Based on the microphone array geometry, configuring an array processor of the product to be controlled; Based on the array processor, construct a multi-microphone array of the product to be controlled; The product features refer to the characteristic attributes of the product to be controlled, including product shape, product type, and product volume; the microphone type refers to selecting a suitable microphone type according to environmental noise, signal frequency range, and spatial resolution, including condenser, electret, and dynamic; the microphone array geometry refers to selecting a suitable microphone array geometry according to the size and shape of the product, including linear array, ring array, and spherical array; the array processor refers to a microphone processor with signal synchronization, beamforming, noise suppression, and echo cancellation functions; The evaluating the speech direction of the target speech comprises: Extracting target speech features of the target speech; Based on the target speech features, constructing a signal covariance matrix and a noise covariance matrix of the target speech; Performing eigenvalue decomposition on the signal covariance matrix and the noise covariance matrix to obtain signal matrix eigenvalues, signal matrix eigenvectors, noise matrix eigenvalues, and noise matrix eigenvectors; Based on the signal matrix eigenvalues, the signal matrix eigenvectors, the noise matrix eigenvalues, and the noise matrix eigenvectors, evaluating the speech direction of the target speech; The step of evaluating the speech direction of the target speech based on the signal matrix eigenvalue, the signal matrix eigenvector, the noise matrix eigenvalue, and the noise matrix eigenvector comprises: Constructing a signal subspace of the target speech based on the signal matrix eigenvalues ​​and the signal matrix eigenvectors; Constructing a noise subspace of the target speech based on the noise matrix eigenvalues ​​and the noise matrix eigenvectors; The orthogonal coefficients of the signal subspace and the noise subspace are calculated using the following formula: in, represents the orthogonal coefficients of the signal subspace and the noise subspace, represents the noise subspace The noise matrix eigenvectors, represents transpose, represents the noise subspace The noise matrix eigenvalues ​​corresponding to the noise matrix eigenvectors are Represents the signal subspace signal matrix eigenvectors, Represents the signal subspace The signal matrix eigenvalues ​​corresponding to the signal matrix eigenvectors are represents the number of eigenvectors of the noise matrix, represents the number of eigenvectors of the signal matrix; Based on the orthogonal coefficient, evaluating the speech direction of the target speech; The constructing the speech beam of the multi-microphone array based on the speech direction includes: Based on the speech direction, the weight matrix of the multi-microphone array is calculated using the following formula: in, represents the weight matrix of the multi-microphone array, Represents the signal covariance matrix of the target speech corresponding to the speech direction, Represents the noise covariance matrix of the target speech corresponding to the speech direction, represents the inverse of the signal covariance matrix multiplied by the 2-norm of the noise covariance matrix; Applying the weight matrix to corresponding microphones of the multi-microphone array to obtain a speech beam of the multi-microphone array; The converting the constrained target speech into speech text by using the speech recognition model based on the acoustic features includes: identifying a feature vector of the acoustic feature; Constructing an eigenvector matrix of the eigenvector; Based on the feature vector matrix, using the speech recognition model to calculate the word sequence probability of the constrained target speech; Based on the word sequence probability, analyzing the constrained target speech and converting it into speech text; The step of parsing the speech text to obtain the parsed speech text includes: Segmenting the speech text to obtain segmented text; Marking the part of speech of the segmented text; Identifying a segmented text entity of the segmented text; Analyzing the segmentation dependency relationship of the segmentation text based on the segmentation part of speech and the segmentation text entity; Based on the word segmentation parts of speech, the word segmentation text entities and the word segmentation dependencies, a parsed speech text of the speech text is constructed.