Household equipment control method, household equipment and storage medium

Abstract

The invention discloses a household equipment control method, household equipment and a storage medium, and relates to the technical field of household equipment, the disclosed household equipment control method comprises the following steps: pre-processing a received voice signal to obtain an audio frame sequence; processing the audio frame sequence through a voice semantic joint coding model to obtain a voice semantic feature vector, and processing the audio frame sequence through a pinyin semantic embedding pre-training model to obtain a pinyin semantic feature vector; performing intention recognition on the voice signal based on the voice semantic feature vector and the pinyin semantic feature vector, and determining a control instruction of the household equipment; and controlling the household equipment to operate by adopting the control instruction. The technical problem that the control effect of the household equipment becomes poor due to the fact that the recognition precision of the voice signal for controlling the household equipment is not high at present is solved, and the control effect of the household equipment is improved.

Description

Technical Field

[0001] This application relates to the field of household equipment technology, and more particularly to control methods for household equipment, household equipment, and storage media. Background Technology

[0002] Currently, smart home appliances are increasingly becoming a part of people's daily lives. Users can control these appliances via voice commands to achieve corresponding functions. In the voice control interaction process, voice signal recognition is the core component determining the performance of the entire control system; its accuracy directly affects the control precision of the home appliances. Most related speech recognition technologies focus on the detailed analysis of speech pronunciation, such as the extraction and matching of basic acoustic features like syllables, tones, and speech rate. However, different users may have different accents or pronunciation habits. Relying solely on recognizing the user's pronunciation makes it difficult to accurately identify the user's intentions, leading to reduced control precision of the home appliances. Summary of the Invention

[0003] The main objective of this application is to provide a control method, a home appliance, and a storage medium for home appliances, aiming to solve the technical problem that the recognition accuracy of voice signals used to control home appliances is not high, resulting in a deterioration in the control effect of home appliances.

[0004] To achieve the above objectives, this application proposes a method for controlling a household appliance, comprising:

[0005] The received speech signal is preprocessed to obtain an audio frame sequence;

[0006] The audio frame sequence is processed by a speech-semantic joint coding model to obtain a speech-semantic feature vector, and the audio frame sequence is processed by a pinyin semantic embedding pre-training model to obtain a pinyin semantic feature vector.

[0007] Based on the speech semantic feature vector and the pinyin semantic feature vector, the speech signal is used to perform intent recognition to determine the control command of the home appliance;

[0008] The control commands are used to control the operation of the household appliances.

[0009] In one embodiment, processing the audio frame sequence using a joint speech-semantic coding model to obtain a speech-semantic feature vector includes:

[0010] The audio frame sequence is input into the speech-semantic joint coding model to extract the acoustic features of the audio frame sequence;

[0011] The acoustic features are mapped to the semantic space of a pre-trained large model to obtain the speech semantic feature vector.

[0012] In one embodiment, the step of inputting the audio frame sequence into the speech-semantic joint coding model and extracting acoustic features from the audio frame sequence includes:

[0013] Local feature extraction is performed on the audio frame sequence using the convolutional neural network of the speech-semantic joint coding model to obtain tone features;

[0014] The acoustic features are extracted using a recurrent neural network of the speech-semantic joint coding model to obtain the tone variation pattern features.

[0015] The tone features and the tone change patterns are determined as acoustic features in the audio frame sequence.

[0016] In one embodiment, the step of processing the audio frame sequence using a pre-trained pinyin semantic embedding model to obtain a pinyin semantic feature vector includes:

[0017] The audio frame sequence is input into the Pinyin semantic embedding pre-trained model, and the audio frame sequence is converted into Pinyin form by the speech-to-Pinyin module of the Pinyin semantic embedding pre-trained model;

[0018] The byte pair encoding module of the pre-trained model for pinyin semantic embedding splits the pinyin form into multiple sub-word units;

[0019] The sub-word units are captured through the multi-layer Transformer structure of the pre-trained Pinyin semantic embedding model to obtain the Pinyin semantic feature vector.

[0020] In one embodiment, after processing the audio frame sequence using a speech-semantic joint coding model to obtain a speech-semantic feature vector, and processing the audio frame sequence using a pinyin semantic embedding pre-trained model to obtain a pinyin semantic feature vector, the method further includes:

[0021] By fusing the speech semantic feature vector and the pinyin semantic feature vector, a multimodal fusion feature vector is obtained;

[0022] Accent detection is performed on the multimodal fusion feature vector using a gated routing module;

[0023] Based on the accent detection results, the multimodal fusion feature vector is assigned to the accent expert module and the scene expert module. The accent expert module is configured to convert dialect pronunciation variants into standard pronunciation representations, and the scene expert module is configured to fuse environmental parameters and user behavior data to generate scene auxiliary feature vectors.

[0024] The intention of the voice signal is recognized based on the multimodal fusion feature vector and the scene auxiliary feature vector to determine the control command of the home device.

[0025] In one embodiment, the step of allocating the multimodal fusion feature vector to the accent expert module and the scene expert module based on the accent detection results includes:

[0026] If an accent is present, increase the weight of the accent expert module to a first weight and decrease the weight of the scene expert module to a second weight, wherein the first weight is greater than the second weight;

[0027] If there is no accent, reduce the weight of the accent expert module to the third weight and increase the weight of the scene expert module to the fourth weight, wherein the fourth weight is greater than the third weight;

[0028] The multimodal fusion feature vector is assigned to the accent expert module and the scene expert module after weight adjustment.

[0029] In one embodiment, the step of performing intent recognition on the speech signal based on the multimodal fusion feature vector and the scene-assisted feature vector to determine the control command of the home device includes:

[0030] The multimodal fusion feature vector and the scene auxiliary feature vector are concatenated into the same dimension, and positional encoding is added to obtain a semantic feature vector representation;

[0031] Intent parsing is performed on the semantic feature vector representation using a pre-trained Transformer model;

[0032] Fill semantic slots based on intent parsing results;

[0033] The filled semantic slots are converted into control commands for the home appliances.

[0034] In one embodiment, the method for controlling the home appliance further includes:

[0035] Environmental noise features are extracted from the audio frame sequence to obtain an environmental noise feature vector;

[0036] Based on the speech semantic feature vector, the pinyin semantic feature vector, and the environmental noise feature vector, the intention of the speech signal is recognized to determine the control command of the home appliance.

[0037] In one embodiment, the step of extracting environmental noise features from the audio frame sequence to obtain an environmental noise feature vector includes:

[0038] Extract noise frames from the audio frame sequence;

[0039] Mel spectrum analysis was performed on the noise frame to obtain the Mel spectrum;

[0040] The environmental noise feature vector is obtained by extracting features from the Mel spectrum using a preset deep residual network.

[0041] In one embodiment, the method for controlling the home appliance further includes:

[0042] The mapping relationship between the voice signal and the control command of the home device is stored in a dynamic knowledge base, so that when the same voice signal as that in the dynamic knowledge base is received, the corresponding control command is obtained based on the mapping relationship to control the home device.

[0043] In one embodiment, the preprocessing of the received speech signal to obtain an audio frame sequence includes:

[0044] The speech signal is subjected to analog-to-digital conversion to obtain a digital speech signal;

[0045] The digital audio signal is processed by framing, and the short-time energy and zero-crossing rate of each audio signal frame are determined.

[0046] The short-time energy and zero-crossing rate of each audio signal frame are smoothed respectively;

[0047] Based on the short-time energy and zero-crossing rate of each audio signal frame after smoothing, the speech frames and noise frames in all audio signal frames are determined.

[0048] The signal-to-noise ratio of the speech signal is calculated based on the average energy of all the speech frames and the average energy of all the noise frames.

[0049] Based on the signal-to-noise ratio, each audio signal frame is processed using the corresponding audio frame processing method to obtain the audio frame sequence.

[0050] In one embodiment, determining the speech frames and noise frames in all audio signal frames based on the short-time energy and zero-crossing rate after smoothing each audio signal frame includes:

[0051] If the short-time energy of an audio signal frame after smoothing is greater than the first preset noise value and the zero-crossing rate after smoothing is less than the second preset noise value, the audio signal frame is determined to be a speech frame.

[0052] If the short-time energy of an audio signal frame after smoothing is less than or equal to the first preset noise value, or the zero-crossing rate after smoothing is greater than or equal to the second preset noise value, the audio signal frame is determined to be a noise frame.

[0053] In one embodiment, the step of processing each audio signal frame using a corresponding audio frame processing method based on the signal-to-noise ratio to obtain the audio frame sequence includes:

[0054] If the signal-to-noise ratio is greater than or equal to the preset signal-to-noise ratio, the Hamming window function is used to weight each audio signal frame to obtain the audio frame sequence.

[0055] If the signal-to-noise ratio is less than the preset signal-to-noise ratio, the preprocessing parameters are adjusted, and the received speech signal is reprocessed using the adjusted preprocessing parameters to obtain the audio frame sequence. The preprocessing parameters include frame segmentation parameters, pre-emphasis coefficients, and preset noise values.

[0056] In one embodiment, the method for controlling the home appliance further includes:

[0057] Perform a Fourier transform on the noise frame to obtain the spectrum of the noise frame;

[0058] Based on the spectrum of the noise frame, determine the proportion of high-frequency energy in the noise frame;

[0059] If the proportion of high-frequency energy is greater than the preset noise proportion, the pre-emphasis coefficient is adjusted;

[0060] The received speech signal is reprocessed using the adjusted pre-emphasis coefficient.

[0061] In addition, to achieve the above objectives, this application also proposes a home appliance, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the control method for the home appliance as described above.

[0062] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the control method for the home appliance as described above.

[0063] This application extracts speech semantic feature vectors from audio frame sequences using a joint speech-semantic coding model. This maps speech acoustic features to a semantic space, rather than simply distinguishing pronunciation details. Even with differences in acoustic features, it can learn the similarity of different pronunciations in the semantic space, achieving a connection between acoustic features and semantics. Furthermore, it extracts pinyin semantic feature vectors from audio frame sequences using a pre-trained pinyin semantic embedding model, converting audio frames into semantic representations of pinyin sequences. This bypasses acoustic interference, allowing the model to infer correct semantics even if pronunciation leads to pinyin recognition errors. By upgrading speech signal recognition from pronunciation detail matching to semantic reasoning, even if the speech signal is distorted due to pronunciation, the model can still reconstruct the user's true intent through the pinyin sequence and contextual semantic association, significantly reducing the semantic reasoning error rate and improving the accuracy of home device control. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 A flowchart illustrating the first embodiment of the control method for household appliances according to this application;

[0067] Figure 2 A flowchart illustrating the second embodiment of the control method for household appliances provided in this application;

[0068] Figure 3 A flowchart illustrating the third embodiment of the control method for household appliances provided in this application;

[0069] Figure 4 A flowchart illustrating the fifth embodiment of the control method for household appliances of this application;

[0070] Figure 5 This is a schematic diagram of the speech signal preprocessing process in this application.

[0071] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0072] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0073] Currently, smart home appliances are increasingly becoming a part of people's daily lives. Users can control these appliances via voice commands to achieve corresponding functions. In the voice control interaction process, voice signal recognition is the core component determining the performance of the entire control system; its accuracy directly affects the control precision of the home appliances. Most related speech recognition technologies focus on the detailed analysis of speech pronunciation, such as the extraction and matching of basic acoustic features like syllables, tones, and speech rate. However, different users may have different accents or pronunciation habits. Relying solely on recognizing the user's pronunciation makes it difficult to accurately identify the user's speech signal, leading to reduced control precision of the home appliances.

[0074] To address the aforementioned problems, this application proposes a control method for home appliances. The main technical solution includes preprocessing the received speech signal to obtain an audio frame sequence; processing the audio frame sequence using a speech-semantic joint coding model to obtain a speech-semantic feature vector; and processing the audio frame sequence using a pinyin semantic embedding pre-training model to obtain a pinyin semantic feature vector; performing intent recognition on the speech signal based on the speech-semantic feature vector and the pinyin semantic feature vector to determine the control command for the home appliance; and using the control command to control the operation of the home appliance.

[0075] This application extracts speech semantic feature vectors from audio frame sequences using a joint speech-semantic coding model. This maps speech acoustic features to a semantic space, rather than simply distinguishing pronunciation details. Even with differences in acoustic features, it can learn the similarity of different pronunciations in the semantic space, achieving a connection between acoustic features and semantics. Furthermore, it extracts pinyin semantic feature vectors from audio frame sequences using a pre-trained pinyin semantic embedding model, converting audio frames into semantic representations of pinyin sequences. This bypasses acoustic interference, allowing the model to infer correct semantics even if pronunciation leads to pinyin recognition errors. By upgrading speech signal recognition from pronunciation detail matching to semantic reasoning, even if the speech signal is distorted due to pronunciation, the model can still reconstruct the user's true intent through the pinyin sequence and contextual semantic association, significantly reducing the semantic reasoning error rate and improving the accuracy of home device control.

[0076] It should be noted that the executing entity in this application embodiment can be a control device for a home appliance, which can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions. This computing service device or electronic device can communicate with the home appliance to control its operation based on the identified control commands. The control device can also be the home appliance itself, or a controller used with the home appliance, such as a remote control. The following description uses a home appliance, specifically an air conditioner, as an example to illustrate this embodiment and the subsequent embodiments.

[0077] It should be noted that the household appliances in the embodiments of this application can be refrigeration appliances such as refrigerators and air conditioners. They can also be cleaning appliances such as washing machines, dryers, and robot vacuum cleaners. They can also be kitchen appliances such as rice cookers, induction cookers, and dishwashers. Furthermore, they can be entertainment appliances such as smart TVs and smart door locks. The household appliances in this application can also be other devices capable of executing the control methods of the household appliances described in this application, and are not specifically limited here.

[0078] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0079] Based on this, embodiments of this application provide a control method for household appliances, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the control method for household appliances according to this application.

[0080] In this embodiment, the control method for the household equipment includes steps S10 to S40:

[0081] Step S10: Preprocess the received speech signal to obtain an audio frame sequence;

[0082] The speech signal refers to the sound wave signal emitted by the user that carries language information. It usually exists in analog form and includes characteristics such as pitch, timbre, and volume, as well as the expressed language content. This language content may not include specific control commands or parameters; it can be content reflecting the user's feelings, such as "It's so hot!" or "It's so cold!" The language content can be standard Mandarin or non-standard Mandarin. For example, it can be a speech with an accent or a dialect, such as Cantonese, Teochew, Hakka, or other dialects.

[0083] An audio frame sequence refers to a series of discrete audio signal frames obtained by dividing a continuous speech signal into segments of a certain time length. Each audio signal frame contains speech feature information within that time segment, facilitating subsequent analysis and processing of the speech signal.

[0084] In one feasible approach, the received speech signal can be pre-emphasized to enhance the high-frequency components and improve clarity. The speech signal is then framed according to a fixed frame length and frame shift, dividing the continuous speech signal into multiple discrete audio frames. Each audio frame is multiplied by a window function to reduce signal abrupt changes at both ends of the frame, minimizing spectral leakage, thus obtaining a sequence of frame-segmented and windowed audio frames. Frame-segmentation and windowing is a commonly used preprocessing method in speech signal processing, effectively preserving local features of the speech signal, reducing errors in spectral analysis, and providing a good foundation for subsequent feature extraction and processing.

[0085] In another feasible approach, the short-time energy and zero-crossing rate of the speech signal can be calculated first. Short-time energy reflects the energy level of the speech signal over a certain period, while the zero-crossing rate indicates the number of times the speech signal waveform crosses zero. By setting appropriate short-time energy and zero-crossing rate thresholds, endpoint detection is performed on the speech signal to determine its start and end points, removing silent and noisy segments to obtain valid speech segments. Within the detected valid speech segments, frame processing is performed according to a set frame length and frame shift to obtain an audio frame sequence. This method can remove invalid parts of the speech signal first, reducing noise interference to subsequent processing and improving the quality of the audio frame sequence. It is particularly suitable for speech signal preprocessing scenarios with significant background noise.

[0086] Step S20: Process the audio frame sequence using a speech-semantic joint coding model to obtain a speech-semantic feature vector; and process the audio frame sequence using a pinyin semantic embedding pre-training model to obtain a pinyin semantic feature vector.

[0087] Among them, the speech-semantic joint coding model is a deep learning model that can simultaneously extract acoustic features such as pronunciation and intonation, and semantic features from speech signals. Here, semantic features refer to the semantic content expressed by the speech. By jointly encoding speech and semantic information, a feature vector that comprehensively reflects the speech semantics is generated, which helps to understand and process speech information more accurately. For example, this speech-semantic joint coding model can be the CLIP-Voiced speech-semantic joint coding model. CLIP-Voiced encoding processing is based on a contrastive learning framework, using multiple sets of dialect speech-text pairs for pre-training, such as the semantic labels "houyitaa" and "temperature rise". By minimizing the loss, different accents with the same semantic meaning are brought closer together in the feature space, generating a speech-semantic feature vector containing acoustic details and semantic labels. In the pre-training stage of this CLIP-Voiced speech-semantic joint coding model, the contrastive learning framework is used to cluster different accents with the same semantic meaning in the feature space, improving the accuracy of semantic understanding. Noise feature discrimination loss is introduced into the contrastive learning framework, constructing a multi-task loss function: total loss L total =L 语音相似性 +L 语义一致性 +L 噪声区别 , where L 语音相似性 Used to minimize the distance between identical speech semantic feature vectors; L 语义一致性 Used to ensure the accuracy of the mapping between acoustic feature vectors and text semantic tags; L 噪声区别 This is used to enable the model to learn a noise robustness representation by comparing the same acoustic features under different noise environments.

[0088] The speech semantic feature vector is a vector representation obtained by processing the audio frame sequence using a joint speech and semantic coding model. This speech semantic feature vector contains speech and semantic information related to the speech signal, and can represent the semantic content of the speech in numerical form, making it convenient for home devices to perform calculations and analysis for subsequent tasks such as intent understanding.

[0089] Among them, the Pinyin semantic embedding pre-trained model is a deep learning model pre-trained on a large amount of Pinyin data. It can map Pinyin sequences to a low-dimensional semantic space and generate corresponding Pinyin semantic feature vectors. These Pinyin semantic feature vectors can capture the semantic relationships and contextual information between Pinyin, which helps to understand the semantics expressed by Pinyin in speech signals. For example, this Pinyin semantic embedding pre-trained model can be PinyinBERT, which constructs a Pinyin-semantic mapping based on DeepSeek pre-trained weights, splits Pinyin into sub-word units, captures pronunciation similarity through 768-dimensional embedding vectors, forces the alignment of Pinyin with semantic vectors, and supports dynamic expansion of dialect Pinyin.

[0090] The pinyin semantic feature vector is obtained by first converting the audio frame sequence into a pinyin sequence using a pinyin semantic embedding pre-trained model, and then processing the pinyin sequence. This pinyin semantic feature vector reflects the semantic information of the pinyin sequence, providing important semantic basis for subsequent intent recognition by combining speech semantic features.

[0091] The speech-semantic feature vector obtained by processing the audio frame sequence through a speech-semantic joint coding model includes:

[0092] In one feasible approach, convolutional neural networks (CNNs) can be used to extract features from audio frame sequences. The convolutional layers of a CNN automatically learn local features in the audio frames, such as spectral and temporal features, and pooling layers reduce and abstract these features. The features extracted by the CNN are then fed into a recurrent convolutional neural network (RNN). The RNN can process sequential data, capture temporal dependencies in the audio frame sequence, and further extract semantic features of the speech. The output of the RNN is then mapped and integrated through fully connected layers to obtain a speech semantic feature vector. Because CNNs effectively extract local audio features, while RNNs excel at processing sequential information, their combination leverages their respective strengths to comprehensively extract speech and semantic features from the speech signal, improving the expressive power of the feature vector.

[0093] In another feasible approach, the audio frame sequence can be positionally encoded to preserve the positional information of each frame. The positionally encoded audio frame sequence is then input into a Transformer encoder. The Transformer encoder consists of multiple self-attention mechanism layers and feedforward neural network layers. The self-attention mechanism captures global dependencies between audio frames, while the multi-head attention mechanism can simultaneously focus on different aspects of features. After processing by multiple Transformer encoder layers, the output of the last layer is used as the speech semantic feature vector. Because the Transformer model possesses powerful parallel computing capabilities and the ability to capture long-range dependencies, it can more effectively handle complex relationships in audio frame sequences, giving it an advantage in processing long-sequence speech signals and generating more representative speech semantic feature vectors.

[0094] The process of processing the audio frame sequence using a pre-trained model for pinyin semantic embedding yields the following pinyin semantic feature vectors:

[0095] In one feasible approach, audio frame sequences can be converted into pinyin sequences using speech recognition technology. A Word2Vec model is pre-trained using a large-scale pinyin corpus. The Word2Vec model learns the contextual relationships of pinyin within the corpus, mapping each pinyin to a low-dimensional vector space to obtain a semantic embedding vector. For the input pinyin sequence, the semantic embedding vectors corresponding to each pinyin are concatenated or averaged to obtain a pinyin semantic feature vector.

[0096] In another feasible approach, the audio frame sequence can be converted into a pinyin sequence first. The pinyin sequence is then specially marked with start and end markers. This processed pinyin sequence is then input into a pre-trained BERT model. The BERT model employs a bidirectional Transformer structure, capable of simultaneously considering the contextual information of the pinyin. Through multi-layer self-attention mechanisms and feedforward neural network learning, it generates a context-related representation for each pinyin syllable. The output of the last layer of the BERT model is taken as the pinyin semantic feature vector, or the output of the last layer can be further processed, such as by average pooling, to obtain the final pinyin semantic feature vector. Because the BERT model has achieved excellent results in natural language processing, it can deeply understand the semantic information of pinyin and capture richer contextual relationships through bidirectional encoding, generating higher-quality pinyin semantic feature vectors, which helps improve the accuracy of subsequent intent recognition.

[0097] Step S30: Based on the speech semantic feature vector and the pinyin semantic feature vector, perform intent recognition on the speech signal to determine the control command of the home appliance;

[0098] Control commands are generated based on the intent recognition results of voice signals and are specific commands used to control the operation of home appliances. Control commands are typically represented in a specific format or encoding, enabling the home appliance's control system to recognize and execute them, thus achieving remote control of the home appliance via voice. These control commands can include specific control actions or specific control parameters. Control actions can include starting the appliance, turning it off, reducing fan speed, reducing temperature, increasing fan speed, etc. Control parameters can include turning it off after running for 5 minutes, increasing the fan speed to the highest setting, or reducing the temperature to 20°C, etc.

[0099] In one feasible approach, the speech semantic feature vector and the pinyin semantic feature vector can be concatenated or weighted and fused to obtain a comprehensive feature vector. A large number of labeled speech samples are collected, and the comprehensive feature vector of each sample is extracted to construct training and test sets. The SVM model is trained using the training set, selecting appropriate kernel functions and parameters. During training, the SVM model classifies samples with different intentions by finding the optimal hyperplane. The trained SVM model is then used to predict on the test set, and the control commands for home appliances are determined based on the prediction results. Since the SVM model is a classic classification algorithm, it exhibits good classification performance on small sample datasets, can handle high-dimensional feature vectors, and is robust to noise and overfitting, making it suitable for classification tasks such as speech intent recognition.

[0100] In another feasible approach, the speech semantic feature vector and the pinyin semantic feature vector are also fused first. A multi-layer DNN model is constructed, including an input layer, multiple hidden layers, and an output layer. The input layer receives the fused feature vector, the hidden layers transform and extract the features using a non-linear activation function, and the output layer uses a softmax activation function to output the probability of each intent category. The DNN model is trained using a labeled speech sample dataset, and the backpropagation algorithm and optimization algorithms such as stochastic gradient descent are used to adjust the model parameters to minimize the loss function between the DNN model's predictions and the true labels. After training, the speech semantic feature vector and the pinyin semantic feature vector are input into the trained DNN model, and the control command for the home device is determined based on the intent category with the highest output probability. Because the DNN model has powerful non-linear modeling capabilities and can automatically learn complex relationships between features, the expressive power of the DNN model can be improved by increasing the number of network layers and neurons. It has advantages in handling large-scale data and complex intent recognition tasks, and can achieve high recognition accuracy.

[0101] Step S40: Use the control command to control the operation of the household equipment.

[0102] In one feasible approach, the wireless communication protocols supported by the home appliance can be determined, such as Wi-Fi, Bluetooth, and ZigBee. Control commands are encoded and encapsulated according to the format of the selected communication protocol. For example, for Wi-Fi communication, control commands can be converted into a specific data packet format, containing information such as device address, command type, and command parameters. The encoded control commands are then sent to the home appliance via a corresponding wireless communication module, such as a Wi-Fi module or a Bluetooth module. Upon receiving the commands, the home appliance uses its built-in communication protocol parsing module to parse them, extract the valid control information, and execute corresponding operations based on that information, such as adjusting the temperature or turning lights on and off. Because wireless communication protocols have wide applicability, most modern home appliances support at least one wireless communication method. Remote control can be achieved through wireless communication, allowing users to operate home appliances from different locations, and the communication stability is relatively high.

[0103] In another feasible approach, an infrared remote control code library corresponding to home appliances can be collected. Different brands and models of appliances have different infrared code formats. Control commands are mapped to the corresponding infrared codes. For example, by looking up the code library, the control command "turn on the TV" is converted into a specific infrared pulse sequence. An infrared transmitting module then transmits infrared signals according to the converted infrared pulse sequence. After receiving the infrared signals, the infrared receiving module on the home appliance decodes them, identifies the control command, and executes the corresponding operation.

[0104] In another possible implementation, after the home appliance identifies the control command through intent recognition, it retrieves the corresponding control components and control parameters of these control components from a preset database, and then controls the corresponding control components based on these control parameters.

[0105] In this embodiment, a speech-semantic joint coding model is used to extract speech-semantic feature vectors from the audio frame sequence. This maps speech acoustic features to a semantic space, rather than simply distinguishing pronunciation details. Even if there are differences in acoustic features, the model can learn the similarity of different pronunciations in the semantic space, thus achieving a connection between acoustic features and semantics. A pre-trained model using pinyin semantic embedding is used to extract pinyin semantic feature vectors from the audio frame sequence, converting the audio frames into a semantic representation of the pinyin sequence. This bypasses acoustic interference, and even if pronunciation leads to pinyin recognition errors, the model can still infer the correct semantics. This approach upgrades speech signal recognition from pronunciation detail matching to semantic reasoning. Even if the speech signal is distorted due to pronunciation, the model can still reconstruct the user's true intent through the pinyin sequence and contextual semantic association, thereby significantly reducing the semantic reasoning error rate and improving the accuracy of home device control.

[0106] In one feasible implementation, step S20 processes the audio frame sequence using a speech-semantic joint coding model to obtain a speech-semantic feature vector, including:

[0107] Step S21: Input the audio frame sequence into the speech-semantic joint coding model to extract the acoustic features in the audio frame sequence;

[0108] Acoustic features are feature representations obtained by extracting and quantifying attributes related to the physical properties of sound in a speech signal. They reflect information such as pitch, intensity, and timbre, and are fundamental to speech signal processing. Common acoustic features include tone features and tone variation patterns. These features help computers understand the basic physical properties of sound, providing crucial information for subsequent intent recognition.

[0109] In one feasible approach, the audio frame sequence can be input into the speech-semantic joint coding model to extract tone features and tone change pattern features from the audio frame sequence; and the tone features and tone change pattern features can be determined as acoustic features in the audio frame sequence.

[0110] Step S22: Map the acoustic features to the semantic space of the pre-trained large model to obtain the speech semantic feature vector.

[0111] The pre-trained large model can be a large language model trained in advance using knowledge from the relevant domain, or it can be a large model that has been fine-tuned. The type of pre-trained large model can be one of GPT, Claude, Grok, DeepSeek, or Gemini.

[0112] Semantic space is an abstract mathematical space used to represent the semantic information contained in language or speech. In this space, each point or vector corresponds to a specific semantic concept or meaning. Similar semantic concepts are located close to each other in the space, while different semantic concepts are located far apart. By mapping acoustic features to semantic space, the physical properties of sound can be transformed into semantically meaningful representations, thus facilitating semantic understanding and analysis by computers.

[0113] In one feasible approach, the acoustic features can be represented as vectors using a pre-trained large model to obtain acoustic feature vectors; the similarity between the acoustic feature vectors and various preset semantic vectors can be determined; a preset semantic vector matching the acoustic features can be determined based on the similarity; and the preset semantic vector can be determined as the speech semantic feature vector.

[0114] In this embodiment, the acoustic features are extracted by inputting the audio frame sequence into the speech-semantic joint coding model, and then mapped to the pre-trained large model semantic space to obtain the speech-semantic feature vector. This achieves a leap from the original low-level acoustic signal of the audio to the high-level semantic feature representation, enabling home devices to better understand speech semantics.

[0115] Furthermore, step S21 includes:

[0116] Step S211: Local feature extraction is performed on the audio frame sequence through the convolutional neural network of the speech-semantic joint coding model to obtain tone features;

[0117] Among them, tone features refer to the characteristics in an audio frame sequence that directly reflect the basic attributes of pitch changes. In speech, tone is usually manifested as a change in fundamental frequency. Different languages ​​or dialects have different tone patterns. For example, Mandarin Chinese has four tones: high level, rising, falling-rising, and falling, each corresponding to a specific fundamental frequency trend and range. Tone features describe the static or relatively static aspects of tone in speech, such as the fundamental frequency value of a certain audio frame or the average fundamental frequency of a short speech segment. They provide basic information for subsequent analysis of tone categories and emotional expression in speech.

[0118] In one feasible approach, the audio frame sequence is converted into a spectrogram. For each audio frame, its spectrum is calculated using STFT, and the spectrogram visually displays the energy distribution of the audio at different frequencies. Specific convolutional kernels are designed to extract tone-related features. Since tone is primarily related to the fundamental frequency, which manifests as energy peaks in the low-frequency region of the spectrum, low-pass filter-type convolutional kernels can be designed to convolve the spectrogram. For example, using bandpass convolutional kernels with different center frequencies and bandwidths, the spectrogram can be scanned to extract feature information in the low-frequency region. This information includes the approximate range and energy distribution of the fundamental frequency, serving as a preliminary representation of tone features. The convolutional results are further processed, such as taking the maximum or average value, to obtain the tone feature vector for each audio frame. Simultaneously, to consider the correlation between adjacent frames, the feature vectors of multiple consecutive frames can be concatenated or fused to form a more comprehensive tone feature representation.

[0119] In another feasible approach, MFCC features are extracted from the audio frame sequence. MFCC is a commonly used speech feature that simulates the auditory characteristics of the human ear and can effectively represent the spectral features of speech. The extraction process includes steps such as pre-emphasis, framing, windowing, Fast Fourier Transform, Mel filter bank filtering, logarithmic operation, and Discrete Cosine Transform. The extracted MFCC features are input into a convolutional neural network, which can be designed with multiple convolutional and pooling layers. The convolutional layers use convolutional kernels of different sizes to capture local patterns and structures in the MFCC features. For example, smaller convolutional kernels can capture short-term spectral changes, while larger kernels can capture long-term spectral features. Pooling layers are used to reduce the feature dimensionality and enhance the translation invariance of the features. In the last layer of the convolutional network, a fully connected layer is used to map the features extracted by the convolution to the tone feature space. By training the model, the output of the fully connected layer can accurately represent the tone features of each audio frame, such as the estimate of the fundamental frequency or the probability distribution of the tone category.

[0120] Step S212: Using the recurrent neural network of the speech-semantic joint coding model, the acoustic features are extracted to obtain the tone change pattern features.

[0121] Among them, the characteristics of tone variation focus on describing the dynamic patterns and regularities of tone changes over time. It not only focuses on the tone of a single audio frame, but also on the evolutionary trend of tone throughout the entire audio frame sequence, such as whether the tone gradually rises, falls, or remains stable, and the rate of rise or fall. Characteristics of tone variation help in understanding the rhythm and intonation of speech, as well as the speaker's intention and emotional state.

[0122] In one feasible approach, the extracted tone features are used as the input sequence for an LSTM. These tone features can be the fundamental frequency value, tone category vector, etc., for each audio frame. A multi-layer LSTM network is constructed, where LSTM units control the flow of information through input gates, forget gates, and output gates, effectively handling temporal dependencies in long sequence data. In each layer, the LSTM unit updates its hidden state and cell state based on the current input and the hidden state from the previous time step. Through training the LSTM network, the network learns the changing patterns of tone features over time. In the final layer of the network, the hidden states of the LSTM can be extracted as tone change pattern features. These hidden states contain dynamic information about the tone throughout the sequence, such as the rising or falling trend of the tone and the rate of change. Further analysis of the hidden states can be performed, such as calculating their mean, variance, slope, and other statistics, to more intuitively represent the tone change patterns.

[0123] In another feasible approach, the tone feature sequence is also used as input to the GRU. GRU is a simplified variant of LSTM, with only reset and update gates, resulting in relatively low computational cost while effectively capturing temporal dependencies. The parameters of the GRU network are initialized, including the weight matrix and bias vector. The tone feature sequence is input into the GRU network one by one. At each time step, the GRU unit updates the current hidden state based on the current input and the hidden state of the previous time step using the reset and update gates. The reset gate determines how much information from the previous hidden state needs to be forgotten, while the update gate determines the fusion ratio between the current input and the previous hidden state. After processing through multiple layers of GRU, the hidden state of the last layer contains information about tone variation patterns. Similar to LSTM, statistical analysis and feature extraction can be performed on these hidden states to obtain feature vectors representing tone variation patterns. Furthermore, an attention mechanism can be used to weight the GRU output, highlighting key moments and features of tone changes, further improving the expressive power of tone variation pattern features.

[0124] Step S213: Determine the tone features and tone change pattern features as acoustic features in the audio frame sequence.

[0125] In one feasible approach, to ensure that the tone features and tone variation pattern features are aligned in the time dimension, if the feature dimensions extracted in the two steps differ or their time resolutions are inconsistent, appropriate interpolation or downsampling is required to ensure a one-to-one correspondence. The tone features and tone variation pattern features are concatenated in chronological order. For example, for each time step, the tone feature vector and the tone variation pattern feature vector are joined into a longer vector to form the final acoustic feature vector. This concatenation method is simple and direct, preserving the original information of both features. However, if the concatenated feature dimension is too high, it may lead to increased computational complexity and overfitting. Dimensionality reduction methods such as principal component analysis can be used to reduce the dimensionality of the concatenated features, extracting the most representative feature subset as the final acoustic features.

[0126] In another feasible approach, the weights of tone features and tone variation patterns can be determined based on their importance to the acoustic features. These weights can be set experimentally or based on prior knowledge. For example, if more emphasis is placed on the static characteristics of tone, tone features can be given a larger weight; if more emphasis is placed on the dynamic changes in tone, the weight of tone variation patterns can be increased. The tone features and tone variation patterns are then multiplied by their respective weights, and the weighted features are summed to obtain the final acoustic features. This weighted fusion method allows for flexible adjustment of the contribution levels of the two features according to different application scenarios and needs, improving the adaptability and effectiveness of the acoustic features.

[0127] In this embodiment, a convolutional neural network is used to extract tone features from an audio frame sequence, which can effectively capture key information such as the local spectrum and time domain of the audio, and accurately characterize the basic attributes of pitch changes. By using a recurrent neural network to extract tone change patterns from acoustic features, the dynamic evolution patterns of tone over time can be explored in depth, and the trends and rates of tone rise and fall can be grasped. The tone features and tone change patterns are jointly determined as acoustic features, which not only preserve the static information of tone, but also cover its dynamic patterns, providing accurate feature representation for subsequent speech recognition and helping to improve the model's understanding of speech.

[0128] In one feasible implementation, step S20 involves processing the audio frame sequence using a pre-trained pinyin semantic embedding model to obtain a pinyin semantic feature vector, including:

[0129] Step S23: Input the audio frame sequence into the Pinyin semantic embedding pre-trained model, and convert the audio frame sequence into Pinyin form through the speech-to-Pinyin module of the Pinyin semantic embedding pre-trained model;

[0130] The core function of the speech-to-pinyin module is to convert the input audio frame sequence into the corresponding pinyin form. An audio frame sequence is a collection of segments of a speech signal divided along a time axis according to a certain frame length and frame shift; each frame contains speech feature information within a specific time period. The speech-to-pinyin module analyzes and processes these audio frames, extracts their acoustic features, and maps them to corresponding pinyin symbols, thus achieving the conversion from speech to pinyin.

[0131] In one feasible approach, an end-to-end speech-to-pinyin model can be constructed using a combination of convolutional neural networks (CNNs) and recurrent neural networks (RNNs) and their variants. This model directly takes an audio frame sequence as input and outputs a pinyin sequence through non-linear transformations over multiple layers of neural networks. For example, a CNN can first be used to extract features from the audio frame sequence, then a RNN or its variants can be used to perform temporal modeling on the extracted features, and finally, a fully connected layer outputs the probability distribution of the pinyin, selecting the pinyin with the highest probability as the recognition result, thus achieving the conversion from the audio frame sequence to pinyin.

[0132] Step S24: The pinyin form is split into multiple sub-word units by the byte pair encoding module of the pinyin semantic embedding pre-trained model;

[0133] The byte pair encoding module's role is to break down the pinyin into multiple sub-word units. It calculates the frequency of characters or character pairs in the pinyin sequence, merging the most frequent character pairs into a new sub-word unit, repeating this process until a preset vocabulary size is reached. For example, for the pinyin sequence "beijing", the BPE module might break it down into two sub-word units, "bei" and "jing". Further merging might result in "beijing" as a single sub-word unit.

[0134] Sub-word units are the result of the byte-to-byte encoding module processing the pinyin form; they are text units between characters and words. The size and number of sub-word units can be adjusted according to actual needs. They can effectively handle out-of-vocabulary words and rare words, improving the model's generalization ability. For example, for some uncommon Chinese pinyin characters, they can be broken down into smaller sub-word units for processing, avoiding recognition errors caused by the absence of the word in the vocabulary.

[0135] In one feasible approach, the frequency of all characters and character pairs in a Pinyin sequence can be statistically analyzed. Starting with the most frequent character pair, it is merged into a new sub-word unit, and this new sub-word unit is added to the vocabulary. This process is repeated, merging the most frequent character pairs each time, until the vocabulary reaches a preset size. When splitting the Pinyin form, the Pinyin sequence is divided into corresponding sub-word unit sequences based on the trained vocabulary. The frequency-based method is simple and intuitive, and can automatically learn common sub-word combinations based on the distribution of training data, effectively reducing the vocabulary size and improving the model's training efficiency and generalization ability. For large-scale Pinyin data, it can quickly generate suitable sub-word units.

[0136] Step S25: The sub-word units are captured through the multi-layer Transformer structure of the Pinyin semantic embedding pre-trained model to capture the association between sub-words, thereby obtaining the Pinyin semantic feature vector.

[0137] The Transformer is a deep neural network architecture based on a self-attention mechanism. A multi-layer Transformer structure consists of stacked Transformer layers. Each Transformer layer contains a self-attention sub-layer and a feedforward neural network sub-layer. Through the self-attention mechanism, the model can capture long-distance dependencies between elements in the input sequence, i.e., the associations between word units. Stacking multiple layers further enhances the model's expressive power, enabling it to learn more complex semantic and syntactic information. For example, when processing Pinyin semantic features, a multi-layer Transformer structure can analyze the semantic combinations and syntactic structures between word units, thus obtaining more accurate Pinyin semantic feature vectors.

[0138] In this embodiment, the audio frame sequence is first converted into pinyin form using the speech-to-pinyin module of the pinyin semantic embedding pre-trained model, effectively bridging the formal gap between speech and text and laying the foundation for subsequent processing. The pinyin is then split into sub-word units through the byte pair encoding module, which can flexibly handle various pinyin combinations, reduce vocabulary size, improve model generalization ability, and facilitate the model's understanding of the local semantic structure of pinyin. By using a multi-layer Transformer structure to capture the relationship between sub-words, and with its powerful self-attention mechanism, the complex connections between sub-word units at the semantic and grammatical levels can be fully explored, thereby accurately obtaining the pinyin semantic feature vector.

[0139] Based on the above embodiments of this application, in the second embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Furthermore, this embodiment addresses the issue that existing technologies only process the acoustic features of the speech modality, without integrating scene knowledge and user historical habits, and cannot combine environmental data to optimize control strategies, resulting in inaccurate control commands. For this problem, please refer to... Figure 2 After step S20, the control method for the household appliance further includes:

[0140] Step S50: Fuse the speech semantic feature vector and the pinyin semantic feature vector to obtain a multimodal fusion feature vector;

[0141] Among them, the multimodal fusion feature vector is a vector obtained by fusing feature information from the speech semantic modality and the pinyin semantic modality. It integrates features from multiple information sources, and can more comprehensively and accurately represent the semantics and related information contained in the speech signal, which helps to improve the accuracy of subsequent accent detection.

[0142] In one feasible approach, the speech semantic feature vector and the pinyin semantic feature vector are directly concatenated in sequence to form a new, longer vector as the multimodal fusion feature vector.

[0143] In another feasible approach, corresponding weights are assigned to each dimension of the speech semantic feature vector and the pinyin semantic feature vector. Then, the corresponding dimensions of the speech semantic feature vector and the pinyin semantic feature vector are multiplied and summed to obtain the multimodal fusion feature vector. These weights can be set empirically or learned from training data.

[0144] Step S60: Accent detection is performed on the multimodal fusion feature vector through the gated routing module;

[0145] The gated routing module is used for decision-making and routing based on the multimodal fusion feature vector. It can determine which processing paths or modules to assign the input features to based on specific rules or learned patterns. In this scenario, it is used to assign the multimodal fusion feature vector to the accent expert module or the scene expert module based on the accent detection results.

[0146] In one feasible approach, a series of rules can be predefined, and the multimodal fusion feature vector can be matched against these rules based on certain key features, such as the tone features of specific phonemes. For example, if the tone features of specific phonemes in certain dialects differ significantly from the standard tone features, when this difference exceeds a certain threshold, the gating routing module determines that a dialect accent exists and routes the multimodal fusion feature vector to the accent expert module; otherwise, it routes it to the scene expert module.

[0147] Step S70: Based on the accent detection results, the multimodal fusion feature vector is allocated to the accent expert module and the scene expert module. The accent expert module is configured to convert dialect pronunciation variants into standard pronunciation representations, and the scene expert module is configured to fuse environmental parameters and user behavior data to generate scene auxiliary feature vectors.

[0148] The accent detection result is a judgment about the speech accent obtained by the gated routing module after analyzing and processing the multimodal fusion feature vector. It is usually represented by some form of label or probability value to indicate whether the speech is standard pronunciation or a dialect, so that subsequent processing can be performed according to different accent characteristics. The accent detection result includes both the presence and absence of an accent, or it can be a probability value, with higher values ​​indicating a more pronounced accent.

[0149] The Accent Expert module is specifically designed to address accent issues. Its main function is to convert dialect pronunciation variations into standard pronunciation representations. By learning and modeling the pronunciation characteristics of different dialects, it can identify and correct dialect pronunciations in speech, making them closer to standard pronunciation, thereby improving the accuracy and versatility of speech processing.

[0150] The scene expert module is a module that processes speech signals by combining environmental parameters and user behavior data. It can fuse this multi-source information to generate scene-aided feature vectors. These feature vectors can reflect the specific scene in which the speech occurs and the user's usage habits, providing richer contextual information for subsequent tasks such as intent recognition.

[0151] The scene-aided feature vector is generated by the scene expert module through analysis and processing of environmental parameters such as noise levels and room size, and user behavior data such as operation history and usage time. Valuable information is extracted and encoded into a feature vector. This scene-aided feature vector can provide additional scene-related information for subsequent tasks such as intent recognition, improving the accuracy of recognition.

[0152] In one feasible approach, a specific accent detection threshold can be set. When the accent detection result exceeds this threshold, the multimodal fusion feature vector is directly assigned to the accent expert module; when the accent detection result is below the threshold, the feature vector is assigned to the scene expert module. The allocation result of this strategy is explicit: a feature vector can only be assigned to one module.

[0153] In another feasible approach, an assignment weight is calculated based on the accent detection result. For example, let the accent detection result be p, representing the probability of belonging to a certain accent. Then, the weight assigned to the accent expert module is p, and the weight assigned to the scene expert module is 1-p. The multimodal fusion feature vector is then weighted and assigned according to these two weights, i.e., the feature vector is multiplied by the corresponding weight and then input into the corresponding module.

[0154] Step S80: Based on the multimodal fusion feature vector and the scene auxiliary feature vector, perform intent recognition on the voice signal to determine the control command of the home device;

[0155] In one feasible approach, key features can be extracted from speech semantic feature vectors, pinyin semantic feature vectors, and scene-aided feature vectors, and these features can be combined into a new feature set. Then, traditional machine learning algorithms are used to train and classify the feature set, and the intent of the speech signal is determined based on the classification results, thereby generating control commands for home appliances.

[0156] In another feasible approach, a deep neural network model can be constructed, taking speech semantic feature vectors, pinyin semantic feature vectors, and scene-aided feature vectors as input. Through multi-layer nonlinear transformations, the model automatically learns the complex relationships and semantic representations between these features. The model outputs the intent classification result of the speech signal and generates corresponding control commands for home appliances based on the classification result.

[0157] Step S40: Use the control command to control the operation of the household equipment.

[0158] In this embodiment, intent recognition of speech signals can be performed based on multimodal fusion feature vectors and scene-assisted feature vectors. Since the scene-assisted feature vectors integrate environmental parameters and user behavior data, cross-module fusion analysis of speech signals using multimodal fusion feature vectors and scene-assisted feature vectors can achieve accurate intent recognition and improve the accuracy of control commands for home devices.

[0159] The relevant technologies do not dynamically adjust computing resources according to the input type. Both standard Mandarin and dialects undergo full feature calculation, causing the processor of home devices to overheat due to computational overload during dialect matching. Furthermore, unused language control configuration files consume storage. To address this issue, in a feasible implementation, step S70 includes:

[0160] Step S71: If an accent exists, increase the weight of the accent expert module to a first weight and decrease the weight of the scene expert module to a second weight, wherein the first weight is greater than the second weight;

[0161] Here, the first weight refers to the higher weight value assigned to the accent expert module when an accent is detected in the speech signal. This weight reflects the system's reliance on the processing capabilities of the accent expert module in the presence of an accent. A higher first weight means that the accent expert module will play a more dominant role in the subsequent processing of multimodal fusion feature vectors, so as to better convert dialect pronunciation variants into standard pronunciation representations.

[0162] The second weight refers to the lower weight value assigned to the scene expert module when an accent is detected in the speech signal. Since the focus is on handling accent issues at this point, the role of the scene expert module is relatively weakened. The lower second weight reflects this change in priority; the scene expert module will still participate in the processing, but its influence will be smaller.

[0163] In one feasible approach, a set of weight values ​​is pre-defined as the benchmark for the first and second weights. For example, the first weight is set to 0.8, and the second weight to 0.2. When the accent detection module determines that an accent exists in the speech signal, it directly sets the weight of the accent expert module to 0.8 and the weight of the scene expert module to 0.2. These weights can be adjusted according to the prevalence of accents in the actual application scenario and the requirements for processing effect.

[0164] Step S72: If there is no accent, reduce the weight of the accent expert module to the third weight and increase the weight of the scene expert module to the fourth weight, wherein the fourth weight is greater than the third weight;

[0165] The third weight refers to the lower weight value assigned to the accent expert module when no accent is detected in the speech signal. In the absence of an accent, the processing requirements of the accent expert module decrease, and a smaller third weight indicates its lower involvement in subsequent processing.

[0166] The fourth weight refers to the higher weight value assigned to the scene expert module when no accent is detected in the speech signal. In this case, the system pays more attention to the scene auxiliary feature vector generated by fusing environmental parameters and user behavior data. The higher fourth weight allows the scene expert module to dominate the processing of multimodal fusion feature vectors, so as to better generate auxiliary features that conform to the current scene.

[0167] In one feasible approach, a step size and initial weight values ​​can be set for weight adjustment. For example, initially, the accent expert module has a weight of 0.6, and the scene expert module has a weight of 0.4. When no accent is detected, the weight of the accent expert module is decreased by a certain step size, while the weight of the scene expert module is increased.

[0168] Step S73: Assign the multimodal fusion feature vector to the accent expert module and the scene expert module after weight adjustment.

[0169] In this embodiment, the weights of the accent expert module and the scene expert module are dynamically adjusted based on the accent detection results to adapt to the processing scenarios of standard Mandarin and dialects, thereby reducing computing power consumption and saving resources.

[0170] In one feasible implementation, step S80 includes: concatenating the multimodal fusion feature vector and the scene auxiliary feature vector into the same dimension and adding position encoding to obtain a semantic feature vector representation; performing intent parsing on the semantic feature vector representation using a pre-trained Transformer model; filling semantic slots based on the intent parsing results; and converting the filled semantic slots into control commands for the home appliance.

[0171] Specifically, multimodal fusion feature vectors and scene-aided feature vectors can be input into the DeepSeek-MoE inference engine. This engine is based on the DeepSeek large model and uses a multi-layer Transformer structure to perform deep semantic parsing of the feature vectors and generate control instructions. The multi-layer Transformer structure here refers to a pre-trained Transformer model. The specific steps are as follows:

[0172] First, the structured processing of the input feature vectors: The fused feature vector input to the inference engine is a hybrid modality feature set containing three core pieces of information: speech and semantic feature vectors from the speech-semantic joint coding model; pinyin semantic feature vectors from the pinyin semantic embedding pre-trained model; and scene-aided feature vectors from the scene expert module. The inference engine first concatenates the three types of vectors into a unified dimension through a linear projection layer and adds positional encoding to obtain a semantic feature vector representation, ensuring that the pre-trained Transformer model can capture the temporal and logical relationships between features.

[0173] Second, a 12-layer Transformer encoder model can be used, with each layer containing an 8-head self-attention mechanism and a feedforward neural network. Each of the 8 self-attention heads has a 256-dimensional dimension, allowing for parallel capture of the correlations between different semantic dimensions in the feature vectors. The feedforward neural network can employ two layers of linear transformation and the GELU activation function to perform non-linear mapping on the features output by the attention, enhancing semantic representation capabilities. The specific analysis process is as follows:

[0174] Basic semantic extraction is performed through layers 1-4: Self-attention computation is used to initially associate acoustic semantics (e.g., "temperature increase"), core word semantics (e.g., "heat"), environmental data (e.g., 30℃), and historical preferences (e.g., 25℃) in the semantic feature vector representation, filtering out irrelevant noise such as background noise and redundant information from accent variants. Key semantic focus is achieved through layers 5-8: A temperature regulation semantic anchor vector library is introduced. This library, built during pre-training, contains core association vectors such as "heat → cooling," "cold → heating," and "temperature → numerical value." The self-attention mechanism dynamically adjusts weights by calculating the cosine similarity between the current feature vector and the anchor vector. For example, the weights of features related to "heat" and "temperature increase" are increased to above 0.7, while the weights of features related to accent variants and non-temperature semantics are reduced to below 0.3, achieving precise focus on the core semantic of "temperature regulation." Semantic slot filling and entity binding are performed through layers 9-12: Key entities are extracted and bound using structured semantic slot templates, such as preset slots for "action type," "target temperature," and "execution object." For example, the action type slot is parsed from the semantics of "temperature rise" to match "cooling." The target temperature slot is extracted from "user's historical average set temperature is 25℃" provided by the scenario expert. Combined with the current temperature of 30℃, the target temperature is determined to be 25℃ by comparing the matching degree between "25℃" and user habits. The execution object slot is bound, which points to the current air conditioning device by default and can be associated through the device's unique identifier.

[0175] Third, after semantic parsing is completed, the inference engine enters the control command generation stage, which needs to meet three constraints: operational feasibility constraint, that is, verifying whether the target temperature of 25℃ is within the air conditioner's operating range. Assuming the air conditioner's operating range is 16-30℃, 25℃ is within a reasonable range and does not require adjustment; user preference consistency constraint, that is, by calculating the deviation between the target temperature of 25℃ and the historical average set temperature, assuming the deviation is 0, confirming that it conforms to user habits; and format standardization constraint, that is, according to the requirements of the air conditioner control protocol, the parsed result is formatted into JSON commands.

[0176] Fourth, control commands are sent to the control module of the home appliance through the communication protocols inside the air conditioner, such as Wi-Fi or Bluetooth, to control the home appliance.

[0177] Based on the above embodiments of this application, in the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. On this basis, this embodiment addresses the lack of robust processing mechanisms in existing technologies for strong noise environments such as compressor high-frequency noise and fan airflow noise during the operation of household appliances like air conditioners. When the air conditioner's own noise is too high, the effective signal in the collected voice signal is submerged by the noise, making it impossible to effectively determine control commands. For this problem, please refer to... Figure 3 The control method for the household equipment further includes:

[0178] Step S90: Extract environmental noise features from the audio frame sequence to obtain an environmental noise feature vector;

[0179] The environmental noise feature vector is a set of numerical vectors extracted from the audio frame sequence to characterize the environmental noise features of the speech signal. It contains various information about the environmental noise, such as its frequency distribution, energy intensity, and noise type-related characteristic parameters. By quantifying these features and assembling them into vector form, computer systems can better understand and process the impact of environmental noise on speech signals, playing a crucial role in tasks such as intent understanding. This helps home devices distinguish between speech content and environmental noise, improving recognition accuracy.

[0180] In one feasible approach, the spectrum is obtained by performing a Fourier transform on each audio signal frame in the audio frame sequence. Key features are extracted from the spectrum, such as calculating the energy proportion of different frequency bands, selecting frequency bands with relatively concentrated energy as the main feature bands, and statistically analyzing parameters such as the energy value and frequency center of these frequency bands. These parameters are then combined into a vector form in a certain order to obtain the environmental noise feature vector.

[0181] In another possible approach, noise frames are extracted from the audio frame sequence; Mel spectrum analysis is performed on the noise frames to obtain the Mel spectrum; features are extracted from the Mel spectrum using a pre-defined deep residual network to obtain the environmental noise feature vector. Mel spectrum analysis is a feature extraction method that simulates the characteristics of human hearing. The pre-defined deep residual network can be a ResNet-18 network, containing four residual blocks, convolutional layers, and global average pooling. The environmental noise feature vector reflects the frequency distribution and energy fluctuations of the noise; for example, the feature vectors of high-frequency fan noise and low-frequency compressor noise show significant differences.

[0182] For example, Mel spectrum analysis is performed on the noisy frame to convert the 16kHz noise signal into a 40-dimensional Mel spectrum; noise features are extracted based on the ResNet-18 network: the Mel spectrum is input into the ResNet-18 structure, and the output is a 128-dimensional environmental noise feature vector.

[0183] Step S100: Based on the speech semantic feature vector, the pinyin semantic feature vector, and the environmental noise feature vector, perform intent recognition on the speech signal to determine the control command of the home appliance.

[0184] In one feasible approach, a deep neural network model can be constructed, comprising multiple input layers that receive speech semantic feature vectors, pinyin semantic feature vectors, and environmental noise feature vectors, respectively. Within the network, features from different sources are integrated and further extracted through nonlinear transformations and feature fusion of multiple layers of neurons. For example, a convolutional neural network can be used for local feature extraction, followed by a recurrent neural network or its variants to model the sequence features. Classification or regression is then performed through fully connected layers and an output layer to determine control commands for home appliances.

[0185] In another feasible approach, a set of rules can be defined based on common correspondences between speech semantics, pinyin semantics, and environmental noise features and control commands for home appliances. For example, if the speech semantics contain "open" and the pinyin semantics correspond to the name of an appliance, while the environmental noise features indicate a relatively quiet environment, then the command is determined to be an instruction to open the corresponding appliance. Simultaneously, different weights are assigned to different feature vectors, and the matching degree of the rules is comprehensively evaluated based on these weights. For instance, the weight of the speech semantic feature vector may be higher, followed by the pinyin semantic feature vector, while the environmental noise feature vector has a relatively lower weight but plays a supporting role in the judgment. The final control command is determined by calculating a comprehensive score.

[0186] In another possible implementation, a multimodal fusion feature vector can be obtained by fusing the speech semantic feature vector, the pinyin semantic feature vector, and the environmental noise feature vector; the multimodal fusion feature vector is then subjected to accent detection by a gating routing module; based on the accent detection results, the multimodal fusion feature vector is allocated to an accent expert module and a scene expert module, wherein the accent expert module is configured to convert dialect pronunciation variants into standard pronunciation representations, and the scene expert module is configured to fuse environmental parameters and user behavior data to generate a scene auxiliary feature vector; the speech signal is then subjected to intent recognition based on the multimodal fusion feature vector and the scene auxiliary feature vector to determine the control commands for the home appliance.

[0187] The step of allocating the multimodal fusion feature vector to the accent expert module and the scene expert module based on the accent detection results includes: if an accent exists, increasing the weight of the accent expert module to a first weight and decreasing the weight of the scene expert module to a second weight, wherein the first weight is greater than the second weight; if no accent exists, decreasing the weight of the accent expert module to a third weight and increasing the weight of the scene expert module to a fourth weight, wherein the fourth weight is greater than the third weight; and allocating the multimodal fusion feature vector to the accent expert module and the scene expert module after weight adjustment.

[0188] The step of performing intent recognition on the speech signal based on the multimodal fusion feature vector and the scene auxiliary feature vector to determine the control command of the home device includes: concatenating the multimodal fusion feature vector and the scene auxiliary feature vector into the same dimension and adding position encoding to obtain a semantic feature vector representation; parsing the semantic feature vector representation using a pre-trained Transformer model; filling semantic slots based on the intent parsing results; and converting the filled semantic slots into control commands for the home device.

[0189] Step S40: Use the control command to control the operation of the household equipment.

[0190] In this embodiment, by introducing environmental noise feature vectors for intent recognition and determination of control commands for home appliances, a robust processing mechanism can be designed for strong noise environments such as compressor high-frequency noise and fan airflow noise when home appliances such as air conditioners are running. This avoids interference of environmental noise with effective signals in speech signals and improves the effectiveness of control commands.

[0191] Based on the above embodiments of this application, in the fourth embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Furthermore, the control method for the home appliance further includes: storing the mapping relationship between the voice signal and the control commands of the home appliance in a dynamic knowledge base, so that when the same voice signal as that in the dynamic knowledge base is received, the corresponding control command is obtained based on the mapping relationship to control the home appliance.

[0192] For example, after completing this speech signal processing, the mapping relationship between "houyitaa" and "cooling" is recorded in the dynamic knowledge base. The dynamic knowledge base is stored using an SQLite database, and the stored table structure includes fields such as "speech expression," "semantic category," "mapping instruction," "usage count," and "noise environment marker." In this recording, "houyitaa" is stored in the "speech expression" field, "temperature adjustment - cooling" is stored in the "semantic category" field, "cooling to 25℃" is stored in the "mapping instruction" field, the "usage count" field is initialized to 1, and the "noise environment marker" field records noise-related information such as the detected signal-to-noise ratio. The next time the user speaks a similar speech, during the MoE expert dynamic routing stage, the mapping relationship will be searched first from the dynamic knowledge base. If a matching record is found, routing will be performed directly according to the recorded mapping relationship, without the need for complex cross-modal feature encoding and intent parsing processes, thus improving routing efficiency and achieving incremental learning.

[0193] In this embodiment, the above method can improve the efficiency of determining control commands, reduce the processing pressure on home appliances, and reduce resource waste.

[0194] Based on the above embodiments of this application, in the fifth embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Step S10 includes:

[0195] Step S11: Perform analog-to-digital conversion on the speech signal to obtain a digital speech signal;

[0196] Digital speech signals are discrete digital signals obtained by converting continuously varying analog speech signals into digital signals through analog-to-digital conversion. They represent the amplitude information of speech in binary code.

[0197] Step S12: Perform frame segmentation on the digital audio signal and determine the short-time energy and zero-crossing rate of each audio signal frame;

[0198] Short-time energy reflects the energy level of a digital speech signal over a short period of time. It is calculated by segmenting the digital speech signal into frames and then determining the energy value of each frame. Short-time energy can be used to distinguish between speech and noise because digital speech signals typically have high short-time energy, while noise has relatively low and stable short-time energy.

[0199] Zero-crossing rate refers to the number of times a digital audio signal crosses zero within one frame. It reflects the frequency characteristics of the audio signal; for example, unvoiced sounds have a higher zero-crossing rate, while voiced sounds have a lower rate. Analyzing the zero-crossing rate can help identify different phonemes and speech segments.

[0200] In one feasible approach, a rectangular window can be used to frame the digital audio signal, dividing the signal into segments with a fixed frame length and frame shift. For each frame, when calculating its short-time energy, the sum of the squares of the amplitude values ​​of all sampling points within the frame is used as the short-time energy; when calculating the zero-crossing rate, the number of times the signal value crosses zero within the frame is counted.

[0201] In another feasible approach, a Hamming window is used to weight the digital speech signal before framing. The window function of the Hamming window gradually decays at the edges, reducing spectral leakage. The methods for calculating short-time energy and zero-crossing rate are the same as in the rectangular window framing method, but due to the use of the Hamming window, the calculation results better reflect the true characteristics of the signal. Compared to the rectangular window, this method better preserves the spectral characteristics of the signal, improves the accuracy of feature calculation, and is suitable for scenarios requiring high precision in speech signal analysis.

[0202] Step S13: Smooth the short-time energy and zero-crossing rate of each audio signal frame;

[0203] In one feasible approach, the short-time energy or zero-crossing rate of an audio signal frame is averaged across several preceding and following frames, and this average is used as the smoothed result for the current frame. For example, a 5-frame moving average can be used, where the smoothed value of the current frame equals the average of the short-time energy or zero-crossing rate of the current frame and the two frames preceding and following it. This method effectively smooths data fluctuations, reduces the impact of noise, and preserves the overall trend of the data.

[0204] In another feasible approach, when smoothing the short-time energy or zero-crossing rate of an audio signal frame, a larger weight is given to the current frame's value, and a smaller weight is given to the values ​​of past frames, with the weights decreasing exponentially. The calculation formula is: Sn = α * Xn + (1 - α) * Sn-1, where Sn is the smoothed value of the current frame, Xn is the original value of the current frame, Sn-1 is the smoothed value of the previous frame, and α is the smoothing coefficient, with a value ranging from 0 < α < 1. This method can respond to data changes more quickly and also has a certain smoothing effect on noise, making it suitable for scenarios with high real-time requirements and rapid data changes.

[0205] Step S14: Based on the short-time energy and zero-crossing rate of each audio signal frame after smoothing, determine the speech frames and noise frames in all audio signal frames;

[0206] In audio signal framing, frames containing speech information are identified based on certain characteristics such as short-time energy and zero-crossing rate. Speech frames are the basic units of speech signal processing and analysis, containing both pronunciation features and semantic information.

[0207] In contrast to speech frames, noise frames refer to frames that, after framing digital speech signals, are determined to contain no valid speech information and mainly consist of background noise. The presence of noise frames affects the quality of the speech signal and the accuracy of subsequent processing, requiring appropriate handling.

[0208] In one possible implementation, if the short-time energy of an audio signal frame after smoothing is greater than a first preset noise value and the zero-crossing rate after smoothing is less than a second preset noise value, the audio signal frame is determined to be a speech frame; if the short-time energy of an audio signal frame after smoothing is less than or equal to the first preset noise value or the zero-crossing rate after smoothing is greater than or equal to the second preset noise value, the audio signal frame is determined to be a noise frame.

[0209] In another possible implementation, a maximum and a minimum threshold value for short-time energy, as well as a maximum and a minimum threshold value for the zero-crossing rate, can be set. If the short-time energy of a signal frame is greater than the maximum threshold value and the zero-crossing rate is less than the maximum threshold value, it is determined to be a speech frame; if the short-time energy is less than the minimum threshold value and the zero-crossing rate is greater than the minimum threshold value, it is determined to be a noise frame.

[0210] Step S15: Calculate the signal-to-noise ratio of the speech signal based on the average energy of all the speech frames and the average energy of all the noise frames;

[0211] Average energy is the value obtained by averaging the short-time energy of a set of audio signal frames, such as speech frames or noise frames. Average energy can reflect the overall energy level of speech or noise and is used to calculate indicators such as signal-to-noise ratio.

[0212] Signal-to-noise ratio (SNR) is the ratio of the average energy of a speech signal to the average energy of a noise signal, usually expressed in decibels (dB). SNR is an important indicator of speech signal quality; a higher SNR indicates less noise in the speech signal and better speech quality.

[0213] In one feasible approach, the average short-time energy of all speech frames (Espeech) and the average short-time energy of all noise frames (Enoise) are calculated separately, and then the signal-to-noise ratio (SNR) is calculated according to the definition formula of SNR.

[0214] Step S16: Based on the signal-to-noise ratio, process each audio signal frame using the corresponding audio frame processing method to obtain the audio frame sequence.

[0215] The audio frame processing method uses different methods to process audio frames based on the signal-to-noise ratio of the speech signal in order to improve the quality of the speech signal.

[0216] In one feasible approach, different signal-to-noise ratio (SNR) thresholds, SNR1 and SNR2, can be set, where SNR1 is less than SNR2. When the SNR is greater than SNR2, the speech signal quality is considered good, and the audio frame is not processed or only slightly adjusted for gain. When the SNR is between SNR1 and SNR2, a noise suppression algorithm is used to process the audio frame to reduce the impact of noise. When the SNR is less than SNR1, the speech signal quality is considered poor, and the audio frame undergoes stronger noise suppression or gain enhancement.

[0217] In another possible implementation, if the signal-to-noise ratio (SNR) is greater than or equal to a preset SNR, a Hamming window function is used to weight each audio signal frame to obtain the audio frame sequence; if the SNR is less than the preset SNR, the preprocessing parameters are adjusted, and the received speech signal is reprocessed using the adjusted preprocessing parameters to obtain the audio frame sequence. The preprocessing parameters include framing parameters, pre-emphasis coefficients, and preset noise values. Adjusting the preprocessing parameters includes: shortening the frame length of each audio signal frame and shortening the frame shift between adjacent audio signal frames; reducing the pre-emphasis coefficient of the noise frame; and increasing the first preset noise value and the second preset noise value.

[0218] In this embodiment, by preprocessing the voice signal in the above manner, the accuracy of the generated audio frame sequence is improved, thereby improving the accuracy of subsequent feature extraction. At the same time, by converting the voice signal into an audio frame sequence that can be processed and understood by subsequent models, it is ensured that subsequent feature extraction can proceed smoothly.

[0219] In other embodiments, perform a Fourier transform on the noise frame to obtain the spectrum of the noise frame; based on the spectrum of the noise frame, determine the high-frequency energy ratio of the noise frame; if the high-frequency energy ratio is greater than a preset noise ratio, adjust the pre-emphasis coefficient; use the adjusted pre-emphasis coefficient to preprocess the received voice signal again.

[0220] Exemplarily, taking the example of a user saying "It's so hot" in Cantonese with the pronunciation "houyitaa", the conversion process between the voice signal and the audio frame sequence is illustrated:

[0221] After the user says "It's so hot" in Cantonese with the pronunciation "houyitaa", an audio acquisition device such as an in-air conditioner microphone converts the analog voice signal into a digital voice signal with a sampling rate set to 16 kHz and a quantization precision of 16 bits. Subsequently, frame division processing is performed on the digital voice signal, a real-time noise detection link is added, a dual-parameter detection method based on short-time energy and zero-crossing rate is used, and the sliding average filtering algorithm is combined to smooth data fluctuations and monitor the signal-to-noise ratio of the input voice signal in real time. The specific implementation details are as Figure 5 :

[0222] (1) Short-time energy calculation.

[0223] For each audio signal frame after frame division, such as the "houyitaa" audio signal frame with a frame length of 25 ms, calculate the short-time energy through the formula where \(x n (m)\) is the amplitude of the \(m\)th sampling point in the \(n\)th frame, \(N\) is the number of sampling points per frame, and at a sampling rate of 16 kHz, a 25-ms frame contains 400 sampling points. The short-time energy can reflect the signal strength, and the energy of the voice frame is usually higher than that of the noise frame. For example, the energy of the air conditioner operation noise is relatively stable and lower than the voice burst period.

[0224] (2) Zero-crossing rate calculation.

[0225] The zero-crossing rate is calculated through the following formula:

[0226]

[0227] where \(sgn\) is the sign function, which reflects the number of times the signal crosses the zero level within a frame. The zero-crossing rate of high-frequency noise such as the high-frequency vibration of the compressor is usually higher than that of low-frequency voice such as the low-frequency tone of Cantonese "hou".

[0228] (3) Moving average filtering.

[0229] To avoid misjudgment caused by sudden changes in energy or zero-crossing rate in a single frame, a moving average filter is applied to the short-time energy and zero-crossing rate of 10 consecutive audio signal frames. The filter formula is as follows:

[0230]

[0231] The function of moving average filtering is to smooth data fluctuations, eliminate instantaneous noise interference, such as a sudden door closing sound, and make the signal-to-noise ratio (SNR) calculation more stable.

[0232] (4) Signal-to-noise ratio (SNR) estimation.

[0233] Based on the smoothed short-time energy and zero-crossing rate, the SNR is calculated using a preset noise baseline, where the noise baseline is the average energy E of the pure ambient noise collected during system initialization over a 3-second period. noise and the mean zero-crossing rate Z noise E noise Z is the first preset noise value. noise This is the second preset noise value. When and If a frame is in the specified state, it is considered a speech frame; otherwise, it is considered a noise frame. The signal-to-noise ratio (SNR) is estimated by comparing the energy ratio of speech frames to noise frames.

[0234]

[0235] in, The average energy of consecutive speech frames.

[0236] (5) Analysis of the proportion of high-frequency noise.

[0237] Perform a Fourier transform on the noisy frames to calculate the energy percentage of high-frequency components above 2kHz. When the high-frequency energy percentage is greater than 40%, it is determined to be dominated by high-frequency noise, such as the airflow noise of a high-speed fan, triggering the pre-emphasis coefficient adjustment mechanism. When local noise is low, such as an SNR higher than 10dB, the frame duration is set to 25ms, the frame shift is 10ms, and a Hamming window function is used to weight the audio of each frame to avoid signal distortion at frame boundaries. When local noise is high, such as an SNR lower than 10dB, the preprocessing parameters are dynamically adjusted according to the following rules: The adjustment process for each preprocessing parameter is as follows:

[0238] Framing parameters: shorten the frame length from 25ms to 15ms and the frame shift from 10ms to 8ms to reduce the interference of noise on speech features within a single frame;

[0239] Pre-emphasis factor: Automatically adjusted based on noise frequency characteristics. If the detected high-frequency noise energy accounts for more than 40% of the total noise energy, the pre-emphasis factor will be dynamically reduced from 0.95 to 0.9 to avoid excessive amplification of high-frequency noise.

[0240] Endpoint detection thresholds: The high threshold is increased to 1.2 times the original threshold, and the low threshold is increased to 1.1 times the original threshold. A consecutive frame determination mechanism is also introduced; only when three consecutive frames meet the threshold condition is the frame determined to be a speech frame, effectively reducing the misidentification of noise as speech. Here, the threshold can be either the first preset noise value mentioned above or the second preset noise value.

[0241] By dynamically adjusting these parameters, the preprocessing effect is comprehensively optimized, significantly improving the system's robustness in noisy environments.

[0242] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the control method of the household equipment of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0243] Based on the same inventive concept, this application provides a home appliance, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the control method of the home appliance in the above embodiments.

[0244] The home appliance provided in this application, employing the control method described in the above embodiments, can solve the technical problem of low recognition accuracy of voice signals used in current home appliance control systems, leading to poor control performance. Compared with the prior art, the beneficial effects of the home appliance provided in this application are the same as those of the control method described in the above embodiments, and other technical features of this home appliance are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0245] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0246] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0247] Based on the same inventive concept, this application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, the computer-readable program instructions being used to execute the control method of the home appliance in the above embodiments.

[0248] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0249] The aforementioned computer-readable storage medium may be included in a home appliance or may exist independently and not incorporated into a home appliance.

[0250] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a home device, enable the home device to upgrade the recognition of voice signals from pronunciation detail matching to semantic reasoning. Even if the voice signal is distorted due to pronunciation, the user's true intention can still be restored through the pinyin sequence and contextual semantic association, thereby significantly reducing the semantic reasoning error rate and improving the accuracy of home device control.

[0251] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0252] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0253] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0254] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the control method of the aforementioned home appliances. This solves the technical problem that the recognition accuracy of voice signals used to control home appliances is currently low, leading to a deterioration in the control effect of the home appliances. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the control method of the home appliances provided in the above embodiments, and will not be repeated here.

[0255] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for controlling a household appliance, characterized in that, The method includes: The received speech signal is preprocessed to obtain an audio frame sequence; The audio frame sequence is processed by a speech-semantic joint coding model to obtain a speech-semantic feature vector, and the audio frame sequence is processed by a pinyin semantic embedding pre-training model to obtain a pinyin semantic feature vector. Based on the speech semantic feature vector and the pinyin semantic feature vector, the speech signal is used to perform intent recognition to determine the control command of the home appliance; The control commands are used to control the operation of the household appliances.

2. The control method for household appliances as described in claim 1, characterized in that, The process of processing the audio frame sequence using a joint speech-semantic coding model to obtain the speech-semantic feature vector includes: The audio frame sequence is input into the speech-semantic joint coding model to extract the acoustic features of the audio frame sequence; The acoustic features are mapped to the semantic space of a pre-trained large model to obtain the speech semantic feature vector.

3. The control method for household appliances as described in claim 2, characterized in that, The step of inputting the audio frame sequence into the speech-semantic joint coding model and extracting the acoustic features of the audio frame sequence includes: Local feature extraction is performed on the audio frame sequence using the convolutional neural network of the speech-semantic joint coding model to obtain tone features; The acoustic features are extracted using a recurrent neural network of the speech-semantic joint coding model to obtain the tone variation pattern features. The tone features and the tone change patterns are determined as acoustic features in the audio frame sequence.

4. The control method for household appliances as described in claim 1, characterized in that, The process of processing the audio frame sequence using a pre-trained model for pinyin semantic embedding to obtain the pinyin semantic feature vector includes: The audio frame sequence is input into the Pinyin semantic embedding pre-trained model, and the audio frame sequence is converted into Pinyin form by the speech-to-Pinyin module of the Pinyin semantic embedding pre-trained model; The byte pair encoding module of the pre-trained model for pinyin semantic embedding splits the pinyin form into multiple sub-word units; The sub-word units are captured through the multi-layer Transformer structure of the pre-trained Pinyin semantic embedding model to obtain the Pinyin semantic feature vector.

5. The control method for a household appliance as described in any one of claims 1 to 4, characterized in that, After processing the audio frame sequence using a speech-semantic joint coding model to obtain a speech-semantic feature vector, and processing the audio frame sequence using a pinyin semantic embedding pre-trained model to obtain a pinyin semantic feature vector, the method further includes: By fusing the speech semantic feature vector and the pinyin semantic feature vector, a multimodal fusion feature vector is obtained; Accent detection is performed on the multimodal fusion feature vector using a gated routing module; Based on the accent detection results, the multimodal fusion feature vector is assigned to the accent expert module and the scene expert module. The accent expert module is configured to convert dialect pronunciation variants into standard pronunciation representations, and the scene expert module is configured to fuse environmental parameters and user behavior data to generate scene auxiliary feature vectors. The intention of the voice signal is recognized based on the multimodal fusion feature vector and the scene auxiliary feature vector to determine the control command of the home device.

6. The control method for household appliances as described in claim 5, characterized in that, The step of allocating the multimodal fusion feature vector to the accent expert module and the scene expert module based on the accent detection results includes: If an accent is present, increase the weight of the accent expert module to a first weight and decrease the weight of the scene expert module to a second weight, wherein the first weight is greater than the second weight; If there is no accent, reduce the weight of the accent expert module to the third weight and increase the weight of the scene expert module to the fourth weight, wherein the fourth weight is greater than the third weight; The multimodal fusion feature vector is assigned to the accent expert module and the scene expert module after weight adjustment.

7. The control method for household appliances as described in claim 5, characterized in that, The step of performing intent recognition on the speech signal based on the multimodal fusion feature vector and the scene auxiliary feature vector to determine the control commands for the home device includes: The multimodal fusion feature vector and the scene auxiliary feature vector are concatenated into the same dimension, and positional encoding is added to obtain a semantic feature vector representation; Intent parsing is performed on the semantic feature vector representation using a pre-trained Transformer model; Fill semantic slots based on intent parsing results; The filled semantic slots are converted into control commands for the home appliances.

8. The control method for a household appliance as described in any one of claims 1 to 4, characterized in that, The control method for the household equipment further includes: Environmental noise features are extracted from the audio frame sequence to obtain an environmental noise feature vector; Based on the speech semantic feature vector, the pinyin semantic feature vector, and the environmental noise feature vector, the intention of the speech signal is recognized to determine the control command of the home appliance.

9. The control method for household appliances as described in claim 8, characterized in that, The step of extracting environmental noise features from the audio frame sequence to obtain an environmental noise feature vector includes: Extract noise frames from the audio frame sequence; Mel spectrum analysis was performed on the noise frame to obtain the Mel spectrum; The environmental noise feature vector is obtained by extracting features from the Mel spectrum using a preset deep residual network.

10. The control method for household appliances as described in claim 1, characterized in that, The control method for the household equipment further includes: The mapping relationship between the voice signal and the control command of the home device is stored in a dynamic knowledge base, so that when the same voice signal as that in the dynamic knowledge base is received, the corresponding control command is obtained based on the mapping relationship to control the home device.

11. The control method for household appliances as described in claim 1, characterized in that, The preprocessing of the received speech signal to obtain the audio frame sequence includes: The speech signal is subjected to analog-to-digital conversion to obtain a digital speech signal; The digital audio signal is processed by framing, and the short-time energy and zero-crossing rate of each audio signal frame are determined. The short-time energy and zero-crossing rate of each audio signal frame are smoothed respectively; Based on the short-time energy and zero-crossing rate of each audio signal frame after smoothing, the speech frames and noise frames in all audio signal frames are determined. The signal-to-noise ratio of the speech signal is calculated based on the average energy of all the speech frames and the average energy of all the noise frames. Based on the signal-to-noise ratio, each audio signal frame is processed using the corresponding audio frame processing method to obtain the audio frame sequence.

12. The control method for household appliances as described in claim 11, characterized in that, The step of determining the speech frames and noise frames in all audio signal frames based on the short-time energy and zero-crossing rate after smoothing each audio signal frame includes: If the short-time energy of an audio signal frame after smoothing is greater than the first preset noise value and the zero-crossing rate after smoothing is less than the second preset noise value, the audio signal frame is determined to be a speech frame. If the short-time energy of an audio signal frame after smoothing is less than or equal to the first preset noise value, or the zero-crossing rate after smoothing is greater than or equal to the second preset noise value, the audio signal frame is determined to be a noise frame.

13. The control method for household appliances as described in claim 11, characterized in that, The step of processing each audio signal frame using a corresponding audio frame processing method based on the signal-to-noise ratio to obtain the audio frame sequence includes: If the signal-to-noise ratio is greater than or equal to the preset signal-to-noise ratio, the Hamming window function is used to weight each audio signal frame to obtain the audio frame sequence. If the signal-to-noise ratio is less than the preset signal-to-noise ratio, the preprocessing parameters are adjusted, and the received speech signal is reprocessed using the adjusted preprocessing parameters to obtain the audio frame sequence. The preprocessing parameters include frame segmentation parameters, pre-emphasis coefficients, and preset noise values.

14. The control method for a household appliance as described in claim 11, characterized in that, The control method for the household equipment further includes: Perform a Fourier transform on the noise frame to obtain the spectrum of the noise frame; Based on the spectrum of the noise frame, determine the proportion of high-frequency energy in the noise frame; If the proportion of high-frequency energy is greater than the preset noise proportion, the pre-emphasis coefficient is adjusted; The received speech signal is reprocessed using the adjusted pre-emphasis coefficient.

15. A household appliance, characterized in that, The home appliance includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the control method for the home appliance as claimed in any one of claims 1 to 14.

16. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the control method for the home appliance as described in any one of claims 1 to 14.

Images