An intelligent home interaction system and method

Abstract

This application provides a smart home interaction system and method. The system includes a terminal and a server. The terminal is configured to obtain a first smart home interaction command and its confidence level; execute the first smart home interaction command and send the first smart home interaction command and an audio signal to the server; the server is configured to convert the audio signal into text and use a large language model to infer a second smart home interaction command represented by the text; send a prohibition command to the terminal regarding the first smart home interaction command; and the terminal is further configured to suspend the use of the first smart home interaction command upon receiving the prohibition command. This provides users with low-latency, high-accuracy smart home interaction, and can reduce or even prevent infinite loops through the terminal's model self-iteration, with each user having their own exclusive audio classification model.

Description

Technical Field

[0001] This application relates to the field of edge-cloud collaboration and smart home technology, and in particular to a smart home interaction system and method. Background Technology

[0002] Users can send voice commands to terminals such as mobile phones, tablets, and smart speakers to control smart home devices. For example, a user can send the voice command "Turn on the living room TV" to a smart speaker, and the smart speaker will respond to the voice command and turn on the TV in the living room.

[0003] For a terminal to accurately respond to user voice commands, it must first recognize audio signals. One method is to deploy an audio recognition model locally on the terminal. However, limited by the terminal's hardware capabilities, only relatively simple audio recognition models can be deployed locally, resulting in low recognition accuracy. Another method is to deploy an audio recognition model in the cloud. After receiving the audio signal, the terminal sends it to the cloud for recognition. While the cloud can deploy more complex audio recognition models to improve the results, this method involves uploading the audio signal to the cloud for recognition, thus incurring higher latency. Summary of the Invention

[0004] The purpose of this application is to provide a smart home interaction system and method to offer users low-latency, highly accurate smart home interaction. The specific technical solution is as follows:

[0005] In a first aspect of this application, a smart home interaction system is provided, the system comprising:

[0006] Terminals and servers;

[0007] The terminal is equipped with an audio classification model, which is used to classify audio signals into one of a plurality of preset smart home interaction commands. The terminal is configured to, upon receiving an audio signal, use the audio classification model to classify the audio signal to obtain a first smart home interaction command and a confidence level of the first smart home interaction command; if the confidence level is greater than a preset confidence threshold and the first smart home interaction command is not suspended, execute the first smart home interaction command and send the first smart home interaction command and the audio signal to the server.

[0008] The server is deployed with a large language model and is configured to, upon receiving the first smart home interaction command and the audio signal, convert the audio signal into text and use the large language model to infer the second smart home interaction command represented by the text; if the first smart home interaction command and the second smart home interaction command are different, send a prohibition command for the first smart home interaction command to the terminal; and if a preset update condition is met, train a new audio classification model and send it to the terminal.

[0009] The terminal is also configured to, upon receiving the prohibition instruction, suspend the use of the first smart home interaction instruction; and upon receiving the new audio classification model, deploy the new audio classification model and resume the use of the first smart home interaction instruction.

[0010] In one possible embodiment, the server is configured to train a new audio classification model by:

[0011] Multiple smart home interaction commands inferred from the audio signals sent by the terminal within a historical period are statistically analyzed to obtain the frequency of occurrence of each smart home interaction command among the multiple smart home interaction commands;

[0012] From the plurality of smart home interaction commands, at least one smart home interaction command whose frequency of occurrence meets the preset high-frequency occurrence condition is determined as the high-frequency smart home interaction command.

[0013] A new audio classification model is trained to categorize audio signals into one of the aforementioned high-frequency smart home interaction commands.

[0014] In one possible embodiment, the server is configured to determine at least one smart home interaction instruction from the plurality of smart home interaction instructions whose frequency of occurrence satisfies a preset high-frequency occurrence condition, as a high-frequency smart home interaction instruction, including:

[0015] From the plurality of smart home interaction commands, the smart home interaction commands that appear in the top A% of frequency are identified as the first candidate smart home interaction commands;

[0016] From the first set of smart home interaction commands, smart home interaction commands with a frequency higher than B% are identified as second candidate smart home interaction commands; wherein, A and B are preset values ​​within the range of (0, 100);

[0017] If the number of the second candidate smart home interaction commands does not exceed a preset threshold, the second candidate smart home interaction commands will be used as high-frequency smart home interaction commands.

[0018] If the number of the second candidate smart home interaction commands exceeds the preset number threshold, then among the second candidate smart home interaction commands, the smart home interaction commands whose frequency of occurrence is at the preset number threshold position are determined as high-frequency smart home interaction commands.

[0019] In one possible embodiment, the server is configured to record the received audio signal when the first smart home interaction command differs from the second smart home interaction command; and is configured to train a new audio classification model by:

[0020] The audio signal recorded after the most recent training to obtain a new audio classification model is used as sample data to train a new audio classification model.

[0021] In one possible embodiment, the server is configured to train a new audio classification model using audio signals recorded after the most recent training of a new audio classification model as sample data, including:

[0022] The audio signal recorded after the most recent training to obtain a new audio classification model is used as the first sample data;

[0023] The first sample data is augmented using the large language model to obtain the second sample data;

[0024] A new audio classification model is trained based on the first sample data and the second sample data.

[0025] In one possible embodiment, the server is further configured to perform a quality assessment on the new audio classification model after training; if the quality assessment passes, send the data to the terminal; and if the quality assessment fails, issue an alarm.

[0026] In one possible embodiment, the server is further configured to mark the first smart home interaction instruction as exempt from verification if the first smart home interaction instruction is the same as the second smart home interaction instruction; upon receiving the first smart home interaction instruction and the audio signal, determine whether the first smart home interaction instruction is marked as exempt from verification; if not, perform the step of converting the audio signal into text and using the large language model to infer the text representation of the second smart home interaction instruction.

[0027] In one possible embodiment, the terminal is further configured to send a decision request containing the audio signal to the server when the confidence level is not greater than a preset confidence threshold or the first smart home interaction command is suspended; and to execute the smart home interaction command fed back by the server in response to the decision request.

[0028] The server is also configured to, upon receiving the decision request, convert the audio signal in the decision request into text; use the large language model to infer the third smart home interaction instruction represented by the text and send it to the terminal.

[0029] In a second aspect of this application, a smart home interaction method is provided, applied to a smart home interaction system. The system includes a terminal and a server. The terminal is equipped with an audio classification model, and the server is equipped with a large language model. The audio classification model is used to classify audio signals into one of a plurality of preset smart home interaction commands. The method includes:

[0030] The terminal uses the audio classification model to classify the received audio signals to obtain the first smart home interaction command and the confidence level of the first smart home interaction command.

[0031] When the confidence level is greater than a preset confidence threshold and the first smart home interaction command is not suspended, the terminal executes the first smart home interaction command and sends the first smart home interaction command and the audio signal to the server.

[0032] The server converts the audio signal into text and uses the large language model to infer the second smart home interaction command represented by the text.

[0033] When the first smart home interaction command and the second smart home interaction command are different, the server sends a prohibition command for the first smart home interaction command to the terminal; and when the preset update conditions are met, it trains a new audio classification model and sends it to the terminal.

[0034] The terminal suspends the use of the first smart home interaction command; and upon receiving the new audio classification model, deploys the new audio classification model and resumes the use of the first smart home interaction command.

[0035] In a third aspect of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the smart home interaction method described in the second aspect above.

[0036] This application also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the smart home interaction method described in the second aspect above.

[0037] Beneficial effects of the embodiments in this application:

[0038] The smart home interaction system and method provided in this application can first use a relatively simple audio classification model deployed on the terminal to recognize and execute the user's voice commands, thus ensuring low latency in smart home interaction. Furthermore, a more complex large language model on the server side asynchronously verifies the recognition results of the audio classification model. If the verification fails, the corresponding voice command is paused to avoid executing erroneous voice commands subsequently. Simultaneously, the server updates the audio classification model for the terminal, thereby improving the accuracy of smart home interaction. In other words, it can provide users with low-latency, highly accurate smart home interaction.

[0039] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0041] Figure 1 This application provides a schematic diagram of the structure of a smart home interaction system.

[0042] Figure 2 A flowchart illustrating a smart home interaction method provided in this application;

[0043] Figure 3 A flowchart illustrating the classification and preprocessing process of an audio classification model provided in this application;

[0044] Figure 4 A flowchart illustrating a novel audio classification model training method provided in this application;

[0045] Figure 5 A schematic diagram illustrating the principle of the iterative process of an audio classification model provided in this application;

[0046] Figure 6 A flowchart illustrating a large language model reasoning method provided in this application;

[0047] Figure 7This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0049] The smart home interaction system provided in this application embodiment is as follows: Figure 1 As shown, the system includes a terminal 100 and a server 200. The terminal 100 is equipped with an audio classification model 110, which categorizes audio signals into one of multiple preset smart home interaction commands. The terminal 100 is configured to, upon receiving an audio signal, use the audio classification model to categorize the audio signal, obtaining a first smart home interaction command and its confidence level. If the confidence level is greater than a preset confidence threshold and the first smart home interaction command is not suspended, the terminal executes the first smart home interaction command and sends the first smart home interaction command and the audio signal to the server 200. The server 200 is equipped with a large language model 210. Terminal 100 is configured to, upon receiving a first smart home interaction command and an audio signal, convert the audio signal into text and use a large language model 210 to infer a second smart home interaction command represented by the text. If the first and second smart home interaction commands differ, a prohibition command for the first smart home interaction command is sent to the terminal. If preset update conditions are met, a new audio classification model is trained and sent to terminal 100. Terminal 100 is also configured to, upon receiving a prohibition command, suspend the use of the first smart home interaction command; and, upon receiving a new audio classification model, deploy the new audio classification model and resume the use of the first smart home interaction command. Using this embodiment, a relatively simple audio classification model deployed on the terminal can be used to recognize and execute the user's voice commands, thus ensuring low latency in smart home interaction. Furthermore, a more complex large language model on the server side asynchronously verifies the recognition results of the audio classification model. If the verification fails, the corresponding voice command is paused to avoid subsequent execution of incorrect voice commands. Simultaneously, the server updates the audio classification model for the terminal, thereby improving the accuracy of smart home interaction. In other words, it can provide users with low-latency and highly accurate smart home interaction.

[0050] Furthermore, the smart home interaction system provided in this application allows the audio classification model deployed on the terminal to achieve self-iteration with the cooperation of the server, thereby enabling more accurate classification of previously misclassified smart home interaction commands. This effectively reduces or even avoids the terminal's audio classification model from continuously misclassifying some smart home commands, thus effectively reducing or even avoiding the terminal's audio classification model from falling into a vicious cycle of continuous classification errors.

[0051] Furthermore, since the audio classification model in this application is deployed on the user's terminal, the audio classification model on different users' terminals will iterate itself in a targeted manner according to the user's own usage habits, thereby enabling more accurate identification of the audio signals emitted by the respective user. In other words, each user in this application can have their own exclusive audio classification model, improving the targeting of the audio classification model.

[0052] The terminal in this application can be any terminal device that supports smart home interaction, including but not limited to mobile phones, tablets, smart speakers, etc. The server can be a physical server or a virtual server; for example, the server can be an intelligent agent deployed in the cloud.

[0053] The audio classification model in this application takes an audio signal as input and outputs a category and its confidence level, where the category consists of multiple preset smart home interaction commands. For example, the audio classification model can extract features from the input audio signal and feed the extracted features into a Softmax function. The Softmax function outputs the probabilities of the multiple preset smart home interaction commands, and the audio classification model outputs the smart home interaction command with the highest probability and its probability. These multiple preset smart home interaction commands can be preset based on user needs or obtained in advance through dynamic statistical methods. How to perform dynamic statistical analysis will be explained below and will not be repeated here. For example, the multiple preset smart home interaction commands can be: turn on the air conditioner, play a piece of music, turn off the lights, and others. Here, "others" broadly refers to smart home interaction commands other than turning on the air conditioner, playing a piece of music, and turning off the lights.

[0054] It is understandable that, since the audio classification model is used to categorize an audio signal into one of several preset smart home interaction commands, the first smart home interaction command in this paper must belong to one of these preset smart home interaction commands. The aforementioned "others" do not refer to any specific smart home interaction command, but are general terms. Therefore, the "others" smart home interaction command cannot be executed. Thus, since the "others" are included among the multiple preset smart home interaction commands, the first smart home interaction command in this application should not be one of the aforementioned "others." When the audio classification model categorizes the audio signal into "others," it will be considered equivalent to the following situation where the confidence level is not greater than the preset confidence threshold or the first smart home interaction command is suspended.

[0055] The large language model in this application is an arbitrary structure large language model, and this application does not impose any restrictions on it.

[0056] To better illustrate the smart home interaction system provided in this application, the steps performed by the terminal and server, excluding the training of the new audio classification model, will be described below. (See [link to relevant documentation]). Figure 2 ,include:

[0057] Step S11: The user sends an audio signal to the terminal;

[0058] Step S12: The terminal uses an audio classification model to classify the audio signal to obtain the first smart home interaction command and the confidence level of the first smart home interaction command.

[0059] Step S13: When the confidence level is greater than the preset confidence threshold and the first smart home interaction command is not suspended, the terminal executes the first smart home interaction command.

[0060] Step S14: The terminal sends the first smart home interaction command and audio signal to the server;

[0061] Step S15: The server converts the audio signal into text;

[0062] Step S16: The server uses a large language model to infer the second smart home interaction command in text representation;

[0063] Step S17: If the first smart home interaction command and the second smart home interaction command are different, the server sends a prohibition command to the terminal for the first smart home interaction command.

[0064] In step S18, the terminal suspends the use of the first smart home interaction command until a new audio classification model is deployed.

[0065] In step S12, the terminal can directly input the audio signal into the audio classification model for classification, or it can preprocess the audio signal before inputting the preprocessed audio signal into the audio classification model for classification. The preprocessing process can be as follows: Figure 3 As shown, the process sequentially includes converting the audio signal into single-channel data, sampling at a frequency of 16kHz, normalization, and format conversion. Furthermore, in... Figure 3 In the example shown, the classification process of the audio classification model includes: the audio classification model extracts the features of the audio signal through model inference, then uses the Softmax function to obtain the confidence of each preset smart home interaction command, and outputs the smart home interaction command with the highest confidence and its confidence.

[0066] Please watch it again. Figure 2 In step S13, each smart home interaction command is initially not disabled; the terminal will only disable the corresponding smart home interaction command upon receiving a disabling command. However, as shown in step S17, the disabling command is sent by the server when it discovers an inconsistency between the audio classification model's classification result and the large language model's inference result. Compared to the large language model, the audio classification model is a relatively simple model. Therefore, when the audio classification model's classification result is inconsistent with the large language model's inference result, it can be considered that the audio classification model has incorrectly classified the first smart home interaction command. Consequently, it can be considered that the audio classification model's classification result for the first smart home interaction command is not reliable. Without loss of generality, it can be considered that if a smart home interaction command is disabled, then the audio classification model is not reliable when it classifies the command as such. Conversely, it can be considered that if a smart home interaction command is not disabled, then the audio classification model is sufficiently reliable when it classifies the command as such.

[0067] If the confidence level is greater than the preset confidence threshold, it indicates that the audio classification model considers the first smart home interaction command sufficiently credible. Furthermore, if the first smart home interaction command is not disabled, as explained above, the audio classification model can be considered sufficiently credible at this point. Therefore, if both the confidence level is greater than the preset confidence threshold and the first smart home interaction command is not disabled, the audio classification model can be considered sufficiently credible, and since the audio classification model considers the first smart home interaction command sufficiently credible, the user's voice command can be considered the first smart home interaction command, and thus the first smart home interaction command can be executed.

[0068] The confidence threshold in this application can be set based on user experience or actual needs. For example, the confidence threshold can be 60%, 70%, or other values.

[0069] In step S14, although the voice command issued by the user can be considered as the first smart home interaction command as explained in step S13 above, its accuracy is still questionable due to the limitations of the audio classification model itself. Therefore, in this application, the first smart home interaction command and audio signal are still sent to the server for verification.

[0070] Although the classification results of the audio classification model are verified by the server, this verification process is asynchronous with the execution of the first home interaction command, and therefore does not affect the latency of smart home interaction. Furthermore, although... Figure 2 Step S14 is executed after step S13, but step S14 can also be executed simultaneously with step S13. For example, if the terminal determines that the confidence level is greater than the preset confidence threshold and the first smart home interaction command has not been suspended, the first smart home command is executed, and the first smart home interaction command and audio signal are sent to the server at the same time.

[0071] In step S15, the server can convert the audio signal into text using any speech recognition method, such as using ASR (Automatic Speech Recognition) technology.

[0072] In step S17, as explained in step S13 above, when the first smart home interaction command and the second smart home interaction command are different, the server needs to send a prohibition command to prevent the subsequent terminal device from incorrectly executing the voice command given by the user.

[0073] In step S18, as explained in step S13 above, the reason for prohibiting the use of the first smart home interaction command is the audio classification model. After the terminal updates the new audio classification model, it can be assumed that the new audio classification model can accurately classify the first smart home command. Therefore, the use of the first smart home interaction command can be restored at this time.

[0074] The following section will continue to explain how to train a new audio classification model on the server:

[0075] The timing for training a new audio classification model on the server in this application is based on meeting a preset update condition. This preset update condition can be a preset time elapsed since the last training of a new audio classification model, or it can be the number of times a first smart home interaction command differs from a second smart home interaction command after a preset number of detections. The preset time elapsed and the preset number of detections can be set based on user experience or needs, but the preset number of detections should not exceed the total number of preset smart home interaction commands. For example, the preset time elapsed could be one week, one month, or other values; the preset number of detections could be one, three, or other values.

[0076] As can be seen from the aforementioned preset update conditions, this application does not necessarily execute the aforementioned conditions every time. Figure 2 The smart home interaction method shown requires the server to train a new audio classification model. In other words, the training of the audio classification model and the smart home interaction are asynchronous.

[0077] The following will explain how the server specifically trains the new audio classification model:

[0078] In one possible embodiment, the server is configured to train a new audio classification model in the following manner: Figure 4 As shown, the method includes: Step S21, statistically analyzing multiple smart home interaction commands inferred from audio signals sent from the terminal within a historical time period to obtain the frequency of occurrence of each smart home interaction command; Step S22, identifying at least one smart home interaction command whose frequency of occurrence meets a preset high-frequency occurrence condition from the multiple smart home interaction commands, and designating it as a high-frequency smart home interaction command; Step S23, training a new audio classification model to categorize audio signals into one of the high-frequency smart home interaction commands. By selecting this embodiment, since only high-frequency smart home interaction commands are chosen, the efficiency of the audio classification model can be improved while ensuring that the audio classification model can recognize the user's voice commands as much as possible.

[0079] In step S21, the historical time period is any time period before the start of training the new audio classification model, such as one month or half a year before the start of training the new audio classification model.

[0080] In this article, the frequency of a smart home interaction command can be defined as: the number of times the smart home interaction command appears / the total number of the aforementioned smart home interaction commands. For example, assuming a total of 10,000 smart home interaction commands are counted, and 1,000 of them are "turn on the air conditioner", then the frequency of "turn on the air conditioner" could be 10%.

[0081] In step S22, the preset high-frequency occurrence condition can be set according to the user's needs or experience. For example, the preset high-frequency condition can be that the occurrence frequency is in the top A%, or that the occurrence frequency is higher than B%, or that the occurrence frequency is in the top A% and the occurrence frequency is higher than B%. Here, A and B are preset values, and their values ​​can be set according to the user's experience or needs, and the value range is within (0,100). For example, A=90, B=3, or A=85, B=5. This application does not limit this.

[0082] Understandably, audio classification models are limited by their performance, making it difficult to accurately identify all smart home interaction commands. Furthermore, the more smart home interaction commands preset for a model, the lower its accuracy and efficiency. Therefore, it is necessary to control the number of preset smart home interaction commands as much as possible.

[0083] Meanwhile, the smart home interaction commands inferred from the audio signals sent by the terminal can be considered as smart home interaction commands issued by the user during historical periods. The frequency of these smart home interaction commands reflects how often the user uses them. Therefore, if a smart home interaction command meets the high-frequency occurrence condition, it can be considered that the user uses the smart home interaction command relatively frequently, and the audio classification model needs to be able to accurately identify these smart home interaction commands. Conversely, if a smart home interaction command does not meet the high-frequency occurrence condition, it can be considered that the user uses the smart home interaction command less frequently, so even if the audio classification model cannot accurately identify these smart home interaction commands, the impact on the user experience is relatively small.

[0084] Therefore, in order to control the number of smart home interaction commands preset as much as possible, this application only presets the aforementioned high-frequency smart home interaction commands for the audio classification model. This minimizes the impact on user experience while controlling the number of commands, thus achieving a balance between user experience and the accuracy and efficiency of the audio classification model.

[0085] In one possible embodiment, in Figure 4Based on the example shown, the server is configured to determine at least one smart home interaction command from multiple smart home interaction commands whose frequency of occurrence meets a preset high-frequency occurrence condition, as a high-frequency smart home interaction command, in the following manner: determining smart home interaction commands whose frequency of occurrence is in the top A% from the multiple smart home interaction commands, as first candidate smart home interaction commands; determining smart home interaction commands whose frequency of occurrence is higher than B% from the first candidate smart home interaction commands, as second candidate smart home interaction commands; if the number of second candidate smart home interaction commands does not exceed a preset number threshold, designating the second candidate smart home interaction commands as high-frequency smart home interaction commands; if the number of second candidate smart home interaction commands exceeds the preset number threshold, determining smart home interaction commands whose frequency of occurrence is in the top preset number threshold position from the second candidate smart home interaction commands, as high-frequency smart home interaction commands. Using this embodiment, the number of high-frequency smart home interaction commands can be truncated using the preset number threshold, further limiting the number of smart home interaction commands preset for the audio classification model, thereby further improving the efficiency of the audio classification model.

[0086] In this embodiment, the preset number threshold can be set according to the user's needs or experience. For example, the preset number threshold can be 10, 8, or other values. This application does not limit this.

[0087] Understandably, if the number of second-candidate smart home interaction commands exceeds a preset threshold, then treating all second-candidate smart home interaction commands as high-frequency smart home interaction commands would result in an excessive number of smart home interaction commands being preset for the audio classification model, affecting its accuracy and efficiency. Therefore, when the number of second-candidate smart home interaction commands exceeds the preset threshold, this application further selects smart home interaction commands within the first preset threshold from the second-candidate smart home interaction commands to limit the number of smart home interaction commands preset for the audio classification model.

[0088] If the number of second-candidate smart home interaction commands does not exceed a preset threshold, then even if all second-candidate smart home interaction commands are treated as high-frequency smart home interaction commands, the accuracy and efficiency of the newly trained audio classification model will still be high enough. Therefore, all second-candidate smart home interaction commands can be treated as high-frequency smart home interaction commands. Furthermore, when the number of second-candidate smart home interaction commands is 1, it can be considered that the user has just started using the terminal and not yet enough data has been collected.

[0089] In one possible embodiment, the server is configured to record the received audio signal when the first smart home interaction command differs from the second smart home interaction command, and is configured to train a new audio classification model by using audio signals recorded after the most recent training of the new audio classification model as sample data to train the new audio classification model. Using this embodiment, the server can automatically collect audio signals that are misclassified by the audio classification model and use these audio signals to specifically train the weak points of the audio classification model, thereby further improving the accuracy of the newly trained audio classification model.

[0090] As explained above, if the first smart home interaction command and the second smart home interaction command are different, it can be assumed that the audio classification model's classification result is inaccurate. In other words, the audio classification model cannot accurately classify the audio signal received by the server this time. Therefore, it can be considered that the audio classification model lacks the knowledge to accurately classify the received audio signal. Thus, using this audio signal as sample data allows the audio classification model to learn the previously lacking knowledge during training, thereby effectively improving the accuracy of the new audio classification model obtained through training.

[0091] In one possible embodiment, the server is configured to train a new audio classification model using audio signals recorded after the most recently trained audio classification model as sample data, including: acquiring audio signals recorded after the most recently trained audio classification model as first sample data; augmenting the first sample data using a large language model to obtain second sample data; and training a new audio classification model based on the first and second sample data. Choosing this embodiment allows the use of a large language model to enrich the sample data, thereby further improving the accuracy of the trained new classification model.

[0092] Furthermore, the large language model can not only expand the sample data, but in some possible embodiments, the server can also use the large language model to clean the sample data and train a new audio classification model based on the cleaned sample data.

[0093] In one possible embodiment, the server is further configured to perform instruction evaluation on the new audio classification model after it has been trained; if the quality evaluation passes, the aforementioned steps of sending to the terminal are executed; if the quality evaluation fails, an alarm is issued. Using this embodiment, the quality of the newly trained audio classification model can be automatically evaluated, and the user is promptly alerted to intervene if the evaluation fails.

[0094] Understandably, during the training of a new audio classification model, the server can only change the parameters of the audio classification model, but not its hyperparameters, such as the model architecture. Therefore, the performance of the newly trained audio classification model will still be limited. If the quality assessment fails to meet the standards, it can be assumed that the current hyperparameters are insufficient to train a high-quality audio classification model. In this case, an alert should be issued to remind the user to manually upgrade the hyperparameters, such as the audio classification model's architecture.

[0095] The above sections have already described the methods for training the new audio classification model, the sample data used, and the quality evaluation after training. In one possible embodiment, the foregoing embodiments can be combined. For example, the method for training the new audio classification model can be as follows: Figure 5 As shown.

[0096] Figure 5 In the example shown, the device is the aforementioned terminal, the cloud is the aforementioned server, and the statistical identification of intent errors means that when the first smart home interaction command and the second smart home interaction command are different, the server records the received audio signal as sample data. LLM data augmentation means that the first sample data is augmented using a large language model to obtain the second sample data. Online data collection means that the server further supplements the sample data through online collection. Figure 5 The sample data used in the example shown includes sample data obtained by statistically identifying intentional errors, sample data obtained by LLM data augmentation, and sample data obtained by online data collection.

[0097] Figure 5 In the example shown, cloud-based model cleaning refers to the server using a large language model to clean the sample data. Edge-side model training refers to the server training a new audio classification model based on the cleaned sample data. Edge-side model evaluation refers to the server evaluating the new audio classification model after training. Manual intervention refers to issuing an alert if the quality evaluation fails, and edge-side model delivery refers to sending the model to the terminal if the quality evaluation passes. Test set expansion refers to adding sample data that the new audio classification model misidentified during the quality evaluation to the test set to expand it.

[0098] In one possible embodiment, the server is further configured to mark the first smart home interaction command as exempt from verification if the first smart home interaction command is identical to the second smart home interaction command; upon receiving the first smart home interaction command and an audio signal, determine whether the first smart home interaction command is marked as exempt from verification; if not, perform the aforementioned step of converting the audio signal into text. Using this embodiment, for each smart home interaction command, the server only needs to perform verification the first time, effectively reducing the server's computational load.

[0099] It is understandable that, similar to step S13 above, if the first smart home interaction command and the second smart home interaction command are the same, the audio classification model can be considered sufficiently reliable when the audio classification model classifies the command as the first smart home interaction command. Therefore, the server does not need to verify the classification result when the audio classification model classifies the command as the first smart home interaction command in subsequent instances. Thus, in this application, when the first smart home interaction command and the second smart home interaction command are the same, the server marks the first smart home interaction command as exempt from verification to avoid unnecessary repeated verification in the future.

[0100] In one possible embodiment, the terminal is further configured to send a decision request containing an audio signal to the server when the confidence level is not greater than a preset confidence threshold or when the first smart home interaction command is suspended, and to execute the smart home interaction command responded to by the server in response to the decision request. The server is further configured to, upon receiving the decision request, convert the audio signal in the decision request into text, use a large language model to infer a text representation of a third smart home interaction command, and send it to the terminal. By using this embodiment, when the terminal finds that it cannot accurately recognize the audio signal, it can rely on the server to complete the recognition, thereby improving accuracy.

[0101] In any of the examples shown above, the reasoning process of a large language model can be as follows: Figure 6 As shown, the text is analyzed by its internal NLU (Natural Language Understanding) module. In this embodiment, the NLU module used is JointBERT (Joint Bidirectional Encoder Representations from Transformers) to obtain the intent and slot, and normalize the two to obtain smart home interaction commands, such as turn on - living room - air conditioner.

[0102] Corresponding to the aforementioned smart home interaction system, this application also provides a smart home interaction method applied to the smart home interaction system. The system includes a terminal and a server. The terminal is equipped with an audio classification model, and the server is equipped with a large language model. The audio classification model is used to classify audio signals into one of a plurality of preset smart home interaction commands. The method includes:

[0103] The terminal uses the audio classification model to classify the received audio signals to obtain the first smart home interaction command and the confidence level of the first smart home interaction command.

[0104] When the confidence level is greater than a preset confidence threshold and the first smart home interaction command is not suspended, the terminal executes the first smart home interaction command and sends the first smart home interaction command and the audio signal to the server.

[0105] The server converts the audio signal into text and uses the large language model to infer the second smart home interaction command represented by the text.

[0106] When the first smart home interaction command and the second smart home interaction command are different, the server sends a prohibition command for the first smart home interaction command to the terminal; and when the preset update conditions are met, it trains a new audio classification model and sends it to the terminal.

[0107] The terminal suspends the use of the first smart home interaction command; and upon receiving the new audio classification model, deploys the new audio classification model and resumes the use of the first smart home interaction command.

[0108] In one possible embodiment, the server trains a new audio classification model, including:

[0109] Multiple smart home interaction commands inferred from the audio signals sent by the terminal within a historical period are statistically analyzed to obtain the frequency of occurrence of each smart home interaction command among the multiple smart home interaction commands;

[0110] From the plurality of smart home interaction commands, at least one smart home interaction command whose frequency of occurrence meets the preset high-frequency occurrence condition is determined as the high-frequency smart home interaction command.

[0111] A new audio classification model is trained to categorize audio signals into one of the aforementioned high-frequency smart home interaction commands.

[0112] In one possible embodiment, the server determines at least one smart home interaction instruction from the plurality of smart home interaction instructions whose frequency of occurrence satisfies a preset high-frequency occurrence condition, as a high-frequency smart home interaction instruction, including:

[0113] From the plurality of smart home interaction commands, the smart home interaction commands that appear in the top A% of frequency are identified as the first candidate smart home interaction commands;

[0114] From the first set of smart home interaction commands, smart home interaction commands with a frequency higher than B% are identified as second candidate smart home interaction commands; wherein, A and B are preset values ​​within the range of (0, 100);

[0115] If the number of the second candidate smart home interaction commands does not exceed a preset threshold, the second candidate smart home interaction commands will be used as high-frequency smart home interaction commands.

[0116] If the number of the second candidate smart home interaction commands exceeds the preset number threshold, then among the second candidate smart home interaction commands, the smart home interaction commands whose frequency of occurrence is at the preset number threshold position are determined as high-frequency smart home interaction commands.

[0117] In one possible embodiment, the method further includes: the server recording the received audio signal when the first smart home interaction command is different from the second smart home interaction command;

[0118] The server trains a new audio classification model, including:

[0119] The audio signal recorded after the most recent training to obtain a new audio classification model is used as sample data to train a new audio classification model.

[0120] In one possible embodiment, the server uses the audio signal recorded after the most recent training of a new audio classification model as sample data to train a new audio classification model, including:

[0121] The audio signal recorded after the most recent training to obtain a new audio classification model is used as the first sample data;

[0122] The first sample data is augmented using the large language model to obtain the second sample data;

[0123] A new audio classification model is trained based on the first sample data and the second sample data.

[0124] In one possible embodiment, the method further includes: after the server trains a new audio classification model, it performs a quality assessment on the new audio classification model; if the quality assessment passes, it sends the data to the terminal; if the quality assessment fails, it issues an alarm.

[0125] In one possible embodiment, the method further includes: when the first smart home interaction instruction and the second smart home interaction instruction are the same, the server marks the first smart home interaction instruction as exempt from verification; when the first smart home interaction instruction and the audio signal are received, the server determines whether the first smart home interaction instruction is marked as exempt from verification; if not, the server performs the step of converting the audio signal into text and using the large language model to infer the text representation of the second smart home interaction instruction.

[0126] In one possible embodiment, the method further includes:

[0127] When the confidence level is not greater than a preset confidence threshold or the first smart home interaction command is suspended, the terminal sends a decision request containing the audio signal to the server; and executes the smart home interaction command fed back by the server in response to the decision request.

[0128] Upon receiving the decision request, the server converts the audio signal in the decision request into text; it then uses the large language model to infer the third smart home interaction command represented by the text and sends it to the terminal.

[0129] This application also provides an electronic device, such as... Figure 7 As shown, it includes:

[0130] Memory 701 is used to store computer programs;

[0131] When the processor 702 executes the program stored in the memory 701, it performs the steps executed by the terminal or server as follows:

[0132] The terminal uses the audio classification model to classify the received audio signals to obtain the first smart home interaction command and the confidence level of the first smart home interaction command.

[0133] When the confidence level is greater than a preset confidence threshold and the first smart home interaction command is not suspended, the terminal executes the first smart home interaction command and sends the first smart home interaction command and the audio signal to the server.

[0134] The server converts the audio signal into text and uses the large language model to infer the second smart home interaction command represented by the text.

[0135] When the first smart home interaction command and the second smart home interaction command are different, the server sends a prohibition command for the first smart home interaction command to the terminal; and when the preset update conditions are met, it trains a new audio classification model and sends it to the terminal.

[0136] The terminal suspends the use of the first smart home interaction command; and upon receiving the new audio classification model, deploys the new audio classification model and resumes the use of the first smart home interaction command.

[0137] In one possible embodiment, the server trains a new audio classification model, including:

[0138] Multiple smart home interaction commands inferred from the audio signals sent by the terminal within a historical period are statistically analyzed to obtain the frequency of occurrence of each smart home interaction command among the multiple smart home interaction commands;

[0139] From the plurality of smart home interaction commands, at least one smart home interaction command whose frequency of occurrence meets the preset high-frequency occurrence condition is determined as the high-frequency smart home interaction command.

[0140] A new audio classification model is trained to categorize audio signals into one of the aforementioned high-frequency smart home interaction commands.

[0141] In one possible embodiment, the server determines at least one smart home interaction instruction from the plurality of smart home interaction instructions whose frequency of occurrence satisfies a preset high-frequency occurrence condition, as a high-frequency smart home interaction instruction, including:

[0142] From the plurality of smart home interaction commands, the smart home interaction commands that appear in the top A% of frequency are identified as the first candidate smart home interaction commands;

[0143] From the first set of smart home interaction commands, smart home interaction commands with a frequency higher than B% are identified as second candidate smart home interaction commands; wherein, A and B are preset values ​​within the range of (0, 100);

[0144] If the number of the second candidate smart home interaction commands does not exceed a preset threshold, the second candidate smart home interaction commands will be used as high-frequency smart home interaction commands.

[0145] If the number of the second candidate smart home interaction commands exceeds the preset number threshold, then among the second candidate smart home interaction commands, the smart home interaction commands whose frequency of occurrence is at the preset number threshold position are determined as high-frequency smart home interaction commands.

[0146] In one possible embodiment, the method further includes: the server recording the received audio signal when the first smart home interaction command is different from the second smart home interaction command;

[0147] The server trains a new audio classification model, including:

[0148] The audio signal recorded after the most recent training to obtain a new audio classification model is used as sample data to train a new audio classification model.

[0149] In one possible embodiment, the server uses the audio signal recorded after the most recent training of a new audio classification model as sample data to train a new audio classification model, including:

[0150] The audio signal recorded after the most recent training to obtain a new audio classification model is used as the first sample data;

[0151] The first sample data is augmented using the large language model to obtain the second sample data;

[0152] A new audio classification model is trained based on the first sample data and the second sample data.

[0153] In one possible embodiment, the method further includes: after the server trains a new audio classification model, it performs a quality assessment on the new audio classification model; if the quality assessment passes, it sends the data to the terminal; if the quality assessment fails, it issues an alarm.

[0154] In one possible embodiment, the method further includes: when the first smart home interaction instruction and the second smart home interaction instruction are the same, the server marks the first smart home interaction instruction as exempt from verification; when the first smart home interaction instruction and the audio signal are received, the server determines whether the first smart home interaction instruction is marked as exempt from verification; if not, the server performs the step of converting the audio signal into text and using the large language model to infer the text representation of the second smart home interaction instruction.

[0155] In one possible embodiment, the method further includes:

[0156] When the confidence level is not greater than a preset confidence threshold or the first smart home interaction command is suspended, the terminal sends a decision request containing the audio signal to the server; and executes the smart home interaction command fed back by the server in response to the decision request.

[0157] Upon receiving the decision request, the server converts the audio signal in the decision request into text; it then uses the large language model to infer the third smart home interaction command represented by the text and sends it to the terminal.

[0158] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 702, communication interface, and memory 701 communicating with each other via the communication bus.

[0159] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0160] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0161] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0162] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0163] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described smart home interaction methods.

[0164] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the smart home interaction methods described in the above embodiments.

[0165] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0166] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0167] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, embodiments of methods, electronic devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0168] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A smart home interaction system, characterized in that, The system includes: Terminals and servers; The terminal is equipped with an audio classification model, which is used to classify audio signals into one of a plurality of preset smart home interaction commands. The terminal is configured to, upon receiving an audio signal, use the audio classification model to classify the audio signal to obtain a first smart home interaction command and a confidence level of the first smart home interaction command; if the confidence level is greater than a preset confidence threshold and the first smart home interaction command is not suspended, execute the first smart home interaction command and send the first smart home interaction command and the audio signal to the server. The server is deployed with a large language model. The server is configured to, upon receiving the first smart home interaction command and the audio signal, convert the audio signal into text and use the large language model to infer the second smart home interaction command represented by the text; if the first smart home interaction command and the second smart home interaction command are different, send a prohibition command for the first smart home interaction command to the terminal; and if a preset update condition is met, train a new audio classification model and send it to the terminal. The terminal is also configured to, upon receiving the prohibition instruction, suspend the use of the first smart home interaction instruction; and upon receiving the new audio classification model, deploy the new audio classification model and resume the use of the first smart home interaction instruction.

2. The system according to claim 1, characterized in that, The server is configured to train a new audio classification model in the following ways: Multiple smart home interaction commands inferred from the audio signals sent by the terminal within a historical period are statistically analyzed to obtain the frequency of occurrence of each smart home interaction command among the multiple smart home interaction commands; From the plurality of smart home interaction commands, at least one smart home interaction command whose frequency of occurrence meets the preset high-frequency occurrence condition is determined as the high-frequency smart home interaction command. A new audio classification model is trained to categorize audio signals into one of the aforementioned high-frequency smart home interaction commands.

3. The system according to claim 2, characterized in that, The server is configured to determine at least one smart home interaction command from the plurality of smart home interaction commands whose frequency of occurrence satisfies a preset high-frequency occurrence condition, as a high-frequency smart home interaction command, including: From the plurality of smart home interaction commands, the smart home interaction commands that appear in the top A% of frequency are identified as the first candidate smart home interaction commands; From the first set of smart home interaction commands, smart home interaction commands with a frequency higher than B% are identified as second candidate smart home interaction commands; wherein, A and B are preset values ​​within the range of (0, 100); If the number of the second candidate smart home interaction commands does not exceed a preset threshold, the second candidate smart home interaction commands will be used as high-frequency smart home interaction commands. If the number of the second candidate smart home interaction commands exceeds the preset number threshold, then among the second candidate smart home interaction commands, the smart home interaction commands whose frequency of occurrence is at the preset number threshold position are determined as high-frequency smart home interaction commands.

4. The system according to claim 1, characterized in that, The server is configured to record the received audio signal when the first smart home interaction command is different from the second smart home interaction command; And it is configured to train a new audio classification model in the following ways, including: The audio signal recorded after the most recent training to obtain a new audio classification model is used as sample data to train a new audio classification model.

5. The system according to claim 4, characterized in that, The server is configured to train a new audio classification model using audio signals recorded after the most recent training of a new audio classification model as sample data, including: The audio signal recorded after the most recent training to obtain a new audio classification model is used as the first sample data; The first sample data is augmented using the large language model to obtain the second sample data; A new audio classification model is trained based on the first sample data and the second sample data.

6. The system according to claim 1, characterized in that, The server is also configured to perform a quality assessment on the new audio classification model after it has been trained; to send the model to the terminal if the quality assessment is passed; and to issue an alarm if the quality assessment fails.

7. The system according to claim 1, characterized in that, The server is further configured to mark the first smart home interaction instruction as exempt from verification if the first smart home interaction instruction is the same as the second smart home interaction instruction; to determine whether the first smart home interaction instruction is marked as exempt from verification upon receiving the first smart home interaction instruction and the audio signal; and to perform the step of converting the audio signal into text and using the large language model to infer the text representation of the second smart home interaction instruction.

8. The system according to claim 1, characterized in that, The terminal is also configured to send a decision request containing the audio signal to the server when the confidence level is greater than a preset confidence threshold or the first smart home interaction command is suspended; and to execute the smart home interaction command fed back by the server in response to the decision request. The server is also configured to, upon receiving the decision request, convert the audio signal in the decision request into text; use the large language model to infer the third smart home interaction instruction represented by the text and send it to the terminal.

9. A smart home interaction method, characterized in that, An application is made in a smart home interaction system, the system comprising a terminal and a server, the terminal deploying an audio classification model, and the server deploying a large language model, the audio classification model being used to categorize audio signals into one of multiple preset smart home interaction commands, the method comprising: The terminal uses the audio classification model to classify the received audio signals to obtain the first smart home interaction command and the confidence level of the first smart home interaction command. When the confidence level is greater than a preset confidence threshold and the first smart home interaction command is not suspended, the terminal executes the first smart home interaction command and sends the first smart home interaction command and the audio signal to the server. The server converts the audio signal into text and uses the large language model to infer the second smart home interaction command represented by the text. When the first smart home interaction command and the second smart home interaction command are different, the server sends a prohibition command for the first smart home interaction command to the terminal; and when the preset update conditions are met, it trains a new audio classification model and sends it to the terminal. The terminal suspends the use of the first smart home interaction command; and upon receiving the new audio classification model, deploys the new audio classification model and resumes the use of the first smart home interaction command.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of claim 9.

Images