Smart home interaction system and method thereof

Abstract

The invention provides a smart home interaction system and method, and the system comprises a digital human interaction screen which is used for an interaction interface between a user and a smart home; the model establishing module is used for establishing a large language module and analyzing and understanding a user instruction; the identity recognition module is used for recognizing and confirming the identity of the user and detecting the action that the user approaches the digital human interaction screen; the voice recognition module is used for capturing a voice command of a user and converting the voice command into a text form to be analyzed and understood by a large language model; the executor module is used for executing a user instruction analyzed by the large language model; after the infrared sensor senses the illumination signal change and the signal duration reaches the set threshold value, the camera is opened through the actuator module to authenticate the user identity, the training method of the infrared sensor is optimized, the accuracy and response speed of user approach detection are improved, the situations of false alarm and missing alarm are reduced, and the user experience is improved. In the interaction process, user identity authentication needs to be carried out firstly, so that smart home electric appliances are prevented from being turned on randomly.

Description

Technical Field

[0001] The present invention relates to the field of smart home technology, and in particular to a smart home interaction system and method thereof. Background Art

[0002] With the rapid development of mobile smart terminals, home appliances are becoming more and more intelligent. The smart home system uses residential space as a platform and integrates home life-related equipment using integrated wiring technology, network communication technology, security technology, automatic control technology, audio and video technology, etc., to build an efficient management system for residential facilities and family schedule affairs, improve the safety, convenience, comfort, and artistry of home appliances, and achieve an environmentally friendly and energy-saving living environment.

[0003] Infrared sensors are a commonly used sensor technology in smart home interactions. Their working principle is to determine whether an object is approaching by detecting the infrared radiation emitted by the object. Although infrared sensors can work effectively in many situations, there are still some problems in actual smart home interaction applications: infrared sensors may produce false alarms due to changes in ambient light, causing unnecessary operations to be triggered. In addition, identity recognition is also very important during the interaction process. If identity recognition is not performed, it is easy for smart home appliances to be opened at will by anyone. How to use it to improve the safety factor of the smart home environment is one of the problems that need to be solved in the field of smart home technology today.

[0004] Therefore, it is necessary to provide a new smart home interaction system and method thereof to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a smart home interaction system and method thereof.

[0006] The present invention provides a smart home interactive system in one aspect, comprising a digital human interactive screen, which is used as an interactive interface between a user and a smart home;

[0007] Model building module, used to build a large language module and analyze and understand user instructions;

[0008] The identity recognition module is used to identify and confirm the user's identity and detect the user's action of approaching the digital human interactive screen. When the user approaches the interactive screen, the digital human responds;

[0009] The speech recognition module is used to capture the user's voice commands and convert them into text form so that the large language model can analyze and understand the instructions;

[0010] The executor module is used to execute user instructions for large language model analysis;

[0011] The identity recognition module includes an infrared sensor and a camera. When the infrared sensor senses that the change in the light signal and the signal duration have reached a set threshold, the camera is turned on through the actuator module to authenticate the user's identity.

[0012] Another aspect of the present invention provides an interaction method applied to the above-mentioned smart home interaction system, the method comprising:

[0013] S1, System initialization: deploy large language model, speech recognition engine, and face recognition algorithm software components;

[0014] S2, identification and processing: The infrared sensor detects the change of the light signal and the duration of the signal between the user and the interactive screen in real time. When the change of the light signal and the duration of the signal reach the preset threshold, the camera face recognition is started to confirm the user's identity;

[0015] S3, user interaction: When the user issues a voice command, the voice recognition module captures the sound signal, converts the voice signal into text, the large language model parses the text, and then converts the text generated by the large language model into voice output, and displays the text information on the digital human interaction screen waiting for operation confirmation;

[0016] S4, output processing: The actuator module controls the smart home devices to perform corresponding operations according to the user's instructions.

[0017] Furthermore, the infrared sensor needs to be optimized and trained before being used for detection. The optimization and training of the infrared sensor includes the following steps: S301, data collection: placing the infrared sensor in different environments and recording the data generated in various situations;

[0018] S302, feature extraction: extracting light signal features from the collected data, wherein the light signal features include real activity data under different light conditions;

[0019] S303, model training: using the extracted features to train a classification model for identifying infrared features of different objects.

[0020] Furthermore, the face recognition algorithm includes:

[0021] S401, using a camera to obtain a video stream, and processing each frame of the image, wherein the video stream contains face images under various lighting conditions;

[0022] S402, scaling, cropping and normalizing the face image;

[0023] S403, inputting the preprocessed image into the ResNet-50 model to optimize the model;

[0024] S404: Apply the optimized model to capture the facial features of the person in front of the digital human interactive screen in real time and obtain the model authentication result.

[0025] Furthermore, the deployment of the large language model includes integrating and deploying a local large language model using LangChain, and then integrating a data source, wherein the data source includes a vector database based on domain text.

[0026] Furthermore, the facial images in the video stream cover different lighting conditions and expression features.

[0027] Furthermore, the speech recognition engine deployment includes the following steps:

[0028] S601, calling the microphone device through the API to record the user's voice in real time;

[0029] S602, audio preprocessing: using a spectral subtraction noise suppression algorithm to remove background noise, determine the start and end time points of the speech segment, and extract useful features from the audio signal;

[0030] S603, converting the recognized speech into text form according to the Google Cloud Speech-toAPI speech recognition engine;

[0031] S604: Text parsing: using a large language model to perform text parsing.

[0032] Furthermore, displaying text information on the digital human interaction screen to wait for operation confirmation includes verbal confirmation, using voice recognition to identify the user's voice instructions.

[0033] Furthermore, displaying text information on the digital human interaction screen to wait for operation confirmation also includes clicking to confirm, and adding a confirmation button on the digital human interaction screen interface for the user to click.

[0034] Compared with the related art, the smart home interaction system and method provided by the present invention have the following beneficial effects:

[0035] 1. The present invention achieves a more natural and smooth user interaction experience by combining the digital human interaction screen with a large language model. By combining the function of capturing user voice commands, a more personalized service experience can be provided, such as automatically adjusting the home environment according to the user's preferences, thereby improving the usability of the smart home system.

[0036] 2. The present invention turns on the camera to authenticate the user's identity through the actuator module after the infrared sensor senses the change in the light signal and the signal duration reaches a set threshold, optimizes the training method of the infrared sensor, improves the accuracy and response speed of user approach detection, reduces false alarms and missed alarms, and requires user identity authentication during the interaction process to prevent smart home appliances from being opened at will. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A structural block diagram of the smart home interaction system provided by the present invention;

[0038] Figure 2 A flowchart of the smart home interaction method provided by the present invention;

[0039] Figure 3 A flowchart of the optimization training of the infrared sensor provided by the present invention;

[0040] Figure 4 A flowchart of the face recognition algorithm provided by the present invention;

[0041] Figure 5 A flowchart of the speech recognition engine deployment provided by the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0043] Please refer to Figure 1 , Figure 2 , Figure 3 , Figure 4 as well as Figure 5 ,in, Figure 1 A structural block diagram of the smart home interaction system provided by the present invention; Figure 2 A flowchart of the smart home interaction method provided by the present invention; Figure 3 A flowchart of the optimization training of the infrared sensor provided by the present invention; Figure 4 A flowchart of the face recognition algorithm provided by the present invention; Figure 5 A flowchart of the speech recognition engine deployment provided by the present invention.

[0044] Embodiment 1

[0045] In the specific implementation process, refer to Figure 1 As shown, the smart home interaction system provided by the present invention on one hand includes a digital human interaction screen, which is used as an interaction interface between a user and a smart home;

[0046] The model building module is used to build a large language module and analyze and understand user instructions. The deployment of the large language model includes integrating and building a local large language model using LangChain, and then integrating data sources, where the data source includes a vector database based on domain text;

[0047] The identity recognition module is used to identify and confirm the user's identity and detect the user's action of approaching the digital human interactive screen. When the user approaches the interactive screen, the digital human responds;

[0048] The speech recognition module is used to capture the user's voice commands and convert them into text form so that the large language model can analyze and understand the instructions;

[0049] The executor module is used to execute user instructions for large language model analysis;

[0050] The identity recognition module includes an infrared sensor and a camera. When the infrared sensor senses that the light signal change and the signal duration reach a set threshold, the actuator module turns on the camera to authenticate the user's identity.

[0051] Embodiment 2

[0052] In the specific implementation process, refer to Figure 2-Figure 5 As shown, another aspect of the present invention provides an interaction method applied to the above-mentioned smart home interaction system, the method comprising:

[0053] S1. System initialization: deploy the large language model, speech recognition engine, and face recognition algorithm software components. It should be noted that the deployment of the large language model includes integrating and deploying the local large language model using LangChain, and then integrating the data source, where the data source includes a vector database based on domain text;

[0054] S2, identification and processing: The infrared sensor detects the change of the light signal and the duration of the signal between the user and the interactive screen in real time. When the change of the light signal and the duration of the signal reach the preset threshold, the camera face recognition is started to confirm the user's identity;

[0055] S3, user interaction: when the user issues a voice command, the voice recognition module captures the sound signal, converts the voice signal into text, the large language model parses the text, and then converts the text generated by the large language model into voice output, and displays the text information on the digital human interaction screen waiting for operation confirmation, displays the text information on the digital human interaction screen waiting for operation confirmation including verbal confirmation, and uses voice recognition to recognize the user's voice command;

[0056] S4, output processing: The actuator module controls the smart home devices to perform corresponding operations according to the user's instructions.

[0057] It should be noted that the deployment of the speech recognition engine includes the following steps:

[0058] S601, calling the microphone device through the API to record the user's voice in real time;

[0059] S602, audio preprocessing: using a spectral subtraction noise suppression algorithm to remove background noise, determine the start and end time points of the speech segment, and extract useful features from the audio signal;

[0060] S603, converting the recognized speech into text form according to the Google Cloud Speech-toAPI speech recognition engine;

[0061] S604: Text parsing: using a large language model to perform text parsing.

[0062] Spectral subtraction:

[0063]

[0064] Among them, s{n} is the original speech signal, is the estimated noise signal, α is the subtraction coefficient, is the signal after noise reduction;

[0065] Extract useful features from audio signals:

[0066]

[0067] Where E[n] is the energy of the signal, θ is the threshold value, and T[n] is a binary flag indicating whether there is speech;

[0068] For example: assuming that the signal energy E[n] = [100, 150, 200, 100, 50, 100, 150, 200, 100, 50] and the threshold value θ = 120; then: T[n] = [0, 1, 1, 0, 0, 0, 1, 1, 0, 0].

[0069] Audio signal feature extraction using: Mel frequency cepstrum coefficients

[0070]

[0071] Where s|n| is the preprocessed speech signal, F{s|n|} is the Fourier transform of the signal, N is the number of discrete cosine transform points, k is the index of MFCC, and MFCC[k] is the kth MFCC coefficient.

[0072] It should be noted that the infrared sensor needs to be optimized and trained before it is used for detection. The optimization training of the infrared sensor includes the following steps:

[0073] S301, data collection: placing infrared sensors in different environments and recording the data generated in various situations;

[0074] S302, feature extraction: extracting light signal features from the collected data, wherein the light signal features include real activity data under different light conditions;

[0075] S303, model training: using the extracted features to train a classification model for identifying infrared features of different objects.

[0076] It should be noted that the face recognition algorithm includes:

[0077] S401, using a camera to obtain a video stream, and processing each frame of the image, wherein the video stream contains facial images under various lighting conditions, and the facial images cover different lighting conditions and expression features;

[0078] S402, scaling, cropping and normalizing the face image;

[0079] S403, inputting the preprocessed image into the ResNet-50 model to optimize the model;

[0080] S404: Apply the optimized model to capture the facial features of the person in front of the digital human interactive screen in real time and obtain the model authentication result.

[0081] Face recognition problem definition:

[0082] Task: Classification problem, distinguish the faces of two different people.

[0083] Dataset: Suppose we have a very small dataset containing 100 28x28 pixel grayscale images of two types of people.

[0084] ResNet-50 model architecture:

[0085] Input layer: The input image size is 28x28 pixels.

[0086] Convolutional layer: Use 32 5x5 convolution kernels with a stride of 1 and padding of "same" to keep the output size unchanged.

[0087] Activation layer: ReLU activation function.

[0088] Pooling layer: Use a 2x2 max pooling layer with a stride of 2 to reduce the spatial dimension.

[0089] Fully connected layer: A hidden layer with 128 nodes and ReLU activation function.

[0090] Output layer: The output layer has 2 nodes, Softmax Activation function.

[0091] Specific steps:

[0092] Step 1: Convolutional Layer

[0093] For an input image I, the convolution operation is:

[0094]

[0095] Among them, F is the size of the filter (here is 5), W is the weight matrix, b is the bias term, ij represents the position of the output feature map, and I i+m，j+n Represents the pixel value of the input image at position i+m, j+n, O ij is the value of the output feature map at position ij;

[0096] Step 2: ReLU Activation

[0097] The ReLU function is defined as:

[0098] f(x) = max(0, x)

[0099] Step 3: Pooling Layer

[0100] The maximum pooling function can be described as:

[0101] P ij =max m,n (O i+m,j+n )

[0102] Among them, m, n are changes within the pooling window (here is 2x2);

[0103] Step 4: Fully connected layer

[0104] The forward propagation calculation of the fully connected layer can be described as:

[0105] H=σ(W h O p +b h )

[0106] Among them, O p is the flattened vector of the pooled feature map, W h and b h are the weight matrix and bias vector, σ is the activation function (ReLU in this case), and H is the output of the hidden layer;

[0107] Step 5: Output Layer

[0108] The forward propagation calculation of the output layer can be described as:

[0109] Y = Softmax(Wy H+b y )

[0110] Among them, W y and b y are the weight matrix and bias vector of the output layer, and Y is the final prediction probability vector.

[0111] Embodiment 3

[0112] During the specific implementation process, in order to avoid the situation where the user does not hear the prompt information when confirming verbally, a text message is displayed on the digital human interaction screen waiting for operation confirmation. The verbal confirmation also includes click confirmation, and a confirmation button is added on the digital human interaction screen interface for the user to click.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any figure mark in the claims should not be regarded as limiting the claims involved.

[0115] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A smart home interactive system, characterized in that: Includes a digital human interactive screen, which is used as the interactive interface between the user and the smart home; Model building module, used to build a large language module and analyze and understand user instructions; The identity recognition module is used to identify and confirm the user's identity and detect the user's action of approaching the digital human interactive screen. When the user approaches the interactive screen, the digital human responds; The speech recognition module is used to capture the user's voice commands and convert them into text form so that the large language model can analyze and understand the instructions; The executor module is used to execute user instructions for large language model analysis; The identity recognition module includes an infrared sensor and a camera. When the infrared sensor senses that the change in the light signal and the signal duration have reached a set threshold, the camera is turned on through the actuator module to authenticate the user's identity.

2. An interactive method applied to the smart home interactive system described in right 1, characterized in that: The method comprises: S1, System initialization: deploy large language model, speech recognition engine, and face recognition algorithm software components; S2, identification and processing: The infrared sensor detects the change of the light signal and the duration of the signal between the user and the interactive screen in real time. When the change of the light signal and the duration of the signal reach the preset threshold, the camera face recognition is started to confirm the user's identity; S3, user interaction: When the user issues a voice command, the voice recognition module captures the sound signal, converts the voice signal into text, the large language model parses the text, and then converts the text generated by the large language model into voice output, and displays the text information on the digital human interaction screen waiting for operation confirmation; S4, output processing: The actuator module controls the smart home devices to perform corresponding operations according to the user's instructions.

3. The smart home interaction method according to claim 2, characterized in that: Before using the infrared sensor for detection, it needs to be optimized and trained. The optimization training of the infrared sensor includes the following steps: S301, data collection: placing the infrared sensor in different environments and recording the data generated in various situations; S302, feature extraction: extracting light signal features from the collected data, wherein the light signal features include real activity data under different light conditions; S303, model training: using the extracted features to train a classification model for identifying infrared features of different objects.

4. The smart home interaction method according to claim 3, characterized in that: The face recognition algorithm includes: S401, using a camera to obtain a video stream, and processing each frame of the image, wherein the video stream contains face images under various lighting conditions; S402, scaling, cropping and normalizing the face image; S403, inputting the preprocessed image into the ResNet-50 model to optimize the model; S404: Apply the optimized model to capture the facial features of the person in front of the digital human interactive screen in real time and obtain the model authentication result.

5. The smart home interaction method according to claim 4, characterized in that: The deployment of the large language model includes integrating and deploying a local large language model using LangChain, and then integrating a data source, wherein the data source includes a vector database based on domain text.

6. The smart home interaction method according to claim 4, characterized in that: The face images in the video stream cover different lighting conditions and expression features.

7. The smart home interaction method according to claim 5, characterized in that: The speech recognition engine deployment includes the following steps: S601, calling the microphone device through the API to record the user's voice in real time; S602, audio preprocessing: using a spectral subtraction noise suppression algorithm to remove background noise, determine the start and end time points of the speech segment, and extract useful features from the audio signal; S603, converting the recognized speech into text form according to the Google Cloud Speech-toAPI speech recognition engine; S604: Text parsing: using a large language model to perform text parsing.

8. The smart home interaction method according to claim 7, characterized in that: Displaying text information on the digital human interaction screen and waiting for operation confirmation includes verbal confirmation and using voice recognition to recognize the user's voice instructions.

9. The smart home interaction method according to claim 8, characterized in that: Displaying text information on the digital human interaction screen and waiting for operation confirmation also includes clicking to confirm, adding a confirmation button on the digital human interaction screen interface for the user to click.

Images