Family knowledge base oriented to smart home and large language model and construction method

Abstract

The invention discloses a family knowledge base and a construction method oriented to smart home and a large language model, and the method comprises the steps: storing family personalized data in a local family based on the basic information of family equipment, the sensing information of a family environment, the daily behavior pattern of family members, and the personalized information of equipment use habits; in this way, a new conclusion can be derived from existing knowledge or a potential rule can be found, then possible requirements or preferences of the user can be deduced, and home personalized services can be achieved in combination with the large language model.

Description

Technical Field

[0001] The present invention relates to the field of smart home technology, and in particular to a family knowledge base and a construction method thereof for smart homes and large language models. Background Art

[0002] In recent years, with the rapid development of technologies such as the Internet of Things (IoT), artificial intelligence (AI), and big data, the smart home industry has never stopped upgrading. General large language models have shown excellent performance in translation, question answering, text generation, and other fields. By using large models to improve the language understanding and text generation capabilities in smart home systems, the intelligence level of smart home systems can be improved, and the user experience can be improved. This has become an emerging hotspot in the field of smart home research.

[0003] At the same time, data privacy issues have become increasingly prominent and have become one of the key factors restricting the further development of the industry. Smart home systems use various sensors, smart devices, and Internet connections to continuously collect and analyze users' living habits, preferences, and even personal privacy information, thereby providing more personalized and convenient services. However, many users are still worried about privacy data leakage and other issues. In order to solve this problem and achieve the goal of providing users with personalized and intelligent services, the edge home knowledge base came into being.

[0004] The combination of smart home, AI big model and edge family knowledge base not only improves the quality of life of users, but also enhances the living experience and meets personalized needs. Currently, smart home has the following technical pain points:

[0005] ① The level of intelligence and personalization needs to be improved: there are significant deficiencies in understanding user intentions and adapting to changes in the home environment. There are also obvious deficiencies in learning user personalized habits, and the update of personalized data lags behind.

[0006] ② Inadequate protection of data privacy: Users’ private home data is at risk of being leaked or attacked.

[0007] ③ Response speed and accuracy need to be improved: There is an obvious delay between the user issuing a command and the actual response, the user experience is poor, and there is a possibility of misoperation.

[0008] ④ Complex home scenes need to be developed: Currently, most of the scenes that can be implemented are simple life scenes, and complex and personalized home scenes are difficult to implement. Summary of the invention

[0009] The purpose of the present invention is to provide a family knowledge base and construction method for smart home and large language model in order to solve the above problems. The construction of the family knowledge base of the present invention relies on multi-dimensional information such as basic information of home devices, sensor data of home environment, daily behavior patterns of family members and equipment usage habits. In combination with the application of large language model, the present invention accurately judges the user's intention by collecting multimodal data such as text, images, videos, audio, etc. in the home environment. When the user's intention is identified as a custom quick instruction, the system will be directly handed over to the Internet of Things control module for processing to achieve accurate and rapid response to the user in the home scene. If the user's intention involves complex instructions other than quick instructions, the large model interaction module will be entered. In this module, the system will combine the constructed family knowledge vector library to enhance the intelligence level of the general language large model in the field of smart home, thereby providing users with a highly personalized home service experience.

[0010] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0011] A family knowledge base for smart home and large language model, characterized by comprising a family knowledge vector library, a multimodal data module, an intention judgment module, an Internet of Things control module and a large model interaction module;

[0012] The family knowledge vector library is used to collect sensor data inside the home, including personalized information such as device logs, user behavior records, user living habits, and environmental data, to achieve all-round smart home question-answering responses and intelligently infer possible user needs or preferences;

[0013] The multimodal data module is used to receive multimodal family data, including images, audio, video, and text;

[0014] The intention judgment module classifies the sentences input by the user into predefined intention categories through natural language processing technology, and judges the user's intention based on the user's input and the information in the family knowledge base, and distributes the judgment result to the Internet of Things control module or the large model interaction module;

[0015] The IoT control module supports user-defined home quick response commands to achieve fast and seamless intelligent response;

[0016] The large model interaction module has a highly intelligent semantic understanding capability, which is used to understand the user's questions and provide personalized and intelligent responses in combination with the family knowledge vector library built within the family.

[0017] A further solution is that the Internet of Things control module is used to support users to customize home quick response commands to achieve fast and seamless intelligent response. When the intention recognition module determines that the user's intention is to customize home control commands, it will directly control the smart devices involved in the commands to achieve remote control and automatic adjustment.

[0018] A further solution is that the large model interaction module interacts with the intention judgment module and the home knowledge vector library. When the user's intention is not a customized home control command, the large model interaction module will be entered, and the similarity comparison with the data in the home knowledge library will be performed, the prompt words will be assembled, the large model will be input, and the final result will be output as voice.

[0019] Another aspect of the present invention provides a method for constructing a household knowledge base for smart homes and large language models, comprising the following steps:

[0020] Step S1: First, data is collected, then the collected data is cleaned, then the data is divided into blocks, and then the text is vectorized, and finally a family knowledge vector library is constructed;

[0021] Step S2: during the interaction with the user, using different sensors or devices to obtain multimodal data in the interaction process in real time, where the sources of the multimodal data include voice, text, pictures, and videos;

[0022] Step S3: input the obtained multimodal data into the intention judgment module to judge the user's intention; if the customized quick response family command is hit, it will directly go to the Internet of Things control module; otherwise, it will enter the large model interaction module;

[0023] Step S4: When the intention in step S3 is directed to the IoT control module, an instruction is immediately issued to achieve a direct response;

[0024] Step S5: If the intention-oriented big model interaction module in step S3, the text input by the user is first vectorized, and then the similarity is compared with the data in the family knowledge base; then the prompt words are assembled and input into the big model, and finally the result is output as voice;

[0025] Step S6: Asynchronously update the family vector knowledge base.

[0026] A further solution is that step S1 includes the following steps:

[0027] Step S1.1: Data collection: Collect basic information of home devices, sensor information of home environment, daily behavior patterns of family members and personalized information of device usage habits;

[0028] Step S1.2: Data cleaning: remove noise from the data, including inaccurate or erroneous data; remove special formats, special characters, stop words and punctuation to simplify the data; perform stemming or word segmentation to reduce vocabulary and improve processing efficiency;

[0029] Step S1.3: Data segmentation: Multiple adjacent phrases or sentences describing the same thing are grouped into text blocks, whose granularity is larger than phrases but smaller than paragraphs. In the RAG method, large documents are divided into "chunks" for storage, which will help improve retrieval efficiency on the one hand, and on the other hand, the lack of retrieval accuracy can be compensated to a certain extent through contextual retrieval methods based on specific "chunks".

[0030] It can be seen that the result of segmentation will directly affect the final application effect, so it is crucial to choose a suitable segmentation method.

[0031] The commonly used methods of text segmentation are as follows:

[0032] ① Split by number of characters: Split according to the character length of the set block. This is the simplest block splitting method, which is relatively "foolproof" and lacks semantic understanding;

[0033] ② Segmentation by paragraph: Segmentation is performed based on the hierarchical structure of the document. This requires a high level of document parsing capability and has better segmentation effects than segmentation by the number of characters.

[0034] ③Segmentation by semantics: It is necessary to combine the model capabilities to understand the document content, and segment and splice it according to semantics. The segmentation effect is good, but the technology is difficult.

[0035] The present invention adopts a text segmentation method based on semantic segmentation and statistics:

[0036] This method uses a binary model and assumes that the probability of a word appearing is only related to the word immediately preceding it, and has nothing to do with other words. The method is described as follows:

[0037] Assume {T1, T2, T3, ..., T n} is the result of word segmentation of all texts in the corpus, where T i is a word, n is the total number of words in the word segmentation result, T i It consists of one or more words, expressed as w1w2...w m , then the word T i+1 The probability of occurrence depends only on T i The formula for conditional probability is:

[0038]

[0039] Among them, P(T i+1|T i ) indicates that at T i In case of T i+1 The probability of occurrence, P(T i T i+1 ) indicates T i and T i+1 The probability of simultaneous occurrence. The specific calculation method is as follows:

[0040]

[0041] Among them, count(T i T i+1 ), count(T i ) represent T i T i+1 and T i Number of occurrences.

[0042] The specific implementation steps of the algorithm are as follows:

[0043] Step 1: Divide the segmented text into bigrams by grouping adjacent words in pairs. Each bigram is in the form of: i T i+1 .

[0044] Step 2: Count the probability P(T) of each word appearing in the word segmentation results, and count each bigram T i T i+1 The frequency of occurrence P(T i T i+1 ).

[0045] Step 3: Calculate the i The probability of each word appearing when (i∈1, 2, ..., n) appears.

[0046] Step 4: Traverse the results of word segmentation and convert P(T i+1 |T i )Words whose values ​​are greater than a given threshold α are merged into phrases.

[0047] If the value of α is too small, the granularity of word division will be too large; if the value of α is too large, the granularity of word division will be too small, which will also cause the cutting of real phrases, thus affecting subsequent results.

[0048] Step S1.4: Embeddings are a way to transform real-world objects, such as words, images, or videos, into a form that computers can process. Embeddings are a way to represent words and phrases in a high-dimensional space, and how they can be used to measure similarities between different texts.

[0049] Knowledge text vectorization is to convert words into vector representations, which can express the similarity between words in a continuous vector space. This vectorized representation method can not only effectively capture the semantic connection between words, but also provide a unified metric. This means that even if the search keywords are not exactly the same as the words in the document, as long as they are semantically similar, the system can still give a high similarity score, thereby accurately retrieving relevant documents. This representation method helps to improve retrieval efficiency and accuracy, and adapts to the needs of deep learning models.

[0050] The present invention adopts a continuous bag of words (CBOW) model to vectorize the family knowledge text. The model is a typical three-layer neural network, and its structure is divided into three layers: input, output and hidden.

[0051] Assume that the jth word in the family knowledge text is x j , the continuous bag of words model intercepts the context words adjacent to the word as the input vector of the input layer. The input layer receives the word vectors of the context words within a certain range before and after the target word, and uses one-hot encoding to encode the intercepted words into a vector set with multi-dimensional features and convert them into the initial word vector. The encoded word vector is input to the hidden layer, and all input word vectors are accumulated or averaged. Assuming that the number of encoded context words is n, the dimension of the word vector is v, and the size of the dictionary is R, each context word is One-Hot encoded and multiplied by the embedding matrix to obtain the corresponding word vector. The formula is:

[0052]

[0053] Where h represents the hidden layer output vector of the continuous bag-of-words model; x j，* Represents the initial word vector after one-hot encoding. The obtained word vector is sent to the output layer, which is a softmax layer used to predict the target word. The output vector is the target word and the model is trained using the loss function softmax.

[0054] During training, the model will traverse each word in the corpus, take it as the target word, and select the context words before and after it as input. The model will calculate the probability distribution between the word vector of the input context word and the output target word, and update the value of the word vector through the back propagation algorithm and gradient descent optimization method, which is expressed as follows:

[0055]

[0056] Among them, y represents the probability that the word vector is the target word; u(h) represents the score of the word vector.

[0057] After the segmented documents are represented by vectors, they are usually queried using similarity retrieval. Similarity formulas are often used in data mining to evaluate the differences between individuals. The similarity calculation formulas for scenarios are as follows:

[0058] (1) Euclidean distance

[0059] Euclidean distance is also called Euclidean distance, which is usually used to calculate the distance between two points or two vectors. The formula is as follows:

[0060]

[0061] (2) Cosine similarity

[0062] Cosine similarity is different from Euclidean distance in that it calculates the distance between two vectors. Cosine similarity focuses more on the difference in direction between vectors. The formula is as follows:

[0063]

[0064] Among them, R u，i represents the rating of user u on item i, represents the average rating of all items by user u.

[0065] (3) Pearson correlation coefficient

[0066] The Pearson correlation coefficient is used to quantify the linear relationship between two variables. It depends on the covariance of the two variables and their respective standard deviations. Its value is between -1 and 1.

[0067]

[0068] A further solution is that in step S2, the collected multimodal data can be further subjected to multimodal data fusion. Multimodal data fusion is the process of integrating data and information from different types. These data and information come from different senses and sensors. For example, vision corresponds to pictures and video data collected by the camera, hearing corresponds to voice data collected by the microphone, and touch may involve some sensor data on the surface of smart home devices (if there are relevant tactile sensors), etc. By fusing the data from these different senses and sensors, more accurate and comprehensive information can be obtained. Multimodal data fusion includes feature fusion, depth fusion, etc.; so that the fused information can play a greater role in the smart home system, help to improve the precision of the data, and then have a positive effect on the entire smart home system.

[0069] A further solution is that step S3 includes the following steps:

[0070] Step S3.1: The intention judgment module uses natural language processing technology to classify the sentences input by the user into pre-defined intention categories.

[0071] Step S3.2: Create custom shortcuts for commands that are frequently used by users and require quick responses, such as turning on and off lights, air conditioning control, and other home control commands. If the user inputs a custom control command, the user input content will enter the IoT control module;

[0072] Step S3.3: If the user input is not the custom control instruction described in step S3.2, then the user input content enters the large model interaction module.

[0073] A further solution is that in step S4, the IoT control module is a key part to achieve interconnection and intelligent control and management of IoT devices. The module uses integrated sensors, actuators, communication interfaces, and microprocessors to enable the device to automatically sense environmental changes and respond accordingly to these changes;

[0074] In the smart home system, the IoT control module can connect to various smart devices in the home, including door locks, lights, air conditioners, etc.

[0075] When the intention recognition module determines that the user's intention is a customized home control command, it will directly control the smart devices involved in the command to achieve the functions of remote control and automatic adjustment.

[0076] A further solution is that step S5 includes the following steps:

[0077] Step S5.1: Text vectorization aims to convert the text input by the user into a vector that can reflect the text semantics, that is, to use a numerical vector to represent the text semantics, which belongs to the category of text vectorization processing. Common text vector and word embedding methods include the One Hot Model, Bag of Words Model, TF-IDF, N-Gram, Word2vec, and Doc2vec.

[0078] Step S5.2: Similarity comparison is to compare the user input with the vectors stored in the smart home knowledge base one by one, so as to calculate the similarity score. Common similarity calculation methods include cosine similarity, Euclidean distance, Manhattan distance, etc.

[0079] Step S5.3: prompt word assembly: The prompt words not only cover the basic task instructions asked by the user, but also contain the background knowledge with the highest similarity score retrieved from the smart home knowledge base, which can greatly optimize the output of the large model.

[0080] Step S5.4: Input the assembled prompt words into the large model, and then wait for the large model to return the result.

[0081] A further solution is that step S6 includes the following steps:

[0082] Step S6.1: When the amount of locally cached data reaches the set cache threshold, an asynchronous home knowledge base update operation is triggered.

[0083] Step S6.2: Based on the latest family data, check whether the data in the existing family knowledge base is outdated or conflicting, and delete or update the data in a timely manner.

[0084] Step S6.3: As new data is added and old data is updated, the vector value of the new data in the knowledge base is recalculated when necessary.

[0085] The beneficial effects of the present invention are:

[0086] The present invention stores personalized family data locally based on basic information of home devices, sensor information of the home environment, daily behavior patterns of family members, and personalized information such as device usage habits. This makes it possible to derive new conclusions or discover potential rules from existing knowledge, and then infer possible needs or preferences of users. Combined with a large language model, personalized home services can be achieved.

[0087] The present invention uses a universal language large model to improve the understanding ability and intelligence level of the smart home "brain".

[0088] The present invention asynchronously updates the family vector knowledge base, checks whether the data in the existing family knowledge base is outdated or conflicting, and deletes or updates the data in a timely manner.

[0089] The Internet of Things control module of the present invention supports customized convenient home control instructions, and as the hub of home control response instructions, it can shorten the waiting time and response delay, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0091] Figure 1 This is a module structure diagram of the present invention. DETAILED DESCRIPTION

[0092] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0093] Example 1

[0094] This embodiment provides a family knowledge base and application for smart home and large language model:

[0095] Arrange smart home products such as millimeter-wave radar, voice panel, smart TV, etc. in the home space.

[0096] The IoT control module allows you to customize household quick response commands, including turning on lights, turning off lights, turning on air conditioners, opening curtains, turning off air conditioners, and closing curtains, for a total of 6 quick response commands.

[0097] Interaction scenario 1:

[0098] Step 1: The user issues a voice command "I want to turn on the light"

[0099] Step 2: The multimodal data processing module converts the user's voice into a text command "I want to turn on the light" and sends it to the intent recognition module.

[0100] Step 3: The intention recognition module uses natural language processing technology to classify "I want to turn on the light" into the predefined quick response command "turn on the light" category. After successful positioning, it is directly sent to the IoT control module.

[0101] Step 4: The IoT control module will generate the corresponding control signal and send it to the target device to perform the actual action.

[0102] Interaction scenario 2:

[0103] Step 1: The user uses the mobile phone APP to take a photo of the medicine and upload it, and asks about precautions for taking the medicine.

[0104] Step 2: The multimodal data processing module processes the image through multiple operations such as image preprocessing, feature extraction, and target recognition, extracts the text features in the image, and sends it to the intent recognition module.

[0105] Step 3: The intent recognition module uses natural language processing technology to first try to classify it into the predefined quick response instruction category. If the positioning fails, it is directly sent to the large model interaction module.

[0106] Step 4: After vectorizing the input text, compare the similarity between the text and the text in the smart home knowledge vector library. The knowledge base text with the highest similarity, such as "suffering from high blood pressure" and "take medicine regularly in the morning and evening", are assembled together with the input text to form prompt words.

[0107] Step 5: Input the prompt word into the large language model. After getting the response result from the large model, respond to the user in the form of voice or text.

[0108] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further explain various possible combinations. In addition, the various different embodiments of the present invention can also be combined arbitrarily, as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A family knowledge base for smart home and large language model, characterized in that: It includes family knowledge vector library, multimodal data module, intention judgment module, Internet of Things control module and large model interaction module; The family knowledge vector library is used to collect sensor data inside the home, including device logs, user behavior records, user living habits, environmental data and other personalized information, to achieve all-round smart home question-answering response and intelligently infer the user's possible needs or preferences; The multimodal data module is used to receive multimodal family data, including multiple forms such as images, audio, video, and text; The intention judgment module classifies the sentences input by the user into predefined intention categories through natural language processing technology, and judges the user's intention based on the user's input and the information in the family knowledge base, and distributes the judgment result to the Internet of Things control module or the large model interaction module; The IoT control module supports user-defined home quick response commands to achieve fast and seamless intelligent response; The large model interaction module has a highly intelligent semantic understanding capability, which is used to understand the user's questions and provide personalized and intelligent responses in combination with the family knowledge vector library built within the family.

2. A family knowledge base for smart home and large language model as claimed in claim 1, characterized in that: The Internet of Things control module is used to support users to customize home quick response instructions and realize fast and seamless intelligent response. When the intention recognition module determines that the user's intention is a customized home control instruction, it will directly control the smart devices involved in the instruction to realize remote control and automatic adjustment.

3. A family knowledge base for smart home and large language model as claimed in claim 1, characterized in that: The large model interaction module interacts with the intention judgment module and the home knowledge vector library. When the user's intention is not a customized home control command, the large model interaction module will be entered, and the similarity comparison with the data in the home knowledge library will be performed, the prompt words will be assembled, the large model will be input, and the final result will be output by voice.

4. A method for constructing a family knowledge base for smart home and large language model as described in any one of claims 1 to 3, characterized in that: The following steps are involved: Step S1: First, data is collected, then the collected data is cleaned, then the data is divided into blocks, and then the text is vectorized, and finally a family knowledge vector library is constructed; Step S2: during the interaction with the user, using different sensors or devices to obtain multimodal data in the interaction process in real time, where the sources of the multimodal data include voice, text, pictures, and videos; Step S3: input the obtained multimodal data into the intention judgment module to judge the user's intention; if the customized quick response family command is hit, it will directly go to the Internet of Things control module; otherwise, it will enter the large model interaction module; Step S4: When the intention in step S3 is directed to the IoT control module, an instruction is immediately issued to achieve a direct response; Step S5: If the intention-oriented big model interaction module in step S3, the text input by the user is first vectorized, then the similarity is compared with the data in the family knowledge base, and then the prompt words are assembled and input into the big model, and finally the result voice is output; Step S6: Asynchronously update the family vector knowledge base.

5. A method for constructing a family knowledge base for smart home and large language model as claimed in claim 4, characterized in that: Step S1 includes the following steps: Step S1.1: Data collection: Collect basic information of home devices, sensor information of home environment, daily behavior patterns of family members and personalized information of device usage habits; Step S1.2: Data cleaning: remove noise from the data, including inaccurate or erroneous data; remove special formats, special characters, stop words and punctuation; perform stemming or word segmentation to reduce vocabulary; Step S1.3: Data segmentation: Multiple adjacent phrases or sentences describing the same thing are grouped into text blocks, whose granularity is larger than phrases but smaller than paragraphs; Step S1.4: Knowledge text vectorization: The continuous bag-of-words model is used to vectorize the family knowledge text to effectively capture the semantic connections between words.

6. A method for constructing a family knowledge base for smart home and large language model as claimed in claim 4, characterized in that: In step S2, multimodal data is fused, that is, different types of data and information are integrated. These data and information come from different senses and sensors. Multimodal data fusion includes feature fusion and depth fusion.

7. A method for constructing a family knowledge base for smart home and large language model as claimed in claim 4, characterized in that: Step S3 includes the following steps: Step S3.1: The intention judgment module classifies the sentence input by the user into a predefined intention category; Step S3.2: Create custom shortcut commands for commands that are frequently used by users and require quick responses. If the user inputs a custom control command, the user input content will enter the IoT control module; Step S3.3: If the user input is not the custom control instruction described in step S3.2, then the user input content enters the large model interaction module.

8. A method for constructing a family knowledge base for smart home and large language model as claimed in claim 4, characterized in that: In step S4, the IoT control module integrates sensors, actuators, communication interfaces and microprocessor technologies, so that the device can automatically sense environmental changes and respond accordingly to these changes; When the intention recognition module determines that the user's intention is a customized home control command, it will directly control the smart devices involved in the command to achieve the functions of remote control and automatic adjustment.

9. A method for constructing a family knowledge base for smart home and large language model as claimed in claim 4, characterized in that: Step S5 includes the following steps: Step S5.1: Text vectorization converts the text input by the user into a vector that can reflect the text semantics, that is, using a numerical vector to represent the text semantics; Step S5.2: Similarity comparison, comparing the user input with the vectors stored in the smart home knowledge base one by one, thereby calculating a similarity score; Step S5.3: prompt word assembly: The prompt words not only cover the basic task instructions asked by the user, but also contain the background knowledge with the highest similarity score retrieved from the smart home knowledge base, which is used to optimize the output of the large model; Step S5.4: Input the assembled prompt words into the large model, and then wait for the large model to return the result.

10. A method for constructing a family knowledge base for smart homes and large language models as claimed in claim 4, characterized in that: Step S6 includes the following steps: Step S6.1: When the amount of data in the local cache reaches the set cache threshold, the asynchronous family knowledge base update operation is triggered; Step S6.2: Based on the latest family data, check whether the data in the existing family knowledge base is outdated or conflicting, and delete or update the data in a timely manner; Step S6.3: As new data is added and old data is updated, the vector value of the new data in the knowledge base is recalculated when necessary.

Images