Scene-driven smart home large model cue word adaptive fine tuning method and system

Abstract

The invention discloses a scene-driven smart home large model cue word adaptive fine tuning method and system, and the method comprises the steps: understanding image data of a scene through a visual multi-mode large model, and obtaining user data through a sensor; constructing a user portrait in a vertical domain scene according to the understanding content of the visual multi-modal large model for the image data and the user data, generating a personalized vertical domain scene corpus based on cue word engineering according to the user portrait, and performing fine tuning on the basic language large model by taking the vertical domain scene corpus as input data to obtain a vertical domain scene fine tuning model; updating the feature data of the user portrait in real time to adjust the construction specification of the cue word, generating language interaction content based on the vertical domain scene fine tuning model and the cue word construction specification updated in real time, and realizing self-adaptive adjustment of the cue word on the content generated by the fine tuning model; the method has the technical advantages of strong scene adaptability, high personalization, excellent user experience, high practicability and flexibility, high vertical domain scene practicability and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart home, and more specifically, relates to a scene-driven smart home large model prompt word adaptive fine-tuning method and system. Background Art

[0002] Prompt tuning is a technique for fine-tuning large models. The principle is to add a series of trainable "prompt words" or "prompt vectors" before the input text, and only train the embedding vectors of these prompt words, while other parameters of the model remain unchanged. This method can customize the task without updating the model parameters on a large scale.

[0003] Although the large model cue word fine-tuning technology performs well in specific tasks, it still has some shortcomings, especially when it comes to non-scene-driven and adaptive adjustment capabilities. The following is a detailed description of several major shortcomings:

[0004] (1) Non-scenario driven: Large models may not always be able to understand and adapt to specific scenarios or contexts well. This means that the model may not fully understand the user needs in a specific scenario, resulting in the answers it generates or the tasks it performs not being in line with the user's expectations, which reduces the user experience and satisfaction. For example, in a smart home environment, if the model is not optimized for the behavior patterns of specific users, it may not be able to accurately predict the user's needs at a specific time or situation, and thus cannot provide personalized services.

[0005] (2) The fine-tuning technology of prompt words lacks adaptive adjustment capabilities: Although large models can be fine-tuned to adapt to specific tasks or fields, they may not have the ability to adaptively adjust in real time. They lack the ability to learn and adjust in real time, and cannot adapt to dynamic changes in user behavior or changes in the environment. They cannot respond to changes in user habits or preferences in a timely manner. This limits the flexibility and responsiveness of the model in a dynamic environment, affecting the continuity of user experience and the quality of personalized services. For example, if the user's habits or preferences change, the model may not be able to adjust its behavior in time to adapt to these changes, thereby affecting the user experience. Summary of the invention

[0006] The present invention proposes a smart home scene understanding and interaction method and system based on multimodal fusion, which jointly constructs a user portrait through the scene understanding of a visual multimodal large model and sensor data, and updates the feature data of the user portrait in real time, realizes adaptive adjustment of the prompt words to the content generated by the fine-tuning model, and promotes the development and popularization of smart home technology in an efficient, accurate, safe and user-friendly manner.

[0007] The present invention is implemented by the following technical solutions:

[0008] A scenario-driven adaptive fine-tuning method for prompt words of a large smart home model is proposed, including:

[0009] Collect vertical scene data, including image data that can be understood by the visual multimodal large model and sensor data that collects user data;

[0010] Build user portraits in vertical scenarios based on the understanding of image data and user data by the visual multimodal big model;

[0011] Generate personalized vertical scene corpus based on prompt word engineering;

[0012] Use personalized vertical scenario corpus to fine-tune the basic language model to obtain a vertical scenario fine-tuning model;

[0013] Adjust the construction specifications of prompt words in real time based on user scenarios and user portraits;

[0014] Generate language interaction content based on the vertical scenario fine-tuning model and real-time updated prompt word construction specifications.

[0015] In some embodiments of the present invention, generating personalized vertical scene corpus based on prompt word engineering includes:

[0016] Arrange the content of prompt words according to user portraits;

[0017] Obtain user portrait feature data for a specified user, and complete fine-tuning questions and answers for smart home scenarios based on the prompt word engineering according to the user portrait characteristics.

[0018] In some embodiments of the present invention, the construction specification of the prompt words is adjusted in real time in combination with the user scenario and the user portrait, including:

[0019] Query the user's current user profile;

[0020] Construct generation constraint specifications for prompt words based on the queried information.

[0021] A scenario-driven smart home large model prompt word adaptive fine-tuning system is proposed, including:

[0022] The acquisition unit is used to collect vertical scene data, including image data that can be understood by the visual multimodal large model and sensor data that collects user data;

[0023] A user portrait construction unit is used to construct a user portrait in a vertical domain scenario based on the understanding content of the image data and user data of the visual multimodal large model;

[0024] A vertical scene corpus generation unit, used to generate personalized vertical scene corpus based on the prompt word project;

[0025] A vertical scenario fine-tuning unit is used to fine-tune the basic language model using personalized vertical scenario corpus to obtain a vertical scenario fine-tuning model;

[0026] A prompt word updating unit, used to adjust the construction specifications of prompt words in real time based on user scenarios and user portraits;

[0027] The interactive content generation unit is used to generate language interactive content based on the vertical domain scenario fine-tuning model and real-time updated prompt word construction specifications.

[0028] In some embodiments of the present invention, the vertical scenario corpus generation unit generates personalized vertical scenario corpus based on the prompt word engineering, including:

[0029] Arrange the prompt words according to the user portrait;

[0030] Obtain user portrait feature data for a specified user, and complete fine-tuned question and answering for smart home scenarios based on the prompt word engineering according to the user portrait characteristics.

[0031] In some embodiments of the present invention, the prompt word updating unit adjusts the construction specification of the prompt word in real time in combination with the user scenario and the user portrait, including:

[0032] Query the user's current user profile;

[0033] Construct generation constraint specifications for prompt words based on the queried information.

[0034] Compared with the prior art, the advantages and positive effects of the present invention are: the scene-driven smart home big model prompt word adaptive fine-tuning method and system proposed in the present invention, based on the understanding of vertical domain scenes, adopts a visual multimodal big model to understand the image data of the scene, adopts sensors to obtain user data such as voice, user heart rate, user mood, etc., and constructs a user portrait in the vertical domain scene according to the understanding content of the image data and the user data of the visual multimodal big model, generates personalized vertical domain scene corpus based on the prompt word project according to the user portrait, and uses the vertical domain scene corpus as input data to fine-tune the basic language big model to obtain a vertical domain scene fine-tuning model, and then updates the feature data of the user portrait in real time to adjust the construction specifications of the prompt words, generates language interaction content based on the vertical domain scene fine-tuning model and the real-time updated prompt word construction specifications, and realizes the adaptive adjustment of the prompt words to the content generated by the fine-tuning model; it has the technical advantages of strong scene adaptability, high personalization, excellent user experience, high practicality and flexibility, and high practicality of vertical domain scenes.

[0035] Other features and advantages of the present invention will become more apparent after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0037] Figure 1 The following is a schematic diagram of the steps of the scenario-driven smart home large model prompt word adaptive fine-tuning method proposed by the present invention;

[0038] Figure 2 It is a schematic diagram of the execution process of the scenario-driven intelligent large model prompt word adaptive fine-tuning method proposed by the present invention;

[0039] Figure 3 This is an example of prompt word generation actually applied to a large model intelligent agent in an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of the adaptive effect of a prompt word after fine-tuning a model in the implementation of the present invention;

[0041] Figure 5 This is a system structure diagram of the scenario-driven smart home large model prompt word adaptive fine-tuning system proposed in the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The present invention proposes a scenario-driven smart home large model prompt word adaptive fine-tuning method and system, aiming at solving the following problems in view of the determination of the existing large model prompt word fine-tuning technology:

[0044] (1) Enhanced scene understanding capabilities: Develop a mechanism that enables large models to better understand and adapt to specific scenes or contexts. This involves the real-time acquisition and image understanding capabilities of multimodal visual models for smart home scenes.

[0045] (2) Improving adaptive adjustment capabilities: Designing a method that can monitor user behavior and environmental changes in real time and automatically adjust model parameters accordingly to achieve faster and more accurate model adjustment.

[0046] (3) Optimization of personalized services: By collecting and analyzing users’ long-term behavior data, we develop a model that can predict user needs and provide personalized services.

[0047] (4) Adaptability to dynamic environments: Ability to quickly learn and adapt to changes in user habits or preferences to maintain the relevance and effectiveness of services.

[0048] The present invention aims to solve the above-mentioned technical problems and provide an efficient, accurate, safe and user-friendly solution for scenario-driven adaptive fine-tuning technology of large model prompt words for smart home, so as to promote the development and popularization of smart home technology, and at the same time solve the shortcomings of the existing technology in data utilization, data quality, model generalization, computing resource consumption, etc., and be more suitable for the vertical scenarios of smart homes.

[0049] The following first describes the background technology related to the present invention.

[0050] 1. LoRA (Low-Rank Adaptation): is a technique for fine-tuning models, especially in large pre-trained models. It achieves adaptive adjustment of the model by introducing low-rank matrices in specific parts of the model instead of retraining the entire model. This approach can significantly reduce the demand for computing resources while maintaining or improving the performance of the model.

[0051] 2. Prompt word technology: It is the process of creating prompts or guiding the output of language models like ChatGPT, which allows users to control the output of the model and generate text that meets their specific needs.

[0052] A good question (prompt word) should at least include four key elements: task, requirement, instruction (constraint rules or conditions), and role.

[0053] (1) Task: The language model is required to complete a task and generate content in a clear and concise manner.

[0054] (2) Requirements: Background requirements for the task.

[0055] (3) Instructions: (Constraints or conditions) Instructions that the language model should follow when generating text, satisfying certain conditions or rules to prevent it from diverging too much.

[0056] (4) Role: The role that the model should play when generating text, which can be professional, emotional, or identity-based.

[0057] like Figure 3 Shown is an example of prompt word generation actually applied to a large model agent.

[0058] 3. In-context Learning (ICL): It is a method for a large language model to learn on a specific task through a small number of labeled samples. The core idea of ​​this method is to form a prompt template by designing task-related instructions, and use a small number of labeled samples as prompts to guide the model to generate prediction results on new test data. The advantage of ICL is that it does not require fine-tuning of the model, thus saving a lot of computing resources and time. It allows the model to learn through examples in context, which are part of the input, helping the model understand the task and generate corresponding output. ICL has shown performance comparable to or even exceeding that of full-resource fine-tuning methods in many NLP benchmark tests.

[0059] 4. Prompt Engineering: A technique for designing and optimizing prompts used to train AI models. A prompt is a form of input that instructs an AI model what action to take or what output to generate when performing a specific task. In Prompt Engineering, the user provides a short prompt, and then the model generates a text in response based on this prompt. This prompt can be a question, a keyword, a sentence, a paragraph of text, and so on. Prompt Engineering provides a very flexible and efficient approach to natural language generation because it can generate high-quality text based on the specific context provided by the user, resulting in more accurate and meaningful interactions.

[0060] Based on the above background knowledge, the present invention is implemented according to the following innovative ideas on the basis of the existing technology:

[0061] (1) Increase the real-time collection and processing of image data and sensor data for vertical scenes.

[0062] (2) Construct (multi-)user portraits for vertical scenarios.

[0063] (3) Based on user portrait and prompt word engineering technology, personalized generation of corpus for vertical scenarios is achieved.

[0064] (4) Input the personalized vertical scenario corpus into the basic large language model, and fine-tune the prompt words of the vertical scenario based on the In-context Learning (ICL) technology.

[0065] (5) The fine-tuning model obtained in the vertical domain scenario updates the user portrait (feature data) in real time during the actual call process to update the constraint template of the prompt word project and finally complete the generation control of language interaction.

[0066] From the above perspectives, the scenario-driven smart home big model prompt word adaptive fine-tuning method proposed in the present invention is as follows: Figure 1 and Figure 2 As shown, including:

[0067] S1: Collect vertical scene data.

[0068] This data mainly includes two parts. One part is the image data that can be understood by the large visual multimodal model. For example, the image data of the living room, kitchen, and bedroom lighting scenes in the smart home scene are collected in real time. The other part is the sensor data, which is used to collect the user's voice data, physical sign data, etc.

[0069] These two parts of data are used to build a user profile, which can be a user profile of a specific user or a user profile of multiple users.

[0070] S2: Build user portraits in vertical scenarios based on the understanding of image data and user data by the visual multimodal large model.

[0071] In the embodiment of the present invention, a user portrait is constructed for multiple or a specific user, as shown below:

[0072] Basic attributes: including basic information of users, such as age, gender, education, occupation, income level, city, etc.

[0073] Behavioral characteristics: User behavior habits, as shown in the following table:

[0074] Table 1: User behavior characteristics Active time Active frequency Use the function Browse content kitchen …… …… …… …… living room …… …… …… …… bedroom …… …… …… …… …… …… …… …… ……

[0075] Interest preferences: User interests, such as music, movies, books, sports, etc.

[0076] Psychological characteristics: user’s personality traits, lifestyle, values, etc.

[0077] Including but not limited to the above contents, for multiple users, they are created separately.

[0078] S3: Generate personalized vertical scenario corpus based on prompt word engineering.

[0079] Some specific question-and-answer examples of smart home scenario corpus generated using the basic language large model are as follows:

[0080] Table 2: Question 1 User: Will the lights in the living room turn on automatically when I get home? Assistant: Of course, you can set an automation rule. When you get home, the smart door lock recognizes your key or mobile phone, which will automatically trigger the living room lights to turn on and other home devices you prefer to adjust to a comfortable state.

[0081] Table 3: Question and Answer 2 User: I want to watch movies in the living room, what should I do? Assistant: You can activate "Movie Mode" in your living room, which will automatically dim the lights, close the curtains, and start your home theater system. You can also select the movie you want to watch through voice commands or the smart remote control.

[0082] Table 4: Question 3 User: If the air conditioning temperature in the living room is too low, how should I adjust it? Assistant: You can adjust the air conditioning temperature through the smart thermostat or voice commands. The system will automatically adjust according to your settings to ensure that your living room remains at the comfortable temperature you prefer.

[0083] Table 5: Question 4 User: I want my living room to be smarter, what devices should I add? Assistant: To improve the intelligence level of your living room, you can consider adding smart lights, smart curtains, smart sockets, smart TVs, smart speakers, smart door locks and other devices. These devices can be centrally controlled through the smart home system to achieve automated scenes and voice control.

[0084] Table 6: Question 5 User: How do I keep my living room safe when I'm not home? Assistant: You can use smart cameras and door and window sensors to monitor the safety of your living room. Once abnormal activities are detected, the system will immediately send an alert to your mobile phone. At the same time, you can also remotely view the real-time picture of the living room through your mobile phone.

[0085] Table 7: Question 6 User: How can the smart devices in the living room save energy? Assistant: Smart devices can save energy through automation and user behavior learning. For example, smart lights can automatically turn off when not needed, and smart thermostats can automatically adjust the air conditioning temperature based on the indoor temperature and your living habits. User: Can I remotely control devices in other rooms from the living room? Assistant: Yes, with the smart home system, you can remotely control smart devices in any room of your home through the smart central control screen in the living room or through mobile phone applications to achieve smart linkage throughout the house.

[0086] Table 8: Question 7 User: How to set the "meeting mode" in the living room? Assistant: You can set a "reception mode" in the smart home system. When it is activated, the system will automatically adjust the lighting, music and temperature in the living room to create a warm and comfortable reception environment.

[0087] Through the questions and answers in the above examples, we can see that the smart home system can provide personalized intelligent control and automation services based on the user's living habits and preferences, thereby improving the comfort and convenience of living.

[0088] Regarding how to realize the generation of corpus for personalized vertical scenarios through prompt word engineering, the following design is performed in the embodiment of the present invention:

[0089] First, based on the obtained user portrait, the following prompt word content arrangement is given:

[0090] "You are an expert in training prompt word fine-tuning models and have experience in writing scenario question and answer descriptions for prompt word fine-tuning. You can complete the writing of prompt word questions based on living room scenarios and their corresponding answer descriptions based on the user portraits of a single or multiple users.

[0091] Your task is to give question-and-answer examples about the smart home living room scene for model fine-tuning based on the user's current user profile features.

[0092] What you need to do next is to first obtain the user portrait feature data of the specified user, and then complete the arrangement of prompt word questions and answers based on the characteristics of the user portrait.

[0093] Assume that the characteristic data of the current user portrait is as follows:

[0094] 1. Dad is used to dimming the lights and closing the curtains when watching movies in the living room, but his son likes bright lights when watching movies in the living room.

[0095] 2. Dad likes the air-conditioning temperature to be 26 degrees, and son likes the air-conditioning temperature to be 22 degrees.

[0096] Then, based on the above user portrait content, and based on the fine-tuning of the smart home scene completed by the prompt word project, the question and answer will become as follows (you can compare it with the smart home scene corpus generated using the basic language large model):

[0097] Table 9: Question 8 User (Dad): Can the lights in the living room be dimmed when I get home? Assistant: Of course, we have set up a personalized home scene for you. When you arrive home, the smart system will recognize you and automatically dim the living room lights to your preferred brightness. User (son): Can the lights in the living room stay on when I get home? Assistant: No problem, we have set up a special home scene for you. When the smart system recognizes you, it will keep the living room lights brighter.

[0098] Table 10: Question 9 User (Dad): I want to watch a movie in the living room. How can I set it to my favorite mode? Assistant: You can activate the "Dad Movie Mode" in the living room, and the system will automatically dim the lights, close the curtains, and set the air conditioner to your preferred 26 degrees. You can also select your favorite movie through voice commands. User (son): I want to watch a movie in the living room at night. How should I adjust it? Assistant: You can use the "Son's Movie Viewing Mode" and the system will adjust the bright lights and your preferred air-conditioning temperature of 22 degrees to ensure that you are comfortable and happy while watching the movie.

[0099] Table 11: Question 10 User (Dad): Can the air conditioner in the living room be automatically adjusted according to my habits? Assistant: Yes, the smart thermostat will automatically adjust the air conditioning temperature according to your usage habits. When it detects that you are in the living room, it will automatically adjust to your favorite 26 degrees. User (son): I often play games in the living room. Can I set a mode that suits me? Assistant: Yes, we can set up a "game mode" for you. In this mode, the lights in the living room will be adjusted to a brightness suitable for the game, the air conditioner will be set to your favorite 22 degrees, and the sound system will be ready to provide an immersive gaming experience.

[0100] S4: Use personalized vertical scenario corpus to fine-tune the basic language model to obtain the vertical scenario fine-tuning model.

[0101] The user-personalized vertical scenario corpus obtained from S3 is used as a data set and input into the basic language model for fine-tuning. After fine-tuning, a fine-tuned language model for the vertical scenario can be obtained.

[0102] S5: Adjust the construction specifications of prompt words in real time based on user scenarios and user portraits.

[0103] When calling the vertical domain scene fine-tuning language model obtained in S4, the embodiment of the present invention is designed as follows: it is necessary to combine the prompt word project to perform the next prompt word generation control. Here, an operation of querying the user's current user portrait is required, for example, calling the interface to query the user's current mood state, heart rate and other user portrait information, and then constructing the prompt word generation constraint specification based on the queried information, for example:

[0104] The user portrait of the current user is obtained as follows: in the living room, in a good mood, a little sweaty, the air conditioner setting dimension preference is 26 degrees, it is evening, and he is watching TV... Then, the above information is used to update the current prompt word specification. An example after the update is as follows:

[0105] "You are a life assistant in the 'living room' scenario. Your task is to give better suggestions and serve users based on their current user profile status. You need to follow the steps below:

[0106] 1. Check the user's current physical condition. If the user is sweating, the system will prompt the user whether to turn on the air conditioner. If the air conditioner is turned on, the system will automatically adjust the temperature to the user's usual setting.

[0107] 2. Check the user's salary. If the user is watching TV, turn on the lights or brighten / dim the lights according to the user's preferences or the current light intensity in the room.

[0108] 3,…”.

[0109] S6: Generate language interaction content based on the vertical scenario fine-tuning model and real-time updated prompt word construction specifications.

[0110] According to the setting in S5, we can get Figure 4 The output of the fine-tuned model after adaptive adjustments is shown.

[0111] Based on the above steps, the present invention implements a prompt word fine-tuning technology driven by smart home scenarios, and achieves the following technical effects:

[0112] Strong scene adaptability: Through the content understanding and recognition technology of the visual multimodal large model, the present invention can better understand and adapt to specific scenes, thereby providing services that are more in line with user expectations.

[0113] Highly personalized: The present invention monitors user behavior and environmental changes in real time and automatically adjusts model parameters, so that the service can quickly respond to changes in user habits or preferences. This adaptive adjustment capability ensures the personalization of services and improves user satisfaction and loyalty.

[0114] Optimized user experience: By analyzing the user's long-term behavior data, the present invention can predict user needs and provide personalized services. This predictive ability makes the service more intimate and can meet the user's needs in advance, thereby improving the overall user experience.

[0115] Practicality and flexibility: This invention aims to improve the practicality and flexibility of large models in various application scenarios. This means that the model not only performs well on specific tasks, but also can adapt to changing environments and needs, providing users with stable and reliable services.

[0116] Promoting technological development: By providing an efficient, accurate, secure and user-friendly solution, the present invention helps promote the development and popularization of smart home technology, which will enable more users to enjoy the convenience brought by smart homes.

[0117] Applicability to vertical scenarios: The present invention is particularly suitable for vertical scenarios of smart homes, which means that it can provide customized solutions for specific problems in smart homes, thereby better meeting the needs of users in this specific area.

[0118] To sum up, the advantages of the present invention are that it can provide more intelligent, personalized and adaptable services, while promoting the development of related technologies, solving the shortcomings of existing technologies, and especially optimizing the application experience of smart home scenarios.

[0119] Based on the above-mentioned scenario-driven smart home big model prompt word fine-tuning method, the present invention also proposes a scenario-driven smart home big model prompt word fine-tuning system to execute the above-mentioned method, such as Figure 5 As shown, including:

[0120] The acquisition unit 1 is used to acquire vertical scene data, including image data that can be understood by the visual multimodal large model and sensor data for acquiring user data.

[0121] The user portrait construction unit 2 is used to construct a user portrait in a vertical domain scenario based on the understanding content of the image data and the user data of the visual multimodal large model.

[0122] The vertical scenario corpus generation unit 3 is used to generate personalized vertical scenario corpus based on the prompt word project.

[0123] The vertical scenario fine-tuning unit 4 is used to fine-tune the basic language model using personalized vertical scenario corpus to obtain a vertical scenario fine-tuning model.

[0124] The prompt word updating unit 5 is used to adjust the construction specification of the prompt word in real time in combination with the user scenario and the user portrait.

[0125] The interactive content generation unit 6 is used to generate language interactive content based on the vertical domain scenario fine-tuning model and the prompt word construction specification updated in real time.

[0126] In some embodiments of the present invention, the vertical scenario corpus generation unit 3 generates personalized vertical scenario corpus based on the prompt word engineering, including: arranging the prompt word content according to the user portrait; obtaining user portrait feature data of the specified user, and completing fine-tuning questions and answers of the smart home scenario based on the prompt word engineering according to the characteristics of the user portrait.

[0127] In some embodiments of the present invention, the prompt word updating unit 5 adjusts the construction specification of the prompt word in real time in combination with the user scenario and the user portrait, including: querying the user's current user portrait; and constructing the generation constraint specification of the prompt word according to the queried information.

[0128] The specific method of adaptive fine-tuning of prompt words implemented by the system will not be described in detail here.

[0129] The present invention described above realizes:

[0130] (1) Enhanced scenario-driven capabilities: By introducing advanced scene recognition and context understanding technologies, large models can better understand and adapt to specific scenarios or contexts, thereby improving the model’s performance and user satisfaction in specific scenarios.

[0131] (2) Improving adaptive adjustment capabilities: We have developed a mechanism that monitors user behavior and environmental changes in real time and automatically adjusts model parameters accordingly, enabling the model to quickly respond to changes in user habits or preferences and provide more personalized and dynamic services.

[0132] (3) Optimize personalized services: By collecting and analyzing users’ long-term behavior data, we develop models that can predict user needs and provide personalized services, thereby improving the user experience.

[0133] It should be noted that, in the specific implementation process, part of the above-mentioned method can be implemented by a hardware processor executing computer execution instructions in software form stored in the memory, which will not be elaborated here, and the programs corresponding to the actions executed can be stored in the system's computer-readable storage medium in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0134] The computer-readable storage medium mentioned above may include volatile memory, such as random access memory; may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; may also include a combination of the above types of memory.

[0135] The processor mentioned above may also be a general term for multiple processing elements. For example, the processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc., or may be a dedicated processor.

[0136] It should be pointed out that the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A scenario-driven adaptive fine-tuning method for prompt words of a large model of smart home, characterized in that: include: Collect vertical scene data, including image data that can be understood by the visual multimodal large model and sensor data that collects user data; Build user portraits in vertical scenarios based on the understanding of image data and user data by the visual multimodal big model; Generate personalized vertical scene corpus based on prompt word engineering; Use personalized vertical scenario corpus to fine-tune the basic language model to obtain a vertical scenario fine-tuning model; Adjust the construction specifications of prompt words in real time based on user scenarios and user portraits; Generate language interaction content based on the vertical scenario fine-tuning model and real-time updated prompt word construction specifications.

2. The scenario-driven smart home large model prompt word adaptive fine-tuning method according to claim 1 is characterized in that: Generate personalized vertical scene corpus based on prompt word engineering, including: Arrange the content of prompt words according to user portraits; Obtain user portrait feature data for a specified user, and complete fine-tuned question and answering for smart home scenarios based on the prompt word engineering according to the user portrait characteristics.

3. The scenario-driven smart home large model prompt word adaptive fine-tuning method according to claim 1 is characterized in that: Combine user scenarios and user portraits to adjust the construction specifications of prompt words in real time, including: Query the user's current user profile; Construct generation constraint specifications for prompt words based on the queried information.

4. The scene-driven smart home large model prompt word adaptive fine-tuning system is characterized by: include: The acquisition unit is used to collect vertical scene data, including image data that can be understood by the visual multimodal large model and sensor data that collects user data; A user portrait construction unit is used to construct a user portrait in a vertical domain scenario based on the understanding content of the image data and user data of the visual multimodal large model; A vertical scene corpus generation unit, used to generate personalized vertical scene corpus based on the prompt word project; A vertical scenario fine-tuning unit is used to fine-tune the basic language model using personalized vertical scenario corpus to obtain a vertical scenario fine-tuning model; A prompt word updating unit, used to adjust the construction specifications of prompt words in real time based on user scenarios and user portraits; The interactive content generation unit is used to generate language interactive content based on the vertical domain scenario fine-tuning model and real-time updated prompt word construction specifications.

5. The scenario-driven smart home large model prompt word adaptive fine-tuning system according to claim 4 is characterized in that: The vertical domain scenario corpus generation unit generates personalized vertical domain scenario corpus based on the prompt word engineering, including: Arrange the content of prompt words according to user portraits; Obtain user portrait feature data for a specified user, and complete fine-tuned question and answering for smart home scenarios based on the prompt word engineering according to the user portrait characteristics.

6. The scenario-driven smart home large model prompt word adaptive fine-tuning system according to claim 4 is characterized in that: The prompt word updating unit adjusts the construction specification of the prompt word in real time in combination with the user scenario and the user portrait, including: Query the user's current user profile; Construct generation constraint specifications for prompt words based on the queried information.

Images