Smart home large model vertical domain adaptation method based on low-rank matrix decomposition

Abstract

The invention relates to a smart home large model vertical domain adaptation method based on low-rank matrix decomposition. The method comprises the steps of 1, preparing vertical domain data; the vertical domain data comprises corpora, images and models; step 2, extracting image label data; step 3, cleaning image label data; 4, labeling the vertical domain scene label data; 5, performing fine adjustment on the visual LoRA model; and step 6, tag data prompt word processing.

Description

Technical Field

[0001] The present invention relates to a vertical domain adaptation method for a large model of a smart home based on low-rank matrix decomposition. Background Art

[0002] With the continuous advancement of big model technology, its influence has gradually expanded to all areas of our daily life and production. Although general big model technology has broad generalization capabilities, when facing specific fields, targeted vertical domain model training is usually required. However, traditional vertical domain training methods have some shortcomings: (1) Data scarcity: Data in a specific field may be difficult to obtain, resulting in insufficient training data. (2) Uneven data quality: The collected data may be noisy, inaccurate or incomplete. (3) Poor model generalization ability: The model performs well on data in a specific field, but performs poorly on other fields or unseen samples. (4) High computing resource consumption: Training large models usually requires a lot of computing resources.

[0003] For existing vertical model training and adaptation solutions, obtaining sufficient training data is a challenge in certain areas, especially in the smart home field, because such data may be difficult or costly to collect. Even if data can be collected, it may contain noisy, inaccurate or incomplete information, which will affect the quality and effect of model training. Training large models usually requires a lot of computing resources, which not only increases costs but also limits its application in resource-constrained environments. Summary of the invention

[0004] The technical problem to be solved by the present invention is generally to provide a vertical domain adaptation method for a large model of a smart home based on low-rank matrix decomposition, aiming to solve the challenges encountered in the training and adaptation process of the smart home domain model in the prior art. Thus, an efficient, accurate, safe and user-friendly vertical domain adaptation solution for a large model of a smart home is provided to promote the development and popularization of smart home technology, while solving the deficiencies of the prior art in data utilization, data quality, model generalization, computing resource consumption, etc. It is more suitable for the vertical domain scenario of smart homes.

[0005] To solve the above problems, the technical solution adopted by the present invention is:

[0006] A vertical domain adaptation method for a large model of a smart home based on low-rank matrix decomposition, the method comprising the following steps;

[0007] Step 1: Prepare vertical domain data; vertical domain data includes corpus, images and models;

[0008] Step 2: extract image label data;

[0009] Step 3: Clean the image label data;

[0010] Step 4: label the vertical scene data;

[0011] Step 5: Fine-tune the visual LoRA model;

[0012] Step 6: Wordify the label data prompts.

[0013] Further, in step 1, the corpus, the corpus prompt words used for vertical domain fine-tuning model training, includes question description and content response;

[0014] Images, images used for training the vertical scene vision fine-tuning model;

[0015] Model, a large base model or multimodal vision model used for model fine-tuning.

[0016] Furthermore, the method is based on a basic large model, which includes a language large model and a visual large model.

[0017] Furthermore, in step 2, the vertical scene image of step 1 is sent to the basic visual model or the multimodal visual model, and the data label content is obtained through image content understanding or prompt word reverse deduction;

[0018] The rule for arranging the data label content is that the one with the highest weight is placed first, and then the one with the highest weight is placed last.

[0019] Identify the keywords of the vertical scene in the current image data.

[0020] Furthermore, in step three, the data labels that have achieved image content recognition are cleaned to remove data labels that are considered to be interference or to be ignored.

[0021] Further, in step 2, data extraction is performed on the kitchen data scene;

[0022] In step 4, the label data is cleaned or cleared according to step 3; the content that is considered important or needs to be identified is re-labeled and sorted;

[0023] It customizes the data labels extracted from the kitchen data scene in step 2. For example, it customizes the originally identified "Refrigerator" to "Haier_regrigerator".

[0024] Furthermore, in step five, fine-tuning model training is performed on the vertical scene data that has been re-labeled in step four.

[0025] Furthermore, in step six, the data labels newly added or adjusted in step four are lexicalized, and based on the keywords appearing in step four, descriptions and questions are given for the current vertical domain scenario, i.e., the step language LoRA model is fine-tuned.

[0026] Furthermore, by fine-tuning the visual model, the language model is synchronously aligned with the effect of the fine-tuning of the visual model to achieve large-model training for vertical scenarios and its adaptation to vertical scenarios.

[0027] Furthermore, the visual model and language model are fine-tuned synchronously, and the prompt word labels recognized by the visual model image content are transformed and processed to achieve alignment of the generated effects.

[0028] Furthermore, the prompt words include key information; the key information includes tasks, requirements, instructions and roles;

[0029] in,

[0030] Tasks,require ChatGPT to complete the set tasks, generate content and make presentations;

[0031] Requirements, explain the background requirements of the task.

[0032] Instructions, which ChatGPT follows when generating text, satisfying set conditions or rules;

[0033] Role, the role that the model is set to play when generating text.

[0034] The present invention solves the following technical problems:

[0035] (1) Improving data utilization efficiency: Through low-rank matrix decomposition technology, the present invention can more effectively utilize limited specific field data, overcome the problem of data scarcity, and thus improve the training effect of the model.

[0036] (2) Improve data quality: The present invention improves data quality through advanced data processing techniques, such as noise reduction and data cleaning, to ensure the accuracy and reliability of model training.

[0037] (3) Enhance the generalization ability of the model: By utilizing low-rank matrix decomposition, the present invention can extract key features and optimize the model structure, thereby improving the generalization ability of the model in different fields and unknown samples.

[0038] (4) Reducing computing resource consumption: The low-rank matrix decomposition method adopted in the present invention can effectively reduce the computational burden in the model training process and reduce the demand for computing resources, so that effective model training can be performed even in a resource-constrained environment.

[0039] (5) Improving model adaptation speed: By optimizing the model structure and training process, the present invention can speed up the adaptation speed of the smart home large model in a specific vertical domain, shorten user waiting time, and improve user experience.

[0040] (6) Improving the scalability and flexibility of the model: The method of the present invention can adapt to the ever-changing technical requirements and user preferences in the smart home field, so that the model can quickly adapt to new devices and scenarios.

[0041] (7) Customized scene annotation: By adding image data processing and customized scene annotation for vertical scenes, the present invention can more accurately understand and respond to the needs of specific scenes.

[0042] (8) Fine-tuning the visual and language models: By implementing fine-tuning in the basic visual model and the basic language model, the present invention can achieve more precise control and response, and improve the overall performance and user experience of the smart home system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the prompt word generation arrangement before improvement.

[0044] Figure 2 It is a schematic diagram of the existing vertical domain model training scheme of the present invention.

[0045] Figure 3 It is a processing flow diagram of the present invention.

[0046] Figure 4 It is a schematic diagram of the overall process of the present invention.

[0047] Figure 5 It is a schematic diagram of an example image of a vertical domain scene of the present invention.

[0048] Figure 6 It is a schematic diagram of obtaining image content labels of the present invention.

[0049] Figure 7 This is an example diagram of the large model capabilities after fine-tuning the kitchen scene of the present invention. DETAILED DESCRIPTION

[0050] like Figure 1-7 , as an example of the actual application of prompt word generation for large model agents: The process of generating CAD models before improvement (there are two types as follows) is as follows Figure 1 shown.

[0051] Compared with the existing scheme, the processing flow of this scheme is as follows: (1) Add image data processing for vertical domains (scenes); (2) Extract content identification labels from the image information of vertical domains (scenes), clean the label data and perform customized scene annotation on the images; (3) Fine-tune the image data samples after scene annotation in the basic visual model; (4) Use the image labels of customized scene annotation as prompt word materials to implement fine-tuning in the basic language model; (5) Finally, the generation control in the current vertical domain scene is jointly realized through the fine-tuned visual model and the aligned fine-tuned language model.

[0052] The process of the present invention is described as follows:

[0053] Step 1: Prepare vertical domain data; vertical domain data includes corpus, images and models;

[0054] Corpus, which is used for corpus prompts for vertical domain fine-tuning model training, including question descriptions and content responses;

[0055] Images, used for training the vertical scene visual fine-tuning model, such as living room scenes, kitchen scenes, pictures of a certain home decoration style, etc.

[0056] Model, a large basic model used for model fine-tuning.

[0057] This solution is based on two basic large models, namely a language large model (for example, chatgpt4) and a visual large model (stable diffusion).

[0058] Step 2: Extract image label data; Figure 5 , Figure 7 The image data of the kitchen scene is shown in the figure. The vertical scene image is sent to the basic visual model (stable diffusion) or multimodal visual model, and the following data label content can be obtained through image content understanding or prompt word reverse deduction. The rule for the order of the label content is: the highest weight is at the front, and then push it back.

[0059] For the current image data, keywords of the vertical scene such as Kitchen and Refrigerator can be identified.

[0060] Step three: image label data cleaning; it can clean the data labels that have already realized image content recognition and remove data labels that are considered to be interfering or less important.

[0061] Step 4: Labeling the vertical scene data; in addition to cleaning or clearing the label data according to step 3, you can also re-label and sort the content that is considered important or needs to be recognized. For example, it can customize the data labels extracted from the kitchen data scene in step 2, for example, customize the original recognized "Refrigerator" to "Haier_regrigerator".

[0062] Step 5: Fine-tune the visual LoRA model; the vertical scene data that has been re-labeled can be used for fine-tuning model training.

[0063] Step six, label data prompt wordization; the newly added or adjusted data labels in step four are wordized, and based on the keywords appearing in step four, descriptions and questions about the current vertical domain scenario are developed, that is, the step language LoRA model is fine-tuned.

[0064] Based on the above steps 1 to 6, the vertical domain large model training and adaptation of this patent can be realized. The final effect is illustrated by an example.

[0065] In such Figure 7 In the kitchen scene (the image does not exist in step 2, and the image used to fine-tune the model is a reflection of the generalization ability of the trained vertical domain model), after fine-tuning the visual model and language of this patent and aligning with the image data, the following scene can be realized:

[0066] This patent can identify the location of "Haier Refrigerator" in the scene and the information near "Haier Refrigerator", and can also perform targeted command interaction with the "Haier Refrigerator" through language model question and answer interaction, for example, sending instructions through a large language model: let the sweeping robot clean the location of "Haier Refrigerator".

[0067] By fine-tuning the visual model, the language model is synchronously aligned with the effect of the fine-tuning of the visual model to achieve large-model training for vertical scenarios and its adaptation to vertical scenarios.

[0068] The visual model and language model are fine-tuned synchronously, and the prompt word labels recognized by the visual model image content are transformed and processed to achieve alignment of the generated effects.

[0069] Glossary of terms:

[0070] LoRA (Low-Rank Adaptation) is a technique for model fine-tuning, especially in large pre-trained models. It achieves adaptive adjustment of the model by introducing low-rank matrices in specific parts of the model instead of retraining the entire model. This approach can significantly reduce the demand for computing resources while maintaining or improving the performance of the model.

[0071] Prompt technology is the process of creating prompts or guiding the output of a language model like ChatGPT. It allows users to control the output of the model and generate text that meets their specific needs.

[0072] The prompt words should contain at least the following four key information: task, requirement, instruction (constraint rules or conditions) and role; among them,

[0073] Task: ChatGPT is required to complete a task and generate content that is clear and concise.

[0074] Requirements: Explain the background requirements of the task.

[0075] Instructions: (Constraints or conditions) Instructions that ChatGPT should follow when generating text, satisfying certain conditions or rules to prevent it from being too divergent.

[0076] Role: The role that the model should play when generating text, which can be professional, emotional, or identity-based.

[0077] The present invention is fully described for a clearer disclosure, and the prior art is not listed one by one.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents; it is obvious for those skilled in the art to combine multiple technical solutions of the present invention. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention. The technical contents not described in detail in the present invention are all known technologies.

Claims

1. A vertical domain adaptation method for a large smart home model based on low-rank matrix decomposition, characterized by: The method comprises the following steps: Step 1: Prepare vertical domain data; vertical domain data includes corpus, images and models; Step 2: extract image label data; Step 3: Clean the image label data; Step 4: label the vertical scene data; Step 5: Fine-tune the visual LoRA model; Step 6: Wordify the label data prompts.

2. According to claim 1, the vertical domain adaptation method of the smart home large model based on low-rank matrix decomposition is characterized by: In step 1, the corpus, the corpus prompt words used for vertical domain fine-tuning model training, includes question descriptions and content responses; Images, images used for training the vertical scene vision fine-tuning model; Model, a large base model or multimodal vision model used for model fine-tuning.

3. The vertical domain adaptation method of a large smart home model based on low-rank matrix decomposition according to claim 1 is characterized in that: The method is based on a basic large model, which includes a language large model and a visual large model.

4. According to claim 1, the vertical domain adaptation method of the smart home large model based on low-rank matrix decomposition is characterized by: In step 2, the vertical scene image of step 1 is sent to the basic visual model or the multimodal visual model, and the data label content is obtained through image content understanding or prompt word reverse deduction; The rule for arranging the data label content is that the one with the highest weight is placed first, and then the one with the highest weight is placed last. Identify the keywords of the vertical scene in the current image data.

5. According to claim 1, the vertical domain adaptation method of the smart home large model based on low-rank matrix decomposition is characterized by: In step three, the data labels that have achieved image content recognition are cleaned to remove data labels that are considered to be interference or to be ignored.

6. The vertical domain adaptation method of a large smart home model based on low-rank matrix decomposition according to claim 1 is characterized by: In step 2, data extraction is performed on the kitchen data scene; In step 4, the label data is cleaned or cleared according to step 3; the content that is considered important or needs to be identified is re-labeled and sorted; It customizes the data labels extracted from the kitchen data scene in step 2. For example, it customizes the originally identified "Refrigerator" to "Haier_regrigerator".

7. The vertical domain adaptation method of a large smart home model based on low-rank matrix decomposition according to claim 1 is characterized by: In step five, fine-tuning model training is performed on the vertical scene data that has been re-labeled in step four.

8. The vertical domain adaptation method of a large smart home model based on low-rank matrix decomposition according to claim 1 is characterized by: In step six, the newly added or adjusted data labels in step four are lexicalized, and based on the keywords that appear in step four, descriptions and questions about the current vertical domain scenario are developed, that is, the step language LoRA model is fine-tuned.

9. The vertical domain adaptation method of a large smart home model based on low-rank matrix decomposition according to claim 1, characterized in that: Through fine-tuning of the visual model, the language model is aligned with the fine-tuning effect of the visual model to achieve large-scale model training and vertical-domain scene adaptation; The visual model and language model are fine-tuned synchronously, and the prompt word labels recognized by the visual model image content are transformed and processed to achieve alignment of the generated effects.

10. The vertical domain adaptation method of a large smart home model based on low-rank matrix decomposition according to claim 1, characterized in that: The prompt words include key information; key information includes tasks, requirements, instructions and roles; among them, Tasks,require ChatGPT to complete the set tasks, generate content and make presentations; Requirements, explain the background requirements of the task. Instructions, which ChatGPT follows when generating text, satisfying set conditions or rules; Role, the role that the model is set to play when generating text.

Images