Vehicle-mounted scene slot extraction method and system, electronic equipment and storage medium

Abstract

The invention discloses a vehicle-mounted scene slot extraction method and system, electronic equipment and a storage medium, and the method comprises the steps: extracting vehicle-mounted application page information, and packaging the vehicle-mounted application page information into priori knowledge; a prompt word project is constructed, a lora method is used for fine tuning of the large language model, a training set and a test set are set, a fine tuning base model is selected, and model training and parameter debugging are carried out; loading the fine-tuned fine-tuning base model, carrying out entity slot extraction on the basis of a large language model in combination with a cue word project, calculating semantic similarity by utilizing a Skip-gram model, weighting extracted slot scores, setting a threshold value for filtering, and obtaining final slot information. According to the invention, by improving the language understanding ability in the vehicle-mounted scene, not only can a driver carry out navigation, entertainment and other operations more conveniently, but also more personalized and intelligent services can be provided on the premise of ensuring safety.

Description

Technical Field

[0001] The present invention relates to a slot extraction method, system, electronic device and storage medium, and in particular to a vehicle-mounted scene slot extraction method, system, electronic device and storage medium. Background Art

[0002] In the car cockpit dialogue scene, the user's voice commands are often related to the content displayed on the screen, especially in the fields of navigation and entertainment. For example: the user is on the navigation page, and the page displays multiple navigation destinations. At this time, the user is likely to say "navigate to Wudaokou, Jingkai Road" or "add Ruyuan Southwest Gate as a waypoint"; the user is on the entertainment page, and the page displays multiple singers and songs. In this case, whether the entity slot can be successfully extracted from the user query is very important for subsequent voice hits. However, because the data volume of place names and song names is huge, cannot be enumerated, and the statements are complicated, it is very challenging for traditional voice slot extraction. The existing technology cannot meet people's requirements and is in urgent need of improvement. Summary of the invention

[0003] The purpose of the present invention is to provide a vehicle-mounted scene slot extraction method, system, electronic device and storage medium, aiming at the problem that the voice slot extraction in the prior art cannot meet the application of vehicle-mounted voice scenes, and solving the shortcomings of the prior art.

[0004] The present invention provides the following scheme:

[0005] A vehicle scene slot extraction method based on a large language model, applied to a vehicle scene slot extraction system, comprising:

[0006] Extracting in-vehicle application page information, the in-vehicle application page information including in-vehicle screen information, screen applications and page controls, and encapsulating the in-vehicle application page information as prior knowledge;

[0007] Build a prompt word project, use the Lora method to fine-tune the large language model, set the training set and test set, select the fine-tuning base model, perform model training, and debug parameters;

[0008] Load the fine-tuned base model, combine it with the prompt word project, extract entity slots based on the large language model, use the Skip-gram model to calculate the semantic similarity, weight the extracted slot scores, set the threshold for filtering, and obtain the final slot information.

[0009] Furthermore, the extracting of the in-vehicle application page information, the in-vehicle application page information including the in-vehicle screen information, the screen application and the page control, and encapsulating the in-vehicle application page information as prior knowledge, further comprises:

[0010] Identify vehicle screen information of vehicle screens at different locations in the vehicle, wherein the vehicle screen information includes main driver screen information, co-driver screen information or rear screen information;

[0011] Determine a current application type of each screen, wherein the application type includes a navigation type, an entertainment type, and a communication type;

[0012] Extracting page controls from the application on each screen, wherein the page controls include a location selection control, a song interaction control, and a contact selection control;

[0013] The in-vehicle screen information, current application type and page controls are structured and encapsulated to form a priori knowledge base based on the in-vehicle scenario.

[0014] Furthermore, the identification of vehicle screen information of vehicle screens at different positions in the vehicle, wherein the vehicle screen information includes main driver screen information, co-driver screen information or rear screen information, further includes:

[0015] Utilize the operating system interface of the vehicle-mounted device to monitor and capture the display content of the vehicle-mounted screen information in real time, analyze the vehicle-mounted screen information through image recognition technology, and identify the vehicle-mounted screen information and application type;

[0016] Through natural language processing technology, semantic analysis is performed on the text information in the application interface, and page controls are extracted. The extracted vehicle screen information, application type and page controls are organized and stored according to a predefined data structure to form a structured prior knowledge base.

[0017] Furthermore, the construction of the prompt word project, using the lora method to fine-tune the large language model, setting the training set and the test set, selecting the fine-tuning base model, performing model training and debugging parameters, further includes:

[0018] Build prompt word engineering based on prior knowledge, and use Few-shot and / or Instruct-Prompt natural language processing technology to perform role positioning and task description for large language models;

[0019] The selected large language model is fine-tuned using the LoRa method. The fine-tuning process includes setting a training set and a test set, selecting a corresponding fine-tuning base model according to the training set and the test set, and performing model training and parameter debugging.

[0020] Load the fine-tuned base model, input the content of the prompt word project, use the large language model to perform entity slot extraction, combine the in-vehicle cockpit dialogue scene, extract the entity slots in the user's instructions and output them.

[0021] Furthermore, the extracting and outputting the entity slot in the user instruction further includes:

[0022] The Skip-gram model is used to obtain word vectors, and the cosine similarity is calculated as the semantic similarity. The final slot score is obtained based on the weighted result of the semantic similarity.

[0023] Furthermore, the lora method is used to fine-tune the selected large language model. The fine-tuning process includes setting a training set and a test set, selecting a corresponding fine-tuning base model according to the training set and the test set, performing model training and debugging parameters, and further includes:

[0024] Performing rank decomposition on the weight matrix of the selected base model to identify and optimize the low-rank structure of the model parameters, and defining a rank decomposition matrix, wherein the rank decomposition matrix is ​​used to represent the low-rank part of the weight matrix of the base model;

[0025] Introducing an adapter layer into the base model, combining the adapter layer with the rank decomposition matrix to form a first rank decomposition matrix;

[0026] The adapter layer is trained using the training set, and the gradient of the adapter layer is calculated by the back propagation algorithm, and the rank decomposition matrix is ​​updated to generate a second rank decomposition matrix;

[0027] The test set is used to evaluate the performance of the fine-tuned base model, and the second rank decomposition matrix is ​​continuously iterated to complete the fine-tuning of the model.

[0028] A vehicle-mounted scene slot extraction system based on a large language model, used to implement the vehicle-mounted scene slot extraction method based on a large language model, comprising:

[0029] A priori knowledge encapsulation module is used to extract in-vehicle application page information, wherein the in-vehicle application page information includes in-vehicle screen information, screen applications and page controls, and encapsulate the in-vehicle application page information as priori knowledge;

[0030] Model training and parameter debugging module, used to build prompt word projects, use the lora method to fine-tune the large language model, set training sets and test sets, select fine-tuning base models, perform model training and debug parameters;

[0031] The slot information calculation module is used to load the fine-tuned base model after fine-tuning, combine the prompt word project, extract entity slots based on the large language model, calculate the semantic similarity using the Skip-gram model, weight the extracted slot scores, set thresholds for filtering, and obtain the final slot information.

[0032] A smart cockpit is provided with the vehicle-mounted scene slot extraction system based on a large language model, and executes the vehicle-mounted scene slot extraction method based on a large language model.

[0033] An electronic device comprises: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method.

[0034] A computer-readable storage medium stores a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the method.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] The present invention encapsulates automobile scene information as prior knowledge into the prompt word project of the large language model, thereby significantly improving the accuracy of semantic slot extraction in the vehicle-mounted scene. By analyzing the screen page information in different positions, such as the main driver's screen, the co-driver's screen, and the rear entertainment screen, the possible interaction intentions and needs of the user can be accurately captured. By identifying the text and controls displayed on the vehicle-mounted screen, a comprehensive automobile scene knowledge base is constructed, providing rich contextual information for subsequent language models.

[0037] The present invention adopts a priori encapsulation technology to convert the extracted automobile scene information into a format that can be understood and processed by the large language model. Through multiple links such as information extraction, semantic understanding and knowledge representation in natural language processing, combined with prompt word engineering, it can guide the large language model to more accurately identify and extract keywords and slots in user instructions, which not only improves the accuracy of slot extraction, but also makes the entire interaction process smoother and more natural.

[0038] In summary, the present invention improves the language comprehension ability in the vehicle-mounted scenario, which not only enables the driver to perform operations such as navigation and entertainment more conveniently, but also provides more personalized and intelligent services while ensuring safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of the vehicle scene slot extraction method based on the large language model.

[0041] Figure 1AIt is a flow chart of the optimization technical solution from step S11 to step S14.

[0042] Figure 1B It is a flow chart of the optimization technical solution from step S21 to step S23.

[0043] Figure 1C It is a flow chart of the optimization technical solution from step S221 to step S224.

[0044] Figure 2 This is the architecture diagram of the vehicle scene slot extraction system based on the large language model.

[0045] Figure 3 It is an implementation method of the embodiment of the present invention in a specific application scenario.

[0046] Figure 4 It is a structural diagram of an electronic device. DETAILED DESCRIPTION

[0047] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] like Figure 1 The vehicle scene slot extraction method based on the large language model is applied to the vehicle scene slot extraction system, including:

[0049] Step S1, extracting the in-vehicle application page information, wherein the in-vehicle application page information includes the in-vehicle screen information, screen applications and page controls, and encapsulating the in-vehicle application page information as prior knowledge; wherein the in-vehicle screen information may include screen position information, and the screen position information refers to which screen in the vehicle needs to be obtained or operated in a certain scenario, for example, the screen position information may include the main driver's screen, the co-driver's screen, the rear screen, and so on.

[0050] Step S2, build a prompt word project, use the lora method to fine-tune the large language model, set the training set and test set, select the fine-tuning base model, perform model training and debug parameters;

[0051] Step S3, load the fine-tuned base model after fine-tuning, combine the prompt word project, extract entity slots based on the large language model, use the Skip-gram model to calculate the semantic similarity, weight the extracted slot scores, set the threshold for filtering, and obtain the final slot information.

[0052] In steps S1 to S3, slot refers to the key information fragment that needs to be extracted in the intent recognition or entity recognition task. The slot score refers to the slot position confidence score, which is a measure to evaluate the model's accuracy in recognizing a certain slot.

[0053] In the optimization technical solution provided in steps S1 to S3, by extracting and encapsulating the in-vehicle application page information as prior knowledge, the system can more accurately understand the user's instructions and context environment, and enhance the context understanding ability. The fine-tuning process enables the large language model to better adapt to the language characteristics in the in-vehicle scenario, improves the model's understanding and processing capabilities for specific tasks, and optimizes the model performance. Using the Skip-gram model and semantic similarity calculation, the system can more accurately identify and evaluate key information fragments (slots), and assign a confidence score to each recognized entity, which significantly improves the accuracy of entity recognition. By setting a threshold to filter the slot score, the system can exclude low-confidence recognition results, ensure that the output slot information has high accuracy, and achieve result filtering and precision control.

[0054] To sum up, the optimization technical solution of steps S1 to S3 improves the response quality of the in-vehicle system to user commands, allowing users to enjoy a more intelligent, accurate and personalized interactive experience, and provides an efficient, accurate and user-friendly processing flow for the in-vehicle interactive system, significantly improving the intelligence level of the in-vehicle system and user experience.

[0055] like Figure 1A As shown, in the optimization technical solution of step S1, the in-vehicle application page information is extracted, and the in-vehicle application page information includes in-vehicle screen information, screen applications and page controls, and the in-vehicle application page information is encapsulated as prior knowledge, further comprising:

[0056] Step S11, identifying the in-vehicle screen information of the in-vehicle screens at different positions in the vehicle, wherein the in-vehicle screen information includes the main driver's screen information, the co-driver's screen information or the rear screen information;

[0057] Step S12, determining the current application type of each screen, wherein the application type includes a navigation type, an entertainment type, and a communication type;

[0058] Step S13, extracting page controls from the application on each screen, wherein the page controls include a location selection control, a song interaction control, and a contact selection control;

[0059] Step S14, encapsulating the in-vehicle screen information, the current application type and the page controls in a structured manner to form a priori knowledge base based on the in-vehicle scenario.

[0060] The optimization technical solution provided by steps S11 to S14 can intelligently identify and process information from different screens in the vehicle environment, and effectively integrate this information into a priori knowledge base of a large language model, realizing functions such as vehicle-mounted screen information recognition, application type determination, page control extraction and structured encapsulation, and improving the interaction accuracy of vehicle-mounted devices. By accurately identifying and encapsulating vehicle-mounted screen information, the system can more accurately understand the user's voice commands, thereby providing more accurate feedback and operations, and integrating information from different screens and application types, so that the vehicle-mounted system can provide more personalized and context-related services, enhancing the user's interactive experience. The structured priori knowledge base enables the large language model to more efficiently utilize vehicle-mounted scene data, reduce the demand for computing resources, improve processing speed, achieve resource utilization of vehicle-mounted equipment, and expand the scope of vehicle-mounted applications. The constructed priori knowledge base is not limited to current vehicle-mounted applications, but can also be extended to other smart devices and scenarios, and has good versatility and extensibility.

[0061] Preferably, in the optimization technical solution of step S1, the identifying of the vehicle screen information of the vehicle screens at different positions in the vehicle, the vehicle screen information including the main driver screen information, the co-driver screen information or the rear screen information, further comprises:

[0062] Utilize the operating system interface of the vehicle-mounted device to monitor and capture the display content of the vehicle-mounted screen information in real time, analyze the vehicle-mounted screen information through image recognition technology, and identify the vehicle-mounted screen information and application type;

[0063] Through natural language processing technology, semantic analysis is performed on the text information in the application interface, and page controls are extracted. The extracted vehicle screen information, application type and page controls are organized and stored according to a predefined data structure to form a structured prior knowledge base.

[0064] The optimization technology solution provided in this embodiment is different from the optimization technology solution provided in step S11 to step S14. The optimization technology solution in this embodiment is mainly based on natural language processing technology, using the operating system interface of the vehicle-mounted equipment and advanced image and natural language processing technology to achieve real-time monitoring, parsing and semantic understanding of the vehicle-mounted screen information. In this embodiment, by real-time monitoring of the vehicle-mounted screen, the system can quickly respond to changes in the screen content and provide real-time data for subsequent processing. Through semantic analysis, the vehicle-mounted voice system can more deeply understand the user's possible intentions and needs, thereby providing more accurate services, and using a structured prior knowledge base to make the retrieval and use of information more efficient, which is convenient for subsequent data processing and decision support.

[0065] like Figure 1BAs shown, in the optimization technical solution of step S2, the construction of the prompt word project, the use of the lora method to fine-tune the large language model, setting the training set and the test set, selecting the fine-tuning base model, performing model training and debugging parameters further include:

[0066] Step S21, constructing a prompt word project based on prior knowledge, and performing role positioning and task description on the large language model through Few-shot and / or Instruct-Prompt natural language processing technology;

[0067] Step S22, using the lora method to fine-tune the selected large language model, the fine-tuning process includes setting a training set and a test set, selecting a corresponding fine-tuning base model according to the training set and the test set, performing model training and debugging parameters;

[0068] Step S23, load the fine-tuned base model, input the content of the prompt word project, use the large language model to perform entity slot extraction, combine the vehicle cockpit dialogue scene, extract the entity slot in the user command and output it.

[0069] For example, based on information extraction technology (IE), a large language model is used to perform entity extraction.

[0070] Exemplarily, in step S23, extracting and outputting the physical slot in the user instruction further includes:

[0071] The Skip-gram model is used to obtain word vectors, and the cosine similarity is calculated as the semantic similarity. The final slot score is obtained based on the weighted result of the semantic similarity.

[0072] The optimization technical scheme of step S21 to step S23 improves the accuracy of understanding and executing user instructions in the vehicle cockpit dialogue scenario. Based on prior knowledge, the Few-shot and / or Instruct-Prompt technology is used to perform role positioning and task description on the large language model to help the model better understand the specific task to be processed. The selected large language model is fine-tuned by the LoRA method, including setting a training set and a test set, selecting a fine-tuning base model, and performing model training and parameter debugging to adapt to specific vehicle scenarios. The fine-tuned model is loaded, and the constructed prompt word engineering content is input. The large language model is used to perform entity extraction and extract entity slots in user instructions. Then, the information extraction technology is used, combined with the word vector obtained by the Skip-gram model and the cosine similarity calculation, and the slots are weighted by semantic similarity to determine the final slot score.

[0073] The optimization technical solution of steps S21 to S23 enhances the model's ability to understand in-vehicle dialogue scenarios, enabling it to more accurately identify the intention of user commands. The parameters of the model are optimized through the LoRA fine-tuning method, making it more suitable for specific in-vehicle applications, improving the performance and accuracy of the model in actual scenarios. It can accurately extract key information from user commands, such as entities such as locations and songs, to provide users with more precise services, enhance the understanding of word meanings and contexts, thereby improving the accuracy and reliability of slot extraction, and significantly improving the interaction quality of the in-vehicle system, allowing users to enjoy a smoother and more natural dialogue experience.

[0074] In this embodiment, Few-shot technology is a machine learning paradigm in the field of natural language processing, which enables the model to learn new tasks, such as text classification or entity recognition, with only a small amount of labeled data, and is suitable for scenarios where data is scarce.

[0075] Instruct-Prompt technology is a natural language processing technology that guides the language model to perform specific tasks by giving clear instructions or prompts. It can significantly improve the accuracy of the model's understanding and execution of tasks by directly using task requirements as part of the input to guide the model to generate the expected output.

[0076] LoRA (Low-Rank Adaptation) is a technology for fine-tuning large pre-trained models. By introducing a low-rank structure in the model's weight matrix to update parameters, it can reduce the demand for computing resources while maintaining the model's performance on specific tasks. It is suitable for model matching implementation.

[0077] Fine-tuning the base model refers to the process of further training a pre-trained large language model for a specific task or application scenario. By fine-tuning the base model, specific features and patterns related to the task can be learned, thereby improving its performance on the task.

[0078] The Skip-gram model is a prediction model in the Word2Vec algorithm that is used to generate word embeddings. It can learn the vector representation of a word by predicting the context of a word. In the Skip-gram model, given a target word, the model needs to predict the context words around it. This model can capture the semantic and grammatical relationships between words, generate high-quality word vectors, and is widely used in various natural language processing tasks.

[0079] like Figure 1CAs shown, in the optimization technical solution of step S22, the lora method is used to fine-tune the selected large language model. The fine-tuning process includes setting a training set and a test set, selecting a corresponding fine-tuning base model according to the training set and the test set, performing model training and debugging parameters, and further includes:

[0080] Step S221, performing rank decomposition on the weight matrix of the selected base model to identify and optimize the low-rank structure of the model parameters, and defining a rank decomposition matrix, wherein the rank decomposition matrix is ​​used to represent the low-rank part of the weight matrix of the base model;

[0081] Step S222, introducing an adapter layer into the base model, combining the adapter layer with the rank decomposition matrix to form a first rank decomposition matrix; introducing the adapter layer in step S222 allows the (fine-tuning) base model to be adjusted for specific tasks while keeping most of the original parameters unchanged.

[0082] Step S223, use the training set to train the adapter layer, calculate the gradient of the adapter layer through the back propagation algorithm, and update the rank decomposition matrix to generate a second rank decomposition matrix; generating the second rank decomposition matrix helps to further optimize the model parameters to adapt to the task.

[0083] Step S224, using the test set to evaluate the performance of the fine-tuned base model, and continuously iterating to generate a second rank decomposition matrix to complete the fine-tuning of the model.

[0084] Steps S221 to S224 use the base model to effectively optimize the parameters of the large pre-trained model and adjust it to specific tasks. The concepts of low-rank decomposition and adapter layer are used to reduce the consumption of computing resources while improving the performance of the model on specific tasks. Through low-rank decomposition, the number of parameters that need to be trained is reduced, and the efficiency of parameter updates is improved. The introduction of adapter layers and rank decomposition matrices reduces the demand for computing resources, making the fine-tuning process more efficient and able to quickly adapt to new tasks without retraining the entire model. Through continuous iteration and optimization of the back-propagation algorithm, the performance of the model on specific tasks has been significantly improved. It also significantly improves the generalization ability. The test set is used to evaluate and adjust the model to ensure that the model not only performs well on the training set, but also maintains a high generalization ability on unseen data.

[0085] like Figure 2 The vehicle-mounted scene slot extraction system based on a large language model is used to implement the vehicle-mounted scene slot extraction method based on a large language model, including:

[0086] A priori knowledge encapsulation module is used to extract in-vehicle application page information, wherein the in-vehicle application page information includes in-vehicle screen information, screen applications and page controls, and encapsulate the in-vehicle application page information as priori knowledge;

[0087] Model training and parameter debugging module, used to build prompt word projects, use the lora method to fine-tune the large language model, set training sets and test sets, select fine-tuning base models, perform model training and debug parameters;

[0088] The slot information calculation module is used to load the fine-tuned base model after fine-tuning, combine the prompt word project, extract entity slots based on the large language model, calculate the semantic similarity using the Skip-gram model, weight the extracted slot scores, set thresholds for filtering, and obtain the final slot information.

[0089] The implementation methods of the system described above are merely illustrative, for example, the various functional modules, units or subsystems in the system may or may not be physically separated, or may or may not be physical units, that is, they may be located in the same place, or they may be distributed to multiple different systems and their subsystems or modules. Those skilled in the art may select some or all of the functional modules, units or subsystems according to actual needs to achieve the purpose of the embodiments of the present invention, and those of ordinary skill in the art may understand and implement the above-mentioned situations without creative work.

[0090] like Figure 4 The embodiment of the present invention shown is implemented in a specific application scenario.

[0091] In view of the recent excellent performance of the large language model in many natural language processing tasks, the embodiment of the present invention aims to extract the current car scene information (such as screen page information of different locations and different applications) as prior knowledge, encapsulate it into the prompt word project of the large language model, and then use the large language model to extract semantic slots to improve the accuracy of vehicle scene slot extraction.

[0092] 1: Extract application page information and encapsulate it as prior knowledge:

[0093] 1Extract application page information;

[0094] 2 Extract screen position information;

[0095] 3. Vehicle scene information processing and packaging;

[0096] Example:

[0097] Input: application information, screen location information

[0098] Output:

[0099]

[0100] In a real car environment, there are usually multiple screens displayed at the same time, such as the main driver's screen and the rear screen. At this time, the screen location information, page application, and page control name are usually of great reference value.

[0101] For example, the navigation application is currently being used on the main driving screen, and the location controls displayed on the page include "Xizhimen, Wudaokou, Tsinghua University", and the entertainment application is being used on the rear screen, and the song controls displayed on the page include "Qilixiang, Blue and White Porcelain, Nocturne, Thousands of Miles Away". Based on the navigation application on the main driving screen, we can infer that the user is more likely to have issued a navigation intention, and the verbs involved include "navigate, depart, go, arrive" and so on.

[0102] In this step, the page information is structured and packaged based on three dimensions: screen position, current application, and page controls.

[0103] 2. Construct a scientific and reasonable prompt word project based on prior knowledge. The prompt word project here integrates the Few-shot + Instruct idea:

[0104] 1. Position the role and describe the tasks of the large language model;

[0105] 2. Give an example to help the large language model understand the task goal;

[0106] 3 Input prior knowledge and user instructions into the large language model;

[0107] Example:

[0108] Input: Prior knowledge of the in-vehicle dialogue scenario output from the first step

[0109] Output:

[0110]

[0111] 1) Zero-shot:

[0112] The idea of ​​zero-shot construction is to directly tell the big model the task goal. In this case, the model only infers how to respond based on the task description or question. The advantage of this method is that it is extremely flexible, but the disadvantage is that the model may not accurately grasp the nuances or specific requirements of the task.

[0113] This kind of prompt word construction idea is relatively simple and direct, and is suitable for simple tasks with short token input, such as classification or slot extraction. However, the tasks in this study involve rich background information and complex output, so a simple Zero-shot is not suitable.

[0114] 2) Few-shot:

[0115] Few-shot learning helps the model understand the task by providing a small number of labeled samples (i.e., examples), which is equivalent to demonstrating to the LLM and asking him to perform word chain according to the above template. These examples usually include input and expected output so that the model can grasp the nature and requirements of the task more accurately. For the target task. Since the model first sees good examples, it can better understand human intentions and the criteria for what type of answers are needed. Therefore, Few-shot learning often has better performance than zero-shot learning. However, this comes at the cost of consuming more tokens, and when the input and output texts are long, the context length limit may be reached.

[0116] This idea of ​​constructing prompt words is to use a small number of examples to help the model understand the task. However, the task in this study involves rich background information, and both the input and output are complex. Simple and direct examples make it difficult for a large model to accurately understand the task objectives.

[0117] 3)Instruct-shot:

[0118] Instruct Prompt is to directly describe the task we want in words, so that the model can understand what we want to do. This method is more effective in most cases, reducing the length of context and the cost of model reasoning. By describing the application scenario, task objectives, output guidance, etc., the large model can understand the task as accurately and completely as possible. However, the effect of this method alone is not ideal, especially for large models with weak understanding ability.

[0119] For a model with strong comprehension ability like GPT4, there is little difference between the two methods of Zero-shot+Instruct and Few-shot+Instruct. However, for a model with slightly poor comprehension ability like Llama2-70B, the effect of the prompt word engineering that combines Few-shot and Instruct is significantly better and more stable, which is obviously helpful for large models to accurately understand tasks and backgrounds. Therefore, the present invention combines the ideas of Few-shot and Instruct-Prompt to construct its own prompt word engineering.

[0120] 3: Based on the prompt word engineering, use the lora method to fine-tune the large model:

[0121] 1. Prepare training set and test set;

[0122] 2. Select the appropriate fine-tuning base model Qwen1.5-7B;

[0123] 3. Choose a suitable fine-tuning method, and finally choose Lora;

[0124] 4. Training and parameter adjustment;

[0125] 5. Model saving;

[0126] Example:

[0127] Input: Qwen1.5-7B open source model, training set based on the prompt word project of this task

[0128] Output: Fine-tuned Qwen1.5-7B model

[0129] Adapt Tuning: This method adds an Adapter layer to the model, freezes the original parameters during fine-tuning, and only updates the Adapter layer. Specifically, it inserts parameters for downstream tasks, namely the Adapter module, into each layer of the pre-trained model, freezes the main body of the model during fine-tuning, and only trains task-specific parameters.

[0130] Prefix Tuning: The traditional fine-tuning paradigm uses pre-trained models to fine-tune different downstream tasks. For each task, a copy of the fine-tuned model weights must be saved. On the one hand, fine-tuning the entire model takes a long time; on the other hand, it also takes up a lot of storage space. Based on the above two points, Prefix Tuning proposes a fixed pre-trained LM (Language Model) to add trainable, task-specific prefixes to the LM, so that different prefixes can be saved for different tasks, and the fine-tuning cost is also small.

[0131] LoRA fine-tuning: The cost of fine-tuning all parameters is high, Adapter Tuning has training and inference delays, Prefix Tuning is difficult to train and will reduce the effective text length in the original training data. In order to improve these shortcomings, someone proposed a fine-tuning method called LoRA (Low-Rank Adaptation). LoRA is actually an improved Adapt Tuning method. If a large model maps data to a high-dimensional space for processing, it is assumed here that when processing a small subdivided task, such a complex large model is not needed. It may only be solved within a certain subspace range, so there is no need to optimize the full amount of parameters. Therefore, it can be defined that when optimizing a certain subspace parameter, a certain level of performance (such as 90% accuracy) of full parameter optimization can be achieved, then the rank of this subspace parameter matrix can be called the intrinsic rank corresponding to the current problem to be solved. The pre-trained model itself implicitly reduces the intrinsic rank. When fine-tuned for a specific task, the weight matrix in the model actually has a lower intrinsic rank. At the same time, the simpler the downstream task, the lower the corresponding intrinsic rank. Therefore, although the parameter matrix for weight update is randomly projected to a smaller subspace, it can still be effectively learned. It can be understood that these weight matrices do not require full rank for specific downstream tasks. Some dense layers in the neural network can be indirectly trained by optimizing the rank decomposition matrix of the dense layer that changes during the adaptation process, thereby achieving fine-tuning effects by optimizing only the rank decomposition matrix of the dense layer.

[0132] P-tuning: P-tuning was proposed mainly to solve the following problem: the Prompt construction method of large models seriously affects the effect of downstream tasks. For example, GPT-3 uses artificially constructed templates for incontext learning, but the changes of artificially designed templates are particularly sensitive. Adding a word or removing a word, or changing the position will cause relatively large changes. At the same time, the cost of recent automated template search is also relatively high. The results of the previous discrete token search may not be optimal, resulting in unstable performance. Based on this, the industry proposed P-tuning and designed a continuously differentiable virtual token. This method converts the Prompt into a learnable Embedding layer, and uses the PLM+LSTM method to process the PromptEmbedding layer.

[0133] P-tuning v2: Improves Prompt Tuning and P-Tuning with Deep Prompt Optimization as a general solution across scales and NLU tasks.

[0134] Fine-tuning is a powerful tool that enables large pre-trained models to adapt to specific tasks and application scenarios. Correctly selecting and applying fine-tuning strategies is critical to achieving efficient and effective model performance. Fine-tuning is actually an instance of transfer learning, where a pre-trained model (usually trained on a large general dataset) is used as a starting point for a specific task. This approach enables efficient learning even for tasks with small datasets.

[0135] The choice of fine-tuning method depends on multiple factors, including the complexity of the task, the amount of data available, computing resources, and the expected performance. For complex tasks that require fine-grained control, P-Tuningv2 or LSTM-based P-Tuning is more suitable because P-Tuning and P-Tuning v2 have more learnable parameters and deeper structures. For situations where computing resources and data resources are limited, it is appropriate to choose methods such as LoRA or Adapter Tuning. The LoRA method uses a parallel structure, which will not affect the computational efficiency of the original base large model, and the trained "plug-in parameters" can be directly merged into the large model parameters during inference. Overall, the LoRA method is the best in terms of computational efficiency and fine-tuning effect, so the mainstream first choice is to fine-tune the large model based on LoRA. In this study, because the equipment and data resources are limited, the task field is single and the goal is clear, and there is no need for strong generalization ability for the time being, so the LoRA fine-tuning method is selected.

[0136] 4: Using a large language model based on the prompt word project to analyze the query:

[0137] 1 Load the fine-tuned Qwen1.5-7B;

[0138] 2 Input prompt word engineering;

[0139] 3. Use a large language model to perform entity extraction.

[0140]

[0141]

[0142] 5. Calculation of the score of the groove cutting result and threshold filtering

[0143] 1 Based on the scene prior knowledge, the Skip-gram model is used to obtain word vectors, and then the cosine similarity is calculated as the semantic similarity. After weighting the semantic similarity, the final slot score extracted by the large language model is obtained.

[0144] 2. Set the threshold and perform threshold filtering on the groove score.

[0145] Example:

[0146] Input: {Location: Forbidden City, Page Control Name: Forbidden City Museum, Score: 0.8}

[0147] Output: {query: depart for the Forbidden City, entity slot: Forbidden City}

[0148] The Skip-gram model is a method in Word2Vec for learning word vectors. It trains word vector representation by predicting the context words of a word. The following is a detailed introduction to the Skip-gram model:

[0149] Goal: Given a target word, predict its context words.

[0150] Model structure:

[0151] Input layer: a one-hot encoded target word vector.

[0152] Projection layer: A hidden layer, usually a linear transformation, maps the one-hot vector to a low-dimensional word vector space.

[0153] Output layer: Through the softmax function, the probability distribution of the predicted context words is output.

[0154] Training process:

[0155] Input: target word and context word pairs, forming training samples.

[0156] Loss function: Negative Log Likelihood is usually used, which maximizes the probability of the target word predicting the context word.

[0157] Optimization: Use optimization methods such as gradient descent to update model parameters.

[0158] The Skip-gram model is an efficient and effective word embedding learning method that generates high-quality word embeddings by capturing the contextual relationships of vocabulary. Despite some shortcomings, such as the selection of context window size and computing resource requirements, it performs well in practical applications and is widely used in natural language processing tasks.

[0159] like Figure 4 As shown, the present invention not only provides a vehicle scene slot extraction method and system based on a large language model, but also provides a corresponding intelligent cockpit, electronic equipment and storage medium:

[0160] A smart cockpit is provided with the vehicle-mounted scene slot extraction system based on a large language model, and executes the vehicle-mounted scene slot extraction method based on a large language model.

[0161] An electronic device comprises: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method.

[0162] A computer-readable storage medium stores a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the method.

[0163] Figure 4 A schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application is shown. The computer device may be a device that executes the information recommendation method and / or model training method provided by the present application. Specifically:

[0164] The computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including a random access memory (RAM) 1002 and a read only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the central processing unit 1001. The computer device 1000 also includes a basic input / output system (I / O system) 1006 for facilitating information transmission between various components in the computer, and a large-capacity storage device 1007 for storing an operating system 1013, application programs 1014, and other program modules 1015.

[0165] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 such as a mouse and a keyboard for user inputting information. The display 1008 and the input device 1009 are connected to the central processing unit 1001 through an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include an input / output controller 1010 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, a printer, or other types of output devices.

[0166] The mass storage device 1007 is connected to the central processing unit 1001 through a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer readable media provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include a computer readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0167] Computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash memory or other solid-state storage technologies, CD-ROM, Digital Versatile Disc (DVD) or Solid State Drive (SSD), other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Among them, random access memory may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM). Of course, those skilled in the art will know that computer storage media are not limited to the above. The above-mentioned system memory 1004 and large-capacity storage device 1007 can be collectively referred to as memory.

[0168] According to various embodiments of the present application, the computer device 1000 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1000 can be connected to the network 1012 through the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer systems (not shown).

[0169] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0170] In an optional embodiment, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the information recommendation method and / or model training method as described above.

[0171] In an optional embodiment, a computer-readable storage medium is provided, in which at least one instruction, at least one program, code set or instruction set is stored, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the information recommendation method and / or model training method as described above.

[0172] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0173] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0174] The present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the information recommendation method and / or model training method provided by the above-mentioned method embodiments.

[0175] The present application also provides a computer program product, the computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the information recommendation method and / or model training method as described above.

[0176] It should be understood that the term "plurality" herein refers to two or more than two.

[0177] "or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can be represented by: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0178] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0179] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0180] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the embodiments claimed in the claims can be used in any combination in the embodiments of the present invention.

[0182] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0183] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle scene slot extraction method based on a large language model, applied to a vehicle scene slot extraction system, characterized in that: include: Extracting in-vehicle application page information, the in-vehicle application page information including in-vehicle screen information, screen applications and page controls, and encapsulating the in-vehicle application page information as prior knowledge; Build a prompt word project, use the Lora method to fine-tune the large language model, set the training set and test set, select the fine-tuning base model, perform model training, and debug parameters; Load the fine-tuned base model, combine it with the prompt word project, extract entity slots based on the large language model, use the Skip-gram model to calculate the semantic similarity, weight the extracted slot scores, set the threshold for filtering, and obtain the final slot information.

2. The vehicle scene slot extraction method based on a large language model according to claim 1 is characterized in that: The extracting of the in-vehicle application page information, wherein the in-vehicle application page information includes in-vehicle screen information, screen applications, and page controls, and encapsulating the in-vehicle application page information as prior knowledge, further includes: Identify vehicle screen information of vehicle screens at different locations in the vehicle, wherein the vehicle screen information includes main driver screen information, co-driver screen information or rear screen information; Determine a current application type of each screen, wherein the application type includes a navigation type, an entertainment type, and a communication type; Extracting page controls from the application on each screen, wherein the page controls include a location selection control, a song interaction control, and a contact selection control; The in-vehicle screen information, current application type and page controls are structured and encapsulated to form a priori knowledge base based on the in-vehicle scenario.

3. The vehicle scene slot extraction method based on a large language model according to claim 2 is characterized in that: The in-vehicle screen information of the in-vehicle screens at different positions in the vehicle is identified, wherein the in-vehicle screen information includes the main driver's screen information, the co-driver's screen information or the rear screen information, further includes: Utilize the operating system interface of the vehicle-mounted device to monitor and capture the display content of the vehicle-mounted screen information in real time, analyze the vehicle-mounted screen information through image recognition technology, and identify the vehicle-mounted screen information and application type; Through natural language processing technology, semantic analysis is performed on the text information in the application interface, and page controls are extracted. The extracted vehicle screen information, application type and page controls are organized and stored according to a predefined data structure to form a structured prior knowledge base.

4. The vehicle scene slot extraction method based on a large language model according to claim 1 is characterized in that: The construction of the prompt word project, using the lora method to fine-tune the large language model, setting the training set and the test set, selecting the fine-tuning base model, performing model training and debugging parameters, further includes: Build prompt word engineering based on prior knowledge, and use Few-shot and / or Instruct-Prompt natural language processing technology to perform role positioning and task description for large language models; The selected large language model is fine-tuned using the LoRa method. The fine-tuning process includes setting a training set and a test set, selecting a corresponding fine-tuning base model according to the training set and the test set, and performing model training and parameter debugging. Load the fine-tuned base model, input the content of the prompt word project, use the large language model to perform entity slot extraction, combine the in-vehicle cockpit dialogue scene, extract the entity slots in the user's instructions and output them.

5. The vehicle scene slot extraction method based on a large language model according to claim 4 is characterized in that: The extracting and outputting the entity slot in the user instruction further includes: The Skip-gram model is used to obtain word vectors, and the cosine similarity is calculated as the semantic similarity. The final slot score is obtained based on the weighted result of the semantic similarity.

6. The vehicle scene slot extraction method based on a large language model according to claim 4 is characterized in that: The lora method is used to fine-tune the selected large language model. The fine-tuning process includes setting a training set and a test set, selecting a corresponding fine-tuning base model according to the training set and the test set, performing model training and debugging parameters, and further includes: Performing rank decomposition on the weight matrix of the selected base model to identify and optimize the low-rank structure of the model parameters, and defining a rank decomposition matrix, wherein the rank decomposition matrix is ​​used to represent the low-rank part of the weight matrix of the base model; Introducing an adapter layer into the base model, combining the adapter layer with the rank decomposition matrix to form a first rank decomposition matrix; The adapter layer is trained using the training set, and the gradient of the adapter layer is calculated by the back propagation algorithm, and the rank decomposition matrix is ​​updated to generate a second rank decomposition matrix; The test set is used to evaluate the performance of the fine-tuned base model, and the second rank decomposition matrix is ​​continuously iterated to complete the fine-tuning of the model.

7. A vehicle scene slot extraction system based on a large language model, used to implement the vehicle scene slot extraction method based on a large language model as described in any one of claims 1 to 6, characterized in that: include: A priori knowledge encapsulation module is used to extract in-vehicle application page information, wherein the in-vehicle application page information includes in-vehicle screen information, screen applications and page controls, and encapsulate the in-vehicle application page information as priori knowledge; Model training and parameter debugging module, used to build prompt word projects, use the lora method to fine-tune the large language model, set training sets and test sets, select fine-tuning base models, perform model training and debug parameters; The slot information calculation module is used to load the fine-tuned base model after fine-tuning, combine the prompt word project, extract entity slots based on the large language model, calculate the semantic similarity using the Skip-gram model, weight the extracted slot scores, set thresholds for filtering, and obtain the final slot information.

8. A smart cockpit, characterized in that: The smart cockpit is provided with the vehicle-mounted scene slot extraction system based on the large language model as described in claim 7, and executes the vehicle-mounted scene slot extraction method based on the large language model as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: It stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the method described in any one of claims 1 to 6.

Images