An in-vehicle scene slot extraction method and system, electronic equipment and storage medium

Abstract

The application discloses a kind of vehicle-mounted scene slot extraction method, system, electronic equipment and storage medium, comprising: extracting vehicle-mounted application page information, vehicle-mounted application page information is encapsulated as prior knowledge;Construct prompt word engineering, using lora method is fine-tuned to large language model, set training set and test set, select fine-tuning base model, carry out model training and debugging parameter;Load fine-tuned fine-tuning base model, in combination with prompt word engineering, based on large language model carries out entity slot extraction, utilizes Skip-gram model to calculate semantic similarity, to the slot score extracted is weighted, set threshold value is filtered, obtains final slot information.The application can not only make driver more conveniently navigate, entertainment and other operations by improving language understanding ability under vehicle-mounted scene, but also can provide more personalized and intelligent service under the premise of ensuring safety.

Description

TECHNICAL FIELD

[0001] The present application relates to a slot extraction method, system, electronic device and storage medium, in particular to a vehicle scene slot extraction method, system, electronic device and storage medium. BACKGROUND

[0002] In the automobile cabin dialogue scene, the user's voice instruction is often related to the screen display content, especially in the fields of navigation and entertainment. For example: the user is on the navigation page, and the page shows multiple navigation destinations. At this time, the user may say "navigate to Jinqiao Road Five Intersection" or "add Xiyuan Southwest Gate as a passing point". In this case, whether the entity slot can be successfully extracted from the user query is very important for subsequent voice hits. However, because the data volume of place names and song names is large, unenumerable, and the ways of saying are various, it is very challenging for traditional voice slot extraction. The existing technology cannot meet people's requirements and needs to be improved. SUMMARY

[0003] The purpose of the present application is to provide a vehicle scene slot extraction method, system, electronic device and storage medium, which solves the problem that the voice slot extraction in the prior art cannot meet the application of vehicle voice scene.

[0004] The present application provides the following scheme:

[0005] A vehicle scene slot extraction method based on a large language model, applied to a vehicle scene slot extraction system, comprising:

[0006] Extracting vehicle application page information, the vehicle application page information including vehicle screen information, screen application and page control, and encapsulating the vehicle application page information as priori knowledge;

[0007] Building a prompt word project, using a lora method to fine-tune a large language model, setting a training set and a test set, selecting a fine-tuning base model, training the model and adjusting parameters;

[0008] Loading the fine-tuned fine-tuning base model, combining the prompt word project, extracting entity slots based on the large language model, calculating semantic similarity using the Skip-gram model, weighting the extracted slot scores, setting a threshold for filtering, and obtaining the final slot information.

[0009] Further, the vehicle application page information is extracted, the vehicle application page information includes vehicle screen information, screen application and page control, and the vehicle application page information is encapsulated as priori knowledge, which further comprises:

[0010] The vehicle screen information of the vehicle screen in different positions in the vehicle is recognized, and the vehicle screen information includes the main driver screen information, the co-driver screen information, or the rear row screen information.

[0011] The current application type of each screen is determined, and the application type includes a navigation type, an entertainment type, and a communication type.

[0012] The page control is extracted from the application of each screen, and the page control includes a location selection control, a song interaction control, and a contact selection control.

[0013] The vehicle screen information, the current application type, and the page control are structured and packaged to form a priori knowledge base based on the vehicle scene.

[0014] Further, the vehicle screen information of the vehicle screen in different positions in the vehicle is recognized, and the vehicle screen information includes the main driver screen information, the co-driver screen information, or the rear row screen information, and further includes:

[0015] The display content of the vehicle screen information is monitored and captured in real time by using the operating system interface of the vehicle equipment, the vehicle screen information is analyzed by using the image recognition technology, and the vehicle screen information and the application type are recognized;

[0016] The text information in the application interface is analyzed by using the natural language processing technology, the page control is extracted, the extracted vehicle screen information, the application type, and the page control are organized and stored according to the predefined data structure, and a structured priori knowledge base is formed.

[0017] Further, the prompt word engineering is constructed, the large language model is fine-tuned by using the lora method, the training set and the test set are set, the fine-tuned base model is selected, the model training and the parameter debugging are performed, and further include:

[0018] The prompt word engineering is constructed based on the priori knowledge, the large language model is positioned and the task is described by using the Few-shot and / or Instruct-Prompt natural language processing technology;

[0019] The selected large language model is fine-tuned by using the lora method, and the fine-tuning process includes setting the training set and the test set, selecting the corresponding fine-tuned base model according to the training set and the test set, performing the model training and the parameter debugging;

[0020] The fine-tuned fine-tuned base model is loaded, the content of the prompt word engineering is input, the entity slot is extracted by using the large language model, the entity slot in the user instruction is extracted and output in combination with the vehicle cabin dialogue scene.

[0021] Further, the entity slot in the user instruction is extracted and output, and further includes:

[0022] The word vector is obtained by using the Skip-gram model, the cosine similarity is calculated as the semantic similarity, and the final slot score is obtained according to the weighted result of the semantic similarity.

[0023] Further, the selected large language model is fine-tuned by using the lora method, and the fine-tuning process includes setting a training set and a test set, selecting a corresponding fine-tuning base model according to the training set and the test set, performing model training and parameter debugging, and further including:

[0024] The weight matrix of the selected base model is rank decomposed to identify and optimize the low-rank structure of the model parameters, and a rank decomposition matrix is defined, which is used to represent the low-rank part of the base model weight matrix.

[0025] An adapter layer is introduced in the base model, the adapter layer is combined with the rank decomposition matrix to form a first rank decomposition matrix;

[0026] The adapter layer is trained using the training set, the gradient of the adapter layer is calculated through a back propagation algorithm, and the rank decomposition matrix is updated to generate a second rank decomposition matrix;

[0027] The performance of the fine-tuned base model is evaluated using the test set, and the second rank decomposition matrix is continuously iterated to complete the fine-tuning of the model.

[0028] A vehicle-mounted scene slot extraction system based on a large language model is used to implement the vehicle-mounted scene slot extraction method based on a large language model, and includes:

[0029] A priori knowledge encapsulation module is used to extract vehicle-mounted application page information, including vehicle-mounted screen information, screen applications and page controls, and encapsulate the vehicle-mounted application page information as priori knowledge.

[0030] A model training and parameter debugging module is used to construct a prompt word project, fine-tune a large language model using a lora method, set a training set and a test set, select a fine-tuning base model, perform model training and parameter debugging.

[0031] A slot information calculation module is used to load the fine-tuned fine-tuning base model, combine the prompt word project, extract entity slots based on a large language model, calculate semantic similarity using a Skip-gram model, weight the extracted slot scores, set a threshold for filtering, and obtain the final slot information.

[0032] An intelligent cockpit is provided with the vehicle-mounted scene slot extraction system based on a large language model, and the vehicle-mounted scene slot extraction method based on a large language model is executed.

[0033] An electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method.

[0034] A computer-readable storage medium stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the method.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] The present application significantly improves the accuracy of semantic slot extraction in the vehicle scene by encapsulating the vehicle scene information as prior knowledge into the prompt word engineering of the large language model. By analyzing the screen page information at different positions, such as the main driver screen, the co-driver screen and the rear entertainment screen, the user's possible interactive intent and demand can be accurately captured. By recognizing the text and controls displayed on the vehicle screen, a comprehensive vehicle scene knowledge base is constructed, providing rich context information for subsequent language models.

[0037] The present application adopts prior encapsulation technology to convert the extracted vehicle scene information into a format that can be understood and processed by the large language model. Through information extraction, semantic understanding and knowledge representation in natural language processing, combined with prompt word engineering, the large language model can more accurately identify and extract keywords and slots in user instructions. Not only does this improve the accuracy of slot extraction, but also makes the entire interaction process smoother and more natural.

[0038] In summary, by improving the language understanding ability in the vehicle scene, the present application not only enables the driver to more conveniently perform navigation, entertainment and other operations, but also provides more personalized and intelligent services under the premise of ensuring safety. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0040] Figure 1 is a flowchart of the vehicle scene slot extraction method based on the large language model.

[0041] Figure 1AIs the flow chart of the optimization technical solution of steps S11 to S14.

[0042] Figure 1B Is the flow chart of the optimization technical solution of steps S21 to S23.

[0043] Figure 1C Is the flow chart of the optimization technical solution of steps S221 to S224.

[0044] Figure 2 Is the architecture diagram of the vehicle-mounted scene slot extraction system based on a large language model.

[0045] Figure 3 Is the embodiment of the present application in a specific application scenario.

[0046] Figure 4 Is the structural schematic diagram of an electronic device. DETAILED DESCRIPTION

[0047] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0048] As Figure 1 The large language model-based vehicle-mounted scene slot extraction method shown in the figure is applied to a vehicle-mounted scene slot extraction system and includes the following steps.

[0049] Step S1, extract vehicle-mounted application page information, the vehicle-mounted application page information includes vehicle-mounted screen information, screen application and page control, and the vehicle-mounted application page information is packaged as priori knowledge; wherein the vehicle-mounted screen information can include screen position information, which refers to which screen in the vehicle needs to be obtained or operated in a certain scheme, for example, the screen position information can include the main driver screen, the co-driver screen, the rear screen, etc.

[0050] Step S2, build a prompt word project, fine-tune the large language model using the lora method, set the training set and test set, select the fine-tuned base model, perform model training and debug parameters;

[0051] Step S3, load the fine-tuned fine-tuned base model, combine the prompt word project, perform entity slot extraction based on the large language model, calculate the semantic similarity using the Skip-gram model, weight the extracted slot score, set the threshold for filtering, and obtain the final slot information.

[0052] In steps S1 to S3, the slot refers to a key information segment that needs to be extracted in an intent recognition or entity recognition task. The slot score refers to a slot position confidence score, which is a measure of the accuracy of the model in identifying a certain slot.

[0053] In the optimization scheme provided in steps S1 to S3, by extracting and packaging the vehicle application page information as prior knowledge, the system can more accurately understand the user's instructions and context environment, enhancing the context understanding ability. The fine-tuning process enables the large language model to better adapt to the language characteristics in the vehicle scene, improving the model's understanding and processing ability for specific tasks, optimizing the model performance. By using the Skip-gram model and semantic similarity calculation, the system can more accurately identify and evaluate key information segments (slots), and assign a confidence score to each identified entity, significantly improving the accuracy of entity recognition. By setting a threshold to filter the slot scores, the system can exclude low-confidence recognition results, ensuring that the output slot information has high accuracy, achieving result filtering and precision control.

[0054] In summary, the optimization scheme of steps S1 to S3 improves the response quality of the vehicle system to user instructions, allowing users to enjoy a more intelligent, accurate, and personalized interaction experience, providing an efficient, accurate, and user-friendly processing flow for vehicle interaction systems, significantly improving the intelligence level and user experience of the vehicle system.

[0055] As shown in Figure 1A In the optimization scheme of step S1, the vehicle application page information is extracted, including vehicle screen information, screen application, and page control. The vehicle application page information is packaged as prior knowledge, further including:

[0056] Step S11, identify the vehicle screen information of the vehicle screen at different positions in the vehicle, including the main driver screen information, the co-driver screen information, or the rear screen information;

[0057] Step S12, determine the current application type of each screen, including navigation type, entertainment type, and communication type;

[0058] Step S13, extract the page control from the application of each screen, including the location selection control, the song interaction control, and the contact selection control;

[0059] Step S14, structure and package the vehicle screen information, the current application type, and the page control to form a prior knowledge base based on the vehicle scene.

[0060] The optimization technical solution provided by steps S11 to S14 can intelligently identify and process information of different screens in a vehicle-mounted environment, and effectively integrate the information into a prior knowledge base of a large language model, realizing functions such as vehicle-mounted screen information identification, application type determination, page control extraction and structured packaging, improving the interaction accuracy of vehicle-mounted devices, accurately identifying and packaging vehicle-mounted screen information, so that the system can more accurately understand the user's voice instructions, thereby providing more accurate feedback and operation, integrating information of different screens and application types, so that the vehicle-mounted system can provide more personalized and context-related services, enhancing the user's interaction experience, and the structured prior knowledge base enables the large language model to more efficiently utilize vehicle-mounted scene data, reducing the demand for computing resources, improving processing speed, and realizing the utilization rate of resources of vehicle-mounted devices, and expanding the scope of vehicle-mounted applications, the constructed prior knowledge base is not limited to current vehicle-mounted applications, but can be extended to other intelligent devices and scenarios, having good universality and expansibility.

[0061] Preferably, in the optimization technical solution of step S1, the vehicle-mounted screen information of the vehicle-mounted screen in different positions in the vehicle includes the main driver screen information, the co-driver screen information or the rear row screen information, and further includes:

[0062] The display content of the vehicle-mounted screen information is monitored and captured in real time by using the operating system interface of the vehicle-mounted device, and the vehicle-mounted screen information is analyzed by using image recognition technology to identify the vehicle-mounted screen information and the application type;

[0063] The text information in the application interface is analyzed by using natural language processing technology to extract the page control, and the extracted vehicle-mounted screen information, application type and page control are organized and stored according to a predefined data structure to form a structured prior knowledge base.

[0064] The optimization technical solution provided by the embodiment is different from the optimization technical solution provided by steps S11 to S14, and the optimization technical solution of the embodiment is mainly based on natural language processing technology, uses the operating system interface of the vehicle-mounted device and advanced image and natural language processing technology to realize real-time monitoring, analysis and semantic understanding of the vehicle-mounted screen information. The system can quickly respond to changes in screen content by monitoring the vehicle-mounted screen in real time to provide real-time data for subsequent processing, and through semantic analysis, the vehicle-mounted voice system can more deeply understand the user's possible intentions and needs, thereby providing more accurate services, and using the structured prior knowledge base makes information retrieval and use more efficient, facilitating subsequent data processing and decision support.

[0065] As Figure 1BAs shown, in the optimization technical solution of step S2, the construction prompt word engineering uses the lora method to fine-tune the large language model, sets the training set and the test set, selects the fine-tuning base model, performs model training and parameter debugging, and further includes:

[0066] Step S21, based on prior knowledge, the prompt word engineering is constructed, and the Few-shot and / or Instruct-Prompt natural language processing technology is used to position the role and describe the task of the large language model;

[0067] Step S22, fine-tune the selected large language model using the lora method, the fine-tuning process includes setting the training set, the test set, selecting the corresponding fine-tuning base model according to the training set and the test set, performing model training and parameter debugging;

[0068] Step S23, load the fine-tuned fine-tuning base model, input the content of the prompt word engineering, use the large language model to perform entity slotting, extract the entity slot in the user instruction and output in combination with the vehicle cabin dialogue scene.

[0069] For example, based on information extraction technology (Information Extraction, IE), the large language model is used to perform entity slotting,

[0070] For example, in step S23, the entity slot in the user instruction is extracted and output, further including:

[0071] The word vector is obtained by using the Skip-gram model, the cosine similarity is calculated as the semantic similarity, and the final slot score is obtained according to the weighted result of the semantic similarity.

[0072] The optimization technical solution of steps S21 to S23 improves the accuracy of understanding and executing user instructions in the vehicle cabin dialogue scene. Based on prior knowledge, the Few-shot and / or Instruct-Prompt technology is used to position the role and describe the task of the large language model to help the model better understand the specific task to be processed. The LoRA method is used to fine-tune the selected large language model, including setting the training set and the test set, selecting the fine-tuning base model, and performing model training and parameter debugging to adapt to specific vehicle scenes. The fine-tuned model is loaded, the content of the constructed prompt word engineering is input, the large language model is used to perform entity slotting, the entity slot in the user instruction is extracted, and the information extraction technology is used in combination with the word vector and cosine similarity calculated by the Skip-gram model to weight the semantic similarity of the slot to determine the final slot score.

[0073] The optimization scheme of steps S21 to S23 enhances the model's understanding of the vehicle conversation scene, enabling it to more accurately identify the user's instruction intent. The LoRA fine-tuning method optimizes the model's parameters, making it more suitable for specific vehicle applications and improving the model's performance and accuracy in real-world scenarios. It can accurately extract key information such as locations and songs from user instructions, providing more precise services to users and enhancing the understanding of word meanings and contexts, thereby improving the accuracy and reliability of slot extraction. This significantly improves the interaction quality of the vehicle system, allowing users to enjoy a smoother and more natural conversation experience.

[0074] In this embodiment, Few-shot technology is a machine learning paradigm in the field of natural language processing, enabling models to learn new tasks such as text classification or entity recognition with only a small amount of labeled data, making it suitable for data-scarce scenarios.

[0075] Instruct-Prompt technology is a natural language processing technique that guides language models to perform specific tasks by providing explicit instructions or prompts. By incorporating task requirements directly as part of the input, it guides the model to generate the expected output, significantly improving the model's understanding and execution accuracy for the task.

[0076] LoRA (Low-Rank Adaptation) is a technique for fine-tuning large pre-trained models by introducing low-rank structures into the weight matrices of the model. This reduces the computational resource requirements while maintaining the model's performance on specific tasks, making it suitable for model matching implementations.

[0077] Fine-tuning a base model involves further training a pre-trained large language model for specific tasks or application scenarios. By fine-tuning the base model, it learns specific features and patterns related to the task, improving its performance on that task.

[0078] Skip-gram model is a prediction model in the Word2Vec algorithm used to generate word embeddings. It learns vector representations of words by predicting the context of a given word. In the Skip-gram model, given a target word, the model needs to predict the surrounding context words. This model can capture semantic and syntactic relationships between words, generating high-quality word vectors, and is widely used in various natural language processing tasks.

[0079] As Figure 1CAs shown, in the optimization scheme of step S22, the selected large language model is fine-tuned using the lora method, and the fine-tuning process includes setting a training set and a test set, selecting a corresponding fine-tuning base model according to the training set and the test set, performing model training and adjusting parameters, and further including:

[0080] In step S221, the weight matrix of the selected base model is rank decomposed to identify and optimize the low-rank structure of the model parameters, and a rank decomposition matrix is defined, which is used to represent the low-rank part of the base model weight matrix.

[0081] In step S222, an adapter layer is introduced into the base model, the adapter layer is combined with the rank decomposition matrix to form a first rank decomposition matrix; the introduction of the adapter layer in step S222 allows the (fine-tuning) base model to be adjusted for specific tasks while keeping most of the original parameters unchanged.

[0082] In step S223, the adapter layer is trained using the training set, and the gradient of the adapter layer is calculated through the backpropagation algorithm, and the rank decomposition matrix is updated to generate a second rank decomposition matrix; the generation of the second rank decomposition matrix helps to further optimize the model parameters to adapt to the task.

[0083] In step S224, the performance of the fine-tuned base model is evaluated using the test set, and the second rank decomposition matrix is continuously generated to complete the fine-tuning of the model.

[0084] Steps S221 to S224 use the base model to effectively optimize the parameters of the large pre-trained model and adjust it to specific tasks, using the concepts of low-rank decomposition and adapter layers, reducing the consumption of computing resources while improving the performance of the model on specific tasks. Through low-rank decomposition, the number of parameters that need to be trained is reduced, improving the efficiency of parameter updates, and the introduction of adapter layers and rank decomposition matrices reduces the demand for computing resources, making the fine-tuning process more efficient and quickly adapting to new tasks without the need to retrain the entire model. Through continuous iteration and optimization of the backpropagation algorithm, the performance of the model on specific tasks has been significantly improved. It also significantly improves the generalization ability, uses the test set to evaluate and adjust the model, ensuring that the model not only performs well on the training set, but also maintains high generalization ability on unseen data.

[0085] As Figure 2 The vehicle-mounted scene slot extraction system based on a large language model is used to implement the vehicle-mounted scene slot extraction method based on a large language model, and includes:

[0086] The prior knowledge encapsulation module is configured to extract vehicle-mounted application page information, the vehicle-mounted application page information including vehicle-mounted screen information, screen applications and page controls, and encapsulate the vehicle-mounted application page information as prior knowledge.

[0087] The model training and parameter debugging module is configured to construct a prompt word project, fine-tune a large language model using a lora method, set a training set and a test set, select a fine-tuning base model, perform model training and parameter debugging.

[0088] The slot information calculation module is configured to load the fine-tuned fine-tuning base model, combine the prompt word project, extract entity slots based on the large language model, calculate semantic similarity using a Skip-gram model, weight the extracted slot scores, set a threshold for filtering, and obtain final slot information.

[0089] The above-described embodiments of the system are merely illustrative, for example: wherein each functional module, unit or subsystem in the system can or can not be physically separated, or can or can not be a physical unit, i.e., can be located in the same place or distributed to multiple different systems and their subsystems or modules. A person skilled in the art can select part or all of the functional modules, units or subsystems to achieve the purpose of the embodiments of the present application according to actual needs, and those of ordinary skill in the art can understand and implement without creative labor.

[0090] As Figure 4 shown in the embodiments of the application in specific application scenarios,

[0091] In view of the excellent performance of large language models in recent natural language processing tasks, the embodiments of the present application aim to extract current car scene information (such as screen page information of different positions and different applications) as prior knowledge, encapsulate it into the prompt word project of the large language model, and then use the large language model to extract semantic slots to improve the accuracy of car scene slot extraction.

[0092] One: extract application page information and encapsulate it as prior knowledge:

[0093] 1. Extract application page information;

[0094] 2. Extract screen position information;

[0095] 3. Vehicle scene information processing and encapsulation;

[0096] Example:

[0097] Input: application information, screen position information

[0098] Output: [

[0100] {

[0101] "screen_position": "main driver",

[0102] "screen_app": "map",

[0103] "page_control": ["Xizhimen", "Wudaokou", "Tsinghua University"]

[0104] },

[0105] {

[0106] "screen_position": "back seat",

[0107] "screen_app": "entertainment",

[0108] "page_control": ["Qilixiang", "Qinghuapoci", "Night Music", "Qianli Zhixia"]

[0109] }

[0110] In a real vehicle environment, there are usually multiple screens such as the main driver screen and the back seat screen displaying at the same time. At this time, the position information of the screen, the page application, and the page control name usually have important reference value.

[0111] For example: the current main driver screen is in navigation application, and the location control on the page includes "Xizhimen, Wudaokou, Tsinghua University", and the back seat screen is in entertainment application, and the song control on the page includes "Qilixiang, Qinghuapoci, Night Music, Qianli Zhixia". According to the navigation application of the main driver screen, we can infer that the user is more likely to issue a navigation intent, and the verbs involved include "navigation, departure, go, to" and so on.

[0112] In this step, based on the three dimensions of screen position, current application, and page control, the page information is structured and packaged.

[0113] Two: Based on prior knowledge, construct a scientific and reasonable prompt word engineering, which integrates Few-shot + Instruct idea:

[0114] 1. Role positioning and task description of large language model;

[0115] 2. Give an example to help the large language model understand the task goal;

[0116] 3. Input prior knowledge and user instructions into the large language model;

[0117] Example:

[0118] Input: Prior knowledge of vehicle dialogue scene output in the first step

[0119] Output:

[0120] You: Assuming you are a car-mounted intelligent voice assistant, the following dialogues are based on the car cockpit dialogue scenario. The goal is to extract entity slots in user instructions and output based on the provided scenario information. Here's an example: Given that the main driver screen application is a map, and the display is Xizhimen, Wudaokou, and Tsinghua University. The user instruction is: Navigate me to Wudaokou. Qwen1.5: You: Good, please follow this approach to extract entity slots in user instructions. Given that the rear screen application is entertainment, and the display is Qilixiang, Qinghuapai, Night Music, and Qianli Zhiwai. The user instruction is: Listen to the original song of the singer A for Qianli Zhiwai. Qwen1.5:

[0121] 1)Zero-shot:

[0122] Zero-shot is a method of directly telling the large model the task target. In this case, the model only infers how to respond through the task description or question. The advantage of this method is high flexibility, and the disadvantage is that the model may not accurately grasp the subtle differences or specific requirements of the task.

[0123] This prompt construction idea is simple and direct, suitable for simple tasks with short token input, such as classification or slotting. However, the task in this study involves rich background information and complex output, so pure Zero-shot is not suitable.

[0124] 2)Few-shot:

[0125] Few-shot learning is to help the model understand the task by providing a small number of labeled samples (i.e. examples), which is equivalent to demonstrating to the LLM to follow the above template to play word chain. These examples usually include input and expected output, so that the model can better understand the nature and requirements of the task. Since the model first sees good examples, it can better understand human intentions and what type of answer is required. Therefore, Few-shot learning often has better performance than zero-shot learning. However, the cost is to consume more tokens, and when the input and output text are long, it may reach the context length limit.

[0126] This prompt construction idea is to help the model understand the task through a small number of examples. However, the task in this study involves rich background information, and the input and output are complex, so simple and direct examples are difficult for the large model to accurately understand the task target.

[0127] 3)Instruct-shot:

[0128] Instruct Prompt is to directly describe the task we want in language, and let the model understand what we want to do. This method is more effective in most cases, reducing the length of the context and the cost of model reasoning. Through the description of the application scenario, task target, output guide, etc., the large model can accurately and completely understand the task. However, the effect of this method is not ideal, especially for large models with insufficient understanding ability.

[0129] For GPT4, a model with strong understanding ability, the difference between Zero-shot+Instruct and Few-shot+Instruct is not big. However, for Llama2-70B, a model with slightly poor understanding ability, the effect of the prompt engineering combined with Few-shot and Instruct is obviously better, and the effect is more stable. It is obviously helpful for large models to accurately understand tasks and backgrounds. Therefore, the present application combines the Few-shot and Instruct-Prompt ideas to construct its own prompt engineering.

[0130] Three: based on the prompt engineering, the lora method is used to fine-tune the large model:

[0131] 1. Prepare the training set and test set;

[0132] 2. Select a suitable fine-tuning base model Qwen1.5-7B;

[0133] 3. Select a suitable fine-tuning method, finally select lora;

[0134] 4. Training and parameter tuning;

[0135] 5. Model saving;

[0136] Example:

[0137] Input: Qwen1.5-7B open source model, training set based on task prompt engineering

[0138] Output: fine-tuned Qwen1.5-7B model

[0139] Adapt Tuning: This method adds an Adapter layer in the model, freezes the original parameters during fine-tuning, and only updates the Adapter layer. Specifically, it inserts parameters for downstream tasks in each layer of the pre-trained model, i.e. Adapter module, freezes the model body during fine-tuning, and only trains task-specific parameters.

[0140] Prefix Tuning: The traditional fine-tuning paradigm uses pre-trained models to fine-tune different downstream tasks, and a fine-tuned model weight is saved for each task. On the one hand, fine-tuning the entire model takes a long time; on the other hand, it also occupies a lot of storage space. Based on the above two points, Prefix Tuning proposes to fix the pre-trained LM (Language Model) and add trainable, task-specific prefixes to the LM. In this way, different prefixes can be saved for different tasks, and the fine-tuning cost is small.

[0141] LoRA fine-tuning: Full-parameter fine-tuning is costly, Adapter Tuning has training and inference delays, and PrefixTuning is difficult to train and reduces the effective text length in the original training data. To improve these shortcomings, LoRA (Low-Rank Adaptation) is proposed as a fine-tuning method. LoRA is actually an improved Adapt Tuning method. If a large model is used to map data to a high-dimensional space for processing, it is assumed that a complex large model is not needed when processing a small task, and it may only be necessary to solve within a certain subspace. Therefore, when optimizing the parameters of a subspace, the performance of the full parameter optimization can be achieved to a certain level (such as 90% accuracy). At this time, the rank of the subspace parameter matrix can be referred to as the intrinsic rank of the current problem to be solved. The pre-training model itself implicitly reduces the intrinsic rank, and after fine-tuning for a specific task, the weight matrix in the model actually has a lower intrinsic rank. At the same time, the simpler the downstream task, the lower the intrinsic rank. Therefore, although the parameter matrix of the weight update is randomly projected to a smaller subspace, it can still effectively learn. It can be understood that for specific downstream tasks, these weight matrices do not require full rank, and the rank decomposition matrix of the dense layer can be optimized to indirectly train some dense layers in the neural network during the adaptation process, thereby achieving the effect of fine-tuning by optimizing only the rank decomposition matrix of the dense layer.

[0142] P-tuning: P-tuning is mainly proposed to solve the problem that the construction method of the Prompt of the large model seriously affects the effect of the downstream task. For example: GPT-3 uses a manually constructed template for in-context learning, but the manually designed template is very sensitive to changes. Adding or removing a word, or changing the position will cause a large change. At the same time, the cost of recent automatic search template work is also relatively high. The results of this discrete token search may not be optimal, resulting in unstable performance. Based on this, the industry proposes P-tuning, which designs a continuous differentiable virtual token. This method converts the Prompt into a learnable Embedding (word embedding) layer and uses a PLM+LSTM method to process the Prompt Embedding.

[0143] P-tuning v2: Deep prompt optimization improves Prompt Tuning and P-Tuning as a general solution across scales and NLU tasks.

[0144] Fine-tuning is a powerful tool that enables large pre-trained models to adapt to specific tasks and application scenarios. Proper selection and application of fine-tuning strategies are crucial for achieving efficient and effective model performance. Fine-tuning is actually an instance of transfer learning, where a pre-trained model (usually trained on large general-purpose datasets) is used as a starting point for a specific task. This approach enables efficient learning even for tasks with small datasets.

[0145] The choice of fine-tuning method depends on multiple factors, including the complexity of the task, the amount of available data, computational resources, and desired performance. For complex tasks that require fine-grained control, P-Tuning v2 or LSTM-based P-Tuning is more suitable due to the more learnable parameters and deeper structure of P-Tuning and P-Tuning v2. For cases where computational resources and data resources are limited, methods such as LoRA or Adapter Tuning are suitable. The LoRA method uses a parallel structure that does not affect the computational efficiency of the original base large model, and the trained "plug-in parameters" can be directly combined with the large model parameters during inference. Overall, the LoRA method is the most optimal in terms of computational efficiency and fine-tuning effect, so it is the preferred method for fine-tuning large models based on LoRA. In this study, due to limited device and data resources, and the single task domain and clear target, there is no need for strong generalization ability at the moment, so the LoRA fine-tuning method is chosen.

[0146] Four: Large language model based on prompt engineering for query slot extraction:

[0147] 1 Load the fine-tuned Qwen1.5-7B;

[0148] 2 Input prompt engineering;

[0149] 3 Use large language model for entity slot extraction.

[0150] You: Assuming you are a car-mounted intelligent voice assistant, the following dialogues are based on the car cockpit dialogue scenario. The goal is to extract entity slots in user instructions and output based on the provided scenario information. Here's an example: Given that the main driver screen application is a map, and the display is Xizhimen, Wudaokou, and Tsinghua University. The user instruction is: Navigate me to Wudaokou. Qwen1.5: Understand, I will extract entity slots in user instructions and output based on the car cockpit dialogue scenario. Please provide the user's instruction and related scenario information. You: Good, please follow this approach to extract entity slots in user instructions. Given that the rear screen application is entertainment, and the display is Qilixiang, Qinghuapai, Night Music, and Qianli Zhiwai. The user instruction is: Listen to the original song of the singer A for Qianli Zhiwai. Qwen1.5: Scene information extraction result: Entity slot "Song" value "Qianli Zhiwai"

[0151] Five: Slot extraction result score calculation and threshold filtering

[0152] 1 Based on prior knowledge of the scene, use the Skip-gram model to obtain word vectors, then calculate the cosine similarity as the semantic similarity. After weighting the semantic similarity, the final slot score extracted by the large language model is obtained.

[0153] 2 Set the threshold value and perform threshold filtering on the slot score.

[0154] Example:

[0155] Input: {Location: The Forbidden City, Page Control Name: The Palace Museum, score: 0.8}

[0156] Output: {query: Go to the Forbidden City, Entity Slot: The Forbidden City}

[0157] The Skip-gram model is a method used in Word2Vec for learning word vectors. It trains word vector representations by predicting the context words of a target word. Here is a detailed introduction to the Skip-gram model:

[0158] Objective: Given a target word, predict its context words.

[0159] Model structure:

[0160] Input layer: A one-hot encoded target word vector.

[0161] Projection layer: A hidden layer, usually a linear transformation, that maps the one-hot vector to a lower-dimensional word vector space.

[0162] Output layer: Through the softmax function, output the probability distribution of predicted context words.

[0163] Training process:

[0164] Input: Target word and context word pairs, forming training samples.

[0165] Loss function: Usually use negative log likelihood (Negative Log Likelihood), that is, maximize the probability of predicting the context word of the target word.

[0166] Optimization: Use optimization methods such as gradient descent to update model parameters.

[0167] The Skip-gram model is an efficient and effective word vector learning method that generates high-quality word vectors by capturing the context relationships of words. Although there are some shortcomings, such as the selection of context window size and the demand for computing resources, it performs well in practical applications and is widely used in natural language processing tasks.

[0168] As shown in Figure 4 , the present application provides a large language model-based vehicle scene slot extraction method and system, and also provides a corresponding intelligent cockpit, electronic device and storage medium:

[0169] An intelligent cockpit, wherein the intelligent cockpit is provided with the large language model-based vehicle scene slot extraction system, and executes the large language model-based vehicle scene slot extraction method.

[0170] An electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method.

[0171] A computer readable storage medium stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the method.

[0172] Figure 4 A structural schematic diagram of a computer device provided by an example embodiment of the present application is shown. The computer device can be a device for executing the information recommendation method and / or the model training method provided by the present application. Specifically:

[0173] The computer device 1000 comprises a central processing unit (CPU) 1001, a system memory 1004 comprising a random access memory (RAM) 1002 and a read-only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the central processing unit 1001. The computer device 1000 further comprises a basic input / output system (I / O system) 1006 to help transfer information between various devices in the computer, and a mass storage device 1007 for storing an operating system 1013, application programs 1014 and other program modules 1015.

[0174] The basic input / output system 1006 comprises a display 1008 for displaying information and an input device 1009 such as a mouse, a keyboard or the like for inputting information by a user. The display 1008 and the input device 1009 are both connected to the central processing unit 1001 through an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 can also comprise the input / output controller 1010 for receiving and processing input from a keyboard, a mouse, an electronic stylus or other devices. Similarly, the input / output controller 1010 also provides output to a display screen, a printer or other types of output devices.

[0175] The mass storage device 1007 is connected to the central processing unit 1001 through a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer readable media provide nonvolatile storage for the computer device 1000. That is, the mass storage device 1007 can include a computer readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0176] The computer readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory or other solid state memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Random access memory can include resistance random access memory (ReRAM) and dynamic random access memory (DRAM). Of course, the computer readable medium is not limited to the foregoing examples. The system memory 1004 and the mass storage device 1007 described above can be collectively referred to as memory.

[0177] According to various embodiments of the present application, the computer device 1000 can also operate in connection with a remote computer through a network such as the Internet. That is, the computer device 1000 can connect to the network 1012 through the network interface unit 1011 connected to the system bus 1005, or can be connected to other types of networks or remote computer systems (not shown) using the network interface unit 1011.

[0178] The above-described memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0179] In an optional embodiment, a computer device is provided, which includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the information recommendation method and / or the model training method as described above.

[0180] In an optional embodiment, a computer readable storage medium is provided, which stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the information recommendation method and / or the model training method as described above.

[0181] Optionally, the computer readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a solid state disk (SSD), an optical disk, etc. Among them, the random access memory can include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The above-mentioned application embodiment serial number is only for description, not representing the pros and cons of the embodiments.

[0182] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0183] The present application also provides a computer readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the information recommendation method and / or the model training method provided by the above-mentioned method embodiments.

[0184] The present application also provides a computer program product, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the information recommendation method and / or the model training method as described above.

[0185] It should be understood that "multiple" mentioned in this article refers to two or more than two. "And / or" describes the association between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship.

[0186] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program to instruct related hardware, and the program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0187] The above is only an optional embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0188] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0189] In addition, those skilled in the art can understand that although some embodiments described herein include certain features included in other embodiments rather than other features, the combination of features of different embodiments means that it is within the scope of the present application and forms different embodiments. For example: any one of the embodiments claimed in the claims can be used in any combination of the embodiments of the present application.

[0190] In the description of the present application, the description of the terms "one embodiment", "example", "specific example" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0191] In addition, the technical solutions of each embodiment of the present application can be combined with each other, but it must be based on the fact that a person of ordinary skill in the art can realize it, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor is it within the protection scope required by the present application.

[0192] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for extracting slots in vehicle scenes based on a large language model, applied to a vehicle scene slot extraction system, characterized in that, include: Extract in-vehicle application page information, which includes in-vehicle screen information, screen applications, and page controls, and encapsulate the in-vehicle application page information as prior knowledge; Based on prior knowledge, a prompt word project is constructed. The LoRa method is used to fine-tune the large language model. Training and test sets are set, a fine-tuning base model is selected, and model training and parameter debugging are performed. The fine-tuned base model is loaded, and entity slots are extracted based on the large language model in conjunction with the prompt word engineering. The semantic similarity is calculated using the Skip-gram model, the extracted slot scores are weighted, and a threshold is set for filtering to obtain the final slot information.

2. The method for extracting vehicle scene slots based on a large language model according to claim 1, characterized in that, The step of extracting in-vehicle application page information, which includes in-vehicle screen information, screen applications, and page controls, further includes encapsulating the in-vehicle application page information as prior knowledge. The vehicle screen information can be identified from different locations within the vehicle, including information from the driver's screen, passenger's screen, or rear-seat screen. Determine the current application type for each screen, including navigation, entertainment, and communication types; Extract page controls from the application on each screen, including location selection controls, song interaction controls, and contact selection controls; The information on the in-vehicle screen, the current application type, and page controls are encapsulated in a structured manner to form a prior knowledge base based on the in-vehicle scenario.

3. The method for extracting vehicle scene slots based on a large language model according to claim 2, characterized in that, The method of identifying in-vehicle screen information in different locations within the vehicle, including driver's side screen information, passenger side screen information, or rear-seat screen information, further includes: By utilizing the operating system interface of the vehicle equipment, the display content of the vehicle screen is monitored and captured in real time. Image recognition technology is used to analyze the vehicle screen information and identify the vehicle screen information and application type. Natural language processing technology is used to perform semantic analysis on the text information in the application interface, extract page controls, and organize and store the extracted in-vehicle screen information, application type and page controls according to a predefined data structure to form a structured prior knowledge base.

4. The method for extracting vehicle scene slots based on a large language model according to claim 1, characterized in that, The aforementioned project for constructing prompt words based on prior knowledge, using the LoRa method to fine-tune the large language model, setting up training and testing sets, selecting a fine-tuning base model, performing model training and parameter debugging, further includes: Based on prior knowledge, a prompt word engineering is constructed, and natural language processing techniques such as Few-shot and / or Instruct-Prompt are used to perform role localization and task description on a large language model. The LoRa method is used to fine-tune a selected large language model. The fine-tuning process includes setting up a training set and a test set, selecting a corresponding fine-tuning base model based on the training set and test set, training the model, and adjusting the parameters. Load the fine-tuned base model, input the content of the prompt word project, use the large language model to perform entity slot extraction, combine the in-vehicle cockpit dialogue scenario, extract the entity slots in the user commands and output them.

5. The method for extracting vehicle scene slots based on a large language model according to claim 4, characterized in that, The step of extracting and outputting the physical slot from the user command further includes: Word vectors are obtained using the Skip-gram model, and cosine similarity is calculated as semantic similarity. The final slot score is obtained based on the weighted result of semantic similarity.

6. The method for extracting vehicle scene slots based on a large language model according to claim 4, characterized in that, The fine-tuning of the selected large language model using the LoRa method includes setting up training and testing sets, selecting a corresponding fine-tuning base model based on the training and testing sets, training the model, and adjusting parameters. Further, it includes: The weight matrix of the selected base model is decomposed into rank to identify and optimize the low-rank structure of the model parameters. A rank decomposition matrix is ​​defined to represent the low-rank part of the weight matrix of the base model. An adapter layer is introduced into the base model, and the adapter layer is combined with the rank decomposition matrix to form the first rank decomposition matrix; The adapter layer is trained using the training set, and the gradient of the adapter layer is calculated using the backpropagation algorithm. The rank decomposition matrix is ​​then updated to generate the second rank decomposition matrix. The performance of the fine-tuned pedestal model is evaluated using a test set, and the second-rank decomposition matrix is ​​continuously generated iteratively to complete the fine-tuning of the model.

7. A vehicle scene slot extraction system based on a large language model, used to implement the vehicle scene slot extraction method based on a large language model as described in any one of claims 1 to 6, characterized in that, include: The prior knowledge encapsulation module is used to extract in-vehicle application page information, which includes in-vehicle screen information, screen applications, and page controls, and encapsulate the in-vehicle application page information into prior knowledge. The model training and parameter debugging module is used to build prompt word projects based on prior knowledge, fine-tune the large language model using the LoRa method, set training and test sets, select fine-tuning base models, and perform model training and parameter debugging. The slot information calculation module is used to load the fine-tuned base model, combine it with the prompt word engineering, extract entity slots based on the large language model, calculate semantic similarity using the Skip-gram model, weight the extracted slot scores, set a threshold for filtering, and obtain the final slot information.

8. An intelligent cockpit, characterized in that, The intelligent cockpit is equipped with the in-vehicle scene slot extraction system based on a large language model as described in claim 7, and executes the in-vehicle scene slot extraction method based on a large language model as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: The system includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, It stores a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the method according to any one of claims 1 to 6.

Images