Voice interaction sales agent based on large language model and sales vehicle

Abstract

The invention discloses a voice interaction sales agent based on a large language model and a sales vehicle. The voice interaction sales agent is constructed through the following steps: step 1, collecting multi-modal data; step 2, constructing a retrieval enhancement generation framework; 3, finely adjusting the large language model; 4, designing an intelligent voice interaction strategy; 5, multi-modal data processing and enhancement are carried out; step 6, designing a functional module; and step 7, system integration and deployment: the voice interaction sales agent is in butt joint with the vehicle-mounted hardware equipment so as to acquire multi-modal data, and all the function modules in the step 6 are integrated. According to the voice interaction sales agent, the intelligent level, the sales efficiency and the user experience of the self-service vending vehicle are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent voice interaction technology, and in particular to a voice interaction sales agent and a sales vehicle based on a large language model. Background Art

[0002] With the rapid development of artificial intelligence technology, unmanned vending trucks have gradually entered specific scenarios for pilot operations. However, most traditional unmanned vending trucks rely on preset simple voice recognition and response mechanisms, which have shown many limitations in practical applications. First, traditional systems have limited capabilities in processing multimodal complex information and cannot effectively integrate multimodal data such as voice and vision, resulting in an insufficient understanding of user needs. Secondly, these systems mainly rely on active user input, lack natural interactive capabilities, and are difficult to achieve active sales promotion, resulting in a passive sales process and failure to fully capture the needs of different target groups. In addition, traditional vending trucks lack automatic analysis of sales data and experience accumulation mechanisms, making it difficult to optimize services in long-term operations and unable to fully stimulate users' desire to buy and sense of participation.

[0003] In recent years, the rapid development of large language models (LLMs) has brought new opportunities for intelligent voice interaction. LLMs not only have powerful natural language understanding and generation capabilities, but can also adapt to the knowledge needs of specific fields through fine-tuning, and perform reasoning and execution of complex tasks. Compared with traditional speech recognition systems, the intelligent vending voice interaction sales agent based on LLM can not only understand multimodal complex information, such as combining the user's voice, expression and posture to more deeply capture the user's real needs, but also provide a more humane interactive experience. During the interaction process, LLM can actively guide the conversation, make personalized recommendations and sales promotions according to the preferences and needs of different target groups, and effectively improve the sales conversion rate. More importantly, by integrating the automatic analysis module of sales data, the intelligent vending truck can continuously accumulate experience while optimizing the recommendation strategy and sales plan, further enhancing the service effect.

[0004] Therefore, applying LLM to the voice interactive sales agent of unmanned vending trucks can significantly improve their interactive capabilities and sales results, enabling them to combine rich domain knowledge to provide personalized and in-depth services in specific scenarios. By understanding multimodal complex information, achieving natural interaction and active sales promotion, and automatically analyzing sales data, unmanned vending trucks based on LLM not only surpass traditional models in interactive experience, but also effectively promote corresponding featured products in specific environments, such as cultural and creative products on campus, etc., to stimulate users' enthusiasm for purchase and bring new changes to the operation model of unmanned vending trucks. Summary of the invention

[0005] The purpose of the present invention is to provide a voice interactive sales agent based on a large language model to address the technical problems in the prior art that unified sales trucks lack automatic analysis of sales data and experience accumulation mechanisms, making it difficult to optimize services in long-term operations.

[0006] The technical solution adopted to achieve the purpose of the present invention is:

[0007] A voice interactive sales agent based on a large language model is constructed through the following steps:

[0008] Step 1, multimodal data collection: extensively collect multiple types of data related to the target sales environment, including text data, voice data, image data, and video data;

[0009] Step 2: First, the text data collected in step 1 is cleaned. The cleaned text data is divided into structured knowledge and unstructured knowledge according to content attributes. For structured knowledge, a conceptual model in the field is established to define the categories, attributes and relationships of entities. The structured knowledge is converted into a knowledge graph and stored in a graph database to support fast queries based on entities and relationships. The unstructured knowledge is converted into an embedded vector through an embedding model and stored in a FAISS database for efficient vector similarity search. The graph database and the FAISS database are used to build a knowledge base.

[0010] When the agent is actually running, the user's text input is converted into a vector expression through the embedding model. The vector expression is searched in the knowledge vectors stored in the FAISS database to find the multiple knowledge vectors with the highest matching degree. The text slices related to these knowledge vectors will be recalled and re-ranked by ReRanker. After sorting, the most relevant one or more text slices will be passed to the large language model through the prompt word generator for generation tasks; the structured knowledge in the graph database supports queries through entity relationships, and the query results are also passed to the large language model through the prompt word generator for generation tasks;

[0011] Step 3, fine-tuning the large language model: select a pre-trained large language model, fine-tune the large language model in combination with the knowledge base constructed in step 2, and formulate a task set to evaluate and optimize the large language model, so that it has the knowledge background of a specific sales scenario and can accurately process sales words, product information and multimodal data; the fine-tuned large language model generates relevant responses based on user input;

[0012] Step 4, designing intelligent voice interaction strategies: In diversified sales scenarios, the intelligent voice interaction strategies include noise filtering and prompt word templates of a large language model. The prompt word templates are dynamically adjusted based on sentiment analysis technology. The noise filtering is used for the voice model of step 5. The prompt word templates are stored in a prompt word generator, which interacts with a large language model. When the agent is running, based on the content input by the user, the voice interaction sales agent will combine historical interaction data, user behavior analysis, and the current scenario to adjust its sales strategy and recommended content in real time;

[0013] Step 5: Multimodal data processing and enhancement: Use the multimodal data collected in step 1 to train the visual model and speech model and integrate them with the large language model;

[0014] Step 6, designing function modules, integrating and encapsulating the functions of the function modules to form customized function functions, all function functions are stored in a function library, through function calls, the voice interactive sales agent can perform semantic matching according to the user's semantic needs, select the most reasonable function from the function library, and generate corresponding instructions; the large language model interacts with the sales data automatic analysis module, and the sales data automatic analysis module deeply mines the user's interactive data to accumulate sales experience and optimize sales strategies;

[0015] Step 7, system integration and deployment: The voice interactive sales agent is connected to the vehicle-mounted hardware equipment to collect multimodal data, integrate the functional modules of step 6, and each functional module interacts with the database. The large language model interacts with the database through the formatting and conversion module and the query construction and execution module.

[0016] In the above technical solution, in step 2, structured knowledge includes product attributes, promotional information, and user purchase preferences, which can be directly used in sales scenarios, and unstructured knowledge includes background stories, brand history, and user feedback, which are used to enhance the cultural background understanding and interactive experience of the voice interactive sales agent. The embedding model includes but is not limited to embedding-2.

[0017] In the above technical solution, in step 3, the large language model includes but is not limited to GLM-4.0.

[0018] In the above technical solution, in step 3, a task test set is customized, and the task test set includes user interaction scenario simulation, multiple sales speech scenarios, generation quality assessment, and personalized recommendation scenarios. The task test set is customized through the following steps:

[0019] Step 31, define core tasks: First, determine the tasks that the large language model needs to perform in actual applications. For example, the core tasks in a sales scenario may be product recommendations, answering user questions, pushing promotional information, etc.

[0020] Step 32, design different scenarios: Design different scenarios for each task, such as users actively asking questions or passively accepting recommendations, users being interested in the recommended products or having objections, etc.

[0021] Step 33, write questions and task objectives: Write questions that users may ask and expected outputs of tasks according to different scenarios. The test set should contain a variety of question types, such as open-ended questions, yes-no questions, specific product inquiries, etc.

[0022] Step 34, set evaluation criteria: define clear evaluation criteria for each task, such as accuracy of answer, natural fluency, whether sales targets are achieved, etc.

[0023] In the above technical solution, in step 5, the visual model includes but is not limited to the target detection model of Yolov8, which is trained with pictures with marked target bounding boxes on the basis of Yolov8, so that it can recognize the corresponding targets in the image or video data, and output the text data of the bounding box coordinates and categories, and at the same time integrate it into the prompt word template to communicate with the large language model.

[0024] In the above technical solution, in step 5, the voice model uses but is not limited to whisper. The voice recognition model therein is first used to convert the user's voice data into text, and questions are asked to the large language model. After obtaining the text answer from the large language model, the voice synthesis model in the whisper library is used to synthesize it into voice for playback, thereby realizing voice interaction with the user.

[0025] In the above technical solution, in step 5, during docking and integration, the prompt word generator uses a prompt word template with corresponding placeholders therein, and after obtaining the text converted by the visual model and the speech recognition model, replaces the corresponding placeholders in the template to generate text that actually communicates with the large language model.

[0026] In the above technical solution, in step 6, the functional modules include a personalized recommendation module, a cultural story interpretation module, an activity reminder module, a payment module, and a product order management module.

[0027] In the above technical solution, in step 7, the voice interactive sales agent is connected to the backend management system in the cloud to obtain sales data, user interaction records, and system operation status in real time.

[0028] Another aspect of the present invention also includes a sales vehicle, on which the voice interactive sales agent is mounted.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The present invention has developed a voice-interactive sales agent by introducing a large language model, which can efficiently process complex natural language input in various environments and conduct comprehensive analysis in combination with multimodal data (including voice, images, and videos). This enables the voice-interactive sales agent to not only understand the user's language, but also accurately interpret their expressions, gestures, and behaviors, thereby providing more personalized and accurate product recommendations. Compared with traditional sales trucks that rely on fixed rules and keyword matching, the present invention significantly improves the flexibility and intelligence level of the sales system in processing multimodal information, especially in a variety of scenarios, it can better meet user needs and improve user experience.

[0031] 2. The large language model used in the present invention has natural interactive capabilities and can actively promote products rather than just passively responding to user inquiries. By capturing and analyzing user behavior, emotional state, and needs in real time, the voice interactive sales agent can make customized product recommendations based on different target groups. This proactive sales method not only improves sales results, but also enhances user experience, making sales strategies more targeted and effective, which has significant advantages over the passive sales method of traditional systems.

[0032] 3. The sales data automatic analysis module in the present invention accumulates sales experience and optimizes sales strategies through in-depth mining of user interaction data. The system can analyze sales data in real time, dynamically adjust sales strategies and improve functional modules, thereby improving sales performance and user experience. This automated data analysis capability enables the voice interactive sales agent to continuously optimize according to actual sales conditions, enhancing the flexibility and effectiveness of sales strategies. Compared with traditional sales systems, the automatic analysis and optimization capabilities of the present invention significantly improve the intelligence level and overall efficiency of the sales system.

[0033] In summary, the innovations of the present invention in intelligent voice interaction and multimodal data processing not only significantly improve the sales efficiency of unmanned vending trucks, but also provide users with a more natural and intimate shopping experience. Combined with the characteristics of unmanned vending trucks actively looking for people and intelligently promoting sales, this technology is particularly suitable for flexible mobile sales in open public places such as campuses and parks. The system can effectively carry out sales activities in a variety of environments and provide accurate product recommendations and interactive services. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Shown is a structural diagram of the voice interactive sales agent. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] A voice interactive sales agent based on a large language model is constructed through the following steps:

[0037] Step 1, multimodal data collection:

[0038] According to the functional requirements of the voice interactive sales agent, various types of data related to the target sales environment are widely collected, including text, voice, image and video data, to improve the intelligence level and interaction ability of the voice interactive sales agent. First, collect text data related to the sales scenario, including but not limited to specific brand product information, industry-related promotion strategies, user interaction data, and scenario-related background knowledge. These multi-level data collections ensure that the voice interactive sales agent can master a comprehensive knowledge system, so as to provide accurate and efficient services in actual sales interactions. In addition, special attention is paid to the collection of multimodal information, which includes user voice data (such as Mandarin, dialects, and voices of different age groups), image data (such as faces, body language), and video data (such as user behavior dynamics). These data can help the voice interactive sales agent recognize different voice features (such as Mandarin, dialects, voice differences between the elderly, children, men, and women), user gestures, crowds, behavior dynamics, etc. By integrating these multimodal data, the voice interactive sales agent can not only provide accurate answers and suggestions, but also achieve efficient user interaction and personalized recommendations in complex sales environments, enhancing its overall interactive experience.

[0039] Step 2: Build the Retrieval Augmented Generation (RAG) framework

[0040] First, the text data collected in step 1 is cleaned to remove redundancy, noise and irrelevant content. The cleaned text data is divided into structured knowledge and unstructured knowledge according to content attributes.

[0041] Structured knowledge includes product attributes, promotional information, user purchase preferences, etc., which can be directly used in sales scenarios. For structured knowledge, a conceptual model in the field is established to define the categories, attributes and relationships of entities. For example, entity categories include products, brands, users, promotional activities, etc. Taking the "product" entity as an example, attributes include product name, price, inventory, color, size, material, origin, rating, etc.; relationships include belonging to (brand), associated with (promotional activities), and applicable to (user groups). Convert structured knowledge into a knowledge graph and use a graph database for storage, supporting fast queries based on entities and relationships.

[0042] Unstructured knowledge includes background stories, brand history, user feedback, etc., which are used to enhance the cultural background understanding and interactive experience of voice interactive sales agents. Graph databases (such as Neo4j) are used for storage and management to support fast retrieval and logical reasoning. For unstructured knowledge, an embedding model is used to embed text data and convert text into vectors. These vectors will be stored in the FAISS database for efficient retrieval. The graph database and FAISS database are used to build a knowledge base.

[0043] When the agent is actually running, the user's text input is converted into a vector expression through the embedding model. The vector expression is searched and matched in the knowledge vectors stored in the FAISS database to find the knowledge vectors with the highest matching degree (the first K, K is an adjustable parameter). The text slices related to these knowledge vectors will be recalled and re-ranked by ReRanker. The most relevant text slices after sorting (can be the first N results, N is an adjustable parameter) will be passed to the large language model through the prompt word generator for generation tasks; the structured knowledge in the graph database supports queries through entity relationships, and the query results are passed to the large language model through the prompt word generator for generation tasks;

[0044] Step 3, fine-tune the large language model:

[0045] Select a pre-trained large language model (including but not limited to GLM-4.0, etc.), and fine-tune the large language model in combination with the knowledge base constructed above, so that it has the knowledge background of specific sales scenarios and can accurately process sales words, product information and multimodal data. During the fine-tuning process, formulate a task test set to evaluate and optimize the generation quality and interaction effect of the large language model, to ensure that the large language model can not only interact with users naturally and smoothly, but also actively promote products, and accumulate experience by analyzing sales data to continuously optimize sales strategies.

[0046] The task test set is used to verify whether the model can effectively perform tasks in the expected usage scenario. In the sales scenario, the task test set can be the following:

[0047] 1. User interaction scenario simulation: The test set includes simulated user questions, conversations, shopping decisions, etc. For example, a user may ask "Does your product have a discount?" or "What are the features of this product?".

[0048] 2. Various sales scenarios: The test set includes different sales scenarios, such as promoting new products, handling user objections, recommending related products, etc., to verify whether the large language model responds naturally and smoothly in various situations.

[0049] 3. Generation quality evaluation: The test set contains specific task objectives to evaluate whether the answers generated by the large language model meet the expected standards. For example, when promoting products, whether the large language model can accurately and effectively recommend products and whether it can make reasonable recommendations based on user needs.

[0050] 4. Personalized recommendation scenario: The test set contains the user's purchase history or preferences to test whether the large language model can generate personalized product recommendations based on the user's personal information.

[0051] The task test set can be used to evaluate the model's interactive effects and the quality of generated content in a variety of scenarios to ensure that the performance of the large language model in the sales environment meets actual needs.

[0052] The steps to develop a task test set are as follows:

[0053] 1. Define core tasks: First, determine the tasks that the large language model needs to perform in actual applications. For example, the core tasks in a sales scenario may be product recommendations, answering user questions, pushing promotional information, etc.

[0054] 2. Design different scenarios: Design different scenarios for each task, such as users actively asking questions or passively accepting recommendations, users being interested in the recommended products or having objections, etc.

[0055] 3. Write questions and task objectives: Write questions that users may ask and expected outputs of tasks according to different scenarios. The test set should contain a variety of question types, such as open-ended questions, yes-no questions, specific product inquiries, etc.

[0056] 4. Set evaluation criteria: Define clear evaluation criteria for each task, such as accuracy of the answer, natural fluency, whether the sales target is achieved, etc.

[0057] The fine-tuned large language model retrieves matches from the vectors stored in the built knowledge base based on user input (the user's text input is converted into vector expression through the embedding model). The most relevant text slices are recalled and relevant responses are generated. Specifically, the prompt word generator generates appropriate prompt words based on user input, and then drives the large language model to generate relevant responses to improve the interactive effect. The relevant responses are played through the speech synthesis model.

[0058] Step 4: Design intelligent voice interaction strategy:

[0059] Intelligent voice interaction strategies include noise filtering and prompt word templates for large language models:

[0060] 1. Noise filtering (speech model):

[0061] In noisy and open environments such as unmanned vending machines, noise filtering is the foundation of voice interaction systems, ensuring that the system can accurately recognize the user's voice input. When training a voice model, noise filtering technology can help the system maintain efficient voice recognition capabilities in the presence of high background noise. Specifically involved:

[0062] Background noise suppression: By introducing a background noise filtering algorithm, the system can remove environmental noise in real time, making the user's voice input clearer and avoiding misrecognition. For example, in outdoor or large-scale event scenes, the intelligent body of the unmanned vending car can filter out the surrounding traffic or crowd noise.

[0063] Multi-channel microphone array: Through hardware optimization (such as multi-channel microphone array), the noise processing capability is further enhanced, allowing the system to accurately capture the user's voice signal.

[0064] Speech recognition model enhancement: Train a more robust speech recognition model to adapt it to various noise scenarios, thereby improving the overall speech recognition accuracy.

[0065] 2. Prompt word template (large language model):

[0066] The prompt word template is the key to the large language model generating high-quality responses during the conversation. It helps the system maintain the continuity of context in multiple rounds of conversations and adjust the response content based on user input. Specifically, it includes:

[0067] Prompt word templates: Design specific prompt word templates based on different sales scenarios. For example, when a user asks about product details, the prompt generated by the model can be: "Please introduce the functions and features of this product." Or when the user is hesitant, the prompt template can be: "Do you want to know more about the discount?" These prompt words help the large language model generate responses relevant to the current situation.

[0068] Context understanding and retention: The large language model ensures the continuity of the conversation through prompt word templates, and can generate answers that conform to the context logic based on the user's historical input. For example, after a user asks about a product, the prompt word template can guide the system to generate a more detailed introduction without deviating from the topic.

[0069] Sentiment analysis combined with prompt generation: Based on sentiment analysis technology (TextBlob), prompt word templates can be dynamically adjusted to enhance the personalized experience of interaction. If the system detects emotional hesitation in users, the template can guide the large language model to generate encouraging language or recommend preferential information, thereby promoting sales.

[0070] Step 5: Multimodal data processing and enhancement:

[0071] In the design of unmanned vending machines, multimodal data processing technology is introduced. The collected multimodal data is used to train visual models and speech models and integrate them with the large language model to enhance the intelligence of the vending machines and the user interaction experience.

[0072] The visual model is trained on the basis of Yolov8 with images of labeled target (crowd / waving / male / female / old / young, etc.) external frames, so that it can recognize the corresponding targets in the image or video data, and output the text data of the external frame coordinates and categories, which are integrated into the prompt word template and communicate with the large language model;

[0073] The voice model uses whisper. First, the voice recognition model in it is used to convert the user's voice data into text, and then questions are asked to the large language model. After obtaining the text answer from the large language model, the voice synthesis model in the whisper library is used to synthesize it into voice for playback, thereby realizing voice interaction with the user.

[0074] During docking and integration, the prompt word generator uses a prompt word template with a corresponding placeholder. After obtaining the text converted by the visual model and speech recognition model (the text of the scene description and the text of speech recognition), it replaces the corresponding placeholder in the template to generate text that actually communicates with the large language model.

[0075] In this way, in addition to processing text data, the system can also recognize and analyze a variety of data modes including voice, images, and videos, making the sales car more intelligent and considerate in various complex sales environments. Specifically, the sales car is equipped with high-definition cameras and microphone arrays to capture and analyze crowd dynamics in the surrounding environment. When the sales car detects a crowd gathering, it will actively approach potential customers and identify customers who are interested in buying through technologies such as facial recognition, gesture recognition, and body language analysis. When the system recognizes that the user waves or sends a certain designated gesture signal, the sales car will take the initiative to interact with the user. In addition, combined with image recognition of the user's gender, age, etc., and sentiment analysis of the user's speech using natural language processing technology, the voice interactive sales agent can infer the user's interests and emotional state, and make more accurate product recommendations to enhance the sales effect.

[0076] Through multimodal data processing technology, the sales car can achieve a comprehensive analysis of user behavior in terms of vision, hearing and perception, so that it can not only interact with the customer through voice, but also actively adjust sales strategies through visual and perceptual feedback, promote related products, and enhance the naturalness and affinity of the interaction, so that the sales car can better understand user needs, provide a more personalized service experience, and ultimately promote sales performance.

[0077] Step 6, integrate function call functionality:

[0078] In order to meet the specific sales needs of unmanned vending cars in various scenarios, a series of customized functional modules are designed and integrated. The functional modules include personalized recommendation modules, cultural story interpretation modules, activity reminder modules, payment modules, commodity order management modules, etc. First, list all the supported functions in the voice interactive sales agent, including but not limited to payment processing, commodity query, order creation, modification and deletion, etc. According to the user's interactive needs, the functions of these functional modules are integrated and encapsulated to form customized functional functions. Each functional function will be written with a standardized description, including properties such as name, default value, explanation and associated functions / methods. All functional functions and their descriptions will be stored in a function library. Through function calls, the voice interactive sales agent can perform semantic matching according to the user's semantic needs, select the most reasonable function from the function library, and generate corresponding instructions to ensure the accuracy and efficiency of function calls. This design not only improves the user experience, but also optimizes the sales effect and interactive function of the vending car.

[0079] In terms of specific implementation, the payment function will integrate existing payment systems, such as WeChat Pay or Alipay, to ensure that users can complete purchases by scanning codes or paying with NFC. The product order management function interacts with the product database to query product inventory and add, delete, modify, and query orders. In addition, the product recommendation algorithm will make personalized recommendations based on the user's historical purchase records and interest preferences.

[0080] The large language model interacts with the sales data automatic analysis module, which will conduct in-depth mining of user interaction data to accumulate sales experience and optimize sales strategies. At the same time, the voice interactive sales agent will continuously improve functional modules based on user feedback and behavior, such as optimizing payment processes and increasing payment success rates. Through these measures, the sales data automatic analysis module can not only improve the intelligence level of the system, but also significantly improve sales performance and user satisfaction, making the voice interactive sales agent more adaptable and competitive in various sales scenarios. Step 7, system integration and deployment:

[0081] In the unmanned vending car system, comprehensive integration and deployment of voice interactive sales agents are carried out.

[0082] First, ensure that the voice interactive sales agent can be seamlessly connected with the vehicle hardware equipment, including microphones, cameras, display screens and other devices, to achieve the smooth implementation of voice recognition, image analysis, and multimodal interaction functions. At the same time, integrate various functional modules, such as the payment module, to ensure that users can conveniently perform payment operations after completing product selection. By developing an API interface with the database system, the large language model can dynamically obtain and update product information, inventory status, and current promotional activities so that the sales data automatic analysis module can play a role. Each functional module interacts with the database, and the data in the database interacts with the large language model through the formatting and conversion module. The large language model obtains data from the database through the query construction and execution module. Specifically, a prompt word template for generating SQL statements is used to allow the large language model to generate the corresponding SQL statement based on the user's natural language, and then call the function that executes the SQL statement (python connects to the database to execute the SQL statement to implement the operation on the database). The function returns the result returned by the database to the large language model to guide the model to the next dialogue.

[0083] Finally, the voice interactive sales agent on the sales vehicle is also connected to the backend management system in the cloud, so that operators can monitor the sales data, user interaction records, and system operation status of the sales vehicle in real time, and conveniently perform data analysis and business optimization. The above comprehensive system integration and deployment strategy ensures the efficiency, stability, and scalability of the voice interactive sales agent in actual applications, so that it can provide users with a continuously optimized shopping experience.

[0084] The above is only a preferred embodiment of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A voice interactive sales agent based on a large language model, characterized in that: Build it with the following steps: Step 1, multimodal data collection: extensively collect multiple types of data related to the target sales environment, including text data, voice data, image data, and video data; Step 2, building a retrieval enhancement generation framework: first, clean the text data collected in step 1, and divide the cleaned text data into structured knowledge and unstructured knowledge according to content attributes. For structured knowledge, establish a conceptual model in the field, define the categories, attributes and relationships of entities, convert structured knowledge into a knowledge graph, and use a graph database for storage, supporting fast queries based on entities and relationships; convert unstructured knowledge into embedded vectors through an embedding model, store them in the FAISS database for efficient vector similarity search, and use the graph database and FAISS database to build a knowledge base; When the agent is actually running, the user's text input is converted into a vector expression through the embedding model. The vector expression is searched in the knowledge vectors stored in the FAISS database to find multiple knowledge vectors with the highest matching degree. The text slices related to these knowledge vectors will be recalled and re-ranked by ReRanker. After sorting, the most relevant one or more text slices will be passed to the large language model through the prompt word generator for generation tasks; the structured knowledge in the graph database supports queries through entity relationships, and the query results are also passed to the large language model through the prompt word generator for generation tasks; Step 3, fine-tuning the large language model: select a pre-trained large language model, fine-tune the large language model in combination with the knowledge base constructed in step 2, and formulate a task set to evaluate and optimize the large language model, so that it has the knowledge background of a specific sales scenario and can accurately process sales words, product information and multimodal data; the fine-tuned large language model generates relevant responses based on user input; Step 4, designing intelligent voice interaction strategies: In diversified sales scenarios, the intelligent voice interaction strategies include noise filtering and prompt word templates of a large language model. The prompt word templates are dynamically adjusted based on sentiment analysis technology. The noise filtering is used for the voice model of step 5. The prompt word templates are stored in a prompt word generator, which interacts with a large language model. When the agent is running, based on the content input by the user, the voice interaction sales agent will combine historical interaction data, user behavior analysis, and the current scenario to adjust its sales strategy and recommended content in real time; Step 5: Multimodal data processing and enhancement: Use the multimodal data collected in step 1 to train the visual model and speech model and integrate them with the large language model; Step 6, designing function modules, integrating and encapsulating the functions of the function modules to form customized function functions, all function functions are stored in a function library, through function calls, the voice interactive sales agent can perform semantic matching according to the user's semantic needs, select the most reasonable function from the function library, and generate corresponding instructions; the large language model interacts with the sales data automatic analysis module, and the sales data automatic analysis module deeply mines the user's interactive data to accumulate sales experience and optimize sales strategies; Step 7, system integration and deployment: The voice interactive sales agent is connected to the vehicle-mounted hardware equipment to collect multimodal data, integrate the functional modules of step 6, and each functional module interacts with the database. The large language model interacts with the database through the formatting and conversion module and the query construction and execution module.

2. The voice interactive sales agent according to claim 1, characterized in that: In step 2, the structured knowledge includes product attributes, promotional information, and user purchase preferences, which can be directly used in sales scenarios. The unstructured knowledge covers background stories, brand history, and user feedback, which are used to enhance the cultural background understanding and interactive experience of the voice interactive sales agent. The embedding model includes but is not limited to embedding-2.

3. The voice interactive sales agent according to claim 1, characterized in that: In step 3, the large language model includes but is not limited to GLM-4.

0.

4. The voice interactive sales agent according to claim 1, characterized in that: In step 3, a task test set is customized, and the task test set includes user interaction scenario simulation, multiple sales speech scenarios, generation quality assessment, and personalized recommendation scenarios. The task test set is customized through the following steps: Step 31, define core tasks: first determine the tasks that the large language model needs to perform in practical applications; Step 32, design different scenarios: design different scenarios for each task; Step 33, write questions and task objectives: write questions that users may ask and the expected output of the task according to different scenarios; Step 34, Set evaluation criteria: Define clear evaluation criteria for each task.

5. The voice interactive sales agent according to claim 1, characterized in that: In step 5, the visual model includes but is not limited to the target detection model of Yolov8, which is trained with pictures with marked target bounding boxes based on Yolov8, so that it can recognize the corresponding targets in the image or video data, and output the text data of the bounding box coordinates and categories, and at the same time integrate them into the prompt word template to communicate with the large language model.

6. The voice interactive sales agent according to claim 1, characterized in that: In step 5, the voice model uses but is not limited to whisper. The voice recognition model therein is first used to convert the user's voice data into text, and questions are asked to the large language model. After obtaining the text answer from the large language model, the voice synthesis model in the whisper library is used to synthesize it into voice for playback, thereby realizing voice interaction with the user.

7. The voice interactive sales agent according to claim 1, characterized in that: In step 5, during the docking and integration, the prompt word generator uses a prompt word template with corresponding placeholders therein, and after obtaining the text converted by the visual model and the speech recognition model, replaces the corresponding placeholders in the template to generate text that actually communicates with the large language model.

8. The voice interactive sales agent according to claim 1, characterized in that: In step 6, the functional modules include a personalized recommendation module, a cultural story interpretation module, an activity reminder module, a payment module, and a product order management module.

9. The voice interactive sales agent according to claim 1, characterized in that: In step 7, the voice interactive sales agent connects to the backend management system in the cloud to obtain sales data, user interaction records, and system operation status in real time.

10. A vending vehicle, characterized in that: The vending vehicle is equipped with a voice interactive sales agent as described in any one of claims 1-9.

Images