Retail customer service response method and retail customer service system

Abstract

The invention relates to the technical field of information interaction, in particular to a retail customer service response method and a retail customer service system. The response method comprises the following steps: based on retail data, constructing a retail large language model by using a Transform model architecture, and carrying out training and fine tuning, so that the retail large language model has a question and answer ability in the retail field; performing feature extraction based on the multi-modal data input by the user to generate question and answer text data; based on the question and answer text data, topK information with most matched semantics is retrieved from a preset vector knowledge base; and based on the text data and the topK information, calling the retail large language model to respond. According to the response method of the retail customer service, questions of the user can be deeply understood, intelligent response is carried out, and the shopping experience of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of information interaction technology, and in particular to a retail customer service response method and a retail customer service system. Background Art

[0002] With the acceleration of people's pace of life and the increase in labor costs, automation has gradually been integrated into people's lives. The vending system, as a convenient equipment that can effectively save labor costs, has gradually been promoted and used. It provides 24-hour service and meets the needs of consumers anytime and anywhere, and is increasingly favored by the market.

[0003] However, the sales systems in the prior art are only equipped with simple sales programs, which can only allow users to simply choose the goods to buy, and cannot interact with users at more information levels, resulting in a failure to explore users' real needs through interaction and provide users with a good shopping experience and service. Summary of the invention

[0004] In order to address the deficiencies in the prior art, the purpose of this application is to provide a retail customer service response method, a retail customer service system, an electronic device and a storage medium to better understand user needs and provide better services to users.

[0005] To achieve the above objectives, the present application provides a retail customer service response method, including: Based on retail data, a retail language model is constructed using the Transformer model architecture and trained and fine-tuned to enable the retail language model to have the ability to answer questions in the retail field. Based on the multimodal data input by the user, feature extraction is performed to generate question-answer text data; Based on the question and answer text data, retrieve the topK information with the best semantic matching in a preset vector knowledge base; Based on the text data and topK information, the retail large language model is called to respond.

[0006] The response method adopts the Retrieval Augmented Generation (RAG) technology. When receiving user input, relevant information is first retrieved from the data provided by the retail store in the vector knowledge base, and then the retail large language model responds, thereby improving the quality and accuracy of the prediction.

[0007] The advantages of RAG technology also include its high customization and flexibility, reducing the training time and cost of large language models, and its ability to reduce hallucinations and improve the reliability of generated text.

[0008] Furthermore, the vector knowledge base is constructed by the following steps: Load and preprocess the retail-related dataset to standardize the retail data in the dataset; Split the standardized retail data into text blocks; Perform vector embedding on the text block to obtain the embedding representation vector of the text block in the vector space; The embedded representation vectors are stored in a database to form a vector knowledge base.

[0009] Furthermore, the method further comprises: Based on the current question and answer text data, topK information and historical question and answer data, the retail large language model is called to respond.

[0010] Furthermore, the Transformer model architecture includes several layers of encoders, each layer having several hidden units; in the process of building a large retail language model using the Transformer model architecture and performing training and fine-tuning, some front-layer encoders are frozen, and the remaining back-layer encoders are used to perform targeted training using retail data suitable for specific tasks; the specific tasks include: inventory query, customer service, and historical data recommendation.

[0011] Furthermore, the multimodal data includes three modal types: text, image and speech. For image multimodal data, OpenAI's GPT 4 Vision Preview pre-trained model is used for feature extraction; for speech multimodal data, Web Speech API is used for feature extraction.

[0012] Furthermore, the Transformer model architecture includes 48 layers of encoders, each layer has 1024 hidden units; each layer of encoder contains 16 attention heads for processing different parts of input data in parallel; when using retail data suitable for a specific task for targeted training, the first 40 layers of encoders are frozen and the remaining 8 layers of encoders are used for training.

[0013] Furthermore, the steps of building a retail big language model based on retail data using the Transformer model architecture and training and fine-tuning the retail big language model so that the retail big language model has the question-answering capability in the retail field include: Use Adam optimizer for optimization and set the initial learning rate to 0.001; The batch size used for each training is 128; Use the gradient descent method to gradually optimize the model parameters and adopt a gradually decreasing learning rate strategy; Use the cross entropy loss function to evaluate the prediction error of the model; Multiple iterations are performed until the loss converges or the preset number of training times is reached.

[0014] To achieve the above objectives, the present application provides a retail customer service system, including: A multimodal input unit, used to receive multimodal data input by a user; A data conversion unit, connected to the multimodal input unit, for converting the multimodal data into question-answer text data; A search enhancement generation unit is connected to the data conversion unit and is configured with a vector knowledge base, wherein the vector knowledge base is configured with retail data related to the retail store and its corresponding vector representation, and is used to retrieve the topK information with the best semantic matching in the vector knowledge base according to the question and answer text data; The retail large language model unit is connected to the multi-retrieval enhancement generation unit and is used to respond according to the topK information and question and answer text data.

[0015] To achieve the above purpose, the electronic device provided by the present application includes: processor; a memory having stored thereon one or more computer program instructions executed on the processor; Wherein, when the processor runs the computer instructions, it executes the retail customer service response method as described above.

[0016] To achieve the above objectives, the present application provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the steps of the retail customer service response method described above are executed.

[0017] The retail customer service response method provided in this application can deeply understand the user's questions, needs and intentions through multi-modal data interaction with the user, and make intelligent responses in a natural and coherent manner, thereby improving the user's shopping experience and increasing user stickiness; at the same time, it can better assist the management of retail stores and reduce labor costs.

[0018] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or may be understood by practicing the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings: Figure 1 A flowchart of a response method for retail customer service of this application; Figure 2 A schematic diagram of the process of building a vector knowledge base; Figure 3 A flowchart of a response method for continuous question and answer; Figure 4 This is a schematic diagram of the structure of the retail customer service system of this application; Figure 5 A working diagram of the retail customer service system of this application. DETAILED DESCRIPTION

[0020] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0021] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0022] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in this application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more". "Plurality" should be understood as two or more.

[0024] Retrieval Augmented Generation (RAG) is a technique that uses information from private or proprietary data sources to assist in text generation. It combines retrieval models (designed to search large datasets or knowledge bases) with generative models, such as large language models (LLMs), which use the retrieved information to generate readable text responses. Retrieval Augmented Generation improves the relevance of the search experience by adding contextual information from more data sources and supplementing the original knowledge base of the LLM through training. This improves the output of the large language model without retraining the model. The additional information sources can range from new information on the Internet that was not used when training the LLM, to proprietary business background information, or confidential internal documents belonging to the company. RAG is extremely valuable for tasks such as question answering and content generation because it enables generative AI systems to use external information sources to generate more accurate and contextual responses. It implements search retrieval methods (usually semantic search or hybrid search) to respond to user intent and provide more relevant results.

[0025] Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0026] Example 1 An embodiment of the present application provides a retail customer service response method to better understand user needs and provide better services to users.

[0027] Figure 1 This is a flow chart of the retail customer service response method of this application, which will be referred to below Figure 1 The retail customer service response method of this application is described in detail, including: Step S101: Based on the retail data, a retail big language model is constructed using the Transformer model architecture and trained and fine-tuned so that the retail big language model has the ability to answer questions in the retail field.

[0028] In this embodiment, the retail data includes: Product information data, including product name, description, category, price, brand, etc.

[0029] Inventory data, including the inventory quantity, inventory location, and warehousing time of the goods.

[0030] Sales data, including sales records, sales time, sales channels, discount information, etc.

[0031] Customer review data, including customer evaluations, ratings, and feedback on products.

[0032] It is understandable that data needs to be cleaned, including: deduplication, removing duplicates to ensure the uniqueness of each piece of data; error correction, correcting erroneous data, such as spelling errors, misclassification, etc., to ensure data accuracy; filling missing values: using interpolation, mean filling and other methods to fill missing values ​​to ensure data integrity. Standardization: converting all data into a unified format, such as a unified date format, unified units, etc., to ensure data consistency.

[0033] In this embodiment, the model architecture of the large language model adopts the Transformer architecture, and the characteristics of the Transformer architecture are as follows: number of layers: a 48-layer encoder is adopted, and each layer has 1024 hidden units; attention head: each layer contains 16 attention heads for parallel processing of different parts of the input data; position encoding: position encoding is used to retain sequence information so that the model can understand the order of the data; activation function: ReLU activation function is used to enhance the nonlinear expression ability of the model; residual connection: residual connection is used between each layer to avoid the gradient disappearance problem and promote the training of deep networks.

[0034] In this embodiment, the pre-training process is optimized using the Adam optimizer, and the initial learning rate is 0.001. Batch size: The batch size used for each training is 128. Gradient descent: Use the gradient descent method to gradually optimize the model parameters, using a gradually decreasing learning rate strategy (such as learning rate decay). Loss function: Use the cross entropy loss function to evaluate the prediction error of the model.

[0035] Specifically, the training steps include: Data loading: Loading large-scale retail datasets into the model in batches.

[0036] Forward propagation: The input data is forward propagated through the model to calculate the prediction results.

[0037] Loss calculation, calculate the loss between the predicted results and the actual results.

[0038] Back propagation, update the model parameters through the back propagation algorithm.

[0039] Iterate the training for multiple iterations until the loss converges or the preset number of training times is reached.

[0040] In this embodiment, the fine-tuning process is performed for different specific tasks. In this embodiment, the specific tasks include inventory query, customer service, and historical data recommendation. The fine-tuning process freezes most of the layers of the model, especially the deeper layers of the model, to maintain basic language understanding capabilities, retaining only the bottom 8 layers, and only adjusts the part close to the output layer to adapt to specific retail tasks. The optimization goal of the fine-tuning process is to optimize the performance of the model on a specific task to improve the accuracy and adaptability of the task.

[0041] Specifically, the fine-tuning steps include: Prepare annotated datasets corresponding to specific tasks; load pre-trained models; freeze the first 40 layers of the pre-trained model; fine-tune the remaining 8 layers using task-specific datasets; model evaluation: use the validation set to evaluate the performance of the fine-tuned model.

[0042] Step S102: Perform feature extraction based on the multimodal data input by the user to generate question and answer text data.

[0043] In this embodiment, the multimodal data includes three modal types: text, image, and speech. For image multimodal data, OpenAI's GPT 4 Vision Preview pre-trained model is used for feature extraction. For speech multimodal data, Web Speech API is used for feature extraction.

[0044] In some other embodiments, for multimodal data such as images, a feature extraction model obtained by training and fine-tuning based on the CNN model ResNet and Inception can also be used for feature extraction; for multimodal data such as speech, an automatic speech recognition model can also be used for feature extraction.

[0045] For example, for multimodal data of image type, the training process of the corresponding feature extraction model is as follows: Image preprocessing includes: Image enhancement, which enhances the input image, including adjusting brightness, contrast, and color balance to improve image quality; noise removal, which uses a filtering algorithm (Gaussian filtering) to remove noise from the image and improve image clarity; image cropping: automatically cropping the image according to the image content, removing excess background parts, and highlighting the main body of the product.

[0046] Specifically, the preprocessing steps are implemented through the OpenCV library, for example, using cv2.equalizeHist() for histogram equalization and using cv2.GaussianBlur() for Gaussian blur processing.

[0047] Feature extraction, use the pre-trained CNN model ResNet to extract features from the image. It can be understood that when an image is input into a trained model, the model will automatically extract the high-level features of the image, namely the feature vector. The output of the CNN model is a high-dimensional feature vector, usually called an embedding vector or Embeddings. Each image can be represented as an n-dimensional vector, where n is 2048. For example, the input image of size 224x224 is converted into a 2048-dimensional feature vector.

[0048] Object detection uses the pre-trained target detection algorithm YOLO to locate and identify specific products in the image and mark their locations. The detected products are marked in the image through bounding boxes, and the output is the product category, location coordinates (x, y, width, height) and confidence.

[0049] Data conversion, namely feature vector conversion, converts the extracted image features and detected product information into feature vectors in a standard format, which is convenient for subsequent processing and as input to the retail language model; feature fusion, which fuses the image feature vector with information such as product category and location coordinates to form a comprehensive feature representation, namely a multimodal vector. For example, the image feature vector and the detection result are combined into a vector containing 2048+4 dimensions, of which 2048 dimensions are image features and 4 dimensions are location coordinates.

[0050] In this embodiment, a dataset containing a large number of images and labels of retail goods is used to train the feature extraction model corresponding to the modal data of the image class. Our training dataset mainly includes two datasets: ImageNet and MS-COCO. The training process uses the pre-trained CNN models ResNet and Inception for fine-tuning to adapt to specific product categories. Combined with the target detection algorithm, the object recognition ability of the model is enhanced. The optimization method uses the Adam optimizer and the cross entropy loss function to evaluate the performance of the model on the validation dataset and perform hyperparameter tuning.

[0051] Exemplarily, the training process of the feature extraction model corresponding to multimodal data of speech is as follows: Speech preprocessing includes: noise removal, using filtering algorithms (Wiener filtering) to remove background noise and improve the clarity of speech signals; speech enhancement, using speech enhancement technology (beamforming) to improve the quality of speech signals; signal normalization: normalizing speech signals to keep their amplitude within a standard range.

[0052] Exemplarily, the Librosa library is used for speech preprocessing, and librosa.effects.preemphasis() is used for pre-emphasis processing.

[0053] Speech recognition, which converts speech signals into text using a pre-trained ASR model. This model represents the input speech signal as audio features and then converts it into text through a neural network.

[0054] Exemplarily, the Wav2Vec 2.0 model is used for speech recognition.

[0055] Step S103: Based on the question and answer text data, retrieve the topK information with the best semantic matching in a preset vector knowledge base.

[0056] See also Figure 2 , the vector knowledge base is built using the following steps: Step S201: loading and preprocessing a retail-related data set to standardize the retail data in the data set; Among them, retail-related data sets include: Product information data, including product name, description, category, price, brand, etc.; Inventory data, including the inventory quantity, inventory location, and warehousing time of the goods.

[0057] Sales data, including sales records, sales time, sales channels, discount information, etc.

[0058] Customer review data, including customer evaluations, ratings, and feedback on products.

[0059] User membership information, including: purchase history, preferences, etc.

[0060] It should be noted that the retail-related dataset used by the vector knowledge base is a real retail dataset local to the retail store.

[0061] Among them, the preprocessing steps generally include: deduplication, removing duplicates to ensure the uniqueness of each retail data; error correction, correcting erroneous data, such as spelling errors, misclassification, etc., to ensure the accuracy of the data; filling missing values: using interpolation, mean filling and other methods to fill missing values ​​to ensure data integrity. Standardization: converting all data into a unified format, such as a unified date format, unified units, etc., to ensure data consistency.

[0062] Step S202: Segment the standardized retail data into text blocks; use a document segmenter to segment the retail data into smaller text blocks to achieve text segmentation. This helps with subsequent processing and embedding generation, and is a very important step when processing large documents.

[0063] It should be noted that for tabular data, the text segmentation step is skipped.

[0064] Step S203: embed the text block into vectors to obtain the embedding representation vector of the text block in the vector space; the text block after word segmentation will use the vector embedding model to convert each text block into a high-dimensional embedding representation vector. These embedding representation vectors are the representation of the text block in the vector space, which retains the semantic information of the text block.

[0065] It should be noted that the setting of the vector embedding model is a conventional technical means and will not be repeated here.

[0066] Step S204: storing the embedded representation vector and the retail-related data set in a database to form a vector knowledge base.

[0067] In this embodiment, ChromaDB is used to store embeddings and set up retrievers; the database supports fast retrieval and comparison of vectors, providing support for subsequent queries and similarity searches. Vector stores are databases designed specifically for efficient storage and retrieval of vector embeddings. They are needed because traditional databases like SQL are not optimized for storing and querying large vector data. Embeddings represent data (usually unstructured data such as text) in a digital vector format in a high-dimensional space. Traditional relational databases are not well suited for storing and searching these vector representations. Vector stores can index and quickly search for similar vectors using similarity algorithms. It allows applications to find related vectors given a target vector query.

[0068] When the user inputs multimodal data and generates corresponding question-and-answer text data, the question-and-answer text data will be converted into corresponding embedding representation vectors by the vector embedding model and compared with the embedding representation vectors in the vector knowledge base. The similarity degree is used to complete semantic search and retrieve the most matching documents or information, that is, to obtain the topK information with the most semantic matching.

[0069] It is understandable that in this embodiment, the construction of the vector knowledge base is an important component of Retrieval Augmented Generation (RAG). The design of the vector knowledge base allows the user to first retrieve relevant information from the data provided by the retail store in the vector knowledge base when the user inputs multimodal data, thereby improving the quality and accuracy of the prediction. The advantages of RAG technology also include its extremely high customization and flexibility, reducing the training time and cost of large language models, and reducing hallucinations and improving the reliability of generated text.

[0070] Step S104: Based on the text data and topK information, the retail large language model is called to respond.

[0071] In this embodiment, for the continuous questions and answers of the user in a whole round of questions and answers, the retail customer service response method of the present application also includes: Based on the current question and answer text data, topK information and historical question and answer data, the retail large language model is called to respond.

[0072] In this embodiment, the specific steps include: S301: Obtaining historical question and answer data of the user; It should be noted that historical question and answer data refers to the historical question and answer data of an entire round of question and answer process.

[0073] S302: Combining the current question and answer text data with the historical question and answer data, generating a contextualized query; Specifically, a contextualized query prompt template is used to combine the user query and the chat history to generate a contextualized query.

[0074] S303: Based on the contextualized query, retrieve the topK information with the best semantic matching in the vector knowledge base; S304: Use the question-answer prompt template to combine the system prompt, chat history, and user input to form a comprehensive prompt so that the retail large language model can respond.

[0075] Corresponding to continuous question and answer, by combining the context of user input, it is possible to generate accurate and user-friendly answers when receiving user queries and related documents, thereby helping users find the information they need.

[0076] In this embodiment, after each answer is given, the retail language model will also ask the user in a friendly manner whether he needs more help until the user expresses his desire to terminate the conversation.

[0077] It is understandable that when a user asks a question, that is, inputs multimodal data, the user can refresh or delete the historical question record to start a new round of question and answer.

[0078] In this embodiment, when a user wants to know the information of a certain retail product, the retail customer service will search for related products in the knowledge vector library. If found, the retail customer service will provide specific product information and specific location information in the retail store. If not found, the retail customer service will intelligently recommend similar products.

[0079] In this implementation, the method for recommending similar products is implemented through a recommendation model, user historical question and answer data, and current question and answer data.

[0080] In this implementation, the training process of the recommendation model is as follows: Training data collection, the training data set includes: user historical purchase data, including purchase time, purchased items, item category and other information; behavioral data, user behavior data, such as browsing history, click history, items added to the shopping cart, etc.; preference data, including customer ratings, comments and feedback on different products.

[0081] Data preprocessing: data cleaning, cleaning the collected data, removing duplicate and erroneous data, filling missing values, and standardizing the data format; data fusion, fusing data from different sources to form a comprehensive customer portrait.

[0082] Feature extraction: Historical data feature extraction, using collaborative filtering model to extract features from customers' historical data and generate customer preference vectors; real-time data feature extraction, extract features from real-time input product images and voices to generate product feature vectors and text feature vectors.

[0083] Training Optimization: The training is performed by combining collaborative filtering models and neural network models. The collaborative filtering model can recommend products based on the behaviors of similar users, and the neural network model can perform deep feature extraction and learning on high-dimensional data.

[0084] Among them, the collaborative filtering model uses the user-item rating matrix for training to generate the user's preference vector. The algorithm includes user-based collaborative filtering (User-based CF) and item-based collaborative filtering (Item-based CF). The rating matrix is ​​decomposed into a user matrix and an item matrix using matrix decomposition technology (such as SVD). The optimization goal is to minimize the error between the predicted rating and the actual rating. The parameter settings of the collaborative filtering model are: Learning Rate: 0.01, Regularization Parameter: 0.1, Number of LatentFactors: 50. During the training process, stochastic gradient descent (SGD) is used to optimize and update the matrix parameters.

[0085] The neural network model uses a deep neural network (DNN) to extract and learn deep features from high-dimensional data. It combines the image and text features of the product and uses a multi-layer perceptron (MLP) model for training. The optimization goal is to minimize the recommendation error. Model training parameter settings: Learning Rate: 0.001, Regularization Parameter: 0.01, Number of Hidden Layers and Neurons: 3 layers, 128 neurons per layer. During the training process, the Adam optimizer is used for training to update the network weights. A comprehensive recommendation algorithm that combines collaborative filtering and content recommendation uses cosine similarity to calculate the similarity between users and other users and the similarity between the user's preference vector and the product's feature vector. The similarity is calculated weighted comprehensively, a weight of 1:1 is set, and a recommendation list is generated.

[0086] Understandably, when making product recommendations, the Retail Big Language Model can generate natural language descriptions to explain the reasons for the recommendation. For example, "Based on your purchase history and recent browsing history, we recommend this smartwatch because it has a high-resolution display and long battery life, which is suitable for your needs.

[0087] In this implementation, the recommendation model is continuously optimized based on user feedback and click behavior to improve the accuracy of recommendations and customer satisfaction. Each time a user makes a purchase or clicks, the system updates the user's preference vector and product feature vector, and adjusts the recommendation strategy in real time.

[0088] In this embodiment, the retail big language model can also query the product information and inventory status in the back-end database in real time; the data transmission format between the database and the retail big language model is JSON to ensure data standardization and consistency. The retail big language model's services to the back-end database include: real-time query, according to the customer's query request, real-time query of product inventory information in the database, return inventory status (sufficient inventory, insufficient inventory, out of stock, etc.); product information query, query detailed information of the product, including product description, specification parameters, price, promotion information, etc.

[0089] For example, a customer queries the inventory status of product ID 12345. The database response is: { "product_id":"12345", "stock_status": "in_stock", "quantity": 50} It is understandable that the queried data will be processed, including: Data cleaning: Cleaning and formatting the data returned by the query to remove redundant information and ensure the accuracy and consistency of the data. Data conversion: Converting the cleaned data into a format that can be understood and processed by the large language model to facilitate the subsequent generation of natural language responses.

[0090] It is understandable that when the retail language model can also query the backend database in real time, when the retail language model provides customers with real-time inventory information and personalized product recommendations, it will also use natural language to provide information including product name, picture, inventory status, recommendation reason, etc.

[0091] Through the retail customer service response method of this embodiment 1, the user can communicate with the retail customer service system in a variety of ways, and realize the search of retail products and other retail product-related consulting services in the form of a question-and-answer session with the retail customer service system, which provides a good user experience.

[0092] Example 2 An embodiment of the present application provides a retail customer service system to better understand user needs and provide better services to users.

[0093] Figure 4 This is a schematic diagram of the structure of the retail customer service system of this application, which will be referred to below Figure 4 The retail customer service system of this application is described in detail, including: The multimodal input unit 100 is used to receive multimodal data input by a user; The data conversion unit 200 is connected to the multimodal input unit 100 and is used to convert the multimodal data into question and answer text data; The search enhancement generation unit 300 is connected to the data conversion unit 200 and is configured with a vector knowledge base 301, wherein the vector knowledge base is configured with retail data related to the retail store and its corresponding vector representation, and is used to retrieve the topK information with the best semantic matching in the vector knowledge base according to the question and answer text data; The retail large language model unit 400 is connected to the multi-search enhancement generation unit 300, and is used to respond according to the topK information and question and answer text data.

[0094] For example, see Figure 5, a working diagram of the retail customer service system of the present application. Generally, at the beginning of the conversation, the retail customer service system will actively and enthusiastically introduce itself, and ask the user if he needs any help, and start a question-and-answer session with the user. The retail customer service system will intelligently determine which part each input of the user belongs to and make a corresponding answer; after each output of the answer, the retail customer service system will also friendly ask the user if he needs more help, thereby improving the user experience; if the user needs more help and enters new product search information, the retail customer service system will recycle the multimodal retail product search process; if the user needs more help but does not need to find products, the retail customer service system will recycle the conversation process; if the user does not need more help, the system will terminate the conversation.

[0095] The customer service system of this application can also pass the question and answer set to the big model for learning according to the specific needs of the store, so as to better improve the question and answer performance, help the management of retail stores, reduce labor costs and improve user stickiness.

[0096] The customer service system of this application can also be configured to support multiple languages.

[0097] Example 3 In this embodiment, an electronic device is also provided, the electronic device comprising a processor and a memory. The memory is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor is used to execute the non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are executed by the processor, one or more steps of the above retail customer service response method can be executed.

[0098] Example 4 In this embodiment, a computer-readable storage medium is also provided, and the storage medium is used to store non-transitory computer-readable instructions. For example, when the non-transitory computer-readable instructions are executed by a computer, one or more steps in the retail customer service response method described above can be executed.

[0099] The above description is only a partial embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0100] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0101] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A retail customer service response method, characterized in that: include: Based on retail data, a retail language model is constructed using the Transformer model architecture and trained and fine-tuned to enable the retail language model to have the ability to answer questions in the retail field. Based on the multimodal data input by the user, feature extraction is performed to generate question-answer text data; Based on the question and answer text data, retrieve the topK information with the best semantic matching in a preset vector knowledge base; Based on the text data and topK information, the retail large language model is called to respond.

2. The retail customer service response method according to claim 1, characterized in that: The vector knowledge base is constructed by the following steps: Load and preprocess the retail-related dataset to standardize the retail data in the dataset; Split the standardized retail data into text blocks; Perform vector embedding on the text block to obtain the embedding representation vector of the text block in the vector space; The embedded representation vectors are stored in a database to form a vector knowledge base.

3. The retail customer service response method according to claim 1, characterized in that: The method further comprises: Based on the current question and answer text data, topK information and historical question and answer data, the retail large language model is called to respond.

4. The retail customer service response method according to claim 1, characterized in that: The Transformer model architecture includes several layers of encoders, each layer having several hidden units; in the process of building a large retail language model using the Transformer model architecture and performing training and fine-tuning, some front-layer encoders are frozen, and the remaining back-layer encoders are used to perform targeted training using retail data suitable for specific tasks; the specific tasks include: inventory query, customer service, and historical data recommendation.

5. The retail customer service response method according to claim 1, characterized in that: The multimodal data includes three modal types: text, image and speech. For image multimodal data, OpenAI's GPT 4 VisionPreview pre-trained model is used for feature extraction. For speech multimodal data, Web Speech API is used for feature extraction.

6. The retail customer service response method according to claim 1, characterized in that: The Transformer model architecture includes 48 layers of encoders, each layer has 1024 hidden units, and each layer of encoder contains 16 attention heads for processing different parts of input data in parallel; when using retail data suitable for specific tasks for targeted training, the first 40 layers of encoders are frozen and the remaining 8 layers of encoders are used for training.

7. The retail customer service response method according to claim 6, characterized in that: The steps of building a retail language model based on retail data using a Transformer model architecture and training and fine-tuning the retail language model so that the retail language model has the ability to answer questions in the retail field include: Use Adam optimizer for optimization and set the initial learning rate to 0.001; The batch size used for each training is 128; Use the gradient descent method to gradually optimize the model parameters and adopt a gradually decreasing learning rate strategy; Use the cross entropy loss function to evaluate the prediction error of the model; Multiple iterations are performed until the loss converges or the preset number of training times is reached.

8. A retail customer service system, characterized in that: include: A multimodal input unit, used to receive multimodal data input by a user; A data conversion unit, connected to the multimodal input unit, for converting the multimodal data into question-answer text data; A search enhancement generation unit is connected to the data conversion unit and is configured with a vector knowledge base, wherein the vector knowledge base is configured with retail data related to the retail store and its corresponding vector representation, and is used to retrieve the topK information with the best semantic matching in the vector knowledge base according to the question and answer text data; The retail large language model unit is connected to the multi-retrieval enhancement generation unit and is used to respond according to the topK information and question and answer text data.

9. An electronic device, characterized in that: include: processor; a memory having stored thereon one or more computer program instructions executed on the processor; Wherein, when the processor runs the computer program instructions, it executes the retail customer service response method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, the steps of the retail customer service response method described in any one of claims 1-7 are executed.

Images