Multi-modal dialogue analysis system and method based on digital e-commerce

Abstract

The invention relates to the technical field of electronic commerce, and discloses a multi-modal dialogue analysis system and method based on digital electronic commerce. The invention constructs a complete multi-mode intelligent dialogue analysis system oriented to a digital e-commerce scene. The system comprises a keyword library, a multi-mode fusion module, an association mapping module, a self-service reply module, a semantic optimization module and an auxiliary optimization module. Firstly, user multi-modal input is unified and standardized through a unified multi-modal message representation pipeline, a unified intention vector capable of accurately expressing the complex intention of a user is generated based on a multi-modal fusion strategy of an attention mechanism, and a keyword result with confidence is output according to a matching mechanism of vector semantic similarity. The reply feedback is executed accordingly; and if automatic feedback cannot be carried out, the manual customer service is called, new knowledge is automatically extracted after the service is finished to update the keyword library, different requirements of pursuing efficiency and autonomous exploration are met at the same time, and the customer satisfaction and the problem solving efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of e-commerce technology, specifically to a multimodal dialogue parsing system based on digital e-commerce. Background Technology

[0002] With the rapid development of online shopping, digital e-commerce has become the mainstream shopping method. Digital e-commerce refers to a fully online business environment driven by data. It breaks down the limitations of time and space, allowing consumers to access goods from anywhere anytime, anywhere, enjoying the convenience of one-click ordering and home delivery, greatly reducing shopping costs in terms of time, effort, and price. At the same time, it also provides businesses with powerful capabilities to accurately understand the market, optimize operations, and develop new channels. Therefore, digital e-commerce is indispensable for modern life and the socio-economic system. Current digital e-commerce uses suggestion dialogue models that only handle text input during customer conversations, failing to accommodate other modal inputs such as images and voice, resulting in weak comprehension capabilities. When other modal inputs are used, human customer service is required, which leads to slow response times and high costs, especially during peak periods, causing long waiting times and increasing the risk of customer churn. Summary of the Invention

[0003] In response to the problems in related technologies, this invention provides a multimodal dialogue parsing system based on digital e-commerce to overcome the aforementioned technical problems in existing related technologies.

[0004] To solve the aforementioned technical problem, the present invention is achieved through the following technical solution: On the one hand, the present invention provides a multimodal dialogue parsing system based on digital e-commerce, specifically including: a keyword library, a multimodal fusion module, an association mapping module, a self-service response module, a semantic optimization module, and an auxiliary optimization module; The multimodal fusion module is used to extract the multimodal input uploaded by the customer in the chat box and convert it into message representations. The message representations include text representation vectors and visual representation vectors. The message representations are fused using a weighted fusion strategy based on the attention mechanism to generate an intention vector sequence. The association mapping module, based on a pre-built keyword library, maps and matches the intention vector sequence with the keywords in the keyword library to output its corresponding associated keywords. If the number of associated keywords is zero, it jumps to the auxiliary optimization module; if the number of associated keywords is one, it directly uses this unique keyword as the final keyword of the intention vector sequence, and its corresponding similarity value is used as the confidence of the final keyword, and it jumps to the self-service response module; if the number of associated keywords is ≥2, it jumps to the semantic optimization module. The semantic optimization module performs deduplication and semantic optimization based on the related keywords of all keyword sequences to output the keyword sequence and send it to the self-service reply module; The self-service reply module is used to provide feedback. Specifically, it matches the corresponding reply with the keyword library based on the final keyword and provides it to the customer in the conversation. At the same time, it creates a keyword jump link based on the keyword sequence and provides it to the customer in the conversation. The auxiliary optimization module is used to call human customer service for customers and update the keyword database after the session ends.

[0005] Preferred multimodal input conversion process: Extract text, image, voice, and video inputs uploaded by the customer in the chat window and transform them into a unified multimodal message representation, which includes text representation vectors and visual representation vectors; specifically: (1) Remove meaningless special characters and extra spaces from the text input, and use a pre-trained language model to convert the entire sentence into a standardized text representation vector; (2) Input the image, adjust the image size to the model requirements, encode the image using a pre-trained visual model, and extract its visual features to obtain a visual representation vector; (3) The speech input is processed by noise reduction, silence removal and splicing, and then the speech recognition engine is used to convert the speech input into text. Then, the pre-trained language model is used to convert the text into a text representation vector. (4) Deconstruct the video input into a series of image frames, encode each image frame using a pre-trained visual model, and output the visual representation of the image frame. Sort the visual representations of each image frame into a set of visual representations in the order they correspond.

[0006] Preferred intention vector sequence generation process: Step 201: Use the text representation vector as the query and the visual representation vector as the key and value. The key is used to calculate the relevance with the query, and the value is the visual information itself to be weighted. Step 202: Calculate the similarity between the text query vector and each visual key vector to obtain a similarity score. Then, input all the calculated similarity scores into the Softmax function for normalization to obtain a set of attention weight coefficients. Step 203: Use the attention weight coefficient as the weight of the corresponding visual vector. Each visual value vector will be multiplied by its corresponding attention weight coefficient and then summed to generate an aggregate vector. Step 204: The generated aggregate vector is concatenated with the original text representation vector to form a fused vector. The fused vector is then passed through a fully connected neural network layer for dimensionality reduction and nonlinear transformation, ultimately outputting a multimodal intention vector sequence.

[0007] Preferred keyword mapping process: Arbitrarily select an intention vector from the intention vector sequence and calculate its similarity to each keyword in the keyword library. This gives the similarity between the intention vector and each keyword. A similarity threshold is preset. If the similarity between the intention vector and the keyword is greater than or equal to the similarity threshold, then the keyword is taken as the associated keyword of the intention vector. Similarly, the similarity between the intention vector and all keywords is compared with the similarity threshold respectively, so as to obtain all associated keywords corresponding to the intention vector and the similarity corresponding to each associated keyword; the number of associated keywords corresponding to all intention vectors is counted.

[0008] The preferred deduplication and semantic optimization process: The initial keywords are obtained by deduplicating the associated keywords of all intention vectors. The similarity scores of each initial keyword are traversed, and the highest similarity score is taken as the confidence score of the initial keyword. According to the preset business logic rules, keywords with mutual exclusion relationships or hierarchical structures are screened and optimized, and the ones with the highest confidence scores are retained. Thus, the initial keywords can be conflict-resolved to obtain the final keywords after semantic purification and their corresponding confidence scores. The final keywords are then sorted in descending order of their corresponding confidence scores to obtain a structured keyword sequence.

[0009] Preferred responses and feedback: Extract the final keyword or keyword sequence. If it is the final keyword, select the answer that matches the keyword from the keyword library as the final response and provide feedback to the customer in the conversation. If the customer is not satisfied, multimodal input can be repeated and the process can be returned to the multimodal fusion module. If it is the keyword sequence, select the answer that matches each final keyword in the keyword sequence from the keyword library and create a keyword jump link. At the same time, if the customer is not satisfied, multimodal input can be repeated and the process can be returned to the multimodal fusion module.

[0010] Preferably, the process of calling a human customer service representative is as follows: The system selects currently online customer service representatives as candidate representatives, pre-determines a historical service coefficient for each candidate representative and denots it as K; selects the number of customers to be served for each candidate representative and denots it as n, eliminates candidates with n≥5, and calculates and analyzes the historical service coefficient K and the number of customers to be served for the remaining representatives using a formula to obtain the call value, where K≥1; selects the candidate representative with the largest call value as the target customer service representative, and simultaneously sends the customer's original multimodal input to the workbench application interface used by the target customer service representative in real time through an internal interface. The workbench interface of the target customer service representative will immediately receive visual and auditory prompts and automatically pop up the newly connected session window, with the synchronous session context data packet preloaded in the session window.

[0011] Preferred historical service coefficient generation process: After the entire interaction session between the target customer service representative and the customer concludes, the service evaluation is obtained, and a service history event is generated. Each historical service event includes a timestamp and an evaluation score. All historical service events, along with their corresponding timestamps and scores, are aggregated. The time interval Δt is calculated by comparing the timestamp of each historical service event with the current time. i Where i = 1, 2, 3...M, i is a positive integer representing the index of any historical service time, and M represents the total number of historical service events; the evaluation score of the historical service event is denoted as F. i According to the formula The historical service coefficient K is calculated; where λ is the decay rate.

[0012] Preferred keyword database update process: After the entire interaction session between customer service and customer ends, the system automatically captures the complete dialogue log, conversation summary, and conversation tags. The complete dialogue log includes the customer's initial multimodal input, intent vector sequence, and all Q&A records between customer service and customer. The conversation summary is a standardized field summarizing the solution manually by customer service after the service ends. The conversation tags are question categories manually filled in by customer service after the service ends. Using conversation tags as new keywords, natural language processing technology is used to standardize and format the final response provided by customer service in the conversation to generate a reply. The extracted new keywords are then matched with the generated standard reply to form a new keyword-response knowledge entry. Each keyword in the keyword library is used as a comparison keyword. The cosine similarity is used to calculate the similarity between the new keyword and the comparison keywords. If there is a similarity greater than or equal to the preset similarity threshold of 2, it is considered semantic duplication. The comparison keyword with the highest similarity is selected as the synonym of the new keyword, and the new keyword is added to the synonym keyword list. At the same time, the corresponding answer of the new keyword is added to the answer of the synonym keyword list. If the similarity is less than the preset similarity threshold of 2, it is considered semantic innovation. The new keyword is encoded using a pre-trained language model to output its corresponding vector, and the new keyword, its corresponding answer, and vector representation are added directly to the keyword library as a new entry.

[0013] On the other hand, the present invention provides a multimodal dialogue parsing method based on digital e-commerce, specifically including the following steps: Step 100: Extract the multimodal input uploaded by the customer in the chat box and convert it into a message representation; wherein the message representation includes a text representation vector and a visual representation vector; Step 200: The message representations are fused using a weighted fusion strategy based on the attention mechanism to generate an intention vector sequence; Step 300: Based on the pre-built keyword library, the intention vector sequence is mapped and matched with the keywords in the keyword library to output its corresponding associated keywords. If the number of associated keywords is zero, proceed to step 600; if the number of associated keywords is one, this unique keyword is directly used as the final keyword of the intention vector sequence, and its corresponding similarity value is used as the confidence of the final keyword, and proceed to step 500; if the number of associated keywords is ≥2, proceed to step 400. Step 400: Perform deduplication and semantic optimization based on the related keywords of all keyword sequences to output the keyword sequence, and send it to step 500; Step 500: Based on the final keywords, match the corresponding responses in the keyword library and provide feedback to the customer in the conversation; at the same time, create keyword jump links based on the keyword sequence and provide feedback to the customer in the conversation; if the customer is not satisfied, directly extract the customer's new multimodal input and return to step 100. Step 600: Call a live customer service representative for the customer and update the keyword database after the conversation ends.

[0014] The present invention has the following beneficial effects: 1. By utilizing the attention mechanism, deep fusion of multimodal information is achieved, generating a unified intent vector sequence that can accurately express the user's complex intentions, thereby improving the depth and accuracy of intent recognition.

[0015] 2. By mapping intent vector sequences to a pre-built structured keyword library to obtain associated keywords, different response strategies are executed based on the number of associated keywords. If the number of associated keywords is zero, a human customer service representative will respond; if the number of associated keywords is one, a direct response will be given; if the number of associated keywords is ≥2, semantic optimization will be performed before responding. This constitutes a coherent process from precise intent matching and decision optimization to final response execution, achieving efficient, accurate, and user-friendly automated services.

[0016] 3. By comprehensively evaluating the service quality of human customer service representatives and combining it with the number of customers to be served, the system calls the appropriate human customer service representative to each customer. After the service is completed, the system automatically extracts new knowledge to update the knowledge base, achieving dual self-evolution in terms of service quality and knowledge breadth.

[0017] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, the drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A schematic diagram of a multimodal dialogue parsing system based on digital e-commerce is provided for this invention; Figure 2 The present invention provides a flowchart of a multimodal dialogue parsing method based on digital e-commerce. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Current digital e-commerce uses suggestion dialogue models that only handle text input during customer conversations, failing to accommodate other modal inputs such as images and voice, resulting in weak comprehension capabilities. When other modal inputs are used, human customer service is required, which leads to slow response times and high costs, especially during peak periods, causing long waiting times and increasing the risk of customer churn.

[0022] To address the problems mentioned in the background section, such as Figure 1As shown, this embodiment of the invention provides a multimodal dialogue parsing system based on digital e-commerce, specifically including: a multimodal fusion module, a keyword database, an association mapping module, a self-service response module, a semantic optimization module, and an auxiliary optimization module; The multimodal fusion module is used to extract the multimodal input uploaded by the customer in the chat box and convert it into message representations. The message representations include text representation vectors and visual representation vectors. The message representations are fused using a weighted fusion strategy based on the attention mechanism to generate an intention vector sequence. The association mapping module, based on a pre-built keyword library, maps and matches the intention vector sequence with the keywords in the keyword library to output its corresponding associated keywords. If the number of associated keywords is zero, it jumps to the auxiliary optimization module; if the number of associated keywords is one, it directly uses this unique keyword as the final keyword of the intention vector sequence, and its corresponding similarity value is used as the confidence of the final keyword, and it jumps to the self-service response module; if the number of associated keywords is ≥2, it jumps to the semantic optimization module. The semantic optimization module performs deduplication and semantic optimization based on the related keywords of all keyword sequences to output the keyword sequence and send it to the self-service reply module; The self-service reply module is used to provide feedback. Specifically, it matches the corresponding reply with the keyword library based on the final keyword and provides it to the customer in the conversation. At the same time, it creates a keyword jump link based on the keyword sequence and provides it to the customer in the conversation. The auxiliary optimization module is used to call human customer service for customers and update the keyword database after the session ends.

[0023] In its specific implementation, the above embodiments construct a multimodal dialogue parsing and closed-loop knowledge system for digital e-commerce. The core processes include multimodal input standardization, cross-modal semantic fusion, vectorized retrieval mapping, semantic cleansing, automated response and human-computer collaboration, and self-evolution of the keyword database. First, text is encoded into semantic vectors using a pre-trained language model; visual vectors are extracted from images and video frames using a pre-trained visual model; and speech, after noise reduction and speech recognition, is also encoded by a language model, forming a unified multimodal message representation. Second, a text-led attention mechanism selects the visual context most relevant to the text semantics from the visual information, aggregating it to obtain multimodal intention vectors, supporting parallel representation of multiple intentions. Subsequently, the intention vector is matched with the pre-built structured keyword library in a unified vector space using cosine similarity. The process involves deduplication, synonym merging, mutual exclusion, and hierarchical conflict resolution. Finally, a standard response or interactive redirect is provided based on a single or sequence of keywords. In case of automation failure, the system intelligently routes the user to a human agent. The conversation tags generated by the human agent and the standardized responses are updated into the keyword library after similarity comparison, forming a controllable closed loop for online knowledge updates.

[0024] like Figure 2 As shown, this embodiment of the invention provides a multimodal dialogue parsing method based on digital e-commerce, specifically including the following steps: Step 100: Extract the text input, image input, voice input, and video input uploaded by the customer in the chat box, and convert them into a unified multimodal message representation, where the message representation includes text representation vectors and visual representation vectors; specifically: (1) Remove meaningless special characters, extra spaces, etc. from the text input, and use a pre-trained language model to convert the whole sentence into a standardized text representation vector; (2) Input the image and adjust the image size to meet the model requirements. Use the pre-trained visual model to encode the image and extract its high-level visual features to obtain the visual representation vector. (3) The speech input is denoised, silence removed and concatenated, and then the speech recognition engine is used to convert the speech input into text. Then, a pre-trained language model is used on the text to convert the text into a text representation vector. (4) Deconstruct the video input into a series of image frames, encode each image frame using a pre-trained visual model, and output the visual representation of the image frame. Sort the visual representations of each image frame into a set of visual representations in the order they correspond. It should be noted that, in order to ensure that the system can perform unified and standardized message representation of multimodal input (text, images, voice, and video) uploaded by customers, language models and visual models are pre-built and trained to provide high-quality vector representations for subsequent multimodal information processing and fusion; The language model is built and trained as follows: The language model uses a deep neural network as its core and employs a multi-layered stacked attention mechanism structure. It can capture contextual semantic relationships in the input text. The model input is a cleaned text sequence (removing meaningless symbols, redundant spaces, and special characters), and the output is a high-dimensional text representation vector. Training corpora are prepared, with sources including e-commerce dialogue logs, product documentation, FAQ databases, customer review data, etc. During collection, anonymization is performed to ensure no sensitive personal information is included. The corpora are then organized into a standardized training dataset through word segmentation, lexical normalization, and stop word filtering. During training, the semantic distribution patterns between words are gradually learned by predicting contextual words, sentence relationships, and paragraph connections. Model optimization uses gradient descent methods to minimize prediction error, combined with regularization constraints to avoid overfitting. Validation and test sets are set to evaluate the model's performance on tasks such as intent recognition, synonym discrimination, and keyword recall. Based on the evaluation results, the model size, learning rate, and training epochs are dynamically adjusted to ensure the generation of stable and semantically accurate text representation vectors. The visual model's construction and training process: The visual model uses a deep convolutional neural network as its core, combining multi-layer feature extraction and non-linear transformations to gradually extract features at different levels, such as local texture, global structure, and semantic concepts, from the original image. The model input is a standardized image (size adjustment, pixel normalization), and the output is a high-dimensional visual representation vector. Training data is prepared, including real-life product photos, user-uploaded images of after-sales issues, and official product catalogs. The data needs to be labeled, such as with product category, material, color, and appearance defects, to supervise model learning. Data augmentation methods include rotation... Image enhancement techniques such as rotation, cropping, and lighting adjustment are used to improve the robustness of the model. During model training, by minimizing classification error or feature contrast loss, the vector representations of images of the same type of goods are made to converge in space, while the vector representations of different types of goods maintain discriminative power. Training adopts a batch input and multi-round iterative optimization approach to continuously update the model parameters, enabling it to gradually master visual patterns. Performance tests are conducted on tasks such as product recognition, similar image retrieval, and anomaly detection to evaluate whether the model can accurately extract visual semantic features. For parts with insufficient performance, improvements are made by expanding the data or introducing deeper feature extraction structures.

[0025] Step 200: A weighted fusion strategy using an attention mechanism is employed to fuse message representations from different modalities, generating a unified intent vector sequence, such as: shipping time, size recommendation, genuine product authorization certificate, return requirements, logistics tracking, etc.; specifically: Step 201: Use the text representation vector as the query, as it usually contains the most direct and explicit user instructions or questions, and use it as the basis to query relevant information in other modalities; use the visual representation vector as the key and value, where the key is used to calculate the relevance with the query, and the value is the visual information to be weighted; use the text representation as the active query carrier, because text usually contains the most direct user intent; use the visual representation as the key and value respectively, where the key is used to calculate the relevance with the text query, and the value is used to provide the visual information to be aggregated after attention weighting. The goal of this step is to establish a mapping table between text and vision. Step 202: Calculate the similarity between the text query vector and each visual key vector. To eliminate the bias caused by the vector magnitude, those skilled in the art use cosine similarity for calculation. Its essence is to measure the difference between two vectors in direction. The more consistent the direction, the higher the similarity. The calculation result is in the range of [-1, 1]. Then, input all the calculated similarity scores into the Softmax function for normalization to obtain a set of attention weight coefficients. The sum of the set of attention weight coefficients is 1. Each attention weight coefficient represents the degree of attention of the current text representation vector to different parts of the visual representation vector. The higher the attention weight coefficient, the closer the semantic association between the visual information and the text query is, and the more it needs to be adopted in subsequent fusion. Step 203: The attention weight coefficients obtained in step 202 are used as weights for the corresponding visual vectors for refinement and aggregation. Each visual value vector is multiplied by its corresponding attention weight coefficient. This operation is equivalent to rescaling the visual information according to its importance. Specifically, highly relevant visual features are enhanced, while irrelevant or weakly relevant features are weakened. The visual value vectors after attention weight scaling are then summed to generate an aggregated vector. It should be noted that the aggregated vector is the refinement and aggregation of visual information that is most relevant to the current text semantics after being guided and filtered by text information.

[0026] Step 204: The generated aggregate vector is concatenated with the original text representation vector to form a higher-dimensional fusion vector. The fusion vector is then passed through a fully connected neural network layer for dimensionality reduction and nonlinear transformation, ultimately outputting a multimodal intent vector sequence. This intent vector sequence integrates core textual and visual information and can deeply express the user's intent. By leveraging the attention mechanism, deep fusion of multimodal information is achieved, generating a unified intent vector sequence that can accurately express the user's complex intentions, thereby improving the depth and accuracy of intent recognition.

[0027] Step 300: For this product, a structured keyword library covering all digital e-commerce business scenarios is pre-built. Each keyword in this library (such as size recommendation, product authenticity verification, return requirements, etc.) corresponds to a unique vector representation, and each keyword corresponds to a specific response. The intent vector sequence is mapped and matched with the keywords in the keyword library to output the keyword sequence and the confidence level of each keyword. Furthermore, association analysis is performed based on the keyword sequence to output its associated keywords in the keyword library. Specifically: Arbitrarily select an intention vector from the intention vector sequence and calculate its similarity with each keyword in the keyword library. The calculation method follows the cosine similarity calculation in step 202. Thus, the similarity between the intention vector and each keyword can be obtained. The cosine similarity value range is [-1, 1]. The closer the value is to 1, the more consistent the directions of the two in the vector space and the more similar their semantics. A similarity threshold of 0.7 is preset. Both the intention vector and the vector corresponding to the keyword reside in the same high-dimensional semantic space. Cosine similarity reflects the directional proximity of two vectors, with a value range of -1 to 1. The closer the similarity is to 1, the closer the two are semantically. Since the similarity of most unrelated vectors in a high-dimensional semantic space is concentrated near zero, while the similarity of vectors with genuine semantic connections is usually above 0.7, setting 0.7 as the baseline threshold effectively distinguishes between significant and weak correlations. If the similarity between the intention vector and the keyword is greater than or equal to the similarity threshold, it indicates a relatively significant relationship between the intention vector and the keyword. If a keyword is found to be related to an intention vector, it is used as the associated keyword of the intention vector. This process is repeated, comparing the similarity between the intention vector and each keyword with a similarity threshold to obtain all associated keywords corresponding to the intention vector and the similarity score for each associated keyword. The number of associated keywords corresponding to all intention vectors is counted. If the number of associated keywords is zero, proceed to step 600. If the number of associated keywords is one, this unique keyword is directly used as the final keyword of the intention vector sequence, and its corresponding similarity value is used as the confidence score of the final keyword. Proceed to step 500. If the number of associated keywords is ≥2, proceed to step 400. Step 400: All related keywords of the intent vectors are deduplicated to obtain preliminary keywords. The highest similarity score for each preliminary keyword is taken as the confidence score. To ensure the final output accurately reflects customer intent, semantic optimization of the preliminary keywords is required, specifically: Based on pre-defined business logic rules, keywords with mutually exclusive relationships or hierarchical structures are filtered and optimized. For mutually exclusive term pairs (such as "consult after-sales" and "consult pre-sales"), the one with higher confidence is retained as the valid intent representation. For terms with hierarchical inclusion relationships (such as "return application" being a subcategory of "after-sales consultation"), if the confidence of a specific term is significantly higher than or equal to its generalized parent category (e.g., the confidence of "return application" is 0.94 compared to the confidence of "after-sales consultation" is 0.7), the specific term is retained first, while its generalized parent category terms are filtered to ensure the accuracy of intent expression; if the confidence of a specific term is significantly lower... For its generalized parent class (e.g., the confidence level of "return request" is 0.94 compared to the confidence level of "after-sales consultation" is 0.7), the generalized parent class terms are retained. Thus, conflict resolution processing can be performed on the initial keywords to obtain the final keywords after semantic purification and their corresponding confidence levels. Then, the final keywords are sorted in descending order of their corresponding confidence levels to obtain a structured keyword sequence. For example: [(return request, 0.92), (logistics query, 0.84)]. This result shows that the user's core intent is determined to be a return request with a confidence level of 92%, while the related intent of logistics query is associated with a confidence level of 84%.

[0028] Step 500: Extract the final keyword or keyword sequence. If it is the final keyword (meaning the customer's inquiry is only related to one keyword), select the answer that matches the keyword from the keyword library as the final reply and provide feedback to the customer in the conversation. If the customer is not satisfied, multimodal input can be repeated, and the process returns to step 100. If it is a keyword sequence, select the answers that match each final keyword in the keyword sequence from the keyword library and create keyword jump links for the customer to click and view. Based on the final keyword clicked by the customer, the customer will jump to the corresponding reply. At the same time, if the customer is not satisfied, multimodal input can be repeated, and the process returns to step 100. By mapping intent vector sequences to a pre-built structured keyword library to obtain associated keywords, different response strategies are executed based on the number of associated keywords. If the number of associated keywords is zero, a human customer service representative will respond; if the number of associated keywords is one, a direct response will be given; if the number of associated keywords is ≥2, semantic optimization will be performed before responding. This constitutes a coherent process from precise intent matching and decision optimization to final response execution, achieving efficient, accurate, and user-friendly automated services.

[0029] Step 600: If the number of associated keywords is zero, it means that no keyword can be matched with the existing keyword library. In this case, manual customer service is required. The appropriate human customer service representative will be called to assist the customer. Specifically: Step 601: Select currently online customer service representatives as candidate representatives. Each candidate representative is assigned a historical service coefficient, denoted as K. This historical service coefficient is a measure of service quality based on a comprehensive analysis of past service events. The number of customers to be served for each candidate representative is denoted as n. Candidate representatives with n ≥ 5 are eliminated. The historical service coefficient K and the number of customers to be served n for the remaining representatives are then calculated using the formula... The call value H is calculated, where K≥1. In this formula, K is chosen in a squared form to emphasize the importance of historical service quality. The call value of high-quality customer service representatives (the higher the K value) will increase quadratically, ensuring that high-quality customer service representatives are given priority. An exponential decay function is used to handle the number of pending services. As n increases, the call value decays exponentially, effectively avoiding customer service overload. The exponential function provides a natural decay gradient, producing clear differentiation when n=0,1,2. This formula achieves the optimal balance between service quality and load status through the ingenious combination of quality quadratic amplification and load exponential decay, ensuring that customers can be assigned to customer service representatives who are both high-quality and capable of handling inquiries. The candidate customer service representative corresponding to the largest call value is selected as the target customer service representative. At the same time, the customer's original multimodal input is sent to the workbench application interface used by the target customer service representative in real time through an internal interface. The workbench interface of the target customer service representative will immediately receive visual and auditory prompts (such as highlighting and flashing, sound reminders) and automatically pop up the newly connected session window. The previously synchronized session context data packet has been preloaded in this window, so that the customer service representative does not need to manually search or request anything, so that the target customer service representative can provide timely shopping guidance to the customer. Step 602: After the entire interaction session between the target customer service representative and the customer ends, the customer can rate the service provided by the human customer service representative. Specific service ratings include "Good," "Average," and "Poor," with each rating corresponding to a score of 2, 0.5, and -2, respectively. The customer then retrieves this service rating and generates a service history event. The customer then summarizes all historical service events, along with the timestamp and rating score for each event. The time interval Δt is calculated by comparing the timestamp of each historical service event with the current time. i Let i = 1, 2, 3, ..., M, where i is a positive integer representing the index of any historical service time, and M represents the total number of historical service events; the evaluation score of each historical service event is denoted as F. i According to the formula The historical service factor K is calculated and updated in step 601; where λ is the decay rate, with a value of 0.01≤λ≤0.05; time decay is introduced in this formula. This formula emphasizes the importance of recent service evaluations, while the weight of long-term events decreases exponentially over time, with λ controlling the rate of decay. As shown in the formula, the historical service coefficient K≥1, and the K value changes with service quality, with the K value increasing significantly when the service quality is good. This formula reasonably balances historical evaluations and time factors, providing a scientific basis for customer service performance evaluation and allocation decisions. Step 603: Analyze the conversation between the target customer service representative and the customer to extract new keywords and their corresponding responses, and send the updated keyword database to step 300; specifically: After the entire interaction between customer service and customer concludes, the system automatically retrieves complete dialogue logs, conversation summaries, and conversation tags. The complete dialogue logs include the customer's initial multimodal input, intent vector sequence, and all Q&A records between customer service and customer. The conversation summary is a standardized field summarizing the solution manually by customer service after the service ends. Conversation tags are question categories manually filled in by customer service after the service ends. These conversation tags are directly used as new keyword candidates, as this is the most accurate and reliable source. Using Natural Language Processing (NPL) technology, the final, verified, and effective responses provided by customer service during the conversation are standardized and formatted to generate a clear, accurate, and friendly standard response. The extracted new keywords are then paired with the generated standard response to form a new keyword-response knowledge entry. Before entering the core knowledge base, the new entry must undergo quality control, comparing it against keywords in the keyword library. The similarity between new keywords and compared keywords is calculated using cosine similarity. If a similarity ≥ a preset similarity threshold of 2 exists, it is considered semantic duplication. The compared keyword with the highest similarity is selected as a synonym of the new keyword, and the new keyword is added to the synonym list. At the same time, the response corresponding to the new keyword is added to the response of the synonym. If the similarity is less than the preset similarity threshold of 2, it is considered semantic innovation. The new keyword is encoded using a pre-trained language model in the same way as in step 100 to output its corresponding unique vector. The new keyword and its corresponding response and vector representation are added directly to the keyword library as a new entry. The similarity threshold of 2 is 0.88, which is a system parameter determined by those skilled in the art based on business objectives, model characteristics, semantic granularity requirements, and management strategies. It serves as a key decision threshold for semantic duplication determination in the knowledge base update process. By comprehensively evaluating the service quality of human customer service representatives and combining it with the number of customers waiting to be served, the system calls the appropriate human customer service representative to each customer. After the service is completed, the system automatically extracts new knowledge to update the knowledge base, achieving dual self-evolution in terms of service quality and knowledge breadth.

[0030] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0031] The preferred embodiments of the invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A multimodal dialogue parsing system based on digital e-commerce, including a keyword database that stores keywords and their corresponding vectors and responses; characterized in that, Also includes: The multimodal fusion module is used to extract the multimodal input uploaded by the customer in the chat box and convert it into a message representation; The message representation includes text representation vectors and visual representation vectors; a weighted fusion strategy using an attention mechanism is used to fuse the message representations to generate an intention vector sequence. The association mapping module, based on a pre-built keyword library, maps and matches the intention vector sequence with the keywords in the keyword library to output its corresponding associated keywords. If the number of associated keywords is zero, it jumps to the auxiliary optimization module; if the number of associated keywords is one, it directly uses this unique keyword as the final keyword of the intention vector sequence, and its corresponding similarity value is used as the confidence of the final keyword, and it jumps to the self-service response module; if the number of associated keywords is ≥2, it jumps to the semantic optimization module. The semantic optimization module performs deduplication and semantic optimization based on the related keywords of all keyword sequences to output the keyword sequence and send it to the self-service reply module; The self-service reply module is used to provide feedback. Specifically, it matches the corresponding reply with the keyword library based on the final keyword and provides it to the customer in the conversation. At the same time, it creates a keyword jump link based on the keyword sequence and provides it to the customer in the conversation. The auxiliary optimization module is used to call human customer service for customers and update the keyword database after the session ends.

2. The multimodal dialogue parsing system based on digital e-commerce according to claim 1, characterized in that, Multimodal input conversion process: Extract text, image, voice, and video inputs uploaded by the customer in the chat window and transform them into a unified multimodal message representation, which includes text representation vectors and visual representation vectors; specifically: Remove meaningless special characters and extra spaces from the text input, and use a pre-trained language model to transform the entire sentence into a standardized text representation vector; The image is input and its size is adjusted to meet the model requirements. The image is then encoded using a pre-trained visual model, and its visual features are extracted to obtain a visual representation vector. The voice input is processed by noise reduction, silence removal and splicing, then the voice recognition engine is used to convert the voice input into text, and then a pre-trained language model is used on the text to convert the text into a text representation vector. The video input is decomposed into consecutive image frames. Each image frame is encoded using a pre-trained visual model, and a visual representation of the image frame is output. The visual representations of each image frame are sorted into a set of visual representations in their corresponding order.

3. The multimodal dialogue parsing system based on digital e-commerce according to claim 2, characterized in that, Intention vector sequence generation process: Step 201: Use the text representation vector as the query and the visual representation vector as the key and value. The key is used to calculate the relevance with the query, and the value is the visual information itself to be weighted. Step 202: Calculate the similarity between the text query vector and each visual key vector to obtain a similarity score. Then, input all the calculated similarity scores into the Softmax function for normalization to obtain a set of attention weight coefficients. Step 203: Use the attention weight coefficient as the weight of the corresponding visual vector. Each visual value vector will be multiplied by its corresponding attention weight coefficient and then summed to generate an aggregate vector. Step 204: The generated aggregate vector is concatenated with the original text representation vector to form a fused vector. The fused vector is then passed through a fully connected neural network layer for dimensionality reduction and nonlinear transformation, ultimately outputting a multimodal intention vector sequence.

4. The multimodal dialogue parsing system based on digital e-commerce according to claim 3, characterized in that, Related keyword mapping process: Arbitrarily select an intention vector from the intention vector sequence and calculate its similarity to each keyword in the keyword library. This gives the similarity between the intention vector and each keyword. A similarity threshold is preset. If the similarity between the intention vector and the keyword is greater than or equal to the similarity threshold, then the keyword is taken as the associated keyword of the intention vector. Similarly, the similarity between the intention vector and all keywords is compared with the similarity threshold respectively, so as to obtain all associated keywords corresponding to the intention vector and the similarity corresponding to each associated keyword; the number of associated keywords corresponding to all intention vectors is counted.

5. The multimodal dialogue parsing system based on digital e-commerce according to claim 4, characterized in that, Deduplication and semantic optimization process: The initial keywords are obtained by deduplicating the associated keywords of all intention vectors. The highest similarity score is taken as the confidence score of the initial keyword. Based on preset business logic rules, keywords with mutually exclusive relationships or hierarchical structures are screened and optimized, and those with the highest confidence are retained. Therefore, conflict resolution processing can be performed on the initial keywords to obtain the final keywords after semantic purification and their corresponding confidence levels. Then, the final keywords can be sorted in descending order of their corresponding confidence levels to obtain a structured keyword sequence.

6. The multimodal dialogue parsing system based on digital e-commerce according to claim 5, characterized in that, the response... feedback: Extract the final keyword or keyword sequence. If it is the final keyword, select the answer that matches the keyword from the keyword library as the final response and provide feedback to the customer in the conversation. If the customer is not satisfied, multimodal input can be repeated and the process can be returned to the multimodal fusion module. If it is a keyword sequence, select the answer that matches each final keyword in the keyword sequence from the keyword library and create a keyword jump link. At the same time, if the customer is not satisfied, multimodal input can be repeated and the process can be returned to the multimodal fusion module.

7. The multimodal dialogue parsing system based on digital e-commerce according to claim 6, characterized in that, The process of calling a live customer service representative is as follows: The system selects currently online human customer service representatives as candidate representatives, and pre-defines a historical service coefficient (K) for each candidate representative. It also selects the number of customers to be served for each candidate representative (n), eliminates those with n≥5, and calculates the call value using a formula based on the historical service coefficient K and the number of customers to be served (n), where K≥1. Finally, it selects the candidate representative with the highest call value as the target customer service representative, and simultaneously sends the customer's original multimodal input to the target customer service representative's workbench application interface in real time via an internal interface. The target customer service representative's workbench interface will immediately receive visual and auditory cues and automatically pop up a newly connected session window, with the synchronized session context data packet preloaded into this session window.

8. The multimodal dialogue parsing system based on digital e-commerce according to claim 7, characterized in that, Historical service factor generation process: After the entire interaction session between the target customer service representative and the customer ends, obtain the service evaluation and generate a service history event; each historical service event includes a timestamp and evaluation score. Summarize historical service events and their corresponding timestamps and evaluation scores. Calculate the time interval by comparing the timestamp of each historical service event with the current time. Based on the evaluation scores of the historical service events, calculate the historical service coefficient.

9. The multimodal dialogue parsing system based on digital e-commerce according to claim 8, characterized in that, Keyword database update process: After the entire interaction session between customer service and customer ends, the system automatically captures the complete dialogue log, conversation summary, and conversation tags. The complete dialogue log includes the customer's initial multimodal input, intent vector sequence, and all Q&A records between customer service and customer. The conversation summary is a standardized field summarizing the solution manually by customer service after the service ends. The conversation tags are question categories manually filled in by customer service after the service ends. Using conversation tags as new keywords, natural language processing technology is used to standardize and format the final response provided by customer service in the conversation to generate a reply. The extracted new keywords are then matched with the generated standard reply to form a new keyword-response knowledge entry. Each keyword in the keyword library is used as a comparison keyword. The cosine similarity is used to calculate the similarity between the new keyword and the comparison keywords. If there is a similarity greater than or equal to the preset similarity threshold of 2, it is considered semantic duplication. The comparison keyword with the highest similarity is selected as the synonym of the new keyword, and the new keyword is added to the synonym keyword list. At the same time, the corresponding answer of the new keyword is added to the answer of the synonym keyword list. If the similarity is less than the preset similarity threshold of 2, it is considered semantic innovation. The new keyword is encoded using a pre-trained language model to output its corresponding vector, and the new keyword, its corresponding answer, and vector representation are added directly to the keyword library as a new entry.

10. A multimodal dialogue parsing method based on digital e-commerce, characterized in that... Implementing the multimodal dialogue parsing system based on digital e-commerce as described in any one of claims 1-9 includes the following steps: Step 100: Extract the multimodal input uploaded by the customer in the chat box and convert it into a message representation; wherein the message representation includes a text representation vector and a visual representation vector; Step 200: The message representations are fused using a weighted fusion strategy based on the attention mechanism to generate an intention vector sequence; Step 300: Based on the pre-built keyword library, the intention vector sequence is mapped and matched with the keywords in the keyword library to output its corresponding associated keywords. If the number of associated keywords is zero, proceed to step 600; if the number of associated keywords is one, this unique keyword is directly used as the final keyword of the intention vector sequence, and its corresponding similarity value is used as the confidence of the final keyword, and proceed to step 500; if the number of associated keywords is ≥2, proceed to step 400. Step 400: Perform deduplication and semantic optimization based on the related keywords of all keyword sequences to output the keyword sequence, and send it to step 500; Step 500: Based on the final keywords, match the corresponding responses in the keyword library and provide feedback to the customer in the conversation; at the same time, create keyword jump links based on the keyword sequence and provide feedback to the customer in the conversation; if the customer is not satisfied, directly extract the customer's new multimodal input and return to step 100. Step 600: Call a live customer service representative for the customer and update the keyword database after the conversation ends.

Images