A large model-based multi-modal intelligent customer service interaction system

Abstract

The application provides a multi-modal intelligent customer service interaction system based on a large model, and relates to the technical field of intelligent interaction, comprising: a multi-modal receiving module for language detection on text, picture and voice input; a target language recognition module for marking different modalities according to language differences, calculating cross-language semantic correlation of each modal directional vocabulary, and selecting the one with the maximum correlation as the target language modality; a generative reply module for unifying each modality text in the target language and extracting consultation intent semantic vector, fusing context perception, knowledge retrieval and historical consistency to generate a preliminary reply word by word and back translation; a back translation correction module for calculating the deviation of the back translation text and the intent vector, replacing the first candidate vocabulary with the second candidate vocabulary according to the correction ratio when the deviation exceeds the limit to obtain an optimized reply. The application effectively solves the deficiencies of the prior art in multi-modal input semantic unified processing, cross-language semantic correlation calculation and generative reply accuracy.

Description

Technical Field

[0001] This invention relates to the field of intelligent interaction technology, and more specifically, to a multimodal intelligent customer service interaction system based on a large model. Background Technology

[0002] In the current technological development of intelligent customer service interaction systems, customer service platforms relying on natural language processing and speech recognition technologies have been widely applied in various industries. These systems can typically receive single-modal inputs such as text and speech, and generate standardized and logically consistent responses based on user needs. However, with the increasing complexity of knowledge bases and the escalating demands of user interaction, single-modal interaction has gradually revealed several limitations and shortcomings. For example, it has relatively weak processing capabilities for cross-language and multimodal (such as text, speech, and images) inputs, and performs poorly in multi-source information fusion and semantic consistency maintenance. Therefore, in recent years, some research has begun to explore the application of multimodal interaction technologies, gradually expanding the application boundaries of multimodal systems by combining image processing (such as optical character recognition, OCR), speech analysis, and deep language modeling. At the same time, the development of Large Language Model (LLM) technology has given more possibilities to the core capabilities of multimodal interaction systems, especially in semantic parsing, multilingual processing, and contextual reasoning, where significant progress has been made.

[0003] Despite this, existing multimodal customer service systems still face several significant challenges. First, cross-language semantic compatibility remains a major technical hurdle when processing user input. Existing systems often employ discrete language translation modules, neglecting the semantic connections within multimodal inputs. Particularly in generative responses, there is a lack of consistent optimization mechanisms tailored to the multimodal context, making it difficult to accurately understand the user's true intent. Second, existing systems lack a unified standard for integrating multimodal information. The processing of images, text, and speech often uses separate pipelines rather than a true semantic unification mechanism, resulting in a fragmented and inefficient cross-modal interaction experience. Furthermore, when generating multilingual responses, existing technologies struggle to effectively address potential semantic biases in back-translation, leading to unsatisfactory accuracy and naturalness in the generated content. This is particularly evident in complex scenarios such as multinational corporate customer service or international user support. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multimodal intelligent customer service interaction system based on a large model. This system can, to some extent, solve the problems of difficulty in accurately identifying cross-language and multimodal consultation intentions due to the lack of unified semantic processing for multimodal inputs, as well as the semantic deviations and insufficient contextual consistency in the response generation process, which affect the accuracy and user experience of the multimodal intelligent customer service interaction system.

[0005] According to one aspect of the present invention, a multimodal intelligent customer service interaction system based on a large model is provided, comprising: The multimodal receiving module is used to receive consultation input in text, image and voice modes, perform optical character recognition on images to obtain character text, and perform language type detection on each modal input to obtain the corresponding language tag; The target language recognition module is used to mark language difference modalities based on the language differences of language tags, calculate the cross-language semantic correlation of each modality-oriented vocabulary, and select the language difference modality with the highest correlation as the target language modality. The generative response module is used to unify the text of each modality based on the target language, extract the semantic vector of consultation intent, calculate the generation probability of candidate words by weighted fusion of context awareness, knowledge retrieval and historical consistency, generate a preliminary response word by word and translate it back to each source language to obtain the back-translated text; The back-translation correction module is used to calculate the deviation between the semantic vector of the back-translated text and the semantic vector of the consultation intent. When the deviation exceeds the limit, the first candidate word is replaced with the second candidate word according to the correction ratio to obtain the optimized response content.

[0006] Furthermore, when the multimodal receiving module performs language type detection for the speech modality, it extracts the Mel frequency cepstral coefficients of the speech signal frame by frame and calculates the weighted time-series average of the inter-frame changes to obtain the acoustic feature score. The Unicode encoding range of the speech-transcribed text is statistically analyzed and normalized to obtain the text statistical feature score. The language with the highest predicted probability is selected as the language type of the speech modality after the two scores are linearly weighted and fused.

[0007] Furthermore, when the target language recognition module calculates the cross-language semantic relevance, it maps the directional words to intermediate language words through a multilingual dictionary and inputs them into the word vector model to obtain semantic vectors. The cumulative cosine similarity between the target modality semantic vector set and the source modality semantic vector set is calculated as the cross-language semantic relevance.

[0008] Furthermore, the generative response module at each time step Calculate candidate words generation probability The formula is:

[0009] in, For time steps Generate vocabulary The normalized probability value, For the current time step The target vocabulary to be generated; For any candidate words traversed in the vocabulary list; The system has a pre-defined vocabulary list of the target language; For the decoder at time step The hidden state vector; For the semantic vector of the consultation intent; For time step The corresponding context representation vector set; For context-aware word matching functions; This is a function to enhance knowledge retrieval. This is the weighting coefficient for knowledge retrieval. For historical sequence consistency function, This is the historical consistency weighting coefficient. Indicates time step The previously generated word sequence.

[0010] Furthermore, the knowledge retrieval enhancement function The calculation formula is:

[0011] in, The number of candidate knowledge fragments, This is the index number of the candidate knowledge fragment. For the first The semantic vectors of the candidate knowledge fragments, with dimension . , To match temperature parameters to knowledge, To reference a knowledge fragment, the indicator function is used when the first... A knowledge fragment at time step It takes the value of 1 when referenced by the decoder, and 0 otherwise.

[0012] Furthermore, the formula for calculating the historical sequence consistency function Ξh is as follows:

[0013] in, Assign time step index numbers to historical terms. For time step The generated historical vocabulary; Historical vocabulary The embedding vector, with dimension , Current candidate words The embedding vector, with dimension ; For time decay parameters, The embedding vector dimension for words.

[0014] Furthermore, when the back-translation correction module calculates the correction ratio, it compares the deviation of each source language with the deviation of the target language to obtain the deviation difference, selects the maximum deviation difference and subtracts the deviation tolerance parameter to obtain the excess deviation, and multiplies the excess deviation by the mapping slope coefficient to obtain the correction ratio.

[0015] Furthermore, when the back-translation correction module performs word replacement, it calculates the generation probability difference between the original output words and the second candidate words at each time step, sorts them in ascending order of the difference, determines the number of positions to be replaced according to the correction ratio, selects the corresponding number of time steps from the first position of the sorted sequence, and replaces the first candidate words at these positions with the second candidate words.

[0016] Compared with existing technologies, this invention, through a target language recognition module, accurately selects the target language modality with the highest semantic relevance among cross-language and multimodal inputs, enhancing the system's ability to recognize complex cross-language consultation intentions. Combined with a generative response module, based on context awareness, knowledge retrieval, and historical consistency, it generates highly semantically consistent response content, significantly enhancing the accuracy and logical completeness of the response. Simultaneously, relying on a back-translation correction module, the semantic deviation between the generated response and the consultation intention is controlled within a reasonable range, ensuring the naturalness and semantic accuracy of the response content. This effectively addresses the shortcomings of existing technologies in unified processing of multimodal input semantics, cross-language semantic association calculation, and the accuracy of generated responses. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart of a multimodal intelligent customer service interaction system based on a large model according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the network service and interface calls of a multimodal intelligent customer service interaction system based on a large model, according to an embodiment of the present invention. Detailed Implementation

[0018] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0019] As mentioned in the background section, existing technologies have two main problems: First, they lack the ability to process multimodal inputs semantically in a unified manner. Existing systems typically process text, speech, and images separately, failing to effectively utilize the semantic associations of cross-modal data, resulting in low accuracy when interpreting complex consultation intentions. Second, when generating response content, existing technologies struggle to address the issues of insufficient cross-language semantic associations and semantic biases generated during back-translation, leading to a lack of logical consistency and coherence in the generated content.

[0020] Figure 1 This is a system block diagram of a multimodal intelligent customer service interaction system based on a large model according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, the multimodal intelligent customer service interaction system based on a large model includes: a multimodal receiving module, a target language recognition module, a generative response module, and a back-translation correction module.

[0021] S1: The multimodal receiving module is used to receive consultation input in text, image and voice modes, perform optical character recognition on the image to obtain character text, and perform language type detection on each modal input to obtain the corresponding language tag.

[0022] Specifically, the intelligent customer service system receives multimodal inquiry inputs submitted by customers, including text-based, image-based, and voice-based inquiry inputs.

[0023] Language type detection is performed on text-based, image-based, and speech-based query inputs to obtain text language labels, image language labels, and speech language labels. The specific processing procedure is as follows: For image-based modal input, the multimodal receiving module performs text recognition processing on the image through the optical character recognition module. This includes: performing binarization on the image to convert the color image into a binary image; performing connected component analysis on the binary image to extract the bounding boxes of individual characters; extracting feature parameters, including stroke direction, stroke density, and character outline, for the character region within each bounding box; performing similarity matching between the feature parameters and a pre-stored multilingual character template library, selecting the character code corresponding to the character template with the highest similarity as the recognition result for that bounding box; and arranging the recognition results according to the spatial position of each bounding box in the image, from left to right and from top to bottom, to generate the image character text.

[0024] After completing the above preprocessing, the multimodal receiving module performs language type detection on the text content of each modality: For text-based query input, the multimodal receiving module extracts the Unicode encoding value of each character in the input text and counts the percentage of characters whose Unicode encoding values ​​fall into preset character ranges such as Chinese, Japanese, Korean, and English. The language corresponding to the character range with the highest percentage is determined as the language type of the text-based query input.

[0025] For image-based text, since the text has been converted into character encoding form through optical character recognition, the multimodal receiving module uses the same Unicode encoding range statistical method as the text modal input to identify the language type of the image-based text.

[0026] For voice-based consultation input, the multimodal receiving module uses a cross-modal detection mechanism that fuses acoustic features and text statistical features to determine its language type. The specific steps are as follows: The multimodal receiving module first preprocesses the speech signal, dividing it into frames of 20 milliseconds each, and extracting Mel-frequency cepstral coefficients from each frame of speech signal to form a sequence of acoustic feature parameters.

[0027] In parallel, the multimodal receiving module inputs the speech signal to a speech recognition module, converts the speech signal into text, and obtains speech-transcribed text. The speech-transcribed text is then calculated using the Unicode encoding interval statistical method to obtain the text statistical features corresponding to the speech modality.

[0028] The acoustic feature parameter sequence is input into a trained language classification model, which is built on a deep neural network architecture. The training process includes: constructing a deep neural network containing an input layer, multiple fully connected hidden layers, and an output layer; inputting the Mel-frequency cepstral coefficients of the sample speech into the deep neural network, where the output layer generates the predicted probability for each language using a standard Softmax function; and iteratively adjusting the connection weight parameters of each layer in the network based on the difference between the predicted probability and the true language label using a backpropagation algorithm until the model's language prediction accuracy for the training samples reaches a stable state, thus obtaining the pre-trained model and saving the weight parameters of the output layer corresponding to each language. ; During the inference phase, the multimodal receiver module enables a cross-modal fusion inference mechanism to determine the speech language. This mechanism first utilizes the weight parameters of the output layer of the pre-trained model. , and the Mel frequency cepstral coefficient matrix of the current input speech The acoustic feature score is obtained by performing a weighted temporal average of the inter-frame variations in the cepstral coefficients. The formula is expressed as:

[0029] in, For the first Acoustic feature scores for each language category The total number of frames of the input speech signal. Let be the dimension of the Mel frequency cepstral coefficients. For the first The first frame of the speech signal Mel frequency cepstral coefficients, For the first The first frame of the speech signal Mel frequency cepstral coefficients, The global mean of the standard deviations of all cepstral coefficients in the training set. is the regularization constant.

[0030] Simultaneously, the multimodal receiving module acquires the text statistical features of the speech-to-text and calculates the text statistical feature score. The formula is expressed as:

[0031] in, For the first Text statistical feature scores for each language category For the first The percentage of characters in each language category in the current sample, calculated based on Unicode encoding ranges. This is the arithmetic mean of the percentage of characters in all language categories. The total number of language categories, is the regularization constant.

[0032] The multimodal receiver module scores acoustic features. Text statistical feature scores Linear weighted fusion is performed, and the result is input into an improved synthesis formula to replace the standard Softmax function during the training phase, thus calculating the final prediction probability of cross-modal fusion. The formula is expressed as:

[0033] in, To integrate acoustic features and text statistical features, the first Predicted probability of language category These are the acoustic feature fusion weighting coefficients. Weighting coefficients are fused to the statistical features of the text.

[0034] Select The language with the highest value is the language type used as input for the speech modality consultation.

[0035] Based on the above detection, the multimodal receiving module assigns language labels to each modal input. The language labels are represented using international standard language codes. For example, if Chinese characters account for the highest proportion in the text modal consultation input, its language label is "zh"; if Japanese characters account for the highest proportion in the image text, its language label is "ja"; and if Korean characters are most likely to be output after fusion inference in the speech modal, its language label is "ko".

[0036] The aforementioned language tags collectively constitute a language tag set, which includes language tags for text-based query input, image character text, and speech-based query input. This set is used to characterize the distribution of language differences between different modal inputs within the same query. The multimodal receiving module outputs the language tag set to the target language recognition module.

[0037] S2: The target language recognition module is used to mark the language difference modality according to the language difference of the language tag, calculate the cross-language semantic correlation of each modality-oriented word, and select the language difference modality with the highest correlation as the target language modality.

[0038] The target language recognition module receives a set of language tags output by the multimodal receiving module, including text language tags, image language tags, and speech language tags. Based on the linguistic differences among the language tags, it performs word frequency statistics on the text modality consultation input, image character text, and speech modality consultation input, calculates the cross-linguistic semantic correlation degree of directional words in each modality input, and identifies the target language modality carrying the true consultation objective based on the cross-linguistic semantic correlation degree. The specific processing procedure is as follows: The target language recognition module compares the strings of the three language tags pairwise. If a language tag string is different from the strings of the other two language tags, the modality corresponding to that language tag is marked as a "language difference modality," and the language tag is recorded as a "difference language tag." If the strings of the three language tags are exactly the same, no modality is marked. For example, when the text language tag is "zh," the image language tag is "ja," and the speech language tag is "zh," the image modality is marked as a "language difference modality" because the language tag string "ja" is different from the other two tags "zh," and its "difference language tag" is "ja."

[0039] For text-based query input, the Chinese word segmentation tool corresponding to the text language tag "zh" is used for word segmentation to obtain a text vocabulary sequence. For image-based text, the Japanese word segmentation tool corresponding to the image language tag "ja" is used for word segmentation to obtain an image vocabulary sequence. For speech-based query input, it is first input into a speech recognition engine to obtain speech-to-text, and then the word segmentation tool corresponding to the speech language tag is used to segment the speech-to-text to obtain a speech vocabulary sequence. Then, the above three vocabulary sequences are iterated through, and the frequency of each word in each sequence is accumulated to generate a text vocabulary frequency table, an image vocabulary frequency table, and a speech vocabulary frequency table. Each row in each table consists of two columns: "vocabulary" and "frequency".

[0040] The target language recognition module pre-stores a directional vocabulary database. This database stores words with clear referential or query functions, categorized by language such as Chinese, Japanese, Korean, and English. Specific categories include interrogative words, demonstrative pronouns, product attribute words, and action request words. For example, the Chinese section stores words like "what," "this," "ingredients," and "query," while the Japanese section stores corresponding Japanese words. The target language recognition module iterates through the text vocabulary frequency table, image vocabulary frequency table, and speech vocabulary frequency table, judging each row: if the word in that row exists in the directional vocabulary database corresponding to that modality, then the word is extracted. After the iteration is complete, text-oriented vocabulary sets, image-oriented vocabulary sets, and speech-oriented vocabulary sets are formed respectively.

[0041] Furthermore, the target language recognition module constructs a mapping table of "modalities and their corresponding vocabulary sets" to record the correspondence between each modality name and its corresponding vocabulary set. Then, it iterates through this mapping table, identifies all modalities marked as "language-difference modalities," sets each language-difference modality as the target modality, sets the remaining two modalities in the mapping table as source modalities, and performs the following calculations: A pre-stored multilingual dictionary is invoked, containing word mappings from Chinese to English, Japanese to English, and Korean to English. Each word in the target modality-directed vocabulary set is traversed, and its corresponding English word is searched in the dictionary. Simultaneously, each word in each source modality-directed vocabulary set is traversed, and its corresponding English word is also searched in the dictionary. Upon successful search, each directed word is replaced with an intermediate English word; if a word search fails, that word is discarded and not included in subsequent calculations. All the intermediate English words obtained in the above process are input one by one into a pre-trained cross-lingual word vector model. This model is pre-built in the following way: a multilingual parallel corpus containing Chinese, Japanese, Korean, English, and their translations is collected to ensure that sentences such as "What are the ingredients of a product?" have corresponding multilingual translations; a deep neural network containing an embedding layer and a shared representation layer is constructed, where the embedding layer maps the word numbers of each language to initial numerical vectors, and the shared representation layer maps the initial vectors of different languages ​​to a unified semantic space; a joint loss function is used during training. Among them, comparative loss The calculation formula is:

[0042]

[0043] Orientation alignment loss The formula is:

[0044] in, To compare the learning loss terms, For cosine similarity alignment loss term, To align the weight coefficients of the loss term, This represents the total number of word pairs in the parallel corpus that have a translation relationship with each other. Number the index of the currently processed vocabulary. vocabulary of the source language Vector representation in a unified semantic space Vocabulary for the target language The corresponding vector representation of the translation in the unified semantic space. The bandwidth parameter of the Gaussian kernel function is... For the negative sample vocabulary set, For negative sample index, The negative sample word vector representation.

[0045] Training stops when the Euclidean and cosine distances of semantically identical words output by the shared representation layer stabilize and decrease, while the distances of semantically different words stabilize and increase. This yields a pre-trained cross-lingual word vector model. For any input English word, this model outputs a fixed-dimensional cross-lingual semantic vector. The target language recognition module records the cross-lingual semantic vector corresponding to each directional word and constructs a target modality semantic vector set and various source modality semantic vector sets according to their respective affiliations.

[0046] Traverse the target modality semantic vector set and any source modality semantic vector set, calculate the cosine similarity between all pairs of vectors in the two sets, and obtain a set of semantic associations between word pairs.

[0047] In one embodiment, the semantic relevance of all word pairs between a target modality and a source modality is summed to obtain the overall semantic relevance between the target modality and the source modality; In another embodiment, to eliminate the influence of the difference in the number of modal-oriented words on the summation result, the summation value is divided by the number of word pairs involved in the calculation, and the average semantic relevance is used as the overall semantic relevance. Subsequently, the overall semantic relevance of a target modality with all source modalities is summed again to obtain the cumulative cross-linguistic semantic relevance of the target modality.

[0048] Compare the cumulative cross-linguistic semantic relevance values ​​of all modalities labeled as "language difference modalities," and select the language difference modal with the highest cumulative value as the target language modal to carry the actual consultation goal; if there is only one language difference modality, then that modality is directly determined as the target language modality. The "difference language label" corresponding to this target language modality is the target language to carry the actual consultation goal.

[0049] The target language recognition module outputs the identifier of the target language modality and its target language tag, for example, the target language modality is "image character text" and the target language tag is "ja".

[0050] S3: The generative response module is used to unify the text of each modality based on the target language, extract the semantic vector of consultation intent, calculate the generation probability of candidate words by weighted fusion of context awareness, knowledge retrieval and historical consistency, generate a preliminary response word by word and translate it back to each source language to obtain the back-translated text.

[0051] The generative response module compares the language tags of text, image, and speech modalities with the target language tags one by one: modalities with the same target language tag are marked as "target language modalities" and their original text content is preserved; modalities with different target language tags are marked as "modalities to be translated" and their source language tags are recorded. For example, if the output target language modal is image text with the target language tag "ja", the text modal tag is "zh", and the speech modal tag is "zh", then the text and speech modalities are marked as "modalities to be translated", and the source language tag for both is "zh"; the image text is marked as "target language modalities", and the original Japanese text remains unchanged.

[0052] For each modality to be translated, the source language tags, target language tags, and the text content of the modality to be translated are submitted as input parameters to a general neural machine translation service interface. This interface receives the source language tags, target language tags, and the text to be translated, and returns the translated string in the target language. The original text of the corresponding modality is replaced with the returned translated string.

[0053] Based on the timestamps of each modality's submission on the client side, the delimiters and content corresponding to each modality are concatenated into a unified consultation text in chronological order. Predefined text delimiters "[TXT]", "[IMG]", and "[AUD]" correspond to text, image, and audio modalities, respectively. Following the order of time from earliest to latest, the delimiters, the current text content of that modality, and a newline character are concatenated to obtain the complete string, which serves as the consultation text in the unified language.

[0054] The generative response module takes the consultation text in a unified language as input and submits it to a general dependency parsing tool to obtain a syntactic dependency tree. Each node in the syntactic dependency tree contains a word, part-of-speech tags, dependency relation tags, and a parent node index. By traversing all nodes in the syntactic dependency tree, the node with the part-of-speech tag of a verb and whose parent node is the root node is located as the core predicate verb. Nodes with the dependency relation tag "nsubj" between the core predicate verb and the core predicate verb are extracted as agent arguments, nodes with the dependency relation tag "obj" as patient arguments, and nodes with the dependency relation tag "nmod" as modifier arguments, forming a semantic framework.

[0055] The generative response module transforms the semantic framework into a structured representation of the consultation intent, specifically through: a pre-defined intent category mapping table, a product attribute dictionary, and a condition dictionary. It matches the core predicate verb in the intent category mapping table, filling the result into the "intent type" slot; it fills the patient argument into the "query object" slot; it iterates through modifying arguments, filling the "query attribute" slot with words appearing in the product attribute dictionary and the "query condition" slot with words appearing in the condition dictionary. If no valid content is matched in a slot, a pre-defined placeholder is used. The final slot is: {Intent Type: Information Query, Query Object: Component, Query Attribute: Component, Query Condition: none}.

[0056] The generative response module performs word segmentation on the text of each slot, obtaining a number of words, and pre-sets a word semantic mapping table. This mapping table uses word strings as lookup keys and fixed-length numerical vectors as storage values. For each word obtained from word segmentation, it searches in the mapping table: if a word exists in the mapping table, its corresponding numerical vector is read; if a word does not exist in the mapping table, a pre-set default vector is used instead. The arithmetic mean of all word vectors corresponding to the slot is calculated dimensionally, and the resulting vector is used as the semantic vector of that slot. After the four slots have obtained their respective semantic vectors, each vector is multiplied by a pre-set weight coefficient corresponding to the importance of that slot, according to the order of intent type, query object, query attribute, and query condition. The weighted vectors are then concatenated end to end to form the consultation intent semantic vector.

[0057] Obtain the semantic vector of consultation intent Subsequently, the generative response module generates an initial response in the target language word by word through the following generation mechanism at each time step. The generation probability of each candidate word in the vocabulary is calculated using the following formula:

[0058] in, For time steps Generate vocabulary The normalized probability value, For the current time step The target vocabulary to be generated; For any candidate words traversed in the vocabulary list; The system's pre-defined target language vocabulary; For the decoder at time step The hidden state vector; For the semantic vector of the consultation intent; For time steps The corresponding set of context representation vectors contains the multi-layer semantic encoding results of the encoder on the customer consultation content; For context-aware word matching functions; This is a function to enhance knowledge retrieval. This is the knowledge retrieval weight coefficient. For historical sequence consistency function, This is the historical consistency weighting coefficient. Indicates time step The previously generated word sequence.

[0059] Furthermore, the formula for calculating the context-aware word matching function is as follows:

[0060]

[0061] in, Candidate words The corresponding output embedding vector has a dimension of , Let be the hidden state transformation matrix, with dimension . Multiply Used to hide the decoder state from Linear transformation of dimensional space to Dimensional output space; For the decoder at time step The hidden state vector, with dimension ; The intention vector transformation matrix has dimensions of . Multiply Used to vectorize consultation intent from Linear transformation of dimensional space to Dimensional output space; The context vector transformation matrix has dimensions of . Multiply Used to average the context vector from Linear transformation of dimensional space to Dimensional output space.

[0062] It should be noted that, To output the spatial dimension, The dimension of the hidden state vector is... For the consultation intent vector dimension, The context vector dimension is determined based on the actual encoder output dimension used.

[0063] Furthermore, the formula for calculating the knowledge retrieval enhancement function is as follows:

[0064] in, The number of candidate knowledge fragments represents the total number of knowledge fragments related to the consultation intent retrieved from the knowledge base by the knowledge retrieval module. This is the index number of the candidate knowledge fragment. For the first The semantic vectors of the candidate knowledge fragments, with dimension . , To match temperature parameters to knowledge, To reference a knowledge fragment, the indicator function is used when the first... A knowledge fragment at time step It takes the value of 1 when referenced by the decoder, and 0 otherwise.

[0065] Furthermore, the formula for calculating the historical sequence consistency function is as follows:

[0066] in, Assign time step index numbers to historical terms. For time steps The generated historical vocabulary; Historical vocabulary The embedding vector, with dimension , Current candidate words The embedding vector, with dimension ; For time decay parameters, The embedding vector dimension for words.

[0067] It should be noted that the knowledge retrieval weight coefficient Consistency weighting coefficient with historical data The values ​​were determined as follows: Before system deployment, the model was trained using a labeled dataset of customer inquiries and standard responses, and the weight coefficients were... and As trainable hyperparameters of the model, during initialization The value range is set to 0.1 to 1.0. The value range is set to 0.05 to 0.5. During training, the weight coefficient values ​​are automatically optimized using the gradient descent algorithm by maximizing the generation probability of the standard response on the training set or minimizing the cross-entropy loss between the generated response and the standard response. After training, the converged weight coefficient values ​​are fixed for actual consultation response generation. If different consultation scenarios have significant differences in the degree of knowledge dependence or historical coherence requirements, the corresponding weight coefficient configuration is trained and saved for each scenario. The system selects the appropriate weight coefficient for calculation based on the consultation type during runtime.

[0068] During the generation process, the function Embedding vectors of candidate words and the current decoder hidden state Consultation Intent Semantic Vector and context representation The results are fused and a scalar score is output, which reflects the degree of matching between the candidate words and the current semantic environment. function by As a query, relevant knowledge fragments are retrieved from a pre-built product knowledge text database, and calculations are performed. The similarity between the semantic vectors of each segment and the data is calculated by taking the logarithm of the weighted sum of the segments with the highest similarity, which provides a positive gain for candidate words related to knowledge. function Calculate candidate words and generated word sequences The semantic vector cosine similarity between each word is calculated and then summed after applying a weight that decays over time step, providing a positive gain for candidate words that are semantically coherent with the generated content. The natural index is obtained by weighted summing of the three scores and dividing by the sum of the natural indices of all candidate words in the vocabulary. This yields the normalized generation probability of the candidate word, and the candidate word with the highest probability is selected as the current time step. The output.

[0069] Append the selected words to the generated sequence and update the decoder's hidden state. The process continues until the preset end symbol is generated or the maximum length limit is reached. All generated words are then concatenated in sequence to obtain the initial response text in the target language.

[0070] The generative response module iterates through all source language tags of non-target language modalities, calls the neural machine translation interface one by one to back-translate the target language as the source language and each source language as the target language, and merges and stores the target language original text with the back-translated text of each source language version to form a multilingual back-translated text set.

[0071] The generative response module will generate the initial response text and the semantic vector of the consultation intent. The multilingual back-translation text set, along with the first and second candidate words and their generation probabilities for each time step, are output to the back-translation correction module.

[0072] S4: The back-translation correction module is used to calculate the deviation between the semantic vector of the back-translated text and the semantic vector of the consultation intent. When the deviation exceeds the limit, the first candidate word is replaced with the second candidate word according to the correction ratio to obtain the optimized response content.

[0073] The back-translation and correction module receives the initial response text and the semantic vector of the consultation intent. The multilingual back-translated text set and the first and second candidate words and their generation probabilities for each time step.

[0074] For each language version of the back-translated text in the multilingual back-translated text collection, its semantic feature vector is extracted. The extraction method is similar to the semantic vector of the consultation intent. If the approach remains consistent and uses the same pre-set lexical semantic mapping table, then the initial response in the target language version will be used to extract the target language semantic feature vector. The back-translated texts of each source language version are extracted sequentially to obtain the corresponding source language semantic feature vectors. , wait.

[0075] Calculate each semantic feature vector and the semantic vector of consultation intent one by one. The cosine distance between them is used as a measure of cross-linguistic semantic consistency deviation. Specifically, the target language semantic feature vector is used as a measure of cross-linguistic semantic consistency deviation. and The cosine distance is denoted as the target language deviation. Calculate the semantic feature vectors of each source language and The cosine distances are denoted as the corresponding source language deviations. .

[0076] The cosine distance ranges from zero to two. A smaller value indicates that the two vectors are closer in direction in the semantic space, while a larger value indicates a greater semantic deviation. The reason for using cosine distance instead of Euclidean distance as a deviation measure is that cosine distance only measures the difference in vector direction and is not affected by the scaling of vector magnitudes, making it more reliable in assessing semantic closeness.

[0077] Target language deviation As a baseline deviation, the deviation of each source language Deviation from the reference amount respectively Numerical comparisons are performed, and the difference between the source language deviation and the benchmark deviation for each source language is recorded as the corresponding deviation difference. And read the pre-stored deviation tolerance parameters. Its value is pre-set based on the tolerance for semantic deviation in the actual application scenario, and its range is from 0.05 to 0.2. This range corresponds to a reasonable fluctuation range of cosine distance. The deviation difference is then iterated through for all source language versions. Determine whether the difference between each deviation is less than or equal to the deviation tolerance parameter. ; If all deviations differ If no significant semantic shift is found during the back-translation process, the initial response content is directly used as the optimized response content and proceeds to the subsequent output translation step. If there is a deviation difference between at least one source language version The system determined that the translated text of the source language version had a significant semantic shift, triggering a correction process to perform weighted correction on the initial response content.

[0078] After triggering the correction process, select all parameters greater than the tolerance parameter. The largest deviation difference among the source language deviation differences is recorded as the maximum deviation difference. ; Read the pre-stored correction scale mapping parameters, which include the mapping slope coefficients. Maximum correction ratio limit , in It is a positive real number, ranging from 0.5 to 2.0, and is used to control the rate at which the correction ratio increases with the amount of deviation exceeding the limit. It is a positive real number, ranging from 0.3 to 0.5, used to limit the upper limit of the correction ratio and prevent excessive correction from causing the response content to lose its original semantics; Will Subtract the deviation tolerance parameter The out-of-limit deviation is obtained, and then multiplied by the mapping slope coefficient. A preliminary correction ratio is obtained; if the preliminary correction ratio exceeds the upper limit of the maximum correction ratio... Then take As the final adjustment ratio, if the initial adjustment ratio does not exceed The initial correction ratio is then taken as the final correction ratio, denoted as . .

[0079] The back-translation correction module reads the generation record for each time step. The generation record contains the first candidate word, the second candidate word, and the generation probability of both for that time step.

[0080] For each time step, the generation probability difference is calculated by subtracting the generation probability of the second candidate word from the generation probability of the first candidate word. All time steps in the initial response are then sorted in ascending order of generation probability difference, with the position having the smallest difference listed first. The total number of words read from the initial response text is... ,Will Multiply by the correction ratio The number of positions to be replaced is obtained. If the product is a decimal, it is rounded up to the nearest integer, denoted as . Starting from the first position of the sorted sequence, select sequentially... Each time step position is used as a set of positions to be replaced. For each position in the set of positions to be replaced, the original first candidate word of that position is replaced with the second candidate word corresponding to that position in the generated record. After all positions to be replaced are replaced, they are concatenated in the original time step order to form the corrected complete text, which is used as the optimized response content.

[0081] The back-translation correction module counts the frequency of language tags used by customers in each modality recorded in the S1 phase, and uses the language tag with the highest frequency as the customer input language tag; if multiple language tags have the same frequency and are all the highest frequency, the language tag corresponding to the text modality is selected first as the customer input language tag. Read the language tag corresponding to the optimized response content and denote it as the response language tag. Then, perform a string comparison between the response language tag and the customer's input language tag. If the two are completely identical, the optimized response content will be used as the final response text. If the two are inconsistent, the neural machine translation interface is invoked, using the response language tag as the source language parameter and the customer input language tag as the target language parameter. The optimized response content is translated to obtain the final response text, which is then returned to the client device through the customer service interface.

[0082] In summary, the multimodal intelligent customer service interaction system based on a large model, as described in this invention, is explained. Through a target language recognition module, it accurately selects the target language modality with the highest semantic relevance among cross-language and multimodal inputs, enhancing the system's ability to recognize complex cross-language consultation intentions. Combined with a generative response module, based on context awareness, knowledge retrieval, and historical consistency, it generates highly semantically consistent response content, significantly improving the accuracy and logical completeness of the response. Simultaneously, relying on a back-translation correction module, the semantic deviation between the generated response and the consultation intention is controlled within a reasonable range, ensuring the naturalness and semantic accuracy of the response content. This effectively addresses the shortcomings of existing technologies in unified processing of multimodal input semantics, cross-language semantic association calculation, and the accuracy of generated responses.

Claims

1. A multimodal intelligent customer service interaction system based on a large model, characterized in that, include: The multimodal receiving module is used to receive consultation input in text, image and voice modes, perform optical character recognition on images to obtain character text, and perform language type detection on each modal input to obtain the corresponding language tag; The target language recognition module is used to mark language difference modalities based on the language differences of language tags, calculate the cross-language semantic correlation of each modality-oriented vocabulary, and select the language difference modality with the highest correlation as the target language modality. The generative response module is used to unify the text of each modality based on the target language, extract the semantic vector of consultation intent, calculate the generation probability of candidate words by weighted fusion of context awareness, knowledge retrieval and historical consistency, generate a preliminary response word by word and translate it back to each source language to obtain the back-translated text; The back-translation correction module is used to calculate the deviation between the semantic vector of the back-translated text and the semantic vector of the consultation intent. When the deviation exceeds the limit, the first candidate word is replaced with the second candidate word according to the correction ratio to obtain the optimized response content.

2. The multimodal intelligent customer service interaction system based on a large model according to claim 1, characterized in that, When the multimodal receiving module performs language type detection for the speech modality, it extracts the Mel frequency cepstral coefficients of the speech signal frame by frame and calculates the weighted time-series average of the inter-frame changes to obtain the acoustic feature score. The Unicode encoding range of the speech-transcribed text is statistically analyzed and normalized to obtain the text statistical feature score. The language with the highest predicted probability is selected as the language type of the speech modality after the two scores are linearly weighted and fused.

3. The multimodal intelligent customer service interaction system based on a large model according to claim 2, characterized in that, When the target language recognition module calculates the cross-language semantic relevance, it maps the directional words to intermediate language words through a multilingual dictionary and then inputs them into the word vector model to obtain semantic vectors. The cumulative cosine similarity between the target modality semantic vector set and the source modality semantic vector set is calculated as the cross-language semantic relevance.

4. The multimodal intelligent customer service interaction system based on a large model according to claim 3, characterized in that, The generative response module at each time step Calculate candidate words generation probability The formula is: ； in, For time step Generate vocabulary The normalized probability value, For the current time step The target vocabulary to be generated; For any candidate words traversed in the vocabulary list; The system has a pre-defined vocabulary list of the target language; For the decoder at time step The hidden state vector; For the semantic vector of the consultation intent; For time step The corresponding context representation vector set; For context-aware word matching functions; This is a function to enhance knowledge retrieval. This is the weighting coefficient for knowledge retrieval. For historical sequence consistency function, This is the historical consistency weighting coefficient. Indicates time step The previously generated word sequence.

5. The multimodal intelligent customer service interaction system based on a large model according to claim 4, characterized in that, The knowledge retrieval enhancement function The calculation formula is: ； in, The number of candidate knowledge fragments, This is the index number of the candidate knowledge fragment. For the first The semantic vectors of the candidate knowledge fragments, with dimension . , To match temperature parameters to knowledge, To reference a knowledge fragment, the indicator function is used when the first... A knowledge fragment at time step It takes the value of 1 when referenced by the decoder, and 0 otherwise.

6. The multimodal intelligent customer service interaction system based on a large model according to claim 5, characterized in that, The formula for calculating the historical sequence consistency function Ξh is as follows: ； in, Assign time step index numbers to historical terms. For time step The generated historical vocabulary; Historical vocabulary The embedding vector, with dimension , Current candidate words The embedding vector, with dimension ; For time decay parameters, The embedding vector dimension for words.

7. The multimodal intelligent customer service interaction system based on a large model according to claim 6, characterized in that, When the back-translation correction module calculates the correction ratio, it compares the deviation of each source language with the deviation of the target language to obtain the deviation difference, selects the maximum deviation difference and subtracts the deviation tolerance parameter to obtain the excess deviation, and multiplies the excess deviation by the mapping slope coefficient to obtain the correction ratio.

8. The multimodal intelligent customer service interaction system based on a large model according to claim 7, characterized in that, When the back-translation correction module performs word replacement, it calculates the generation probability difference between the original output words and the second candidate words at each time step, sorts them in ascending order of the difference, determines the number of positions to be replaced according to the correction ratio, selects the corresponding number of time steps from the first position of the sorted sequence, and replaces the first candidate words at these positions with the second candidate words.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal intelligent customer service interaction system based on a large model as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal intelligent customer service interaction system based on a large model as described in any one of claims 1 to 8.

Images