A multi-modal large model intelligent customer service training method, device and storage medium

Abstract

The present application discloses a method, device and storage medium for intelligent customer service training of a multimodal large model, and relates to the field of artificial intelligence technology. The present application method includes: collecting multimodal data; checking for duplicate data on multimodal data; preprocessing the checked duplicate data; performing data enhancement on the preprocessed data; initializing a multimodal large model based on the Transformer architecture; initializing a multimodal large model based on enhanced data training to obtain a target multimodal large model; obtaining user query information; extracting weight text in the user query information; inputting the weight text into the target multimodal large model to obtain associated information; preprocessing the associated information to generate a feature vector; obtaining a similarity value between the feature vector and the weight text according to a similarity algorithm; arranging the similarity values ​​in order to obtain a sorting result; extracting the sorting result; determining target associated information based on the target similarity value; and displaying the target associated information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multi-modal large-scale model intelligent customer service training method, device and storage medium. Background Art

[0002] With the rapid development of Internet technology and the increasing prosperity of e-commerce platforms, consumers often find it difficult to quickly find products that meet their needs when faced with massive amounts of product information. Traditional product recommendation systems mostly rely on rule-based recommendations, collaborative filtering recommendations, or content-based recommendations. Although these methods have improved the accuracy of recommendations to a certain extent, they still have many limitations. For example, rule-based recommendation systems require manual setting of rules and are difficult to adapt to diverse user needs; collaborative filtering recommendation systems rely on users' historical behavior data and have cold start problems for new users or new products; content-based recommendation systems mainly rely on the matching degree between product descriptions and user inputs, ignoring users' potential needs and preferences.

[0003] Deep learning models can automatically learn the feature representation of data and capture the complex relationships between data through a hierarchical structure. Therefore, applying deep learning technology to product recommendation systems can achieve accurate understanding of user input information, efficient retrieval, and personalized recommendations. However, how to effectively combine deep learning technology with product recommendation systems and how to optimize the similarity calculation process are still hot and difficult issues in current research. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a multi-modal large model intelligent customer service training method, device and storage medium.

[0005] The technical solution provided in this application is described below:

[0006] The first aspect of the present application provides a multi-modal large model intelligent customer service training method, the method comprising:

[0007] Collecting multimodal data, the multimodal data including text, voice and words;

[0008] Performing data duplication check on the multimodal data to obtain duplicate check data;

[0009] Preprocessing the duplicate checking data to obtain preprocessed data;

[0010] Performing data enhancement on the preprocessed data by using an enhanced algorithm and obtaining enhanced data, wherein the data enhancement processing includes performing synonym replacement, random insertion and deletion of text on the preprocessed data;

[0011] Initialize a large multimodal model based on the Transformer architecture;

[0012] Training the initialized multimodal large model according to the enhanced data to obtain a target multimodal large model;

[0013] Obtain user query information;

[0014] Extracting weighted text from the user query information;

[0015] Inputting the weighted text into the target multimodal macromodel, so that the target multimodal macromodel obtains associated information according to the weighted text, wherein the associated information includes text, video, voice and image information corresponding to the weighted text;

[0016] Preprocessing the association information to generate a feature vector;

[0017] Obtaining a similarity value between the feature vector and the weighted text according to a similarity algorithm;

[0018] Arrange the similarity values ​​in order to obtain a sorting result;

[0019] Extracting the target similarity value ranked first in the sorting results;

[0020] Determining target association information based on the target similarity value;

[0021] The target association information is displayed.

[0022] Optionally, training the initialized multimodal large model according to the enhanced data to obtain a target multimodal large model includes:

[0023] Acquire a plurality of enhanced feature vectors of the enhanced data;

[0024] Inputting the enhanced feature vector into the initialized multimodal large model to obtain a multimodal feature vector;

[0025] Obtaining a prediction result according to the multimodal feature vector;

[0026] Using the cross entropy loss function to calculate the loss between the predicted result and the actual label, and obtaining the loss value;

[0027] According to the loss value, obtaining update parameters of the initialized multimodal large model through a back propagation algorithm;

[0028] The initialized multimodal large model is updated by using the update parameters to obtain the target multimodal large model.

[0029] Optionally, a cross entropy loss function is used to calculate the loss between the prediction result and the actual label, and the loss value is obtained, including:

[0030] Using the cross entropy loss function to calculate the loss between the predicted result and the actual label, and obtaining the loss value;

[0031] The cross entropy loss function is:

[0032]

[0033] Where N is the number of categories, yi is the one-hot encoding of the actual label, and pi is the probability of belonging to the i-th category predicted by the model.

[0034] Optionally, performing a data duplication check on the multimodal data to obtain duplicate check data includes:

[0035] Clean multimodal data to remove invalid, incomplete or erroneous data;

[0036] Classifying the cleaned multimodal data, wherein the multimodal data includes: text data, image data, audio data, and video data;

[0037] Using a first processing method to check the text data for duplicates, so as to obtain first duplicate checking data;

[0038] Using a second processing method to check the image data for duplicates, so as to obtain second duplicate checking data;

[0039] Using a third processing method to check the audio data for duplication, so as to obtain third duplication checking data;

[0040] Using a fourth processing method to check the video data for duplication, so as to obtain fourth duplication checking data;

[0041] The first duplicate checking data, the second duplicate checking data, the third duplicate checking data and the fourth duplicate checking data are merged to generate a duplicate checking data set.

[0042] Optionally, extracting the weight text in the user query information includes:

[0043] Extracting words from the user query information as nodes;

[0044] Constructing an undirected graph based on the co-occurrence relationship of the words in the user's query information;

[0045] Calculate the weight value of each word according to the number of co-occurrences of the words in the undirected graph;

[0046] The weight text in the user query information is extracted according to the weight value.

[0047] Optionally, a softmax classification algorithm is configured in the multimodal large model, and the softmax classification algorithm is defined as:

[0048]

[0049] Among them, c represents the classification category, represents the probability that sample i belongs to category c, Represents the exponential sum of all category scores.

[0050] Optionally, the similarity algorithm is a Tanimoto similarity algorithm, and the Tanimoto similarity algorithm is defined as:

[0051]

[0052] Among them, X and Y are two sets, X∩Y is the intersection of X and Y, and are the number of elements of X and Y respectively.

[0053] The second aspect of the present application provides a multi-modal large-model intelligent customer service training device, the device comprising:

[0054] A collecting unit, used to collect multimodal data, wherein the multimodal data includes text, voice and words;

[0055] A duplicate checking unit, used for performing a duplicate checking on the multimodal data to obtain duplicate checking data;

[0056] A preprocessing unit, used for preprocessing the duplicate checking data to obtain preprocessed data;

[0057] Construction unit, used to build and initialize a large multimodal model based on the Transformer architecture;

[0058] An acquisition unit, used for acquiring user query information;

[0059] A first extraction unit, used to extract weighted words from the user query information;

[0060] An input unit, used to input the weighted text into the target multimodal large model, so that the target multimodal large model obtains associated information according to the weighted text, wherein the associated information includes text, video, voice and image information corresponding to the weighted text;

[0061] A preprocessing unit, used for preprocessing the associated information to generate a feature vector;

[0062] A similarity calculation unit, used for obtaining a similarity value between the feature vector and the weighted text according to a similarity algorithm;

[0063] A sorting unit, used to arrange the similarity values ​​in order to obtain a sorting result;

[0064] A second extraction unit, used to extract the target similarity value ranked first in the sorting result;

[0065] A determination unit, configured to determine target association information based on the target similarity value;

[0066] An output unit is used to display the target related information.

[0067] The third aspect of the present application provides a multi-modal large-model intelligent customer service training device, the device comprising:

[0068] Processor, memory, input-output unit, and bus;

[0069] The processor is connected to the memory, the input and output unit, and the bus;

[0070] The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0071] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0072] It can be seen from the above technical solutions that this application has the following advantages:

[0073] 1. Accurately understand user input information through deep learning models, and combine users' historical behavior data and preference information to achieve personalized recommendations and provide users with a product list that better meets their needs.

[0074] 2. Use optimized deep learning algorithms and cross entropy loss functions to calculate similarity. Continuously optimize model parameters through iterative training to improve the accuracy and efficiency of similarity calculation and ensure the accuracy of recommendations.

[0075] 3. By integrating multi-dimensional information such as user historical behavior data and social media data, user portraits can be built to provide users with more accurate recommendations even when there is less historical behavior data. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0077] Figure 1 A flowchart of an embodiment of a multi-modal large-model intelligent customer service training method provided in this application;

[0078] Figure 2 A schematic diagram of a process for obtaining a target multimodal large model in an embodiment of the present application;

[0079] Figure 3 A schematic diagram of a process for performing data duplication check on multimodal data and obtaining duplicate check data in an embodiment of the present application;

[0080] Figure 4 A flowchart of an embodiment of extracting weighted text from user query information in this application;

[0081] Figure 5 A schematic diagram of the structure of an embodiment of a multi-modal large-model intelligent customer service training device provided in this application;

[0082] Figure 6 A structural schematic diagram of an embodiment of another multimodal large-model intelligent customer service training device provided in this application. DETAILED DESCRIPTION

[0083] It should be noted that the multimodal large-model intelligent customer service training method, device and storage medium provided in this application can be applied to a terminal, a system, or a server. For example, the terminal can be a smart phone or a computer, a tablet computer, a smart TV, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses the terminal as an example for explanation.

[0084] See also Figure 1 The present application first provides an embodiment of a multi-modal large model intelligent customer service training method, which includes:

[0085] S101, collecting multimodal data, where the multimodal data includes text, voice and words;

[0086] The text in this embodiment refers to a written language form composed of a series of characters or symbols for expressing information or ideas. In the field of computer science and information processing, text generally refers to a sequence of characters stored in an electronic device that can be read and processed by a computer program. Voice refers to the sound signals of human speech, which can be captured by a recording device and converted into a digital format for storage, transmission and processing. Speech recognition technology can convert voice signals into text, while speech synthesis technology can convert text into speech. Words refer to characters or symbols in written form, which can represent words, phrases or sentences in a language. Unlike text, words focus more on the written form of a single character or symbol, while text is a complete sentence or paragraph composed of these characters or symbols.

[0087] When building a multimodal large-model intelligent customer service system, it is first necessary to collect a large amount of multimodal data as the input and training basis of the system. These data come from a variety of channels and forms, including but not limited to user conversation records, social media content, website text, audio files, handwriting recognition results, etc. Directly obtain user input text, voice and text data through user interaction interfaces, social media platforms, websites, etc. And obtain relevant text data from third-party data sources such as news websites, forums, blogs, etc.; convert audio files into text data through voice recognition technology; and convert handwritten text into digital text data through handwriting recognition technology. During the collection process, the data needs to be screened to remove invalid, repeated or noisy data to ensure the accuracy and diversity of the data. The collected data is sorted and classified into three modes: text, voice and text for subsequent data processing and model training.

[0088] S102, performing a data duplication check on the multimodal data to obtain duplicate check data;

[0089] In this embodiment, data duplication checking refers to comparing the content of the submitted document or data set with a huge database through specific software or system, so as to identify possible plagiarism, duplication or improper citation parts and present them in an intuitive way. The core of this process lies in the efficiency of the algorithm and the wide coverage of the database.

[0090] Input the collected multimodal data into the duplicate checking system. For non-text data, preprocessing is required to convert it into text or comparable vector representation for duplicate checking. Using the algorithm and database inside the duplicate checking system, the input data is compared with the existing data in the system. Based on the comparison results, the similarity between the input data and the existing data is calculated. Based on the calculation results of the similarity, the parts that are similar or repeated with the existing data are identified.

[0091] S103, preprocessing the duplicate checking data to obtain preprocessed data;

[0092] In this embodiment, the main purpose of preprocessing the data after duplication check is to make the data more standardized and consistent for subsequent data processing and model training. Remove special characters, invalid characters, extra spaces, etc. in the data, unify the date and time format, fill in missing values, etc. Through these steps, the accuracy and completeness of the data can be ensured, and the errors in subsequent processing can be reduced. Convert the data into a unified format and unit, such as converting all dates into the corresponding format, converting numerical data into a unified unit of measurement, etc., to ensure the consistency of data between different systems and platforms. For unstructured or semi-structured data, such as voice, images, etc., it is necessary to convert them into structured data for subsequent processing and analysis. Convert voice data into text data by using voice recognition technology; convert image data into numerical or vector representation by using image recognition technology. Integrate and integrate data from different sources and in different formats to form a unified data set. Ensure the comprehensiveness and diversity of the data, and provide more input information for subsequent data analysis and model training.

[0093] S104, performing data enhancement on the preprocessed data by using an enhancement algorithm, and obtaining enhanced data, wherein the data enhancement processing includes performing synonym replacement, random insertion, and deletion of text on the preprocessed data;

[0094] In this embodiment, the enhanced algorithm is a technical method for optimizing a data set. In the fields of natural language processing, image processing, etc., the enhanced algorithm generates new data samples through a series of transformations and processing operations, thereby increasing the diversity and richness of the data.

[0095] In the text of the preprocessed data, a certain number of words are randomly selected, and then the synonyms of these words are found in the synonym dictionary and replaced. It is used to increase the diversity of the text while keeping the basic meaning of the original text unchanged, which helps the multimodal large model learn richer language representation. In the text of the preprocessed data, some new words are randomly selected to insert into the positions. Here, new words can be synonyms of existing words in the original text, or other related or unrelated words. By introducing new words and sentence structures, the complexity and diversity of the text are increased, and the model's ability to handle unknown words and sentences is improved. In the text of the preprocessed data, some words or phrases are randomly deleted with a certain probability. Simulating noise and redundant information in the text enhances the model's ability to handle incomplete data and noisy data, and also helps to improve the model's generalization ability.

[0096] S105, Initialize a large multimodal model based on the Transformer architecture;

[0097] In this embodiment, the Transformer architecture is a deep learning architecture. The core of the Transformer architecture is the self-attention mechanism, which can effectively capture the long-distance dependencies between sequence data. The Transformer architecture consists of an encoder and a decoder, and each encoder and decoder layer contains multiple layers of self-attention mechanisms and feedforward neural networks. This architecture has achieved great success in the field of NLP and has gradually expanded to the field of multimodal learning.

[0098] Before building a large multimodal model, multimodal data such as text and images are preprocessed. This includes steps such as text segmentation and image cropping in order to extract effective feature representations. Text encoders and image encoders are used to extract feature representations of text and images respectively, and the feature representations are used as the basis for subsequent processing of the model. The cross-modal attention mechanism can learn the weights between different modalities and dynamically fuse information to capture cross-modal representations and relationships. It is one of the core parts of the large multimodal model. Training is performed on multimodal data to learn cross-modal representations and relationships, which is achieved through tasks such as image-text pair classification and image description generation. The training strategy improves model performance by utilizing massive data. Subsequently, the trained large multimodal model is fine-tuned to specific downstream tasks, such as image question answering, visual retrieval, etc., so that the model can be optimized for specific tasks and improve model performance.

[0099] S106, training the initialized multimodal large model according to the enhanced data to obtain a target multimodal large model;

[0100] Select a large multimodal model based on the Transformer architecture as the initial model and initialize the model parameters according to the task requirements and data characteristics. Input the multimodal data after data enhancement into the model. During the training process, the large multimodal model adapts to the changes in input data by continuously adjusting its internal parameters. Use an optimization algorithm to minimize the loss function to improve the performance of the model, and pass the gradient information to each layer of the model through the back propagation algorithm for weight update. The large multimodal model achieves a comprehensive understanding and processing of multimodal data by learning the associations and complementary information between different modalities.

[0101] During the training process, the multimodal large model is evaluated regularly to check whether its performance meets expectations. If the evaluation results are not ideal, the multimodal large model needs to be adjusted and optimized, such as modifying the model structure and adjusting parameter settings. When the multimodal large model achieves the expected performance on the training set, the training process ends. At this point, the model has learned how to process and understand multimodal data and has a certain degree of generalization ability.

[0102] S107, obtaining user inquiry information;

[0103] The intelligent customer service system allows users to easily input their inquiries by designing an intuitive and easy-to-use user interface. The user interface has functions such as text input boxes, voice input buttons, and image upload areas. The necessary prompts and guidance are provided in the UI to help users clarify the content and format of the input. When users enter information through the UI, the intelligent customer service system captures these inputs in real time. For text input, the intelligent customer service system listens to keyboard events and captures the characters entered by the user; for voice input, the intelligent customer service converts voice into text by using voice recognition technology; for image input, the intelligent customer service system captures and stores image data. When the data is captured, the input data is parsed to ensure that the input data meets the requirements of the system processing. For text input, extra spaces and punctuation marks are removed and word segmentation is performed; for image input, scaling, cropping, or denoising are performed. If text and image data are entered at the same time, the intelligent customer service system will integrate the information of these different modalities. Align, fuse, or associate text and image features for subsequent processing. In the process of obtaining user input, the intelligent customer service system will detect and correct possible formal errors. For example, for text input, the intelligent customer service system uses a spell-checking algorithm to identify and correct spelling errors; for voice input, the intelligent customer service system uses the confidence score of voice recognition to filter out low-quality recognition results. In the process of obtaining user inquiry information, the privacy protection principle must be strictly followed to ensure the secure storage and transmission of user data, as well as compliance with relevant laws, regulations and privacy policies when processing data. The information entered by the user is stored in an appropriate database or data structure for subsequent processing and analysis. This information is passed to the next part of the system for further processing and answer generation.

[0104] S108, extracting weighted words from the user query information;

[0105] The intelligent customer service system receives the query information input by the user. The query information here is usually a string that may contain multiple characters such as text, numbers, symbols, etc. The user input is preprocessed, including removing unnecessary blank characters, HTML tags or special characters. Preprocessing is used to ensure that the data processed later is clean and standardized text data. The intelligent customer service system maintains a predefined weighted vocabulary, which contains a series of words that are considered to be of high importance in user queries. The preprocessed user query information is traversed, and the words or phrases therein are checked one by one to see if they match the words in the weighted vocabulary. If the match is successful, the word or phrase is marked as a weighted text, and its position and number of occurrences are recorded. After extracting all matching weighted texts, the intelligent customer service system can generate a list or data structure containing these texts and their related information. The intelligent customer service system will use the generated results for further analysis or directly answer the user's questions. Finally, the system will extract the weighted text and its related information and output the final result to the user interface.

[0106] S109, inputting the weighted text into the target multimodal macromodel, so that the target multimodal macromodel obtains associated information according to the weighted text, wherein the associated information includes text, video, voice and image information corresponding to the weighted text;

[0107] The intelligent customer service system first inputs the weighted text into the target multimodal big model. The weighted text here refers to one or more keywords, phrases or sentences, each of which carries a specific weight. The multimodal big model first parses the input weighted text to understand its meaning and context. Through natural language processing technology, the auxiliary model accurately grasps the intention of the input information. Based on the parsed weighted text, the multimodal big model searches for multimodal information associated with these texts in its internal database or externally connected database. Including: information in various forms such as text, video, voice and image. If the weight information is explicitly provided at the time of input or the multimodal big model is able to identify the weight difference, the multimodal big model will consider these weights when retrieving and selecting related information, and give priority to returning multimodal information that is more relevant to the text with higher weight. The multimodal big model organizes and generates a set of multimodal information associated with the weighted text. The multimodal information set contains directly related text fragments, related video links or summaries, links or transcripts of voice fragments, and images or charts related to the input text.

[0108] S110, preprocessing the association information to generate a feature vector;

[0109] Preprocess the associated information by removing noise and irrelevant content from the associated information, such as advertisements, duplicate information, invalid links, etc. Then convert the associated information into a standardized information format, such as unified text encoding, adjust the image size, etc., and classify the associated information, such as text, video, voice, and image. Index each type of information and extract keywords, topics, or summaries for subsequent processing. If the associated information involves multiple modalities, it is necessary to ensure their consistency in time or space for subsequent fusion processing. For text information, methods such as bag of words, TF-IDF, and word embedding can be used to extract features. These methods can convert text into vector form and capture key information in the text. For image information, methods such as convolutional neural networks can be used to extract features. Convolutional neural networks can automatically learn local and global features in images and represent them as vectors. For voice information, methods such as Mel frequency cepstral coefficients and linear prediction coding can be used to extract features. Mel frequency cepstral coefficients and linear prediction coding can capture the pitch and rhythm information in voice. For video information, features such as key frames, motion trajectories, and color histograms can be extracted. These features can reflect the content and dynamic information of the video. Convert the extracted features into a unified feature representation, such as one-hot encoding, word vector, image feature vector, etc. Normalize the feature vectors to ensure that they have the same scale in subsequent processing. Combine feature vectors of different modes to form a comprehensive feature vector. This feature vector can contain multiple modal information in the associated information, providing rich feature representation for subsequent processing.

[0110] S111, obtaining a similarity value between the feature vector and the weighted text according to a similarity algorithm;

[0111] The similarity value between the feature vector and the weighted text is obtained according to the similarity algorithm. The working principle of this step is mainly based on the vector space model and the similarity calculation method. The feature vector is extracted from the associated information through the preprocessing step. The feature vector is used to represent the core content or key features of the information. The feature vector is a point in a high-dimensional space. Each dimension corresponds to a feature, and the value of the dimension represents the importance or weight of the feature in the information. Weighted text refers to text with a specific weight or importance. These texts are usually closely related to the target of the query or analysis. The weighted text can be converted into a feature vector form for comparison with the feature vector of the associated information.

[0112] The similarity algorithm may be a Tanimoto similarity algorithm, which is defined as:

[0113] ;

[0114] Among them, X and Y are two sets, X∩Y is the intersection of X and Y, and are the number of elements of X and Y respectively.

[0115] First, calculate the similarity of sets X and Y, and then calculate the intersection X∩Y of sets X and Y. The intersection contains the elements that appear in both X and Y. Then calculate the number of elements in sets X and Y respectively. and . According to the definition of Tanimoto similarity, the similarity between the two sets is obtained by dividing the size of the intersection by the sum of the number of elements in the two sets minus the size of the intersection. Finally, the calculated similarity value is output. This value is a real number between 0 and 1, indicating the degree of similarity between the two sets. The closer the value is to 1, the more similar the two sets are; the closer the value is to 0, the less similar the two sets are. The Tanimoto similarity algorithm can calculate the similarity between two sets. The calculation process is simple and suitable for processing large-scale data sets, which significantly improves the efficiency of data processing. The Tanimoto similarity algorithm can accurately reflect the degree of similarity between two sets by calculating the relationship between the intersection of the sets and the total number of elements.

[0116] S112, arranging the similarity values ​​in order to obtain a sorting result;

[0117] Input a set of multiple similarity values ​​to the multimodal large model for sorting. The similarity value is calculated by comparing the feature vector with the weight text. According to the specific needs and performance considerations, select the appropriate sorting algorithm. When the sorting is completed, a list or array containing the sorted similarity values ​​is generated. The elements in the list or array are arranged in order of similarity values, and the sorted similarity value set is output. The similarity value set is used for subsequent data analysis, visualization or decision support.

[0118] S113, extracting the target similarity value ranked first in the sorting results;

[0119] The intelligent customer service system will access this sorted similarity value set. Since the set has been sorted, the elements are arranged in order of similarity value. In the sorted set, the first-ranked element is usually located at the first position of the set. During the sorting process, the algorithm has ensured that the element with the largest or smallest similarity value is at the beginning of the set. Next, the intelligent customer service system will extract the element from the first position of the set and output it. The output element is the first-ranked element in the sorted similarity value set, that is, the target similarity value.

[0120] S114, determining target association information based on the target similarity value;

[0121] First, input the target similarity value ranked first in the similarity value set, which represents the similarity corresponding to the feature vector that is most similar to the weighted text. The similarity threshold is a preset value used to determine whether the target similarity value is high enough to confirm the correlation between its corresponding feature vector and the weighted text. By comparing the target similarity value with the preset similarity threshold. The comparison process is to determine whether the target similarity value has reached a high enough level to confirm the strong correlation between its corresponding feature vector and the weighted text. If the target similarity value is greater than or equal to the similarity threshold, it indicates that there is a strong correlation between the corresponding feature vector and the weighted text, indicating that the information represented by the feature vector and the information represented by the weighted text have a high similarity in content. Once it is confirmed that there is a strong correlation between the feature vector and the weighted text, the target association information can be determined based on the feature vector, and this information can be used as the output of the target association information.

[0122] S115: Display the target related information.

[0123] In this embodiment, this step first inputs the target association information determined in the previous step, selects a suitable display format according to the type of the target association information and the user's preference, and finally presents the processed target association information to the user.

[0124] As can be seen from the above embodiments, the beneficial effects brought by this solution are as follows: by collecting and processing multimodal data, the method can train a more comprehensive and accurate intelligent customer service model. The fusion of multimodal data enables the model to understand the user's intention more accurately, thereby improving the accuracy and relevance of the response. Data enhancement processing can increase the diversity of training data, help the multimodal large model learn more language patterns and features, and help improve the generalization ability of the model when facing new user inquiries. Using the Transformer architecture to build an initialized multimodal large model can improve the efficiency of model training by utilizing the parallel processing capability and self-attention mechanism of the architecture, and help the model better understand and process complex user inquiries. The method provided by this solution can provide users with a more personalized service experience by extracting weighted texts in user inquiry information and obtaining related information based on these weighted texts. By calculating the similarity value between the feature vector and the weighted text and sorting them according to the similarity, the method can give priority to displaying the most relevant related information to the user's inquiry, which helps to improve the user's information acquisition efficiency and optimize the overall user experience. The method supports multiple modal interaction methods such as text, voice, and image, so that users can choose the most appropriate interaction method according to their preferences and habits.

[0125] See also Figure 2The present application provides an embodiment of a method for obtaining a target multimodal large model, the embodiment comprising:

[0126] S201, obtaining a plurality of enhanced feature vectors of the enhanced data;

[0127] S202, inputting the enhanced feature vector into the initialized multimodal large model to obtain a multimodal feature vector;

[0128] S203, obtaining a prediction result according to the multimodal feature vector;

[0129] S204, using a cross entropy loss function to calculate the loss between the prediction result and the actual label, and obtaining a loss value;

[0130] S205, obtaining update parameters of the initialized multimodal large model through a back propagation algorithm according to the loss value;

[0131] S206. Update the initialized multimodal large model using the update parameters to obtain the target multimodal large model.

[0132] This embodiment describes a process of optimizing a multimodal large model for prediction through training. This process involves steps such as processing of enhanced data, feature extraction, model prediction, loss calculation, and model parameter update. The intelligent customer service system first processes the enhanced data. The enhanced data is the data obtained after preprocessing. The intelligent customer service system extracts multiple feature vectors from these data, and then inputs these feature vectors into an initialized multimodal large model. After the multimodal large model processes the input feature vector, it outputs a multimodal feature vector. The intelligent customer service system uses the multimodal feature vector to generate prediction results. The multimodal feature vector is passed to the output layer of the model, and the prediction result is obtained by calculation. In order to evaluate the prediction performance of the model, the system uses the cross entropy loss function to calculate the difference between the prediction result and the actual label. The cross entropy loss function is a method to measure the difference between two probability distributions. By calculating the loss value, the intelligent customer service system can quantify the degree of error in the model prediction.

[0133] Using the cross entropy loss function to calculate the loss between the predicted result and the actual label, and obtaining the loss value;

[0134] The cross entropy loss function is:

[0135]

[0136] Where N is the number of categories, yi is the one-hot encoding of the actual label, and pi is the probability of belonging to the i-th category predicted by the model.

[0137] The prediction result and actual label of the model. The loss value is calculated by using the cross entropy loss function, which represents the difference between the prediction result and the actual label. The cross entropy loss function is used to measure the difference between two probability distributions. In this embodiment, the two probability distributions are the prediction result pi of the model and the actual label yi. First, the prediction result and actual label of the target multimodal large model are obtained, and the actual label is converted into a one-hot encoding. According to the formula of the cross entropy loss function, the loss value of each category is calculated, and then the loss values ​​of all categories are added and divided by the number of categories N to obtain the final loss value L, which is used to evaluate the performance of the model. During the training process, the target multimodal large model will continuously adjust its parameters to minimize the loss value L. Through the back propagation algorithm, the gradient of the loss value L will be used to update the weights and biases of the model. In order to minimize the loss value L, an optimization algorithm is usually used to update the parameters of the model. In each iteration, the current loss value L is calculated, and the parameters of the model are updated according to its gradient.

[0138] After obtaining the loss value, the intelligent customer service system uses the back propagation algorithm to update the parameters of the model. The error signal is back propagated according to the loss value, the gradient of each parameter is calculated by the chain rule, and then the parameters of the model are updated according to these gradients. Finally, the intelligent customer service system uses the updated parameters to update the initialized multimodal large model to obtain the optimized target multimodal large model. After training, the multimodal large model can process input data more accurately and generate prediction results.

[0139] As can be seen from the above embodiments, the beneficial effects brought by this solution are as follows: by using enhanced data, the diversity and complexity of training data can be increased, thereby helping the multimodal large model to learn more features and patterns, which helps to improve the accuracy of the model when processing actual tasks. The multimodal large model in this embodiment can fuse data features from different modes and generate multimodal feature vectors, so that the model can understand the input data more comprehensively and improve the accuracy and relevance of the prediction results. The cross entropy loss function can accurately measure the difference between the model prediction results and the actual labels, providing a clear direction for the optimization of the model. By minimizing the loss value, the model can gradually adjust its parameters to improve the accuracy of the prediction. The loss value can directly reflect the performance of the model, so the training progress can be evaluated by monitoring the change of the loss value, so as to adjust the training strategy in time. The cross entropy loss function is particularly suitable for multi-classification tasks, can process data sets with multiple categories, and output the predicted probability of each category, making the model more flexible and accurate when dealing with complex multi-classification problems. During the training process, even if there are some wrong labels or noise data, the cross entropy loss function can be used to guide the model to learn the correct data distribution, thereby improving the generalization ability of the model.

[0140] See also Figure 3 The present application provides an embodiment of a method for performing data duplication check on multimodal data and obtaining duplicate check data, the embodiment comprising:

[0141] S301, cleaning the multimodal data to remove invalid, incomplete or erroneous data;

[0142] S302, classifying the cleaned multimodal data, wherein the multimodal data includes: text data, image data, audio data, and video data;

[0143] S303, using a first processing method to perform a duplicate check on the text data to obtain first duplicate check data;

[0144] S304, using a second processing method to perform a duplicate check on the image data to obtain second duplicate check data;

[0145] S305, using a third processing method to check for duplicates on the audio data to obtain third duplicate checking data;

[0146] S306, using a fourth processing method to check for duplicates on the video data to obtain fourth duplicate checking data;

[0147] S307: Merge the first duplicate checking data, the second duplicate checking data, the third duplicate checking data and the fourth duplicate checking data to generate a duplicate checking data set.

[0148] This embodiment mainly performs a series of preprocessing and classification duplication checking operations on multimodal data to obtain a non-repetitive, high-quality data set, ensuring the integrity and accuracy of the data. Through a specific duplication checking method for each modal data, duplicate content is effectively removed, and finally the duplication checking results of each modality are merged to form a unified duplication checking data set. The cleaned data is classified according to its modality. In this embodiment, multimodal data is divided into four categories: text data, image data, audio data, and video data. Cleaning multimodal data is mainly to remove invalid, incomplete or erroneous data. Invalid data may include null values, missing values, or data that is obviously not in line with expectations; incomplete data refers to data that is incomplete and cannot be used for subsequent analysis; erroneous data may be caused by encoding problems, storage problems, or transmission problems. The first processing method is to check for duplicate text data, which is used in conjunction with string matching, semantic similarity calculation, and hashing technology. String matching can be based on direct comparison of keywords, phrases or entire documents; semantic similarity calculation uses natural language processing technology to evaluate the similarity of meaning between texts; hash technology converts text into a fixed-length hash value, and determines whether the text is repeated by comparing the hash value. The second processing method is to check for duplicate image data, mainly through image feature extraction and matching technology. Feature extraction methods include: color histogram, edge detection, texture analysis, and feature vectors extracted by deep learning models, which are completed by calculating the similarity between feature vectors. The third processing method is to check for duplicate audio data. Audio duplication checking technology includes audio feature extraction, audio fingerprint generation and matching. By analyzing the waveform, spectrum and other features of the audio, a unique audio fingerprint is generated. By comparing the fingerprints of different audios, the system can identify duplicate audio content. The fourth processing method checks the video data for duplicate video. Video duplication checking usually combines the techniques of image duplication checking and audio duplication checking. By extracting key frames and audio features in the video, and checking for duplicate video separately.

[0149] It can be seen from the above embodiments that the beneficial effects brought by this solution are as follows: Through the data cleaning stage, invalid, incomplete or erroneous data are effectively removed to ensure the accuracy and reliability of the data used for subsequent analysis. Through duplicate checking, duplicate items in the data set are reduced, the impact of redundant information on the analysis results is avoided, and the overall quality of the data is improved. Classification of data allows each type of data to use the most suitable duplicate checking method, avoiding the inefficiency that may be caused by a single method for processing multiple types of data. Targeted duplicate checking reduces unnecessary computing overhead and improves the overall efficiency of data processing.

[0150] See also Figure 4 The present application provides an embodiment of a method for extracting weighted text in user query information, the embodiment comprising:

[0151] S401, extracting words from the user query information as nodes;

[0152] S402, constructing an undirected graph according to the co-occurrence relationship of the words in the user query information;

[0153] S403, calculating the weight value of each word according to the number of co-occurrences of the words in the undirected graph;

[0154] S404: Extract weight text in the user query information according to the weight value.

[0155] First, the user query information is preprocessed, such as word segmentation, stop word removal, etc., to obtain the vocabulary that constitutes the query information. These words are used as nodes to provide a basis for the subsequent construction of an undirected graph. According to the co-occurrence relationship of the extracted words in the user query information, an undirected graph is constructed. In this undirected graph, if two words appear in the user query information at the same time, there will be an edge connecting them, which can reflect the correlation between the words and provide a basis for the subsequent calculation of the weight value. For each node in the undirected graph, its weight value is calculated according to the number of co-occurrences with other nodes. The more co-occurrences, the more important the word is in the user query information, so its weight value is also greater. According to the calculated weight value, the words in the user query information are sorted, and the words with higher weight values ​​are selected as weight texts to reflect the focus or key information of the user query. The extracted weight texts can be used for subsequent analysis, processing or answer generation tasks to improve the accuracy and efficiency of the system.

[0156] It can be seen from the above embodiments that the beneficial effects brought about by this solution are as follows: By considering the co-occurrence relationship of words in user query information, this method can more accurately capture the correlation between words and more accurately extract key information that reflects user intentions. The calculation of weight values ​​can improve the recognition of key information, so that the extracted weighted text meets the focus and emphasis of the user's inquiry. This method realizes the rapid processing of a large amount of user query information by constructing an undirected graph and calculating weight values, thereby improving the efficiency of information processing. The extracted weighted text can be directly used for subsequent analysis, processing or answer generation tasks, reducing unnecessary information screening and filtering steps, and further improving processing efficiency.

[0157] In the above embodiment, the multimodal large model is configured with a softmax classification algorithm, and the softmax classification algorithm is defined as:

[0158] ;

[0159] Among them, c represents the classification category, represents the probability that sample i belongs to category c, Represents the exponential sum of all category scores.

[0160] In a multimodal large model, the softmax classification algorithm is usually used in the output layer of the model to convert the processing results of the multimodal large model on the input data of multiple modalities such as text, image, audio, etc. into a probability distribution. The multimodal large model receives input data from different modalities, which are converted into classification scores after being processed inside the multimodal large model. The classification scores are processed by the softmax function and converted into probability distribution. According to the probability distribution, the category with the highest probability is selected as the final classification result.

[0161] It can be seen from the above embodiments that the beneficial effects brought by this solution are as follows: The Softmax classification algorithm can convert the output of the multimodal large model into a probability distribution, making the classification result more intuitive and accurate. By comparing the probabilities of different categories, the multimodal large model can select the category with the highest probability as the prediction result, thereby improving the accuracy of classification. The Softmax classification algorithm is suitable for processing multimodal data and can combine the features and information of different modes for joint classification. During the model training process, the Softmax classification algorithm can be combined with the cross entropy loss function algorithm to achieve efficient model training. Through continuous iterative optimization, the classification performance of the multimodal large model can be improved.

[0162] See also Figure 5 The present application also provides a multi-modal large-model intelligent customer service training device, the device comprising:

[0163] A collecting unit 501 is used to collect multimodal data, where the multimodal data includes text, voice and words;

[0164] A duplicate checking unit 502 is used to perform a duplicate checking on the multimodal data to obtain duplicate checking data;

[0165] A first preprocessing unit 503 is used to preprocess the duplicate checking data to obtain preprocessed data;

[0166] A construction unit 504 is used to construct an initial multimodal large model based on the Transformer architecture;

[0167] The acquisition unit 505 is used to acquire user query information;

[0168] A first extraction unit 506, configured to extract weighted text from the user query information;

[0169] An input unit 507, configured to input the weighted text into the target multimodal large model, so that the target multimodal large model obtains associated information according to the weighted text, wherein the associated information includes text, video, voice and image information corresponding to the weighted text;

[0170] A second preprocessing unit 508, configured to preprocess the associated information to generate a feature vector;

[0171] A similarity calculation unit 509, configured to obtain a similarity value between the feature vector and the weighted text according to a similarity algorithm;

[0172] A sorting unit 510, configured to arrange the similarity values ​​in order to obtain a sorting result;

[0173] A second extraction unit 511 is used to extract the target similarity value ranked first in the sorting result;

[0174] A determination unit 512, configured to determine target association information based on the target similarity value;

[0175] The output unit 513 is used to display the target association information.

[0176] See also Figure 6 The present application also provides a multi-modal large-model intelligent customer service training device, the device comprising:

[0177] Processor 601, memory 602, input and output unit 603, bus 604;

[0178] The processor 601 is connected to the memory 602, the input and output unit 603 and the bus 604;

[0179] The memory 602 stores a program, and the processor 601 calls the program to execute any of the above methods.

[0180] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above methods.

[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0182] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0183] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0185] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A multi-modal large model intelligent customer service training method, characterized in that: The training method comprises: Collecting multimodal data, the multimodal data including text, voice and words; Performing data duplication check on the multimodal data to obtain duplicate check data; Preprocessing the duplicate checking data to obtain preprocessed data; Performing data enhancement processing on the preprocessed data by using an enhanced algorithm to obtain enhanced data, wherein the data enhancement processing includes performing synonym replacement, random insertion and deletion of text on the preprocessed data; Initialize a large multimodal model based on the Transformer architecture; Training the initialized multimodal large model according to the enhanced data to obtain a target multimodal large model; Obtain user query information; Extracting weighted text from the user query information; Inputting the weighted text into the target multimodal large model, so that the target multimodal large model obtains associated information according to the weighted text, wherein the associated information includes text, video, voice and image information corresponding to the weighted text; Preprocessing the association information to generate a feature vector; Obtaining a similarity value between the feature vector and the weighted text according to a similarity algorithm; Arrange the similarity values ​​in order to obtain a sorting result; Extracting the target similarity value ranked first in the sorting results; Determining target association information based on the target similarity value; The target association information is displayed.

2. The multimodal large model intelligent customer service training method according to claim 1 is characterized in that: The step of training the initialized multimodal large model according to the enhanced data to obtain a target multimodal large model includes: Acquire multiple enhanced feature vectors of the enhanced data; Inputting the enhanced feature vector into the initialized multimodal large model to obtain a multimodal feature vector; Obtaining a prediction result according to the multimodal feature vector; Using the cross entropy loss function to calculate the loss between the predicted result and the actual label, and obtaining the loss value; According to the loss value, obtaining update parameters of the initialized multimodal large model through a back propagation algorithm; The initialized multimodal large model is updated by using the update parameters to obtain a target multimodal large model.

3. The multimodal large model intelligent customer service training method according to claim 2 is characterized in that: The cross entropy loss function is used to calculate the loss between the prediction result and the actual label, and the loss value is obtained, including: Using the cross entropy loss function to calculate the loss between the predicted result and the actual label, and obtaining the loss value; The cross entropy loss function is: Where N is the number of categories, yi is the one-hot encoding of the actual label, and pi is the probability of belonging to the i-th category predicted by the model.

4. The multimodal large model intelligent customer service training method according to claim 1, characterized in that: The performing data duplication checking on the multimodal data to obtain duplicate checking data includes: Cleaning the multimodal data to remove invalid, incomplete or erroneous data; Classifying the cleaned multimodal data, wherein the multimodal data includes: text data, image data, audio data, and video data; Using a first processing method to check the text data for duplicates, so as to obtain first duplicate checking data; Using a second processing method to check the image data for duplicates, so as to obtain second duplicate checking data; Using a third processing method to check the audio data for duplication, so as to obtain third duplication checking data; Using a fourth processing method to check the video data for duplication, so as to obtain fourth duplication checking data; The first duplicate checking data, the second duplicate checking data, the third duplicate checking data and the fourth duplicate checking data are merged to generate a duplicate checking data set.

5. The multimodal large model intelligent customer service training method according to claim 1, characterized in that: The step of extracting the weighted text in the user query information includes: Extracting words from the user query information as nodes; Constructing an undirected graph based on the co-occurrence relationship of the words in the user's query information; Calculating a weight value of each word according to the number of co-occurrences of the words in the undirected graph; The weight text in the user query information is extracted according to the weight value.

6. The multi-modal large model intelligent customer service training method according to claim 1, characterized in that: The multimodal large model is configured with a softmax classification algorithm, which is defined as: Among them, c represents the classification category, represents the probability that sample i belongs to category c, Represents the exponential sum of all category scores.

7. The multimodal large model intelligent customer service training method according to claim 1, characterized in that: The similarity algorithm is the Tanimoto similarity algorithm, and the Tanimoto similarity algorithm is defined as: Among them, X and Y are two sets, X∩Y is the intersection of X and Y, and are the number of elements of X and Y respectively.

8. A multi-modal large-model intelligent customer service training device, characterized in that: The device comprises: A collecting unit, used to collect multimodal data, wherein the multimodal data includes text, voice and words; A duplicate checking unit, used for performing a duplicate checking on the multimodal data to obtain duplicate checking data; A preprocessing unit, used for preprocessing the duplicate checking data to obtain preprocessed data; Construction unit, used to build and initialize a large multimodal model based on the Transformer architecture; An acquisition unit, used for acquiring user query information; A first extraction unit, used to extract weighted words from the user query information; An input unit, used to input the weighted text into the target multimodal large model, so that the target multimodal large model obtains associated information according to the weighted text, wherein the associated information includes text, video, voice and image information corresponding to the weighted text; A preprocessing unit, used for preprocessing the associated information to generate a feature vector; A similarity calculation unit, used for obtaining a similarity value between the feature vector and the weighted text according to a similarity algorithm; A sorting unit, used to arrange the similarity values ​​in order to obtain a sorting result; A second extraction unit, used to extract the target similarity value ranked first in the sorting result; A determination unit, configured to determine target association information based on the target similarity value; An output unit is used to display the target related information.

9. A multi-modal large-model intelligent customer service training device, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 7.

Images