Service quality monitoring method and device based on multi-modal model sentiment analysis

Abstract

The invention discloses a service quality monitoring method and device based on multi-modal model sentiment analysis, and relates to the technical field of data analysis, and the method comprises the steps: collecting text modal evaluation data and non-text modal evaluation data of a to-be-monitored service; performing feature extraction mapping on the non-text modal evaluation data, and determining a target non-text modal evaluation representation; combining the text modal evaluation data with a predefined cue word template to generate a text evaluation cue word; inputting the text evaluation prompt words and the target non-text modal evaluation representation into a preset large language model base for sentiment analysis, and outputting a sentiment analysis result; and calculating a service quality monitoring score according to the sentiment analysis result and the text evaluation prompt, and decoding the sentiment analysis result to obtain a service quality monitoring result explanation. The service quality monitoring scheme has good universality, fine monitoring granularity and self-interpretation, and the service quality monitoring reliability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a service quality monitoring method and device based on multimodal model sentiment analysis. Background Art

[0002] With the development of science and technology, physical services combined with Internet technology have derived a series of digital services, such as Internet e-commerce, online taxis (online car-hailing), etc. The convenience brought by digital services to people's lives has led to a sharp increase in their scale. However, the rapid growth in scale has also led to uneven service quality in digital services, and even brought personal safety to consumers. In order to create a good digital service atmosphere and improve the quality of digital services, it is necessary to build a service quality monitoring algorithm to analyze and supervise the service process and its evaluation.

[0003] Existing service quality monitoring methods mainly monitor the quality of related services by pre-defining quality monitoring indicators or training specific quality monitoring models. As far as pre-defined monitoring indicators are concerned, existing technologies mainly build service quality evaluation systems around indicators related to traditional networks and communication services. Commonly used monitoring indicators include service response time, service accuracy, service reliability, etc., but they are often only applicable to a single data modality and fixed scenarios, and it is difficult to cover settings such as digital services that have a wide range of scenarios. In terms of training specific quality monitoring models, existing technologies regard service monitoring as a binary or multi-classification task, and construct a corresponding data set to train a quality monitoring classification model. Due to the limitations of the classification categories of the quality monitoring classification model and the data set, the service quality rating of this type of method is often rough, and it is difficult to give relevant explanations for the service quality rating results. Therefore, the service quality monitoring reliability of existing service quality monitoring methods is low. Summary of the invention

[0004] The present invention provides a service quality monitoring method and device based on multimodal model sentiment analysis, which solves the technical problem that the existing service quality monitoring method is usually only applicable to a single data modality and a fixed scenario, the service quality rating is often rough, and it is difficult to give relevant explanations for the service quality rating results, resulting in low reliability of service quality monitoring.

[0005] A first aspect of the present invention provides a service quality monitoring method based on multimodal model sentiment analysis, comprising:

[0006] Collecting text modality evaluation data and non-text modality evaluation data of the service to be monitored;

[0007] Performing feature extraction and mapping on the non-text modality evaluation data to determine a target non-text modality evaluation representation;

[0008] Combining the text modality evaluation data with a predefined prompt word template to generate a text evaluation prompt word;

[0009] Input the text evaluation prompt and the target non-text modal evaluation representation into a preset large language model base for sentiment analysis, and output a sentiment analysis result;

[0010] A service quality monitoring score is calculated according to the sentiment analysis result and the text evaluation prompt, and the sentiment analysis result is decoded to obtain a service quality monitoring result explanation.

[0011] Optionally, the non-text modality evaluation data includes image modality evaluation data and audio modality evaluation data; and the step of performing feature extraction and mapping on the non-text modality evaluation data to determine a target non-text modality evaluation representation includes:

[0012] Using a preset visual representation extractor to extract features from the image modality evaluation data, and outputting an intermediate image modality evaluation representation;

[0013] Perform feature extraction on the audio modal evaluation data based on a preset audio representation extractor, and output an intermediate audio modal evaluation representation;

[0014] The intermediate image modality evaluation representation and the intermediate audio modality evaluation representation are dimensionally aligned through a preset representation mapper to generate a target image modality evaluation representation and a target audio modality evaluation representation.

[0015] Optionally, the predefined prompt word template includes a task definition, a sentiment definition and an evaluation text.

[0016] Optionally, the calculation process of the service quality monitoring score includes:

[0017] ;

[0018] in, ;

[0019] In the formula, For service quality monitoring scores, is the token sequence of sentiment analysis results, The token sequence of the sentiment analysis result Tokens, is the length of the token sequence of the sentiment analysis result, is the probability score, For predefined prompt word templates, For the task definition, Define emotion, To evaluate the text, is the target image modality evaluation data, is the target audio modality evaluation data, is the weight of the multimodal large model.

[0020] Optionally, it also includes:

[0021] The mapping training dataset is constructed using non-text modality training evaluation data, question text, and model reply truth values;

[0022] Performing feature extraction on the non-text modality training evaluation data by the visual representation extractor or the audio representation extractor, and outputting a corresponding first non-text modality training representation;

[0023] Based on the initial representation mapper, feature mapping is performed on each of the first non-text modality training representations using a mapping matrix of a corresponding non-text modality to obtain a corresponding second non-text modality training representation;

[0024] Using the large language model base in turn, the question text and the second non-text modality training representation are used as input, and outputting a predicted value of a response to the question text;

[0025] Based on each of the response prediction values ​​and the corresponding model response true value, each mapping matrix is ​​reversely iterated and updated in turn to determine the representation mapper of each non-text modality.

[0026] Optionally, it also includes:

[0027] Obtain text modality fine-tuning evaluation data, non-text modality fine-tuning evaluation data, and corresponding fine-tuning reply true values;

[0028] After extracting features of the non-text modality fine-tuning evaluation data by the visual representation extractor or the audio representation extractor, the representation mapper is used to perform feature mapping to output a corresponding non-text modality fine-tuning evaluation representation;

[0029] The text modality fine-tuning evaluation data is used to construct a fine-tuning prompt, and the fine-tuning prompt and the corresponding non-text modality fine-tuning evaluation representation and fine-tuning reply truth value are used to construct a fine-tuning dataset;

[0030] The mapping matrix of the representation mapper and the large language model base are fine-tuned based on the low-rank fine-tuning method using the fine-tuning dataset to obtain a new representation mapper and a new large language model base.

[0031] A second aspect of the present invention provides a service quality monitoring device based on multimodal model sentiment analysis, comprising:

[0032] A data acquisition module, used to collect text modality evaluation data and non-text modality evaluation data of the service to be monitored;

[0033] A feature processing module, used to perform feature extraction and mapping on the non-text modality evaluation data to determine a target non-text modality evaluation representation;

[0034] A prompt word construction module, used to combine the text modality evaluation data with a predefined prompt word template to generate a text evaluation prompt word;

[0035] A sentiment analysis module, used for inputting the text evaluation prompt and the target non-text modal evaluation representation into a preset large language model base for sentiment analysis, and outputting a sentiment analysis result;

[0036] The service quality monitoring module is used to calculate the service quality monitoring score according to the sentiment analysis result and the text evaluation prompt, and decode the sentiment analysis result to obtain the service quality monitoring result explanation.

[0037] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the service quality monitoring method based on multimodal model sentiment analysis as described in any one of the above items.

[0038] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the service quality monitoring method based on multimodal model sentiment analysis as described in any one of the above items.

[0039] A fifth aspect of the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements a service quality monitoring method based on multimodal model sentiment analysis as described in any one of the above items.

[0040] It can be seen from the above technical solutions that the present invention has the following advantages:

[0041] The above scheme of the present invention provides a service quality monitoring method based on multimodal model sentiment analysis, including: collecting text modal evaluation data and non-text modal evaluation data of the service to be monitored; performing feature extraction and mapping on the non-text modal evaluation data to determine the target non-text modal evaluation representation; combining the text modal evaluation data with a predefined prompt word template to generate a text evaluation prompt; inputting the text evaluation prompt and the target non-text modal evaluation representation into a preset large language model base for sentiment analysis, and outputting the sentiment analysis result; calculating the service quality monitoring score according to the sentiment analysis result and the text evaluation prompt, and decoding the sentiment analysis result to obtain the service quality monitoring result explanation. By setting a specific predefined prompt word template to guide the large language model base to complete the sentiment analysis task on the multimodal evaluation data, the service quality score is obtained based on the probability score statistics to determine the service quality rating, and the service quality monitoring result can be attached with explanation information. The above service quality monitoring scheme has good universality, fine monitoring granularity and self-explanatory properties, which improves the reliability of service quality monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 A flowchart of the steps of a service quality monitoring method based on multimodal model sentiment analysis provided by an embodiment of the present invention;

[0044] Figure 2 A framework diagram of a service quality monitoring model based on multimodal model sentiment analysis provided by an embodiment of the present invention;

[0045] Figure 3 A flow chart of model fine-tuning provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of model fine-tuning provided by an embodiment of the present invention;

[0047] Figure 5 A structural block diagram of a service quality monitoring device based on multimodal model sentiment analysis provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The embodiments of the present invention provide a service quality monitoring method and device based on multimodal model sentiment analysis, which are used to solve the technical problems that the existing service quality monitoring methods are usually only applicable to a single data modality and a fixed scenario, the service quality rating is often rough, and it is difficult to give relevant explanations for the service quality rating results, resulting in low reliability of service quality monitoring.

[0049] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] See also Figure 1 , Figure 1 A flowchart of the steps of a service quality monitoring method based on multimodal model sentiment analysis provided in an embodiment of the present invention.

[0051] The present invention provides a service quality monitoring method based on multimodal model sentiment analysis, comprising:

[0052] Step 101: Collect text modality evaluation data and non-text modality evaluation data of the service to be monitored.

[0053] It should be noted that in order to implement service quality monitoring from the perspective of consumers, evaluations of the service to be monitored can be obtained from consumers to determine the quality of the service. In this embodiment, multimodal evaluation data of the service to be monitored is collected. The multimodal evaluation data includes text modal evaluation data and non-text modal evaluation data, wherein the non-text modal evaluation data may include evaluation data of modalities such as images, videos, and audio.

[0054] Step 102: Perform feature extraction and mapping on the non-text modality evaluation data to determine the target non-text modality evaluation representation.

[0055] The non-text modality evaluation data includes image modality evaluation data and audio modality evaluation data; step 102 includes the following sub-steps:

[0056] Using a preset visual representation extractor to extract features from the image modality evaluation data, and outputting an intermediate image modality evaluation representation;

[0057] Perform feature extraction on the audio modal evaluation data based on a preset audio representation extractor, and output an intermediate audio modal evaluation representation;

[0058] The intermediate image modal evaluation representation and the intermediate audio modal evaluation representation are dimensionally aligned through a preset representation mapper to generate a target image modal evaluation representation and a target audio modal evaluation representation.

[0059] It should be noted that, since the evaluation data needs to be analyzed by a large language model base in subsequent steps, and the large language model base is mainly pre-trained based on text data and cannot directly process non-text modal data, in this embodiment, the non-text modal evaluation data is first input into the corresponding representation extractor for feature extraction. It can be understood that the representation extractor is generally a model that has completed pre-training and can be used without further training; in specific implementation, when the non-text modal evaluation data includes image modal evaluation data, it corresponds to feature extraction through a visual representation extractor to obtain an intermediate image modal evaluation representation. The visual representation extractor can use a visual encoder trained by CLIP, and its main architecture is a visual Transformer; when the non-text modal evaluation data includes audio modal evaluation data, it corresponds to an audio representation extractor for feature extraction to obtain an intermediate audio modal evaluation representation. The audio representation extractor can be a WHISPER-BASE model, which is a multilingual speech recognition model trained based on a large number of audio data sets;

[0060] Since the intermediate evaluation representations of different modalities come from different representation extractors, there are differences in the representation feature spaces of different modalities. For this reason, this embodiment introduces a trainable mapping matrix of the same size but not sharing parameters for different modalities into the representation mapper to map the intermediate evaluation representations of different modalities into the same feature space, thereby generating a unified representation after mapping, namely, a target image modality evaluation representation and a target audio modality evaluation representation. In specific implementation, the representation mapper can use multi-layer perceptrons (MLPs), such as two-layer MLPs; the above feature mapping conversion process can be summarized as follows: , ,in, is the image modality representation after feature mapping, is the image modality mapping matrix, is the image modality representation before feature mapping, is the audio modality representation after feature mapping, is the audio mode mapping matrix, It is the audio modality representation before feature mapping.

[0061] Step 103: Combine the text modality evaluation data with a predefined prompt word template to generate a text evaluation prompt.

[0062] Optionally, the predefined prompt word template includes a task definition, a sentiment definition, and an evaluation text.

[0063] It should be noted that, in this embodiment, the purpose of setting the predefined prompt word template is to guide the large language model base to complete the emotion classification task. In the specific implementation, the predefined prompt word template includes task definition, emotion definition and evaluation text, wherein the task definition mainly guides the large language model base to complete the emotion classification task of multimodal evaluation data and gives corresponding explanations to the results, and the emotion definition assists the large language model base to understand the difference between good emotions and bad emotions, so as to complete accurate classification; the reference format of the predefined prompt word template is as follows:

[0064] “Please infer the emotion category contained in the following evaluation data based on the definition of good / bad emotion. Please directly answer "good emotion" or "bad emotion" and give corresponding explanations.

[0065] <Definition of good / bad emotions>: Good emotions refer to consumers' satisfaction with the quality of the service, and the evaluation data reveals emotions such as happiness, excitement, and gratitude. Bad emotions are the opposite, and the evaluation data reveals emotions such as disappointment, anger, and helplessness.

[0066] <Evaluation data>: xxx (text data)"

[0067] Step 104: input the text evaluation prompt and the target non-text modal evaluation representation into a preset large language model base for sentiment analysis, and output the sentiment analysis result.

[0068] It should be noted that the comprehensive text evaluation prompts and the target non-text modal evaluation representation are input into the large language model base, the emotional situation in the evaluation data is analyzed and then the sentiment analysis results are output. It can be understood that the sentiment analysis results output by the large language model base are in the form of token sequences; in the specific implementation, relevant open source large language models can be used as the large language model base in this embodiment. For example, GLM-4-9B is selected as the large language model base. GLM-4-9B is the open source version of the latest generation of pre-trained models GLM-4 series launched by Zhipu AI.

[0069] Step 105: Calculate the service quality monitoring score according to the sentiment analysis result and the text evaluation prompt, and decode the sentiment analysis result to obtain the service quality monitoring result explanation.

[0070] It should be noted that this embodiment obtains a fine-grained service quality monitoring score by counting the probability of the content generated by the large language model base, that is, the sentiment analysis result, similar to the regression method; in terms of the service monitoring score, the text evaluation prompt generated according to the predefined prompt word template can also provide the content required for probability statistics, so it has the function of assisting in completing the regression task. For the text evaluation prompt, the stronger the emotional expression of good or bad emotions in the evaluation text, the higher the confidence of the large language model base in the reply. Therefore, each selected token in the generated reply will have a larger probability score. According to this principle, this embodiment obtains the intensity of emotional expression by counting the average probability score of the generated content of the large language model base as the service quality monitoring score; the calculation process of the service quality monitoring score includes:

[0071] ;

[0072] in, ;

[0073] In the formula, For service quality monitoring scores, is the token sequence of sentiment analysis results, The token sequence of the sentiment analysis result Tokens, is the length of the token sequence of the sentiment analysis result, is the probability score, For predefined prompt word templates, For the task definition, Define emotion, To evaluate the text, is the target image modality evaluation data, is the target audio modality evaluation data, is the multimodal large model weight (including the weights of the large language model base, representation extractor, and representation mapper mapping matrix);

[0074] In terms of result interpretation, the content generated by the large language model base when completing the sentiment classification task includes the explanation of the classification results. By decoding the token sequence of the sentiment analysis results back into natural language text, the explanation of the service quality monitoring results is obtained.

[0075] Optionally, the method further comprises:

[0076] The mapping training dataset is constructed using non-text modality training evaluation data, question text, and model reply truth values;

[0077] Performing feature extraction on the non-text modality training evaluation data by a visual representation extractor or an audio representation extractor, and outputting a corresponding first non-text modality training representation;

[0078] Based on the initial representation mapper, feature mapping is performed on each first non-text modality training representation using a mapping matrix of the corresponding non-text modality to obtain a corresponding second non-text modality training representation;

[0079] A large language model base is used to sequentially take the question text and the second non-text modality training representation as input, and output a predicted value of the response to the question text;

[0080] Based on each response prediction value and the corresponding model response true value, each mapping matrix is ​​reversely iterated and updated in turn to determine the representation mapper of each non-text modality.

[0081] It should be noted that since the representation extractors of each non-text modality and the large language model base are models that have been pre-trained on a large amount of data, the mapping matrix in the untrained representation mapper, i.e., the initial representation mapper and It has not been trained at all, so it is necessary to train the mapping matrix in the initial representation mapper to obtain the corresponding representation mapper;

[0082] In the specific implementation, in order to realize the training of the mapping matrix, it is necessary to first construct a mapping training data set. Each sample in the mapping training data set consists of a question text Q, non-text modality training evaluation data X, and a model reply truth value A corresponding to QX. The non-text modality training evaluation data can be a regular image or short audio data. The text modality question Q is set to guide the large language model base to understand the second non-text modality training representation content, which can better train the model, such as "What is the content of the image?", "Please generate a title for the image", etc.

[0083] Secondly, the weights of each representation extractor and the large language model base are fixed and remain unchanged. The non-text modality training evaluation data X is input into the representation extractor of the corresponding modality to obtain the first non-text modality training representation, and then input into the initial representation mapper for feature processing to obtain the second non-text modality training representation H. According to the following format, the second non-text modality training representation is combined with the question text Q at the representation layer of the large language model base. The question Q of the text modality is responsible for guiding the large language model base to output a summary response about the non-text modality data, and output the corresponding response prediction value:

[0084]

[0085] In order to ensure the training of mapping matrices of different modalities, only one non-text modality training evaluation data is input in one forward propagation, and only one modality mapping matrix is ​​updated in one back propagation. Therefore, based on the training loss value determined by each response prediction value and the corresponding model response true value, each mapping matrix is ​​reversely iterated and updated in turn to determine the representation mapper of each non-text modality.

[0086] Optionally, it also includes:

[0087] Obtain text modality fine-tuning evaluation data, non-text modality fine-tuning evaluation data, and corresponding fine-tuning reply true values;

[0088] After extracting features of the non-text modality fine-tuning evaluation data through a visual representation extractor or an audio representation extractor, a representation mapper is used to perform feature mapping, and the corresponding non-text modality fine-tuning evaluation representation is output;

[0089] The fine-tuning prompts are constructed using text modality fine-tuning evaluation data, and the fine-tuning dataset is constructed using the fine-tuning prompts and the corresponding non-text modality fine-tuning evaluation representations and fine-tuning reply truth values;

[0090] The fine-tuning dataset is used to fine-tune the mapping matrix of the representation mapper and the large language model base based on the low-rank fine-tuning method to obtain a new representation mapper and a new large language model base.

[0091] It should be noted that the service quality monitoring model framework constructed in this embodiment is as follows Figure 2 As shown, the multimodal large model consists of a large language model base, representation extractors of each non-text modality and their corresponding representation mapper mapping matrix. In order to ensure that the multimodal large model can successfully complete the emotion classification and result interpretation, it is also necessary to fix each representation extractor and fine-tune the large language model base and mapping matrix:

[0092] First, you need to build a fine-tuning dataset. In this embodiment, a single-round question-answering method can be used to build a fine-tuning dataset. The specific format is as follows:

[0093]

[0094] Based on the above format, obtain text modality fine-tuning evaluation data, non-text modality fine-tuning evaluation data and corresponding fine-tuning reply true values, perform feature extraction and mapping on the non-text modality fine-tuning evaluation data through the representation extractor and the representation mapper, output the corresponding non-text modality fine-tuning evaluation representation, generate fine-tuning prompts using the text modality fine-tuning evaluation data according to the aforementioned predefined prompt word template, and construct a fine-tuning dataset using each fine-tuning prompt and the corresponding non-text modality fine-tuning evaluation representation and fine-tuning reply true value; taking the Internet e-commerce service evaluation as an example, the format of the specific fine-tuning prompt is as follows:

[0095] { "question": "Please infer the emotion categories contained in the following evaluation data based on the definition of good / bad emotions. Please directly answer "good emotion" or "bad emotion" and give corresponding explanations.

[0096] <Definition of good / bad emotions>: Good emotions refer to consumers' satisfaction with the quality of the service, and the evaluation data reveals emotions such as happiness, excitement, and gratitude. Bad emotions are the opposite, and the evaluation data reveals emotions such as disappointment, anger, and helplessness.

[0097] <Evaluation data>: The hanging scroll poster made by the company for the exhibition is good, the actual color is very good, and it is very convenient",

[0098] "data_path1": "",

[0099] "data_path2": ". / data / evaluate_video.mp4",

[0100] "answer": "Good sentiment, the review shows the usefulness of the purchased service (item) and praises the quality of the service (item)."}

[0101] Then, the fine-tuning dataset is used to fine-tune the large language model base and the mapping matrix based on the low-rank fine-tuning method; taking the fine-tuning of the large language model base as an example, assuming that the GLM-4-9B model is used in this embodiment, the model has 9 billion parameters. If all parameters of the model are fine-tuned, the computing power and time cost will be very high. The basic idea of ​​fine-tuning is to add a bypass to the original large language model base as a representation of the change in parameters during model fine-tuning. This embodiment introduces the parameters that need to be updated in the model that needs to be fine-tuned. , when fine-tuning, only update the parameters that need to be updated , assuming that the weight parameter of the pre-trained large language model to be fine-tuned in the downstream task is W, and the pre-trained GLM-4-9B model parameter W is fixed, the model parameters that need to be updated are , It consists of two parts, A and B. The parameter dimension of part A is r×k, which is initialized using a random Gaussian distribution, and the parameter dimension of part B is d×k, which is initialized using a full zero matrix. The process of fine-tuning the model is as follows: Figure 3 and Figure 4 As shown:

[0102] S1. Initialize the parameters that need to be updated ;

[0103] S2, the fine-tuning prompt and the corresponding non-text modality fine-tuning evaluation representation are input into the model as vector x and respectively compared with W, Perform the operation and add the results to output the intermediate result of the forward propagation The intermediate result is then processed by softmax to get the prediction of the next word :

[0104] ;

[0105] ;

[0106] S3, according to the preset loss function, through the prediction of the next word The loss is calculated with the label value, i.e., the fine-tuned restored true value y; assuming that the preset loss function is the cross entropy loss, for a true value sentence consisting of n words, is the predicted value of each word in the sentence, is the true value of each word in the sentence, the loss function is calculated as follows:

[0107] ;

[0108] S4. Based on the calculated loss value, update the parameters through gradient descent using the back propagation algorithm .

[0109] In an embodiment of the present invention, a specific predefined prompt word template is set to guide the large language model base to complete the task of sentiment analysis on multimodal evaluation data, and a service quality score is obtained based on probability score statistics to determine the service quality rating, and explanatory information can be attached to the service quality monitoring results. The above-mentioned service quality monitoring scheme has good universality, fine monitoring granularity and self-explanatoryness, which improves the reliability of service quality monitoring.

[0110] See also Figure 5 , Figure 5 A structural block diagram of a service quality monitoring device based on multimodal model sentiment analysis provided in an embodiment of the present invention.

[0111] The present invention provides a service quality monitoring device based on multimodal model sentiment analysis, comprising:

[0112] A data acquisition module 501 is used to collect text modality evaluation data and non-text modality evaluation data of the service to be monitored;

[0113] A feature processing module 502 is used to perform feature extraction and mapping on the non-text modality evaluation data to determine the target non-text modality evaluation representation;

[0114] The prompt word construction module 503 is used to combine the text modality evaluation data with the predefined prompt word template to generate a text evaluation prompt word;

[0115] Sentiment analysis module 504, used for inputting text evaluation prompts and target non-text modal evaluation representation into a preset large language model base for sentiment analysis, and outputting sentiment analysis results;

[0116] The service quality monitoring module 505 is used to calculate the service quality monitoring score according to the sentiment analysis result and the text evaluation prompt, and decode the sentiment analysis result to obtain the service quality monitoring result explanation.

[0117] Optionally, the non-text modality evaluation data includes image modality evaluation data and audio modality evaluation data; the feature processing module 502 is specifically used for:

[0118] Using a preset visual representation extractor to extract features from the image modality evaluation data, and outputting an intermediate image modality evaluation representation;

[0119] Perform feature extraction on the audio modal evaluation data based on a preset audio representation extractor, and output an intermediate audio modal evaluation representation;

[0120] The intermediate image modal evaluation representation and the intermediate audio modal evaluation representation are dimensionally aligned through a preset representation mapper to generate a target image modal evaluation representation and a target audio modal evaluation representation.

[0121] Optionally, the predefined prompt word template includes a task definition, a sentiment definition, and an evaluation text.

[0122] Optionally, the calculation process of the service quality monitoring score includes:

[0123] ;

[0124] in, ;

[0125] In the formula, For service quality monitoring scores, is the token sequence of sentiment analysis results, The token sequence of the sentiment analysis result Tokens, is the length of the token sequence of the sentiment analysis result, is the probability score, For predefined prompt word templates, For the task definition, Define emotion, To evaluate the text, is the target image modality evaluation data, is the target audio modality evaluation data, is the weight of the multimodal large model.

[0126] Optionally, a model training module is also included, for:

[0127] The mapping training dataset is constructed using non-text modality training evaluation data, question text, and model reply truth values;

[0128] Performing feature extraction on the non-text modality training evaluation data by a visual representation extractor or an audio representation extractor, and outputting a corresponding first non-text modality training representation;

[0129] Based on the initial representation mapper, feature mapping is performed on each first non-text modality training representation using a mapping matrix of the corresponding non-text modality to obtain a corresponding second non-text modality training representation;

[0130] The large language model base is used to sequentially take the question text and the second non-text modality training representation as input, and output the predicted value of the response to the question text;

[0131] Based on each response prediction value and the corresponding model response true value, each mapping matrix is ​​reversely iterated and updated in turn to determine the representation mapper of each non-text modality.

[0132] Optionally, a model fine-tuning model is also included to:

[0133] Obtain text modality fine-tuning evaluation data, non-text modality fine-tuning evaluation data, and corresponding fine-tuning reply true values;

[0134] After extracting features of the non-text modality fine-tuning evaluation data through a visual representation extractor or an audio representation extractor, a representation mapper is used to perform feature mapping, and the corresponding non-text modality fine-tuning evaluation representation is output;

[0135] The fine-tuning prompts are constructed using text modality fine-tuning evaluation data, and the fine-tuning dataset is constructed using the fine-tuning prompts and the corresponding non-text modality fine-tuning evaluation representations and fine-tuning reply truth values;

[0136] The fine-tuning dataset is used to fine-tune the mapping matrix of the representation mapper and the large language model base based on the low-rank fine-tuning method to obtain a new representation mapper and a new large language model base.

[0137] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the service quality monitoring method based on multimodal model sentiment analysis as in any of the above embodiments.

[0138] An embodiment of the present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the service quality monitoring method based on multimodal model sentiment analysis as described in any of the above embodiments are implemented.

[0139] An embodiment of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a service quality monitoring method based on multimodal model sentiment analysis as described in any of the above embodiments.

[0140] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0141] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0144] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0145] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A service quality monitoring method based on multimodal model sentiment analysis, characterized in that: include: Collecting text modality evaluation data and non-text modality evaluation data of the service to be monitored; Performing feature extraction and mapping on the non-text modality evaluation data to determine a target non-text modality evaluation representation; Combining the text modality evaluation data with a predefined prompt word template to generate a text evaluation prompt word; Input the text evaluation prompt and the target non-text modal evaluation representation into a preset large language model base for sentiment analysis, and output a sentiment analysis result; A service quality monitoring score is calculated according to the sentiment analysis result and the text evaluation prompt, and the sentiment analysis result is decoded to obtain a service quality monitoring result explanation.

2. The service quality monitoring method based on multimodal model sentiment analysis according to claim 1 is characterized in that: The non-text modality evaluation data includes image modality evaluation data and audio modality evaluation data; The performing feature extraction and mapping on the non-text modality evaluation data to determine the target non-text modality evaluation representation includes: Using a preset visual representation extractor to extract features from the image modality evaluation data, and outputting an intermediate image modality evaluation representation; Perform feature extraction on the audio modal evaluation data based on a preset audio representation extractor, and output an intermediate audio modal evaluation representation; The intermediate image modality evaluation representation and the intermediate audio modality evaluation representation are dimensionally aligned through a preset representation mapper to generate a target image modality evaluation representation and a target audio modality evaluation representation.

3. The service quality monitoring method based on multimodal model sentiment analysis according to claim 1 is characterized in that: The predefined prompt word template includes task definition, emotion definition and evaluation text.

4. The service quality monitoring method based on multimodal model sentiment analysis according to claim 1 is characterized in that: The calculation process of the service quality monitoring score includes: ； in, ; In the formula, For service quality monitoring scores, is the token sequence of sentiment analysis results, is the token sequence of the sentiment analysis result Tokens, is the length of the token sequence of the sentiment analysis result. is the probability score, For predefined prompt word templates, For the task definition, Define emotion, To evaluate the text, is the target image modality evaluation data, is the target audio modality evaluation data, is the weight of the multimodal large model.

5. The service quality monitoring method based on multimodal model sentiment analysis according to claim 2 is characterized in that: Also includes: The mapping training dataset is constructed using non-text modality training evaluation data, question text, and model reply truth values; Performing feature extraction on the non-text modality training evaluation data by the visual representation extractor or the audio representation extractor, and outputting a corresponding first non-text modality training representation; Based on the initial representation mapper, feature mapping is performed on each of the first non-text modality training representations using a mapping matrix of a corresponding non-text modality to obtain a corresponding second non-text modality training representation; Using the large language model base in turn, the question text and the second non-text modality training representation are used as input, and outputting a predicted value of a response to the question text; Based on each of the response prediction values ​​and the corresponding model response true value, each mapping matrix is ​​reversely iterated and updated in turn to determine the representation mapper of each non-text modality.

6. The service quality monitoring method based on multimodal model sentiment analysis according to claim 2 is characterized in that: Also includes: Obtain text modality fine-tuning evaluation data, non-text modality fine-tuning evaluation data, and corresponding fine-tuning reply true values; After extracting features of the non-text modality fine-tuning evaluation data by the visual representation extractor or the audio representation extractor, the representation mapper is used to perform feature mapping to output a corresponding non-text modality fine-tuning evaluation representation; The text modality fine-tuning evaluation data is used to construct a fine-tuning prompt, and the fine-tuning prompt and the corresponding non-text modality fine-tuning evaluation representation and fine-tuning reply truth value are used to construct a fine-tuning dataset; The mapping matrix of the representation mapper and the large language model base are fine-tuned based on the low-rank fine-tuning method using the fine-tuning dataset to obtain a new representation mapper and a new large language model base.

7. A service quality monitoring device based on multimodal model sentiment analysis, characterized in that: include: A data acquisition module, used to collect text modality evaluation data and non-text modality evaluation data of the service to be monitored; A feature processing module, used to perform feature extraction and mapping on the non-text modality evaluation data to determine a target non-text modality evaluation representation; A prompt word construction module, used to combine the text modality evaluation data with a predefined prompt word template to generate a text evaluation prompt word; A sentiment analysis module, used for inputting the text evaluation prompt and the target non-text modal evaluation representation into a preset large language model base for sentiment analysis, and outputting a sentiment analysis result; The service quality monitoring module is used to calculate the service quality monitoring score according to the sentiment analysis result and the text evaluation prompt, and decode the sentiment analysis result to obtain the service quality monitoring result explanation.

8. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the service quality monitoring method based on multimodal model sentiment analysis as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by the processor, the steps of the service quality monitoring method based on multimodal model sentiment analysis as described in any one of claims 1-6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, the steps of the service quality monitoring method based on multimodal model sentiment analysis as described in any one of claims 1-6 are implemented.

Images