A method and system for evaluating service quality in public places based on multimodal fusion

Abstract

The present invention discloses a method and system for evaluating the quality of service in public places based on multimodal fusion, and relates to the technical field of multimodal fusion. The method for evaluating the quality of service includes obtaining sound information in a service window environment and voice information of a customer and a service personnel; processing the sound information to obtain sound events in the sound information; processing the voice information to obtain words related to the quality of service in the voice information and words related to the quality of service in text information; the text information is information obtained after the voice information is converted into text; the sound events in the sound information, the words related to the quality of service in the processed voice information, and the words related to the quality of service in the text information are integrated to obtain a service quality grade. The present invention integrates the sound information, voice information, and text information to evaluate the service quality, thereby achieving the comprehensiveness and objectivity of the service quality evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal fusion, and in particular to a method and system for evaluating the quality of public service based on multimodal fusion. Background Art

[0002] Various public service places, such as banks, hotels, hospitals and government agencies, have set up a large number of manual windows to improve customer service satisfaction. At present, the service quality evaluation of manual windows is mostly in the form of manual scoring by customers, which has problems such as incomplete coverage and poor objectivity. Therefore, it is necessary to establish a comprehensive and objective service quality evaluation method.

[0003] How to establish the above means is quite difficult. On the one hand, the evaluation of service quality needs to be considered comprehensively from two dimensions: customers and windows. A single dimension cannot inevitably extract the real situation and make an accurate evaluation. On the other hand, the evaluation of service quality requires in-depth excavation of some deep information, the integration of the background of the situation and various factors, and the joint analysis of audio and video.

[0004] Driven by this demand and technology, multimodal and fusion technologies have emerged and developed rapidly. However, there are few applications and developments in the existing technology. Moreover, it should be noted that although the video mode can provide relatively rich information, its information effectiveness is low for applications such as customer-window communication, which is mainly based on voice communication.

[0005] Chinese patent [CN111754980A] proposes an intelligent scoring method based on semantic recognition. Voice information is recognized and converted into text form to obtain voice text; word vectors are trained to represent voice text in the form of data matrix; semantic features of voice text are extracted using convolutional neural network; and service level is scored based on the truncated and padded voice text and semantic features. The method proposed in this patent uses the conversation between maintenance personnel and users as the research target, and in most cases only voice events exist. In fact, if there is a service window environment in a public place, there may be event information that affects the service. If the event information is not included in the evaluation, the evaluation may not be objective enough. In addition, in the service window environment in a public place, there is usually glass with strong sound insulation, and there may be noise from many other events. It is necessary to consider noise reduction and directional collection of voice information from users and service personnel.

[0006] Chinese patent [CN111311327A] proposes a service evaluation method based on artificial intelligence. Acquire the voice stream to be recognized collected in real time during the service process; perform identity recognition on the voice stream to be recognized, and determine the target identity information corresponding to the voice stream to be recognized; perform text analysis on the voice stream to be recognized, and obtain the text analysis result corresponding to the voice stream to be recognized; perform sentiment analysis on the voice stream to be recognized, and obtain the sentiment analysis result corresponding to the voice stream to be recognized; fuse the text analysis result and the sentiment analysis result corresponding to the voice stream to be recognized, and obtain the service quality score corresponding to the target identity information. The method proposed in this patent only takes the voice of customers and service personnel as the analysis objects. It uses the technology of sentiment analysis, but only uses the single voice mode for analysis, which may cause the problem of incomplete analysis. At the same time, the invention does not consider the impact of sound events on the evaluation, and the service evaluation may not be objective enough.

[0007] Therefore, the existing technology only uses a single mode to evaluate service quality, which has the problem of insufficient evaluation and the lack of inclusion of sound event information in the evaluation, which may result in the evaluation being not objective. The present application provides a method and system for evaluating the service quality of public places based on multimodal fusion. Summary of the invention

[0008] The purpose of the present invention is to provide a method and system for evaluating the quality of service in public places based on multimodal fusion, which can integrate the information of sound, voice and text to evaluate the service quality, thereby achieving the comprehensiveness and objectivity of the service quality evaluation.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] In a first aspect, the present invention provides a method for evaluating the quality of service in public places based on multimodal fusion, the method for evaluating the quality of service in public places being applied to various service window environments in which sound information and voice information exist, and the service quality level is obtained by analyzing the sound information and voice information; the method for evaluating the quality of service comprises:

[0011] Acquiring sound information in the service window environment, and acquiring voice information of the communication between the customer and the service personnel; the service personnel are the personnel located at the service window;

[0012] Processing the sound information to obtain the sound events present in the sound information;

[0013] Processing the voice information to obtain processed voice information;

[0014] Extract keywords from the processed voice information to obtain words related to the service quality in the processed voice information;

[0015] Converting the processed voice information into text to obtain text information, and extracting keywords from the text information to obtain words related to the quality of service in the text information;

[0016] The sound events in the sound information, the words related to the service quality in the processed voice information and the words related to the service quality in the text information are integrated to obtain the service quality level.

[0017] Optionally, processing the sound information to obtain a sound event in the sound information specifically includes:

[0018] Performing pre-emphasis, framing and windowing processing on the sound information to obtain processed sound information; the processed sound information includes multiple frames of processed sound data;

[0019] Perform fast Fourier transform on each frame of processed sound data to obtain sound information after fast Fourier transform;

[0020] Inputting the sound information after the fast Fourier transform into a Mel filter to obtain Mel cepstral coefficient features;

[0021] The Mel cepstral coefficient feature is input into the FDY-SK model to obtain an output result; the FDY-SK model includes a selective kernel convolution network, a frequency dynamic convolution network, a Transformer model and a linear layer; the selective kernel convolution network is used to input the Mel cepstral coefficient, the frequency dynamic convolution network is used to input the Mel cepstral coefficient, the selective kernel convolution network and the frequency dynamic convolution network are both used to input the output data into the Transformer model, the Transformer model is used to input the output data into the linear layer, and the linear layer is used to output the result;

[0022] When there is a sound event in the output result, it is determined that there is a sound event in the sound information.

[0023] Optionally, processing the voice information to obtain processed voice information specifically includes:

[0024] Processing the voice information using voice enhancement and dynamic voice detection technology to obtain first voice information;

[0025] Processing the first voice information using an independent vector analysis algorithm and an echo cancellation technique to obtain second voice information;

[0026] The Pyannote.audio tool is used to obtain the start and end time of the second voice information, and audio segmentation processing is performed on the second voice information according to the start and end time of the second voice information to obtain a plurality of segmented voice activity segments.

[0027] Optionally, keyword extraction is performed on the processed voice information, and the words related to the service quality in the processed voice information specifically include:

[0028] Inputting a plurality of the segmented speech activity segments into the trained Transformer model for negative vocabulary detection, and obtaining negative vocabulary segments in the speech activity segments.

[0029] Optionally, the processed voice information is converted into text to obtain text information, and keywords are extracted from the text information to obtain words related to the quality of service in the text information, specifically including:

[0030] The plurality of segmented speech activity segments are respectively sent to the trained Squeeze-ShareNet network to perform speech-to-text tasks to obtain text information; the Squeeze-ShareNet network includes a shared encoder part, a CTC decoder part and an Attention decoder part; the shared encoder part is composed of a Squeezeformer encoder for inputting the segmented speech activity segments, the shared encoder is used to input the output data into the CTC decoder and the Attention decoder, the CTC decoder is used to input the output data into the Attention decoder for re-scoring, and the Attention decoder is used to output the result;

[0031] Use jieba word segmentation tool to process text information and obtain text vocabulary;

[0032] According to the service types of different places, a benign word library, a negative word library and a neutral word library are established. The benign word library is composed of benign words, including corresponding service professional words and polite words; the negative word library is composed of negative words, including uncivilized words; the neutral word library is composed of neutral words, except for benign words and negative words, all other words are neutral words;

[0033] According to the established benign word library, negative word library and neutral word library, benign words, neutral words and negative words in the text vocabulary are determined.

[0034] Optionally, the sound events in the sound information, the words related to the service quality in the processed voice information, and the words related to the service quality in the text information are integrated to obtain the service quality level, which specifically includes:

[0035] Calculating a speech information score according to the proportion of the negative vocabulary segments to the overall speech activity segments;

[0036] Calculating a text information score based on the sum of all benign word weights, the sum of all neutral word weights, and the sum of all negative word weights in the text vocabulary;

[0037] Combining the voice information score and the text information score, the voice and text weighted scores are calculated;

[0038] Calculate the impact factor of sound events;

[0039] The impact factor of the sound event is integrated into the voice and text weighted scores, the final score is calculated, and the corresponding service quality level is obtained based on the final score.

[0040] Optionally, calculating the text information score according to the sum of all benign word weights, the sum of all neutral word weights, and the sum of all negative word weights in the text vocabulary specifically includes:

[0041] The weight of any word in the text vocabulary is calculated according to the TF-IDF method, and normalized to obtain the normalized value of the sum of the weights of all benign words, the normalized value of the sum of the weights of all neutral words, and the normalized value of the sum of the weights of all negative words;

[0042] According to the formula S T =δ positive +δ neutral -δ negative Calculate the text information score;

[0043] Among them, S T is the text information score, δ positive is the normalized value of the sum of all good word weights, δ neutral is the normalized value of the sum of all neutral word weights, δ negative is the normalized value of the sum of all negative word weights.

[0044] Optionally, the voice information score and the text information score are combined to calculate the voice and text weighted scores, specifically including:

[0045] The voice and text weighted scores are calculated by weighting the voice information score, the text information score, the historical upper and lower limits of the score interval where the voice information score is located, and the historical upper and lower limits of the score interval where the text information score is located; the historical upper and lower limits of the score interval where the voice information score is located are obtained based on the historical voice information score statistics; the historical upper and lower limits of the score interval where the text information score is located are obtained based on the historical text information score statistics; wherein, the calculation formula for the voice and text weighted scores is:

[0046]

[0047] Among them, S S+T is the weighted score of speech and text obtained, S S is the speech information score, S T is the text information score. If the voice information score is higher than the text information score, A is the historical upper limit score of the voice score interval, and B is the historical lower limit score of the text score interval; if the voice information score is lower than the text information score, A is the historical upper limit score of the text score interval, and B is the historical lower limit score of the voice score interval.

[0048] Optionally, the impact factor of the sound event is integrated into the weighted scores of speech and text, and a final score is calculated, and the corresponding service quality level is obtained according to the final score, specifically including:

[0049] According to the formula S Final =F×(S S+T ) Calculate the final score after the fusion of sound, speech and text;

[0050] Among them, S Final is the final score, F is the impact factor of the sound event, S S+T Weighted scores for speech and text obtained;

[0051] The corresponding service quality level is obtained according to the final score and the divided score interval, and the divided score interval is (0.8, 1] for excellent, (0.6, 0.8] for good, (0.4, 0.8] for average, (0.2, 0.4] for poor, and (0, 0.2] for poor.

[0052] In a second aspect, the present invention provides a public place service quality evaluation system based on multimodal fusion, the service quality evaluation system comprising:

[0053] Audio collection module, used to obtain sound information in the service window environment, as well as voice information of the communication between customers and service personnel;

[0054] A sound event detection module, used for processing the sound information to obtain the sound events present in the sound information;

[0055] A voice keyword extraction module is used to process the voice information to obtain processed voice information; extract keywords from the processed voice information to obtain words related to service quality in the processed voice information;

[0056] The speech-to-text and text keyword extraction module is used to convert the processed speech information into text to obtain text information, and to extract keywords from the text information to obtain words related to the service quality in the text information;

[0057] The multimodal fusion and normalization module is used to fuse the sound events in the sound information, the words related to the service quality in the processed voice information and the words related to the service quality in the text information to obtain the service quality level.

[0058] According to the specific embodiment provided by the present invention, the present invention discloses the following technical effects: on the basis of the existing service evaluation method, the present invention takes into account the sound information in the environment that may directly or indirectly affect the window service, and incorporates the sound information into the evaluation index, so that the evaluation of the service is more objective. At the same time, the voice information is converted into text information while removing other noises in the environment, and the sound information, voice information and text information are integrated to evaluate the service quality, achieving a comprehensive evaluation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0060] Figure 1 It is a flowchart of a method for evaluating the quality of service in public places based on multimodal fusion in an embodiment of the present invention;

[0061] Figure 2 It is a flowchart of a method for evaluating the quality of service in public places based on multimodal fusion in an embodiment;

[0062] Figure 3 Schematic diagram of the FDY-SK model structure in an embodiment of the present invention;

[0063] Figure 4 Schematic diagram of the FDY-SK training structure in an embodiment of the present invention;

[0064] Figure 5 Schematic diagram of the Squeeze-ShareNet network structure in an embodiment of the present invention;

[0065] Figure 6 Schematic diagram of the Squeeze-ShareNet training structure in an embodiment of the present invention;

[0066] Figure 7It is a schematic diagram of the composition of a public place service quality evaluation system based on multimodal fusion in an embodiment of the present invention.

[0067] Figure 8 It is a schematic diagram of the structure of a public place service quality evaluation system based on multimodal fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The purpose of the present invention is to provide a method and system for evaluating the quality of service in public places based on multimodal fusion, which evaluates the service quality by integrating the information of sound, voice and text, thereby achieving the comprehensiveness and objectivity of the service quality evaluation.

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] Embodiment 1

[0072] The present invention provides a method for evaluating the quality of service in public places based on multimodal fusion. The method is applied to various service window environments in public places. The service window environment in public places has sound information and voice information. The service quality level is obtained by analyzing the sound information and voice information. Figure 1 and Figure 2 As shown, the service quality evaluation method includes:

[0073] Step 101: Acquire sound information in a service window environment, and acquire voice information of communication between a customer and a service staff; the service staff is a person located at the service window.

[0074] Step 102: Process the sound information to obtain the sound events in the sound information.

[0075] Step 103: Process the voice information to obtain processed voice information.

[0076] Step 104: extract keywords from the processed voice information to obtain words related to the quality of service in the processed voice information.

[0077] Step 105: converting the processed voice information into text to obtain text information, and extracting keywords from the text information to obtain words related to the quality of service in the text information.

[0078] Step 106: The sound events in the sound information, the words related to the service quality in the processed voice information, and the words related to the service quality in the text information are integrated to obtain the service quality level.

[0079] As a further supplement to the above embodiment, the embodiment of the present invention also provides a more detailed public place service quality evaluation method based on multimodal fusion.

[0080] In some embodiments, step 102 specifically includes:

[0081] Step 1021: perform pre-emphasis, framing and windowing on the sound information to obtain processed sound information; the processed sound information includes sound data after multiple frames of processing; the pre-emphasis is to amplify the high-frequency part of the sound signal; the framing is to intercept the audio signal with a fixed length, each section is a frame, and a frame is taken as 25ms here; the windowing is used to enhance the first bit continuity between frames, wherein the window is a Hamming window.

[0082] Step 1022: Perform fast Fourier transform on each frame of processed sound data to obtain sound information after fast Fourier transform.

[0083] Step 1023: Input the sound information after the fast Fourier transform into a Mel filter, obtain a logarithmic Mel spectrum through a logarithmic operation, and then obtain Mel cepstral coefficient features after a discrete cosine transform; the sound information after the fast Fourier transform is a time-frequency feature.

[0084] Step 1024: Input the Mel cepstral coefficient feature into the FDY-SK model to obtain an output result; the FDY-SK model structure is as follows: Figure 3 As shown, the FDY-SK model includes a selective kernel convolution network, a frequency dynamic convolution network, a Transformer model and a linear layer; the selective kernel convolution network is used to input Mel cepstral coefficients, the frequency dynamic convolution network is used to input Mel cepstral coefficients, the selective kernel convolution network and the frequency dynamic convolution network are both used to input output data into the Transformer model, the Transformer model is used to input output data into the linear layer, and the linear layer is used to output the result; and the FDY-SK model is trained by adopting the Mean-Teacher semi-supervised consistency learning method, as shown in Figure 4 shown.

[0085] Step 1025: When there is a sound event in the output result, determine that there is a sound event in the sound information.

[0086] In some embodiments, step 103 specifically includes:

[0087] Step 1031: Use speech enhancement and dynamic speech detection technology to process the speech information to obtain first speech information.

[0088] Step 1032: Use an independent vector analysis algorithm and echo cancellation technology to process the first voice information to obtain second voice information.

[0089] Step 1033: Use the Pyannote.audio tool to obtain the start and end time of the second voice information, and perform audio segmentation processing on the second voice information according to the start and end time of the second voice information to obtain a plurality of segmented voice activity segments.

[0090] In some embodiments, step 104 specifically includes:

[0091] Step 1041: inputting a plurality of the segmented speech activity segments into the trained Transformer model for negative vocabulary detection to obtain negative vocabulary segments in the speech activity segments.

[0092] In some embodiments, step 105 specifically includes:

[0093] Step 1051: Send the plurality of segmented speech activity segments to the trained Squeeze-ShareNet network to perform speech-to-text tasks, and obtain text information; the Squeeze-ShareNet network structure for speech-to-text tasks is as follows: Figure 5As shown, several of the segmented speech activity segments are respectively sent to the trained Squeeze-ShareNet network to perform speech-to-text tasks to obtain text information; the Squeeze-ShareNet network includes a shared encoder part, a CTC decoder part and an Attention decoder part; the shared encoder part is composed of a Squeezeformer encoder for inputting the segmented speech activity segments, the shared encoder is used to input the output data into the CTC decoder and the Attention decoder, the CTC decoder is used to input the output data into the Attention decoder for re-scoring, and the Attention decoder is used to output the result; that is, the Squeeze-ShareNet network is obtained by changing the shared encoder in the WeNet network to a Squeezeformer encoder; the Squeeze-ShareNet training structure is as shown Figure 6 shown.

[0094] Step 1052: Use the Jieba word segmentation tool to process the text information to obtain text vocabulary.

[0095] Step 1053: establishing a benign word library, a negative word library and a neutral word library according to different types of services in different places, wherein the benign word library is composed of benign words, including corresponding service professional words and polite words; the negative word library is composed of negative words, including uncivilized words; the neutral word library is composed of neutral words, and all words except benign words and negative words are neutral words;

[0096] Step 1054: Determine the benign words, neutral words and negative words in the text vocabulary according to the established benign word library, negative word library and neutral word library.

[0097] In some embodiments, step 106 specifically includes:

[0098] Step 1061: Calculate the voice information score according to the proportion of the negative vocabulary segments to the overall voice activity segments; the proportion of the negative vocabulary segments to the overall voice activity segments is regarded as a deduction item, and the proportion of the benign and neutral vocabulary segments to the overall voice activity segments is the obtained score; the voice information score takes a value range of [0, 1].

[0099] Step 1062: Calculate the text information score according to the sum of all benign word weights, the sum of all neutral word weights and the sum of all negative word weights in the text vocabulary.

[0100] The weight of any word in the text vocabulary is calculated according to the TF-IDF method to obtain the sum of all benign word weights, the sum of all neutral word weights and the sum of all negative word weights; the process of calculating the weight of any word in the text vocabulary according to the TF-IDF method is:

[0101] According to the formula TF-IDF word-c =TF word-c ×IDF word-c Calculate the weight of a word in the text vocabulary.

[0102] Among them, word-c represents a certain word, TF-IDF word-c is the keyword weight of word-c. word-c ) = the number of times a word appears in the article / the total number of words in the article, inverse document frequency (IDF word-c ) = log (total number of documents in the corpus / (number of documents containing the word + 1)).

[0103] According to the formula Normalize.

[0104] Among them, δ word-c Represents the normalized value of word-c weight, is the keyword weight of all words;

[0105] According to the formula S T =δ positive +δ neutral -δ negative Calculate the text information score.

[0106] Among them, S T is the text information score, δ positive is the normalized value of the sum of all good word weights, δ neutral is the normalized value of the sum of all neutral word weights, δ negative is the normalized value of the sum of all negative word weights.

[0107] Step 1063: Combine the voice information score and the text information score to calculate the voice and text weighted scores.

[0108] The voice and text weighted scores are calculated by weighting the voice information score, the text information score, the historical upper and lower limits of the score interval where the voice information score is located, and the historical upper and lower limits of the score interval where the text information score is located; the score intervals are (0.8, 1], (0.6, 0.8], (0.4, 0.8], (0.2, 0.4], (0, 0.2]; the historical upper and lower limits of the score interval where the voice information score is located are obtained based on the historical voice information score statistics; the historical upper and lower limits of the score interval where the text information score is located are obtained based on the historical text information score statistics; wherein, the calculation formula for the voice and text weighted scores is:

[0109]

[0110] Among them, S S+T is the weighted score of speech and text obtained, S S is the speech information score, S T is the text information score. If the voice information score is higher than the text information score, A is the historical upper limit score of the voice score interval, and B is the historical lower limit score of the text score interval; if the voice information score is lower than the text information score, A is the historical upper limit score of the text score interval, and B is the historical lower limit score of the voice score interval.

[0111] Step 1064: Calculate the impact factor of the sound event; the impact factor of the sound event is obtained by determining whether the sound event is a malicious event, for example, quarrels, fights, explosions and other events are regarded as malicious events that will affect the service; the value range of the impact factor of the sound event is [0, 1], and the impact factor will reduce the weighted scores of voice and text, thereby reflecting the impact of the sound event on the service quality.

[0112] Step 1065: Integrate the impact factor of the sound event into the voice and text weighted scores, calculate the final score, and obtain the corresponding service quality level based on the final score.

[0113] According to the formula S Final =F×(S S+T ) calculates the final score after the fusion of sound, speech and text.

[0114] Among them, S Final is the final score, F is the impact factor of the sound event, S S+T The weighted scores of the speech and text obtained.

[0115] The corresponding service quality level is obtained according to the final score and the divided score interval, and the divided score interval is (0.8, 1] for excellent, (0.6, 0.8] for good, (0.4, 0.8] for average, (0.2, 0.4] for poor, and (0, 0.2] for poor.

[0116] Embodiment 2:

[0117] The present invention provides a public place service quality evaluation system based on multi-modal fusion, such as Figure 7 and Figure 8 As shown, the service quality evaluation system includes:

[0118] 201: An audio acquisition module, comprising a multi-microphone array deployed in a service window environment in a public place and a cardioid noise reduction microphone deployed respectively at the customer and service staff windows; the multi-microphone array is used to obtain sound information in the service window environment; the cardioid noise reduction microphone is used to obtain voice information of the communication between the customer and the service staff.

[0119] 202: A sound event detection module is used to process the sound information to obtain the sound events existing in the sound information.

[0120] 203: A voice keyword extraction module is used to process the voice information to obtain processed voice information; extract keywords from the processed voice information to obtain words related to service quality in the processed voice information.

[0121] 204: A voice-to-text and text keyword extraction module is used to convert the processed voice information into text to obtain text information, and to extract keywords from the text information to obtain words related to the quality of service in the text information.

[0122] 205: A multimodal fusion and normalization module, used to fuse the sound events in the sound information, the words related to the service quality in the processed voice information, and the words related to the service quality in the text information to obtain a service quality level.

[0123] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0124] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for evaluating the quality of public service based on multimodal fusion, characterized in that: The service quality evaluation method is applied to various public service window environments, where there is sound information and voice information in the public service window environment, and the service quality level is obtained by analyzing the sound information and voice information; The service quality evaluation method comprises: Acquiring sound information in the service window environment, and acquiring voice information of the communication between the customer and the service personnel; the service personnel are the personnel located at the service window; Processing the sound information to obtain the sound events present in the sound information; Processing the voice information to obtain processed voice information; Extract keywords from the processed voice information to obtain words related to the service quality in the processed voice information; Converting the processed voice information into text to obtain text information, and extracting keywords from the text information to obtain words related to the quality of service in the text information; The sound events in the sound information, the words related to the service quality in the processed voice information and the words related to the service quality in the text information are integrated to obtain the service quality level; Processing the sound information to obtain the sound events present in the sound information specifically includes: Performing pre-emphasis, framing and windowing processing on the sound information to obtain processed sound information; the processed sound information includes multiple frames of processed sound data; Perform fast Fourier transform on each frame of processed sound data to obtain sound information after fast Fourier transform; Inputting the sound information after the fast Fourier transform into a Mel filter to obtain Mel cepstral coefficient features; The Mel cepstral coefficient feature is input into the FDY-SK model to obtain an output result; the FDY-SK model includes a selective kernel convolution network, a frequency dynamic convolution network, a Transformer model and a linear layer; the selective kernel convolution network is used to input the Mel cepstral coefficient, the frequency dynamic convolution network is used to input the Mel cepstral coefficient, the selective kernel convolution network and the frequency dynamic convolution network are both used to input the output data into the Transformer model, the Transformer model is used to input the output data into the linear layer, and the linear layer is used to output the result; When there is a sound event in the output result, it is determined that there is a sound event in the sound information.

2. According to claim 1, a method for evaluating the quality of public service based on multimodal fusion is characterized in that: Processing the voice information to obtain processed voice information specifically includes: Processing the voice information using voice enhancement and dynamic voice detection technology to obtain first voice information; Processing the first voice information using an independent vector analysis algorithm and an echo cancellation technique to obtain second voice information; The Pyannote.audio tool is used to obtain the start and end time of the second voice information, and audio segmentation processing is performed on the second voice information according to the start and end time of the second voice information to obtain a plurality of segmented voice activity segments.

3. According to claim 2, a method for evaluating the quality of public service based on multimodal fusion is characterized in that: Keyword extraction is performed on the processed voice information, and the words related to service quality in the processed voice information include: Inputting a plurality of the segmented speech activity segments into the trained Transformer model for negative vocabulary detection, and obtaining negative vocabulary segments in the speech activity segments.

4. The method for evaluating the quality of public service based on multimodal fusion according to claim 3 is characterized in that: The processed voice information is converted into text to obtain text information, and keywords are extracted from the text information to obtain words related to the quality of service in the text information, including: The plurality of segmented speech activity segments are respectively sent to the trained Squeeze-ShareNet network to perform speech-to-text tasks to obtain text information; the Squeeze-ShareNet network includes a shared encoder part, a CTC decoder part and an Attention decoder part; the shared encoder part is composed of a Squeezeformer encoder for inputting the segmented speech activity segments, the shared encoder is used to input the output data into the CTC decoder and the Attention decoder, the CTC decoder is used to input the output data into the Attention decoder for re-scoring, and the Attention decoder is used to output the result; Use jieba word segmentation tool to process text information and obtain text vocabulary; According to the service types of different places, a benign word library, a negative word library and a neutral word library are established. The benign word library is composed of benign words, including corresponding service professional words and polite words; the negative word library is composed of negative words, including uncivilized words; the neutral word library is composed of neutral words, except for benign words and negative words, all other words are neutral words; According to the established benign word library, negative word library and neutral word library, benign words, neutral words and negative words in the text vocabulary are determined.

5. The method for evaluating the quality of public service based on multimodal fusion according to claim 4 is characterized in that: The sound events in the sound information, the words related to the service quality in the processed voice information and the words related to the service quality in the text information are integrated to obtain the service quality level, which specifically includes: Calculating a speech information score according to the proportion of the negative vocabulary segments to the overall speech activity segments; Calculating a text information score based on the sum of all benign word weights, the sum of all neutral word weights, and the sum of all negative word weights in the text vocabulary; Combining the voice information score and the text information score, the voice and text weighted scores are calculated; Calculate the impact factor of sound events; The impact factor of the sound event is integrated into the voice and text weighted scores, the final score is calculated, and the corresponding service quality level is obtained based on the final score.

6. The method for evaluating the quality of public service based on multimodal fusion according to claim 5 is characterized in that: Calculating the text information score according to the sum of all benign word weights, the sum of all neutral word weights, and the sum of all negative word weights in the text vocabulary specifically includes: The weight of any word in the text vocabulary is calculated according to the TF-IDF method, and normalized to obtain the normalized value of the sum of the weights of all benign words, the normalized value of the sum of the weights of all neutral words, and the normalized value of the sum of the weights of all negative words; According to the formula S T =δ positive +δ neutral -δ negative Calculate the text information score; Among them, S T is the text information score, δ positive is the normalized value of the sum of all good word weights, δ neutral is the normalized value of the sum of all neutral word weights, δ negative is the normalized value of the sum of all negative word weights.

7. The method for evaluating the quality of public service based on multimodal fusion according to claim 5 is characterized in that: Combining the voice information score and the text information score, the voice and text weighted scores are calculated, including: The voice and text weighted scores are calculated by weighting the voice information score, the text information score, the historical upper and lower limits of the score interval where the voice information score is located, and the historical upper and lower limits of the score interval where the text information score is located; the historical upper and lower limits of the score interval where the voice information score is located are obtained based on the historical voice information score statistics; the historical upper and lower limits of the score interval where the text information score is located are obtained based on the historical text information score statistics; wherein, the calculation formula for the voice and text weighted scores is: Among them, S S+T is the weighted score of speech and text obtained, S S is the speech information score, S T is the text information score. If the voice information score is higher than the text information score, A is the historical upper limit score of the voice score interval, and B is the historical lower limit score of the text score interval; if the voice information score is lower than the text information score, A is the historical upper limit score of the text score interval, and B is the historical lower limit score of the voice score interval.

8. A method for evaluating the quality of public service based on multimodal fusion according to claim 5 or 7, characterized in that: The impact factor of the sound event is integrated into the weighted scores of speech and text, and the final score is calculated. The corresponding service quality level is obtained according to the final score, including: According to the formula S Final =F×(S S+T ) Calculate the final score after the fusion of sound, speech and text; Among them, S Final is the final score, F is the impact factor of the sound event, S S+T Weighted scores for speech and text obtained; The corresponding service quality level is obtained according to the final score and the divided score interval, and the divided score interval is (0.8, 1] for excellent, (0.6, 0.8] for good, (0.4, 0.8] for average, (0.2, 0.4] for poor, and (0, 0.2] for poor.

9. A public place service quality evaluation system based on multimodal fusion, characterized in that: The method for evaluating the quality of service in public places based on multimodal fusion according to any one of claims 1 to 8 is applied, and the service quality evaluation system comprises: Audio collection module, used to obtain sound information in the service window environment, as well as voice information of the communication between customers and service personnel; A sound event detection module, used for processing the sound information to obtain the sound events present in the sound information; A voice keyword extraction module is used to process the voice information to obtain processed voice information; extract keywords from the processed voice information to obtain words related to service quality in the processed voice information; The speech-to-text and text keyword extraction module is used to convert the processed speech information into text to obtain text information, and to extract keywords from the text information to obtain words related to the service quality in the text information; The multimodal fusion and normalization module is used to fuse the sound events in the sound information, the words related to the service quality in the processed voice information and the words related to the service quality in the text information to obtain the service quality level.

Images