Intelligent customer service recommendation method fusing multi-modal information and application equipment thereof

Abstract

The invention relates to the technical field of language processing, in particular to an intelligent customer service recommendation method fusing multi-modal information and application equipment thereof, and the method comprises the following steps: S1, user question recognition; s2, character recognition processing; s3, carrying out image recognition processing; s4, performing voice recognition processing; s5, user questions are integrated; s6, user preference analysis; s7, analyzing user interaction records; according to the method, character recognition processing, image recognition processing and voice recognition processing are carried out according to the user question-asking modes, then user question-asking associated questions are obtained through analysis, and the recommended reply modes are obtained through analysis according to the preference degree of the user for the reply modes and the degree of interaction understanding difficulty. Through multiple identification processing modes, questions of different forms of the user can be more comprehensively captured, accurate mastering of the intention of the user is ensured, meanwhile, the preference of the user for different reply modes is considered, and habits and preferences of different users can be better adapted.

Description

Technical Field

[0001] The present invention relates to the field of language processing technology, and in particular to an intelligent customer service recommendation method integrating multimodal information and an application device thereof. Background Art

[0002] In today's digital age, customer service faces increasingly higher requirements and challenges. With the continuous advancement of artificial intelligence technology, intelligent customer service has emerged and gradually become an important part of customer service. However, relying solely on single-modal information, such as text, is often difficult to meet complex and diverse user needs.

[0003] In many cases, users may prefer to express problems or needs through voice. Voice information can convey meaning more directly and quickly, especially for some users who are not good at text input or are in specific scenarios. At the same time, image information also has unique value. For example, users can send product pictures to more accurately describe problems or failures. On the other hand, information in different modes has its own characteristics and advantages. Text information has high accuracy and standardization, but there may be incomplete expressions. Voice information is more emotional and real-time. Image information can provide intuitive visual clues. The integration of these multimodal information can complement each other and bring out the best in each other, so as to understand users' intentions and needs more comprehensively and deeply.

[0004] The patent name is a technical solution published in a Chinese patent (patent number is 202210805914.X). The solution uses SMS information to be distributed by customer service to selected clients, selects interested customers and feeds them back to customer service for further communication, and uses the emotion detection unit to timely observe the customer's mood prompts after receiving the message. After the artificial intelligence recommendation, the customer can freely choose the voice package and "package" recommendation module. When the customer chooses, big data is used to instantly recommend the content required by the customer, cater to their preferences, and make the customer's choice more humane. However, there are still some shortcomings, which are specifically manifested in the following aspects: 1. The solution may have the problem of insufficient information richness, and does not make full use of multimodal information such as images and audio to achieve a more comprehensive customer understanding. In some cases, it may cause misunderstandings among users, which makes the overall service quality and efficiency subject to a certain degree of limitation.

[0005] Second, although this solution can instantly recommend the content that customers need, it fails to fully consider the difficulty of customers in understanding the recommended content. This may cause customers to encounter obstacles in receiving and understanding these recommended contents, affecting the customer experience. Summary of the invention

[0006] In order to overcome the shortcomings of the background technology, the embodiments of the present invention provide an intelligent customer service recommendation method and an application device that integrates multimodal information, which can effectively solve the problems involved in the above-mentioned background technology.

[0007] The purpose of the present invention can be achieved through the following technical solutions: an intelligent customer service recommendation method integrating multimodal information, the method comprising the following steps: S1. User question recognition: reading the user's input content and identifying the user's question method, and identifying and processing different question methods respectively, and the question methods include text, images, and voice.

[0008] S2. Text recognition processing: Perform text processing on the text input by the user to obtain the key words of the user text, and then match them with the stored existing questions one by one to obtain the questions related to the user's text questions.

[0009] S3. Image recognition processing: read the image input by the user. If there is text information in the user image, obtain the user image question-related questions by performing text recognition processing on the user image. If there is no text information in the user image, perform image recognition processing on the user image, and obtain the user image question-related questions by extracting feature data for matching.

[0010] S4. Voice recognition processing: Perform text recognition on the voice sent by the user to obtain the user's voice correction text, and then analyze the user's voice questions to obtain related questions.

[0011] S5. User question integration: obtain the user's text-question related questions, the user's image-question related questions, and the user's voice-question related questions, and integrate them to obtain the corresponding related questions of the user's questions.

[0012] S6. User preference analysis: Based on the user's historical behavior, the number of voice replies, text replies, and voice replies of the intelligent customer service are recorded, as well as the scores of each text reply, image reply, and voice reply to obtain the user's preference for text reply, image reply, and voice reply.

[0013] S7. Analysis of user interaction records: Obtain the user's interaction record data and analyze the difficulty of understanding the text replies, image replies, and voice replies. The interaction record data includes the number of times the user repeats each text reply, image reply, and voice reply, and the average reply time of each text reply, image reply, and voice reply.

[0014] S8. Recommended answer analysis: Based on the user's preference for text answers, image answers, and voice answers and the difficulty level of interactive understanding, the recommended answer method is obtained, and the corresponding related questions asked by the user are answered accordingly.

[0015] Preferably, the specific analysis method of the text recognition processing is as follows: the first step is to read the text input by the user, remove punctuation marks and non-alphanumeric characters in the text by using a text preprocessing technique, record the remaining text content as the user text, divide the user text into words by using a natural language processing algorithm, record them as each word of the user text, read a stop word thesaurus and a similar word thesaurus from a management database, perform stop word filtering on the user text, and obtain each word of the user text and each similar word by comparing and screening each word of the user text with the similar word thesaurus, filter them out from the user text, record the remaining words of the user text after screening as each key word of the user text, and count the number of screened similar words corresponding to each key word of the user text, record them as γ m , m represents the number of the mth keyword, m=1,2,...,q, through the formula Get the weight of each keyword in the user text δ m , q represents the number of key words.

[0016] The second step is to extract the existing questions stored in the management database, match the key words in the user text with the existing questions one by one, and filter out the number of overlaps between the key words in the existing questions and the user text, recorded as N. f , f represents the number of the f-th existing question, f = 1, 2, ..., k, and the proportion of the key words of each user text that overlaps with each existing question is extracted from the proportion of each keyword in the user text, denoted as ζ fa , a represents the number of the key words of the a-th user text, a=1,2,...,b, and b≤q, substitute it into the formula Get the similarity between each existing question and the user text ξ f , φ1 and φ2 represent the weight factors of the set number of keyword overlaps and the keyword proportion respectively, k represents the number of existing questions, and the similarity between each existing question and the user text is sorted in descending order, and the first existing question is recorded as the user text question related question.

[0017] Preferably, the specific analysis method for performing text recognition processing on user images is: reading the image input by the user, recorded as the user image, recognizing and extracting the text in the user image through optical character recognition technology, recorded as user image extracted text, and obtaining user image question-related questions according to the method of analyzing user text question-related questions.

[0018] Preferably, the specific analysis method for performing image recognition processing on the user image is: extracting characteristic data of the power grid from the user image, recorded as user question characteristic data, and extracting each related question corresponding to each characteristic data from the management database, comparing the user question characteristic data with each related question corresponding to each characteristic data, obtaining each related question corresponding to the user question characteristic data, recorded as each related question asked by the user, reading each historical related question of the power grid user for each question characteristic data from the management database, and extracting each historical related question of the power grid user for the user question characteristic data, comparing each related question asked by the user with each historical related question asked by the power grid user for the user question characteristic data, and obtaining the number of occurrences of each related question asked by the power grid user, recorded as i represents the number of the i-th related question, i = 1, 2, ..., n. At the same time, the historical questions asked by the user for each question feature data are read from the management database, and the corresponding related questions are compared with the related questions asked by the user. The number of occurrences of each related question asked by the user is counted and recorded as Substituting this into the formula Get the proportion of each related question of the user for the question feature data δ i , respectively represent the total number of questions asked by the set power grid users and the total number of questions asked by the users, μ1 and μ2 represent the weight factors of the set power grid users’ questions and user’s questions, and arrange the proportions of each associated question for the question feature data in order from large to small, and record the first associated question corresponding to the proportion of the associated question for the question feature data as the user image question associated question.

[0019] Preferably, the specific analysis method of the speech recognition processing is: reading the speech sent by the user, performing noise reduction processing on it and recording it as user speech, using speech recognition technology to convert the user speech into text, recording it as user speech recognition text, and reading the user's place of origin from the network background, correcting the user speech recognition text according to common words in the accent area corresponding to the user's place of origin to obtain user speech correction text, analyzing the user speech correction text according to the method of analyzing the similarity between each existing question and the user text, obtaining the similarity between each existing question and the user speech correction text, filtering out the maximum value from the similarity between each existing question and the user speech correction text, and recording the corresponding existing question as the user voice question associated question.

[0020] Preferably, the specific analysis method of the user preference analysis is: reading the user's historical behavior records from the management database, filtering and counting the number of text replies, image replies, and voice replies received by the user from the intelligent customer service, which are recorded as g, l, and b, and the scores of each text reply, image reply, and voice reply are recorded as j represents the number of the jth text reply, j = 1, 2, ..., g, p represents the number of the pth image reply, p = 1, 2, ..., l, a represents the number of the ath voice reply, a = 1, 2, ..., b, and substitute it into the formula Get the user's preference for text replies χ 文字 , η1 and η2 represent the set number of replies and the weight factor of the reply score, respectively. According to the method of analyzing the user's preference for text replies, the user's preference for image replies and the user's preference for voice replies are analyzed in turn, which are recorded as χ 图像 , χ 语音 .

[0021] Preferably, the specific analysis method of the user interaction record is: extract the number of repeated inquiries of the user when receiving each text reply, image reply, and voice reply from the intelligent customer service from the user's historical behavior record, and record it as the number of repeated inquiries of each text reply of the user The number of times the user replies to each image. The number of times the user repeated the question in each voice reply At the same time, the response time of each repeated question in each text response, image response, and voice response is recorded, and the average response time of each text response, image response, and voice response is obtained by taking the average value, which is recorded as

[0022] Preferably, the specific analysis method of the user interaction record analysis is: through the formula

[0023] Interactive understanding difficulty of getting textual responses ρ 文字 , κ1 and κ2 represent the weight factors of the set number of repeated inquiries and the length of response time, respectively. The interactive understanding difficulty of image responses and the interactive understanding difficulty of voice responses are analyzed according to the method of analyzing the interactive understanding difficulty of text responses, which are recorded as ρ 图像 , 语音 .

[0024] Preferably, the specific analysis method of the recommended reply analysis is: reading the user's preference for the text reply x 文字 , Difficulty of interactive understanding of text responses ρ 文字 , substituting it into the formula Recommended level for getting text replies w1 and w2 represent the weight factors of the preference degree of the set reply method and the difficulty of interactive understanding of the reply method respectively. e is a natural constant. The recommendation degree of image reply and the recommendation degree of voice reply are analyzed according to the method of analyzing the recommendation degree of text reply. The maximum value is screened out from the recommendation degree of text reply, image reply and voice reply, and the corresponding reply method is recorded as the recommended reply method.

[0025] A second aspect of the present invention provides an application device, which includes a processor, a memory, and a communication bus, wherein the memory stores a computer-readable program that can be executed by the processor.

[0026] The communication bus realizes the connection and communication between the processor and the memory.

[0027] When the processor executes the computer-readable program, the steps of the chromatography-mass spectrometry analysis method of the present invention are implemented.

[0028] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects: 1. The present invention performs text recognition processing, image recognition processing, and voice recognition processing according to the user's questioning method, and then analyzes and obtains the user's text question-related issues, the user's image question-related issues, and the user's voice question-related issues, thereby improving the adaptability to multiple user input methods, meeting the habits and needs of different users, and avoiding omissions or misunderstandings that may be caused by a single recognition method.

[0029] 2. The present invention obtains a recommended reply method based on the user's preference for text replies, image replies, and voice replies and the difficulty of interactive understanding of text replies, image replies, and voice replies, and uses this method to reply to user questions, thereby improving interaction efficiency, avoiding communication barriers and time waste caused by using reply methods that are not suitable for users, and at the same time better matching users' understanding abilities and preferences to improve the effectiveness of information communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0031] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0032] Figure 2 for Figure 1 Schematic diagram of the method flow of steps S6, S7 and S8.

[0033] Figure 3The figure is a schematic diagram of an embodiment of an application device in an example of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] See also Figure 1 As shown, the present invention provides an intelligent customer service recommendation method that integrates multimodal information, and the method includes the following steps: S1. User question recognition: reading the user's input content and identifying the user's question method, and identifying and processing different question methods respectively, and the question methods include text, image, and voice.

[0036] It should be noted that, in response to the user's image questions, the image uploaded by the user is read and recorded as the user image, and it is determined whether there is text information in the user image. If there is text information in the user image, the text in the user image is extracted, and then text recognition processing is performed on it. If there is no text information in the user image, image recognition processing is performed on the user image.

[0037] S2. Text recognition processing: Perform text processing on the text input by the user to obtain the key words of the user text, and then match them with the stored existing questions one by one to obtain the questions related to the user's text questions.

[0038] The specific analysis method of the text recognition processing is as follows: the first step is to read the text input by the user, remove punctuation marks and non-alphanumeric characters in the text through text preprocessing technology, record the remaining text content as user text, divide the user text into words through natural language processing algorithm, record them as each word of user text, read the stop word thesaurus and the similar word thesaurus from the management database, filter the user text for stop words, and obtain each word of the user text and each similar word by comparing and screening each word of the user text with the similar word thesaurus, filter them out from the user text, record the remaining words of the user text after screening as each key word of the user text, and count the number of similar words screened corresponding to each key word of the user text, record them as γ m , m represents the number of the mth keyword, m=1,2,...,q, through the formula Get the weight of each keyword in the user text δ m, where q represents the number of keyword phrases; it improves the accuracy and efficiency of text processing. By removing unnecessary punctuation and non-alphanumeric characters, the text becomes cleaner, facilitating subsequent analysis. Using stop-word filtering can reduce the interference of irrelevant words and highlight key information. Querying and comparing similar words can help identify semantically similar words and better understand the meaning and potential relationships of the text.

[0039] It should be noted that the stop words refer to some common words in natural language processing that have no practical meaning or little help in understanding the text, such as "de" (的), "le" (了), "shi" (是), "zai" (在), etc. These words usually appear too frequently and will be filtered out in the analysis to simplify the processing and highlight more meaningful words. Similar words refer to words that are semantically close or similar. For example, synonyms of "transformer" can include "power transformation equipment" (变电设备), "voltage transformation device" (变压装置); synonyms of "transmission line" can include "power transmission line" (送电线路), "power supply line" (供电线路), etc. In natural language processing, analyzing synonyms can better understand the semantics and connotations of the text and also help with some semantic-related operations and analyses.

[0040] It should be noted that the specific analysis method for stop-word filtering of the user text is as follows: Read the stop-word library from the management database, traverse and retrieve the user text according to the stop-word library, screen out the words in the user text that exist in the stop-word library, and filter them out from the user text.

[0041] It should be noted that the specific analysis method for each word and its similar words in the user text is as follows: Select any word from each word in the user text as the specified word, query for similar words for the specified word in the similar-word library, record the queried similar words as the similar words of the specified word, and compare the similar words of the specified word with each word in the user text except the specified word. Screen out the words in the user text that are the same as the similar words of the specified word, and record them as the similar words of the specified word in the user text. Analyze each word in the user text according to the method of analyzing the similar words of the specified word in the user text to obtain the similar words of each word in the user text.

[0042] In the second step, extract the existing problems stored in the management database, match each keyword phrase in the user text with each existing problem one by one, screen out the number of keyword phrases in each existing problem that coincide with the user text, and record it as N f , where f represents the number of the f-th existing problem, f = 1, 2,..., k. Extract the proportions of the keyword phrases in the user text that coincide with each existing problem from the proportions of each keyword in the user text, and record them as a represents the number of the a-th keyword phrase in the user text that coincides, a = 1, 2,..., b, and b ≤ q. Substitute it into the formula to obtain the similarity ξ between each existing problem and the user textf , φ1 and φ2 represent the weight factors of the set number of keyword overlaps and the proportion of keywords respectively, k represents the number of existing questions, and the similarity between each existing question and the user text is sorted in order from large to small, and the first existing question is recorded as the user text question related question; it can quickly find the existing question most similar to the user text, saving time and energy, and more accurately determine the related questions through multi-dimensional analysis such as the number of keyword overlaps and proportions, improve relevance, sort by similarity, and give priority to presenting the most relevant questions, which is convenient for further processing and response.

[0043] It should be noted that, in a specific embodiment, φ1 can be set to 0.5, and φ2 can be set to 0.5. The number of keyword overlaps reflects the degree of direct correlation to a certain extent, but may ignore the differences in the importance of some keywords. The keyword proportion is more focused on emphasizing the relative importance of key information in the whole, and can carefully consider the key factors of similarity. Therefore, the weights corresponding to the number of keyword overlaps and the keyword proportion are equal.

[0044] S3. Image recognition processing: read the image input by the user. If there is text information in the user image, obtain the user image question-related questions by performing text recognition processing on the user image. If there is no text information in the user image, perform image recognition processing on the user image, and obtain the user image question-related questions by extracting feature data for matching.

[0045] The specific analysis method for performing text recognition processing on user images is as follows: reading the image input by the user, recorded as the user image, recognizing and extracting the text in the user image through optical character recognition technology, recorded as user image extracted text, and analyzing the user image question-related questions according to the method of analyzing the user text question-related questions; it is possible to find existing questions related to the image just like processing text questions, thereby improving the pertinence of problem solving, allowing users to submit questions in a variety of forms (such as images containing text), thereby increasing the convenience and diversity of interaction.

[0046] The specific analysis method for performing image recognition processing on the user image is as follows: extracting characteristic data of the power grid from the user image, recorded as user question characteristic data, extracting each related question corresponding to each characteristic data from the management database, comparing the user question characteristic data with each related question corresponding to each characteristic data, obtaining each related question corresponding to the user question characteristic data, recorded as each related question asked by the user, reading each historical related question of the power grid user for each question characteristic data from the management database, and extracting each historical related question of the power grid user for each user question characteristic data, comparing each user question related question with each historical related question of the power grid user for each user question characteristic data, and obtaining the number of occurrences of each related question asked by the power grid user, recorded as i represents the number of the i-th related question, i = 1, 2, ..., n. At the same time, the historical questions asked by the user for each question feature data are read from the management database, and the corresponding related questions are compared with the related questions asked by the user. The number of occurrences of each related question asked by the user is counted and recorded as Substituting this into the formula Get the proportion of each related question of the user for the question feature data δ i , respectively represent the total number of questions asked by the set power grid users and the total number of questions asked by the users, μ1 and μ2 represent the set number of questions asked by the power grid users and the weight factors of the questions asked by the users, and the proportion of each associated question for the question feature data is arranged in order from large to small, and the first associated question corresponding to the proportion of the associated question for the question feature data is recorded as the user image question associated question; by extracting the power grid feature data and comparing it with the associated questions, the specific problems related to the user image can be found more accurately, and the pertinence of problem solving can be improved. At the same time, combined with the historical associated problems of power grid users and ordinary users, the frequency and importance of current problems can be analyzed more comprehensively, which is helpful to give priority to key problems.

[0047] It should be noted that, in a specific embodiment, μ1 can be set to 0.4, and μ2 can be set to 0.6. Questions asked by power grid users can provide more extensive reference and trend information, reflecting the behavior and characteristics of the overall user group, and user questions are directly targeted at understanding the user's specific preferences, habits, and behavior patterns. When providing personalized services or analysis for the user, the corresponding weight of user questions is higher.

[0048] It should be noted that the characteristic data of the power grid are various data information that can reflect the characteristics and properties of the power grid's operating status, performance, structure, etc. In a specific embodiment, the shape and color of the transformer are recognized based on the user's image, and it is used as a characteristic data to extract related issues corresponding to the characteristic data from the management database, such as the model, capacity, and operating status of the transformer.

[0049] S4. Voice recognition processing: Perform text recognition on the voice sent by the user to obtain the user's voice correction text, and then analyze the user's voice questions to obtain related questions.

[0050] The specific analysis method of the speech recognition processing is as follows: reading the speech sent by the user, performing noise reduction processing on the speech, and recording it as the user speech; using speech recognition technology to convert the user speech into text, and recording it as the user speech recognition text; and reading the user's place of origin from the network background; correcting the user speech recognition text according to common words in the accent area corresponding to the user's place of origin to obtain the user speech correction text; analyzing the user speech correction text according to the method of analyzing the similarity between each existing question and the user text, and obtaining the similarity between each existing question and the user speech correction text; filtering out the maximum value from the similarity between each existing question and the user speech correction text, and recording the corresponding existing question as the user speech question-related question; correcting by using common words in the accent area can reduce the recognition error caused by the accent, make the converted text more accurate, and more accurately find the existing question with the highest similarity to the user's needs, thereby improving the quality of question matching.

[0051] It should be noted that the specific analysis method for correcting the user speech recognition text is: obtaining the user's geographical location by acquiring the user's IP address, reading common words in different accent areas and their corresponding standard expressions from the management database, and confirming the user's accent area based on the user's geographical location, obtaining common words in the user's accent area and their corresponding standard expressions from the common words in different accent areas and their corresponding standard expressions, reading the user speech recognition text, searching the user speech recognition text based on the common words in the user's accent area, obtaining the common words in the user's accent area that exist in the user speech recognition text, recording them as the accent words in the user speech recognition text, and then matching them with the common words in the user's accent area and their corresponding standard expressions to obtain the standard expressions corresponding to the accent words in the user speech recognition text, replacing the accent words in the user speech recognition text with the standard expressions corresponding to the accent words in the user speech recognition text, and recording the replaced user speech recognition text as the user speech correction text.

[0052] S5. User question integration: obtain the user's text-question related questions, the user's image-question related questions, and the user's voice-question related questions, and integrate them to obtain the corresponding related questions of the user's questions.

[0053] It should be noted that the specific analysis method for the corresponding related questions asked by the user is: obtain the related questions asked by the user in text, the related questions asked by the user in image, and the related questions asked by the user in voice, merge them and record them as the related questions asked by the user, perform sentence structure analysis on each related question asked by the user, obtain the key points of each related question asked by the user, integrate the related questions asked by the user with the same key points, and record the integrated related questions as the corresponding related questions asked by the user.

[0054] It should be noted that the specific analysis method of the sentence structure analysis is: 1. Find the subject: The subject is the executor or subject of the action in the sentence, usually a person, a thing or a concept; 2. Determine the predicate: The predicate is a word that expresses the action or state of the subject; 3. Identify the object: The object is the object of the predicate verb, usually something affected by the action; 4. Pay attention to the attributive: The attributive is a word or phrase used to modify and limit the subject or object, usually to further describe and explain; 5. Pay attention to the adverbial: The adverbial is a component that describes the time, place, method, conditions, etc. of the action; 6. Distinguish the complement: The complement is a part that supplements the predicate to make the expression more complete; By dividing the sentence components, the subject, predicate, object, etc. are marked with symbols to refine the key points of the question.

[0055] It should be noted that, in a specific embodiment, a sentence structure analysis is performed on the question “In the construction of smart grids, how to better realize the efficient transmission and distribution of electricity?”, “In the construction of smart grids” is an adverbial clause, indicating the background, “how” is a question word, guiding the question, “better realize” is the predicate, and “efficient transmission and distribution of electricity” is the object. The analysis yields the key point of the question “Methods for realizing efficient transmission and distribution of electricity in the construction of smart grids”. In another specific embodiment, a sentence structure analysis is performed on the question “What key technologies are needed to support the intelligent upgrade of power grids?”, “Intelligent upgrade of power grids” is the subject, “need” is the predicate, and “which key technologies support” is the object. The analysis yields the key point of the question “Key technologies relied on for the intelligent upgrade of power grids”.

[0056] See also Figure 2 As shown, the analysis of recommended reply methods includes the recommendation degree of text reply, the recommendation degree of image reply, and the recommendation degree of voice reply, among which the recommendation degree of text reply includes the preference degree of text reply and the difficulty degree of interactive understanding, the recommendation degree of image reply includes the preference degree of image reply and the difficulty degree of interactive understanding, and the recommendation degree of voice reply includes the preference degree of voice reply and the difficulty degree of interactive understanding.

[0057] S6. User preference analysis: Based on the user's historical behavior, the number of voice replies, text replies, and voice replies of the intelligent customer service are recorded, as well as the scores of each text reply, image reply, and voice reply to obtain the user's preference for text reply, image reply, and voice reply.

[0058] The specific analysis method of the user preference analysis is as follows: read the user's historical behavior records from the management database, filter and count the number of text replies, image replies, and voice replies received by the user from the intelligent customer service, which are recorded as g, l, and b, and the scores of each text reply, image reply, and voice reply are recorded as j represents the number of the jth text reply, j = 1, 2, ..., g, p represents the number of the pth image reply, p = 1, 2, ..., l, a represents the number of the ath voice reply, a = 1, 2, ..., b, and substitute it into the formula Get the user's preference for text replies χ 文字 , η1 and η2 represent the set number of replies and the weight factor of the reply score, respectively. According to the method of analyzing the user's preference for text replies, the user's preference for image replies and the user's preference for voice replies are analyzed in turn, which are recorded as χ 图像 , χ 语音 Based on user preferences, the intelligent customer service response method can be adjusted more specifically, improving user experience, meeting user needs more accurately, and improving service effectiveness.

[0059] It should be noted that, in a specific embodiment, η1 can be set to 0.4, η2 can be set to 0.6, and the number of replies can reflect, to a certain extent, the frequency of users' use of a certain reply method. If a reply method is used more times, it may mean that the user has a certain preference or habit for it to a certain extent. The reply score more directly reflects the user's satisfaction and recognition of the reply method. A higher reply score indicates that the user is more satisfied with the reply method and can better reflect the user's preference for it. Therefore, the weight corresponding to the reply score is higher.

[0060] S7. Analysis of user interaction records: Obtain the user's interaction record data and analyze the difficulty of understanding the text replies, image replies, and voice replies. The interaction record data includes the number of times the user repeats each text reply, image reply, and voice reply, and the average reply time of each text reply, image reply, and voice reply.

[0061] The specific analysis method of the user's interaction record data is: extract the number of repeated inquiries of the user when receiving each text reply, image reply, and voice reply from the user's historical behavior record, and record it as the number of repeated inquiries of each text reply of the user The number of times the user replies to each image. The number of times the user repeated the question in each voice reply At the same time, the response time of each repeated question in each text response, image response, and voice response is recorded, and the average response time of each text response, image response, and voice response is obtained by taking the average value, which is recorded as By repeating the number of inquiries, we can see in which areas users tend to have more questions, so as to improve the content and method of responses in a targeted manner. The average response time can reflect the efficiency of intelligent customer service in different response forms, which is conducive to discovering problems and optimizing processes.

[0062] The specific analysis method of the user interaction record analysis is:

[0063] Interactive understanding difficulty of getting textual responses ρ 文字 , κ1 and κ2 represent the weight factors of the set number of repeated inquiries and the length of response time, respectively. The interactive understanding difficulty of image responses and the interactive understanding difficulty of voice responses are analyzed according to the method of analyzing the interactive understanding difficulty of text responses, which are recorded as ρ 图像 , 语音 ; Ability to quantify the difficulty of interactive understanding of text, images, and voice responses in order to more accurately understand service difficulties.

[0064] It should be noted that, in a specific embodiment, κ1 can be set to 0.7, and κ2 can be set to 0.3. The number of repeated inquiries directly shows that there are obstacles to understanding during the interaction process. When users frequently repeat the same question, this may indicate that they did not obtain sufficiently clear or accurate information in the previous or previous replies, resulting in an inability to fully understand and the need for further confirmation and clarification. The more repeated inquiries are made, the greater the degree of difficulty in understanding. A shorter response time may mean that the response is relatively concise and clear, and the user can quickly obtain key information and may be relatively easy to understand. However, if the response time is long, on the one hand, it may be that the question itself is more complicated and requires detailed explanation, but it may also be that the content organization or expression of the response is not clear and efficient enough, so that the user needs to spend more time to digest and understand. This also reflects the possible interaction understanding difficulties to a certain extent. Therefore, the weight corresponding to the number of repeated inquiries is higher.

[0065] S8. Recommended answer analysis: Based on the user's preference for text answers, image answers, and voice answers and the difficulty level of interactive understanding, the recommended answer method is obtained, and the corresponding related questions asked by the user are answered accordingly.

[0066] The specific analysis method of the recommended reply analysis is: reading the user's preference for the text reply x 文字 , Difficulty of interactive understanding of text responses ρ 文字 , substituting it into the formula Recommended level for getting text replies w1 and w2 represent the weight factors of the preference for the set reply method and the difficulty of interactive understanding of the reply method respectively. e is a natural constant. The recommendation degree of image reply and the recommendation degree of voice reply are analyzed according to the method of analyzing the recommendation degree of text reply. The maximum value is screened out from the recommendation degree of text reply, image reply and voice reply, and the corresponding reply method is recorded as the recommended reply method. Considering the user's preference for different reply methods, it can provide services that better meet the user's personal needs and improve user satisfaction. Analyzing the difficulty of interactive understanding of different reply methods helps to lower the user's understanding threshold and enhance the interactive effect.

[0067] It should be noted that in a specific embodiment, w1 can be set to 0.5, and w2 can be set to 0.5. If the user has a clear preference for a certain type of reply, such as a concise and direct reply or an illustrative reply, then the reply that meets the user's preference will be more likely to be recognized and accepted by the user, thereby having a greater impact on the recommendation level. The difficulty of interactive understanding of the reply method is also very important. If it is difficult for users to understand in actual use, then it will also be affected to a certain extent in terms of the recommendation level. Therefore, the weights corresponding to the preference for the reply method and the difficulty of interactive understanding of the reply method are equal.

[0068] See also Figure 3 As shown, the present invention proposes an application device, which includes a processor, a memory and a communication bus, and the memory stores a computer-readable program that can be executed by the processor.

[0069] The communication bus realizes the connection and communication between the processor and the memory.

[0070] When the processor executes the computer-readable program, the steps of the chromatography-mass spectrometry analysis method of the present invention are implemented.

[0071] The present invention performs text recognition processing, image recognition processing, and voice recognition processing respectively according to the user's questioning method, and then analyzes to obtain the user's text-related questions, the user's image-related questions, and the user's voice-related questions. According to the user's preference for text replies, image replies, and voice replies and the difficulty level of interactive understanding of text replies, image replies, and voice replies, the recommended reply method is analyzed and the user's question is replied accordingly. Through a variety of recognition and processing methods, different forms of user questions can be captured more comprehensively to ensure accurate grasp of the user's intentions. At the same time, considering the user's preference for different reply methods, the habits and preferences of different users can be better adapted.

[0072] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention and they are still covered by the protection scope of the present invention.

Claims

1. An intelligent customer service recommendation method integrating multimodal information, characterized in that: The steps include: S1. User question recognition: read the user's input content and identify the user's question method, and identify and process different question methods, including text, image, and voice; S2. Text recognition processing: Perform text processing on the text input by the user to obtain the key words of the user text, and then match them with the stored existing questions one by one to obtain the user's text questions related questions; S3. Image recognition processing: reading the image input by the user, if there is text information in the user image, then obtaining the user image question-related questions by performing text recognition processing on the user image, if there is no text information in the user image, then performing image recognition processing on the user image, obtaining the user image question-related questions by extracting feature data matching; S4. Speech recognition processing: Perform text recognition on the speech sent by the user to obtain the user's speech correction text, and then analyze the user's voice question related issues; S5. User question integration: obtain the user's text question-related questions, the user's image question-related questions, and the user's voice question-related questions, and integrate them to obtain the corresponding related questions of the user's questions; S6. User preference analysis: Based on the user's historical behavior, the number of voice replies, text replies, and voice replies of the intelligent customer service are recorded, and the scores of each text reply, image reply, and voice reply are obtained to obtain the user's preference for text reply, image reply, and voice reply; S7. User interaction record analysis: Obtain user interaction record data and analyze the difficulty of understanding text responses, image responses, and voice responses. The interaction record data includes the number of repeated inquiries by the user for each text response, image response, and voice response, as well as the average response time for each text response, image response, and voice response; S8. Recommended answer analysis: Based on the user's preference for text answers, image answers, and voice answers and the difficulty level of interactive understanding, the recommended answer method is obtained, and the corresponding related questions asked by the user are answered accordingly.

2. The intelligent customer service recommendation method integrating multimodal information according to claim 1, characterized in that: The specific analysis method of the text recognition processing is: The first step is to read the text input by the user, remove punctuation marks and non-alphanumeric characters in the text through text preprocessing technology, record the remaining text content as user text, divide the user text into words through natural language processing algorithm, record them as user text words, read the stop word thesaurus and similar word thesaurus from the management database, filter the stop words of the user text, and compare and filter each word of the user text with the similar word thesaurus to obtain each word of the user text and each similar word, filter them out from the user text, record the remaining words of the user text after filtering as user text keywords, and count the number of similar words that are filtered out for each keyword of the user text, record them as γ m , m represents the number of the mth keyword, m=1,2,...,q, through the formula Get the weight of each keyword in the user text q represents the number of key words; The second step is to extract the existing questions stored in the management database, match the key words in the user text with the existing questions one by one, and filter out the number of overlaps between the key words in the existing questions and the user text, recorded as N. f , f represents the number of the fth existing question, f = 1, 2, ..., k, and the proportion of the key words of each user text that overlaps with each existing question is extracted from the proportion of each keyword in the user text, recorded as a represents the number of the key word of the a-th user text, a=1,2,...,b, and b≤q. Substitute it into the formula Get the similarity between each existing question and the user text ξ f , φ1 and φ2 represent the weight factors of the set number of keyword overlaps and the keyword proportion respectively, k represents the number of existing questions, and the similarity between each existing question and the user text is sorted in descending order, and the first existing question is recorded as the user text question related question.

3. The intelligent customer service recommendation method integrating multimodal information according to claim 2 is characterized in that: The specific analysis method for performing text recognition processing on the user image is: The image input by the user is read and recorded as the user image. The text in the user image is recognized and extracted through optical character recognition technology, which is recorded as the user image extracted text. The user image question related questions are analyzed according to the method of analyzing the user text question related questions.

4. The intelligent customer service recommendation method integrating multimodal information according to claim 3 is characterized in that: The specific analysis method for performing image recognition processing on the user image is: Extract the characteristic data of the power grid from the user image, record it as the user question characteristic data, extract the related questions corresponding to each characteristic data from the management database, compare the user question characteristic data with the related questions corresponding to each characteristic data, obtain the related questions corresponding to the user question characteristic data, record it as the user question related questions, read the historical related questions of the power grid user for each question characteristic data from the management database, extract the historical related questions of the power grid user for the user question characteristic data, compare the user question related questions with the historical related questions of the power grid user for the user question characteristic data, and count the occurrence times of each related question asked by the power grid user, record it as i represents the number of the i-th related question, i = 1, 2, ..., n. At the same time, the historical questions asked by the user for each question feature data are read from the management database, and the corresponding related questions are compared with the related questions asked by the user. The number of occurrences of each related question asked by the user is counted and recorded as Substituting this into the formula Get the proportion of each related question of the user for the question feature data δ i , respectively represent the total number of questions asked by the set power grid users and the total number of questions asked by the users, μ1 and μ2 represent the weight factors of the set power grid users’ questions and user’s questions, and arrange the proportions of each associated question for the question feature data in order from large to small, and record the first associated question corresponding to the proportion of the associated question for the question feature data as the user image question associated question.

5. The intelligent customer service recommendation method integrating multimodal information according to claim 4 is characterized in that: The specific analysis method of the speech recognition processing is: The voice sent by the user is read, and after noise reduction processing, it is recorded as the user voice. The user voice is converted into text using voice recognition technology and recorded as user voice recognition text. The user's place of origin is read from the network background, and the user voice recognition text is corrected according to the common vocabulary of the accent area corresponding to the user's place of origin to obtain the user voice correction text. The user voice correction text is analyzed according to the method of analyzing the similarity between each existing question and the user text to obtain the similarity between each existing question and the user voice correction text. The maximum value is selected from the similarities between each existing question and the user voice correction text, and the corresponding existing question is recorded as the user voice question associated question.

6. The intelligent customer service recommendation method integrating multimodal information according to claim 1, characterized in that: The specific analysis method of the user preference analysis is: Read the user's historical behavior records from the management database, filter and count the number of text replies, image replies, and voice replies received by the user from the intelligent customer service, which are recorded as g, l, and b, and the scores of each text reply, image reply, and voice reply are recorded as j represents the number of the jth text reply, j = 1, 2, ..., g, p represents the number of the pth image reply, p = 1, 2, ..., l, a represents the number of the ath voice reply, a = 1, 2, ..., b, and substitute it into the formula Get the user's preference for text replies χ 文字 , η1 and η2 represent the set number of replies and the weight factor of the reply score, respectively. According to the method of analyzing the user's preference for text replies, the user's preference for image replies and the user's preference for voice replies are analyzed in turn, which are recorded as χ 图像 , χ 语音 .

7. The intelligent customer service recommendation method integrating multimodal information according to claim 6, characterized in that: The specific analysis method of the user interaction record is: Extract the number of times the user repeatedly inquires when receiving each text reply, image reply, and voice reply from the user's historical behavior record, and record it as the number of times the user repeatedly inquires for each text reply The number of times the user replies to each image. The number of times the user repeated the question in each voice reply At the same time, the response time of each repeated question in each text response, image response, and voice response is recorded, and the average response time of each text response, image response, and voice response is obtained by taking the average value, which is recorded as 8. The intelligent customer service recommendation method integrating multimodal information according to claim 7, characterized in that: The specific analysis method of the user interaction record analysis is: By formula Interactive understanding difficulty of getting textual responses ρ 文字 , κ1 and κ2 represent the weight factors of the set number of repeated inquiries and the length of response time, respectively. The interactive understanding difficulty of image responses and the interactive understanding difficulty of voice responses are analyzed according to the method of analyzing the interactive understanding difficulty of text responses, which are recorded as ρ 图像 , 语音 .

9. The intelligent customer service recommendation method integrating multimodal information according to claim 8, characterized in that: The specific analysis method of the recommended response analysis is: Read the user's preference for text replies χ 文字 , Difficulty of interactive understanding of text responses ρ 文字 , substituting it into the formula Recommended level for getting text replies w1 and w2 represent the weight factors of the preference degree of the set reply method and the difficulty of interactive understanding of the reply method respectively. e is a natural constant. The recommendation degree of image reply and the recommendation degree of voice reply are analyzed according to the method of analyzing the recommendation degree of text reply. The maximum value is screened out from the recommendation degree of text reply, image reply and voice reply, and the corresponding reply method is recorded as the recommended reply method.

10. An application device, characterized in that , comprising a processor, a memory and a communication bus, wherein the memory stores a computer-readable program executable by the processor; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, the steps of the intelligent customer service recommendation method integrating multimodal information as described in any one of claims 1 to 8 are implemented.

Images