Artificial intelligence-based complaint data identification method, apparatus and device, and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a complaint data identification method and device based on artificial intelligence, equipment and a medium, and the method comprises the steps: calibrating a complaint work order in complaint data as a positive sample or a gray sample; dividing the gray sample into N sub-sample sets, selecting one sub-sample set as a negative sample, and taking the gray samples in the rest N-1 sub-sample sets as to-be-predicted samples; inputting the negative sample and the positive sample into a prediction model for training to obtain a trained prediction model; predicting the to-be-predicted samples based on the trained prediction model to obtain the prediction probability of each to-be-predicted sample; repeating N rounds until all gray samples serve as samples to be predicted to participate in (N-1) times of prediction, and obtaining (N-1) prediction probabilities; calculating the average prediction probability of each gray sample; and judging the gray samples of which the average prediction probabilities exceed a preset risk threshold value as non-real agent complaint work orders. And the recognition accuracy of the non-real agent complaint work order is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based complaint data identification method, device, equipment and medium. Background Art

[0002] With the development of the Internet and mobile technology, the phenomenon of agency complaints has gradually become a prominent problem in various industries, especially in the financial industry. The agency complaint gangs file complaints on behalf of financial industry customers to financial companies and regulatory agencies in a paid manner, and use this to carry out illegal acts such as threats and blackmail, which seriously disrupts the normal market order. These complaints are often made in the capacity of non-directly related persons, which creates great difficulties in identification and evidence collection, causing financial companies to face a large number of invalid complaints, thereby consuming a large amount of customer service resources and affecting normal customer service and company reputation.

[0003] Traditional response methods, such as using voiceprint recognition technology to identify whether the complaint tickets in the complaint data are non-genuine agent complaint tickets (for example, the same person is the agent for multiple people to make complaints). However, since voiceprint recognition technology is limited by many factors, such as noise interference, changes in voice quality, the speaker's emotional state, etc., the recognition accuracy is low. Summary of the invention

[0004] The embodiments of the present invention provide a complaint data recognition method, device, equipment and medium based on artificial intelligence to solve the problem of low recognition accuracy of non-genuine agent complaint work orders in the prior art.

[0005] In the first aspect, a complaint data identification method based on artificial intelligence is provided, including: Obtain complaint data, and mark complaint work orders in the complaint data as positive samples or gray samples, wherein the positive samples represent non-real agent complaint work orders, and the gray samples represent agent complaint work orders to be confirmed; Divide the gray samples into N sub-sample sets, select one of the sub-sample sets as negative samples, and use the gray samples in the remaining N-1 sub-sample sets as samples to be predicted; Inputting the negative sample and the positive sample into a prediction model for training to obtain a trained prediction model; Based on the trained prediction model, the samples to be predicted are predicted to obtain the prediction probability of each sample to be predicted; repeat N rounds, wherein in each round of training and prediction, a different sub-sample set is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted in the current round, until all gray samples are used as samples to be predicted to participate in N-1 predictions, and N-1 prediction probabilities of each gray sample are obtained; Calculate the average predicted probability of each of the gray samples; The gray samples whose average predicted probability exceeds the preset risk threshold are determined to be non-genuine agent complaint work orders.

[0006] In the second aspect, a complaint data identification device based on artificial intelligence is provided, comprising: An acquisition module is used to acquire complaint data, and mark complaint work orders in the complaint data as positive samples or gray samples, wherein the positive samples represent non-real agent complaint work orders, and the gray samples represent agent complaint work orders to be confirmed; A division module is used to divide the gray sample into N sub-sample sets, select one of the sub-sample sets as a negative sample, and select the gray samples in the remaining N-1 sub-sample sets as samples to be predicted; A training module, used for inputting the negative samples and the positive samples into a prediction model for training to obtain a trained prediction model; A prediction module is used to predict the samples to be predicted based on the trained prediction model to obtain the prediction probability of each sample to be predicted; repeat N rounds, wherein in each round of training and prediction, a different sub-sample set is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted in the current round, until all gray samples are used as samples to be predicted to participate in N-1 predictions, and N-1 prediction probabilities of each gray sample are obtained; A calculation module, used for calculating the average prediction probability of each of the gray samples; The first determination module is used to determine the gray samples whose average predicted probability exceeds a preset risk threshold as non-genuine agent complaint work orders.

[0007] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the complaint data identification method based on artificial intelligence described in the first aspect is implemented.

[0008] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the artificial intelligence-based complaint data identification method described in the first aspect is implemented.

[0009] In a solution implemented by the above-mentioned complaint data identification method, device, equipment and medium based on artificial intelligence, by obtaining complaint data, the complaint work orders in the complaint data are calibrated as positive samples or gray samples, wherein the positive samples represent non-real agent complaint work orders, and the gray samples represent agent complaint work orders to be confirmed; the gray samples are divided into N sub-sample sets, one of which is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted; the negative samples and the positive samples are input into the prediction model for training to obtain a trained prediction model; based on the trained prediction model, the samples to be predicted are predicted to obtain the prediction probability of each sample to be predicted; N rounds are repeated, wherein in each round of training and prediction, a different sub-sample set is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted in the current round, until all gray samples are used as samples to be predicted to participate in N-1 predictions, and N-1 prediction probabilities of each gray sample are obtained; the average prediction probability of each gray sample is calculated; and the gray samples whose average prediction probability exceeds the preset risk threshold are determined to be non-real agent complaint work orders. In the present invention, by calibrating multiple complaint work orders in the complaint data, positive samples (non-real agent complaint work orders) and gray samples (agent complaint work orders to be confirmed) are distinguished. Then, a multi-round recommendation method is adopted to obtain N-1 prediction probabilities for each gray sample. Then, by calculating the average prediction probability of each gray sample and comparing it with the preset risk threshold, non-real agent complaint work orders can be effectively identified, which not only improves the recognition accuracy but also reduces the risk of misjudgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 It is a schematic diagram of an application environment of a complaint data identification method based on artificial intelligence in one embodiment of the present invention; Figure 2 is a flow chart of a complaint data identification method based on artificial intelligence in one embodiment of the present invention; Figure 3 is a training diagram of a prediction model in one embodiment of the present invention; Figure 4 is another flow chart of a complaint data identification method based on artificial intelligence in one embodiment of the present invention; Figure 5is another flow chart of a complaint data identification method based on artificial intelligence in one embodiment of the present invention; Figure 6 is another flow chart of a complaint data identification method based on artificial intelligence in one embodiment of the present invention; Figure 7 is another flow chart of a complaint data identification method based on artificial intelligence in one embodiment of the present invention; Figure 8 is a schematic diagram of a complaint data identification device based on artificial intelligence in one embodiment of the present invention; Fig. 9 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The complaint data recognition method based on artificial intelligence provided by the embodiment of the present invention can be applied in Figure 1 In the application environment shown, the client and the server communicate through the network. The server can obtain complaint data from the client, and calibrate multiple complaint work orders in the complaint data, distinguish positive samples from gray samples, and then use multiple rounds of recommendation to obtain N-1 prediction probabilities for each gray sample, and then by calculating the average prediction probability of each gray sample and comparing it with the preset risk threshold, it can effectively identify non-real agent complaint work orders, which not only improves the recognition accuracy, but also reduces the risk of misjudgment. Among them, the client can be but is not limited to various personal computers, laptops, smart phones and tablets. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0014] In one embodiment, if Figure 2 As shown, a complaint data recognition method based on artificial intelligence is provided, and the method is applied in Figure 1 The server in the example is used as an example to illustrate the process, including the following steps: S10. Obtain complaint data, and mark complaint work orders in the complaint data as positive samples or gray samples, wherein the positive samples represent non-real agent complaint work orders, and the gray samples represent agent complaint work orders to be confirmed.

[0015] In step S10, the complaint data includes multiple complaint worksheets within a preset time period, wherein the complaint worksheet includes information such as the complaint call number, customer registration information, complaint voice, complaint time, etc. For example, in the insurance application field, the complaint call is the complainant's phone number; the customer registration information includes the name, contact information (for example, the customer's pre-registered phone number) and account information; the complaint voice includes whether the customer is familiar with the basic situation, what the complaint point is, what the core reason for the complaint is, whether there is a threat to escalate the complaint, what the specific requirements are, and whether there is a request for monetary compensation, etc.; the complaint time includes the specific date and time when the complaint occurred.

[0016] After obtaining the complaint data, the complaint call number in the complaint ticket and the customer's pre-registered phone number in the customer registration information can be classified and calibrated to obtain the calibrated positive samples and gray samples, or the complaint voice in the complaint ticket can be classified and calibrated, and the specific details are not limited. Among them, the positive sample represents the confirmed non-real agent complaint ticket, for example, it is not initiated by the actual affected customer or the customer's "direct relationship", but is maliciously initiated by an agent or a third party, where the direct relationship can be a spouse or child; the gray sample represents the unconfirmed agent complaint ticket, that is, the ticket cannot be distinguished as a non-real agent complaint ticket or a real agent complaint ticket.

[0017] Taking a financial company as an example, the server can obtain a batch of complaint data within a preset time period from the client (such as a customer service center, online platform, etc.). For example, complaint tickets within a week, assuming a total of 180 complaint tickets, after calibration, 30 of them are calibrated as positive samples (non-real agent complaint tickets), and the remaining 150 complaint tickets are calibrated as gray samples (agent complaint tickets to be confirmed).

[0018] The above-mentioned labeling of multiple complaint work orders in the complaint data as positive samples and gray samples respectively provides an accurate data basis for subsequent multi-round recommendation modeling and prediction, and reduces the computational burden of the prediction model.

[0019] S20, dividing the gray samples into N sub-sample sets, selecting one sub-sample set as a negative sample, and using the gray samples in the remaining N-1 sub-sample sets as samples to be predicted.

[0020] In step S20, in order to effectively determine whether each gray sample is a non-real complaint work order, the calibrated gray samples are divided. Preferably, the obtained multiple gray samples can be divided into N parts equally or randomly, which is not limited by the specific present invention.

[0021] Taking the 150 gray samples calibrated in step S10 as an example, the above 150 gray samples are divided into N sub-sample sets. Assume that N=5 is selected, that is, the 150 gray samples are divided into 5 sub-sample sets, each sub-sample set contains 30 gray samples. The purpose of this step is to flexibly select different sub-sample sets as negative samples in the subsequent training process, train the prediction model, and then predict the samples to be predicted, so as to improve the recognition accuracy of gray samples.

[0022] In the specific implementation process, 150 gray samples can be divided into 5 sub-sample sets, denoted as A1, A2, A3, A4 and A5, where each sub-sample set contains 30 gray samples. For example: Subsample set 1 (A1): G1, G2, G3, ..., G30; Subsample set 2 (A2): H1, H2, H3, ..., H30; Subsample set 3 (A3): I1, I2, I3, ..., I30; Subsample set 4 (A4): J1, J2, J3, ..., J30; Subsample set 5 (A5): K1, K2, K3, ..., K30.

[0023] Next, select one of the subsample sets as the negative sample for the current round, and the gray samples in the remaining N-1 subsample sets are used as the samples to be predicted for the current round. Among them, the negative sample represents the real agent complaint ticket. For example, suppose subsample set 1 (A1) is selected as the negative sample, and the gray samples in the remaining N-1 subsample sets (A2, A3, A4, A5) are used as the samples to be predicted.

[0024] S30, inputting the negative sample and the positive sample into a prediction model for training to obtain a trained prediction model.

[0025] In step S30, the prediction model may be a logistic regression model, a random forest model, an xgboost model or a deep learning model, without specific limitation.

[0026] Next, we continue to take the selection of subsample set 1 (A1) as negative samples in step S20 and the gray samples in the remaining subsample sets (A2, A3, A4, A5) as samples to be predicted as an example for explanation, as follows: The 30 gray samples in the negative samples selected in step S20 and the positive samples calibrated in step S10 can be input as test samples into the prediction model for training. During the training process, the prediction model continuously adjusts the internal parameters to correctly distinguish between positive samples and negative samples, thereby obtaining a trained prediction model. Taking the logistic regression model Model = sigmoid (KX + b) as an example, during the training process, by continuously adjusting the internal parameters K and b of the logistic regression model, the logistic regression model makes the prediction of positive samples and negative samples more accurate. Specifically, the learning algorithm inside the logistic regression model will try to find a suitable parameter combination so that when the input is a positive sample, the output value of the logistic regression model (that is, the probability value after processing by the sigmoid function) is close to 1; and when the input is a negative sample, the output value of the logistic regression model is close to 0, thereby obtaining a trained logistic regression model.

[0027] S40, based on the trained prediction model, predict the samples to be predicted to obtain the prediction probability of each sample to be predicted; repeat N rounds, wherein in each round of training and prediction, select a different sub-sample set as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted in the current round, until all gray samples are used as samples to be predicted to participate in N-1 predictions, and N-1 prediction probabilities of each gray sample are obtained.

[0028] In step S40, based on the prediction model trained in step S30, the samples to be predicted are predicted to obtain the prediction probability of each sample to be predicted, and the above process is repeated for N rounds. As an example, it is assumed that the gray samples are divided into 5 sub-sample sets, denoted as A1, A2, A3, A4 and A5. In the first round of model training and prediction, A1 is selected as a negative sample. At this time, G1, G2, G3, ..., G30 in A1 and the multiple positive samples calibrated in step S10 will be used as training samples for training the prediction model. The gray samples in A2, A3, A4 and A5 are used as samples to be predicted, that is, H1, H2, H3, ..., H30 in A2 are all used as samples to be predicted, and so on, K1, K2, K3, ..., K30 in A5 are also used as samples to be predicted, and these samples to be predicted will be input into the trained prediction model for prediction to obtain the prediction probability of each sample to be predicted in the first round.

[0029] After completing the first round of training and prediction, the second round of training and prediction begins. In this round, A2 is selected as a new negative sample, and the gray samples in A1, A3, A4, and A5 are used as samples to be predicted, and the same training and prediction process is continued. This process is repeated for N rounds, and a new negative sample is selected from the remaining sub-sample set in each round, so as to ensure that each gray sample participates in N-1 predictions in N rounds, so that each gray sample obtains N-1 prediction probabilities.

[0030] It should be noted that, in the N rounds of training and prediction process of the present application, the same model can be selected for each round of training and prediction, or multiple models can be selected for training and prediction. For example, the first round selects a logistic regression model for training and prediction, the second round selects a random forest model for training and prediction, the third round selects an xgboost model for training and prediction, and so on, the Nth round selects a deep learning model for training and prediction, or the first to third rounds select a random forest model for training and prediction, the fourth to sixth rounds select an xgboost model for training and prediction, and so on, the N-3 round to the Nth round selects a deep learning model for training and prediction, and the specific present invention is not limited.

[0031] S50, calculating the average prediction probability of each of the gray samples.

[0032] S60: Determine the gray samples whose average predicted probability exceeds a preset risk threshold as non-genuine agent complaint work orders.

[0033] In steps S50 and S60, after N rounds of training and prediction are completed, the N-1 prediction results of each gray sample are counted, and the average prediction probability of each gray sample is calculated. Further, each gray sample is determined to be a non-real agent complaint work order according to the preset risk threshold. If the average prediction probability of a gray sample exceeds the preset risk threshold, the gray sample is determined to be a non-real agent complaint work order. Otherwise, the gray sample will be determined to be a real agent complaint work order.

[0034] For example, in N-1 predictions of a gray sample (e.g., G1), assuming N is 5, the probabilities of the gray sample (G1) for 4 times are 0.6, 0.7, 0.65, and 0.75 respectively, and its average predicted probability is (0.6+0.7+0.65+0.75) / 4 = 0.675. The preset risk threshold is 0.6. At this time, the average predicted probability of the gray sample (G1) exceeds the preset risk threshold, and the gray sample (G1) will be judged as a non-genuine agent complaint ticket.

[0035] Through multiple rounds of prediction, it is possible to more comprehensively and accurately evaluate whether each gray sample is a non-genuine agent complaint ticket. This not only reduces the error, but also effectively reduces the workload of manual judgment and significantly improves the accuracy of the judgment results.

[0036] In summary, by obtaining complaint data, the complaint work orders in the complaint data are calibrated into positive samples and gray samples, wherein the positive samples represent non-real agent complaint work orders, and the gray samples represent agent complaint work orders to be confirmed; the gray samples are divided into N sub-sample sets, one of which is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted; the negative samples and the positive samples are input into the prediction model for training to obtain a trained prediction model; based on the trained prediction model, the samples to be predicted are predicted to obtain the prediction probability of each sample to be predicted; N rounds are repeated, wherein in each round of training and prediction, a different sub-sample set is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted in the current round, until all gray samples are used as samples to be predicted to participate in N-1 predictions, and N-1 prediction probabilities of each gray sample are obtained; the average prediction probability of each gray sample is calculated; the gray samples whose average prediction probability exceeds the preset risk threshold are determined to be non-real agent complaint work orders. In the present invention, by calibrating multiple complaint work orders in the complaint data, positive samples (non-real agent complaint work orders) and gray samples (agent complaint work orders to be confirmed) are distinguished. Then, a multi-round recommendation method is adopted to obtain N-1 prediction probabilities for each gray sample. Then, by calculating the average prediction probability of each gray sample and comparing it with the preset risk threshold, non-real agent complaint work orders can be effectively identified, which not only improves the accuracy of recognition, but also reduces the risk of misjudgment.

[0037] like Figure 3 As shown, taking a customer complaint of a bank as an example, the contents described in the above steps S10 to S60 are summarized as follows with a complete example: First, all gray samples are divided into N sub-sample sets. For example, assuming there are 150 gray samples, select N=5, and divide the 150 gray samples into 5 sub-sample sets on average, each sub-sample set contains 30 gray samples. This process can be discussed with reference to the embodiment of step S20, and will not be described here to avoid repetition.

[0038] Round 1: Select negative samples: Select the first subsample set (A1) as negative samples.

[0039] Samples to be predicted: The gray samples in the remaining sub-sample sets (A2, A3, A4, A5) are used as samples to be predicted in the current round.

[0040] Model training and prediction: Use the trained prediction model to predict the samples to be predicted and obtain the prediction probability of each sample to be predicted.

[0041] Round 2: Select negative samples: Select the second subsample set (A2) as negative samples.

[0042] Samples to be predicted: The gray samples in the subsample set (A1, A3, A4, A5) are taken as samples to be predicted.

[0043] Model training and prediction: Repeat the training and prediction process to obtain new prediction probabilities.

[0044] Round 3: Select negative samples: Select the third subsample set (A3) as negative samples.

[0045] Samples to be predicted: The gray samples in the subsample set (A1, A2, A4, A5) are taken as samples to be predicted.

[0046] Model training and prediction: Continue to make predictions and obtain the corresponding prediction probabilities.

[0047] Round 4: Select negative samples: Select the fourth subsample set (A4) as negative samples.

[0048] Samples to be predicted: The gray samples in the sub-sample set (A1, A2, A3, A5) are taken as samples to be predicted.

[0049] Model training and prediction: Make predictions again to obtain prediction probabilities.

[0050] Round 5: Select negative samples: Select the fifth subsample set (A5) as negative samples.

[0051] Samples to be predicted: The gray samples in the subsample set (A1, A2, A3, A4) are taken as samples to be predicted.

[0052] Model training and prediction: Complete the last round of prediction and obtain the prediction probability.

[0053] After N rounds (5 rounds in this example) of training and prediction, each gray sample participated in N-1 predictions as a sample to be predicted, and obtained N-1 prediction probabilities.

[0054] Next, the average of the N-1 predicted probabilities of each gray sample is calculated to obtain the average predicted probability of the gray sample. Based on this average predicted probability, it is compared with the preset risk threshold: If the average predicted probability exceeds the preset risk threshold, the gray sample is determined to be a non-genuine agent complaint work order.

[0055] If the average predicted probability is lower than the preset risk threshold, the gray sample is determined to be a real agent complaint work order.

[0056] Through this cross-validation method, the limited gray sample data is fully utilized, the reliability and stability of the model prediction are improved, the misjudgment caused by single prediction deviation is avoided, and the performance of the model on different data sets is effectively evaluated, which is conducive to the optimization and improvement of the model.

[0057] In one embodiment, if Figure 4 As shown, in step S10, that is, marking the complaint work orders in the complaint data as positive samples or gray samples, respectively, includes the following steps: S11, obtaining the complaint call number in the complaint work order; S12, determining whether the complaint call number is a telephone number pre-registered by the customer, where the telephone number pre-registered by the customer includes a primary account telephone number and an emergency contact telephone number; S13. If the complaint call number is not the phone number pre-registered by the customer, obtain the number of complaints of different complaint work orders corresponding to the complaint call number; S14. According to the number of complaints and preset data, the complaint work order is marked as a positive sample or a negative sample.

[0058] In steps S11 to S14, the complaint call number is first obtained from the complaint ticket. Then, it is determined whether the complaint call number is a phone number pre-registered by the customer, for example, whether the complaint call number is a primary account phone number or an emergency contact phone number. If the complaint call number is a phone number pre-registered by the customer, it is identified as a complaint ticket to be verified to further determine whether it is the customer's true intention and to avoid malicious complaints by third-party agents using virtual application numbers. Among them, the virtual application number is a customized phone number using virtual phone software.

[0059] If the complaint call number is not the customer's pre-registered phone number, that is, the complaint call number is not the main account phone number (such as the customer's own number) nor the customer's registered emergency contact phone number (such as the spouse or child's phone number), then the complaint call is considered a call from a "non-directly related person". Further analyze the number of other complaint tickets associated with the complaint call number to obtain the number of complaints for different complaint tickets corresponding to the complaint call number, that is, count the number of complaints initiated by the complaint call number for "non-directly related persons". Further, based on the size of the obtained number of complaints and the preset number, the complaint ticket is marked as a positive sample or a negative sample.

[0060] Specifically, the process of marking the complaint work order as a positive sample or a negative sample according to the number of complaints and preset data includes the following steps: S141. If the number of complaints is greater than or equal to the preset number, marking the complaint work order as the positive sample; S142. If the number of complaints is less than the preset number, the complaint work order is marked as the gray sample.

[0061] In steps S141 and S142, if the number of complaints is greater than or equal to the preset number, the complaint ticket is marked as a positive sample. This indicates that the complaint is a malicious complaint from a "non-directly related person", that is, it is not a real agent complaint ticket. If the number of complaints is less than the preset number, the complaint ticket is marked as a gray sample. This indicates that it is impossible to clearly determine whether it is a malicious complaint from a "non-directly related person" or a real agent complaint ticket.

[0062] The following is a specific example to illustrate the process of the above steps S11 to S14: Assume that a financial services company receives a customer complaint ticket. First, obtain the complaint call number in the complaint ticket and record it as number A.

[0063] Next, determine whether this A number is the telephone number pre-registered by the customer in the system (i.e., the telephone number pre-registered by the customer). Assume that the telephone number pre-registered by the customer includes the telephone number of the main account (recorded as B number) and the telephone number of the emergency contact (recorded as C number).

[0064] Case 1: If number A matches number B or number C, it means that the complaint may be initiated by the customer himself or his emergency contact. Therefore, the complaint ticket is preliminarily identified as a complaint ticket to be verified and the subsequent verification process is carried out.

[0065] Case 2: If number A matches neither number B nor number C, it is considered a complaint from a "non-directly related person" customer. Further inquiries are made to all complaint tickets that use number A as the complaint call number, and the number of complaints in these complaint tickets is calculated.

[0066] Sub-case 2 1: Assume that the preset number is 2. If the number of complaint tickets corresponding to number A is greater than or equal to 2, it means that number A is frequently used by a third-party agent to make complaints, so the complaint ticket is marked as a positive sample, that is, a non-genuine agent complaint ticket.

[0067] Sub-case 2: If the number of complaint tickets corresponding to number A is less than 2, for example, only 1, then it is impossible to directly determine whether this is a malicious complaint. Therefore, the complaint ticket is marked as a gray sample, that is, an agent complaint ticket to be confirmed, which requires further verification and analysis.

[0068] The above process can intelligently identify and label complaint tickets, providing strong support for subsequent risk control and response strategies.

[0069] In one embodiment, if Figure 5 As shown, in step S30, that is, inputting the negative sample and the positive sample into the prediction model for training to obtain the trained prediction model, the following steps are included: S31. Use a pre-trained large language model to extract features of the customer complaint speech in the negative sample and the positive sample to obtain first feature information.

[0070] S32: Use the first feature information, the negative sample, and the positive sample as a training data set, and input them into the prediction model for training to obtain a trained prediction model.

[0071] S33, obtaining a pre-calibrated verification data set, and using the verification data set to evaluate the trained prediction model to obtain an evaluation result.

[0072] S34: If the evaluation result meets the expected result, the trained prediction model is used as the trained pre-trained model.

[0073] In steps S31 to S34, the pre-trained large language model is used to extract features from the customer complaint speech in the negative sample and the positive sample. First, the customer complaint speech can be converted into text data. This step can be achieved with the help of speech recognition technology. Then, the large language model is used to extract features from the converted text data. For example, features can be extracted by word embedding (such as Word2Vec, GloVe and other algorithms), context representation (such as Transformer, LSTM and other models), attention mechanism (such as BERT, GPT and other models) and other methods to obtain first feature information, which includes the basic situation of the customer, complaint points, reasons for complaints, threatening remarks and complaint requirements. Then, the extracted first feature information, negative samples and positive samples are used as training data sets. Among them, the first feature information can be used as the input feature of the prediction model, and the labels of the negative samples and positive samples are used for supervised learning. Then, the training data set is input into the prediction model for training. During the training process, the prediction model will continuously optimize its own parameters to obtain a trained prediction model. Next, obtain a pre-calibrated validation data set and use it to evaluate the trained prediction model to obtain evaluation results, which may include accuracy, recall, and F1 score. If the evaluation results meet the expected results, for example, the accuracy is 90%, the recall is 85%, and the F1 score is 87.5%, it means that the evaluation results meet the expected results. At this time, the trained prediction model is used as a trained pre-trained model and used for the actual complaint data recognition task.

[0074] It should be understood that the pre-trained large language model in the present application is obtained by training the large language model based on historical complaint data, and has the ability to extract customer complaint voice.

[0075] In one embodiment, if Figure 6 As shown, in step S40, based on the trained prediction model, the gray samples in the remaining N-1 samples to be predicted are predicted to obtain the prediction probability corresponding to each gray sample, including the following steps: S41, using the large prediction model to extract features from the customer complaint voice of the sample to be predicted to obtain second feature information; S42. The second feature information is input into the trained prediction model to obtain a prediction probability of each sample to be predicted.

[0076] In steps S41 and S42, the large prediction model is used to extract features from the customer complaint voice of the sample to be predicted. After obtaining the second feature information, the second feature information of the sample to be predicted is input into the trained prediction model. The pre-trained prediction model will output a prediction probability for each sample to be predicted, indicating the probability that the sample to be predicted belongs to a non-real agent complaint work order. For example, if the prediction probability of a sample to be predicted is 0.85, then there is an 85% chance that the sample will be judged as a non-real agent complaint work order. The accuracy of the classification is ensured, providing data support for further improving the recognition accuracy of gray samples in the future.

[0077] Taking a financial company as an example, assuming that feature extraction is performed on the complaint voice in the prediction sample, the second feature information obtained is unfamiliarity with the customer's basic situation, the complaint point is super repayment of credit card, the core reason for the complaint is that I don't know why the repayment interest rate of my credit card is so high, there are threatening words, and there is a request for compensation for money, etc.

[0078] Next, this information is input into the trained prediction model. Assuming that in this example, after the trained prediction model is processed, the output prediction probability is 0.90, which means that there is a 90% probability that the complaint ticket is a non-genuine agent complaint ticket maliciously initiated by a third-party agent.

[0079] Through this automated identification process, financial companies can significantly improve the efficiency of identifying non-genuine agent complaints while also saving labor costs.

[0080] In one embodiment, if Figure 7 As shown, the complaint data identification method based on artificial intelligence also includes the following steps: S70: If the complaint work order is marked as the gray sample, use a pre-trained large language model to extract features of the customer complaint voice in the gray sample to obtain third feature information; S80, comparing the third characteristic information with the preset characteristic information to obtain a comparison result; S90. According to the comparison result, determine whether the gray sample is a non-genuine agent complaint work order.

[0081] In steps S70 to S90, when the complaint worksheet is marked as a gray sample, at this time, the pre-trained large language model can also be directly called to extract the features of the customer complaint voice in the gray sample to obtain the third feature information, which includes the basic situation of the customer known by the complainant, the complaint point, whether there are threatening remarks and the complaint requirements, etc. Then, the extracted third feature information is compared with the preset feature information to obtain the comparison result. For example, the basic situation of the customer known by the complainant is compared with the pre-registered customer registration information to obtain the comparison result. Then, according to the comparison result, it is directly judged whether the gray sample belongs to a non-real agent complaint worksheet. For example, if the comparison result includes threatening remarks (for example, if it is not resolved, I will complain to my superiors to the end), there is financial compensation (for example, compensation of 10,000 yuan) and / or the basic situation of the customer known by the complainant is inconsistent with the pre-registered customer registration information, then it is determined whether the gray sample is a non-real agent complaint worksheet.

[0082] By combining the feature extraction capability of a large language model with the comparative analysis capability of preset feature information, it is possible to more accurately identify real and fake agent complaint tickets, which improves processing efficiency and accuracy compared to traditional manual verification and processing by customer service.

[0083] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0084] In one embodiment, a complaint data identification device based on artificial intelligence is provided, and the complaint data identification device based on artificial intelligence corresponds one-to-one to the complaint data identification method based on artificial intelligence in the above embodiment. Figure 8 As shown, the complaint data identification device based on artificial intelligence includes an acquisition module 101, a division module 102, a training module 103, a prediction module 104, a calculation module 105 and a first determination module 106. The functional modules are described in detail as follows: The acquisition module 101 is used to acquire complaint data, and mark the complaint work orders in the complaint data as positive samples or gray samples, wherein the positive samples represent non-real agent complaint work orders, and the gray samples represent agent complaint work orders to be confirmed; A division module 102 is used to divide the gray samples into N sub-sample sets, select one sub-sample set as a negative sample, and select the gray samples in the remaining N-1 sub-sample sets as samples to be predicted; A training module 103 is used to input the negative samples and the positive samples into a prediction model for training to obtain a trained prediction model; The prediction module 104 is used to predict the samples to be predicted based on the trained prediction model to obtain the prediction probability of each sample to be predicted; repeat N rounds, wherein in each round of training and prediction, a different sub-sample set is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted in the current round, until all gray samples are used as samples to be predicted to participate in N-1 predictions, and N-1 prediction probabilities of each gray sample are obtained; A calculation module 105 is used to calculate the average prediction probability of each of the gray samples; The first determination module 106 is used to determine the gray samples whose average predicted probability exceeds a preset risk threshold as non-genuine agent complaint work orders.

[0085] In one embodiment, the acquisition module 101 is specifically used for: Obtain the complaint call number in the complaint ticket; Determine whether the complaint call number is a phone number pre-registered by the customer, where the phone number pre-registered by the customer includes the primary account phone number and the emergency contact phone number; If the complaint call number is not the phone number pre-registered by the customer, then obtaining the number of complaints of different complaint work orders corresponding to the complaint call number; According to the number of complaints and preset data, the complaint work order is marked as a positive sample or a negative sample.

[0086] In one embodiment, the acquisition module 101 is further configured to: If the number of complaints is greater than or equal to the preset number, the complaint work order is marked as a positive sample; If the number of complaints is less than the preset number, the complaint work order is marked as a gray sample.

[0087] In one embodiment, the training module 103 is specifically used for: Using a pre-trained large language model to extract features from the customer complaint speech in the negative sample and the positive sample to obtain first feature information; Input the first feature information, the negative sample, and the positive sample as a training data set into the prediction model for training to obtain a trained prediction model; Obtaining a pre-calibrated validation data set, and using the validation data set to evaluate the trained prediction model to obtain an evaluation result; If the evaluation result meets the expected result, the trained prediction model is used as the trained pre-trained model.

[0088] In one embodiment, the first characteristic information includes basic information of the customer, complaint points, complaint reasons, threatening remarks and complaint requirements.

[0089] In one embodiment, the prediction module 104 is specifically configured to: Using the large prediction model to extract features from the customer complaint speech of the sample to be predicted, to obtain second feature information; The second feature information is input into the trained prediction model to obtain the prediction probability of each sample to be predicted.

[0090] In one embodiment, the complaint data identification device based on artificial intelligence further includes: An extraction module, configured to extract features of the customer complaint speech in the gray sample using a pre-trained large language model to obtain third feature information if the complaint work order is marked as the gray sample; A comparison module, used to compare the third characteristic information with the preset characteristic information to obtain a comparison result; The second determination module is used to determine whether the gray sample is a non-genuine agent complaint work order according to the comparison result.

[0091] For the specific definition of the complaint data identification device based on artificial intelligence, please refer to the definition of the complaint data identification method based on artificial intelligence above, which will not be repeated here. Each module in the above-mentioned complaint data identification device based on artificial intelligence can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0092] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the complaint data required for running the complaint data identification method based on artificial intelligence and the results obtained. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a complaint data identification method based on artificial intelligence is implemented.

[0093] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: Obtain complaint data, and mark complaint work orders in the complaint data as positive samples or gray samples, wherein the positive samples represent non-real agent complaint work orders, and the gray samples represent agent complaint work orders to be confirmed; Divide the gray samples into N sub-sample sets, select one of the sub-sample sets as negative samples, and use the gray samples in the remaining N-1 sub-sample sets as samples to be predicted; Inputting the negative sample and the positive sample into a prediction model for training to obtain a trained prediction model; Based on the trained prediction model, the samples to be predicted are predicted to obtain the prediction probability of each sample to be predicted; repeat N rounds, wherein in each round of training and prediction, a different sub-sample set is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted in the current round, until all gray samples are used as samples to be predicted to participate in N-1 predictions, and N-1 prediction probabilities of each gray sample are obtained; Calculate the average predicted probability of each of the gray samples; The gray samples whose average predicted probability exceeds the preset risk threshold are determined to be non-genuine agent complaint work orders.

[0094] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Obtain complaint data, and mark complaint work orders in the complaint data as positive samples or gray samples, wherein the positive samples represent non-real agent complaint work orders, and the gray samples represent agent complaint work orders to be confirmed; Divide the gray samples into N sub-sample sets, select one of the sub-sample sets as negative samples, and use the gray samples in the remaining N-1 sub-sample sets as samples to be predicted; Inputting the negative sample and the positive sample into a prediction model for training to obtain a trained prediction model; Based on the trained prediction model, the samples to be predicted are predicted to obtain the prediction probability of each sample to be predicted; repeat N rounds, wherein in each round of training and prediction, a different sub-sample set is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted in the current round, until all gray samples are used as samples to be predicted to participate in N-1 predictions, and N-1 prediction probabilities of each gray sample are obtained; Calculate the average predicted probability of each of the gray samples; The gray samples whose average predicted probability exceeds the preset risk threshold are determined to be non-genuine agent complaint work orders.

[0095] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the aforementioned method embodiments. To avoid repetition, they will not be described one by one here.

[0096] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0097] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0098] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A complaint data identification method based on artificial intelligence, characterized in that: include: Obtain complaint data, and mark complaint work orders in the complaint data as positive samples or gray samples, wherein the positive samples represent non-real agent complaint work orders, and the gray samples represent agent complaint work orders to be confirmed; Divide the gray samples into N sub-sample sets, select one of the sub-sample sets as negative samples, and use the gray samples in the remaining N-1 sub-sample sets as samples to be predicted; Inputting the negative sample and the positive sample into a prediction model for training to obtain a trained prediction model; Based on the trained prediction model, the samples to be predicted are predicted to obtain the prediction probability of each sample to be predicted; repeat N rounds, wherein in each round of training and prediction, a different sub-sample set is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted in the current round, until all gray samples are used as samples to be predicted to participate in N-1 predictions, and N-1 prediction probabilities of each gray sample are obtained; Calculate the average predicted probability of each of the gray samples; The gray samples whose average predicted probability exceeds the preset risk threshold are determined to be non-genuine agent complaint work orders.

2. The complaint data identification method based on artificial intelligence according to claim 1, characterized in that: The step of marking the complaint work orders in the complaint data as positive samples or gray samples includes: Obtain the complaint call number in the complaint ticket; Determine whether the complaint call number is a phone number pre-registered by the customer, where the phone number pre-registered by the customer includes the primary account phone number and the emergency contact phone number; If the complaint call number is not the phone number pre-registered by the customer, then obtaining the number of complaints of different complaint work orders corresponding to the complaint call number; According to the number of complaints and preset data, the complaint work order is marked as a positive sample or a negative sample.

3. The complaint data identification method based on artificial intelligence as claimed in claim 2, characterized in that: The step of marking the complaint work order as a positive sample or a negative sample according to the number of complaints and preset data includes: If the number of complaints is greater than or equal to the preset number, the complaint work order is marked as the positive sample; If the number of complaints is less than the preset number, the complaint work order is marked as the gray sample.

4. The complaint data identification method based on artificial intelligence according to claim 1, characterized in that: The step of inputting the negative sample and the positive sample into a prediction model for training to obtain a trained prediction model includes: Using a pre-trained large language model to extract features from the customer complaint speech in the negative sample and the positive sample to obtain first feature information; Input the first feature information, the negative sample, and the positive sample as a training data set into the prediction model for training to obtain a trained prediction model; Obtaining a pre-calibrated validation data set, and using the validation data set to evaluate the trained prediction model to obtain an evaluation result; If the evaluation result meets the expected result, the trained prediction model is used as the trained pre-trained model.

5. The complaint data identification method based on artificial intelligence according to claim 4, characterized in that: The first characteristic information includes basic information of the customer, complaint points, complaint reasons, threatening remarks and complaint requirements.

6. The complaint data identification method based on artificial intelligence according to claim 1, characterized in that: The step of predicting the gray samples in the remaining N-1 samples to be predicted based on the trained prediction model to obtain the prediction probability corresponding to each gray sample includes: Using the large prediction model to extract features from the customer complaint speech of the sample to be predicted, to obtain second feature information; The second feature information is input into the trained prediction model to obtain the prediction probability of each sample to be predicted.

7. The complaint data identification method based on artificial intelligence according to claim 1, characterized in that: The method further comprises: If the complaint work order is marked as the gray sample, then the pre-trained large language model is used to extract features of the customer complaint voice in the gray sample to obtain third feature information; Comparing the third characteristic information with the preset characteristic information to obtain a comparison result; According to the comparison result, it is determined whether the gray sample is a non-genuine agent complaint work order.

8. A complaint data recognition device based on artificial intelligence, characterized in that: include: An acquisition module is used to acquire complaint data, and mark complaint work orders in the complaint data as positive samples or gray samples, wherein the positive samples represent non-real agent complaint work orders, and the gray samples represent agent complaint work orders to be confirmed; A division module is used to divide the gray sample into N sub-sample sets, select one of the sub-sample sets as a negative sample, and select the gray samples in the remaining N-1 sub-sample sets as samples to be predicted; A training module, used for inputting the negative samples and the positive samples into a prediction model for training to obtain a trained prediction model; A prediction module is used to predict the samples to be predicted based on the trained prediction model to obtain the prediction probability of each sample to be predicted; repeat N rounds, wherein in each round of training and prediction, a different sub-sample set is selected as a negative sample, and the gray samples in the remaining N-1 sub-sample sets are used as samples to be predicted in the current round, until all gray samples are used as samples to be predicted to participate in N-1 predictions, and N-1 prediction probabilities of each gray sample are obtained; A calculation module, used for calculating the average prediction probability of each of the gray samples; The first determination module is used to determine the gray samples whose average predicted probability exceeds a preset risk threshold as non-genuine agent complaint work orders.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the artificial intelligence-based complaint data identification method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the artificial intelligence-based complaint data identification method as described in any one of claims 1 to 7 is implemented.

Images