Method for testing verbal skill recommendation system and computer readable storage medium
By using templated requests and multiple vector models to calculate similarity in the speech recommendation system, the speech recommendation system is automated to solve the problems of inefficient testing, strong subjectivity and unclear focus in the existing technology, and realizes that Fast and accurate version evaluation and priority screening.
Patent Information
- Application Number
- CN202411074449.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-20
AI Technical Summary
The prior art has problems such as inefficiency, strong subjectivity and unclear focus when testing speech recommendation systems, which cannot meet the needs of rapid iteration and efficient evaluation of the rationality trends of different versions.
By formulating request templates and designing test data, the similarity between expected speech and actual speech is calculated using cosine similarity threshold, the weighted calculation is performed using TF-IDF, Word2Vec and BERT models, batch requests and result collection is automated, and the similarity is based on the similarity Sort and mark results to enable automated testing and priority filtering.
It significantly improves the testing efficiency, reduces the subjectivity of manual judgment, can automatically identify high-priority issues, and improves the efficiency and accuracy of version testing.
Smart Images
Figure CN119166494A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of system testing, and in particular to a method and a computer-readable storage medium for testing a speech recommendation system. Background Art
[0002] The speech recommendation system is an application of AI technology in customer service. It can intelligently recommend appropriate dialogue strategies based on scenarios and user needs. For example, in sales scenarios, the model can provide the best sales pitch; in customer service, it can guide how to effectively solve customer problems and improve satisfaction. The system learns a large amount of dialogue data, understands the context, and ensures that the recommended speech is both professional and humane, thereby improving communication efficiency and effectiveness.
[0003] The method of testing the recommendation system is to simulate user scenarios, input various questions, and see whether the words returned by the model are appropriate. Due to the characteristics of AI technology, the words recommended for the same question at different times are not exactly the same, and it is impossible to use traditional methods such as exact matching and fuzzy matching to judge the accuracy of the results.
[0004] The conventional method is to manually input various questions and then manually judge whether the recommended words are reasonable one by one. This has the following shortcomings:
[0005] 1. Inefficient testing: The recommendation system requires frequent tuning and has high requirements for regression testing. Manual comparison is inefficient and cannot meet the requirements for rapid iteration.
[0006] 2. Subjective: Manual judgment of whether the recommended words are reasonable is highly subjective, and it is difficult to longitudinally evaluate the rationality trend of different versions of the word recommendation system;
[0007] 3. Unclear focus: It is difficult to confirm the focus of the test by manually comparing one by one. When testing resources are scarce, high-priority issues cannot be identified and confirmed in priority. Summary of the invention
[0008] The purpose of the present invention is to provide a method and a computer-readable storage medium for testing a speech recommendation system in view of the deficiencies in the prior art.
[0009] To achieve the above object, in a first aspect, the present invention provides a method for testing a speech recommendation system, comprising:
[0010] Formulate a request template based on the input parameters of the recommended interface of the tested speech, which will be used for subsequent batch requests, and prepare the designed test data according to the request template, and set the relevant environment and configuration to complete the test data preparation;
[0011] Determine the requested document path and cosine similarity threshold, and submit the request;
[0012] Parse the test data in the uploaded request document, convert it into a request message for the speech recommendation interface, collect the actual speech in the response message, and then calculate the cosine similarity between the expected speech and the actual speech;
[0013] Display the cosine similarity between the requested question and the actual speech, as well as the expected speech and the actual speech. If the cosine similarity between the expected speech and the actual speech is higher than the cosine similarity threshold, the output test result is qualified, otherwise the output test result is unqualified.
[0014] Furthermore, the request template is in Excel spreadsheet format, and all variable fields in the input parameters of the tested speech recommendation interface are treated as a column, and an expected speech column is added. The fixed field values in the input parameters of the tested speech recommendation interface are fixedly assigned when subsequent requests are sent.
[0015] Furthermore, the request document and the cosine similarity threshold are uploaded via an http post request.
[0016] Furthermore, the method of parsing the test data in the uploaded request document is as follows:
[0017] The data in the request document parsed by openpyxl is a python list object. Each list object element is a dict object of key:value, where the key is the table title and the value is the field value.
[0018] Furthermore, the specific method of calculating the cosine similarity between the expected speech and the actual speech is as follows:
[0019] Based on IF-IDF, the cosine similarity between the expected speech and the actual speech is calculated as the first cosine similarity;
[0020] Based on the Word2vec model, the cosine similarity between the expected speech and the actual speech is calculated as the second cosine similarity;
[0021] Based on the BERT model, the cosine similarity between the expected speech and the actual speech is calculated as the third cosine similarity;
[0022] A weighted calculation is performed on the first cosine similarity, the second cosine similarity and the third cosine similarity to obtain a final cosine similarity.
[0023] Furthermore, the final cosine similarity calculated is:
[0024] total_similarity=(tfidf+word2vec+bert*2) / 4
[0025] Among them, total_similarity is the final cosine similarity, tfidf is the first cosine similarity, word2vec is the second cosine similarity, and bert is the third cosine similarity.
[0026] Furthermore, it also includes: selecting fields that the tester is concerned about from the response message, and saving them together with the request-related fields and expected speech fields corresponding to the response in a pandas dataFrame structure.
[0027] Furthermore, it also includes: sorting the test results from low to high according to the cosine similarity between the expected speech and the actual speech, and marking the test results below the cosine similarity threshold with a red background color.
[0028] Furthermore, the data stored in the dataFrame structure can be exported in Excel format.
[0029] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.
[0030] Beneficial effects: The main advantages of the present invention include the following aspects:
[0031] 1. Batch request problem, batch collection of recommended words
[0032] Changing the manual question-and-answer mode, sending all questions and collecting all recommended scripts at once, automating the sending of questions and collecting scripts can significantly improve request efficiency and speed up version testing.
[0033] 2. Use multiple vector methods for quantitative analysis to avoid the limitations of a single vector method
[0034] To avoid the instability and limitations of the single vector method, the present invention adopts three vector methods: TF-IDF / Word2Vec / Bert, and weighted calculation as the final cosine similarity;
[0035] 3. Sort the cosine similarity and filter out high-priority issues that require manual confirmation
[0036] Sort the calculated similarity values from low to high, set the threshold reasonably according to the version quality requirements and test resource investment, mark the issues below the threshold and give priority to manual confirmation. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a structural diagram of a method for testing a speech recommendation system according to an embodiment of the present invention. Figure 2Schematic diagram of uploading request documents and inputting cosine similarity threshold. DETAILED DESCRIPTION
[0038] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. The embodiments are implemented based on the technical solution of the present invention. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.
[0039] like Figure 1 As shown, an embodiment of the present invention provides a method for testing a speech recommendation system, including:
[0040] According to the input parameters of the recommended interface for the tested speech, a request template is formulated for subsequent batch requests, and the designed test data is prepared according to the request template, and the relevant environment and configuration are set to complete the test data preparation. In order to facilitate the tester to edit the request data later, the request template is in Excel table format, and all variable fields in the input parameters of the recommended interface for the tested speech are taken as one column, and an expected speech column is added to facilitate subsequent batch comparison. The fixed field values in the input parameters of the recommended interface for the tested speech are fixedly assigned when subsequent requests are sent.
[0041] Determine the requested document path and cosine similarity threshold, and submit the request. For details, see Figure 2 , the request document can be uploaded through the "Select File" button, and the cosine similarity threshold can be directly entered. The request document and cosine similarity threshold are uploaded through an http post request. The specific message is as follows:
[0042] excel_file:(binary)
[0043] Tereshold: 0.75
[0044] Parse the test data in the uploaded request document, convert it into a speech recommendation interface request message, collect the actual speech in the response message, and then calculate the cosine similarity between the expected speech and the actual speech.
[0045] Specifically, the method of parsing the test data in the uploaded request document is as follows:
[0046] The data in the request document parsed by openpyxl is a python list object, and each list object element is a dict object of key:value, where the key is the table title and the value is the field value. The example is as follows:
[0047] [{'customer_chat_id': '01999', 'customer_msg_id': '202402271007', 'employee_id': 23030016, 'employee_name': 'Liu Huahua', 'customer_id': '01000', 'customer_name': None, 'customer_query': 'Said that the function is not available in the region', 'create_time': datetime.datetime(2024, 5, 11, 17, 39, 30), 'customer_phone': None, 'customer_card_num': None, 'excepted_suggested_script': 'Hello, according to regulatory requirements, city commercial banks should be based on serving the local market. Our bank will handle various deposit operations in accordance with national laws, regulations and regulatory requirements, and will no longer provide deposit product services in other regions. The deposit operations you have already handled will not be affected.'},
[0048] ……
[0049] {'customer_chat_id': '35999', 'customer_msg_id': '202402271007', 'employee_id': 23030016, 'employee_name': 'Liu Huahua', 'customer_id': '35000', 'customer_name': None, 'customer_query': 'How to issue the settlement certificates for different types of loans in your bank?', 'create_time': datetime.datetime(2024, 5, 12, 17, 39, 30), 'customer_phone': None, 'customer_card_num': None, 'excepted_suggested_script': 'May I ask which loan you want to consult or apply for?'}]。
[0050] The specific method for calculating the cosine similarity between the expected script and the actual script is as follows:
[0051] First, calculate the cosine similarity between the expected script and the actual script based on TF-IDF as the first cosine similarity. Specifically, first use the jieba technology to segment the expected recommended script and the actual recommended response respectively, and convert the expected recommended script and the actual script into word lists. For example, after segmenting the sentence "May I ask which loan you want to consult or apply for?", it is converted into the following list.
[0052] ['Excuse me', 'what's the name of the loan you want to consult or apply for? ']
[0053] The list of words is then converted into a document-term matrix. The document-term matrix contains the weight of each word in each document. This weight is called the TF-IDF weight, which measures the importance of a word in a document while also taking into account the importance of the word in the entire corpus.
[0054] Matrix example:
[0055] (0,12)0.2773500981126146
[0056] (0,2)0.2773500981126146
[0057] (0,1)0.2773500981126146
[0058] (0,4)0.2773500981126146
[0059] (0,8)0.2773500981126146
[0060] (0,11)0.2773500981126146
[0061] (0,9)0.2773500981126146
[0062] (0,0)0.2773500981126146
[0063] (0,7)0.2773500981126146
[0064] (0,3)0.2773500981126146
[0065] (0,6)0.2773500981126146
[0066] (0,5)0.2773500981126146
[0067] (0,10)0.2773500981126146
[0068] Finally, the cosine_similarity method of the sklearn.metrics.pairwise library is used to calculate the cosine similarity between the expected speech and the actual speech as the first cosine similarity.
[0069] When two speech descriptions are exactly the same, their TF-IDF vectors are the same, so the first cosine similarity can be directly calculated by cosine_similarity to be 1.0 (indicating complete consistency); when two speech descriptions are inconsistent, if some words are exactly the same but in a different order, or the positions of some words are swapped, their TF-IDF vectors may be the same or even similar, so the first cosine similarity can be calculated by cosine_similarity to be a value close to 1 (indicating partial consistency); when two documents have no similarities or similarities, the first cosine similarity will be a very small number (much less than 1.0), indicating complete inconsistency.
[0070] Based on the Word2vec model, the cosine similarity between the expected speech and the actual speech is calculated as the second cosine similarity. Specifically, first use jieba to segment the speech string to obtain a word list, use the pre-trained Word2vec model to obtain the text vector, and use the above cosine_similarity method to obtain the cosine similarity between the expected speech and the actual speech as the second cosine similarity.
[0071] The cosine similarity between the expected speech and the actual speech is calculated based on the BERT model as the third cosine similarity. Specifically, the speech string is segmented and encoded based on the pre-trained model bert-base-chinese, the output embedding of the BERT model is obtained, and the cosine similarity between the expected speech and the response speech embedding vector is calculated as the third cosine similarity.
[0072] The sample code for calculating the third cosine similarity based on bert-base-chinese is as follows:
[0073] def_get_sentence_embedding(text):
[0074] #Load pre-trained model(weights)
[0075] model_path=".. / llm / bert-base-chinese"
[0076] model=BertModel.from_pretrained(model_path)
[0077] tokenizer=BertTokenizer.from_pretrained(model_path)
[0078] input_ids=torch.tensor([tokenizer.encode(text)])#Add batch dimension
[0079] with torch.no_grad():
[0080] outputs = model(input_ids)
[0081] last_hidden_states=outputs[0]#The last hidden-state is the firstelement of the output tuple
[0082] return torch.mean(last_hidden_states,dim=1).squeeze()
[0083] def bert_cosine_similarity(text1,text2):
[0084] tensor1=_get_sentence_embedding(text1)
[0085] tensor2=_get_sentence_embedding(text2)
[0086] return torch.nn.functional.cosine_similarity(tensor1,tensor2,dim=0).item().
[0087] The first cosine similarity, the second cosine similarity and the third cosine similarity are weighted to obtain the final cosine similarity. The weighted calculation method can be adjusted according to different situations, and the preferred method is:
[0088] total_similarity=(tfidf+word2vec+bert*2) / 4
[0089] Among them, total_similarity is the final cosine similarity, tfidf is the first cosine similarity, word2vec is the second cosine similarity, and bert is the third cosine similarity.
[0090] Display the cosine similarity between the requested question and the actual speech, as well as the expected speech and the actual speech. If the cosine similarity between the expected speech and the actual speech is higher than the cosine similarity threshold, the output test result is qualified, otherwise the output test result is unqualified.
[0091] An embodiment of the present invention also includes selecting fields that the tester is concerned about from the response message, and saving them together with the request-related fields and expected speech fields corresponding to the response in a pandas dataFrame structure. The test results can also be sorted from low to high according to the cosine similarity between the expected speech and the actual speech, and the test results below the cosine similarity threshold are marked with a red background. This facilitates the priority confirmation of request-result records with lower similarity, and increases the probability of discovering defects in the speech recommendation system. In order to facilitate the comparison of cosine similarity changes for the same problem across iterative versions, a data export function is also provided, and the data stored in the dataFrame structure supports export in Excel spreadsheet format. An example of an exported Excel spreadsheet is shown in Table 1:
[0092] Table 1:
[0093]
[0094]
[0095] It should be noted that some unimportant fields are deleted from the exported Excel table shown in Table 1, and field explanations are added.
[0096] Based on the above embodiments, those skilled in the art can easily understand that the present invention also provides a computer-readable storage medium storing a computer program, which implements the above method when executed by a processor.
[0097] The above is only a preferred embodiment of the present invention. It should be noted that for ordinary technicians in this technical field, other parts not specifically described belong to the prior art or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.
Claims
1. A method for testing a speech recommendation system, characterized in that: include: Formulate a request template based on the input parameters of the recommended interface of the tested speech, which will be used for subsequent batch requests, and prepare the designed test data according to the request template, and set the relevant environment and configuration to complete the test data preparation; Determine the requested document path and cosine similarity threshold, and submit the request; Parse the test data in the uploaded request document, convert it into a request message for the speech recommendation interface, collect the actual speech in the response message, and then calculate the cosine similarity between the expected speech and the actual speech; Display the cosine similarity between the requested question and the actual speech, as well as the expected speech and the actual speech. If the cosine similarity between the expected speech and the actual speech is higher than the cosine similarity threshold, the output test result is qualified, otherwise the output test result is unqualified.
2. A method for testing a speech recommendation system according to claim 1, characterized in that: The request template is in Excel spreadsheet format, and all variable fields in the input parameters of the tested speech recommendation interface are treated as one column, and an expected speech column is added. The fixed field values in the input parameters of the tested speech recommendation interface are fixedly assigned when subsequent requests are sent.
3. A method for testing a speech recommendation system according to claim 1, characterized in that: The requested document and the cosine similarity threshold are uploaded via an http post request.
4. A method for testing a speech recommendation system according to claim 2, characterized in that: The method of parsing the test data in the uploaded request document is as follows: The data in the request document parsed by openpyxl is a python list object. Each list object element is a dict object of key:value, where the key is the table title and the value is the field value.
5. The method for testing a speech recommendation system according to claim 1, characterized in that: The specific method for calculating the cosine similarity between the expected speech and the actual speech is as follows: Based on IF-IDF, the cosine similarity between the expected speech and the actual speech is calculated as the first cosine similarity; Based on the Word2vec model, the cosine similarity between the expected speech and the actual speech is calculated as the second cosine similarity; Based on the BERT model, the cosine similarity between the expected speech and the actual speech is calculated as the third cosine similarity; A weighted calculation is performed on the first cosine similarity, the second cosine similarity and the third cosine similarity to obtain a final cosine similarity.
6. A method for testing a speech recommendation system according to claim 1, characterized in that: The final cosine similarity calculated is: total_similarity=(tfidf+word2vec+bert*2) / 4 Among them, total_similarity is the final cosine similarity, tfidf is the first cosine similarity, word2vec is the second cosine similarity, and bert is the third cosine similarity.
7. A method for testing a speech recommendation system according to claim 2, characterized in that: Also includes: The fields that the tester is concerned about are selected from the response message, and are saved in a pandas dataFrame structure together with the request-related fields and expected speech fields corresponding to the response.
8. A method for testing a speech recommendation system according to claim 7, characterized in that: Also includes: The test results are sorted from low to high according to the cosine similarity between the expected speech and the actual speech, and the test results below the cosine similarity threshold are marked with a red background color.
9. A method for testing a speech recommendation system according to claim 7, characterized in that: The data stored in the dataFrame structure can be exported in Excel table format.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.