Verification method, device and storage medium for intermediate indicators of speech comparison

Abstract

The present application discloses a method, apparatus, and storage medium for validating intermediate metrics for conversation script comparison. The method includes: during the process of comparing and monitoring the multiple conversation scripts using candidate metrics and final metrics, performing a significance test on the candidate metrics and final metrics of each comparison group respectively, where each comparison group constitutes a set of comparison groups; when the candidate metrics and final metrics of any one comparison group in the set of comparison groups are significant, determining whether the trends of the candidate metrics and final metrics of all comparison groups in the set of comparison groups are consistent; when it is determined that the trends of the candidate metrics and final metrics of all comparison groups in the set of comparison groups are consistent, performing a correlation analysis on the overall improvement ratios of the candidate metrics and final metrics; and when the result of the correlation analysis shows a strong linear correlation between the overall improvement ratios of the candidate metrics and final metrics, validating that the candidate metrics can be used as intermediate metrics for comparing multiple conversation scripts.

Description

Technical Field

[0001] This application relates to the technical field of conversation optimization, and particularly to a method, device, and storage medium for verifying intermediate metrics in conversation comparison. Background Art

[0002] Due to the development of artificial intelligence technology, there are now many voice communication systems where robots communicate with humans, especially in voice customer service, intelligent telemarketing, intelligent debt collection, intelligent speakers, and other voice interaction scenarios with very wide applications. After robots can communicate with humans in voice, it is necessary to optimize the conversation scripts of the robots to improve the conversation effect. During the process of conversation script optimization, due to limited data volume and few positive samples, the comparison time is too long. How to improve the experimental efficiency and reduce the comparison time has become a difficult problem.

[0003] In some scenarios, there are few positive samples for experimental comparison metrics, resulting in a long time required to obtain significant results. To reduce the experimental time and obtain the same conclusion, it is necessary to find intermediate metrics for comparison. It is necessary to verify the usability of the intermediate metrics, that is, to verify whether the intermediate metrics are consistent with the final metric results and can reduce the experimental time, thus avoiding the problem of incorrect results caused by using invalid intermediate metrics. However, there is currently a lack of a verification method that can verify whether the intermediate metrics are available to assist operation personnel in optimizing conversation scripts, reducing the comparison time, and improving the experimental efficiency.

[0004] In view of the above technical problem in the prior art that there is currently a lack of a verification method that can verify whether the intermediate metrics are available to assist operation personnel in optimizing conversation scripts, reducing the comparison time, and improving the experimental efficiency, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of the present disclosure provide a method, device, and storage medium for verifying intermediate metrics in conversation comparison to at least solve the technical problem in the prior art that there is currently a lack of a verification method that can verify whether the intermediate metrics are available to assist operation personnel in optimizing conversation scripts, reducing the comparison time, and improving the experimental efficiency.

[0006] According to one aspect of the embodiments of the present disclosure, a method for verifying an intermediate metric for speech comparison is provided, which is used to verify whether a candidate metric can be used as an intermediate metric for speech comparison, including: during the process of comparing and monitoring the multiple speeches using the candidate metric and the final metric, performing a significance test on the candidate metric and the final metric of each comparison group respectively, where the comparison group includes any two speeches among the multiple speeches, and each comparison group constitutes a set of comparison groups; when the candidate metric and the final metric of any one comparison group in the set of comparison groups are significant, determining whether the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent; when it is determined that the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent, performing a correlation analysis on the overall improvement ratios of the candidate metric and the final metric; and when the result of the correlation analysis shows that the overall improvement ratios of the candidate metric and the final metric have a strong linear correlation, verifying that the candidate metric can be used as an intermediate metric for comparing multiple speeches.

[0007] According to another aspect of the embodiments of the present disclosure, a storage medium is further provided. The storage medium includes a stored program, wherein, when the program runs, the method described in any one of the above is executed by a processor.

[0008] According to another aspect of the embodiments of the present disclosure, a verification device for an intermediate metric for speech comparison is further provided, which is used to verify whether a candidate metric can be used as an intermediate metric for speech comparison, including: a significance test module, configured to perform a significance test on the candidate metric and the final metric of each comparison group respectively during the process of comparing and monitoring the multiple speeches using the candidate metric and the final metric, where the comparison group includes any two speeches among the multiple speeches, and each comparison group constitutes a set of comparison groups; a trend determination module, configured to determine whether the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent when the candidate metric and the final metric of any one comparison group in the set of comparison groups are significant; a first correlation analysis module, configured to perform a correlation analysis on the overall improvement ratios of the candidate metric and the final metric when it is determined that the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent; and a first determination module, configured to verify that the candidate metric can be used as an intermediate metric for comparing multiple speeches when the result of the correlation analysis shows that the overall improvement ratios of the candidate metric and the final metric have a strong linear correlation.

[0009] According to another aspect of the embodiments of the present disclosure, there is also provided a verification device for intermediate metrics in speech comparison, which is used to verify whether a candidate metric can be used as an intermediate metric in speech comparison, including: a processor; and a memory connected to the processor for providing instructions for the processor to perform the following processing steps: during the comparison and monitoring process of the multiple speech using the candidate metric and the final metric, performing a significance test on the candidate metric and the final metric of each comparison group respectively, where the comparison group includes any two of the multiple speech, and each comparison group constitutes a set of comparison groups; when the candidate metric and the final metric of any one comparison group in the set of comparison groups are significant, determining whether the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent; when it is determined that the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent, performing a correlation analysis on the overall improvement ratios of the candidate metric and the final metric; and when the result of the correlation analysis shows a strong linear correlation between the overall improvement ratios of the candidate metric and the final metric, verifying that the candidate metric can be used as an intermediate metric for multiple speech comparisons.

[0010] In the embodiments of the present disclosure, operations such as significance test, trend analysis, and correlation analysis are used to verify whether a candidate metric can be used as an intermediate metric in speech comparison, that is, to verify the usability of the intermediate metric, verify whether the intermediate metric can obtain the same conclusion as the final metric, and can effectively reduce the experiment time and improve the experiment efficiency. Thus, a verification method that can verify the usability of the intermediate metric is provided, which can assist the operation personnel in optimizing the speech, reducing the comparison time, and improving the experiment efficiency. Furthermore, it solves the technical problem in the prior art that there is currently a lack of a verification method that can verify the usability of the intermediate metric to assist the operation personnel in optimizing the speech, reducing the comparison time, and improving the experiment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of this application. The schematic embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0012] Figure 1 is a hardware structure block diagram of a computing device for implementing the method described in Embodiment 1 of the present disclosure;

[0013] Figure 2 is a flowchart of the verification method for the intermediate metric in speech comparison according to the first aspect of Embodiment 1 of the present disclosure;

[0014] Figure 3 is an overall flowchart of the verification method for the intermediate metric in speech comparison according to Embodiment 1 of the present disclosure;

[0015] Figure 4 It is a schematic flowchart of calculating the experimental time ratio according to Embodiment 1 of the present disclosure;

[0016] Figure 5 It is a schematic diagram of the conversation words related to the intelligent voice robot according to Embodiment 1 of the present disclosure;

[0017] Figure 6 It is a schematic flowchart of verifying whether the login rate can be used as an intermediate index according to Embodiment 1 of the present disclosure;

[0018] Figure 7 It is a schematic diagram of the result of ratio test using the conversion rate and the login rate according to Embodiment 1 of the present disclosure;

[0019] Figure 8 It is a schematic flowchart of calculating the experimental time ratio related to the conversion rate and the login rate according to Embodiment 1 of the present disclosure;

[0020] Figure 9 It is a schematic diagram of the significant result of the conversion rate and the login rate according to Embodiment 1 of the present disclosure;

[0021] Figure 10 It is a schematic diagram of the result of weighted summation of the login rate by the sample size according to Embodiment 1 of the present disclosure;

[0022] Figure 11 It is a schematic diagram of the verification device for the intermediate index of the conversation words comparison according to Embodiment 2 of the present disclosure; and

[0023] Figure 12 It is a schematic diagram of the verification device for the intermediate index of the conversation words comparison according to Embodiment 3 of the present disclosure. Detailed implementation manners

[0024] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure.

[0025] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present disclosure are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described here can be implemented in an order different from those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] Embodiment 1

[0027] According to this embodiment, an embodiment of a method for verifying an intermediate index for speech comparison is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0028] The method embodiment provided in this embodiment can be executed on a server or a similar computing device. Figure 1 A hardware structure block diagram of a computing device for implementing a method for verifying an intermediate index for speech comparison is shown. As Figure 1 shown, the computing device may include one or more processors (the processor may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, and a transmission device for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computing device may further include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0029] It should be noted that one or more of the above-mentioned processors and / or other data processing circuits can generally be referred to as "data processing circuits" herein. The data processing circuit can be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit can be a single independent processing module, or be incorporated in whole or in part into any one of other elements in the computing device. As involved in the embodiments of the present disclosure, the data processing circuit is a kind of processor control (such as the selection of a variable resistor terminal path connected to an interface).

[0030] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage devices corresponding to the verification method of the intermediate index of the conversation comparison in the embodiments of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, to implement the verification method of the intermediate index of the conversation comparison of the above application program. The memory can include high-speed random access memory, and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory can further include memories remotely located relative to the processor, and these remote memories can be connected to the computing device through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0031] The transmission device is used to receive or send data via a network. Specific examples of the above network can include the wireless network provided by the communication provider of the computing device. In one instance, the transmission device includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables the user to interact with the user interface of the computing device.

[0033] It should be noted here that in some alternative embodiments, the above Figure 1 shown computing device can include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. It should be pointed out that Figure 1 is only an example of a specific specific instance and is intended to show the types of components that can exist in the above computing device.

[0034] Under the above operating environment, according to the first aspect of this embodiment, a method for verifying intermediate metrics for conversation comparison is provided, which is used to verify whether a candidate metric can be used as an intermediate metric for conversation comparison. Figure 2 The flowchart of this method is shown. Refer to Figure 2 As shown, this method includes:

[0035] S202: During the process of comparing and monitoring the multiple conversations using the candidate metric and the final metric, perform a significance test on the candidate metric and the final metric of each comparison group respectively, where a comparison group includes any two conversations among the multiple conversations, and each comparison group constitutes a set of comparison groups;

[0036] S204: When the candidate metric and the final metric of any one comparison group in the set of comparison groups are significant, determine whether the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent;

[0037] S206: When it is determined that the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent, perform a correlation analysis on the overall improvement ratios of the candidate metric and the final metric; and

[0038] S208: When the result of the correlation analysis shows a strong linear correlation between the overall improvement ratios of the candidate metric and the final metric, verify that the candidate metric can be used as an intermediate metric for comparing multiple conversations.

[0039] As described in the background art, in some scenarios, the number of positive samples of the experimental comparison metric is small, resulting in a long time required to obtain a significant result. In order to reduce the experimental time and obtain the same conclusion, an intermediate metric needs to be found for comparison. It is necessary to verify the usability of the intermediate metric, that is, to verify whether the intermediate metric is consistent with the final metric result and can reduce the experimental time, so as to avoid the problem of incorrect results caused by using an invalid intermediate metric. However, there is currently a lack of a verification method that can verify whether an intermediate metric is available to assist operation personnel in optimizing conversations, reducing comparison time, and improving experimental efficiency.

[0040] Regarding the technical problems existing in the background art, refer to Figure 2 and Figure 3As shown, after initially selecting a candidate metric in the technical solution of this embodiment, it is necessary to verify that the candidate metric is consistent with the final metric and can effectively reduce the experiment time. Only in this way can the selected candidate metric be usable and become the intermediate metric for conversation comparison. Therefore, it is necessary to monitor the candidate metric and the final metric together for a period of time for multiple conversation comparisons. During the process of using the candidate metric and the final metric to monitor the multiple conversations, it is necessary to perform a significance test on the candidate metric and the final metric of each comparison group respectively, and compare the sizes of the metrics. When the candidate metric and the final metric of any comparison group in the set of comparison groups are significant, it is necessary to determine whether the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent. Trend consistency means that in the same comparison group, the candidate metric and the final metric of object A are both higher than those of object B. If the trends of all comparison groups are consistent, a correlation analysis is performed on the overall improvement ratio of the final metric and the candidate metric. If the overall improvement ratio shows a strong linear correlation, it is verified that the candidate metric can be used as an intermediate metric.

[0041] In this embodiment, through operations such as significance test, trend analysis, and correlation analysis, it is verified whether the candidate metric can be used as the intermediate metric for conversation comparison, that is, the usability of the intermediate metric is verified, whether the intermediate metric can obtain the same conclusion as the final metric, and it can effectively reduce the experiment time and improve the experiment efficiency. Thus, a verification method that can verify whether the intermediate metric is usable is provided, which can assist the operation personnel in optimizing the conversation, reducing the comparison time, and improving the experiment efficiency. Furthermore, it solves the technical problem in the prior art that there is currently a lack of a verification method that can verify whether the intermediate metric is usable to assist the operation personnel in optimizing the conversation, reducing the comparison time, and improving the experiment efficiency.

[0042] Optionally, the method further includes: verifying whether using the intermediate metric to monitor multiple conversations can reduce the comparison time. Specifically, after verifying that the candidate metric can be used as the intermediate metric, it is also necessary to verify whether the intermediate metric can effectively reduce the experiment time. In this way, the problem of incorrect results caused by using an invalid intermediate metric is effectively avoided.

[0043] Optionally, the operation of verifying whether using the intermediate metric to monitor multiple conversations can reduce the comparison time includes: performing a significance test on the candidate metric and the final metric of each comparison group in the set of comparison groups respectively, determining the number of days when the intermediate metrics of all comparison groups in the set of comparison groups reach significance and the number of days when the final metrics of all comparison groups in the set of comparison groups reach significance; and determining whether using the intermediate metric to monitor multiple conversations can reduce the comparison time according to the number of days when the intermediate metric reaches significance and the number of days when the final metric reaches significance.

[0044] Specifically, referring to Figure 4As shown, the final indicators and intermediate indicators of the daily cumulative data of each comparison group are respectively subjected to a significance test to obtain the number of days when the intermediate indicators and final indicators of all comparison groups reach significance. Then, based on the number of days when the intermediate indicators and final indicators of all comparison groups reach significance, it is determined whether using the intermediate indicators for the comparative monitoring of multiple conversation scripts can reduce the comparison time.

[0045] In a preferred embodiment, the operation of determining whether using the intermediate indicators for the comparative monitoring of multiple conversation scripts can reduce the comparison time according to the number of days when the intermediate indicators reach significance and the number of days when the final indicators reach significance includes: determining the average number of days in advance when the intermediate indicators of all comparison groups in the comparison group set reach significance according to the number of days when the intermediate indicators reach significance and the number of days when the final indicators reach significance; determining the average number of significant days when the final indicators of all comparison groups in the comparison group set reach significance according to the number of days when the final indicators reach significance; and comparing the average number of days in advance and the average number of significant days, and determining whether using the intermediate indicators for the comparative monitoring of multiple conversation scripts can reduce the comparison time according to the comparison result.

[0046] Specifically, referring to Figure 4 As shown, according to the number of days when the intermediate indicators and final indicators of all comparison groups reach significance, calculate the number of days in advance when the intermediate indicators of all comparison groups reach significance. If the intermediate indicators reach significance later than the final indicators, the number of days in advance is 0. Then, sum the number of days in advance of the comparison groups that reach significance weighted by the sample size of the comparison groups to obtain the average number of days in advance. The comparison groups whose indicators do not reach significance are not included. Further, sum the number of days when the final indicators reach significance weighted by the sample size of the comparison groups to obtain the average number of significant days. Divide the average number of days in advance by the average number of significant days to obtain the proportion of the experimental time saved. For example, when the experimental time proportion is greater than 1, it is proved that determining to use the intermediate indicators for the comparative monitoring of multiple conversation scripts can reduce the comparison time.

[0047] Thus, by means of the significance test, it can be accurately determined whether using the intermediate indicators for the comparative monitoring of multiple conversation scripts can reduce the comparison time, and the specific proportion of the experimental time saved is given, which assists in the time estimation of subsequent experiments and the evaluation of the importance of intermediate indicators.

[0048] Optionally, before the operation of determining whether the trends of the candidate indicators and the final indicators of all comparison groups in the comparison group set are consistent, it also includes: determining whether the candidate indicators and the final indicators of all comparison groups in the comparison group set are not significant according to the results of the significance test. Specifically, referring to Figure 3 As shown, after respectively performing a significance test on the candidate indicators and the final indicators of each comparison group, it is necessary to determine whether the candidate indicators and the final indicators of all comparison groups in the comparison group set are not significant according to the results of the significance test.

[0049] Optionally, the method further includes: when it is determined that the candidate indicators and the final indicators of all the comparison groups in the set of comparison groups are not significant, continue to use the candidate indicators and the final indicators respectively to monitor the comparison of multiple conversation scripts. Specifically, referring to Figure 3 as shown, if the candidate indicators and the final indicators of all the comparison groups are not significant, continue to perform comparison monitoring for a period of time.

[0050] Optionally, the method further includes: when it is determined that the trends of the candidate indicators and the final indicators of all the comparison groups in the set of comparison groups are inconsistent, determine whether the candidate indicators and the final indicators of the comparison groups with inconsistent trends are both significant; when it is determined that the candidate indicators and the final indicators of the comparison groups with inconsistent trends are not both significant, perform a correlation analysis on the overall improvement ratios of the candidate indicators and the final indicators; and when the result of the correlation analysis shows a strong linear correlation between the overall improvement ratios of the candidate indicators and the final indicators, verify that the candidate indicators can be used as intermediate indicators for the comparison of multiple conversation scripts.

[0051] Specifically, referring to Figure 3 as shown, if the trends of the candidate indicators and the final indicators of all the comparison groups are inconsistent, it is necessary to determine whether the candidate indicators and the final indicators of the comparison groups with inconsistent trends are both significant. When it is determined that the candidate indicators and the final indicators of the comparison groups with inconsistent trends are not both significant, perform a correlation analysis on the overall improvement ratios of the candidate indicators and the final indicators, and when the result of the correlation analysis shows a strong linear correlation between the overall improvement ratios of the candidate indicators and the final indicators, verify that the candidate indicators can be used as intermediate indicators for the comparison of multiple conversation scripts.

[0052] In addition, the overall process of the method for verifying the intermediate indicators for the comparison of conversation scripts proposed by the present invention is as follows:

[0053] 1. After selecting the candidate indicators, it is necessary to verify that the candidate indicators are consistent with the final indicators and can effectively reduce the experiment time. Only in this way can the selected candidate indicators be available and can be used as intermediate indicators. At this time, it is necessary to monitor the comparison of the candidate indicators and the final indicators together for a period of time.

[0054] 2. Perform significance tests on the final indicators and the candidate indicators of different comparison groups respectively, and compare the sizes of the indicators.

[0055] 3. If the candidate indicators and the final indicators of all the comparison groups are not significant, continue to perform comparison monitoring for a period of time, otherwise proceed to the next step.

[0056] 4. Check whether the trends of the candidate indicators and the final indicators of all comparison groups are consistent. Consistent trends mean that in the same comparison group, both the candidate indicator and the final indicator of A are higher than those of B. If the trends of all comparison groups are consistent, proceed to the next step; otherwise, check whether the candidate indicators and the final indicators of the comparison groups with inconsistent trends are both significant. If both are significant, the candidate indicator cannot be used as an intermediate indicator; otherwise, proceed to the next step.

[0057] 5. Conduct a correlation analysis on the overall improvement ratio between the final indicator and the candidate indicator. If the overall improvement ratio shows a strong linear correlation, the candidate indicator can be used as an intermediate indicator; otherwise, it cannot be used as an intermediate indicator.

[0058] 6. After confirming that the candidate indicator can be used as an intermediate indicator, it is necessary to verify whether the intermediate indicator can effectively reduce the experiment time. Conduct a significance test on the final indicator and the intermediate indicator of the daily cumulative data of different comparison groups respectively, and obtain the number of days when the intermediate indicator and the final indicator of all comparison groups reach significance.

[0059] 7. Calculate the number of days in advance when the intermediate indicator of all comparison groups reaches significance. If the intermediate indicator reaches significance later than the final indicator, the number of days in advance is 0.

[0060] 8. Weighted sum the number of days in advance of the comparison groups that reach significance according to the sample size of the comparison groups to obtain the average number of days in advance. Comparison groups where the indicator does not reach significance are not included.

[0061] 9. Weighted sum the number of days when the final indicator reaches significance according to the sample size of the comparison groups to obtain the average number of significant days.

[0062] 10. The ratio of the average number of days in advance to the average number of significant days is the proportion of the experiment time saved.

[0063] Exemplarily, for example, using Figure 5 intelligent voice robots to conduct experiments, and respectively monitor and compare the conversion rate and the login rate for each experimental group. Among them, the conversion rate is the final indicator, and the login rate is the candidate indicator. Referring to Figure 6 as shown, conduct a proportion test on the conversion rate and the login rate for each experimental group respectively, and the results of the proportion test Figure 7As shown. If the conversion rates and login rates of all comparison groups are not significant, continue with comparison monitoring. Otherwise, continue to determine whether the trends of the conversion rate and the login rate are consistent. For example, if the conversion rate of "pause for 2s" is higher than that of "surname identity verification", and the login rate is also higher than that of "surname identity verification", then the trends of the conversion rate and the login rate are consistent. For comparison groups with inconsistent trends, check whether they are all significant. If they are all significant, the login rate cannot be used as an intermediate metric. If they are not significant, it is considered that the inconsistent trend is caused by randomness. If there is no experimental group that cannot be used as an intermediate metric, continue to the next step. After the trends of the conversion rate and the login rate are consistent, it is also necessary to determine whether the ratio of their differences is consistent. Perform correlation analysis with the overall improvement ratio to obtain the correlation coefficient of the overall improvement ratios of the conversion rate and the login rate. If the correlation coefficient is greater than 0.8, it is considered that they have a strong linear correlation, and the login rate can be used as an intermediate metric. Otherwise, the login rate cannot be used as an intermediate metric.

[0064] Further, after confirming that the login rate can be used as an intermediate metric, it is necessary to verify whether its effect can effectively reduce the experiment time. Perform proportion tests on the conversion rates and login rates of the daily cumulative data of different comparison groups respectively to obtain the significance of each day for all comparison groups, as Figure 8 shown. Calculate the number of days by which the login rate becomes significant earlier than the conversion rate, as Figure 9 shown. If the login rate becomes significant later than the conversion rate, the number of days in advance is 0. Refer to Figure 10 shown. Since the sample sizes are different, perform weighted summation with the sample sizes to obtain the average number of days in advance. Similarly, perform weighted summation with the sample sizes for the number of days when the conversion rate becomes significant to obtain the average number of significant days. The average number of days in advance divided by the average number of significant days is the percentage by which the login rate can reduce the experiment time.

[0065] In this embodiment, it can be verified whether the intermediate metric can obtain the same conclusion as the final metric, and it can effectively reduce the experiment time and improve the experiment efficiency. Moreover, a verification method for whether the intermediate metric is available is given, avoiding the problem of incorrect results caused by using an invalid intermediate metric. At the same time, a specific proportion of the saved experiment time is given to assist in the time estimation of subsequent experiments and the evaluation of the importance of the intermediate metric.

[0066] In summary, the verification method for the intermediate metric of the script comparison proposed by the present invention can produce the following effects: 1) Verify the availability of the intermediate metric through methods such as significance testing and correlation analysis. 2) Give the proportion of the saved experiment time, which can be used for the evaluation of the importance of the intermediate metric and as a reference for future experiment time estimation.

[0067] It should be specifically noted that the present solution provides an overall method. Different methods are used in the specific steps, but solving the same problem is also a form of the present solution. For example, when performing a significance test on the intermediate index and the final index, if the proportion test is not used but other methods are adopted, it is also regarded as an infringement of this patent. Moreover, this solution is applicable to all scenarios where intelligent voice requires conversation comparison, including but not limited to intelligent outbound calls, inbound calls, voice collection, return visits, telemarketing, etc., and is also applicable to the scenario of text multi-turn conversation comparison.

[0068] In addition, referring to Figure 1 As shown, according to the second aspect of this embodiment, a storage medium is provided. The storage medium includes a stored program, wherein when the program runs, the method described in any one of the above is executed by a processor.

[0069] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0070] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0071] Embodiment 2

[0072] Figure 11 Fig. shows a verification device 1100 for the intermediate index of conversation comparison according to this embodiment, which is used to verify whether a candidate index can be used as the intermediate index of conversation comparison. The verification device 1100 corresponds to the method described in the first aspect of Embodiment 1. Referring to Figure 11As shown, the verification device 1100 includes: a significance test module 1110, which is used to perform significance tests on the candidate indicators and final indicators of each comparison group respectively during the process of comparing and monitoring the multiple speech scripts using the candidate indicators and final indicators, where a comparison group includes any two speech scripts among the multiple speech scripts, and each comparison group constitutes a comparison group set; a trend determination module 1120, which is used to determine whether the trends of the candidate indicators and final indicators of all comparison groups in the comparison group set are consistent when the candidate indicators and final indicators of any one comparison group in the comparison group set are significant; a first correlation analysis module 1130, which is used to perform a correlation analysis on the overall improvement ratio of the candidate indicators and final indicators when it is determined that the trends of the candidate indicators and final indicators of all comparison groups in the comparison group set are consistent; and a first determination module 1140, which is used to verify that the candidate indicator can be used as an intermediate indicator for comparing multiple speech scripts when the result of the correlation analysis shows a strong linear correlation between the overall improvement ratios of the candidate indicators and final indicators.

[0073] Optionally, the verification device 1100 further includes: a verification module, which is used to verify whether using the intermediate indicator for comparing and monitoring multiple speech scripts can reduce the comparison time.

[0074] Optionally, the verification module includes: a significance test sub-module, which is used to perform significance tests on the candidate indicators and final indicators of each comparison group in the comparison group set respectively, to determine the number of days when the intermediate indicators of all comparison groups in the comparison group set reach significance and the number of days when the final indicators of all comparison groups in the comparison group set reach significance; and a determination sub-module, which is used to determine whether using the intermediate indicator for comparing and monitoring multiple speech scripts can reduce the comparison time according to the number of days when the intermediate indicator reaches significance and the number of days when the final indicator reaches significance.

[0075] Optionally, the determination sub-module includes: a first determination unit, which is used to determine the average number of days in advance when the intermediate indicators of all comparison groups in the comparison group set reach significance according to the number of days when the intermediate indicator reaches significance and the number of days when the final indicator reaches significance; a second determination unit, which is used to determine the average significant number of days when the final indicators of all comparison groups in the comparison group set reach significance according to the number of days when the final indicator reaches significance; and a third determination unit, which is used to compare the average number of days in advance and the average significant number of days, and determine whether using the intermediate indicator for comparing and monitoring multiple speech scripts can reduce the comparison time according to the comparison result.

[0076] Optionally, the verification device 1100 further includes: a first significance determination module, which is used to determine whether the candidate indicators and final indicators of all comparison groups in the comparison group set are not significant according to the result of the significance test before the operation of determining whether the trends of the candidate indicators and final indicators of all comparison groups in the comparison group set are consistent.

[0077] Optionally, the verification device 1100 further includes: a monitoring module, configured to, when it is determined that the candidate metrics and the final metrics of all comparison groups in the comparison group set are not significant, continue to perform comparison monitoring on multiple conversation scripts using the candidate metrics and the final metrics respectively.

[0078] Optionally, the verification device 1100 further includes: a second significance determination module, configured to determine whether the candidate metrics and the final metrics of the comparison groups with inconsistent trends are both significant when it is determined that the trends of the candidate metrics and the final metrics of all comparison groups in the comparison group set are inconsistent; a second correlation analysis module, configured to perform a correlation analysis on the overall improvement ratio of the candidate metrics and the final metrics when it is determined that the candidate metrics and the final metrics of the comparison groups with inconsistent trends are not both significant; and a second determination module, configured to verify that the candidate metrics can be used as intermediate metrics for comparing multiple conversation scripts when the result of the correlation analysis shows a strong linear correlation between the overall improvement ratios of the candidate metrics and the final metrics.

[0079] Thus, according to this embodiment, through operations such as significance testing, trend analysis, and correlation analysis, it is verified whether the candidate metrics can be used as intermediate metrics for conversation script comparison, that is, the usability of the intermediate metrics is verified, whether the intermediate metrics can obtain the same conclusion as the final metrics is verified, and the experiment time can be effectively reduced and the experiment efficiency can be improved. Thus, a verification method capable of verifying the usability of intermediate metrics is provided, which can assist operation personnel in optimizing conversation scripts, reducing comparison time, and improving experiment efficiency. Furthermore, the technical problem in the prior art that there is currently a lack of a verification method capable of verifying the usability of intermediate metrics to assist operation personnel in optimizing conversation scripts, reducing comparison time, and improving experiment efficiency is solved.

[0080] Embodiment 3

[0081] Figure 12 A verification device 1200 for the intermediate metrics of conversation script comparison according to this embodiment is shown, which is used to verify whether the candidate metrics can be used as intermediate metrics for conversation script comparison. The device 1200 corresponds to the method described in the first aspect of Embodiment 1. Refer to Figure 12As shown, the device 1200 includes: a processor 1210; and a memory 1220, connected to the processor 1210, for providing instructions to the processor 1210 to process the following steps: during the process of comparing and monitoring the multiple conversation scripts using the candidate metrics and the final metrics, performing a significance test on the candidate metrics and the final metrics of each comparison group respectively, where a comparison group includes any two conversation scripts among the multiple conversation scripts, and each comparison group constitutes a set of comparison groups; when the candidate metrics and the final metrics of any one comparison group in the set of comparison groups are significant, determining whether the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent; when it is determined that the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent, performing a correlation analysis on the overall improvement ratio of the candidate metrics and the final metrics; and when the result of the correlation analysis shows a strong linear correlation between the overall improvement ratio of the candidate metrics and the final metrics, verifying that the candidate metrics can be used as intermediate metrics for comparing multiple conversation scripts.

[0082] Optionally, the memory 1220 is further configured to provide instructions to the processor 1210 to process the following steps: verifying whether using the intermediate metrics for comparing and monitoring multiple conversation scripts can reduce the comparison time.

[0083] Optionally, the operation of verifying whether using the intermediate metrics for comparing and monitoring multiple conversation scripts can reduce the comparison time includes: performing a significance test on the candidate metrics and the final metrics of each comparison group in the set of comparison groups respectively, determining the number of days when the intermediate metrics of all comparison groups in the set of comparison groups reach significance and the number of days when the final metrics of all comparison groups in the set of comparison groups reach significance; and determining whether using the intermediate metrics for comparing and monitoring multiple conversation scripts can reduce the comparison time according to the number of days when the intermediate metrics reach significance and the number of days when the final metrics reach significance.

[0084] Optionally, the operation of determining whether using the intermediate metrics for comparing and monitoring multiple conversation scripts can reduce the comparison time according to the number of days when the intermediate metrics reach significance and the number of days when the final metrics reach significance includes: determining the average number of days in advance when the intermediate metrics of all comparison groups in the set of comparison groups reach significance according to the number of days when the intermediate metrics reach significance and the number of days when the final metrics reach significance; determining the average number of significant days when the final metrics of all comparison groups in the set of comparison groups reach significance according to the number of days when the final metrics reach significance; and comparing the average number of days in advance and the average number of significant days, and determining whether using the intermediate metrics for comparing and monitoring multiple conversation scripts can reduce the comparison time according to the comparison result.

[0085] Optionally, the memory 1220 is further configured to provide instructions for the processor 1210 to process the following steps: before determining whether the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent, determine whether the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are not significant according to the results of the significance test.

[0086] Optionally, the memory 1220 is further configured to provide instructions for the processor 1210 to process the following steps: in the case where it is determined that the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are not significant, continue to use the candidate metrics and the final metrics respectively for comparison monitoring of multiple conversation scripts.

[0087] Optionally, the memory 1220 is further configured to provide instructions for the processor 1210 to process the following steps: in the case where it is determined that the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are inconsistent, determine whether the candidate metrics and the final metrics of the comparison groups with inconsistent trends are all significant; in the case where the candidate metrics and the final metrics of the comparison groups with inconsistent trends are not all significant, perform a correlation analysis on the overall improvement ratios of the candidate metrics and the final metrics; and in the case where the result of the correlation analysis shows a strong linear correlation between the overall improvement ratios of the candidate metrics and the final metrics, verify that the candidate metrics can be used as intermediate metrics for comparing multiple conversation scripts.

[0088] Thus, according to this embodiment, through operations such as significance test, trend analysis, and correlation analysis, it is verified whether the candidate metrics can be used as intermediate metrics for conversation script comparison, that is, the usability of the intermediate metrics is verified, whether the intermediate metrics can obtain the same conclusion as the final metrics is verified, and the experiment time can be effectively reduced and the experiment efficiency can be improved. Therefore, a verification method capable of verifying the usability of intermediate metrics is provided, which can assist operation personnel in optimizing conversation scripts, reducing comparison time, and improving experiment efficiency. Furthermore, the technical problem in the prior art that there is currently a lack of a verification method capable of verifying the usability of intermediate metrics to assist operation personnel in optimizing conversation scripts, reducing comparison time, and improving experiment efficiency is solved.

[0089] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0090] In the above embodiments of the present invention, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0091] In several embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings, direct couplings, or communication connections shown or discussed with each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0092] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] In addition, in each embodiment of the present invention, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0094] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs and other various media that can store program codes.

[0095] The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A verification method for intermediate indicators in speech comparison, used to verify whether a candidate indicator can be used as an intermediate indicator in speech comparison, characterized in that Including: During the process of comparing and monitoring multiple conversation scripts using the candidate metric and the final metric, performing a significance test on the candidate metric and the final metric of each comparison group respectively, where the comparison group includes any two conversation scripts among the multiple conversation scripts, and all the comparison groups form a comparison group set; When the candidate metric and the final metric of any one comparison group in the comparison group set are significant, determining whether the trends of the candidate metric and the final metric of all the comparison groups in the comparison group set are consistent; When it is determined that the trends of the candidate metric and the final metric of all the comparison groups in the comparison group set are consistent, performing a correlation analysis on the overall improvement ratios of the candidate metric and the final metric; And When the result of the correlation analysis shows that there is a strong linear correlation between the overall improvement ratios of the candidate metric and the final metric, verifying that the candidate metric can be used as an intermediate metric for comparing the multiple conversation scripts; Before the operation of determining whether the trends of the candidate metric and the final metric of all the comparison groups in the comparison group set are consistent, it further includes: according to the result of the significance test, determining whether the candidate metric and the final metric of all the comparison groups in the comparison group set are not significant; it also includes: when it is determined that the candidate metric and the final metric of all the comparison groups in the comparison group set are not significant, continuing to use the candidate metric and the final metric respectively for comparing and monitoring the multiple conversation scripts.

2. The method according to claim 1, wherein After the operation of verifying that the candidate metric can be used as an intermediate metric for comparing the multiple conversation scripts, it further includes: verifying whether using the intermediate metric for comparing and monitoring the multiple conversation scripts can reduce the comparison time.

3. The method according to claim 2, characterized in that, The operation of verifying whether using the intermediate metric for comparing and monitoring the multiple conversation scripts can reduce the comparison time includes: Performing a significance test on the candidate metric and the final metric of each comparison group in the comparison group set respectively, to determine the number of days when the intermediate metric of all the comparison groups in the comparison group set reaches significance and the number of days when the final metric of all the comparison groups in the comparison group set reaches significance; and Based on the number of days when the intermediate metric reaches significance and the number of days when the final metric reaches significance, determining whether using the intermediate metric for comparing and monitoring the multiple conversation scripts can reduce the comparison time.

4. The method according to claim 3, characterized in that, The operation of determining whether using the intermediate metric for comparing and monitoring the multiple conversation scripts can reduce the comparison time based on the number of days when the intermediate metric reaches significance and the number of days when the final metric reaches significance includes: Based on the number of days when the intermediate metric reaches significance and the number of days when the final metric reaches significance, determining the average number of days in advance when the intermediate metric of all the comparison groups in the comparison group set reaches significance; Based on the number of days when the final metric reaches significance, determining the average number of significant days when the final metric of all the comparison groups in the comparison group set reaches significance; and Compare the average number of days in advance and the average number of significant days, and determine whether using the intermediate metric for comparative monitoring of the multiple sales scripts can reduce the comparison time according to the comparison result.

5. The method according to claim 1, wherein Further included are: In the case where it is determined that the trends of the candidate metric and the final metric of all comparison groups in the comparison group set are inconsistent, determine whether both the candidate metric and the final metric of the comparison groups with inconsistent trends are significant; In the case where it is determined that not both the candidate metric and the final metric of the comparison groups with inconsistent trends are significant, perform a correlation analysis on the overall improvement ratios of the candidate metric and the final metric; And In the case where the result of the correlation analysis shows a strong linear correlation between the overall improvement ratios of the candidate metric and the final metric, verify that the candidate metric can be used as an intermediate metric for comparison of the multiple sales scripts.

6. A storage medium, characterized in that, The storage medium includes a stored program, wherein the method according to any one of claims 1 to 5 is executed by a processor when the program runs.

7. An apparatus for verifying an intermediate metric for speech comparison, which is used to verify whether a candidate metric can be used as an intermediate metric for speech comparison, characterized in that Included are: A significance test module for performing a significance test on the candidate metric and the final metric of each comparison group respectively during the process of using the candidate metric and the final metric for comparative monitoring of multiple sales scripts, wherein each comparison group includes any two sales scripts among the multiple sales scripts, and all the comparison groups form a comparison group set; A trend determination module for determining whether the trends of the candidate metric and the final metric of all comparison groups in the comparison group set are consistent in the case where the candidate metric and the final metric of any one comparison group in the comparison group set are significant; A first correlation analysis module for performing a correlation analysis on the overall improvement ratios of the candidate metric and the final metric in the case where it is determined that the trends of the candidate metric and the final metric of all comparison groups in the comparison group set are consistent; And A first determination module for verifying that the candidate metric can be used as an intermediate metric for comparison of the multiple sales scripts in the case where the result of the correlation analysis shows a strong linear correlation between the overall improvement ratios of the candidate metric and the final metric; Before the operation of determining whether the trends of the candidate metric and the final metric of all comparison groups in the comparison group set are consistent, further included are: determining whether both the candidate metric and the final metric of all comparison groups in the comparison group set are not significant according to the result of the significance test; further included are: in the case where it is determined that both the candidate metric and the final metric of all comparison groups in the comparison group set are not significant, continue to use the candidate metric and the final metric respectively for comparative monitoring of the multiple sales scripts.

8. An apparatus for validating an intermediate metric for conversation comparison, which is used to validate whether a candidate metric can be used as an intermediate metric for conversation comparison, characterized in that, Included are: A processor; And A memory, connected to the processor, for providing instructions for the processor to perform the following processing steps: In the process of comparing and monitoring multiple sales pitches using candidate metrics and final metrics, perform a significance test on the candidate metrics and the final metrics of each comparison group respectively, where the comparison group includes any two sales pitches among the multiple sales pitches, and the respective comparison groups form a set of comparison groups; When the candidate metrics and the final metrics of any one comparison group in the set of comparison groups are significant, determine whether the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent; When it is determined that the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent, perform a correlation analysis on the overall improvement ratios of the candidate metrics and the final metrics; and When the result of the correlation analysis shows that there is a strong linear correlation between the overall improvement ratios of the candidate metrics and the final metrics, verify that the candidate metrics can be used as intermediate metrics for comparing the multiple sales pitches; Before the operation of determining whether the trends of the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are consistent, it also includes: according to the result of the significance test, determine whether the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are not significant; it also includes: when it is determined that the candidate metrics and the final metrics of all comparison groups in the set of comparison groups are not significant, continue to use the candidate metrics and the final metrics respectively for comparing and monitoring the multiple sales pitches.

Images