A voiceprint recognition evaluation method and system based on multi-dimensional index and probability distribution comparison

Abstract

The application discloses a kind of based on multi-dimensional index and probability distribution contrast's voiceprint recognition evaluation method and system, constructs two-dimensional confusion matrix based on labeled data and the sample count of four quadrants is counted, to cross orthogonally the subjective feeling of user with the objective determination result of voiceprint model, breaks the limitation of traditional only relying on single accuracy;Further, by calculating misleading rate and bottom rate, the remedial ability of reply strategy when model misjudgment and the error proportion when model is correct are quantified, directly solve the problem that existing technology cannot identify accurate rate and strategy misleading cover real level are high;By calculating voiceprint scene index (i.e. the ratio of control rate and accuracy), the comprehensive evaluation of system overall performance is realized, so that the evaluation result can not only reflect the precision of model itself, but also reveal the real user experience and strategy optimization space of system in complex interactive scene.

Description

Technical Field

[0001] This invention relates to the field of voiceprint recognition technology, specifically to a voiceprint recognition evaluation method and system based on the comparison of multidimensional indicators and probability distributions. Background Technology

[0002] Voiceprint recognition technology is widely used in intelligent dialogue systems, but existing evaluation methods mainly focus on single-dimensional accuracy statistics or general model testing. For example, existing technologies (such as patents CN122090522A and CN122135743A) often utilize multimodal features or specific acoustic models (such as ECAPA-TDNN) to improve recognition accuracy, but their evaluation systems rely solely on recognition accuracy and do not construct evaluation indicators that cover the cross-relationship between user experience and model judgment. Another example is patent CN122176480A, which calculates accuracy by counting the number of correct / incorrect results, but this scheme is only applicable to general image recognition, has a single indicator, and cannot reflect the comprehensive needs of subjective experience in the voiceprint field.

[0003] The core flaw of existing technologies lies in the fact that they only use the accuracy rate under a fixed threshold as an evaluation indicator. When the model confidence fluctuates around the decision threshold, the accuracy rate may still be artificially high even if the actual discrimination ability is weak. It is impossible to judge the true discrimination ability of the model from the global probability distribution. At the same time, there is a lack of device-level differentiated diagnostic methods, making it difficult to discover the experience problems of long-tail devices. Summary of the Invention

[0004] This invention aims to provide a voiceprint recognition evaluation method and system based on the comparison of multi-dimensional indicators and probability distributions, which realizes a comprehensive evaluation of the overall performance of the system. This allows the evaluation results to not only reflect the accuracy of the model itself, but also reveal the real user experience and strategy optimization space of the system in complex interaction scenarios.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a voiceprint recognition evaluation method based on the comparison of multi-dimensional indicators and probability distributions, comprising: S1: Obtain the labeled evaluation record data, which includes the voiceprint model judgment result, confidence score, and true correctness label; S2: Based on the voiceprint model judgment result and the true correctness label, construct a two-dimensional confusion matrix and count the sample counts in four quadrants. The four quadrants are the first quadrant where the voiceprint model is correct and the user's perception is correct, the second quadrant where the voiceprint model is correct but the user's perception is incorrect, the third quadrant where the voiceprint model is incorrect but the user's perception is correct, and the fourth quadrant where the voiceprint model is incorrect and the user's perception is incorrect. S3: Based on the sample counts in the four quadrants, calculate the voiceprint model accuracy and control rate, wherein the voiceprint model accuracy is the ratio of the sum of the sample counts in the first quadrant and the second quadrant to the total number of samples, and the control rate is the ratio of the sum of the sample counts in the first quadrant and the third quadrant to the total number of samples. S4: Based on the accuracy and control rate of the voiceprint model, calculate the misleading rate and the fallback rate, wherein the misleading rate uses the correct samples of the voiceprint model as the denominator to measure the proportion of errors in the response strategy, and the fallback rate uses the incorrect samples of the voiceprint model as the denominator to measure the proportion of remedies in the response strategy. S5: Based on the misleading rate and the fallback rate, and the accuracy of the voiceprint model, calculate the voiceprint scene index. The voiceprint scene index is defined as the ratio of the control rate to the accuracy of the voiceprint model, and is used to characterize the degree to which the system successfully compensates for the deficiencies of the voiceprint model or the existence of misleading effects.

[0006] Preferably, the method further includes S6: extracting the confidence scores from the labeled evaluation record data and constructing an ideal two-component Gaussian mixture distribution based on the central limit theorem; The ideal two-component Gaussian mixture distribution includes registered user components and unregistered user components.

[0007] Preferably, the method further includes S7: fitting the confidence score using the expectation-maximization algorithm to obtain actual distribution parameters, wherein the actual distribution parameters include the weight, mean, and standard deviation of each component.

[0008] Preferably, the method further includes S8: calculating the Wasserstein-1 distance between the actual distribution corresponding to the actual distribution parameters and the ideal two-component Gaussian mixture distribution, wherein the Wasserstein-1 distance is obtained by solving the integral of the quantile function using Monte Carlo sampling combined with the bisection method, and is used to quantify the global discriminative power of the model.

[0009] Preferably, the method further includes S9: grouping the labeled evaluation record data according to the equipment granularity, performing operations S6 to S8 on each equipment group to obtain the Wasserstein-1 distance of each equipment group, and generating an equipment quality ranking list based on the size of the Wasserstein-1 distance of each equipment group.

[0010] Preferably, the voiceprint scene index calculated in S3 satisfies the following relationship: When the voiceprint scene index is less than 1, it indicates that the response strategy is flawed and the misleading effect is dominant. When the voiceprint scene index is greater than 1, it indicates that the system has successfully made up for the deficiencies of the voiceprint model and the fallback effect is dominant.

[0011] Preferably, the formula for calculating the Wasserstein-1 distance is as follows: , Where p is the actual distribution and q is the ideal distribution. and These are the quantile functions for the actual distribution and the ideal distribution, respectively.

[0012] Preferably, when grouping the labeled evaluation record data by equipment granularity, equipment groups with a sample size less than a preset minimum value N_min are filtered out. The preset minimum value N_min is 30, to ensure that the equipment quality ranking list only includes statistically significant equipment.

[0013] Preferably, the sample counts in the four quadrants correspond to the symbols respectively. and The formula for calculating the misleading rate is as follows: The formula for calculating the coverage rate is as follows: .

[0014] On the other hand, this invention proposes a voiceprint recognition evaluation system based on the comparison of multidimensional indicators and probability distributions, comprising: The data acquisition module is used to acquire the labeled evaluation record data; The matrix construction module is used to construct a two-dimensional confusion matrix based on the labeled evaluation record data and count the sample counts in the four quadrants. The indicator calculation module is used to calculate the voiceprint model accuracy, control rate, misleading rate, fallback rate and voiceprint scene index sequentially based on the sample counts in the four quadrants. The distribution evaluation module is used to construct an ideal two-component Gaussian mixture distribution, fit the confidence score using the expectation-maximization algorithm, and calculate the Wasserstein-1 distance between the actual distribution and the ideal distribution. The ranking output module is used to sort the Wasserstein-1 distance of each equipment group according to the equipment granularity and generate a ranking list of equipment quality.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention constructs a two-dimensional confusion matrix based on labeled data and counts samples in four quadrants, thereby orthogonally intersecting the objective judgment results of the voiceprint model with the user's subjective experience, breaking the limitations of traditional reliance solely on accuracy. Furthermore, by calculating the misleading rate and the fallback rate, it quantifies the remedial capability of the response strategy when the model misjudges and the error ratio when the model is correct, directly solving the problem that existing technologies cannot identify inflated accuracy and the masking of true performance by misleading strategies. By calculating the voiceprint scene index (i.e., the ratio of control rate to accuracy rate), it achieves a comprehensive evaluation of the overall system performance, enabling the evaluation results to not only reflect the accuracy of the model itself but also reveal the real user experience and strategy optimization space of the system in complex interaction scenarios. Attached Figure Description

[0016] Figure 1 This is a flowchart of the voiceprint recognition evaluation method based on the comparison of multidimensional indicators and probability distributions according to the present invention. Figure 2 This is a block diagram of the voiceprint recognition evaluation system based on the comparison of multidimensional indicators and probability distributions according to the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a specific posture. If the specific posture changes, the directional indicators will also change accordingly.

[0019] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the use of "and / or" or "and / or" throughout the text includes three parallel solutions. For example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0020] Please see Figure 1 This invention provides a voiceprint recognition evaluation method based on the comparison of multidimensional indicators and probability distributions, comprising: S1: Obtain labeled evaluation record data, which includes voiceprint model judgment results, confidence scores, and true / false accuracy annotations. By obtaining labeled evaluation record data containing voiceprint model judgment results, confidence scores, and true / false accuracy annotations, comprehensive and accurate basic data support is provided for subsequent construction of multidimensional confusion matrix and probability distribution comparison analysis.

[0021] S2: Based on the voiceprint model judgment results and the accuracy labeling, a two-dimensional confusion matrix is ​​constructed, and the sample counts in the four quadrants are counted. The four quadrants are the first quadrant where the voiceprint model is correct and the user's perception is correct, the second quadrant where the voiceprint model is correct but the user's perception is incorrect, the third quadrant where the voiceprint model is incorrect but the user's perception is correct, and the fourth quadrant where the voiceprint model is incorrect and the user's perception is incorrect. By constructing a two-dimensional confusion matrix that includes the cross-relationship between model judgment and user perception and counting the samples in the four quadrants, orthogonal decoupling analysis of objective recognition results and subjective experience is achieved, effectively revealing abnormal scenarios that traditional single accuracy indicators cannot detect, such as models being correct but users being dissatisfied or models being incorrect but users being satisfied.

[0022] S3: Based on the sample counts in the four quadrants, calculate the voiceprint model accuracy and control rate, wherein the voiceprint model accuracy is the ratio of the sum of the sample counts in the first and second quadrants to the total sample count, and the control rate is the ratio of the sum of the sample counts in the first and third quadrants to the total sample count; the calculated voiceprint scene index satisfies the following relationship: When the voiceprint scene index is less than 1, it indicates that the response strategy is flawed and the misleading effect is dominant. When the voiceprint scene index is greater than 1, it indicates that the system has successfully made up for the deficiencies of the voiceprint model and the fallback effect is dominant.

[0023] By calculating the accuracy and control rate of the voiceprint model and constructing a voiceprint scene index, the remedial capability and misleading risk of the response strategy can be dynamically quantified. This accurately identifies whether the system is covering up model defects or effectively covering up, thus upgrading the single model performance evaluation to a comprehensive diagnosis that includes strategy optimization dimensions. This provides a clear quantitative basis for judging the system's real user experience and guiding strategy iteration.

[0024] S4: Based on the accuracy and control rate of the voiceprint model, calculate the misleading rate and the fallback rate, wherein the misleading rate uses the correct samples of the voiceprint model as the denominator to measure the proportion of errors in the response strategy, and the fallback rate uses the incorrect samples of the voiceprint model as the denominator to measure the proportion of remedies in the response strategy. S5: Based on the misleading rate and the fallback rate, and the accuracy of the voiceprint model, calculate the voiceprint scene index. The voiceprint scene index is defined as the ratio of the control rate to the accuracy of the voiceprint model, and is used to characterize the degree to which the system successfully compensates for the deficiencies of the voiceprint model or the existence of misleading effects.

[0025] The above method also includes S6: extracting the confidence scores from the labeled evaluation record data and constructing an ideal two-component Gaussian mixture distribution based on the central limit theorem; the ideal two-component Gaussian mixture distribution includes registered user components and non-registered user components.

[0026] The above method also includes S7: fitting the confidence score using the expectation-maximization algorithm to obtain the actual distribution parameters, wherein the actual distribution parameters include the weight, mean and standard deviation of each component.

[0027] The above method also includes S8: calculating the Wasserstein-1 distance between the actual distribution corresponding to the actual distribution parameters and the ideal two-component Gaussian mixture distribution. The Wasserstein-1 distance is obtained by solving the integral of the quantile function using Monte Carlo sampling combined with the bisection method, and is used to quantify the global discriminative power of the model.

[0028] The above method also includes S9: grouping the labeled evaluation record data according to the equipment granularity, performing operations S6 to S8 on each equipment group to obtain the Wasserstein-1 distance of each equipment group, and generating an equipment quality ranking list based on the size of the Wasserstein-1 distance of each equipment group.

[0029] Specifically, when grouping the labeled evaluation record data by equipment granularity, equipment groups with a sample size less than a preset minimum value N_min are filtered out. The preset minimum value N_min is 30, to ensure that the equipment quality ranking list only includes statistically significant equipment.

[0030] By introducing an ideal two-component Gaussian mixture distribution model based on the central limit theorem, and using the expectation-maximization algorithm to fit the actual confidence distribution and calculate the Wasserstein-1 distance, the limitations of traditional single-threshold evaluation can be overcome. This allows for precise quantification of the inherent discriminative power and robustness of the voiceprint model from the perspective of global probability distribution morphology. Simultaneously, by combining device-level grouping statistics and a minimum sample size filtering mechanism, not only are statistical biases caused by small samples eliminated, but significant device quality rankings are also generated. This effectively identifies the experience bottlenecks of long-tail devices, providing fine-grained diagnostic basis for differentiated optimization and hardware adaptation.

[0031] On the other hand, this invention proposes a voiceprint recognition evaluation system based on the comparison of multidimensional indicators and probability distributions, such as... Figure 2 As shown, it includes: The data acquisition module is used to acquire the labeled evaluation record data; The matrix construction module is used to construct a two-dimensional confusion matrix based on the labeled evaluation record data and count the sample counts in the four quadrants. The indicator calculation module is used to calculate the voiceprint model accuracy, control rate, misleading rate, fallback rate and voiceprint scene index sequentially based on the sample counts in the four quadrants. The distribution evaluation module is used to construct an ideal two-component Gaussian mixture distribution, fit the confidence score using the expectation-maximization algorithm, and calculate the Wasserstein-1 distance between the actual distribution and the ideal distribution. The ranking output module is used to sort the Wasserstein-1 distance of each equipment group according to the equipment granularity and generate a ranking list of equipment quality.

[0032] Furthermore, each module in the above system is also used to implement other steps of the aforementioned voiceprint recognition evaluation method based on the comparison of multidimensional indicators and probability distributions.

[0033] Furthermore, this invention is applied to the evaluation of intelligent dialogue systems with voiceprint recognition capabilities. In such systems, the voiceprint recognition model outputs an identity determination result and a confidence score for each round of user voice input. To evaluate the actual performance of the voiceprint model, its output determination records need to be systematically evaluated.

[0034] This invention assumes that the input data required for the evaluation already exists, meaning that each judgment record already contains the identity determination result output by the voiceprint model, the confidence score, and the accuracy label of the judgment (i.e., whether the voiceprint model's judgment is correct). This invention does not address how to obtain these labels, but rather focuses on: given the aforementioned labeled evaluation data, how to calculate multi-dimensional evaluation indicators and model quality metrics. Specifically, this method performs calculations from two orthogonal perspectives: First-person perspective (index calculation): The objective accuracy of the voiceprint model is cross-referenced with the user's subjective experience in two dimensions to construct an evaluation matrix and derive an index system from it layer by layer. Second perspective (probability distribution comparison): Based on the central limit theorem, an ideal mixture Gaussian distribution is constructed. The EM algorithm is used to fit the actual distribution from the confidence data. The difference between the two is quantified by the Wasserstein distance and ranked at the device granularity.

[0035] Specifically, the evaluation matrix is ​​organized as follows: Receive labeled evaluation data, where each data point includes: (a) whether the voiceprint model is correct (objective correctness labeling); and (b) whether the user's perception is correct (subjective perception labeling). Cross the two to construct a two-dimensional confusion matrix.

[0036] The two dimensions of this matrix are: Line (Voiceprint Model True Prediction): Whether the identity judgment output by the voiceprint model is consistent with the speaker's true identity, with a value of Correct or Error.

[0037] Column (User's Actual Perception): Whether the user felt that their identity was correctly identified in the current dialogue interaction, with a value of Correct or Error.

[0038] The two-dimensional intersection forms four quadrants and their sample counts: The meanings and optimization directions of each quadrant are as follows: The voiceprint and body feedback are correct, the system is performing optimally, and the current strategy can be maintained.

[0039] The voiceprint is correct but the haptic feedback is incorrect, indicating that the voiceprint model is correctly identified but the response strategy is inappropriate, causing the user to perceive it as incorrect. The logic related to identity perception in the response strategy needs to be optimized.

[0040] If the voiceprint is incorrect but the haptic feedback is correct, it indicates that the voiceprint model misidentified the voiceprint but the response strategy successfully covered the error. The pattern of successful coverage needs to be analyzed for strategy reuse.

[0041] The system performed the worst due to both voiceprint and haptic errors, requiring simultaneous optimization of both the voiceprint model accuracy and the response strategy.

[0042] Specifically, the multi-level indicator system is calculated as follows: Based on the four count values ​​of the two-dimensional confusion matrix, the derived index is calculated layer by layer according to the following four-level derivation chain: First level (basic count): Obtained directly Total number of samples .

[0043] Second layer (directly derived indicators): Voiceprint model accuracy The sum of the two quadrants correctly identified by the voiceprint model is divided by N; control rate. Take the sum of the values ​​in the two quadrants indicating a correct user experience, and divide by N. The numerators of these two quadrants differ only in one quadrant. Includes (The model is correct, but the user experience is wrong) Includes (The model is wrong, but the user experience is correct).

[0044] Third layer (condition-derived indicators): Using the samples with correct voiceprint model results as the denominator, this measures the proportion of errors in the response strategy within that subset; catch-all rate. Using samples with voiceprint model errors as the denominator, the proportion of remedial measures taken by the response strategy in that subset is measured.

[0045] Fourth layer (Comprehensive Index): Voiceprint Scene Index The ratio of control rate to voiceprint model accuracy can also be expanded as and The weighted combination encodes both the misleading effect and the catch-all effect into a single scalar. This four-layer derivation chain allows each indicator to be traced back to its original count, and the direction of numerical change between indicators can be predicted through quantitative coupling relationships.

[0046] Specifically, the probability distribution is evaluated as follows: Ideal Gaussian Mixture Distribution Construction: Based on the Central Limit Theorem, the confidence output of the voiceprint model approaches a Gaussian distribution under the comprehensive mapping of a large number of independent acoustic features to construct an ideal distribution. The confidence expected value of the registered user component is concentrated in the high value region (mean such as 0.75), while the confidence of the non-registered user component is concentrated in the low value region (mean such as 0.25). The standard deviations of the two components are controlled to ensure sufficient separation.

[0047] EM algorithm GMM parameter fitting: The expectation-maximization algorithm is used to fit the two-component Gaussian mixture model to the confidence data output by the voiceprint model to obtain the actual distribution parameters.

[0048] Wasserstein Distance Calculation: Calculates the Wasserstein-1 distance between the actual GMM distribution and the ideal GMM distribution, quantifying the global difference between the confidence distribution and the ideal distribution. The smaller the distance, the better the model's discriminative ability. Device-Grained GMM-EMD Ranking: Fits GMMs and calculates EMD at the device level, then sorts them by distance.

[0049] The evaluation report is generated by integrating the calculation results of the indicators and the probability distribution EMD values, and a ranking list of devices is output based on the EMD values.

[0050] Specifically, the workflow of a voiceprint recognition evaluation system based on multidimensional indicators and probability distribution comparison in this embodiment is as follows: S1. Input the labeled evaluation data: Each record includes voiceprint model correctness label (correct / incorrect), user perception label (correct / incorrect), and voiceprint confidence value.

[0051] S2. Cross-statistics are performed according to the two labeled dimensions to construct a two-dimensional confusion matrix and obtain the sample counts in the four quadrants.

[0052] S3. Calculate the derived indicators layer by layer according to the four-layer derivation chain: the first layer is direct statistical counting, the second layer is calculated... and Third layer calculation and Fourth layer calculation .

[0053] S4. Extract voiceprint confidence values ​​from the evaluation data, which can be grouped by device dimension.

[0054] S5. Construct an ideal Gaussian mixture distribution, in which the registered user and unregistered user components are clearly separated in the confidence space.

[0055] S6. Use the EM algorithm to fit a two-component GMM to the confidence data to obtain the actual distribution parameters.

[0056] S7. Calculate the Wasserstein-1 distance between the actual distribution and the ideal distribution using Monte Carlo and bisection methods for efficient calculation.

[0057] S8. Calculate and sort the EMD at the equipment granularity level.

[0058] S9. Summarize the calculation results of the indicators, EMD distance, and equipment ranking, and generate an evaluation report.

[0059] Furthermore, the meanings of the parameters and the formulas are as follows: (1) Confusion matrix notation: (2) Derivative indicator symbols: (3) Probability distribution evaluation symbols: (4) The core formula is as follows: Calculation of derived indicators: In equation (5) =1 is the baseline: less than 1 indicates that the response strategy has defects, and greater than 1 indicates that the system has made up for the deficiencies of the voiceprint model.

[0060] Quantitative coupling relationship: ; Derivation logic: The control rate is composed of two weighted components, with the contribution of successful fallback when the voiceprint model is flawed. Contribution to not being misled when the voiceprint model is correct. : ; Substituting into equation (5), we obtain the VSI expansion: ; when yes In other words, VSI is entirely determined by the misleading rate. When the voiceprint model is good enough, the system bottleneck lies in whether the response strategy incorrectly negates the correct result.

[0061] Ideal Gaussian mixture distribution: ; EM Algorithm: Step E: ; M step: Wasserstein-1 distance: ; Monte Carlo approximation (M uniformly random sampling points): ; GMM cumulative distribution function (dichotomy method for quantiles): ; It is the standard normal cumulative distribution function.

[0062] Parameter meaning: This invention starts with basic counting of a two-dimensional confusion matrix and derives a four-layer derivation chain to establish a system of indicators with clear quantitative relationships, including accuracy, control rate, misleading rate, fallback rate, and VSI, comprehensively reflecting the accuracy of the voiceprint model and the overall system experience. By using the Wasserstein distance between the ideal GMM and the actual GMM, the invention assesses the model's discriminative ability from the global distribution of confidence, overcoming the problem of artificially high accuracy with a fixed threshold.

[0063] The following section further illustrates the application scenario of an intelligent dialogue system with voiceprint recognition capabilities, where the performance of the voiceprint model needs to be evaluated periodically: Scenario 1: Metric calculation guides optimization direction; In one evaluation, the statistics from approximately 10,000 rounds of labeled evaluation data yielded the following results: .

[0064] Calculated .

[0065] because This indicates that the system still has a high proportion of incorrect responses even when the voiceprint model is correct, and the response strategy should be optimized first.

[0066] Scenario 2: Evaluating the model's discriminative ability by comparing probability distributions; Confidence data for a certain period (approximately 5000 samples after removing the ambiguous interval of 0.4-0.6), with preset ideal distribution parameters. .

[0067] After fitting the actual GMM with the EM algorithm, the Wasserstein-1 distance (Monte Carlo method, M=50000) was calculated, and W≈0.087 was obtained, indicating that the current model has good discriminative ability.

[0068] Scenario 3: Ranking of device granularity; By device dimension (as described above) (Filter) Calculate EMD for each device and sort them. If the EMD of a certain device is significantly greater than the average (e.g., the last 5 devices), If so, then the acoustic environment of the device should be investigated.

[0069] The above description is merely an exemplary embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention specification and drawings under the technical concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A voiceprint recognition evaluation method based on the comparison of multidimensional indicators and probability distributions, characterized in that, include: S1: Obtain the labeled evaluation record data, which includes the voiceprint model judgment result, confidence score, and true correctness label; S2: Based on the voiceprint model judgment result and the true correctness label, construct a two-dimensional confusion matrix and count the sample counts in four quadrants. The four quadrants are the first quadrant where the voiceprint model is correct and the user's perception is correct, the second quadrant where the voiceprint model is correct but the user's perception is incorrect, the third quadrant where the voiceprint model is incorrect but the user's perception is correct, and the fourth quadrant where the voiceprint model is incorrect and the user's perception is incorrect. S3: Based on the sample counts in the four quadrants, calculate the voiceprint model accuracy and control rate, wherein the voiceprint model accuracy is the ratio of the sum of the sample counts in the first quadrant and the second quadrant to the total number of samples, and the control rate is the ratio of the sum of the sample counts in the first quadrant and the third quadrant to the total number of samples. S4: Based on the accuracy and control rate of the voiceprint model, calculate the misleading rate and the fallback rate, wherein the misleading rate uses the correct samples of the voiceprint model as the denominator to measure the proportion of errors in the response strategy, and the fallback rate uses the incorrect samples of the voiceprint model as the denominator to measure the proportion of remedies in the response strategy. S5: Based on the misleading rate and the fallback rate, and the accuracy of the voiceprint model, calculate the voiceprint scene index. The voiceprint scene index is defined as the ratio of the control rate to the accuracy of the voiceprint model, and is used to characterize the degree to which the system successfully compensates for the deficiencies of the voiceprint model or the existence of misleading effects.

2. The voiceprint recognition evaluation method based on the comparison of multidimensional indicators and probability distributions according to claim 1, characterized in that, The method further includes S6: extracting the confidence scores from the labeled evaluation record data and constructing an ideal two-component Gaussian mixture distribution based on the central limit theorem; The ideal two-component Gaussian mixture distribution includes registered user components and unregistered user components.

3. The voiceprint recognition evaluation method based on multidimensional index and probability distribution comparison according to claim 2, characterized in that, The method further includes S7: fitting the confidence score using the expectation-maximization algorithm to obtain the actual distribution parameters, wherein the actual distribution parameters include the weight, mean, and standard deviation of each component.

4. The voiceprint recognition evaluation method based on comparison of multidimensional indicators and probability distribution according to claim 3, characterized in that, The method further includes S8: calculating the Wasserstein-1 distance between the actual distribution corresponding to the actual distribution parameters and the ideal two-component Gaussian mixture distribution. The Wasserstein-1 distance is obtained by solving the integral of the quantile function using Monte Carlo sampling combined with the bisection method, and is used to quantify the global discriminative power of the model.

5. The voiceprint recognition evaluation method based on comparison of multidimensional indicators and probability distribution according to claim 4, characterized in that, The method further includes S9: grouping the labeled evaluation record data according to the equipment granularity, performing operations S6 to S8 on each equipment group to obtain the Wasserstein-1 distance of each equipment group, and generating an equipment quality ranking list based on the size of the Wasserstein-1 distance of each equipment group.

6. The voiceprint recognition evaluation method based on the comparison of multidimensional indicators and probability distributions according to claim 1, characterized in that, The voiceprint scene index calculated in S3 satisfies the following relationship: When the voiceprint scene index is less than 1, it indicates that the response strategy is flawed and the misleading effect is dominant. When the voiceprint scene index is greater than 1, it indicates that the system has successfully made up for the deficiencies of the voiceprint model and the fallback effect is dominant.

7. The voiceprint recognition evaluation method based on the comparison of multidimensional indicators and probability distributions according to claim 4, characterized in that, The formula for calculating the Wasserstein-1 distance is as follows: ， Where p is the actual distribution and q is the ideal distribution. and These are the quantile functions for the actual distribution and the ideal distribution, respectively.

8. The voiceprint recognition evaluation method based on the comparison of multidimensional indicators and probability distributions according to claim 5, characterized in that, When grouping the labeled evaluation record data by equipment granularity, equipment groups with a sample size less than a preset minimum value N_min are filtered out. The preset minimum value N_min is 30, to ensure that the equipment quality ranking list only includes statistically significant equipment.

9. The voiceprint recognition evaluation method based on the comparison of multidimensional indicators and probability distributions according to claim 1, characterized in that, The sample counts in the four quadrants correspond to the symbols respectively. and The formula for calculating the misleading rate is as follows: The formula for calculating the coverage rate is as follows: .

10. A voiceprint recognition evaluation system based on the comparison of multidimensional indicators and probability distributions for implementing the method as described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to acquire the labeled evaluation record data; The matrix construction module is used to construct a two-dimensional confusion matrix based on the labeled evaluation record data and count the sample counts in the four quadrants. The indicator calculation module is used to calculate the voiceprint model accuracy, control rate, misleading rate, fallback rate and voiceprint scene index sequentially based on the sample counts in the four quadrants. The distribution evaluation module is used to construct an ideal two-component Gaussian mixture distribution, fit the confidence score using the expectation-maximization algorithm, and calculate the Wasserstein-1 distance between the actual distribution and the ideal distribution. The ranking output module is used to sort the Wasserstein-1 distance of each equipment group according to the equipment granularity and generate a ranking list of equipment quality.

Images