Method and device for measuring output quality of streaming conversation service and storage medium

Abstract

This invention provides a method, apparatus, and storage medium for measuring the output quality of a streaming dialogue service, relating to the field of artificial intelligence technology. The method includes: inputting preset prompt words into a target dialogue service; obtaining the word sequence returned by the target dialogue service through the streaming response mode of its API; calculating the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library based on a pre-built standard decoding reference library and the word sequence returned by the target dialogue service; and determining the output quality of the target dialogue service based on the structural deviation. This invention utilizes the streaming output characteristics of APIs to quantitatively analyze the output sequence based on structural deviation, possessing not only good objectivity and operability but also high scalability of the measurement standard through the construction of a standard decoding reference library.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus and storage medium for measuring the output quality of a streaming dialogue service. Background Technology

[0002] In current AI technology applications, an increasing number of developers are using APIs to connect to third-party large-scale models to build dialogue services for intelligent generation, dialogue assistants, translation, and other business scenarios. To improve the interactive experience, these services generally adopt a streaming output mode, gradually returning the decoded word sequence to achieve a real-time "generating and displaying simultaneously" effect. However, differences in text decoding strategies among different dialogue services lead to inconsistent word segmentation methods, which may negatively impact the smoothness of downstream system display, semantic coherence, and processing stability. Overly fine-grained output from some services can cause disjointed front-end presentation and a poor reading experience; improper splitting of certain mixed Chinese-English or compound word structures can lead to semantic breaks or even abnormal rhythm in downstream speech synthesis.

[0003] Currently, when developers introduce or integrate dialogue services, they primarily rely on methods such as manual spot checks, rule matching, and final content comparison to assess output quality. However, existing methods lack systematic analysis of streaming output boundary delineation and decoding consistency. Furthermore, measuring the performance of different services in terms of display smoothness and structural rationality is reliant on manual methods and difficult to quantify. Especially in scenarios involving service version updates or vendor switching, it is difficult to promptly detect potential changes in output behavior, which can easily lead to front-end rendering anomalies or a decline in user experience, thus failing to meet developers' actual needs in service integration, version verification, and multi-source comparison. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus and storage medium for measuring the output quality of streaming dialogue services. By utilizing the characteristics of API streaming output, the output sequence is quantitatively analyzed based on structural deviation. This not only has good objectivity and operability, but also makes the measurement standard highly scalable by constructing a standard decoding reference library.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a method for measuring the output quality of a streaming dialogue service, comprising: inputting preset prompt words into a target dialogue service; obtaining the word sequence returned by the target dialogue service through the streaming response mode of the API of the target dialogue service; calculating the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library based on a pre-built standard decoding reference library and the word sequence returned by the target dialogue service; and determining the output quality of the target dialogue service based on the structural deviation.

[0007] Optionally, the above method further includes: obtaining multiple reference word segmentation systems and clustering the reference word segmentation systems to obtain a standard decoding reference library; wherein the standard decoding reference library includes multiple reference word segmentation templates.

[0008] Optionally, the reference word segmentation systems are clustered to obtain a standard decoding reference library, including: obtaining test text and inputting the test text into the reference word segmentation systems to obtain the word sequence output by each reference word segmentation system; preprocessing the word sequence output by each reference word segmentation system to obtain the word segmentation boundary of each reference word segmentation system; calculating the consistency between the reference word segmentation systems based on the word segmentation boundary of each reference word segmentation system; clustering the reference word segmentation systems based on the consistency between the reference word segmentation systems to obtain multiple classes of reference word segmentation systems; selecting reference word segmentation templates from each class of reference word segmentation systems, and constructing a standard decoding reference library based on the reference word segmentation templates.

[0009] Optionally, based on a pre-built standard decoding reference library and the lexical sequence returned by the target dialogue service, the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library is calculated, including: concatenating the lexical sequence returned by the target dialogue service to obtain the complete text; performing word segmentation on the complete text based on each reference word segmentation template in the standard decoding reference library to obtain the lexical sequence output by each reference word segmentation template, and determining the reference boundary position set of each reference word segmentation template based on the lexical sequence output by each reference word segmentation template; determining the boundary position set of the target dialogue service based on the lexical sequence returned by the target dialogue service; and calculating the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library based on the reference boundary position set of each reference word segmentation template and the boundary position set of the target dialogue service.

[0010] Optionally, the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library is calculated, including: calculating the symmetric difference between the boundary position set of the target dialogue service and the reference boundary position set of each reference word segmentation template, and calculating the norm of the symmetric difference to obtain the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library.

[0011] Optionally, determining the output quality of the target dialogue service based on structural deviation includes: obtaining the minimum structural deviation value and the number of parameter segmentation templates corresponding to the minimum structural deviation value; comparing the minimum structural deviation value with a preset deviation threshold; and determining the output quality of the target dialogue service based on the comparison result and the number of parameter segmentation templates corresponding to the minimum structural deviation value.

[0012] Optionally, the output quality of the target dialogue service is determined based on the comparison results and the number of parameter segmentation templates corresponding to the minimum structural deviation value, including: if the minimum structural deviation value is less than a preset deviation threshold and the number of parameter segmentation templates corresponding to the minimum structural deviation value is one, then the output structure of the target dialogue service is determined to be reasonable and meets the expected standard; if the minimum structural deviation value is greater than or equal to the preset deviation threshold, then the output structure of the target dialogue service is determined to have deviation; if the minimum structural deviation value is less than the preset deviation threshold and the number of parameter segmentation templates corresponding to the minimum structural deviation value is greater than one, then the output structure of the target dialogue service is determined to have universality and compatibility.

[0013] Secondly, the present invention provides a device for measuring the output quality of a streaming dialogue service, comprising: an output sequence acquisition module, used to input preset prompt words into a target dialogue service and obtain the word sequence returned by the target dialogue service through the streaming response mode of the API of the target dialogue service; a structure deviation calculation module, used to calculate the structure deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library based on a pre-built standard decoding reference library and the word sequence returned by the target dialogue service; and a quality determination module, used to determine the output quality of the target dialogue service based on the structure deviation.

[0014] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the steps of the method provided in any of the first aspects above.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the steps of the method provided in any of the first aspects above.

[0016] This invention brings the following beneficial effects:

[0017] The method, apparatus, and storage medium for measuring the output quality of streaming dialogue services provided by this invention first input preset prompt words into the target dialogue service. Then, through the streaming response mode of the target dialogue service's API, the word sequence returned by the target dialogue service is obtained. Next, based on a pre-built standard decoding reference library and the word sequence returned by the target dialogue service, the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library is calculated. Finally, the output quality of the target dialogue service is determined based on the structural deviation. This method, without relying on the internal implementation details of the model, utilizes the streaming output characteristics of the API to calculate the effective quantitative indicator of the structural quality of the streaming output of the dialogue service—structural deviation—and performs quantitative analysis of the output sequence based on the structural deviation. It not only possesses good objectivity and operability but also, by constructing a standard decoding reference library, makes the measurement standard highly scalable.

[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a method for measuring the output quality of a streaming dialogue service, as provided in an embodiment of the present invention;

[0022] Figure 2 A flowchart illustrating another method for measuring the output quality of a streaming dialogue service, provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a device for measuring the output quality of a streaming dialogue service, provided in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Currently, when developers introduce or integrate dialogue services, they primarily rely on methods such as manual sampling, rule matching, and final content comparison to evaluate output quality. Manual sampling involves observing output effects through manual testing, resulting in subjective standards, low efficiency, limited coverage, and difficulty in forming quantifiable conclusions. Rule matching uses simple text rules to detect anomalies and segmentation, but it has poor adaptability to multilingual mixing and complex contexts. Final content comparison only focuses on the semantic correctness of the generated results, ignoring the structural characteristics of the word sequence during streaming output. Therefore, existing methods lack a systematic analysis of streaming output boundary delineation and decoding consistency, and their measurement of different services' performance in terms of display fluency and structural rationality is reliant on manual methods and difficult to quantify. Especially in scenarios involving service version updates or vendor switching, it is difficult to promptly detect potential changes in output behavior, easily leading to front-end rendering anomalies or a decline in user experience, thus failing to meet developers' actual needs in service integration, version verification, and multi-source comparison.

[0027] Based on this, the present invention provides a method, apparatus and storage medium for measuring the output quality of streaming dialogue services. It utilizes the characteristics of API streaming output and performs quantitative analysis of the output sequence based on structural deviation. It not only has good objectivity and operability, but also makes the measurement standard highly scalable by constructing a standard decoding reference library.

[0028] To facilitate understanding of this embodiment, a method for measuring the output quality of a streaming dialogue service disclosed in this invention will first be described in detail. This method can be executed by an electronic device, such as a smartphone, computer, or tablet. See also... Figure 1 The flowchart shown illustrates a method for measuring the output quality of a streaming dialogue service, indicating that the method mainly includes the following steps S101 to S103:

[0029] Step S101: Input the preset prompt words into the target dialogue service, and obtain the word sequence returned by the target dialogue service through the streaming response mode of the API of the target dialogue service.

[0030] In one implementation, the target dialogue service is the dialogue service to be analyzed. For the target dialogue service, utilizing its API's streaming response capability, and under the control of preset prompts, the word sequence output by the target dialogue service is collected and recorded as a string list, preserving the boundary information of the original output.

[0031] In practical implementation, when collecting the lexical sequence output by the target dialogue service, a specific prompt word needs to be sent to the target dialogue service, such as: "Directly output a 1000-word long script using Latin script and common CJK characters. Emojis must not appear! As random as possible." This prompt aims to guide the model to generate a structurally rich mixed text free of emoji interference. Then, through the API's streaming response mode, the lexical units returned by the target dialogue service are received one by one and recorded sequentially as a list of strings, such as: ["The", "quick", "brown", "fox", "jumps", ...]. This lexical sequence completely preserves the lexical boundary information of the service during the actual decoding process, and this lexical boundary information is the basic data for subsequent metric analysis.

[0032] Step S102: Based on the pre-built standard decoding reference library and the word sequence returned by the target dialogue service, calculate the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library.

[0033] In one implementation, a standard decoding reference library is pre-built. For multiple languages ​​and typical text structures, several existing mainstream dialogue systems are collected as reference word segmentation systems. Then, word segmentation experiments are conducted on the reference word segmentation systems using a unified test text. Based on the system output, word segmentation systems with similar output models are clustered. In each category, a model with reasonable segmentation and high stability is selected as a standard reference template (i.e., a reference word segmentation template) to form a standard decoding reference library.

[0034] Based on this, when calculating the structural deviation, the collected lexical sequence of the target dialogue service is first concatenated to restore the complete text. Then, the complete text is re-segmented using various reference segmentation templates from the standard decoding reference library, and the segmentation boundary position set corresponding to each reference segmentation template is extracted. Next, the structural deviation between the boundary position set of the original output lexical sequence of the target dialogue service and the boundary position set of each re-segmentation result is calculated. This structural deviation is the norm of the symmetric difference between the boundary position set of the original output lexical sequence of the target dialogue service and the boundary position set of each re-segmentation result.

[0035] Step S103: Determine the output quality of the target dialogue service based on the structural deviation.

[0036] In one implementation, after obtaining the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library, a comprehensive judgment can be made based on the comparison between the minimum deviation value and the threshold using an appropriate deviation threshold.

[0037] In practice, the minimum structural deviation value and the number of parameter segmentation templates corresponding to it are first obtained from the structural deviation score. Then, the minimum structural deviation value is compared with a preset deviation threshold. Based on the comparison result and the number of parameter segmentation templates corresponding to the minimum structural deviation value, the output quality of the target dialogue service is determined. Specifically, the minimum structural deviation value is selected from the structural deviation score. Set the deviation threshold to C The typical value is 2 (which can be adjusted flexibly according to actual needs).

[0038] Specifically, if the minimum structural deviation value is less than the preset deviation threshold, and the number of parameter segmentation templates corresponding to the minimum structural deviation value is one, then the output structure of the target dialogue service is determined to be reasonable and meets the expected standards. Less than C Furthermore, if the number of parameter segmentation templates corresponding to the minimum structural deviation value is one, then the output structure of the target dialogue service is determined to be reasonable and meets the expected standards.

[0039] If the minimum structural deviation value is greater than or equal to a preset deviation threshold, then the output structure of the target dialogue service is determined to be deviated. Specifically, if... Greater than or equal to C If the output shows a significant structural bias, it is considered that there is an unreasonable segmentation method in the output, which may interfere with downstream tasks. It is recommended to optimize the model or replace the service.

[0040] If the minimum structural deviation value is less than the preset deviation threshold, and the number of parameter segmentation templates corresponding to the minimum structural deviation value is greater than one, then the output structure of the target dialogue service is determined to have universality and compatibility. Specifically, if multiple parameter segmentation templates simultaneously reach the minimum deviation value... (That is, the number of parameter segmentation templates corresponding to the minimum structural deviation value is greater than one) and Less than C This indicates that the output structure has good versatility and compatibility, and can be adapted to various mainstream word segmentation systems.

[0041] The method for measuring the output quality of streaming dialogue services provided in this embodiment of the invention can, without relying on the internal implementation details of the model, utilize the characteristics of API streaming output to calculate the effective quantitative indicator of structural deviation of the structural quality of streaming dialogue service output, and perform quantitative analysis on the output sequence based on the structural deviation. It not only has good objectivity and operability, but also makes the measurement standard highly scalable by constructing a standard decoding reference library.

[0042] In one embodiment, the method further includes: acquiring multiple reference word segmentation systems and clustering the reference word segmentation systems to obtain a standard decoding reference library; wherein the standard decoding reference library includes multiple reference word segmentation templates.

[0043] In practical implementation, several mainstream word segmentation systems were selected as reference systems. These systems included both open-source models and those providing word segmentation service API interfaces. For open-source models, their vocabulary files (such as tokenizer.json) and accompanying configurations could be downloaded directly. For closed-source services, word segmentation capabilities were obtained by calling their official word segmentation interfaces or analysis tools. Subsequently, word segmentation experiments were conducted on the reference word segmentation systems using a unified test text. Based on the system outputs, word segmentation systems with similar output models were clustered. In each cluster, models with reasonable segmentation and high stability were selected as standard reference templates (i.e., reference word segmentation templates) to form a standard decoding reference library.

[0044] Furthermore, when clustering the reference word segmentation system to obtain the standard decoding reference library, the following methods can be used, including but not limited to:

[0045] First, obtain the test text and input it into the reference word segmentation system to obtain the word sequence output by each system. In practice, a uniform test text is used as input, and the word segmentation functions of each reference system are called to obtain the corresponding word sequence. This test text must contain various elements such as Chinese characters, English words, numbers, and punctuation marks, and its length must be no less than one thousand words.

[0046] Secondly, after preprocessing the word sequence output by each reference word segmentation system, the segmentation boundaries of each system are obtained. In practice, to ensure the purity of the analysis, the outputs of each reference word segmentation system need to be cleaned, removing special control symbols that do not actually appear in the test text, such as "<|begin of sentence|>", "[gMASK]", etc. <sop>"etc.

[0047] Next, the consistency between the reference word segmentation systems is calculated based on the word segmentation boundaries of each reference word segmentation system.

[0048] Then, the reference word segmentation systems are clustered based on the consistency between the reference word segmentation systems, and multiple categories of reference word segmentation systems are obtained.

[0049] Finally, a reference word segmentation template is selected from each category of reference word segmentation systems, and a standard decoding reference library is constructed based on the reference word segmentation template.

[0050] In specific implementation, the cleaned word sequence is subjected to cluster analysis, systems with similar behaviors are classified into a category according to the consistency of the word segmentation boundaries of each reference word segmentation system, and a representative model is selected from each category as a reference word segmentation template, and finally a standard decoding reference library is formed. In calculating the consistency of the word segmentation boundaries of each reference word segmentation system, the symmetric difference method can be used, that is, first, the symmetric difference of the word segmentation boundaries of each two reference word segmentation systems is calculated, then the norm of the symmetric difference is calculated, and the norm is taken as the consistency between the reference word segmentation systems. Then, according to a pre-set evaluation threshold, systems with a consistency less than the evaluation threshold are classified into a category, and finally a model is randomly selected from each category as a reference word segmentation template. The standard decoding reference library can be realized as a program module in actual application, and the user only needs to input the text to be analyzed to obtain a group of word segmentation sequences generated by different reference templates.

[0051] In one embodiment, for the aforementioned step S102, that is, when the structural deviation degree between the target dialogue service and each reference word segmentation template in the standard decoding reference library is calculated based on the word sequence returned by the target dialogue service and the pre-constructed standard decoding reference library, the following methods can be used, including but not limited to:

[0052] First, the word sequence returned by the target dialogue service is spliced to obtain a complete text; second, based on each reference word segmentation template in the standard decoding reference library, the complete text is subjected to word segmentation processing to obtain the word sequence output by each reference word segmentation template, and the reference boundary position set of each reference word segmentation template is determined based on the word sequence output by each reference word segmentation template; next, the boundary position set of the target dialogue service is determined based on the word sequence returned by the target dialogue service; finally, the structural deviation degree between the target dialogue service and each reference word segmentation template in the standard decoding reference library is calculated based on the reference boundary position set of each reference word segmentation template and the boundary position set of the target dialogue service.

[0053] In practice, the collected word sequence of the target dialogue service is concatenated to restore the complete text. Then, each reference segmentation template in the standard decoding reference library is called to re-segment the complete text, and the word sequence output by each reference segmentation template and the reference segmentation boundary positions (represented by character indices) are recorded to form a reference boundary position set. ( i =1,2,..., N , N (The number of reference word segmentation templates). Simultaneously, the set of corresponding boundary positions is calculated based on the original collected sequence (the lexical sequence returned by the target dialogue service). and based on the reference boundary position set and boundary location set Calculate the structural deviation between the target dialogue service and each reference segmentation template in the standard decoding reference library.

[0054] Furthermore, the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library is calculated, including: calculating the symmetric difference between the boundary position set of the target dialogue service and the reference boundary position set of each reference word segmentation template, and calculating the norm of the symmetric difference to obtain the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library.

[0055] In practical implementation, the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library is defined as the set. With sets The norm of the symmetric difference, i.e.:

[0056]

[0057] in, Serving the purpose of dialogue i Structural deviation between reference word segmentation templates.

[0058] For ease of understanding, this embodiment of the invention also provides a flowchart of another method for measuring the output quality of streaming dialogue services, see [link to flowchart]. Figure 2 As shown, it mainly includes three stages: construction of a standard decoding reference library, acquisition of service output sequences, and calculation of structural deviation and quality assessment.

[0059] The first step is to build a standard decoding reference library. For various languages ​​and typical text structures, text processing components (such as open-source model vocabularies, word segmentation APIs, etc.) from multiple existing mainstream dialogue systems are collected. Using a unified test text, the word segmentation functions of the above systems are called to obtain the corresponding word output sequences. After cleaning the output results (including cleaning control symbols), cluster analysis is performed on the word sequences, and representative models are selected from each category to form a standard decoding reference library.

[0060] Secondly, the output sequence of the acquisition service is collected. For the target dialogue service, specific prompt words are sent to the target dialogue service, and the returned words are received one by one through API streaming response. The words are recorded as a string list in sequence, and boundary information is preserved.

[0061] Finally, the structural deviation of the word sequence is collected and a quality assessment is performed. In the quality assessment stage, the collected word sequences are first reassembled into complete text. Then, each template from the standard decoding reference library is called for re-segmentation, and the set of boundary positions corresponding to the original sequence is calculated. Next, the structural deviation of each template is calculated, and the minimum deviation is selected to set a judgment threshold (i.e., the deviation threshold). Then, it is determined whether the minimum deviation is less than the judgment threshold and corresponds to a unique template. If so, the output structure is considered reasonable. If not, it is determined whether multiple templates simultaneously reach the same minimum deviation and are all below the judgment threshold. If so, the output structure has good universality and compatibility; if not, the output has a significant structural deviation.

[0062] The method for measuring the output quality of streaming dialogue services provided in this embodiment of the invention constructs a standard decoding reference library and quantifies the degree of structural deviation by comparing the actual output sequence of the streaming dialogue service with the word segmentation boundaries of each model in the reference library, thereby achieving quality judgment. It not only has good objectivity and operability, but also makes the measurement standard highly scalable.

[0063] In addition to the method for measuring the output quality of streaming dialogue services provided in the foregoing embodiments, this invention also provides a device for measuring the output quality of streaming dialogue services. (See attached document.) Figure 3 The schematic diagram shown illustrates the structure of a device for measuring the output quality of a streaming dialogue service, indicating that the device mainly includes the following parts:

[0064] The output sequence acquisition module 301 is used to input preset prompt words into the target dialogue service and obtain the word sequence returned by the target dialogue service through the streaming response mode of the API of the target dialogue service.

[0065] The structural deviation calculation module 302 is used to calculate the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library based on the pre-built standard decoding reference library and the word sequence returned by the target dialogue service.

[0066] The quality assessment module 303 is used to determine the output quality of the target dialogue service based on the structural deviation.

[0067] The measurement device for the output quality of streaming dialogue services provided in this embodiment of the invention can calculate the effective quantitative indicator of structural deviation of the structural quality of streaming dialogue service output by utilizing the characteristics of API streaming output without relying on the internal implementation details of the model. It can also perform quantitative analysis on the output sequence based on the structural deviation. It not only has good objectivity and operability, but also has high scalability of the measurement standard by constructing a standard decoding reference library.

[0068] In one embodiment, the above apparatus further includes: a standard decoding reference library construction module, used to: acquire multiple reference word segmentation systems and cluster the reference word segmentation systems to obtain a standard decoding reference library; wherein the standard decoding reference library includes multiple reference word segmentation templates.

[0069] In one implementation, the aforementioned standard decoding reference library construction module is specifically used for: acquiring test text and inputting the test text into a reference word segmentation system to obtain the word sequence output by each reference word segmentation system; preprocessing the word sequence output by each reference word segmentation system to obtain the word segmentation boundary of each reference word segmentation system; calculating the consistency between reference word segmentation systems based on the word segmentation boundary of each reference word segmentation system; clustering the reference word segmentation systems based on the consistency between reference word segmentation systems to obtain multiple types of reference word segmentation systems; selecting a reference word segmentation template from each type of reference word segmentation system, and constructing a standard decoding reference library based on the reference word segmentation template.

[0070] In one implementation, the structural deviation calculation module 302 is specifically used for: concatenating the word sequence returned by the target dialogue service to obtain the complete text; performing word segmentation on the complete text based on each reference word segmentation template in the standard decoding reference library to obtain the word sequence output by each reference word segmentation template, and determining the reference boundary position set of each reference word segmentation template based on the word sequence output by each reference word segmentation template; determining the boundary position set of the target dialogue service based on the word sequence returned by the target dialogue service; and calculating the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library based on the reference boundary position set of each reference word segmentation template and the boundary position set of the target dialogue service.

[0071] In one embodiment, the above-mentioned structural deviation calculation module 302 is specifically used to: calculate the symmetry difference between the boundary position set of the target dialogue service and the reference boundary position set of each reference word segmentation template, and calculate the norm of the symmetry difference to obtain the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library.

[0072] In one embodiment, the quality determination module 303 is specifically used to: obtain the minimum structural deviation value in the structural deviation and the number of parameter segmentation templates corresponding to the minimum structural deviation value; compare the minimum structural deviation value with a preset deviation threshold, and determine the output quality of the target dialogue service based on the comparison result and the number of parameter segmentation templates corresponding to the minimum structural deviation value.

[0073] In one embodiment, the quality determination module 303 is specifically used to: if the minimum structural deviation value is less than a preset deviation threshold and the number of parameter segmentation templates corresponding to the minimum structural deviation value is one, then determine that the output structure of the target dialogue service is reasonable and meets the expected standard; if the minimum structural deviation value is greater than or equal to the preset deviation threshold, then determine that the output structure of the target dialogue service has a deviation; if the minimum structural deviation value is less than the preset deviation threshold and the number of parameter segmentation templates corresponding to the minimum structural deviation value is greater than one, then determine that the output structure of the target dialogue service has universality and compatibility.

[0074] It should be noted that the device provided in the embodiments of the invention has the same implementation principle and the same technical effects as the aforementioned method embodiments. For the sake of brevity, any parts not mentioned in the device embodiments can be referred to the corresponding content in the aforementioned method embodiments. The specific numerical values ​​provided in the embodiments of the present invention are merely exemplary and are not intended to limit the scope of the invention.

[0075] This invention also provides an electronic device, specifically, the electronic device includes a processor and a storage device; the storage device stores a computer program, and the computer program, when run by the processor, executes the method described in any of the above embodiments.

[0076] Figure 4 The present invention provides a schematic diagram of the structure of an electronic device 100, which includes a processor 40, a memory 41, a bus 42 and a communication interface 43. The processor 40, the communication interface 43 and the memory 41 are connected through the bus 42. The processor 40 is used to execute executable modules, such as computer programs, stored in the memory 41.

[0077] The memory 41 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0078] Bus 42 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0079] The memory 41 is used to store programs. After receiving an execution instruction, the processor 40 executes the program. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0080] Processor 40 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 40 or by instructions in software form. Processor 40 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 41. The processor 40 reads the information in memory 41 and, in conjunction with its hardware, completes the steps of the above method.

[0081] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0082] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.< / sop>

Claims

1. A method for measuring the output quality of a streaming dialogue service, characterized in that, include: Multiple reference word segmentation systems are obtained, and the reference word segmentation systems are clustered to obtain a standard decoding reference library; wherein, the standard decoding reference library includes multiple reference word segmentation templates; Input the preset prompt words into the target dialogue service, and obtain the word sequence returned by the target dialogue service through the streaming response mode of the API of the target dialogue service; Based on a pre-built standard decoding reference library and the lexical sequence returned by the target dialogue service, the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library is calculated. This calculation includes: concatenating the lexical sequence returned by the target dialogue service to obtain complete text; segmenting the complete text based on each reference word segmentation template in the standard decoding reference library to obtain the lexical sequence output by each template, and determining the reference boundary position set for each template based on the lexical sequence output by each template; determining the boundary position set of the target dialogue service based on the lexical sequence returned by the target dialogue service; and calculating the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library based on the reference boundary position set of each template and the boundary position set of the target dialogue service. The output quality of the target dialogue service is determined based on the structural deviation. This determination includes: obtaining the minimum structural deviation value and the number of reference word segmentation templates corresponding to the minimum structural deviation value; comparing the minimum structural deviation value with a preset deviation threshold; and determining the output quality of the target dialogue service based on the comparison result and the number of reference word segmentation templates corresponding to the minimum structural deviation value.

2. The method according to claim 1, characterized in that, Clustering the aforementioned reference word segmentation system yields a standard decoding reference library, including: Obtain test text and input the test text into the reference word segmentation system to obtain the word sequence output by each reference word segmentation system; After preprocessing the word sequence output by each of the reference word segmentation systems, the word segmentation boundary of each of the reference word segmentation systems is obtained; Based on the word segmentation boundaries of each of the reference word segmentation systems, the consistency between the reference word segmentation systems is calculated; Based on the consistency among the reference word segmentation systems, the reference word segmentation systems are clustered to obtain multiple types of reference word segmentation systems; Select a reference word segmentation template from each type of reference word segmentation system, and build a standard decoding reference library based on the reference word segmentation template.

3. The method according to claim 1, characterized in that, Calculating the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library includes: Calculate the symmetric difference between the boundary position set of the target dialogue service and the reference boundary position set of each reference word segmentation template, and calculate the norm of the symmetric difference to obtain the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library.

4. The method according to claim 1, characterized in that, Based on the comparison results and the number of reference word segmentation templates corresponding to the minimum structural deviation value, the output quality of the target dialogue service is determined, including: If the minimum structural deviation value is less than the preset deviation threshold, and the number of reference word segmentation templates corresponding to the minimum structural deviation value is one, then the output structure of the target dialogue service is determined to be reasonable and meets the expected standard. If the minimum structural deviation value is greater than or equal to the preset deviation threshold, it is determined that the output structure of the target dialogue service has a deviation. If the minimum structural deviation value is less than a preset deviation threshold, and the number of reference word segmentation templates corresponding to the minimum structural deviation value is greater than one, then the output structure of the target dialogue service is determined to have universality and compatibility.

5. A measurement device for the output quality of a streaming dialog service, characterized in that, include: A standard decoding reference library construction module is used to acquire multiple reference word segmentation systems and cluster the reference word segmentation systems to obtain a standard decoding reference library; wherein, the standard decoding reference library includes multiple reference word segmentation templates; The output sequence acquisition module is used to input preset prompt words into the target dialogue service and obtain the word sequence returned by the target dialogue service through the streaming response mode of the API of the target dialogue service; The structural deviation calculation module is used to calculate the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library, based on a pre-built standard decoding reference library and the word sequence returned by the target dialogue service. Specifically, the structural deviation calculation module is used to concatenate the word sequence returned by the target dialogue service to obtain complete text; perform word segmentation on the complete text based on each reference word segmentation template in the standard decoding reference library to obtain the word sequence output by each reference word segmentation template, and determine the reference boundary position set of each reference word segmentation template based on the word sequence output by each reference word segmentation template; determine the boundary position set of the target dialogue service based on the word sequence returned by the target dialogue service; and calculate the structural deviation between the target dialogue service and each reference word segmentation template in the standard decoding reference library based on the reference boundary position set of each reference word segmentation template and the boundary position set of the target dialogue service. A quality determination module is used to determine the output quality of the target dialogue service based on the structural deviation. Specifically, the quality determination module is used to: obtain the minimum structural deviation value and the number of reference word segmentation templates corresponding to the minimum structural deviation value; compare the minimum structural deviation value with a preset deviation threshold, and determine the output quality of the target dialogue service based on the comparison result and the number of reference word segmentation templates corresponding to the minimum structural deviation value.

6. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the method described in any one of claims 1 to 4.

Images