A novel method, system and electronic device for fault diagnosis of speech models

Abstract

This specification relates to the field of computer technology, and in particular to a novel method and apparatus for diagnosing faults in speech models. The novel method for diagnosing faults in speech models includes a data acquisition stage, a clustering stage, a similarity calculation stage, and a data processing and reporting stage. The data acquisition stage extracts data from the speech model; the clustering stage clusters the data to obtain clustered data; the similarity calculation stage calculates the similarity of the clustered data; and the data processing and reporting stage analyzes and reports fault data.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a novel method, system and electronic device for diagnosing faults in speech models. Background Technology

[0002] During online operation, speech models often reveal errors not previously covered in testing. These errors severely impact the normal operation of the speech model. Current technologies are unable to effectively monitor these errors or efficiently analyze them, thus hindering the provision of data support for the proper functioning of the speech model. Summary of the Invention

[0003] In view of this, the purpose of one or more embodiments of this specification is to propose a novel method and system for diagnosing voice model faults, so as to solve the problem that the existing technology cannot effectively monitor errors in voice models, can analyze related errors more efficiently, and provide relevant data support for the normal and healthy operation of voice models.

[0004] The present invention adopts the following technical solution:

[0005] A novel method for diagnosing faults in a speech model includes a data acquisition stage, a clustering stage, a similarity calculation stage, and a data processing and reporting stage; wherein, the data acquisition stage is used to extract the data points of the speech model.

[0006] The clustering step is used to cluster the data collected at the tracking points to obtain clustered data.

[0007] The similarity calculation step is used to calculate the similarity of clustered data.

[0008] The data processing and reporting stage is used to analyze fault data and report it.

[0009] Furthermore, the aforementioned data acquisition step includes:

[0010] Step 1, Save Logs: Run the speech model on the server and save the online running logs returned by the client;

[0011] Step 2, Connect to the database: Connect the database and the virtual private network;

[0012] Step 3, extract logs: Call the function to extract logs from the database;

[0013] Step 4: Extract the event tracking data: Extract the event tracking data of the speech model execution based on the online logs.

[0014] Furthermore, the clustering process includes:

[0015] Step one: Proceed to the next action;

[0016] Step 2, Clustering: Cluster the data with the same tracking points in the next step;

[0017] Step 3, Differentiation Processing: In the case of the same action, differentiate based on whether the speech text is matched by the model.

[0018] Furthermore, the similarity calculation step includes...

[0019] Step 1, Call the function: Call the processing function;

[0020] Step 2, similarity calculation: Perform text semantic similarity calculation on the audio texts that match and those that do not.

[0021] Furthermore, the data processing and reporting process includes...

[0022] Step 1, extract fault data: parse and extract the speech text of the hit models with results higher than the threshold, the speech text of the miss models, and the reported information.

[0023] Step 2, Data Reporting: The extracted speech text is filtered to generate the final non-matching speech text and corresponding reporting information, which serves as the speech text and reporting information for analyzing problems in the online speech interaction model.

[0024] Furthermore, the method also includes a similarity verification step, which includes a basic sub-step and a routine operation sub-step. The basic sub-step includes setting up a verification level library, which includes at least one verification level. The routine operation sub-step includes the following steps: First, selecting a verification level: selecting a verification level from the verification level library as needed; Second, calculating based on the level: calculating the text semantic similarity of the data based on the verification level selected in the first step.

[0025] Furthermore, in step three of the data acquisition process, the `cli ckhouse_dr i ver` function is called to write a Python script to execute SQL statements and retrieve logs from the database.

[0026] Furthermore, the data collected in step four of the data acquisition process includes voice text, page data, control data, and scene data.

[0027] A system is characterized by comprising an online log storage module, a clustering module, a text similarity matching module, and a filtering module; wherein the online log storage module is used to record and store the operation logs of the speech model; wherein the clustering module is used to cluster the speech text reporting information based on the log information and the next execution action; the text similarity matching module is used to perform text similarity matching on the matched and unmatched speech texts using the Simbert model; and wherein the filtering module is used to manually filter out the final speech text and corresponding reporting information for the unmatched speech texts whose text similarity is higher than a threshold.

[0028] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the novel voice model fault diagnosis method.

[0029] The beneficial technical effects of this invention are:

[0030] This invention can expose bugs that were not covered by previous tests during the online operation of a speech model, helping developers to extract data from online tracking data more conveniently and effectively, analyze the problems in the model, and further optimize the online speech model.

[0031] This invention clusters speech text based on the next action, distinguishes between hit and miss speech text, calls the Simbert model to calculate semantic similarity, analyzes model problems in online speech interaction, and finally extracts and reports the miss speech text. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in one or more embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only one or more embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram of the method components of a novel voice model fault diagnosis method according to an embodiment of this specification.

[0034] Figure 2 This is a schematic diagram of the data acquisition steps in Embodiment 2 of a novel voice model fault diagnosis method described in this specification.

[0035] Figure 3 This is a schematic diagram of the clustering steps in Embodiment 3 of a novel speech model fault diagnosis method described in this specification;

[0036] Figure 4 This is a schematic diagram of the similarity calculation steps in Embodiment 4 of a novel speech model fault diagnosis method described in this specification.

[0037] Figure 5 This is a schematic diagram of the data processing and reporting steps in Embodiment 5 of a novel voice model fault diagnosis method described in this specification.

[0038] Figure 6 This is a schematic diagram of the similarity check step in Embodiment Six of a novel speech model fault diagnosis method of this specification;

[0039] Figure 7 This is a schematic diagram of the routine operation sub-steps in Embodiment Six of a novel voice model fault diagnosis method of this specification;

[0040] Figure 8 This is a schematic diagram of the system composition in Embodiment Seven of this specification. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0042] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of this specification should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the element or object listed following the word and its equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0043] In this invention, query = voice text, and the following embodiments are all applicable.

[0044] This specification provides an embodiment of a novel voice model fault diagnosis method, such as... Figure 1 As shown:

[0045] This method includes a data acquisition stage, a clustering stage, a similarity calculation stage, and a data processing and reporting stage; among them, the data acquisition stage is used to extract the data acquisition stage of the speech model.

[0046] The clustering step is used to cluster the data collected at the tracking points to obtain clustered data.

[0047] The similarity calculation step is used to calculate the similarity of clustered data.

[0048] The data processing and reporting stage is used to analyze fault data and report it.

[0049] This specification provides a second embodiment of a novel voice model fault diagnosis method, such as... Figure 2 As shown:

[0050] The data acquisition process includes:

[0051] Step 1: Run the speech model on the server and save the online running logs returned by the client;

[0052] Step two: Connect to the database and the virtual private network;

[0053] Step 3: Call the function to retrieve logs from the database;

[0054] Step 4: Extract the data points used to execute the speech model based on the online logs.

[0055] Preferably, in step three of the data acquisition process, the `cl ickhouse_dr i ver` function is called to write a Python script to execute SQL statements and retrieve logs from the database.

[0056] Preferably, enter: SQL query statement

[0057] Output:

[0058] 2022 / 2 / 7 12:30:01,domai n:mus ic,i intent:pl ay,act i on:pl ay,trace_i d:a72b50c,page_id:GROUP_TELEPHONE.DOMAI N_TELEPHONE.PhoneBTDevi ceList,wi dget_i d:HUAWE I P50_1,behavi or_id:1020010101.

[0059] This specification provides a third embodiment of a novel voice model fault diagnosis method, such as... Figure 3 The clustering process described herein includes

[0060] Step one: Proceed to the next action;

[0061] Step 2, Clustering: Cluster the data with the same tracking points in the next step;

[0062] Step 3, differentiate processing: In the case of the same execution action, differentiate according to whether the query is hit by the model.

[0063] Preferred,

[0064] enter:

[0065] page_i d:GROUP_TELEPHONE.DOMAI N_TELEPHONE.PhoneBTDevi ceList,wi dget_i d:HUAWE I P50_1,behavi or_i d:1020010101

[0066] Output:

[0067] {1020010101:{semant ic:{semantic parsing result},page{page information},query:{sentence information},no_res{missed statements}}}

[0068] Example 4 of a novel voice model fault diagnosis method described in this specification, such as... Figure 4 The similarity calculation process described above includes:

[0069] Step 1: Call the processing function;

[0070] Step two: Calculate the text semantic similarity between the queries that match and those that do not.

[0071] This specification provides a fifth embodiment of a novel voice model fault diagnosis method, such as... Figure 5 The data processing and reporting process shown includes:

[0072] Step 1, extract fault data: parse and extract the queries of the hit models with results higher than the threshold, the queries of the miss models, and the reported information;

[0073] The threshold is based on semantic similarity. Semantic similarity is calculated by feeding the matched and unmatched speech texts into the Simbert model. The actual semantics of speech texts with different similarity levels are manually observed to determine the threshold while ensuring that the semantics are roughly similar; generally, the similarity threshold is between 0.5 and 0.7.

[0074] Step two involves filtering the extracted queries to generate the final list of missed queries and their corresponding reporting information, which serves as the query and reporting information for analyzing problems in the online voice interaction model.

[0075] The method for filtering queries (voice-text): Manually determine whether the query is a user command and whether it should be matched by the model. This involves filtering from the unmatched queries, manually selecting those that should be matched by the model. The reported information includes the page context information on the screen when the user uttered the command. Whether a query should be matched by the model requires a comprehensive judgment combining the semantic parsing results of the query and the current page context information.

[0076] Example six of a novel voice model fault diagnosis method described in this specification, such as... Figure 6 —As shown in 7:

[0077] The method also includes a similarity verification step, which includes a basic sub-step and a routine operation sub-step. The basic sub-step includes setting up a verification level library, which includes at least one verification level. The routine operation sub-step includes a first step, selecting a verification level from the verification level library as needed; and a second step, calculating the text semantic similarity of the data based on the verification level selected in the first step.

[0078] Preferably, the data collected in step four of the data collection process includes voice queries, page data, control data, and scene data.

[0079] One embodiment of the system described in this specification is seven, such as Figure 8 As shown:

[0080] A system comprising an online log storage module, a clustering module, a text similarity matching module, and a filtering module; wherein the online log storage module is used to record and store the running logs of a speech model; the clustering module is used to cluster queries and reporting information based on log information and the next execution action; the text similarity matching module is used to perform text similarity matching on queries that match and those that do not match using a Simbert model; and the filtering module is used to manually filter out the final queries and their corresponding reporting information for queries that do not match with text similarity scores higher than a threshold.

[0081] This specification provides an eighth embodiment of an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the novel voice model fault diagnosis method as described in any of the preceding embodiments.

[0082] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0083] The electronic device comprises a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory. The operating system can be any one or more computer operating systems that control the electronic device through processes, such as Linux, Unix, Android, iOS, or Windows. Furthermore, in this embodiment of the invention, the electronic device can be a smartphone, tablet computer, or other handheld device, or a desktop computer, portable computer, or other electronic device; there is no particular limitation in this embodiment.

[0084] In this embodiment of the invention, the executing entity for electronic device control can be an electronic device itself, or a functional module within an electronic device capable of calling and executing a program. The electronic device can obtain the firmware corresponding to the storage medium. This firmware is provided by the supplier, and different storage media may have the same or different firmware; no limitation is made here. After obtaining the firmware corresponding to the storage medium, the electronic device can write this firmware into the storage medium; specifically, it burns the firmware corresponding to the storage medium into the storage medium. The process of burning the firmware into the storage medium can be implemented using existing technology, and will not be elaborated upon in this embodiment of the invention.

[0085] Electronic devices can also obtain reset commands corresponding to the storage media. The reset commands corresponding to the storage media are provided by the supplier. The reset commands corresponding to different storage media can be the same or different, and no restrictions are imposed here.

[0086] At this time, the storage medium of the electronic device is a storage medium on which the corresponding firmware has been written. The electronic device can respond to the reset command corresponding to the storage medium on which the corresponding firmware has been written, thereby resetting the storage medium on which the corresponding firmware has been written according to the reset command. The process of resetting the storage medium according to the reset command can be implemented by existing technology and will not be described in detail in this embodiment of the invention.

[0087] It should be noted that the methods of one or more embodiments of this specification can be executed by a single device, such as a computer or server. The methods of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the methods of one or more embodiments of this specification, and the multiple devices will interact with each other to complete the method described.

[0088] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0089] For ease of description, the above apparatus is described in terms of function, divided into various modules. Of course, when implementing one or more embodiments of this specification, the functions of each module can be implemented in one or more software and / or hardware.

[0090] The apparatus described above is used to implement the corresponding methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0091] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this specification as described above, which are not provided in detail for the sake of brevity.

[0092] Additionally, to simplify the description and discussion, and to avoid obscuring one or more embodiments of this specification, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be illustrated in block diagram form to avoid obscuring one or more embodiments of this specification, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which one or more embodiments of this specification will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that one or more embodiments of this specification may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0093] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description.

[0094] One or more embodiments of this specification are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of this disclosure.

Claims

1. A novel voice model fault diagnosis method, characterized in that, The method comprises a buried point data acquisition link, a clustering link, a similarity calculation link, and a data processing and reporting link; wherein the buried point data acquisition link is used to extract the buried point data of the voice model; The clustering link is used to cluster the buried point data to obtain clustered data. The similarity calculation link is used to calculate the similarity of the clustered data. The data processing and reporting link is used to analyze fault data and report it. The clustering link comprises: Step 1: Perform the next action; Step 2: Cluster processing: cluster the same buried point data in the next action; Step 3: Differentiate: in the case of the same execution action, differentiate according to whether the voice text is hit by the model. The similarity calculation link comprises Step 1: Call the function: call the processing function; Step 2: Similarity calculation: calculate the text semantic similarity of the hit and non-hit voice texts. The data processing and reporting link comprises Step 1: Extract fault data: extract the voice text of the hit model and the voice text of the non-hit model and the reporting information above the threshold value; Step 2: Data reporting: filter the extracted voice text to generate the final non-hit voice text and corresponding reporting information as the voice text and reporting information for analyzing the online voice interaction model problem.

2. The novel voice model failure diagnosis method according to claim 1, characterized in that, The buried point data acquisition link comprises: Step 1: Save the log: run the voice model on the server side and save the online running log returned by the client; Step 2: Connect the database and virtual private network; Step 3: Extract the log: call the function to extract the log from the database; Step 4: Extract the buried point data: extract the buried point data executed by the voice model according to the online log.

3. The novel voice model failure diagnosis method according to claim 1, characterized in that, The method further comprises a similarity verification link, which comprises a basic sub-link and a regular operation sub-link. The basic sub-link comprises a verification level library, which comprises at least one verification level. The regular operation sub-link comprises a first step of selecting a verification level: selecting a verification level from the verification level library as needed; a second step of calculating by level: calculating the text semantic similarity of the data according to the verification level selected in the first step.

4. The novel voice model failure diagnosis method according to claim 2, characterized in that, In step 3 of the buried point data acquisition link, the clickhouse_driver function is called, a python script is written to execute an sql statement, and the log is obtained from the database.

5. The novel voice model failure diagnosis method according to claim 2, characterized in that, The buried point data in step 4 of the buried point data acquisition link comprises voice text, page, control, and scene data.

6. A system based on the new voice model fault diagnosis method according to any one of claims 1 to 5, characterized in that, The system comprises an online log storage module, a clustering module, a text similarity matching module, and a filtering module. The online log storage module is used to record and store the running log of the voice model. The clustering module is used to cluster the voice text reporting information according to the log information and the next execution action. The text similarity matching module is used to match the text similarity of the hit and non-hit voice texts using the simbert model. The filtering module is used to manually filter the final voice text and corresponding reporting information from the non-hit voice text with a text similarity higher than a threshold value.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the new voice model fault diagnosis method according to any one of claims 1 to 5 when executing the program.

Images