Test method of language rewriting model, storage medium and electronic device

Abstract

This application discloses a testing method, storage medium, and electronic device for a language rewriting model, relating to the field of smart home / intelligent home technology. The method includes: acquiring a test corpus, which is composed of historical data from user interactions with smart devices, data constructed by product development and testing personnel, and data generated by a language generalization model; extracting a subset of data to be tested from the test corpus and inputting this subset into the language rewriting model to obtain the rewriting results; analyzing and processing the rewriting results to obtain multi-dimensional model evaluation metrics; and evaluating the test results of the language rewriting model using these metrics, with the results indicating whether the language rewriting model meets the test requirements. This method, by enriching the corpus sources, simulating real-world business scenarios, and adding different dimensions of evaluation metrics, more comprehensively and accurately measures the effectiveness of the language rewriting model in actual business operations.

Description

Technical Field

[0001] This application relates to the field of smart home / intelligent home technology, and more specifically, to a testing method, storage medium, and electronic device for a language rewriting model. Background Technology

[0002] With the rapid development of artificial intelligence technology, large models are increasingly widely used in the field of natural language processing, especially in language rewriting tasks such as text summarization, text generation, and dialogue system optimization, where large models have demonstrated powerful capabilities. These applications not only improve the efficiency of information processing but also greatly enrich the human-computer interaction experience.

[0003] Before deploying large-scale models to real-world business scenarios, comprehensive and accurate testing is crucial. Currently, testing language rewriting models typically relies on manually constructed corpora as input, using natural language processing techniques to quantitatively evaluate the model-generated text, or on subjective evaluation of the model-generated text by humans.

[0004] However, manually constructed corpora are usually based on the experience and assumptions of testers, making it difficult to fully cover the diversity and complexity of real-world business scenarios. The limited source of the corpus and the limited testing dimensions make it difficult to comprehensively and accurately measure the actual effectiveness of large-scale language rewriting models in business applications. Summary of the Invention

[0005] This application provides a testing method, storage medium, and electronic device for a language rewriting model, in order to solve the problem that in the prior art, the testing scheme in large model context rewriting (i.e., language rewriting model) deviates from the actual use by users and cannot accurately measure the actual effect of the large model in business applications.

[0006] Firstly, this application provides a testing method for a language rewriting model, comprising:

[0007] Obtain a test corpus set; wherein, the test corpus set includes: historical corpus of user interactions with smart devices, corpus constructed by product development and testing personnel, and corpus generated by a language generalization model;

[0008] A subset of test corpus is extracted from the test corpus set and input into the language rewriting model to obtain the corpus rewriting result. The subset of test corpus is extracted according to the proportion of the corpus categories in the test corpus set. The corpus rewriting result is a sentence generated based on the context of the test corpus.

[0009] The rewritten corpus results are analyzed and processed to obtain multi-dimensional model evaluation indicators;

[0010] The test results of the language rewriting model are evaluated using the model evaluation metrics, and the test results are used to indicate whether the language rewriting model meets the test requirements.

[0011] Optionally, before obtaining the test corpus, the method further includes:

[0012] Obtain historical data on user interactions with smart devices;

[0013] The historical corpus is identified and annotated to obtain test corpus; the test corpus is used to indicate the execution intent of each historical corpus.

[0014] The test corpus is classified and parsed according to preset categories to obtain a first test corpus set, which includes: the categories of the test corpus and the information proportion of each category;

[0015] The first test corpus is subjected to construction and generalization processes to obtain a second test corpus and a third test corpus. The test corpus includes the first test corpus, the second test corpus, and the third test corpus.

[0016] Optionally, the step of classifying and parsing the test corpus according to preset categories to obtain a first test corpus includes:

[0017] Based on the preset category, information is extracted from the test corpus to obtain multiple category information, where the category information refers to the specific information in the test corpus corresponding to the preset category.

[0018] Based on the various categories of information, determine the information proportion of each preset category, where the information proportion refers to the proportion of category information in all information of the same preset category;

[0019] The first test corpus set is generated based on the test corpus and the proportion of the information.

[0020] Optionally, the step of performing construction and generalization processing on the first test corpus to obtain the second and third test corpus sets includes:

[0021] Based on the information proportions, a second test corpus constructed by the staff is obtained, wherein the information proportion of each test corpus in the second test corpus is the same as the information proportion of each test corpus in the first test corpus;

[0022] The test corpus is input into the language generalization model to obtain the generalized corpus;

[0023] The generalized corpus is extracted and processed based on the information proportions to obtain the third test corpus set.

[0024] Optionally, the analysis and processing of the corpus rewriting results to obtain multi-dimensional model evaluation index values ​​includes:

[0025] The rewritten result is parsed and judged to determine the result category;

[0026] Statistical processing is performed on the result categories to obtain statistical values ​​for the result categories;

[0027] Based on the statistical values ​​of the aforementioned result categories, multidimensional model evaluation metrics are determined.

[0028] Optionally, the result category includes one or more of the following categories: positive sample rewriting correctly, positive sample rewriting incorrectly, negative sample rewriting, negative sample not rewriting, high-frequency negative sample rewriting, and high-frequency negative sample not rewriting. The evaluation metric includes one or more of the following metrics: recall, precision, negative sample accuracy, and high-frequency negative sample accuracy.

[0029] The recall rate is determined based on the correct rewriting of the positive samples and the statistical values ​​of all positive samples.

[0030] The precision rate is determined based on the statistical values ​​of the positive samples being rewritten correctly, the positive samples being rewritten incorrectly, and the negative samples being rewritten.

[0031] The accuracy of the negative samples is determined based on the fact that the negative samples were not rewritten and the statistical values ​​of all negative samples.

[0032] The accuracy of the high-frequency negative samples is determined based on the statistical values ​​of the high-frequency negative samples that have been rewritten and the high-frequency negative samples that have not been rewritten.

[0033] Optionally, determining the test results of the language rewriting model based on the model evaluation metrics includes:

[0034] Determine whether the model evaluation index meets the preset evaluation index threshold;

[0035] If the model evaluation index meets the preset evaluation index threshold, the test result is determined to be a successful test.

[0036] If the model evaluation index does not meet the preset evaluation index threshold, the test result is determined to be a test failure.

[0037] Secondly, this application provides a testing apparatus for a language rewriting model, comprising:

[0038] The acquisition module is used to acquire a test corpus set; wherein, the test corpus set includes: historical corpus of user interaction with smart devices, corpus constructed by product development testers, and corpus generated by language generalization models;

[0039] The input module is used to extract a subset of test corpus from the test corpus set and input the subset of test corpus into the language rewriting model to obtain the corpus rewriting result. The subset of test corpus is extracted according to the proportion of the corpus categories in the test corpus set. The corpus rewriting result is a sentence generated based on the context of the test corpus.

[0040] The processing module is used to analyze and process the rewritten corpus results to obtain multi-dimensional model evaluation indicators.

[0041] The determination module is used to evaluate the test results of the language rewriting model using the model evaluation metrics, and the test results are used to indicate whether the language rewriting model meets the test requirements.

[0042] Optionally, the acquisition module is further configured to acquire historical data of user interactions with smart devices;

[0043] The processing module is further configured to identify and annotate the historical corpus to obtain test corpus; the test corpus is used to indicate the execution intent of each historical corpus.

[0044] The processing module is further configured to perform classification and parsing processing on the test corpus according to a preset category to obtain a first test corpus set, the first test corpus set including: the category of the test corpus and the information proportion of each category;

[0045] The processing module is further configured to perform construction processing and generalization processing on the first test corpus set to obtain a second test corpus set and a third test corpus set, wherein the test corpus set includes: the first test corpus set, the second test corpus set and the third test corpus set.

[0046] Optionally, the apparatus further includes: a generation module;

[0047] The processing module is also used to extract and process information from the test corpus according to the preset category to obtain multiple category information, wherein the category information refers to the specific information in the test corpus corresponding to the preset category;

[0048] The determining module is further configured to determine the information proportion of each preset category based on the various types of category information, wherein the information proportion refers to the proportion of category information in all information of the same preset category;

[0049] The generation module is further configured to generate the first test corpus set based on the test corpus and the information proportion.

[0050] Optionally, the acquisition module is further configured to acquire a second test corpus constructed by the staff based on the information proportion, wherein the information proportion of each test corpus in the second test corpus is the same as the information proportion of each test corpus in the first test corpus;

[0051] The input module is also used to input the test corpus into the language generalization model to obtain generalized corpus;

[0052] The processing module is further configured to extract and process the generalized corpus according to the information proportion to obtain the third test corpus set.

[0053] Optionally, the processing module is further configured to parse and judge the rewritten result to determine the result category;

[0054] The processing module is also used to perform statistical processing on the result category to obtain statistical values ​​for the result category;

[0055] The determining module is further configured to determine multi-dimensional model evaluation indicators based on the statistical values ​​of the result categories.

[0056] Optionally, the determining module is further configured to determine the recall rate based on the statistical values ​​of the correct rewrite of the positive samples and all positive samples;

[0057] The determining module is further configured to determine the precision rate based on the statistical values ​​of the positive samples being rewritten correctly, the positive samples being rewritten incorrectly, and the negative samples being rewritten.

[0058] The determining module is further configured to determine the accuracy of the negative sample based on the fact that the negative sample was not rewritten and the statistical value of all negative samples;

[0059] The determining module is further configured to determine the accuracy of the high-frequency negative samples based on the statistical values ​​of the high-frequency negative samples being rewritten and the high-frequency negative samples not being rewritten.

[0060] Optionally, the device further includes: a determination module;

[0061] The judgment module is used to determine whether the model evaluation index meets the preset evaluation index threshold.

[0062] The determining module is further configured to determine that the test result is a test pass if the model evaluation index meets the preset evaluation index threshold.

[0063] The determining module is further configured to determine that the test result is a test failure if the model evaluation index does not meet the preset evaluation index threshold.

[0064] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0065] The memory stores computer-executed instructions;

[0066] The processor executes computer execution instructions stored in the memory to implement the test method for the language rewriting model as described in the first aspect and various possible implementations of the first aspect.

[0067] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions thereon, which, when executed by a processor, are used to implement a test method for a language rewriting model as described in the first aspect and various possible implementations of the first aspect.

[0068] Fifthly, this application provides a program product, including a computer program, which, when executed by a processor, implements the testing method for the language rewriting model described above.

[0069] This application provides a testing method, storage medium, and electronic device for a language rewriting model. The method involves acquiring a test corpus from multiple sources, including past user interactions with smart devices, specially constructed data by product development testers, and corpora generated through a language generalization model. Next, a subset of the corpus to be tested is extracted from this comprehensive test corpus and input into the language rewriting model to obtain the rewritten results. Subsequently, these rewritten results undergo multi-dimensional computational analysis to derive a series of model evaluation metrics. Based on these metrics, the test results of the language rewriting model are judged, directly reflecting whether the model has met the testing standards. This method, by broadening the sources of the corpus, simulating real-world business environments, and introducing diverse evaluation dimensions, provides a more comprehensive and accurate assessment of the performance of the language rewriting model in practical applications. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0071] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a schematic diagram of the hardware environment for a test method of a language rewriting model according to an embodiment of this application;

[0073] Figure 2 A flowchart illustrating a testing method for a language rewriting model provided in this application. Figure 1 ;

[0074] Figure 3 A flowchart illustrating a testing method for a language rewriting model provided in this application. Figure 2 ;

[0075] Figure 4 A flowchart illustrating a testing method for a language rewriting model provided in this application. Figure 3 ;

[0076] Figure 5 A schematic diagram of the structure of a test device for a language rewriting model provided in this application;

[0077] Figure 6 A schematic diagram of the structure of a test device for a language rewriting model provided in this application. Detailed Implementation

[0078] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0079] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0080] According to one aspect of the embodiments of this application, a method for testing a language rewriting model is provided. This method for testing language rewriting models is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned method for testing language rewriting models can be applied to scenarios such as... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0081] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0082] With the rapid development of artificial intelligence technology, large-scale models are being applied more and more widely in the field of natural language processing, especially in language rewriting tasks such as text summarization, text creation, and improvement of dialogue systems, where large-scale models have demonstrated outstanding capabilities. These applications have not only significantly improved the speed of information processing, but also greatly enhanced the richness and quality of human-machine interaction.

[0083] Before applying large-scale models to real-world business environments, thorough and accurate testing is crucial. Currently, the common practice for testing language rewriting models is to rely on manually created corpora by testers as input and use natural language processing techniques to quantitatively evaluate the model's output text, or to subjectively evaluate the model's output text manually.

[0084] However, corpora manually created by testers are often limited by their experience and preconceived notions, making it difficult to fully cover the various complex and ever-changing scenarios in real-world business. The sources of these corpora are relatively singular, and the testing perspectives are also limited, making it difficult to comprehensively and accurately evaluate the real-world performance of large-scale models in practical business applications.

[0085] To address the aforementioned issues, this application provides a testing method for a language rewriting model. This method aims to construct a diverse and comprehensive test corpus to accurately evaluate the actual performance of the language rewriting model. Specifically, it integrates multiple corpus sources: first, historical interaction records between users and smart devices; second, corpus constructed by staff in different roles during product development and testing; and third, generalized corpus generated using a language generalization model. Simultaneously, it calculates the proportion of various corpus types to ensure the test corpus is rich and comprehensive in terms of scenario types. Based on the corpus proportions, a comprehensive selection of test corpus data is input into the language rewriting model, and the model's output rewriting results are subjected to multi-dimensional and refined calculations and evaluations, thereby accurately measuring the performance of the language rewriting model across different dimensions.

[0086] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0087] Figure 2 A flowchart illustrating a testing method for a language rewriting model provided in this application embodiment. Figure 1 .like Figure 2 As shown, the testing method for the language rewriting model provided in this embodiment includes:

[0088] S101: Obtain the test corpus.

[0089] The test corpus includes: historical data on user interactions with smart devices, data constructed by product development and testing personnel, and data generated by a language generalization model.

[0090] Understandably, test corpora serve as the basis for testing language generalization models. Large test corpora can cover more linguistic phenomena and text types, thus providing a more comprehensive evaluation of the language rewriting model's performance. By testing on different types of text, we can understand the model's performance in different scenarios, and then optimize the model accordingly. The larger the test corpus, the more accurately the evaluation results reflect the model's true performance. Testing on large datasets can reduce the impact of random errors on the evaluation results and improve the accuracy of the evaluation.

[0091] S102: Extract a subset of the test corpus from the test corpus set and input the subset of the test corpus into the language rewriting model to obtain the corpus rewriting result.

[0092] The test corpus subset is extracted according to the proportion of the corpus categories in the test corpus set, and the corpus rewriting result is the corpus generated based on the expected context of the test.

[0093] Understandably, a test corpus is a collection of text data constructed in advance before model testing, typically used to evaluate the performance of a language model. Since the test corpus can be very large, using the entire corpus directly for testing is not only time-consuming and labor-intensive, but may also affect the accuracy of the evaluation results due to data redundancy. Therefore, we need to extract a subset from it according to certain criteria (such as corpus proportion) as the test corpus.

[0094] Here, "corpus proportion" refers to the proportion of various types of corpora in the test corpus. Extracting test corpora according to their proportions ensures the diversity of corpus types used in testing and guarantees that the corpora match the characteristics of the model's application. For example, if user interaction with a TV is the most common application scenario for the model, then during the model testing phase, the focus should be on testing and optimizing the corpus rewriting for user-TV interaction to ensure that the model maintains high accuracy in rewriting performance during real-world applications, i.e., when users interact with a TV.

[0095] Once the subset of the test corpus is determined, the next step is to input it into a pre-trained language rewriting model. This model is a deep learning model built using large-scale pre-training and self-supervised learning techniques, capable of generating rewritten text based on the input text. The result of the rewriting is to transform incomplete user commands into statements that the device can recognize and execute.

[0096] S103: Analyze and process the corpus rewriting results to obtain multi-dimensional model evaluation indicators.

[0097] Understandably, the rewritten results are multiple statements that can be recognized by smart home devices. Based on the rewritten results and the input test data, we can determine whether the model rewriting is correct. The rewritten results are then statistically analyzed and categorized, and calculated according to a preset formula to obtain multi-dimensional model evaluation metrics. The specific calculation process is detailed below and will not be elaborated here.

[0098] S104: Determine the test results of the language rewriting model based on the model evaluation metrics.

[0099] The test results are used to indicate whether the language rewriting model meets the test requirements.

[0100] Understandably, model evaluation metrics are a set of standards or methods for measuring the performance of a language rewriting model. These metrics may include, for example, the accuracy and precision of the model rewriting. If the calculated evaluation metrics meet the preset evaluation criteria (i.e., the testing requirements), it means that the model has passed the test and can be applied in actual business.

[0101] This embodiment provides a testing method for a language rewriting model. The method involves acquiring a test corpus, which comprises historical data from user interactions with smart devices, data constructed by product development and testing personnel, and data generated by a language generalization model. A subset of the test corpus is extracted from the test corpus and input into the language rewriting model to obtain the rewriting result. The rewriting result is analyzed and processed to obtain multi-dimensional model evaluation metrics. Based on these metrics, the test result of the language rewriting model is determined, indicating whether the model meets the testing requirements. This method, by enriching the corpus sources, simulating real-world business scenarios, and adding different dimensions of evaluation metrics, more comprehensively and accurately measures the effectiveness of the language rewriting model in actual business operations.

[0102] Figure 3 A flowchart illustrating a testing method for a language rewriting model provided in this application embodiment. Figure 2 .like Figure 3 As shown, in Figure 2 Based on the embodiments, a possible implementation method for obtaining test corpus sets through multiple channels is described in detail, including:

[0103] S201: Obtain historical data on user interactions with smart devices.

[0104] Understandably, smart home devices (such as smart speakers, smart lighting systems, and smart thermostats) typically record detailed information about their interactions with users, including timestamps, user commands, and device responses. These logs are the primary source of historical data on user interactions with smart devices. By reading and analyzing these logs, we can extract useful linguistic information. Historical data refers to the language data generated during user interactions with smart home devices over a period of time. This data typically includes user voice commands, device responses, and any relevant contextual information.

[0105] S202: Identify and annotate the historical corpus to obtain the test corpus.

[0106] Understandably, test corpora are used to indicate the execution intent of each historical corpus. After acquiring the historical corpus, information recognition processing is performed on the corpus to determine the execution intent of the historical corpus, and key information of a test corpus segment of historical corpus is annotated to facilitate rapid identification of the corpus content during subsequent corpus classification. For example, a historical corpus could be: User X (device wake word), I want to watch a movie. X: Okay, what movie would you like to watch? User: AA (movie name). X: Okay, would you like to watch it on the living room TV or the bedroom projector? User: Living room TV. X: It's already playing for you on the living room TV. The information identified and annotated for this historical corpus could be: Living room TV, playing movie AA.

[0107] S203: Extract and process information from the test corpus according to preset categories to obtain information of multiple categories.

[0108] The category information indicates the specific information corresponding to each preset category in the test corpus. The preset categories include: speech entry point, execution domain, speech turn, and positive and negative sample information. Speech entry point refers to the device or system that triggers a specific operation; execution domain refers to the type of operation performed by the speech entry point; speech turn refers to the turn in the dialogue; and positive and negative samples indicate whether the test corpus needs to be rewritten by the model.

[0109] Understandably, corpora may differ across different execution devices and operations. Therefore, classifying the corpora allows for a better understanding of the model's performance in various aspects. Furthermore, classification ensures that the model can be tested on corpora extracted from different implementation scenarios during testing, comprehensively reflecting its performance in real-world applications. Category information refers to the specific information within the test corpus that falls under a predefined category. This specific information is extracted and statistically analyzed. For example, taking the corpus in step S201, its corresponding entry point is: living room TV, the execution domain is: playing a movie, the number of speaking rounds is: 3 rounds, and the positive / negative sample information is: positive samples (needs rewriting). For this test corpus, the predefined category is the entry point, and its corresponding category information is: living room TV.

[0110] S204: Determine the proportion of information in each preset category based on multiple categories of information.

[0111] Among them, the information proportion refers to the proportion of category information among all information in the same preset category.

[0112] Understandably, category information describes the specific details of the test corpus within a preset category. Within the same category, there may be multiple instances of specific information. Classifying identical specific information allows us to determine the proportion of each piece of information within that category. For example, taking the preset category of "speech entry point" as an example, in this category, 10 test corpora mention the living room TV, 20 mention the bedroom air conditioner, and 5 mention the living room speaker. The information proportion of the speech entry point is: Living room TV: Bedroom air conditioner: Living room speaker = 10:20:5 = 2:4:1.

[0113] S205: Generate the first test corpus set based on the test corpus and the proportion of information.

[0114] Understandably, after obtaining the test corpus and the information proportion of each preset category, the test corpus is categorized and saved according to the preset categories to generate the first test corpus set. This ensures that when extracting test corpus according to categories, the complete test corpus can be extracted accordingly. It should be noted that a complete test corpus can be extracted from multiple categories. For example, taking the corpus in step S201 as an example, the complete corpus can be extracted through the category of "living room TV," and it can also be extracted through three rounds of interaction.

[0115] S206: Based on the information proportion, obtain the second test corpus constructed by the staff.

[0116] In the second test corpus, the information percentage of each test corpus is the same as that of each test corpus in the first test corpus.

[0117] Understandably, the staff includes personnel from different stages of product development and testing. These different roles can construct various datasets from different perspectives. By extracting data from these constructed datasets based on their information proportions, a second test dataset is created. This ensures that the test dataset fully reflects user needs and problem distribution in actual use, thereby improving the relevance and effectiveness of the testing. By simulating real data distribution, developers can more accurately identify potential problems and vulnerabilities, guiding product design and feature improvements. Constructing test datasets based on historical data can also reduce development costs and time, and increase the automation level of testing.

[0118] S207: Input the test corpus into the language generalization model to obtain the generalized corpus.

[0119] Understandably, the test corpus refers to the historical data of user interactions with the device, while the language generalization model is a machine learning model capable of processing and generating similar data. By learning from the input corpus, it captures the rules and patterns of language and generates new corpus that retains the original language characteristics. This process is called "generalization," meaning the model can apply its learned knowledge to new, unseen data. After inputting historical real-world data into the language generalization model, the model outputs a new batch of data. This new data retains the language characteristics and distribution of the original data, but its specific content may differ.

[0120] S208: Extract and process the generalized corpus according to the information proportion to obtain the third test corpus set.

[0121] Understandably, when processing generalized corpora, to ensure that the test corpus truly reflects user habits and needs, it's necessary to extract generalized corpora based on their proportion in historical real-world data. This process involves extracting and processing the generalized corpora according to their proportions. Through this process, a test corpus set containing various types of corpora while maintaining their original proportions can be obtained—the third test corpus set. This corpus set is used in the product testing phase to verify the product's ability and performance in processing different types of corpora.

[0122] This embodiment provides a testing method for a language rewriting model. The method acquires historical data of user interactions with smart devices. The historical data is identified and annotated to obtain test data. Information from the test data is extracted according to preset categories, resulting in multiple categories of information. The proportion of information in each preset category is determined based on the multiple categories. A first test data set is generated based on the test data and the information proportions. A second test data set constructed by staff is obtained based on the information proportions. The test data is input into a language generalization model to obtain generalized data. The generalized data is extracted and processed according to the information proportions to obtain a third test data set. This method collects test data through diverse channels to enrich the data sources and processes the data according to the actual data situation, ensuring that the selection of test data is both comprehensive and reasonable, thereby achieving a more complete test of the model.

[0123] Figure 4 A flowchart illustrating a testing method for a language rewriting model provided in this application embodiment. Figure 3 .like Figure 4 As shown, in Figure 2 Based on the embodiments, a possible implementation method for determining multi-dimensional evaluation indicators based on the rewriting results is described in detail, including:

[0124] S301: Analyze and judge the rewritten result to determine the result category.

[0125] The result categories include: positive samples rewritten correctly, positive samples rewritten incorrectly, negative samples rewritten, negative samples not rewritten, high-frequency negative samples rewritten, and high-frequency negative samples not rewritten.

[0126] Understandably, rewritten results are the corpus obtained after a language rewriting model rewrites the input corpus. These rewritten results need to be further parsed and evaluated to determine their category. The process of determining the category of the results can, for example, use automated evaluation tools to ensure the accuracy of the evaluation.

[0127] "Positive sample rewriting correct" means that the original positive sample (the corpus that needs model rewriting) has been correctly rewritten by the model, and the rewritten sentence retains its original correctness; "Positive sample rewriting incorrect" means that the original positive sample has been correctly rewritten by the model, but the rewritten sentence has a different meaning from the original; "Negative sample rewriting" means that the original negative sample (the corpus that does not need model rewriting) has been rewritten by the model; "Negative sample rewriting" means that the language rewriting model correctly identifies the negative sample and does not rewrite it; high-frequency negative samples refer to sentences that frequently appear in the training data and do not need to be rewritten. If the language rewriting model incorrectly rewrites these high-frequency negative samples, then this rewriting result is considered "high-frequency negative samples rewritten"; when the language rewriting model correctly identifies high-frequency negative samples and does not rewrite them, this processing result is considered "high-frequency negative samples not rewritten".

[0128] S302: Perform statistical processing on the result categories to obtain statistical values ​​for the result categories.

[0129] Understandably, after determining the outcome category for each rewrite result, it's necessary to perform statistical analysis on these categories. Statistical values ​​refer to the number of different outcome categories within the entire batch of rewritten corpora. For example, we can count the number of correctly rewritten positive samples, the number of rewritten negative samples, and the number of incorrectly rewritten positive samples. These statistical values ​​help us understand the model's performance on different types of corpora, as well as the model's overall performance and stability.

[0130] S303: Determine the multidimensional model evaluation indicators based on the statistical values ​​of the result categories.

[0131] Understandably, multi-dimensional model evaluation metrics refer to a set of metrics that evaluate model performance from multiple angles and dimensions. These metrics include: recall, precision, negative sample accuracy, and high-frequency negative sample accuracy.

[0132] Optionally, the calculation method of the multi-dimensional model evaluation index is explained in detail below:

[0133] The recall rate is determined based on the correct rewrite of the positive samples and the statistical values ​​of all positive samples.

[0134] As is understandable, recall represents the proportion of positive samples correctly rewritten by the model out of all positive samples. Recall can be calculated in multiple rounds. Recall focuses on the model's ability to correctly predict all actual positive samples, i.e., the model's completeness. By calculating the recall in multiple rounds, the model's completeness performance in different rounds and under different conditions can be evaluated more comprehensively.

[0135] Precision is determined based on the statistical values ​​of correctly rewritten positive samples, incorrectly rewritten positive samples, and rewritten negative samples.

[0136] As is understandable, precision refers to the proportion of instances that are actually positive out of all instances predicted as positive. Precision is calculated as: Correctly rewritten positive samples / (Correctly rewritten positive samples + Incorrectly rewritten positive samples + Rewritten negative samples).

[0137] The negative sample accuracy is determined based on the fact that the negative sample was not rewritten and the statistical value of all negative samples.

[0138] As is understandable, negative sample accuracy refers to the ratio of the number of samples correctly identified as negative to the total number of negative samples. It measures the model's ability to distinguish negative samples. The formula is defined as: Negative Sample Accuracy = (Negative Samples Not Rewritten) / Total Number of Negative Samples.

[0139] The accuracy of high-frequency negative samples is determined based on the statistical values ​​of high-frequency negative samples that have been rewritten and those that have not.

[0140] Understandably, high-frequency negative sample accuracy is a subset of negative sample accuracy, focusing specifically on negative samples that appear frequently in the dataset. These high-frequency negative samples can significantly impact model training due to their dominance in the dataset. High-frequency negative sample accuracy measures the model's accuracy in handling these common negative samples. The calculation formula is defined as: High-frequency negative sample accuracy = High-frequency negative samples not rewritten / Total number of high-frequency negative samples.

[0141] For example, the determination process in step S303 is illustrated as follows: Assume there are 200 positive samples and 100 negative samples in the input model to be rewritten. The rewriting results are: 120 positive samples were rewritten correctly, 80 positive samples were rewritten incorrectly, 20 negative samples were rewritten, 80 negative samples were not rewritten, 5 high-frequency negative samples were rewritten, and 20 high-frequency negative samples were not rewritten.

[0142] Recall: 120 correctly rewritten positive samples / 200 positive samples = 0.6; Precision: 120 correctly rewritten positive samples / (120 correctly rewritten positive samples + 80 incorrectly rewritten positive samples + 20 rewritten negative samples) = 0.54; Negative sample precision: 80 negative samples not rewritten / 100 negative samples = 0.8; High-frequency negative sample precision: 20 high-frequency negative samples not rewritten / 25 high-frequency negative samples = 0.8.

[0143] The recall rate can be calculated in multiple rounds:

[0144] First round of rewriting: There are 200 positive samples, 120 of which were correctly rewritten, and 80 were incorrectly rewritten (rewriting errors include omissions and incorrect corrections). First round recall: 120 correctly rewritten positive samples / 200 positive samples = 0.6.

[0145] Second round of rewriting: 10 new correct entries were added (previously missed or incorrectly corrected), bringing the total to 130 correct entries, with 70 remaining. Second round recall: 130 correct positive samples / 200 positive samples = 0.65.

[0146] S304: Determine whether the model evaluation index meets the preset evaluation index threshold. If yes, proceed to step S305; otherwise, proceed to step S306.

[0147] Understandably, to measure whether model performance meets expectations, we typically set a series of preset evaluation metrics and their corresponding thresholds. These metrics and thresholds are determined based on the specific application scenario, business requirements, and data characteristics. During the evaluation process, we compare the model's performance on each evaluation metric with the preset thresholds. These thresholds represent our minimum requirements for model performance. If the model's performance on a certain evaluation metric reaches or exceeds the preset threshold, then we can consider the model to perform well on that metric and meet our expectations.

[0148] S305: The test result is confirmed as passed.

[0149] Understandably, if the model evaluation metrics meet the preset evaluation metric thresholds, that is, if the model reaches or exceeds the corresponding thresholds on all preset evaluation metrics, then we can determine that the test result is passed, indicating that the model has met our performance requirements and can be further considered for deployment in practical applications.

[0150] S306: Determine the test result as a failure and generate a test report based on the model evaluation metrics.

[0151] Understandably, if the model's evaluation metrics do not meet the preset thresholds—that is, if the model's performance on one or more evaluation metrics fails to reach the preset thresholds—then we need to determine the test result as a failure and statistically analyze the results of the model's evaluation metrics failing to meet the thresholds, generating a test report. This report should detail the problems found during the test; for example, in cases where negative samples were rewritten, it should specify which negative sample was rewritten and what the result was, providing concrete examples and relevant data for testers to analyze the reasons for the failure. A failed test indicates that the model's performance is still insufficient and requires further adjustment and optimization.

[0152] This embodiment provides a testing method for a language rewriting model. The method analyzes and judges the rewriting results to determine the result category. Statistical processing is performed on the result categories to obtain statistical values. Appropriate statistical values ​​for the result categories are selected according to a preset calculation method for model evaluation metrics, and multi-dimensional model evaluation metrics are calculated based on multiple statistical values. It is determined whether the model evaluation metrics meet preset evaluation metric thresholds. If yes, the test result is determined to be a pass; otherwise, the test result is determined to be a fail. This method comprehensively and meticulously measures the effectiveness of the rewriting model across different dimensions by performing multi-dimensional evaluation calculations on the rewriting results.

[0153] Figure 5 A schematic diagram of the structure of a testing device for a language rewriting model provided in this application. Figure 5 As shown, this application provides a testing apparatus for a language rewriting model. The testing apparatus 500 for the language rewriting model includes:

[0154] The acquisition module 501 is used to acquire a test corpus set; wherein, the test corpus set includes: historical corpus of user interaction with smart devices, corpus constructed by product development and testing personnel, and corpus generated by language generalization model;

[0155] The input module 502 is used to extract a subset of test corpus from the test corpus set and input the subset of test corpus into the language rewriting model to obtain the corpus rewriting result. The subset of test corpus is extracted according to the proportion of the categories of the corpus in the test corpus set. The corpus rewriting result is the corpus generated based on the context of the test corpus.

[0156] The processing module 503 is used to analyze and process the rewritten corpus results to obtain multi-dimensional model evaluation indicators.

[0157] The determination module 504 is used to evaluate the test results of the language rewriting model using the model evaluation index, and the test results are used to indicate whether the language rewriting model meets the test requirements.

[0158] Optionally, the acquisition module 501 is further configured to acquire historical data of user interactions with smart devices;

[0159] The processing module 503 is further configured to identify and annotate the historical corpus to obtain test corpus; the test corpus is used to indicate the execution intent of each historical corpus.

[0160] The processing module 503 is further configured to perform classification and parsing processing on the test corpus according to a preset category to obtain a first test corpus set, wherein the first test corpus set includes: the category of the test corpus and the information proportion of each category;

[0161] The processing module 503 is further configured to perform construction processing and generalization processing on the first test corpus set respectively to obtain a second test corpus set and a third test corpus set, wherein the test corpus set includes: the first test corpus set, the second test corpus set and the third test corpus set.

[0162] Optionally, the apparatus further includes: a generation module 505;

[0163] The processing module 503 is further configured to extract and process information from the test corpus according to the preset category to obtain multiple category information, wherein the category information refers to the specific information in the test corpus corresponding to the preset category;

[0164] The determining module 504 is further configured to determine the information proportion of each preset category based on the various types of category information, wherein the information proportion refers to the proportion of category information in all information of the same preset category;

[0165] The generation module 505 is further configured to generate the first test corpus set based on the test corpus and the information proportion.

[0166] Optionally, the acquisition module 501 is further configured to acquire a second test corpus constructed by the staff based on the information proportion, wherein the information proportion of each test corpus in the second test corpus is the same as the information proportion of each test corpus in the first test corpus.

[0167] The input module 502 is also used to input the test corpus into the language generalization model to obtain generalized corpus;

[0168] The processing module 503 is further configured to extract and process the generalized corpus according to the information proportion to obtain the third test corpus set.

[0169] Optionally, the processing module 503 is further configured to perform parsing and judgment processing on the rewritten result to determine the result category;

[0170] The processing module 503 is also used to perform statistical processing on the result category to obtain the statistical value of the result category;

[0171] The determining module 504 is further configured to determine multi-dimensional model evaluation indicators based on the statistical values ​​of the result categories.

[0172] Optionally, the determining module 504 is further configured to determine the recall rate based on the statistical values ​​of the correct rewrite of the positive samples and all positive samples;

[0173] The determining module 504 is further configured to determine the precision rate based on the statistical values ​​of the positive samples being rewritten correctly, the positive samples being rewritten incorrectly, and the negative samples being rewritten.

[0174] The determining module 504 is further configured to determine the accuracy of the negative sample based on the fact that the negative sample was not rewritten and the statistical value of all negative samples;

[0175] The determining module 504 is further configured to determine the accuracy of the high-frequency negative samples based on the statistical values ​​of the high-frequency negative samples being rewritten and the high-frequency negative samples not being rewritten.

[0176] Optionally, the device further includes: a determination module 506;

[0177] The judgment module 506 is used to determine whether the model evaluation index meets the preset evaluation index threshold.

[0178] The determining module 504 is further configured to determine that the test result is a test pass if the model evaluation index meets the preset evaluation index threshold.

[0179] The determining module 504 is further configured to determine that the test result is a test failure if the model evaluation index does not meet the preset evaluation index threshold.

[0180] The testing device for the language rewriting model provided in this application embodiment is similar in principle and technical effect to the implementation of each part of the aforementioned testing method for the language rewriting model, and will not be repeated here.

[0181] Figure 6 A schematic diagram of the structure of a test device for a language rewriting model provided in this application. Figure 6 As shown, this application provides a test device for a language rewriting model. The test device 600 for the language rewriting model includes: a receiver 601, a transmitter 602, a processor 603, and a memory 604.

[0182] Receiver 601 is used to receive instructions and data;

[0183] Transmitter 602 is used to send commands and data;

[0184] Memory 604 is used to store instructions executed by the computer;

[0185] The processor 603 is used to execute computer execution instructions stored in the memory 604 to implement the various steps of the test method in the above embodiments. For details, please refer to the relevant descriptions in the foregoing test method embodiments.

[0186] Optionally, the memory 604 can be either standalone or integrated with the processor 603.

[0187] When the memory 604 is set up independently, the electronic device also includes a bus for connecting the memory 604 and the processor 603.

[0188] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0189] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in any of the foregoing embodiments.

[0190] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0191] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0192] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0193] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0194] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0195] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0196] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A testing method for a language rewriting model, characterized in that, include: Obtain a test corpus set; wherein, the test corpus set includes: historical corpus of user interactions with smart devices, corpus constructed by product development and testing personnel, and corpus generated by a language generalization model; A subset of test corpus is extracted from the test corpus set and input into the language rewriting model to obtain the corpus rewriting result. The subset of test corpus is extracted according to the proportion of the corpus categories in the test corpus set. The corpus rewriting result is a sentence generated based on the context of the test corpus. The rewritten corpus results are analyzed and processed to obtain multi-dimensional model evaluation indicators; The test results of the language rewriting model are evaluated using the model evaluation metrics, and the test results are used to indicate whether the language rewriting model meets the test requirements.

2. The method according to claim 1, characterized in that, Before obtaining the test corpus, the method further includes: Obtain historical data on user interactions with smart devices; The historical corpus is identified and annotated to obtain test corpus; the test corpus is used to indicate the execution intent of each historical corpus. The test corpus is classified and parsed according to preset categories to obtain a first test corpus set, which includes: the categories of the test corpus and the information proportion of each category; The first test corpus is subjected to construction and generalization processes to obtain a second test corpus and a third test corpus. The test corpus includes the first test corpus, the second test corpus, and the third test corpus.

3. The method according to claim 2, characterized in that, The first test corpus is obtained by classifying and parsing the test corpus according to preset categories, including: Based on the preset category, information is extracted from the test corpus to obtain multiple category information, where the category information refers to the specific information in the test corpus corresponding to the preset category. Based on the various categories of information, determine the information proportion of each preset category, where the information proportion refers to the proportion of category information in all information of the same preset category; The first test corpus set is generated based on the test corpus and the proportion of the information.

4. The method according to claim 3, characterized in that, The process of constructing and generalizing the first test corpus to obtain the second and third test corpora includes: Based on the information proportions, a second test corpus constructed by the staff is obtained, wherein the information proportion of each test corpus in the second test corpus is the same as the information proportion of each test corpus in the first test corpus; The test corpus is input into the language generalization model to obtain the generalized corpus; The generalized corpus is extracted and processed based on the information proportions to obtain the third test corpus set.

5. The method according to claim 1, characterized in that, The analysis and processing of the rewritten corpus results yields multi-dimensional model evaluation metrics, including: The rewritten result is parsed and judged to determine the result category; Statistical processing is performed on the result categories to obtain statistical values ​​for the result categories; Based on the statistical values ​​of the aforementioned result categories, multidimensional model evaluation metrics are determined.

6. The method according to claim 5, characterized in that, The result categories include one or more of the following categories: positive samples rewritten correctly, positive samples rewritten incorrectly, negative samples rewritten, negative samples not rewritten, high-frequency negative samples rewritten, and high-frequency negative samples not rewritten. The evaluation metrics include one or more of the following metrics: recall, precision, negative sample accuracy, and high-frequency negative sample accuracy. The step of determining multi-dimensional model evaluation indicators based on the statistical values ​​of the result categories includes: The recall rate is determined based on the correct rewriting of the positive samples and the statistical values ​​of all positive samples. The precision rate is determined based on the statistical values ​​of the positive samples being rewritten correctly, the positive samples being rewritten incorrectly, and the negative samples being rewritten. The accuracy of the negative samples is determined based on the fact that the negative samples were not rewritten and the statistical values ​​of all negative samples. The accuracy of the high-frequency negative samples is determined based on the statistical values ​​of the high-frequency negative samples that have been rewritten and the high-frequency negative samples that have not been rewritten.

7. The method according to claim 1, characterized in that, The step of determining the test results of the language rewriting model based on the model evaluation metrics includes: Determine whether the model evaluation index meets the preset evaluation index threshold; If the model evaluation index meets the preset evaluation index threshold, the test result is determined to be a successful test. If the model evaluation index does not meet the preset evaluation index threshold, the test result is determined to be a test failure, and a test report is generated based on the model evaluation index.

8. A testing device for a language rewriting model, characterized in that, include: The acquisition module is used to acquire a test corpus set; wherein, the test corpus set includes: historical corpus of user interaction with smart devices, corpus constructed by product development testers, and corpus generated by language generalization models; The input module is used to extract a subset of test corpus from the test corpus set and input the subset of test corpus into the language rewriting model to obtain the corpus rewriting result. The subset of test corpus is extracted according to the proportion of the corpus categories in the test corpus set. The corpus rewriting result is a sentence generated based on the context of the test corpus. The processing module is used to analyze and process the rewritten corpus results to obtain multi-dimensional model evaluation indicators. The determination module is used to evaluate the test results of the language rewriting model using the model evaluation metrics, and the test results are used to indicate whether the language rewriting model meets the test requirements.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 7 through the computer program.

Images