Sample labeling detection method and device, medium and electronic equipment
By generating a test question bank and distribution strategy, and using the model to detect manual annotation results, the problems of inconsistent manual annotation quality and low efficiency of expert spot checks are solved, and efficient annotation quality control is achieved.
Patent Information
- Application Number
- CN202410986069.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-15
AI Technical Summary
The quality of manual labeling in the prior art is inconsistent, and experts have found problems such as untimely, missed inspections, and it is difficult to meet the quality requirements of labeling.
By generating a test question bank composed of test samples and standard annotations, the distribution strategy of test samples is determined based on the annotation business and test question bank, the training model is used to detect the differences between manual annotation results and standard annotation, and the labeling errors are promptly discovered and corrected.
Improve the accuracy and efficiency of labeling inspection, timely discover and correct labeling problems, and ensure labeling quality.
Smart Images

Figure CN118964909A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to computer technology, and in particular to a sample labeling detection method, device, medium and electronic equipment. Background Art
[0002] At present, with the widespread application of machine learning in various fields, how to train accurate models while protecting personal information has become a key issue. For supervised machine learning methods, accurate sample labeling is one of the main factors affecting the model training effect.
[0003] In the existing technology, manually labeled samples are still an irreplaceable labeling method, and different labelers may have different understandings of labeling, resulting in inconsistent quality of manual labeling. In order to ensure the quality of labeling, the expert spot check process is generally supplemented. Experts will spot check the labeled samples to identify samples or labelers with labeling problems and correct the labeling errors in a timely manner.
[0004] However, there are still problems in expert spot checks, such as untimely detection, missed detection, and false detection, which still cannot meet the labeling quality requirements. Based on this, the present application provides a new sample labeling detection method, device, storage medium, and electronic device. Summary of the invention
[0005] The embodiment of the present specification provides a method for detecting sample annotations. The method can generate a test question bank consisting of test samples and standard annotations of the test samples by annotating standard samples of the business type or annotating historical samples, and determine a distribution strategy of the test samples based on the annotated business and the test question bank. Finally, based on the difference between the manual annotation results of the test samples and the standard annotations, determine the manual annotation detection results, so as to timely determine the problems of the manual annotations. The method includes:
[0006] Determine the business type to which the model trained by the sample to be tested is applied;
[0007] According to at least one of the standard sample and the historical sample of the service type and the service type of the service, a plurality of groups of test samples and standard annotations of the test samples are generated through the trained detection model as a test question bank of the service type;
[0008] Determining a distribution strategy of the test question bank based on the business information of the marked business and the test question bank;
[0009] The test samples are distributed according to the distribution strategy, and the detection results are determined according to the manual annotation results of the test samples and the standard annotations of the test samples.
[0010] Further, in some implementations, based on at least one of the standard sample and the historical sample of the service type and the service type of the service, multiple groups of test samples and standard annotations of the test samples are generated through the generation branch network of the trained detection model, specifically including:
[0011] According to the generated test question bank of each business type, determining whether there is a test question bank of the business type;
[0012] If yes, then generate multiple groups of test samples and standard annotations of the test samples based on the historical samples of the service type and the service type of the service through the generation branch network of the trained detection model;
[0013] If not, multiple groups of test samples and standard annotations of the test samples are generated according to the standard samples of the business type and the business type of the business through the generation branch network of the trained detection model.
[0014] Further, in some implementations, based on the historical samples of the service type and the service type of the service, multiple groups of test samples and standard annotations of the test samples are generated through the generation branch network of the trained detection model, specifically including:
[0015] According to the detection results of the historical samples of the business type, determine the historical samples with manual labeling errors;
[0016] Determine the analysis results of wrong questions based on the sample characteristics of historical samples with manual annotation errors;
[0017] According to the wrong question analysis results and the business type of the business, a prompt text for generating wrong questions is determined, and multiple groups of test samples and standard annotations of the test samples are generated through the generation branch network of the trained detection model.
[0018] Further, in some implementations, based on the standard sample of the service type and the service type of the service, multiple groups of test samples and standard annotations of the test samples are generated through the generation branch network of the trained detection model, specifically including:
[0019] Determine historical business data of normal execution from various historical business data of the business;
[0020] Determine standard samples and annotations of the standard samples based on historical business data of normal execution;
[0021] According to the standard samples, the annotations of the standard samples and the business type of the business, a prompt text for generating wrong questions is determined, and multiple groups of test samples and standard annotations of the test samples are generated through the generation branch network of the trained detection model.
[0022] Furthermore, in some implementations, based on the service information of the marked service and the test question bank, a distribution strategy of the test question bank is determined, specifically including:
[0023] Determine the shifts of each annotation staff and the annotation environment of each annotation staff according to the business information of the annotation business;
[0024] According to the shifts of the labeling personnel and the labeling environments of the labeling personnel, with the goal of the labeling personnel labeling the test samples in different labeling environments, a distribution strategy of the test question bank is determined.
[0025] Further, in some embodiments, the test result is determined according to the manual labeling result of the test sample and the standard labeling of the test sample, specifically including:
[0026] For each test sample, when the labeling result of the labeler is inconsistent with the standard labeling of the test sample, it is determined that the labeler has incorrectly labeled the test sample;
[0027] According to each test sample with incorrect labeling, the test samples with incorrect labeling and / or the labelers with incorrect labeling are counted as the detection result.
[0028] Furthermore, in some embodiments, the method further comprises:
[0029] For each annotator, the number of incorrectly annotated questions by the annotator is cumulatively calculated based on the test samples that the annotator has annotated incorrectly.
[0030] When the number of incorrectly marked questions reaches a preset value, the marking task for the marking person is stopped.
[0031] Further, in some embodiments, the test result is determined according to the manual labeling result of the test sample and the standard labeling of the test sample, specifically including:
[0032] Monitoring the completion of the annotation business, wherein the completion includes at least one of the hit probability of the test samples in the test question bank, the hit probability of the test samples by each annotator, and the total amount of annotations;
[0033] According to the completion status of the labeling business, determine a target sampling strategy that matches the completion status from preset candidate sampling strategies;
[0034] According to the target selection strategy, select the test samples annotated by the annotator;
[0035] The test results are determined based on the labeling results of the selected test samples and the standard labeling.
[0036] Furthermore, in some embodiments, the method further comprises:
[0037] According to a preset ratio, determine some test samples and their annotation results from each test sample whose annotation results are different from the standard annotation;
[0038] The determined test samples and their annotation results are sent to the designated terminal for review.
[0039] The present specification also provides a sample labeling detection device, the device comprising:
[0040] A type determination module determines the business type to which the model trained by the sample to be tested is applied;
[0041] A test question generation module generates multiple groups of test samples and standard annotations of the test samples as a test question bank for the business type through a trained detection model according to at least one of the standard samples and historical samples of the business type and the business type of the business;
[0042] A test question distribution module, which determines a distribution strategy of the test question bank based on the business information of the marked business and the test question bank;
[0043] The detection module distributes the test samples according to the distribution strategy, and determines the detection results according to the manual labeling results of the test samples and the standard labeling of the test samples.
[0044] The embodiments of the present specification also provide a computer program product, wherein the computer program product stores at least one instruction, and the at least one instruction is suitable for being loaded by a processor and executing the above method steps.
[0045] The embodiments of the present specification also provide a storage medium, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.
[0046] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.
[0047] In the embodiments of the present specification, a test sample and a test question bank consisting of standard annotations of the test sample can be generated for the business type of the sample application through the annotation of the standard sample of the business type or the annotation of the historical sample, and the distribution strategy of the test sample can be determined based on the annotation business and the test question bank. Finally, based on the difference between the manual annotation results of the test sample and the standard annotation, the manual annotation detection result is determined. Through the trained model, test samples are generated based on historical samples or standard samples for testing manual annotation, so as to achieve targeted detection of error-prone situations in manual annotation and improve the accuracy of detection. At the same time, based on the task information of the annotation task, the distribution strategy of the test sample is determined, and by monitoring the annotation results of each test sample, the problems of manual annotation are promptly determined. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 A schematic diagram of a flow chart of a sample labeling detection method is provided for the embodiments of this specification;
[0049] Figure 2 A schematic diagram of a detection model training provided in an embodiment of this specification;
[0050] Figure 3 A schematic diagram of a flow chart of a sample labeling detection method provided in an embodiment of this specification;
[0051] Figure 4 A schematic diagram of the structure of a sample labeling detection device provided in an embodiment of this specification;
[0052] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.
[0054] Usually, sample labeling, application of sample training models, and application of trained models to execute business can be performed by different entities respectively. For example, a data processing company labels a large number of samples and provides the labeled samples to an algorithm development company. The algorithm development company trains the model based on the labeled samples collected from different data sources, and the application company executes the task based on the trained model. Of course, the above process can also be implemented by a single party or two parties. In addition, since companies that usually apply models are more accurate in algorithm selection, expected model effects, sample requirements, and labeling requirements, there is usually a greater need to detect sample labeling when a single entity executes the above process from sample labeling to application of model execution tasks.
[0055] Based on this, please see Figure 1 , which is a flow chart of a sample labeling detection method provided in the embodiment of this specification. Figure 1 The process shown is described in detail, and the sample annotation detection method may specifically include the following steps:
[0056] S100: Determine the business type to which the model trained by the sample to be detected is applied.
[0057] In one or more embodiments of this specification, for the convenience of description, the detection process of sample annotation by a single entity server is taken as an example for subsequent description. Of course, the server can be a single device or a distributed system composed of multiple devices, and this specification does not limit this.
[0058] In one or more embodiments of this specification, the server may first determine the target to be detected. Since the labeling service is usually not only for a certain type of sample, but for a variety of labeling requirements, the labeling personnel are assigned samples to be labeled. For sample labeling detection, it is possible to specifically determine that a certain type or several types of samples need to be labeled, so the server can determine the service type based on the detection target.
[0059] Specifically, the server can determine the model used to train the sample to be tested, and determine the business type to which the model is applied after the training is completed. Therefore, according to the determined business type, it can be determined what kind of test sample to generate in the subsequent steps. The test sample is used to test whether the annotator who annotates the sample to be tested meets the annotation requirements, so as to quickly locate the annotator with abnormal annotation by monitoring the annotation of the test sample.
[0060] The samples to be tested are samples that need to be labeled and tested, which means samples that actually need to be assigned to labelers for labeling through the labeling service. Usually, based on different model training objectives, even with the same data, i.e., samples, the correct labeling results are different. For example, an image sample used for target object recognition should be labeled with a label box of the recognized object, while the same image sample, if used for image classification service, should be labeled with the classification result of the object in the image.
[0061] It can be seen that the specific model used to train the sample to be tested is very important for the labeling standard of generating test samples. Therefore, after determining the sample to be tested, the server can further determine the model used to train the sample to be tested, determine the business of the model after the application training is completed, and determine the business type of the business.
[0062] The service type can be set as needed, and this specification does not limit it. For example, whether the service needs to be predicted based on a model or classified based on a model, etc. Alternatively, in this specification, the service type can also be text information describing the service content of the service, that is, different services have different service types.
[0063] S102: Based on at least one of the standard samples and historical samples of the business type and the business type of the business, a plurality of groups of test samples and standard annotations of the test samples are generated through a trained detection model as a test question bank for the business type.
[0064] In one or more embodiments of the present specification, after determining the service type, the server can generate test samples and standard labels for the test samples for checking whether the manual labeling is normal through the pre-trained detection model as a test question bank for the service type. This allows the test question bank to be issued later to test whether the labeler is suitable for labeling samples of the service and to promptly identify labeling problems.
[0065] Specifically, the standard annotation of the test sample generated by the server needs to match the requirements of the subsequent training model. For example, if the trained model is used for image classification, the test sample is an image, and its standard annotation is the classification result of the image. Therefore, the server needs to generate the test sample and the standard annotation of the test sample based on the business type of the business.
[0066] Moreover, in this specification, the server also needs to generate test samples and standard annotations of test samples through the detection model based on at least one of the standard samples or historical samples of the business type. Among them, the standard samples of the business type are samples and annotations set in advance according to the business needs and based on manual experience. In addition to being provided to the annotation personnel for reference when performing the annotation task, they are also the "standard answers" for annotating the samples of the business. Usually, the standard samples and the annotations of the standard samples are set by experts in this field. The historical samples are test samples and their annotations that have been generated for testing for the business type in history.
[0067] In one or more embodiments of the present specification, the server may first determine whether a test question bank has been generated, and determine whether a test question bank of the business type exists from each business type corresponding to the generated test question bank. If so, it means that a test question bank has been generated for the business type in the past, and then multiple groups of test samples and standard annotations of the test samples may be generated based on the historical samples of the business type and the business type of the business through a trained detection model. If not, it means that this is the first time that a test sample has been generated for the business type, and then multiple groups of test samples and standard annotations of the test samples may be generated based on the standard samples of the business type and the business type of the business through a trained detection model.
[0068] It should be noted that in this specification, the historical samples or standard samples input to the detection model are used by the detection model to determine what kind of test samples need to be generated and the standard annotations of the test samples. This specification does not limit what algorithm the detection model uses, and it can be set as needed. For example, the detection model is a large language model (LLM) or reinforcement learning from human feedback (RLHF).
[0069] Taking the detection model as LLM as an example, the server can input prompt words into the detection model so that the detection model generates test samples and standard annotations of the test samples according to the prompt words. The prompt words can be determined by the server according to at least one of the standard samples and historical samples, as well as the business type. For example, the standard sample and the business type are input into a preset prompt word template to generate prompt words. The standard sample is "first text" and its annotation is "first annotation". The business type is "extract keywords that can express user intentions". The prompt words can be "generate test samples for training the model (extract keywords that can express user intentions in the text), for example, generate a test sample such as (first text) and generate a standard annotation such as (first annotation) for the test sample". Among them, the brackets in the above prompt words are the parts of the prompt word template used to fill in the business type and standard samples.
[0070] Furthermore, the server can generate multiple groups of test samples and standard annotations of the test samples according to the preset quantity through the detection model, and use the generated test samples and standard annotations of the test samples as the test question bank for the business type to be detected. Of course, the server can also store the test question bank and the correspondence between the test question bank and the business type for use when the test samples need to be generated again in the future or when repeated testing is required.
[0071] S104: Determine a distribution strategy of the test question bank based on the business information of the marked business and the test question bank.
[0072] In one or more embodiments of the present specification, after generating the test question bank, the server can determine how to distribute each test sample according to the demand to perform the labeling detection. Among them, the test samples in the test question bank are used to be mixed with the samples of the actual labeling task and distributed to each labeler, so the server needs to determine the task information of the labeling task in order to determine the distribution strategy.
[0073] Specifically, the server can first determine the shifts of each labeler and the labeling environment of each labeler based on the business information of the labeling business. Among them, the labeling environment may include the time period for the labeler to label, the equipment environment for the labeler to label, etc. For example, whether the labeler uses a mobile terminal or a personal computer for labeling, and under what system environment the labeling is performed, etc. The shifts of the labelers indicate the working time of each labeler and which samples will be assigned for labeling. Based on the shift information, the server can determine the working hours of each labeler in order to determine how to distribute each test sample.
[0074] Afterwards, the server may determine the distribution strategy of the test question bank according to the shift schedule of each annotator and the annotation environment of each annotator.
[0075] It should be noted that the server can determine the distribution strategy according to the needs. For example, if each labeler is conducting the test for the first time, it is necessary to detect whether the labeling results of each labeler are normal in different environments as much as possible. Then, the server can determine the distribution strategy of the test question bank with the goal of each labeler labeling the test samples in different labeling environments. Among them, the server can use existing optimization algorithms to determine the distribution strategy, such as the optimal path algorithm, evolutionary algorithm, ant colony algorithm, etc. The server only needs to determine when each test sample in the test question bank needs to be distributed to which labeler. When the labeler performs the labeling task, the server adds the test sample to the sample distributed to the labeler for testing according to the determined sending time of the test sample.
[0076] S106: Distribute the test samples according to the distribution strategy, and determine the detection results according to the manual annotation results of the test samples and the standard annotations of the test samples.
[0077] In one or more embodiments of the present specification, the server may ultimately distribute the test samples to the labelers according to the distribution strategy, and determine the detection results according to the testers' manual labeling results of the test samples and the standard labeling of the test samples.
[0078] Specifically, the server mixes the test samples with the samples of the actual labeling task according to the distribution strategy, distributes them to the labelers, and monitors the manual labeling results of each test sample. If the labeler's labeling result is different from the standard labeling, it means that the labeler has made an error in labeling, which is stored as the detection result. The server can then immediately identify the labeler with labeling problems and discover the errors in a timely and effective manner.
[0079] Of course, there may be a situation where the standard labeling of the test sample is inaccurate, in which case the stored test results may be subsequently sent to a designated terminal for review by a corresponding user, for example, by an expert who labels the standard sample.
[0080] Optionally, the server may calculate the number of incorrectly labeled questions for each labeler based on the test samples that the labeler has manually labeled incorrectly. When the number of incorrectly labeled questions reaches a preset value, the labeling task for the labeler is stopped. In other words, when the number of incorrectly labeled test samples by the labeler reaches a preset value, it is determined that the labeler is not suitable for labeling, and the labeler is stopped from performing the labeling task, thereby reducing the noise in the labeled samples.
[0081] Afterwards, the server can also send a prompt message to the designated terminal to remind the annotator that there is a problem. The annotator can then be trained and re-assigned the annotation task to the annotator after he passes the test question bank. The annotator with problems is quickly removed and allowed to work after passing the training.
[0082] Of course, it should be noted that in order to achieve the above functions, it is necessary to ensure that the number of test samples assigned by each annotator is greater than the preset value, and in order to ensure the accuracy of the detection, the number of test samples assigned needs to be much greater than the preset value. For example, according to the preset ratio and preset value, the number of test samples to be assigned is determined. If the preset value is 10 test samples and the ratio is 50%, the number of test samples assigned is determined to be 20, that is, when 10 of the 20 assigned test samples are incorrectly labeled, there is a problem with the annotator, and the annotator should be stopped from performing the labeling task.
[0083] based on Figure 1 The sample annotation detection method shown can generate a test sample and a test question bank consisting of standard annotations of the test sample by annotating the standard sample of the business type or the historical sample of the business type, and determine the distribution strategy of the test sample based on the annotated business and the test question bank. Finally, based on the difference between the manual annotation results of the test sample and the standard annotation, determine the detection result of the manual annotation. Through the trained model, test samples are generated based on historical samples or standard samples for testing manual annotation, so as to achieve targeted detection of error-prone situations of manual annotation and improve the accuracy of detection. At the same time, based on the task information of the annotation task, the distribution strategy of the test sample is determined, and by monitoring the annotation results of each test sample, the problems of manual annotation are timely determined.
[0084] In addition, in one or more embodiments of the present specification, in step S100, the server may determine the service type in response to a detection target input by a user.
[0085] Specifically, the server may receive a detection target input by a user, and determine the business type that matches the detection target based on the detection target. Wherein, the detection target may be a business identifier, and the server may determine a specific business based on the business identifier, and use the business type of the specific business as the business type that matches the detection target. For example, when a user finds that the probability of a business execution exception is high, the server may perform a detection process to confirm whether the business exception is related to the model training sample, so as to eliminate the exception in a timely manner.
[0086] Alternatively, the detection target may also be the identification of the person who needs to detect the annotation. The server may then determine the model trained on the samples annotated by the annotator in the recent period, and determine the business type that matches the detection target based on the business type of the business to which the model is applied.
[0087] Furthermore, the detection target may also include the number of test samples. When the test samples are generated in step S102, the number may also be input into the detection model, and the detection model may generate the number of test samples and the standard annotations of the test samples accordingly. In other words, the number of test samples generated by the detection model may be limited through the process of training the detection model, or through the business process.
[0088] Optionally, in step S102, when the server determines that a test question bank of the business type has been generated in the past, the server can determine historical samples in which the labelers are prone to make mistakes from the historical samples, and generate targeted test samples. In other words, for situations where the labelers are not prone to making mistakes, there is no need to conduct further testing, while situations where labeling errors are prone to occur require more attention, so targeted test samples can be generated.
[0089] Specifically, first, the server can determine the historical samples with manual labeling errors based on the detection results of the historical samples of the business type. Since the historical samples are usually mixed in the labeling tasks and distributed to the labelers after they are generated, the detection results of the historical samples can be determined, and the historical samples with manual labeling errors by the labelers can be determined from them as the labelers' "wrong question set".
[0090] Secondly, the server can determine the prompt text for generating wrong questions based on the historical samples with manual labeling errors and the business type of the business, and generate multiple groups of test samples and standard labels for the test samples through the trained detection model. The server can determine the error labels given by the labelers, the standard labels of the historical samples, and the historical samples based on the detection results of the historical samples with labeling errors, and then determine the prompt words through the prompt word template in combination with the business type, and input the prompt words into the detection model so that the detection model generates each test sample and the standard label of the test sample.
[0091] For example, let's continue to assume that the detection model is trained based on the LLM model. The historical sample is the "first text", its standard annotation is the "first annotation", the wrong annotation is the "second annotation", and the business type is "extracting keywords that can express user intentions". The prompt word can be "generate test samples for training models (extracting keywords in text that can express user intentions), and the labelers are prone to label the test samples of (first text) as (second annotation), while the correct annotation is (first annotation). Please generate test samples that can detect annotation errors for this error". Among them, the brackets in the above prompt words are the contents determined in the previous steps such as filling in the business type in the prompt word template.
[0092] Furthermore, in one or more embodiments of the present specification, since the labeler may make more than one incorrect labeling and the errors may also have tendencies, in order to reduce the amount of input data and improve the efficiency of the detection model in identifying errors, the server may also determine the error question analysis results based on the sample characteristics of historical samples with manual labeling errors.
[0093] Specifically, the server may first determine each historical sample with manual labeling errors, and extract the sample features of each historical sample. Then, the historical samples are clustered according to the sample features of each historical sample, and for each cluster class, the common features of the historical samples with labeling errors in the cluster class are determined as the analysis results. Alternatively, the server may also determine the difference between the incorrect labeling and the standard labeling for the incorrect labeling of the historical samples, and determine the features of the incorrect labeling according to the differences.
[0094] Subsequently, the server may determine the prompt text for generating wrong questions based on the wrong question analysis results and the business type of the business, and generate multiple groups of test samples and standard annotations of the test samples through the trained detection model.
[0095] For example, let's continue to assume that the detection model is trained based on the LLM model. The feature extracted based on the incorrect annotations of each historical sample is "second text", and the business type is "extracting keywords that can express user intentions". The prompt word can be "generate test samples for training models (extracting keywords in texts that can express user intentions), and labelers are prone to labeling errors for samples with (second text) features. Please generate test samples that can detect labeling errors for such errors." Among them, the brackets in the above prompt words are the parts of the prompt word template used to fill in the business type and the results of the wrong question analysis.
[0096] Optionally, in step S102, when the server determines that the test question bank of the business type has not been generated in history, the server can generate targeted test samples based on the standard samples given by the business data of the normal business execution. In other words, if there are no historical samples, the experts can provide standard samples for detection and the annotations of the standard samples so that the detection model can generate test samples based on the "standard".
[0097] Specifically, the server may first determine historical service data that is executed normally from various historical service data of the service.
[0098] Secondly, based on the historical business data of normal execution, determine the standard sample and the annotation of the standard sample. Among them, normal execution refers to historical business without errors or abnormalities. If the historical business of this business type also applies the model during the execution process, then based on the historical business data, the data input into the model and the data output by the model during the execution of the historical business can be determined as the standard sample and the annotation of the standard sample respectively.
[0099] Finally, the server can determine the prompt text for generating wrong questions based on the standard samples, the annotations of the standard samples and the business type of the business, and generate multiple groups of test samples and standard annotations of the test samples through the trained detection model.
[0100] Optionally, in one or more embodiments of the present specification, in step S106, the server may also determine, for each test sample, that the annotator has annotated the test sample incorrectly when the annotator's annotation result is inconsistent with the standard annotation of the test sample. After all the test samples have been annotated, based on the test samples with incorrect annotations, the test samples with incorrect annotations and / or the annotators with incorrect annotations are counted as the detection result. In other words, it is necessary not only to count the annotators with annotation problems, but also to count the test samples that the annotators are prone to mislabeling.
[0101] The server may then repeat the above steps S102 to S106 based on the test samples that are easily mislabeled by the labeling personnel and the business type to continue the detection.
[0102] Optionally, in one or more embodiments of the present specification, the server monitoring the labeling results of each test sample will significantly increase the operating pressure of the server. Therefore, the service area can also determine a sampling strategy and perform random inspections based on the sampling strategy.
[0103] Specifically, the server may monitor the completion status of the annotation business, which includes at least one of the hit probability of the test samples in the test question bank, the hit probability of the test samples of each annotator, and the total amount of annotations.
[0104] Afterwards, the server can determine the target sampling strategy that matches the completion status from the preset candidate sampling strategies according to the completion status of the labeling business. Candidate sampling strategies may include: more sampling for more mistakes, more sampling for more work, and full sampling for all staff. Among them, more sampling for more mistakes means increasing the probability of sampling for labelers who have made labeling errors in the sampling. In other words, the more times the labeler makes labeling errors, the greater the probability of sampling the labeler's labeling results, thereby improving the detection of "error-prone" personnel. More sampling for more work means increasing the probability of sampling for labelers with large labeling volumes. In other words, the more samples the labeler labels, the greater the probability of sampling the labeler's labeling results, thereby avoiding missed detections. At the same time, since the greater the number of labeling, the greater the impact will be after problems occur, the detection intensity should be increased.
[0105] Then, the server can select the test samples annotated by the annotator according to the target selection strategy.
[0106] Finally, the test results are determined based on the labeling results of the selected test samples and the standard labeling. Since each test sample has a standard labeling, it is possible to quickly determine whether a large number of data labels are incorrect after sampling, so that problems such as abnormal labeling personnel operations, abnormal sample data, and abnormal labeling standards can be quickly perceived in the real-time sampling process.
[0107] Optionally, in one or more embodiments of the present specification, in order to further ensure the overall reliability of the test results, the server may also review the test results. Specifically, the server may send the annotation results and the corresponding test samples to the designated terminal for review according to a preset ratio. In other words, the experts at the designated terminal may review the generated test samples and the standard annotations respectively to determine whether there are any errors in the test samples.
[0108] Alternatively, in order to reduce the workload of manual review and improve review efficiency, the server may also determine some test samples and their annotation results from the test samples whose annotation results are different from the standard annotations, and then send the determined test samples and their annotation results to the designated terminal for review.
[0109] In addition, in one or more embodiments of the present specification, the detection model may further include: a generation branch network and an allocation branch network, wherein the generation branch network is used to generate a test sample in step S102, and the allocation branch network is used to determine a distribution strategy in step S104. The generation branch network and the allocation branch network can be trained separately.
[0110] Specifically, each branch network of the detection model can be obtained by training using the architecture of supervised fine-tuning (SFT) + RLHF. The server can input the generated branch network of the trained detection model according to at least one of the standard samples and historical samples of the business type, as well as the business type of the business, to obtain multiple groups of test samples and their standard annotations. When determining the distribution test, the business information of the annotated business and the test question bank are input into the distribution branch network of the trained detection model to determine the distribution strategy of the test question bank.
[0111] Optionally, in one or more embodiments of the present specification, the detection model may further include: a sampling branch network, and when performing sampling, the test samples that need to be sampled are determined through the sampling branch network.
[0112] Specifically, the server can monitor the completion of the annotation service, and the completion status can include at least one of the aforementioned various hit probabilities and the total amount of annotations. Of course, the completion status also includes other contents that will not be repeated in this specification, and reference can be made to the aforementioned contents.
[0113] Afterwards, the server can input the completion status of the labeling business into the sampling branch network of the trained detection model to determine the target sampling strategy output by the sampling branch network.
[0114] Then, according to the target selection strategy, the test samples annotated by the annotators are selected.
[0115] Finally, the test results are determined based on the annotation results of the selected test samples and the standard annotations.
[0116] Furthermore, in one or more embodiments of this specification, the following method may be used to train the detection model, wherein:
[0117] First, obtain a pre-trained detection model, and fine-tune the detection model through supervised training according to historical business data. This process can refer to the SFT training process, that is, obtain the basic model to be trained through pre-training (Pre-Training) and supervised fine-tuning (Supervised Fine-Tuning, SFT).
[0118] In the pre-training stage, the server can train on an unlabeled text corpus through a large neural network (such as the Transformer architecture) to learn the basic structure, vocabulary, and expression patterns of the language. This step usually uses self-supervised learning methods, such as Masked Language Modeling (MLM), to learn contextual relationships by predicting masked (marked) words.
[0119] In the supervised fine-tuning phase, the server can use historical samples or standard samples related to the business type to fine-tune the pre-trained model. This helps the detection model learn the domain-specific knowledge or dialogue patterns of the business type and produce more expected test samples.
[0120] Afterwards, the server may send the historical annotation results of the annotation service to the designated terminal, and receive feedback results of the historical service data returned by the designated terminal, wherein the feedback results are feedback on whether the historical annotation results are correct.
[0121] Specifically, human feedback collection (Human Feedback Collection) is required in this step, that is, through expert evaluation, the correctness of different annotation results is determined, so that the reward function can be determined based on the expert evaluation later for the final reinforcement learning. For example, the server can send the historical annotation results of the annotation business of this business type and the annotation samples corresponding to the historical annotation results to the designated terminal. Then the expert can compare or rate the historical annotation results through the designated terminal, or provide positive or negative feedback on the historical annotation results. That is, the feedback result of the historical business data returned by the designated terminal is received, and the feedback result is feedback on whether the historical annotation result is correct.
[0122] The feedback received by the server reflects the experts' preferences for which annotations are more appropriate, more natural, or more ethical.
[0123] Then, the server can determine the reward function based on the feedback results of the historical annotation results.
[0124] Specifically, the server can train a classification or regression model based on the feedback results as a reward function, which can predict the possible ratings given by humans based on the model output. This allows the model to iterate itself in the subsequent reinforcement learning stage without relying on manual feedback for each decision.
[0125] Finally, based on the reward function, the detection model is trained by a reinforcement learning method.
[0126] Specifically, the server can integrate the reward function into the reinforcement learning framework, and the model learns to take actions (generate test samples and their standard annotations) to maximize the cumulative reward by interacting with the environment, usually with the reward model. This usually involves policy optimization algorithms, such as the proximal policy optimization algorithm (PPO) or the REINFORCE algorithm, to update the model parameters.
[0127] In addition, it should be noted that in the embodiments of this specification, each branch network can be trained separately and is not affected by other branch networks when used. Figure 2 A schematic diagram of the detection model provided for implementation of this specification, Figure 2 The left side in the middle represents the data required for training the detection model, and the bottom is a large model pre-trained with labeled data. Knowledge learning, knowledge understanding, and knowledge reasoning can be implemented using various existing methods in the field of machine learning. The middle is trained through SFT+RLHF, and the top is the branch network obtained through detection model training, which can support the execution of different business contents.
[0128] Optionally, this specification also provides a flow chart of a sample labeling detection method, such as Figure 3 shown. Figure 3 It is divided into the detection system, which executes Figure 1 The device of the process, wherein the database is a device that provides the data required for training, can be in the detection system or a third-party device, and the detection model is trained through the data provided by the database to obtain each branch network and used in different steps of the detection process. When performing the labeling task, the test sample is generated, and then the distribution strategy is generated, and then distributed to different labelers. According to the labeling results and the generated sampling strategy, sampling is carried out, and at the same time, it is sent to the designated terminal for re-inspection, and the detection results are stored in the database, and the detection model can be further updated afterwards.
[0129] See also Figure 4 , is a schematic diagram of a sample labeling detection device provided in an embodiment of this specification. Figure 4 As shown, the sample annotation detection device can be implemented as all or part of the electronic device through software, hardware or a combination of both. According to some embodiments, the sample annotation detection includes a type determination module 401, a question generation module 402, a question distribution module 403, and a detection module 404, which specifically include:
[0130] Type determination module 401, determining the business type to which the model trained by the sample to be detected is applied;
[0131] The test question generating module 402 generates multiple groups of test samples and standard annotations of the test samples as a test question bank of the service type according to at least one of the standard sample and the historical sample of the service type and the service type of the service through the trained detection model;
[0132] The test question distribution module 403 determines a distribution strategy of the test question bank based on the business information of the marked business and the test question bank;
[0133] The detection module 404 distributes the test samples according to the distribution strategy, and determines the detection results according to the manual labeling results of the test samples and the standard labeling of the test samples.
[0134] Optionally, a test question generation module 402 determines whether there is a test question bank for the business type based on the test question banks that have been generated for each business type. If so, multiple groups of test samples and standard annotations of the test samples are generated based on historical samples of the business type and the business type of the business through a trained detection model. If not, multiple groups of test samples and standard annotations of the test samples are generated based on standard samples of the business type and the business type of the business through a trained detection model.
[0135] Optionally, a wrong question generation module 402 is used to determine historical samples with artificially marked errors based on the detection results of historical samples of the business type, determine wrong question analysis results based on sample characteristics of the historical samples with artificially marked errors, determine prompt text for generating wrong questions based on the wrong question analysis results and the business type of the business, and generate multiple groups of test samples and standard annotations for the test samples through the trained detection model.
[0136] Optionally, a wrong question generation module 402 determines historical business data of normal execution from various historical business data of the business, determines standard samples and annotations of the standard samples based on the historical business data of normal execution, determines prompt text for generating wrong questions based on the standard samples, the annotations of the standard samples and the business type of the business, and generates multiple groups of test samples and standard annotations of the test samples through the trained detection model.
[0137] Optionally, the question distribution module 403 determines the schedule of each annotation personnel and the annotation environment of each annotation personnel according to the business information of the annotation business, and determines the distribution strategy of the test question bank according to the schedule of each annotation personnel and the annotation environment of each annotation personnel, with the goal of each annotation personnel annotating the test samples under different annotation environments.
[0138] Optionally, the detection module 404 determines, for each test sample, that the labeler has incorrectly labeled the test sample when the labeler's labeling result is inconsistent with the standard labeling of the test sample, and based on the incorrectly labeled test samples, counts the incorrectly labeled test samples and / or incorrectly labeled labelers as the detection result.
[0139] Optionally, the detection module 404 calculates the number of incorrectly marked questions for each labeler based on the test samples that the labeler has manually marked incorrectly, and stops executing the labeling task for the labeler when the number of incorrectly marked questions reaches a preset value.
[0140] Optionally, the detection module 404 monitors the completion status of the labeling business, the completion status including at least one of the hit probability of the test samples in the test question bank, the hit probability of the test samples by each labeler, and the total labeling amount; and according to the completion status of the labeling business, determines a target sampling strategy that matches the completion status from preset candidate sampling strategies; according to the target sampling strategy, selects the test samples labeled by the labeling personnel; and determines the detection results according to the labeling results of the selected test samples and the standard labeling.
[0141] Optionally, the device further includes: a re-examination module 405, which determines some test samples and their annotation results from the test samples whose annotation results are different from the standard annotation according to a preset ratio, and sends the determined test samples and their annotation results to a designated terminal for review.
[0142] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.
[0143] The present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded and executed by a processor as described above. Figure 1 to Figure 3 The method of the embodiment shown in the figure can be specifically executed by referring to Figure 1 to Figure 3 The specific description of the illustrated embodiment will not be repeated here.
[0144] The present specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figure 1 to Figure 3 The method of the embodiment shown in the figure can be specifically executed by referring to Figure 1 to Figure 3 The specific description of the illustrated embodiment will not be repeated here.
[0145] The embodiments of this specification also provide Figure 5 The structural diagram of the electronic device shown in FIG. Figure 5 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned voice activity detection method.
[0146] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0147] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0148] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0149] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0150] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0151] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0152] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0155] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0156] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0157] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0158] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0159] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0160] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0161] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0162] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.
Claims
1. A method for detecting sample annotation, the method comprising: Determine the business type to which the model trained by the sample to be tested is applied; According to at least one of the standard sample and the historical sample of the service type and the service type of the service, a plurality of groups of test samples and standard annotations of the test samples are generated through the trained detection model as a test question bank of the service type; Determining a distribution strategy of the test question bank based on the business information of the marked business and the test question bank; The test samples are distributed according to the distribution strategy, and the detection results are determined according to the manual annotation results of the test samples and the standard annotations of the test samples.
2. The method according to claim 1, generating multiple groups of test samples and standard annotations of the test samples through a trained detection model according to at least one of the standard sample and the historical sample of the service type and the service type of the service, specifically comprising: According to the generated test question bank of each business type, determining whether there is a test question bank of the business type; If yes, then generate multiple groups of test samples and standard annotations of the test samples based on the historical samples of the service type and the service type of the service through the trained detection model; If not, multiple groups of test samples and standard annotations of the test samples are generated according to the standard samples of the business type and the business type of the business through the trained detection model.
3. The method according to claim 2, generating multiple groups of test samples and standard annotations of the test samples through the trained detection model according to the historical samples of the service type and the service type of the service, specifically comprising: According to the detection results of the historical samples of the business type, determine the historical samples with manual labeling errors; Determine the analysis results of wrong questions based on the sample characteristics of historical samples with manual annotation errors; According to the wrong question analysis results and the business type of the business, a prompt text for generating wrong questions is determined, and multiple groups of test samples and standard annotations of the test samples are generated through the trained detection model.
4. The method according to claim 2, generating multiple groups of test samples and standard annotations of the test samples through the trained detection model according to the standard samples of the service type and the service type of the service, specifically comprising: Determine historical business data of normal execution from various historical business data of the business; Determine standard samples and annotations of the standard samples based on historical business data of normal execution; According to the standard samples, the annotations of the standard samples and the business type of the business, a prompt text for generating wrong questions is determined, and multiple groups of test samples and standard annotations of the test samples are generated through the trained detection model.
5. The method according to claim 1, determining a distribution strategy of the test question bank based on the business information of the marked business and the test question bank, specifically comprising: Determine the shifts of each annotation staff and the annotation environment of each annotation staff according to the business information of the annotation business; According to the shifts of the labeling personnel and the labeling environments of the labeling personnel, with the goal of the labeling personnel labeling the test samples in different labeling environments, a distribution strategy of the test question bank is determined.
6. The method according to claim 1, determining the test result according to the manual labeling result of the test sample and the standard labeling of the test sample, specifically comprising: For each test sample, when the labeling result of the labeler is inconsistent with the standard labeling of the test sample, it is determined that the labeler has incorrectly labeled the test sample; According to each test sample with incorrect labeling, the test samples with incorrect labeling and / or the labelers with incorrect labeling are counted as the detection result.
7. The method of claim 6, further comprising: For each annotator, the number of incorrectly annotated questions by the annotator is calculated cumulatively based on the test samples that the annotator has annotated incorrectly. When the number of incorrectly marked questions reaches a preset value, the marking task is stopped for the marking person.
8. The method of claim 1, determining the test result based on the manual labeling result of the test sample and the standard labeling of the test sample, specifically comprising: Monitoring the completion of the annotation business, wherein the completion includes at least one of the hit probability of the test samples in the test question bank, the hit probability of the test samples by each annotator, and the total amount of annotations; According to the completion status of the labeling business, determine a target sampling strategy that matches the completion status from preset candidate sampling strategies; According to the target selection strategy, select the test samples annotated by the annotator; The test results are determined based on the labeling results of the selected test samples and the standard labeling.
9. The method according to claim 6 or 8, further comprising: According to a preset ratio, determine some test samples and their annotation results from each test sample whose annotation results are different from the standard annotation; The determined test samples and their annotation results are sent to the designated terminal for review.
10. A sample labeling detection device, the device comprising A type determination module determines the business type to which the model trained by the sample to be tested is applied; A test question generation module generates multiple groups of test samples and standard annotations of the test samples as a test question bank for the business type through a trained detection model according to at least one of the standard samples and historical samples of the business type and the business type of the business; A test question distribution module, which determines a distribution strategy of the test question bank based on the business information of the marked business and the test question bank; The detection module distributes the test samples according to the distribution strategy, and determines the detection results according to the manual labeling results of the test samples and the standard labeling of the test samples.
11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.
12. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 9.
13. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 9 are implemented.
