A text processing method and related device

Abstract

The present application provides a text processing method and related device; the method includes: determining a first quality parameter of a first text label of the first text based on a first parameter of the first text, the first text label being obtained by annotating a first object; determining a second quality parameter of the first text label based on the first parameter, an attribute parameter of the first object, a time parameter of the first object annotating, a cognitive state parameter of the first object annotating, and an emotional parameter of the first object annotating; determining a third quality parameter of the first text label based on the cognitive state parameter; and determining a quality verification result of the first text label based on the first quality parameter, the second quality parameter, and the third quality parameter. Through the present application, the accuracy of the quality verification of text labels can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a text processing method and related devices. Background Art

[0002] Natural language processing models are usually trained on large amounts of text data, which can be annotated text data with corresponding text labels. Text labels are usually obtained by annotators who annotate the text data. However, annotators may make mistakes during the annotation process, which may affect the quality of the text labels. Therefore, it is necessary to perform quality verification on the text labels of the annotators. However, in the related art, there is a problem of low accuracy in the quality verification of text labels. Summary of the Invention

[0003] The embodiments of the present application provide a text processing method and related devices, which can improve the accuracy of quality verification of text labels.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides a text processing method, the method comprising:

[0006] Determining a first quality parameter of a first text label of the first text based on a first parameter of the first text, where the first text label is obtained by annotating the first object;

[0007] Determining a second quality parameter of the first text label based on the first parameter, the attribute parameter of the first object, the time parameter of the first object performing the annotation, the cognitive state parameter of the first object performing the annotation, and the emotion parameter of the first object performing the annotation;

[0008] determining a third quality parameter of the first text label based on the cognitive state parameter;

[0009] A quality verification result of the first text label is determined based on the first quality parameter, the second quality parameter, and the third quality parameter.

[0010] The present invention provides a text processing device, including:

[0011] A first quality module, configured to determine a first quality parameter of a first text label of the first text based on a first parameter of the first text, where the first text label is obtained by annotating the first object;

[0012] a second quality module, configured to determine a second quality parameter of the first text label based on the first parameter, an attribute parameter of the first object, a time parameter of the first object performing the annotation, a cognitive state parameter of the first object performing the annotation, and an emotion parameter of the first object performing the annotation;

[0013] a third quality module, configured to determine a third quality parameter of the first text label based on the cognitive state parameter;

[0014] A result determination module is configured to determine a quality verification result of the first text label based on the first quality parameter, the second quality parameter, and the third quality parameter.

[0015] An embodiment of the present application provides an electronic device, comprising:

[0016] a memory for storing computer-executable instructions or computer programs;

[0017] The processor is used to implement the text processing method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the text processing method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the text processing method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects: quality parameters are determined from the dimensions of the difficulty of annotating the first text, the dimension of the cognitive state of the first object during annotation, and the dimension of the multi-dimensional behavior and state of the first object during annotation, thereby obtaining quality parameters of different dimensions; and then, quality parameters of multiple dimensions are simultaneously combined, that is, the first quality parameter, the second quality parameter, and the third quality parameter are simultaneously combined to collaboratively verify the quality of the first text label in multiple dimensions, so as to comprehensively cover various factors that affect the quality of the first text label, reduce the deviation caused by quality verification in a single dimension, thereby achieving accurate quality assessment of the first text label, obtaining more accurate quality verification results, and thus improving the accuracy of quality verification of text labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of an application environment of the text processing method provided in an embodiment of the present application;

[0022] Figure 2 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 1 ;

[0023] Figure 3 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 2 ;

[0024] Figure 4 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 3 ;

[0025] Figure 5 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 4 ;

[0026] Figure 6 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 5 ;

[0027] Figure 7 Schematic diagram of the process of calculating the influencing factors of the parameters provided in the embodiment of the present application;

[0028] Figure 8 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0030] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] In the following description, the terms "first\second\third\fourth" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third\fourth" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0032] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or portion of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal. It can be implemented in whole or in part using software, hardware (such as processing circuits or memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the functionality of the module or unit.

[0033] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0034] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0035] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0036] 1) Annotation refers to the process of labeling, classifying, or annotating text. The purpose of annotation is to assign specific semantic information or structured text labels to the text to facilitate subsequent model training based on the text labels. For example, models used for natural language processing tasks (such as sentiment recognition and entity recognition) can be trained based on text labels.

[0037] 2) Text labels are the result or output of annotating text, used to quantify or categorize certain attributes of the text. In this embodiment, text labels can refer to multiple categories, such as sentiment: positive / neutral / negative. Text labels can also refer to multiple labels in sequence annotation, such as names of people, places, and organizations.

[0038] 3) Cognitive state parameters are comprehensive indicators that describe the annotator’s cognitive activities and neural mechanisms during the annotation process. Therefore, cognitive state parameters reflect the annotator’s cognition of the text from a biological perspective.

[0039] Natural language processing models are typically trained on large amounts of text data, which can be annotated with corresponding text labels. These labels are typically created by human annotators. However, human annotators can make errors during the annotation process, which can affect the quality of the labels. Therefore, quality verification of the labels is necessary.

[0040] In related art, text label quality verification is typically implemented based on static metrics, such as the Kappa coefficient or F1 coefficient. However, accurate quality verification based solely on static metrics is difficult, resulting in low accuracy in text label quality verification.

[0041] In addition, in the related art, the accuracy of the labelers in different time periods is not analyzed, making it difficult to guide the labelers' labeling work based on the accuracy to further ensure the labeling quality of the text labels.

[0042] In response to at least one of the above-mentioned problems existing in the related art, an embodiment of the present application provides a text processing method and related device, which includes: determining a first quality parameter of a first text label of the first text based on a first parameter of the first text, where the first text label is obtained by annotating a first object; determining a second quality parameter of the first text label based on the first parameter, an attribute parameter of the first object, a time parameter of the first object annotation, a cognitive state parameter of the first object annotation, and an emotional parameter of the first object annotation; determining a third quality parameter of the first text label based on the cognitive state parameter; and determining a quality verification result of the first text label based on the first quality parameter, the second quality parameter, and the third quality parameter. In this way, a more accurate quality verification result is determined for the first text label, thereby improving the accuracy of the quality verification of the text label.

[0043] In order to better understand the text processing method and related devices provided in the embodiments of the present application, the application environment applicable to the embodiments of the present application is described below.

[0044] See also Figure 1 , Figure 1 As an implementation method, the audio processing method of the embodiment of the present application can be applied to an electronic device, wherein the electronic device can be such as Figure 1 The server 110 shown in FIG. 1 can be connected to the terminal 120 via a network 130. The network 130 is used to provide a medium for a communication link between the server 110 and the terminal 120. The network 130 can include various connection types, such as wired communication links, wireless communication links, etc., which are not limited in this embodiment of the present application.

[0045] It should be understood that Figure 1 The server 110, terminal 120, and network 130 are merely illustrative. Any number of servers, networks, and terminals may be used as needed. For example, the server 110 may be a physical server or a server cluster consisting of multiple servers, and the terminal 120 may be a smartphone, tablet computer, desktop computer, laptop computer, smartwatch, or other device. It will be appreciated that in embodiments of the present application, multiple terminals 120 may be allowed to access the server 110 simultaneously.

[0046] The embodiment of the present application can be implemented by a server. For example, for the text labels annotated by the annotators for the text, the server 110 will obtain the quality verification results of the text labels based on the text processing method provided by the embodiment of the present application, so as to subsequently determine whether the text labels need to be re-annotated based on the quality verification results.

[0047] The text processing method provided in the embodiment of the present application can be applied to various text label quality verification scenarios, for example, the quality verification of text labels of call texts, the quality verification of text labels of user comments, etc. The following describes the scenarios in which the text processing method provided in the embodiment of the present application can be applied.

[0048] 1) Quality verification of text labels of call texts. For example, the server obtains the call intention labels annotated by the annotators for the call texts of telemarketing calls, and then determines the quality verification results for the call intention labels based on the text processing method provided in the embodiments of the present application, so as to determine whether the call text needs to be re-annotated based on the quality verification results.

[0049] 2) Quality verification of the text labels of user comments. For example, the server obtains the emotion category labels annotated by the annotators for the user comments on a certain product, and then determines the quality verification results for the text labels based on the text processing method provided in the embodiment of the present application, so as to determine whether to use the user comments and their emotion category labels to train the emotion recognition model based on the quality verification results.

[0050] The following describes the text processing method provided in the embodiments of the present application. As previously mentioned, the electronic device implementing the text processing method in the embodiments of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0051] See also Figure 2 , Figure 2 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 1 , will combine Figure 2The steps shown are explained, Figure 2 The main body of the step is the electronic device.

[0052] Step 101: Determine a first quality parameter of a first text label of the first text based on a first parameter of the first text.

[0053] In an embodiment of the present application, the first text label is obtained by annotating the first object. The electronic device first obtains the first text label annotated by the first object for the first text, and simultaneously obtains the first parameter of the first text. Then, combined with the first parameter of the first text, the electronic device performs a quality analysis on the first text label based on the dimension of the difficulty of annotating the first text, thereby obtaining the first quality parameter of the first text label.

[0054] It should be noted that the first text refers to any text that needs to be labeled, which can be a transcript of a call, a review of a product, a Chinese translation of a foreign text, etc., and the embodiments of this application are not limited here. The first object refers to the annotator who annotates the first text, which can be any annotator. The first text label is obtained by the first object annotating the first text based on its understanding of the label system.

[0055] The first parameter is a comprehensive parameter that measures the complexity of the text during the annotation process. It can be determined by the length parameter of the first text and the parameter used to characterize the difficulty of determining the text label, namely the level parameter. Therefore, the first parameter can reflect the impact of the text on the annotator's cognitive load. The first parameter can be a parameter in the form of a score, such as difficulty 90 points, difficulty 50 points, or a parameter in the form of a level, such as difficulty level 1, difficulty level 2, and so on. The first parameter can be obtained by analyzing the first text. The process of determining the first parameter is described below.

[0056] It should be noted that cognitive load refers to the degree of cognitive resource consumption incurred by the first subject when processing information, thereby reflecting the total amount of cognitive abilities, such as attention and decision-making, required by the first subject when annotating. The greater the complexity of the first text during annotation, the higher the cognitive load. The higher the cognitive load, the more prone to errors in the annotation of the first subject, and thus the lower the quality parameter of the first text label. Therefore, the electronic device can analyze the quality parameter of the first text label based on the annotation difficulty dimension of the text itself, combining the first parameter of the first text. Thus, the first quality parameter represents the quality of the first text label based on the annotation difficulty dimension of the text.

[0057] The first quality parameter can be a parameter in the form of a score, such as quality 80 points, quality 60 points, etc., or it can be a parameter in the form of a level, such as quality level 1, quality level 2, etc., which is not limited in the embodiment of the present application.

[0058] In an embodiment of the present application, the first parameter includes at least one of a level parameter and a length parameter of the first text. The level parameter characterizes the difficulty of distinguishing different text labels, that is, the complexity of judging which text label the first text should belong to during the annotation process. Since the first text may have semantic overlap with multiple text labels, that is, the first text may conform to multiple text labels at the same time, thereby affecting the quality of the first text label, therefore, in an embodiment of the present application, the first quality parameter of the first text label can be determined in combination with the level parameter of the first text. The length parameter of the first text refers to the number of characters contained in the first text. The more characters contained in the first text, the larger the length parameter of the first text, the more complex its semantics, the more prone to annotation errors, and the quality of the first text label. Therefore, in an embodiment of the present application, the first quality parameter of the first text label can be determined in combination with the length parameter of the first text.

[0059] See also Figure 3 , Figure 3 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 2 In some embodiments of the present application, the first parameter includes: a level parameter and a length parameter of the first text. In this case, Figure 2 Step 101 in the embodiment, i.e., determining a first quality parameter of a first text label of the first text based on a first parameter of the first text, can be implemented by the following process:

[0060] Step 1011: Obtain a first weighting factor corresponding to the level parameter and a second weighting factor corresponding to the length parameter.

[0061] The electronic device first obtains a corresponding weight factor for the level parameter and simultaneously obtains a corresponding weight factor for the length parameter, records the weight factor corresponding to the level parameter as a first weight factor, and records the weight factor corresponding to the length parameter as a second weight factor.

[0062] It should be noted that the first weighting factor can be a preset weighting factor, the value of which can be set by a human expert or randomly. The first weighting factor can also be a weighting factor determined by the first model, the process of determining the first weighting factor of which will be described below. Similarly, the second weighting factor can be a preset weighting factor or a weighting factor determined by the first model, the process of determining the second weighting factor of which will be described below.

[0063] Step 1012: Sum the level parameter and the length parameter using the first weight factor and the second weight factor, and determine the sum as the first quality parameter of the first text label.

[0064] After obtaining the first weight factor and the second weight factor, the electronic device will use the first weight factor and the second weight factor to perform weighted summation on the level parameter and the length parameter of the first text, and directly use the weighted summation result as the first quality parameter of the first text label.

[0065] For example, if the first weight factor is , the second weight factor is , the level parameter is , the length parameter is , then the first quality parameter can be expressed as .

[0066] It can be understood that in an embodiment of the present application, the electronic device can first obtain the first weight factor and the second weight factor, so that the level parameter and length parameter of the first text can be introduced at the same time through the first weight factor and the second weight factor to determine the quality parameter, so that the annotation quality can be judged more comprehensively in the annotation difficulty dimension of the first text, thereby making the first quality parameter more accurate.

[0067] In some other embodiments of the present application, the first parameter includes: a level parameter, in which case, Figure 2 Step 101, i.e., determining the first quality parameter of the first text label of the first text based on the first parameter of the first text, can also be achieved by the following processing: obtaining a first weight factor corresponding to the grade parameter, adjusting the grade parameter according to the first weight factor, and determining the adjustment result as the first quality parameter of the first text label.

[0068] That is, in the embodiment of the present application, the electronic device can also determine the first quality parameter of the first text label using only the grade parameter. This can make the process of determining the first quality parameter simpler, thereby improving the efficiency of determining the first quality parameter.

[0069] Step 102: Determine a second quality parameter of the first text label based on the first parameter, the attribute parameter of the first object, the time parameter of the first object annotation, the cognitive state parameter of the first object annotation, and the emotion parameter of the first object annotation.

[0070] The electronic device obtains the first parameter of the first text, and also obtains the attribute parameters of the first object itself, the time parameter of the first object when annotating the first text, the cognitive state parameter of the first object when annotating the first text, and the emotional parameter of the first object when annotating the first text. Then, the electronic device combines the obtained parameters to analyze the quality parameters of the first text label and determines the obtained quality parameters as the second quality parameters. Since each of the above parameters reflects the multi-dimensional behavior and state of the first object during the annotation process, the second quality parameter can be understood as a quality parameter determined based on the multi-dimensional behavior and state dimensions of the annotation process.

[0071] It should be noted that the attribute parameters of the first object are parameters that provide the static attributes of the first object, thereby reflecting the individual characteristics of the first object. The attribute parameters may include the object identification parameters of the first object, and may also include the annotation quality parameters of the first object. Among them, the object identification parameter refers to the unique identity of the first object. Different objects can be distinguished by the object identification parameter. The object identification parameter may include at least one of the ID of the first object and the name of the first object; the annotation quality parameter is a parameter that characterizes the annotation ability of the first object. The annotation quality parameter may include at least one of the annotation experience value of the first object and the historical accuracy rate of the first object. Here, the annotation experience value can be obtained by the annotation years of the first object, or by the second object scoring the first object. The second object may be an annotation person other than the first object, or may be the superior of the first object; the historical accuracy rate can be obtained by counting the number of text labels annotated by the first object, and the number of correct labels among the annotated text labels.

[0072] The time parameter is a parameter that describes the time-related aspects of the first object's annotation process, and reflects the specific circumstances of the first object's annotation task in the time dimension. The time parameter may include the start annotation time of the first object, that is, the time when the first object starts to perform the annotation task, and may also include the first object's annotation duration for the first text, that is, the time it takes the first object to complete the annotation of the current first text, and may also include the first object's cumulative annotation time after starting the annotation task, etc. Here, the start annotation time, annotation duration, and cumulative annotation time can all be obtained from the annotation system.

[0073] The cognitive state parameter is a comprehensive parameter that describes the cognitive activity and neural mechanism of the first subject during the annotation process. Thus, the cognitive state parameter reflects the first subject's understanding of the first text from a biometric perspective. The cognitive state parameter may include at least one of an eye movement parameter and a brain region processing parameter of the first subject. The eye movement parameter may be obtained by performing eye movement detection on the first subject using an eye movement detector, and the brain region processing parameter may be obtained by performing brain wave detection on the first subject using an electroencephalogram (EEG) detector.

[0074] The emotion parameter is a parameter describing the personal emotion of the first subject during the annotation process. The emotion parameter may include an identification parameter of the emotion category of the first subject. The emotion parameter may be identified from the image information of the first subject during the annotation process. The image information of the first subject during the annotation process may be obtained by the annotation system using a camera to capture an image of the first subject.

[0075] See also Figure 4 , Figure 4 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 3 In some embodiments of the present application, the attribute parameters include: an object identification parameter of the first object and a labeling quality parameter of the first object. In this case, Figure 2 Step 102 in the embodiment, i.e., determining the second quality parameter of the first text label based on the first parameter, the attribute parameter of the first object, the time parameter of the first object being labeled, the cognitive state parameter of the first object being labeled, and the emotion parameter of the first object being labeled, can be achieved by the following processing:

[0076] Step 1021: Calculate a first sub-quality parameter of the first text label based on the object identification parameter, the annotation quality parameter, the time parameter, and the emotion parameter.

[0077] The electronic device simultaneously estimates a quality parameter for the first text label based on the object identification parameter, the annotation quality parameter, the time parameter, and the emotion parameter, and determines the resulting quality parameter as the first sub-parameter quality. It should be noted that since the object identification parameter, the annotation quality parameter, the time parameter, and the emotion parameter all reflect directly observable or directly quantifiable surface characteristics of the first object, the first sub-quality parameter derived based on these parameters can be considered a quality parameter from the perspective of the surface characteristics of the first object.

[0078] It should be noted that, in an embodiment of the present application, the annotation quality parameter may include at least one of the annotation experience value and the historical accuracy of the first object, and the time parameter may include at least one of the annotation start time, the annotation end time, the annotation duration of the first text, and the cumulative annotation duration of the first object. Therefore, the electronic device can arbitrarily extract the sub-parameters involved in the determination of the first sub-quality parameter from the sub-parameters contained in the above parameters, and then obtain the corresponding weight factors for these sub-parameters, as well as the object identification parameters and the emotion parameters, and perform weighted summation on the sub-parameters using the corresponding weight factors, and determine the weighted summation result as the first sub-quality parameter. These weight factors can all be preset weight factors, such as 0.2, or 0.05, etc., or they can be weight factors learned by the third model. The specific implementation process of the third model learning weight factors will be described below.

[0079] For example, in some embodiments, the electronic device may respectively obtain the weight factors corresponding to the object identification parameter, the annotation experience value, the annotation start time, the annotation duration of the first text, and the emotion parameter, and then use the corresponding weight factors to weight and sum the object identification parameter, the annotation experience value, the annotation start time, the annotation duration of the first text, and the emotion parameter to obtain the first sub-quality parameter. Among them, the weight factor corresponding to the object identification parameter may be 0.3, the weight factor corresponding to the annotation experience value may be 0.25, the weight factor corresponding to the annotation start time may be 0.2, the weight factor for the annotation duration of the first text may be 0.15, and the weight factor corresponding to the emotion parameter may be 0.1.

[0080] For another example, in other embodiments, the electronic device may respectively obtain the object identification parameter, the annotation experience value, the historical accuracy rate, the annotation start time, the annotation end time, the annotation duration of the first text, the cumulative annotation duration, and the weight factor of the emotional parameter, and then perform weighted summation using the weight factor to obtain the first sub-quality parameter. Among them, the weight factor corresponding to the object identification parameter may be 0.1, the weight factor corresponding to the annotation experience value may be 0.2, the weight factor corresponding to the historical accuracy rate may be 0.3, the weight factor corresponding to the annotation start time may be 0.1, the weight factor of the annotation end time may be 0.05, the weight factor of the annotation duration of the first text may be 0.15, the weight factor of the cumulative annotation duration may be -0.05, and the weight factor of the emotional parameter may be 0.15.

[0081] Step 1022: Based on the first parameter, the quality parameter, time parameter, emotion parameter, and cognitive state parameter are annotated to calculate a second sub-quality parameter of the first text label.

[0082] The electronic device does not consider the object identification parameter, but instead combines the first parameter, the annotation quality parameter, the time parameter, the emotion parameter, and the cognitive state parameter to perform a quality parameter analysis on the first text label, and determines the resulting quality parameter as the second sub-quality parameter. It should be noted that because the cognitive state parameter represents the intrinsic attributes of the first object in inference or subjective judgment, it reflects information such as the first object's motivation, attitude, and potential ability. Therefore, by excluding the object identification parameter and considering the first parameter and the cognitive state parameter simultaneously on the basis of the annotation quality parameter, the time parameter, and the emotion parameter, a more in-depth analysis of the first object's processing of the first text can be achieved, thereby enabling a more in-depth judgment of the quality of the first text label.

[0083] It should also be noted that the first parameter may include at least one of a level parameter and a length parameter, the annotation quality parameter may include at least one of an annotation experience value and a historical accuracy rate of the first object, the time parameter may include at least one of an annotation start time, an annotation end time, an annotation duration of the first text, and a cumulative annotation duration of the first object, and the cognitive state parameter may include at least one of an eye movement parameter and a brain slice processing parameter. Therefore, the electronic device may arbitrarily extract the sub-parameters involved in determining the second sub-quality parameter from the sub-parameters contained in the above parameters, and then obtain corresponding weight factors for these sub-parameters and the emotional parameter, and perform weighted summation based on the corresponding weight factors, and determine the weighted summation result as the first sub-quality parameter. These weight factors may all be preset weight factors, such as 0.1, or 0.15, etc., or they may be weight factors learned by the fourth model. The specific process of learning the weight factors by the fourth model will be described below.

[0084] For example, in some embodiments, the electronic device may obtain weight factors corresponding to the level parameter, the annotation experience value, the annotation start time, the annotation duration of the first text, the eye movement parameter, and the emotion parameter, respectively, and then weight the above parameters using the corresponding weight factors to obtain the second sub-quality parameter. The weight factor corresponding to the level parameter may be -0.2, the weight factor corresponding to the annotation experience value may be 0.3, the weight factor corresponding to the annotation start time may be 0.15, the weight factor corresponding to the annotation duration of the first text may be 0.2, the weight factor corresponding to the eye movement parameter may be 0.25, and the weight factor corresponding to the emotion parameter may be 0.3.

[0085] For another example, in other embodiments, the electronic device may respectively obtain the level parameter, length parameter, annotation experience value, historical accuracy, annotation start time, annotation end time, annotation time of the first text, cumulative annotation time, eye movement parameter, brain slice processing parameter, and weight factor of the emotion parameter, and then use the weight factor to perform weighted summation on the above parameters to obtain the second sub-quality parameter. Among them, the weight factor corresponding to the level parameter can be -0.15, the weight factor corresponding to the length parameter can be 0.1, the weight factor corresponding to the annotation experience value can be 0.3, the weight factor corresponding to the historical accuracy can be 0.3, the weight factor corresponding to the annotation start time can be 0.05, the weight factor corresponding to the annotation end time can be 0.05, the weight factor corresponding to the annotation time of the first text can be 0.10, the weight factor corresponding to the cumulative annotation time can be -0.1, the weight factor corresponding to the eye movement parameter can be 0.15, the weight factor corresponding to the brain slice processing parameter can be 0.1, and the weight factor corresponding to the emotion parameter can be 0.1.

[0086] Step 1023: Determine the sum of the first sub-quality parameter and the second sub-quality parameter as the second quality parameter of the first text label.

[0087] After obtaining the first sub-quality parameter and the second sub-quality parameter, the electronic device may directly sum the first sub-quality parameter and the second sub-quality parameter to obtain the second quality parameter, or may perform a weighted sum of the first sub-quality parameter and the second sub-quality parameter to obtain the second quality parameter. The weighting of the first sub-quality parameter and the second sub-quality parameter may be preset or randomly determined by the electronic device.

[0088] It can be understood that in the embodiment of the present application, the electronic device can evaluate the quality parameters of the first text label from two perspectives: the surface features of the first object and the deep features of the first object, and then simultaneously fuse the sub-quality parameters of the two perspectives to obtain the final second quality parameter, thereby improving the accuracy of the second quality parameter.

[0089] In some embodiments of the present application, Figure 2 Step 102, i.e., determining the second quality parameter of the first text label based on the first parameter, the attribute parameter of the first object, the time parameter for labeling the first object, the cognitive state parameter for labeling the first object, and the emotional parameter for labeling the first object, can also be achieved by the following processing: reading the first parameter, attribute parameter, time parameter, cognitive state parameter and emotional parameter through a deep learning model for estimating quality parameters, and determining the output of the deep learning model as the second quality parameter.

[0090] Step 103: Determine a third quality parameter of the first text label based on the cognitive state parameter.

[0091] The electronic device also estimates a quality parameter for the first text label based only on the cognitive state parameter, thereby obtaining a quality parameter of the first object's cognitive state angle for the first text label, and records the quality parameter as a third quality parameter.

[0092] In an embodiment of the present application, the cognitive state parameter includes at least one of an eye movement parameter and a brain slice processing parameter. Among them, the eye movement parameter is used to characterize the eye movement of the first subject when annotating, which may include the number of eye movements, the number of the eye movement path (for example, from front to back, from back to front, multiple cycles, etc.), the eye movement duration and other data. The brain slice processing parameter is used to characterize the highlighting of the brain slice of the first subject when annotating, which may include the number of brain slice highlights, the number of the brain slice highlight position, the number of the highlight order, etc. It should be noted that the eye movement parameter and the brain slice processing parameter are both biological characteristics of the first subject, and the eye movement parameter and the brain slice processing parameter are biological reflections of the cognitive state of the first subject when annotating the first text.

[0093] See also Figure 5 , Figure 5 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 4 In some embodiments of the present application, cognitive state parameters include: eye movement parameters and brain area processing parameters. In this case, Figure 2 Step 103 in the embodiment, i.e., determining the third quality parameter of the first text label based on the cognitive state parameter, can be implemented by the following process:

[0094] Step 1031: Obtain a third weighting factor of the eye movement parameter and a fourth weighting factor of the brain slice processing parameter.

[0095] It should be noted that the third weighting factor for the eye movement parameters can be a weighting factor preset by an expert or a weighting factor learned by the second model, and reflects the influence of the eye movement parameters on the third quality parameter. Similarly, the fourth weighting factor for the brain slice processing parameters can be a weighting factor preset by an expert or a weighting factor learned by the second model, and reflects the influence of the brain slice processing parameters on the third quality parameter.

[0096] Step 1032: sum the eye movement parameters and the brain slice processing parameters using the third weight factor and the fourth weight factor, and determine the sum result as the third quality parameter of the first text label.

[0097] After obtaining the third weight factor and the fourth weight factor, the electronic device will use the third weight factor and the fourth weight factor to weight the eye movement parameters and the brain area processing parameters respectively, and then sum the weighted results. Finally, the summation result is directly used as the third quality parameter of the first text label.

[0098] It can be understood that in the embodiment of the present application, the electronic device can perform quality analysis on the first text label by combining eye movement parameters and brain area processing parameters through weighted summation, so that the eye movement and brain area processing of the first object can be fully considered to obtain the third quality parameter and improve the accuracy of the third quality parameter.

[0099] In other embodiments of the present application, Figure 2 Step 103, i.e., determining the third quality parameter of the first text label based on the cognitive state parameter, can also be achieved by the following processing: obtaining a third weight factor of the eye movement parameter, adjusting the eye movement parameter using the third weight factor, and determining the adjustment result as the third quality parameter of the first text label.

[0100] That is, the electronic device may also determine the third quality parameter using only the eye movement parameter. In this way, the third quality parameter can be obtained more quickly, thereby improving the calculation efficiency of the third quality parameter.

[0101] Step 104: Determine a quality verification result of the first text label based on the first quality parameter, the second quality parameter, and the third quality parameter.

[0102] After obtaining the first quality parameter, the second quality parameter, and the third quality parameter, the electronic device may perform a weighted sum of the first quality parameter, the second quality parameter, and the third quality parameter, and use the weighted sum result as the quality verification result of the first text label, or may take the median or maximum value of the first quality parameter, the second quality parameter, and the third quality parameter as the quality verification result of the first text label.

[0103] It can be understood that, compared with the related art, since the quality verification of text labels usually relies on static indicators, there is a problem of low accuracy in the quality verification of text labels. In the embodiment of the present application, the electronic device will determine the quality parameters from the dimension of the difficulty of marking the first text, the dimension of the cognitive state of the first object when marking, and the multi-dimensional behavior and state dimension of the first object when marking, so as to obtain quality parameters of different dimensions, and then combine the quality parameters of multiple different dimensions at the same time, that is, combine the first quality parameter, the second quality parameter and the third quality parameter at the same time, and collaboratively verify the quality of the first text label in multiple different dimensions to comprehensively cover various factors that affect the quality of the first text label, reduce the deviation caused by quality verification in a single dimension, thereby achieving accurate quality assessment of the first text label, and obtaining more accurate quality verification results, thereby improving the accuracy of quality verification of the text label.

[0104] based on Figure 2 , see Figure 6 , Figure 6 This is a flowchart of the text processing method provided in the embodiment of the present application. Figure 5 In some embodiments of the present application, Figure 2 After step 104 in step 105, that is, after determining the quality verification result of the first text label based on the first quality parameter, the second quality parameter, and the third quality parameter, the method may further include the following processing:

[0105] Step 105: Determine multiple marking time periods for the first object based on the time parameter.

[0106] The electronic device determines multiple marking time periods for the first object based on the time parameters of the first object when marking. Each marking time period is a time period in which the first object performs the marking task. Therefore, the first object performs the marking task in multiple marking time periods.

[0107] In some embodiments of the present application, Figure 6 Step 105, i.e., determining multiple marking time periods for the first object based on the time parameter, can be achieved by the following processing: extracting the marking start time and the marking end time of the first object from the time parameter, and using the marking start time and the marking end time to determine the total marking time period; dividing the total marking time period into multiple marking time periods.

[0108] The electronic device first extracts the first object's annotation start time and annotation end time from the time parameters, and then determines the time period with the annotation start time as the starting point and the annotation end time as the total annotation period. The annotation start time is the time when the first object begins the annotation task, and the annotation end time is the time when the first object ends the annotation task. It can be determined by the time when the first object clicks the "Start Annotation" button and the "End Annotation" button. The electronic device can then divide the total annotation period into multiple annotation time periods. Here, the electronic device can divide the total annotation period into multiple annotation time periods according to a preset duration, or divide the total annotation period into multiple evenly preset numbers to obtain multiple annotation time periods.

[0109] It can be understood that in the embodiment of the present application, the electronic device quickly determines the total marking time period based on the marking start time and the marking end time, and then directly divides the total marking time period to obtain multiple marking time periods. In this way, multiple different marking time periods can be quickly obtained to facilitate subsequent accuracy statistics for each marking.

[0110] In other embodiments of the present application, Figure 6 Step 105, i.e., determining multiple annotation time periods for the first object based on the time parameter, can also be achieved by the following processing: determining whether the first object is annotated with text labels in multiple preset time periods according to the time parameter, and using the multiple preset time periods for annotating text labels as multiple annotation time periods.

[0111] That is, the electronic device will first determine some preset time periods, and then combine the time parameters to determine whether the first object is in a working state during these preset time periods, and then use the preset time periods in the working state as the marked time periods. In this way, the marked time periods can also be obtained.

[0112] Step 106 : Based on the quality verification result, the accuracy of the first object in each marking time period is counted, and the marking time period with the highest accuracy is determined from the multiple marking time periods.

[0113] The electronic device, based on the obtained quality verification results, calculates the accuracy of the first object in each annotation time period to obtain the accuracy of each annotation time period, and then selects the annotation time period with the highest accuracy. The accuracy of the first object in each annotation time period can be obtained by comparing the number of correct text labels in each annotation time period with the number of text labels annotated by the first object. The accuracy of the text labels can be verified by the second object.

[0114] It can be understood that in an embodiment of the present application, the electronic device can first determine multiple marking time periods for the first object, and then, based on the quality verification results obtained, count the accuracy of the first object in each marking time period, and then filter the marking time period with the highest accuracy for the first object. In this way, it can be clear in which marking time period the first object has the highest accuracy, so as to subsequently guide the first object to perform text annotation in the marking time period, thereby further improving the quality of the text labels.

[0115] The following describes the process of determining the first parameter involved in the above content, wherein the first parameter includes: a level parameter and a length parameter of the first text.

[0116] In some embodiments of the present application, Figure 2 Before step 101, that is, before determining the first quality parameter of the first text label of the first text based on the first parameter of the first text, the method may further include the following processing: performing feature encoding on the first text to obtain first text features; determining the grade parameter of the first text based on the first text features; counting the number of characters of the first text, and taking the number of characters as the text length, and obtaining a length parameter based on the text length.

[0117] It should be noted that the level parameter characterizes the difficulty of judging the text label, that is, the difficulty of judging the label to which the first text belongs. First, the electronic device will first perform feature encoding on the first text, and use the obtained encoding feature as the first text feature. Then, based on the first text feature, it will judge the level parameter of the first text. Next, the electronic device will count the number of characters in the first text, and use the obtained number of characters directly as the text length of the first text. Afterwards, the electronic device can directly use the text length of the first text as the length parameter of the first text, or it can normalize the text length and use it as the length parameter.

[0118] The first text feature can be obtained by encoding the first text feature using a deep learning model that can be used for text encoding. The deep learning model can be a bidirectional encoder representation (Bidirectional Encoder Representation from Transformers, BERT) model or a transformer model, which is not limited in this embodiment of the present application. The text length of the first text can be obtained by traversing the text character by character and accumulating a counter.

[0119] In some embodiments of the present application, the above content of determining the level parameter of the first text based on the first text feature can be achieved by the following processing: performing similarity calculation on the first text feature and multiple second text features corresponding to multiple text categories to obtain multiple feature similarities; performing quantitative statistics on the same feature similarities, and determining the difference between the quantitative statistical result and the preset number as the level parameter of the first text.

[0120] The electronic device obtains multiple second text features corresponding to multiple text categories, and then calculates the feature similarity between the first text feature and each second text feature, thereby obtaining multiple feature similarities. The electronic device then extracts the identical feature similarities from the multiple feature similarities and performs a quantitative count (for example, the number of feature similarities with a statistical value of 0.5). The result of the quantitative count is then subtracted from a preset number, and the resulting difference is directly used as a level parameter. Here, the preset number can be set according to actual conditions and is not limited in this embodiment of the present application.

[0121] For example, if the number of identical feature similarities is 3 and the preset number is 1, then 2 is the level parameter of the first text; if the number of identical feature similarities is 2 and the preset number is 1, then 1 is the level parameter of the first text.

[0122] It should be noted that the second text feature of a text category can be that the electronic device performs feature extraction on the text belonging to the text category, and then uses the mean of the extracted features as the second text feature of the text category; the second text feature of a text category can also be that the electronic device performs feature extraction on the text that has not yet been classified, and then clusters these text features to obtain different clusters, each cluster corresponds to a text category, and then determines the center value of the cluster as the second text feature of the text category.

[0123] It can be understood that in the embodiment of the present application, the number of second text features that are sufficiently close to the first text feature of the first text can indicate the number of text categories into which the first text can be classified, and the more text categories the first text can be classified, the more difficult it is to perform label discrimination on the first text. Therefore, based on the number of similarities of the same features, the level parameters of the first text can be accurately estimated, so that the quality parameters can be estimated more accurately subsequently.

[0124] In other embodiments of the present application, the above content of determining the level parameters of the first text based on the first text features can also be achieved through the following processing: using a model for level discrimination, reading in the first text features, and determining the output of the model as the level parameters of the first text.

[0125] Among them, the model can be a large language model or a shallow machine learning model, such as a multi-layer perceptron, etc., which is not limited in the embodiments of this application.

[0126] It can be understood that in the embodiment of the present application, the electronic device can simultaneously determine the level parameter and the length parameter for the first text, so that the first parameter can simultaneously include the level parameter and the length parameter, thereby making the content included in the first parameter more comprehensive.

[0127] Next, the process of learning the weight factors of the first model in the above content is explained.

[0128] In some embodiments of the present application, Figure 2 Before step 101 in , that is, before determining the first quality parameter of the first text label of the first text based on the first parameter of the first text, the method may further include the following processing: obtaining the second text label annotated by the first object for the second text, and the third text label preset for the second text; calculating the semantic similarity between the second text label and the third text label, and determining the accuracy of the second text label using the semantic similarity; calculating the influence of the level parameter and the length parameter on the accuracy respectively through the first model, and determining the influence of the level parameter on the accuracy as the first weight factor, and determining the influence of the length parameter on the accuracy as the second weight factor.

[0129] The electronic device first obtains the text label annotated by the first object for the second text and uses this text label as the second text label. It also obtains the preset text label for the second text and uses this text label as the third text label. That is, the second text label is obtained by the first object annotating the second text, while the third text label is the accurate text label for the second text. The electronic device then calculates the semantic similarity between the second and third text labels and directly uses the semantic similarity as the accuracy of the second text label, or uses the logarithm of the semantic similarity as the accuracy of the second text label, etc. The electronic device then uses the first model to calculate the influence of the second text's level parameter on the accuracy and the influence of the second text's length parameter on the accuracy. The influence of the level parameter on the accuracy is then used as the first weight factor, and the influence of the length parameter on the accuracy is used as the second weight factor. In this way, the first weight factor and the second weight factor are obtained.

[0130] In the embodiment of the present application, the first model can be a large language model or a naive Bayes model. If the first model is a large language model, the electronic device can use the level parameter, length parameter and accuracy to generate a prompt word indicating the relationship between the large language model analysis accuracy and the level parameter and length parameter, and then process the prompt word through the large language model to obtain the degree of influence. If the first model is a naive Bayes model, the electronic device can use the accuracy, level parameter and length parameter as input to provide to the naive Bayes model, so that the naive Bayes model learns the above-mentioned degree of influence.

[0131] Below, taking the naive Bayes model as an example, the process of determining the first weight factor and the second weight factor is explained. If the second text includes text 1, text 2, text 3, and text 4, the corresponding second text labels include: text label 1, text label 2, text label 3, and text label 4. Among them, the rank parameter of text 1 is 1, the length parameter is 0.1, and the accuracy of text label 1 is 0.9; the rank parameter of text 2 is 1, the length parameter is 0.333 (the length parameter can be a normalized value), and the accuracy of text label 2 is 0.7; the rank parameter of text 3 is 3, the length parameter is 0.667, and the accuracy of text label 3 is 0.5; the rank parameter of text 4 is 4, the length parameter is 0.9, and the accuracy of text label 4 is 0.8. Afterwards, the electronic device will use the level parameters and length parameters of text 1 to text 4 as input variables of the naive Bayes model, and the accuracy of text labels 1 to text labels 4 as the target variables of the naive Bayes model to train the naive Bayes model, thereby obtaining the degree of influence of the level parameter on the accuracy, that is, the information gain of the level parameter is 0.6, and the degree of influence of the length parameter on the accuracy, that is, the information gain of the length parameter is 0.4, and then the information gain of the level parameter is used as the first weight factor, and the information gain of the length parameter is used as the second weight factor. In this way, the determined first weight factor can be 0.6, and the second weight factor can be recorded as 0.4. Here, information gain refers to the contribution of a feature to the classification task. The greater the information gain, the more important the feature is. Therefore, information gain can characterize the impact of the feature on accuracy.

[0132] It should be noted that when using the naive Bayes model for training, the prior probability of each accuracy can be calculated, and then the conditional probability of each accuracy given a certain feature, that is, the level parameter or length parameter in the above content, is calculated. Then, according to Bayesian quantification, the posterior probability of each accuracy is calculated, and then based on the posterior probability, the entropy of the data set is calculated, that is, the entropy of text 1 to text 4, and then the conditional entropy of the feature is calculated, that is, the conditional entropy of the level parameter and the length parameter. According to the entropy of the data set and the conditional entropy of the feature, the information gain can be calculated.

[0133] It can be understood that in an embodiment of the present application, the electronic device can determine the accuracy of the second text label based on the annotation result of the first object for the second text that already has the correct text label, and then combine the level parameter of the second text and the length parameter of the second text to analyze the impact of the level parameter and the length parameter on the accuracy, thereby being able to more accurately determine the first weight factor and the second weight factor based on the text that already has the correct text label.

[0134] Next, the process of learning the weight factors of the second model in the above content is explained.

[0135] In some embodiments of the present application, Figure 2 Before step 101 in the method, that is, before determining the first quality parameter of the first text label of the first text based on the first parameter of the first text, the method further includes: obtaining the second text label annotated by the first object for the second text, and the third text label preset for the second text; calculating the semantic similarity between the second text label and the third text label, and determining the accuracy of the second text label by using the semantic similarity; calculating the influence of the accuracy on the eye movement parameters and the brain area processing parameters respectively through the second model, and determining the influence of the accuracy on the eye movement parameters as the third weight factor, and determining the influence of the accuracy on the brain area processing parameters as the fourth weight factor.

[0136] It should be noted that the process of determining the accuracy of the second text label has been described in detail above and will not be repeated here.

[0137] The second model is used to analyze the degree of influence of accuracy on eye movement parameters and brain area processing parameters. It can be a large language model or a Bayesian model. It should be noted that in the embodiment of the present application, the analysis of the influence of accuracy on eye movement parameters and brain area processing parameters can achieve the association between eye movement parameters and brain area processing parameters through accuracy inference, so that when similar eye movement parameters and brain area processing parameters appear during the labeling task, the accuracy of the text label can be obtained. If the influence of eye movement parameters and brain area processing parameters on accuracy is simply analyzed, it is easily affected by other factors, such as duration, etc., which leads to inaccurate judgment.

[0138] The following describes the process of determining the third weight factor and the fourth weight factor using the Naive Bayes model as an example: If the second text includes text 1 to text 6, the second text label includes text label 1 to text label 6. Among them, the accuracy of text label 1 is 0.8, the eye movement parameter of text 1 is 0.5 (the eye movement parameter can be a normalized value), and the brain area processing parameter of text 1 is 0.7 (the brain area processing parameter can be a normalized value); the accuracy of text label 2 is 0.9, the eye movement parameter of text 2 is 0.6, and the brain area processing parameter is 0.8; the accuracy of text label 3 is 0.7, the eye movement parameter of text 3 is 0.4, and the brain area processing parameter is 0.6; the accuracy of text label 4 is 0.85, the eye movement parameter of text 4 is 0.55, and the brain area processing parameter is 0.75; the accuracy of text label 5 is 0.95, the eye movement parameter of text 5 is 0.65, and the brain area processing parameter is 0.85; the accuracy of text label 6 is 0.65, the eye movement parameter of text 6 is 0.35, and the brain area processing parameter is 0.55. Afterwards, the electronic device can use the accuracy of text labels 1 to 6 as the input variables of the Naive Bayes model, and determine the target variables of the Naive Bayes model based on the eye movement parameters and brain area processing parameters of texts 1 to 2, so as to use the Naive Bayes model for training. Here, the electronic device can fuse the eye movement parameters and the brain area processing parameters and use the fusion result as the target variable, or use the eye movement parameters and the brain area processing parameters as target variables respectively. If the fusion result is used as the target variable, then the second model only includes one Naive Bayes model. If the eye movement parameters and the brain area processing parameters are used as target variables respectively, then the second model can include two Naive Bayes models, one of which is used to calculate the degree of influence of accuracy on the eye movement parameters, that is, to determine the third weight factor, and the other Naive Bayes model is used to calculate the degree of influence of accuracy on the brain area processing parameters, that is, to determine the fourth weight factor.

[0139] In this example, the second model includes two naive Bayesian models. The electronic device uses a naive Bayesian model for calculating the influence of accuracy on eye movement parameters to calculate an information gain of 0.7 for accuracy on eye movement parameters. Then, a naive Bayesian model for calculating the influence of accuracy on brain slice processing parameters is used to calculate an information gain of 0.8 for accuracy on eye movement parameters. These two information gains are then linearly normalized, and the normalized result of the information gain of accuracy on eye movement parameters, 0.4667, is determined as the third weight factor. The normalized result of the information gain of accuracy on brain slice processing parameters, 0.5333, is determined as the fourth weight factor.

[0140] It is understandable that in the embodiment of the present application, the electronic device can eliminate the influence of irrelevant factors by analyzing the impact of accuracy on eye movement parameters and brain area processing parameters, thereby obtaining more accurate third weight factors and fourth weight factors.

[0141] It should be noted that in the embodiment of the present application, the process of learning the weight factors by the third model and the process of learning the weight factors by the fourth model are similar to the process of learning the weight factors by the first model, and will not be repeated here. Among them, the third model can be a large language model or a naive Bayes model. Similarly, the fourth model can be a large language model or a naive Bayes model. The embodiment of the present application does not specifically limit this.

[0142] Exemplarily, taking the third model as a naive Bayes model, and taking the parameters involved in determining the first sub-quality parameter including object identification parameter, annotation experience value, annotation start time, annotation duration of the first text and emotion parameter as an example, the learning process of the weight factor is explained. The second text includes text 1 to text 4, and the second text label includes text label 1 to text label 4, among which, the object identification parameter of text 1 is 1, the labeling experience value is 5, the labeling start time is 0 (normalized value), the labeling duration is 0 (normalized value), and the emotion parameter is 1 (normalized value), and the accuracy of text label 1 is 0.9; the object identification parameter of text 2 is 2, the labeling experience value is 7, the labeling start time is 0.333, the labeling duration is 0.25, and the emotion parameter is 0.667, and the accuracy of text label 2 is 0.7; the object identification parameter of text 3 is 3, the labeling experience value is 8, the labeling start time is 0.667, the labeling duration is 0.5, and the emotion parameter is 0.333, and the accuracy of text label 3 is 0.7; the object identification parameter of text 4 is 4, the labeling experience value is 9, the labeling start time is 1, the labeling duration is 1, the emotion parameter is 0, and the accuracy of text label 4 is 0.6. Afterwards, the electronic device can use the object identification parameters, labeling experience values, labeling start time, labeling duration and emotional parameters of texts 1 to 4 as input variables of the naive Bayes model, and use the accuracy of text labels 1 to text labels 4 as target variables of the naive Bayes model to train the naive Bayes model to obtain a weight factor of 0.3 corresponding to the object identification parameters, a weight factor of 0.25 corresponding to the labeling experience values, a weight factor of 0.2 corresponding to the labeling start time, a weight factor of 0.15 corresponding to the labeling duration and a weight factor of 0.1 corresponding to the emotional parameters.

[0143] Exemplarily, the fourth model is taken as a naive Bayes model, and the parameters involved in the determination of the second sub-quality parameter include the grade parameter, the annotation experience value, the annotation start time, the annotation duration of the first text, the eye movement parameter, and the emotion parameter as an example to illustrate the learning process of the weight factor. The second text includes text 1 to text 4, and the second text label includes text label 1 to text label 4. Among them, the grade parameter of text 1 is 1, the annotation experience value is 5, the annotation start time is 0 (the normalized value), the annotation duration is 0 (the normalized value), the eye movement parameter is 0 (the normalized value), the emotion parameter is 1 (the normalized value), and the accuracy of text label 1 is 0.9; the grade parameter of text 2 is 2, the annotation experience value is 5, the annotation start time is 0.333, the annotation duration is 0.25, the eye movement parameter is 0.2, and the emotion parameter is 1 (the normalized value). The value of the annotation experience is 0.667, and the accuracy of text label 2 is 0.8; the level parameter of text 3 is 3, the annotation experience value is 8, the annotation start time is 0.667, the annotation duration is 0.5, the eye movement parameter is 0.4, the emotion parameter is 0.333, and the accuracy of text label 3 is 0.7; the level parameter of text 4 is 4, the annotation experience value is 9, the annotation start time is 1, the annotation duration is 1, the eye movement parameter is 1, the emotion parameter is 0, and the accuracy of text label 2 is 0.6. Afterwards, the electronic device can use the level parameters, annotation experience value, annotation start time, annotation duration, eye movement parameters and emotion parameters of text 1 to text 4 as input variables of the naive Bayes model, and use the accuracy of text label 1 to text label 4 as the target variable of the naive Bayes model to train the naive Bayes model, thereby obtaining a weight factor of -0.2 corresponding to the level parameter, a weight factor of 0.3 corresponding to the annotation experience value, a weight factor of 0.15 corresponding to the annotation start time, a weight factor of 0.2 corresponding to the annotation duration, a weight factor of 0.25 corresponding to the eye movement parameter and a weight factor of 0.3 corresponding to the emotion parameter.

[0144] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0145] The embodiment of the present application is implemented in a scenario where the server performs quality verification on the intent labels (referred to as first text labels) annotated by the annotator (referred to as the first object) on the call text (referred to as the first text).

[0146] First, the server will combine the annotated text and analyze the impact of different parameters of the text and the annotator on the annotation accuracy to obtain the influencing factors of these parameters.

[0147] Figure 7 This is a flow chart of the calculation of the impact factor of the parameters provided in the embodiment of this application. Figure 7 , the process includes:

[0148] Step 201: Initialize the annotation system.

[0149] Step 202: It is detected that the annotator inputs the annotator ID (called object identification information) to start the annotation task.

[0150] Step 203: Obtain the annotator's intention label for the annotated text.

[0151] The labelers will label the call text based on the call text and their understanding of the intent labeling system. The intent labeling system is an explanatory document that classifies, explains, and provides positive and negative examples for intent labels.

[0152] Step 204: Detect the eye movement data (called eye movement parameters) and brain area processing data (called brain area processing parameters) of the annotator using an eye movement detection instrument and an electroencephalogram detection instrument.

[0153] Among them, eye movement data records the number of times the annotator's eyes lingered on different words and the order of their eye movements when annotating. By analyzing the order, number, and duration of the annotator's lingering on different words, it is possible to comprehensively judge the current annotator's thinking path when annotating the data, focus on the words, and whether they are entangled in the annotation results. Brain area processing data records data such as the highlighted parts of the brain area, which can reflect the brain areas activated by the annotator during annotation. Generally, the more brain areas activated, the higher the annotation difficulty.

[0154] Step 205: Determine the model based on the text length and the discrimination difficulty, and analyze the annotated text to obtain the text length (called length parameter) and the discrimination difficulty (called level parameter).

[0155] Here, we determine the text's general difficulty level based on the annotated text, without considering the individual characteristics of the annotators. The text length and difficulty determination model performs the following processing:

[0156] A. Calculate the number of words in the annotated text as the text length;

[0157] B. Extract the text vector of the annotated text (called the first text feature);

[0158] C. Use typical examples and quality-checked texts in the intent labeling system as the golden dataset;

[0159] D. Calculate the similarity between the labeled text vector and the sentence vector (called the second text feature) of each data item in the gold dataset (called the second text). If the labeled text has the same similarity with the sentence vectors of multiple categories of data in the gold dataset, determine the labeling difficulty based on the number of categories (called the quantity statistics result). For example, if the number of categories is 2, the labeling difficulty is 1; if the number of categories is 3, the labeling difficulty is 2, and so on.

[0160] Step 206: Calculate the accuracy of the annotation of the annotated text by the annotator (referred to as the accuracy of the second text label) using the accuracy calculation model.

[0161] The server retrieves the original correct label of the annotated text, and then determines whether the label of the annotator is the same as that of the annotator. If they are the same, the accuracy is 1. If the two are semantically similar (for example, the similarity is greater than 0.8), the accuracy is 0.9*similarity. If they are completely different or the similarity is less than 0.8, the accuracy is 0.

[0162] Step 207: Determine relevant parameters of the annotated text.

[0163] After the above processing, each annotated text will have the following parameters: text, text length, judgment difficulty, intention label, annotation start and end time, annotation time, annotator ID, annotator eye movement record, annotator brain area processing process, annotator cumulative annotation time for the day, annotator emotion, annotator annotation experience value, annotator historical annotation data accuracy (called historical accuracy), and annotation accuracy.

[0164] Among them, the emotions of the annotator can be completed by inputting the expression sequence captured by the camera of the annotation system into the emotion recognition model to complete expression recognition, and the emotion that appears most frequently in the multiple emotion label results of multiple expression recognitions is taken as the final emotion (different emotions can be represented by different numerical values, such as 1 for happy, 2 for angry, etc.); the annotation experience value can be obtained by the annotator's years of work or the score given by the annotation team leader; the accuracy of historical annotation data can be obtained by calculating the accuracy between the annotation results and the correct labels of all the data annotated by the annotator in the past.

[0165] Step 208: Using a naive Bayes model (referred to as the first model), perform pairwise correlation studies on text length, discrimination difficulty, and annotation accuracy.

[0166] That is, the text length can be used as a feature and the discrimination difficulty can be used as a label and provided to the Naive Bayes model so that the Naive Bayes model can learn the weighted relationship equation between the two factors; the discrimination difficulty can be used as a feature and the annotation accuracy can be used as a label and provided to the Naive Bayes model so that the Naive Bayes model can learn the weighted relationship equation between the two factors; the text length and discrimination difficulty can both be used as features and the annotation accuracy can be used as a label and provided to the Naive Bayes model so that the Naive Bayes model can learn the weighted relationship equation between the three factors, that is, annotation accuracy = w1*text length + w2*discrimination difficulty.

[0167] Step 209: Use the start and end times, annotator ID, cumulative annotation time for the day, sentiment, annotation experience, and historical annotation data accuracy as features, and the annotation accuracy as a label, and feed them into the Naive Bayes model to obtain a weighted relationship equation. For example, the obtained annotation accuracy = w1 * annotation start and end times + w2 * annotator ID + w3 * cumulative annotation time for the day + w4 * sentiment + w5 * annotation experience + w6 * historical annotation data accuracy.

[0168] Step 210: Analyze the weighted relationship for the same labeler.

[0169] That is, the text, text length, discrimination difficulty, labeling start and end time, labeling time, labeler's eye movement record, labeler's brain area processing process, labeler's cumulative labeling time on the day, labeler's emotion, labeler's labeling experience value, labeler's historical labeling data accuracy, that is, all parameters except the labeler's ID selected in step 207 are used as features, and the labeling accuracy is used as a label to provide to the naive Bayes model, and a weighted relationship is learned, that is, labeling accuracy = w1*text+w2*text length+w3*discrimination difficulty+w4*intention label+w5*labeling start and end time+w6*labeling time+w7*labeler's eye movement record+w8*labeler's brain area processing process+w9*labeler's cumulative labeling time on the day+w10*emotion+w11*labeling experience value+w12*historical labeling data accuracy.

[0170] Step 211: Use the eye movement records and brain slice processing process as labels, use the labeling accuracy as features, and use the analysis results to determine the weights of the eye movement records and brain slice processing process to obtain a weighted relationship.

[0171] At this point, the influencing factor analysis of these parameters is completed.

[0172] The server then uses these influencing factors, or weights, to perform a weighted summation of the various parameters of the call text to obtain multiple accuracies (called sub-quality parameters). The server then uses voting or mean calculation to obtain the final accuracy, which is the quality verification result of the labels annotated by the annotators on the call text.

[0173] Next, the server will determine multiple time periods (called annotation time periods) according to the annotation start time, such as 9:00-11:00, 11:00-13:00, 13:00-15:00, etc., and then count the accuracy of the annotation personnel in different time periods. By sorting the accuracy rates of different time periods, the server will obtain the time period with the highest accuracy rate, thereby guiding the annotation personnel to perform annotation work in this time period.

[0174] In this way, the factors that affect the quality of the labelers' labels can be analyzed in detail, so as to more accurately analyze the labelers' label quality. At the same time, the labeling quality of the labelers in different time periods can be clarified, so as to guide the labelers to label in the time period with the highest labeling quality.

[0175] It is understandable that in the embodiments of the present application, when user information is involved, such as emotional parameters, cognitive state parameters and other related data, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0176] See also Figure 8 , Figure 8 is a structural diagram of an electronic device provided in an embodiment of the present application, Figure 8 The electronic device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 8 Various buses are labeled as bus system 440 .

[0177] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0178] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0179] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0180] Memory 450 includes volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. Nonvolatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0181] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0182] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0183] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420 . Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0184] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0185] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0186] In some embodiments, the text processing device provided in the embodiments of the present application can be implemented in software. Figure 8 A text processing device 455 stored in memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a first quality module 4551, a second quality module 4552, a third quality module 4553, a result determination module 4554, and a weight determination module 4555. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0187] In other embodiments, the text processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the text processing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the text processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0188] In some embodiments, the electronic device can implement the text processing method provided in the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer program can be a native program or software module in the operating system; it can be a native application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a marking APP; it can also be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to a browser environment to run. In short, the above-mentioned computer-executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module, or plug-in in any form.

[0189] The following continues to describe the exemplary structure of the text processing device 455 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 8 As shown, the software modules stored in the text processing device 455 of the memory 450 may include:

[0190] A first quality module 4551 is configured to determine a first quality parameter of a first text label of the first text based on a first parameter of the first text, where the first text label is obtained by annotating the first object;

[0191] A second quality module 4552 is configured to determine a second quality parameter of the first text label based on the first parameter, the attribute parameter of the first object, the time parameter of the first object performing the annotation, the cognitive state parameter of the first object performing the annotation, and the emotion parameter of the first object performing the annotation;

[0192] A third quality module 4553, configured to determine a third quality parameter of the first text label based on the cognitive state parameter;

[0193] The result determination module 4554 is configured to determine a quality verification result of the first text label based on the first quality parameter, the second quality parameter, and the third quality parameter.

[0194] In the above scheme, the first parameter includes: a level parameter and a length parameter of the first text; the first quality module 4551 is also used to obtain a first weight factor corresponding to the level parameter and a second weight factor corresponding to the length parameter; the level parameter and the length parameter are summed using the first weight factor and the second weight factor, and the sum result is determined as the first quality parameter of the first text label.

[0195] In the above scheme, the first quality module 4551 is also used to perform feature encoding on the first text to obtain the first text feature; based on the first text feature, determine the level parameter of the first text, wherein the level parameter represents the difficulty of judging the text label; count the number of characters in the first text, and use the number of characters as the text length, and obtain the length parameter based on the text length.

[0196] In the above scheme, the first quality module 4551 is also used to calculate the similarity of the first text feature and multiple second text features corresponding to multiple text categories to obtain multiple feature similarities; perform quantitative statistics on the same feature similarities, and determine the difference between the quantitative statistical result and the preset quantity as the level parameter of the first text.

[0197] In the above scheme, the text processing device 455 also includes: a weight determination module 4555, which is used to obtain the second text label marked by the first object for the second text, and the third text label preset for the second text; calculate the semantic similarity between the second text label and the third text label, and use the semantic similarity to determine the accuracy of the second text label; calculate the degree of influence of the level parameter and the length parameter on the accuracy through the first model, and determine the degree of influence of the level parameter on the accuracy as the first weight factor, and determine the degree of influence of the length parameter on the accuracy as the second weight factor.

[0198] In the above scheme, the attribute parameters include: the object identification parameter of the first object and the annotation quality parameter of the first object; the second quality module 4552 is also used to calculate the first sub-quality parameter of the first text label based on the object identification parameter, the annotation quality parameter, the time parameter and the emotion parameter; calculate the second sub-quality parameter of the first text label based on the first parameter, the annotation quality parameter, the time parameter, the emotion parameter and the cognitive state parameter; and determine the sum of the first sub-quality parameter and the second sub-quality parameter as the second quality parameter of the first text label.

[0199] In the above solution, the cognitive state parameters include: eye movement parameters and brain slice processing parameters, the eye movement parameters are used to characterize the eye movement of the first subject during the annotation, and the brain slice processing parameters are used to characterize the highlighting of the brain slice of the first subject during the annotation;

[0200] The third quality module 4553 is also used to obtain a third weight factor of the eye movement parameter and a fourth weight factor of the brain area processing parameter; the eye movement parameter and the brain area processing parameter are summed using the third weight factor and the fourth weight factor, and the summation result is determined as the third quality parameter of the first text label.

[0201] In the above scheme, the weight determination module 4555 is also used to obtain the second text label annotated by the first object for the second text, and the third text label preset for the second text; calculate the semantic similarity between the second text label and the third text label, and use the semantic similarity to determine the accuracy of the second text label; calculate the degree of influence of the accuracy on the eye movement parameters and the brain area processing parameters respectively through the second model, and determine the degree of influence of the accuracy on the eye movement parameters as the third weight factor, and determine the degree of influence of the accuracy on the brain area processing parameters as the fourth weight factor.

[0202] In the above scheme, the result determination module 4554 is also used to determine multiple marking time periods for the first object based on the time parameters; based on the quality verification results, the accuracy of the first object in each marking time period is counted, and the marking time period with the highest accuracy is determined from the multiple marking time periods.

[0203] In the above scheme, the result determination module 4554 is also used to extract the marking start time and marking end time of the first object from the time parameters, and use the marking start time and the marking end time to determine the total marking period; and divide the total marking period into multiple marking time periods.

[0204] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the text processing method described in the embodiment of the present application.

[0205] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the text processing method provided by the embodiment of the present application, for example, Figure 2 The text processing method shown.

[0206] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0207] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0208] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0209] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0210] In summary, through the embodiments of the present application, quality parameters are determined based on the dimension of the difficulty of annotating the first text, the dimension of the cognitive state of the first object during annotation, and the dimension of the multi-dimensional behavior and state of the first object during annotation, thereby obtaining quality parameters of different dimensions. Then, the quality parameters of multiple different dimensions are simultaneously combined, that is, the first quality parameter, the second quality parameter, and the third quality parameter are simultaneously combined to collaboratively verify the quality of the first text label in multiple different dimensions, so as to comprehensively cover various factors affecting the quality of the first text label and reduce the deviation caused by quality verification in a single dimension, thereby achieving accurate quality assessment of the first text label and obtaining more accurate quality verification results, thereby improving the accuracy of the quality verification of the text label; multiple annotation time periods are determined for the first object, and then based on the obtained quality verification results, the accuracy of the first object in each annotation time period is counted, and then the annotation time period with the highest accuracy is screened for the first object. In this way, it is possible to clearly determine in which annotation time period the first object has the highest accuracy, so as to facilitate subsequent guidance of the first object to perform text annotation in the annotation time period, thereby further improving the quality of the text label.

[0211] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A text processing method, characterized in that: The method comprises: Determining a first quality parameter of a first text label of the first text based on a first parameter of the first text, where the first text label is obtained by annotating the first object; Calculating a first sub-quality parameter of the first text label based on an object identification parameter of the first object, a labeling quality parameter of the first object, a time parameter of the first object performing the labeling, and a sentiment parameter of the first object performing the labeling; Calculating a second sub-quality parameter of the first text label based on the first parameter, the annotation quality parameter, the time parameter, the emotion parameter, and a cognitive state parameter of the first object performing the annotation; Determining a sum of the first sub-quality parameter and the second sub-quality parameter as a second quality parameter of the first text label; determining a third quality parameter of the first text label based on the cognitive state parameter; A quality verification result of the first text label is determined based on the first quality parameter, the second quality parameter, and the third quality parameter.

2. The method according to claim 1, characterized in that The first parameter includes: a level parameter and a length parameter of the first text; and determining a first quality parameter of the first text label of the first text based on the first parameter of the first text includes: Obtaining a first weight factor corresponding to the level parameter and a second weight factor corresponding to the length parameter; The level parameter and the length parameter are summed using the first weight factor and the second weight factor, and the summation result is determined as the first quality parameter of the first text label.

3. The method according to claim 2, characterized in that Before determining the first quality parameter of the first text label of the first text based on the first parameter of the first text, the method further includes: Performing feature encoding on the first text to obtain first text features; Determining a level parameter of the first text based on the first text feature, wherein the level parameter represents the difficulty of determining a text label; The number of characters in the first text is counted, and the number of characters is used as the text length, and the length parameter is obtained based on the text length.

4. The method according to claim 3, characterized in that The determining of the level parameter of the first text based on the first text feature includes: Calculating similarity between the first text feature and a plurality of second text features corresponding to a plurality of text categories to obtain a plurality of feature similarities; Quantity statistics are performed for the similarities of the same features, and the difference between the quantity statistics result and a preset quantity is determined as the level parameter of the first text.

5. The method according to any one of claims 2 to 4, characterized in that Before determining the first quality parameter of the first text label of the first text based on the first parameter of the first text, the method further includes: Acquire a second text label annotated by the first object for the second text, and a third text label preset for the second text; Calculating semantic similarity between the second text label and the third text label, and determining accuracy of the second text label using the semantic similarity; Through the first model, the influence of the level parameter and the length parameter on the accuracy are calculated respectively, and the influence of the level parameter on the accuracy is determined as the first weight factor, and the influence of the length parameter on the accuracy is determined as the second weight factor.

6. The method according to any one of claims 1 to 4, characterized in that The cognitive state parameters include: eye movement parameters and brain slice processing parameters, wherein the eye movement parameters are used to characterize the eye movement of the first subject during the annotation process, and the brain slice processing parameters are used to characterize the highlighting of the brain slice of the first subject during the annotation process; The determining, based on the cognitive state parameter, a third quality parameter of the first text label includes: Obtaining a third weight factor of the eye movement parameter and a fourth weight factor of the brain slice processing parameter; The eye movement parameter and the brain slice processing parameter are summed using the third weight factor and the fourth weight factor, and the summation result is determined as the third quality parameter of the first text label.

7. The method according to claim 6, characterized in that Before determining the first quality parameter of the first text label of the first text based on the first parameter of the first text, the method further includes: Acquire a second text label annotated by the first object for the second text, and a third text label preset for the second text; Calculating semantic similarity between the second text label and the third text label, and determining accuracy of the second text label using the semantic similarity; Through the second model, the degree of influence of the accuracy on the eye movement parameters and the brain area processing parameters is calculated respectively, and the degree of influence of the accuracy on the eye movement parameters is determined as the third weight factor, and the degree of influence of the accuracy on the brain area processing parameters is determined as the fourth weight factor.

8. The method according to any one of claims 1 to 4, characterized in that After determining the quality verification result of the first text label based on the first quality parameter, the second quality parameter, and the third quality parameter, the method further includes: determining a plurality of marked time periods for the first object based on the time parameter; Based on the quality verification result, the accuracy of the first object in each marking time period is counted, and the marking time period with the highest accuracy is determined from the multiple marking time periods.

9. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 8 when executing computer-executable instructions or computer programs stored in the memory.

10. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images