Data processing method and apparatus

Abstract

The application discloses a data processing method and device, and relates to the technical field of computer processing. The method comprises the following steps: acquiring a target sample data set, wherein the target sample data set comprises a first sample data set to be processed and a second sample data set for reference, the second sample data set comprises a known processing conclusion; acquiring a first processing result corresponding to a first label and a second processing result corresponding to a second label; acquiring a processing confidence corresponding to the second sample data set based on a matching degree between the second processing result and the known processing conclusion; and determining that the first processing result is credible in a case where the processing confidence satisfies a preset first condition. The quantitative checking mechanism for the first processing result is beneficial to avoiding a large number of controversial labels from repeatedly flowing back to an audit queue for re-distribution and processing, thereby reducing the redundancy of a server data processing link, and being beneficial to reducing the consumption of computing resources and reducing the unnecessary occupation of storage space.

Description

Technical Field

[0001] This application relates to the field of computer processing technology, specifically to data processing methods and apparatus. Background Technology

[0002] In the training of artificial intelligence models, a large amount of labeled sample data is typically required. This labeled data needs to be reviewed after labeling to ensure quality. In related technologies, this review process mainly relies on manual evaluation of each labeling result. However, in scenarios involving large-scale sample processing, the lack of an objective quantitative verification mechanism for the review conclusions leads to a large amount of controversial labeled data needing to be repeatedly returned to the review queue for redistribution and processing. This results in redundancy in server data processing links, increased consumption of computing resources, and unnecessary occupation of storage space. Summary of the Invention

[0003] In view of this, this application provides a data processing method and apparatus to solve the problems of redundant server data processing links, increased consumption of computing resources, and unnecessary occupation of storage space.

[0004] In a first aspect, this application provides a data processing method, the method comprising: Obtain a target sample dataset, which includes a first sample dataset to be processed and a second sample dataset for reference; the first sample dataset includes first sample data and corresponding first annotations; the second sample dataset includes second sample data and corresponding second annotations, and the second sample dataset contains known processing conclusions; Obtain the annotation processing results for the target sample dataset, the annotation processing results including a first processing result for the first annotation and a second processing result for the second annotation; Based on the degree of matching between the second processing result and the known processing conclusion, the processing confidence level corresponding to the second sample dataset is obtained; If the processing confidence level meets a preset first condition, the first processing result is determined to be reliable.

[0005] Secondly, this application provides a data processing apparatus, the apparatus comprising: The data acquisition module is used to acquire a target sample dataset, which includes a first sample dataset to be processed and a second sample dataset for reference; the first sample dataset includes first sample data and corresponding first annotations; the second sample dataset includes second sample data and corresponding second annotations, and the second sample dataset contains known processing conclusions; The result acquisition module is used to acquire the annotation processing results for the target sample dataset, the annotation processing results including a first processing result for the first annotation and a second processing result for the second annotation; The result matching module is used to obtain the processing confidence level corresponding to the second sample dataset based on the degree of matching between the second processing result and the known processing conclusion. The result processing module is used to determine that the first processing result is reliable if the processing confidence level meets a preset first condition.

[0006] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the data processing method described in the first aspect or any corresponding embodiment.

[0007] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the data processing method described in the first aspect or any corresponding embodiment.

[0008] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the data processing method described in the first aspect or any corresponding embodiment thereof.

[0009] The data processing method provided in this application adds a second sample dataset for user reference to the first sample dataset to be processed to obtain a target sample dataset. The second sample dataset contains known processing conclusions. Further, the processing confidence of the second processing result for the second label is measured by the known processing conclusions. If the processing confidence meets the first condition, it is determined that the label processing result for the target sample dataset is accurate, and thus the first processing result for the first label is determined to be credible. On the one hand, the introduction of the second sample data, the second label, and the known processing conclusions can automatically and objectively measure the credibility of the first processing result. Moreover, the quantitative verification mechanism for the first processing result helps to avoid a large number of controversial labels being repeatedly returned to the review queue for redistribution and processing, thereby reducing the redundancy of the server data processing link, reducing the consumption of computing resources, and reducing the unnecessary occupation of storage space. On the other hand, there is no need to continue to introduce manual processing of the first processing result, reducing the consumption of human resources. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this application, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application; Figure 2 This is a schematic flowchart of a first type of data processing method according to an embodiment of this application; Figure 3 This is a schematic diagram of a second type of data processing method according to an embodiment of this application; Figure 4 This is a schematic diagram of a third data processing method according to an embodiment of this application; Figure 5 An exemplary diagram illustrates the complete workflow of a data processing method; Figure 6 This is a structural block diagram of a data processing apparatus according to an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] It should be noted that the information involved in this application (including but not limited to: user input information, such as information entered by the user into an input box), sample data (including but not limited to sample data used for analysis, stored sample data, and displayed sample data, such as context code, all code of the current project, the service pressure corresponding to the operation performed on all code of the current project, and the code development status of the current project), and signals are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant sample data must comply with relevant laws, regulations, and standards. For example, the context code, the operation performed on all code of the current project, the service pressure corresponding to the operation, and the code development status involved in this application were all obtained under full authorization.

[0014] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0015] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained.

[0016] As one optional application scenario in the embodiments of this application, such as Figure 1 As shown, the system may include at least one terminal device and at least one server. Figure 1 The system is illustrated in the example, which includes a computer 101, a mobile terminal 102, and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0017] Specifically, the terminal device can be a smartphone, tablet, laptop, PDA, desktop computer, game console, smart TV, smart wearable device, in-vehicle terminal, VR (Virtual Reality) device, AR (Augmented Reality) device, etc. Server 103 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. Network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranet, local area network, wide area network, mobile communication network, and combinations thereof.

[0018] For example, an application is installed on the terminal device. This application can be one that requires downloading and installation, or it can be used instantly. Optionally, in this embodiment, the application is used to display sample data and corresponding annotations to the user.

[0019] For example, the server is the backend server of the application.

[0020] In related technologies, the number of labeled samples required during model training is gradually increasing. In these technologies, after sample data is labeled, the corresponding labels are manually processed, and the accuracy of the labels is judged based on personal experience. However, in the aforementioned technologies, the processing of labels relies on personal experience and subjective judgment, which cannot guarantee the reliability of the labeling results. To further ensure the reliability of the labeling results, further in-depth manual processing is required, resulting in excessive human resource consumption. Moreover, in scenarios involving large-scale sample processing, the lack of an objective quantitative verification mechanism for review conclusions leads to a large amount of controversial labeled data needing to be repeatedly returned to the review queue for redistribution and processing, causing redundancy in server data processing links, increased consumption of computing resources, and unnecessary occupation of storage space.

[0021] Furthermore, in this embodiment, a second sample dataset for user reference is added to the first sample dataset to be processed to obtain a target sample dataset. The second sample dataset contains known processing conclusions. Further, the processing confidence of the second processing result for the second annotation is measured by the known processing conclusions. If the processing confidence meets the first condition, it is determined that the annotation processing result of the processor for the target sample dataset is accurate, and thus the first processing result for the first annotation is determined to be credible. On the one hand, the introduction of second sample data, second label, and result identifier can automatically and objectively measure the credibility of the first processing result. Moreover, the quantitative verification mechanism for the first processing result helps to avoid a large number of controversial annotations from repeatedly flowing back to the review queue for redistribution and processing, thereby reducing the redundancy of server data processing links, reducing the consumption of computing resources, and reducing the unnecessary occupation of storage space. On the other hand, there is no need to continue to introduce manual processing of the first processing result, reducing the consumption of human resources.

[0022] According to an embodiment of this application, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0023] This embodiment provides a data processing method that can be used in the aforementioned mobile terminal and / or server (hereinafter referred to as electronic device). Figure 2 This is a flowchart of a data processing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the target sample dataset.

[0024] In this embodiment of the application, during data processing, the electronic device acquires target sample data. The target sample dataset includes a first sample dataset to be processed and a second sample dataset for reference.

[0025] The first sample dataset includes the first sample data and the corresponding first annotation. For example, the first sample dataset refers to the sample dataset to be processed after manual annotation.

[0026] The second sample dataset includes second sample data and corresponding second annotations, and contains known processing conclusions. These known processing conclusions characterize the accuracy of the second annotations relative to the second sample data. For example, the known processing results include result identifiers corresponding to the second annotations, which characterize the accuracy of the second annotations. For example, the second sample dataset refers to sample data that has been processed after manual annotation.

[0027] Step S202: Obtain the annotation processing results for the target sample dataset.

[0028] In this embodiment of the application, after acquiring the target sample dataset, the electronic device acquires the annotation processing results for the target sample dataset. The annotation processing results include a first processing result for a first annotation and a second processing result for a second annotation.

[0029] For example, the electronic device obtains the annotation processing results of the user who reviewed the target sample dataset by reviewing the user.

[0030] For example, the first processing result is used to characterize whether the first annotation is accurate relative to the first sample processed data, and the second processing result is used to characterize whether the second annotation is accurate relative to the second sample data.

[0031] Step S203: Based on the degree of matching between the second processing result and the known processing conclusion, obtain the processing confidence level corresponding to the second sample dataset.

[0032] In this embodiment of the application, after obtaining the above-mentioned annotation processing results, the electronic device obtains the processing confidence level corresponding to the second sample dataset based on the degree of matching between the second processing results and the known processing conclusions.

[0033] Step S204: If the processing confidence level meets the preset first condition, the first processing result is determined to be credible.

[0034] In this embodiment of the application, after obtaining the aforementioned processing confidence level, the electronic device determines that the first processing result is reliable if the processing confidence level meets a preset first condition. For example, the first condition is greater than a first preset target value. This first preset target value can be any numerical value and can be flexibly set and adjusted according to actual circumstances.

[0035] For example, if the processing confidence does not meet the first condition, the electronic device sends the first sample dataset to the annotation task initiator of the first sample dataset so that the annotation task initiator can re-annotate the first sample dataset.

[0036] For example, if the first processing result is reliable, and the first processing result indicates that the first label is incorrect, the electronic device generates a labeling suggestion for the first sample data corresponding to the incorrect first label based on the first model; further, the labeling suggestion is sent to the sample labeler and / or the labeling task initiator so that the user can adjust the first label of the first sample data based on the labeling suggestion, which is conducive to realizing rapid adjustment of the incorrect first label.

[0037] For example, the first model can be a pre-trained machine learning model or a general pre-trained model.

[0038] The data processing method provided in this embodiment adds a second sample dataset for user reference to the first sample dataset to be processed to obtain a target sample dataset. The second sample dataset contains known processing conclusions. Further, the processing confidence of the second processing result for the second label is measured by the known processing conclusions. If the processing confidence meets the first condition, it is determined that the label processing result for the target sample dataset is accurate, and thus the first processing result for the first label is determined to be credible. On the one hand, the introduction of the second sample data, the second label, and the known processing conclusions can automatically and objectively measure the credibility of the first processing result. Moreover, the quantitative verification mechanism for the first processing result helps to avoid a large number of controversial labels being repeatedly returned to the review queue for redistribution and processing, thereby reducing the redundancy of the server data processing link, reducing the consumption of computing resources, and reducing the unnecessary occupation of storage space. On the other hand, there is no need to continue to introduce manual processing of the first processing result, reducing the consumption of human resources.

[0039] Furthermore, if the processing confidence level does not meet the first condition, the first sample dataset is sent to the task initiator of the first sample dataset for re-annotation. This return and review mechanism prevents low-confidence processing results from directly entering the downstream model training stage. Instead, it triggers a secondary check at the data source, creating a controllable quality gate in the data processing path and significantly improving the purity of the final corpus used for training. In other words, the user can understand the task publisher as a third party. Generally, task publishers often outsource to third parties due to insufficient personnel. Considering this staffing issue, the processing confidence level is determined using a second sample dataset. Therefore, if the processing confidence level does not meet the first condition, the task publisher then re-annotates the first sample dataset, saving the task publisher's human and computational resources.

[0040] In an exemplary embodiment, step S201 includes: Step S2011: Obtain the first sample dataset and the mixing strategy.

[0041] In this embodiment of the application, when acquiring the target sample dataset, the electronic device acquires the first sample dataset and the mixing strategy.

[0042] For example, the electronic device acquires a first sample dataset and preset injection parameters, including an injection interval and an injection quantity; further, it groups the first sample dataset based on the injection interval to obtain at least one data subset; further, it randomly generates an injection quantity of injection locations corresponding to each data subset. The hybrid strategy includes the injection locations.

[0043] For example, if the mixing strategy is "inject 2 second sample data into every 10 processed data", then the injection interval is 10 and the number of injections is 2. The electronic device divides the 100 first sample data in the first sample dataset into 10 data subsets in groups of 10, and randomly generates 2 injection positions (e.g., the 4th and 7th positions) in the 10 positions of each data subset.

[0044] It should be noted that the above description of data injection is merely illustrative and explanatory. In practical applications, the injection method can be flexibly set and adjusted according to the actual situation. For example, the injection parameters include the injection interval and the total injection quantity. The electronic device allocates the injection quantity to each data subset based on the total injection quantity, and each data subset corresponds to at least one injection quantity. For example, when allocating the injection quantity to each data subset, different data subsets may correspond to the same injection quantity; or, to improve the randomness of data injection, different data subsets may correspond to different injection quantities.

[0045] Step S2012: Obtain the second sample dataset.

[0046] In this embodiment of the application, when acquiring the target sample dataset, the electronic device acquires the second sample dataset.

[0047] For example, the injection parameters mentioned above also include the ratio between the first result identifier and the second result identifier. The electronic device obtains the ratio between the first result identifier and the second result identifier; further, based on the ratio, it obtains a second sample dataset from a preset database. Here, the first result identifier is used to indicate that the second annotation is correct, and the second result identifier is used to indicate that the second annotation is incorrect.

[0048] Step S2013: Based on the mixing strategy, the second sample data is mixed into the first sample dataset to obtain the target sample dataset.

[0049] In this embodiment, after obtaining the aforementioned mixing strategy and the second sample dataset, the electronic device mixes the second sample data into the first sample dataset based on the mixing strategy to obtain the target sample dataset. For example, the electronic device injects the second sample data into the injection position corresponding to the first sample dataset to obtain the target sample dataset.

[0050] For example, the electronic device acquires the real-time position of the currently processed first sample data in the first sample dataset; further, in response to the real-time position matching the injection position, it injects second sample data and the corresponding second annotation into the injection position.

[0051] The data processing method provided in this embodiment injects the second sample data into the first sample dataset through the injection location to obtain the target sample dataset. This achieves automatic acquisition of the target sample dataset, is easy to operate, and helps to improve the overall annotation processing efficiency.

[0052] In addition, by grouping the first sample dataset through the injection interval, at least one data subset is obtained. Then, for each data subset, a number of injection positions are randomly generated, that is, each data subset corresponds to a random injection position. This allows the second sample data and the second label to be injected into the first sample dataset in a dispersed and random manner. This helps to avoid the random, chaotic and unpredictable injection positions that users may discover when processing the second sample data, thereby improving the reliability of the processing confidence.

[0053] In addition, by using the ratio between the first result identifier and the second result identifier, a second sample dataset is selected from a preset database according to the ratio. The correct second label and the incorrect second label with a controllable ratio are injected into the first sample dataset at the same time. This helps to avoid the influence of a single result identifier on the processing confidence, making the calculation basis of the subsequent processing confidence more balanced and comprehensive, and improving the reliability of the processing confidence.

[0054] Furthermore, by using the real-time position of the first sample data in the first sample dataset, and when the real-time position matches the injection position, the second sample data and the corresponding second label are injected into the injection position. The real-time injection of the second sample data improves the randomness of the injection of the second sample data. Moreover, the second sample data and the second label are injected only when the real-time position matches the injection position, instead of directly injecting all the second sample data. There is a certain overlap between the processing process and the second sample data injection process, which saves time and helps to improve the efficiency of obtaining the processing confidence.

[0055] Below, in conjunction with Figure 3 The processing of the target sample dataset and the methods for obtaining users will be introduced.

[0056] This embodiment provides a data processing method that can be used in the aforementioned mobile terminal and / or server (hereinafter referred to as electronic device). Figure 3 This is a flowchart of a data processing method according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps: Step S301: If the response meets the preset triggering conditions, at least one second candidate user is obtained based on the user confidence of each first candidate user.

[0057] In this embodiment, when acquiring a user for processing, the electronic device responds to a preset triggering condition and acquires at least one second candidate user based on the user confidence score of each first candidate user. The user confidence score characterizes the processing confidence score of the first candidate user for the historical sample dataset.

[0058] In one possible implementation, the preset trigger condition is the completion of sample dataset labeling; that is, the preset trigger condition is automatically met after the sample dataset is labeled. For example, after detecting the completion of the first sample data labeling, the electronic device collects the first sample data and its corresponding first label to generate the first sample dataset, and responds by obtaining and processing the user in response to the met preset trigger condition.

[0059] For example, to avoid the burden caused by a large backlog of unprocessed data, the electronic device determines that the labeling of the first sample data is complete when it detects that the number of currently labeled but unprocessed sample data has reached a second preset target value. The second preset target value can be any value and can be flexibly set and adjusted according to actual circumstances.

[0060] In another possible implementation, the preset trigger condition is a manually triggered sample processing instruction. For example, after the first sample data is labeled, the electronic device generates a sample processing instruction for the first sample dataset when it detects a user's sample processing operation on the first sample dataset, and responds by obtaining the processing user when the preset processing condition is met.

[0061] For example, when acquiring a second candidate user, the electronic device acquires a first candidate user whose user confidence level is greater than a first threshold, thus obtaining at least one third candidate user; further, it acquires the annotation type corresponding to the first sample dataset and the grouping information to which each third candidate user belongs; further, among the at least one third candidate user, the third candidate user whose grouping information matches the annotation type is determined as the second candidate user. For example, the first threshold can be any value, and can be flexibly set and adjusted according to the actual situation.

[0062] For example, the classification of annotation types includes, but is not limited to, at least one of the following: Based on the above division of the annotation task initiator, that is, different sample datasets correspond to different sample task annotators; The content domain of the sample data is divided, such as the annotation of different modalities or different task types, such as text classification, named entity recognition, sentiment analysis, image semantic segmentation, and speech transcription; The business scenarios are divided based on sample data, such as annotation of medical texts, annotation of legal documents, annotation of e-commerce reviews, annotation of intelligent customer service dialogues, and other different business scenarios.

[0063] In one possible implementation, to reduce the consumption of human resources, the first candidate users include the users of the aforementioned sample labelers. For example, the first candidate users are the sample labelers whose user confidence level is greater than a second threshold. The second threshold is less than the first threshold. In another possible implementation, to ensure the accuracy of the processing results, the first candidate users do not include the aforementioned sample labelers.

[0064] Step S302: Based on the annotation association relationship between the candidate users and the first sample data, exclude users with annotation association relationships from at least one second candidate user to determine the processing users of the target sample dataset.

[0065] In this embodiment of the application, when acquiring the processing user, the electronic device excludes users with labeled association relationships from at least one second candidate user based on the labeled association relationship between the candidate user and the first sample data, so as to determine the processing user of the target sample dataset.

[0066] For example, users with standard association relationships refer to the labeled users corresponding to the first sample dataset. For example, different first sample datasets may correspond to the same or different labeled users, that is, one sample dataset may correspond to one or more labeled users. For example, based on the labeled association relationship between candidate users and the first sample data, the electronic device excludes the labeled users corresponding to the first sample dataset from at least one second candidate user, and determines the remaining second candidate users as the processing users of the target sample dataset.

[0067] Step S303: Obtain the target sample dataset.

[0068] Step S304: Obtain the annotation processing results for the target sample dataset.

[0069] Step S305: Based on the degree of matching between the second processing result and the known processing conclusion, obtain the processing confidence level corresponding to the second sample dataset.

[0070] Step S306: If the processing confidence level meets the preset first condition, the first processing result is determined to be reliable.

[0071] The above steps S303-S306 and Figure 2 Steps S201-S204 in the embodiment are similar; see details below. Figure 2 Examples are not detailed here.

[0072] The data processing method provided in this embodiment selects a second candidate user based on the confidence level of the first candidate user for the historical sample dataset, and then obtains the processing user for the first sample data by avoiding the labeling user of the first sample data. On the one hand, reliable users are selected as processing personnel based on user confidence level. On the other hand, it ensures that the processing user and the labeling user are different users, so as to avoid subjective bias caused by self-labeling and self-review, and improve the fairness and credibility of the labeling process.

[0073] In addition, a third candidate user with a qualified user confidence level is selected. Then, the labeling type corresponding to the first sample dataset is matched with the grouping information of the third candidate user. The third candidate user that matches the first labeling type is selected as the second candidate user for subsequent processing personnel to choose. Selecting processing users based on both processing ability and processing expertise helps to improve the reliability of label processing.

[0074] Below, in conjunction with Figure 4 This paper introduces the second sample dataset and the method for updating user confidence.

[0075] This embodiment provides a data processing method that can be used in the aforementioned mobile terminal and / or server (hereinafter referred to as electronic device). Figure 4 This is a flowchart of a data processing method according to an embodiment of this application, such as... Figure 4 As shown, the process includes the following steps: Step S401: Obtain the target sample dataset.

[0076] Step S402: Obtain the annotation processing results for the target sample dataset.

[0077] Step S403: Based on the degree of matching between the second processing result and the known processing conclusion, obtain the processing confidence level corresponding to the second sample dataset.

[0078] Step S404: If the processing confidence level meets the preset first condition, determine that the first processing result is reliable.

[0079] The above steps S401-S404 and Figure 2 Steps S201-S204 in the embodiment are similar; see details below. Figure 2 Examples are not detailed here.

[0080] Step S405: Use the first sample data as the new second sample data, use the first annotation as the new second annotation, and obtain the new known processing conclusion based on the second processing result.

[0081] In this embodiment of the application, after determining that the first processing result is credible, the electronic device uses the first sample data as the new second sample data and the first annotation as the new second annotation, and obtains a new known processing conclusion based on the second processing result.

[0082] For example, the new known processing conclusion includes the result identifier corresponding to the new second annotation, and the electronic device obtains the result identifier corresponding to the new second annotation based on the second processing result.

[0083] For example, if the confidence level meets the first condition, indicating that the first processing result for the user is credible, then each first sample data and its corresponding first label in the first sample dataset can be stored as a new second sample. For each first sample data, if the first processing result indicates that the first label is correct, then the new second label is assigned a first result identifier; if the first processing result indicates that the first label is incorrect, then the new second label is assigned a second result identifier.

[0084] Step S406: Store the new second sample data, the new second annotation, and the new known processing conclusions into the preset database.

[0085] In this embodiment of the application, the electronic device stores the new second sample data, the new second annotation, and the new known processing conclusions into a preset database.

[0086] Step S407: Update the user confidence of the processing user based on the degree of matching between the second processing result and the known processing conclusion.

[0087] In this embodiment of the application, after obtaining the second processing result, the electronic device updates the user confidence of the processing user based on the degree of matching between the second processing result and the known processing conclusion.

[0088] For example, after the user completes the processing of the target sample dataset, the electronic device uses the ratio of the matched second processing result to the total second processing result as an update parameter based on the degree of matching between the user's second processing result and the known processing conclusion; further, for the user, the update parameter and the currently stored confidence are equally weighted and averaged to obtain the updated user confidence.

[0089] The data processing method provided in this embodiment determines the first qualified sample data as the new second sample data and adds it to the database to expand the sample database. At the same time, it dynamically updates the user confidence of the processing user based on the processing matching situation, realizes the autonomous iterative expansion of the benchmark sample database and the dynamic updating of the processing personnel's capability profile, continuously optimizes the accuracy of subsequent processing confidence assessment, and forms a closed-loop processing system for bidirectional iterative optimization of sample resources and user capabilities, thereby improving the overall data annotation and processing effect in the long term.

[0090] In addition, combined Figure 5 This section provides a complete introduction to data processing methods.

[0091] For example, the terminal equipment includes an annotation terminal, a processing terminal, and a review terminal. The annotation terminal is the terminal equipment used by the annotation user, the processing terminal is the terminal equipment used by the processing user, and the review terminal is the terminal equipment used by the review user.

[0092] The annotation terminal displays the first sample data to the annotation user through the application; further, in response to the annotation user's annotation operation on the first sample data, it obtains the first annotation corresponding to the first sample data; further, the annotation terminal sends the first sample data and the first annotation to the server.

[0093] Subsequently, in response to the fulfillment of the preset triggering conditions, the server stores the second sample data, the second annotation, and the result identifier. For the first sample dataset containing the first sample data, the server determines the injection position according to the preset injection parameters and dynamically pushes the first sample data and the second sample data to the processing terminal during the processing process. The processing terminal receives the processing operation of the processing user and returns the annotation processing result to the server. The annotation processing result includes the first processing result for the first annotation and the second processing result for the second annotation.

[0094] Subsequently, the server obtains the processing confidence level based on the degree of matching between the second processing result and the result identifier; further, if the processing confidence level meets the preset first condition, the first processing result is determined to be credible; and if the processing confidence level does not meet the preset first condition, the first processing result is determined to be unreliable, and then the server sends the first sample dataset to the review terminal. The review terminal displays the first sample dataset to the review user, and the review user reviews the first annotation contained in the first sample dataset through the review terminal. This embodiment also provides a data processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0095] This embodiment provides a data processing device, such as... Figure 6 As shown, it includes: The data acquisition module 601 is used to acquire a target sample dataset, which includes a first sample dataset to be processed and a second sample dataset for reference; the first sample dataset includes first sample data and corresponding first annotations; the second sample dataset includes second sample data and corresponding second annotations, and the second sample dataset contains known processing conclusions; The result acquisition module 602 is used to acquire the annotation processing results for the target sample dataset, the annotation processing results including a first processing result for the first annotation and a second processing result for the second annotation; The result matching module 603 is used to obtain the processing confidence level corresponding to the second sample dataset based on the degree of matching between the second processing result and the known processing conclusion. The result processing module 604 is used to determine that the first processing result is reliable if the processing confidence level meets a preset first condition.

[0096] The data processing apparatus provided in this application embodiment can execute the data processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0097] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0098] The following is a detailed reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 702 or a program loaded from memory 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and sample data required for the operation of the electronic device. The processor 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0099] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange sample data. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0100] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a memory 708, or installed from a ROM 702. When the computer program is executed by the processor 701, it performs the functions defined in the data processing method of the embodiments of this application.

[0101] Figure 7The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0102] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the data processing methods shown in the above embodiments are implemented.

[0103] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0104] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a target sample dataset, which includes a first sample dataset to be processed and a second sample dataset for reference; the first sample dataset includes first sample data and corresponding first annotations; the second sample dataset includes second sample data and corresponding second annotations, and the second sample dataset contains known processing conclusions; Obtain the annotation processing results for the target sample dataset, the annotation processing results including a first processing result for the first annotation and a second processing result for the second annotation; Based on the degree of matching between the second processing result and the known processing conclusion, the processing confidence level corresponding to the second sample dataset is obtained; If the processing confidence level meets a preset first condition, the first processing result is determined to be reliable.

2. The method according to claim 1, characterized in that, The acquisition of the target sample dataset includes: Obtain the first sample dataset and the mixing strategy; Obtain the second sample dataset; Based on the mixing strategy, the second sample data is mixed into the first sample dataset to obtain the target sample dataset.

3. The method according to claim 2, characterized in that, The process of obtaining the first sample dataset and the mixing strategy includes: Obtain the first sample dataset and preset injection parameters, the injection parameters including injection interval and injection quantity; The first sample dataset is grouped based on the injection interval to obtain at least one data subset; For each of the data subsets, the number of injection positions are randomly generated; wherein, the hybrid strategy includes the injection positions.

4. The method according to claim 2, characterized in that, The process of obtaining the second sample dataset includes: Obtain the ratio between the first result identifier and the second result identifier; wherein, the first result identifier is used to indicate that the second label is correct, and the second result identifier is used to indicate that the second label is incorrect; Based on the stated ratio, the second sample dataset is obtained from a preset database.

5. The method according to claim 2, characterized in that, The step of mixing the second sample data into the first sample dataset based on the mixing strategy includes: Obtain the real-time position of the currently processed first sample data in the first sample dataset; In response to the real-time location matching the injection location, the second sample data and the corresponding second label are injected into the injection location.

6. The method according to claim 1, characterized in that, The method further includes: If the processing confidence does not meet the first condition, the first sample dataset is sent to the annotation task initiator of the first sample dataset so that the annotation task initiator can re-annotate the first sample dataset.

7. The method according to claim 1, characterized in that, The method further includes: If the response meets the preset triggering conditions, at least one second candidate user is obtained based on the user confidence score of each first candidate user; wherein, the user confidence score is used to characterize the processing confidence score of the first candidate user for the historical sample dataset; Based on the annotation association relationship between candidate users and the first sample data, users with annotation association relationships are excluded from the at least one second candidate user in order to determine the processing users of the target sample dataset.

8. The method according to claim 7, characterized in that, The step of obtaining at least one second candidate user based on the user confidence scores of each first candidate user includes: Obtain the first candidate user whose user confidence score is greater than the first threshold, and obtain at least one third candidate user; Obtain the annotation type corresponding to the first sample dataset, and the group information to which each of the third candidate users belongs; Among the at least one third candidate user, the third candidate user whose grouping information matches the labeling type is determined as the second candidate user.

9. The method according to claim 1, 4, or 7, characterized in that, The method further includes: The first sample data is used as the new second sample data, the first annotation is used as the new second annotation, and a new known processing conclusion is obtained based on the second processing result. The new second sample data, the new second annotation, and the new known processing conclusion are stored in a preset database; Based on the degree of matching between the second processing result and the known processing conclusion, the user confidence of the processing user is updated.

10. A data processing apparatus, characterized in that, The device includes: The data acquisition module is used to acquire a target sample dataset, which includes a first sample dataset to be processed and a second sample dataset for reference; the first sample dataset includes first sample data and corresponding first annotations; the second sample dataset includes second sample data and corresponding second annotations, and the second sample dataset contains known processing conclusions; The result acquisition module is used to acquire the annotation processing results for the target sample dataset, the annotation processing results including a first processing result for the first annotation and a second processing result for the second annotation; The result matching module is used to obtain the processing confidence level corresponding to the second sample dataset based on the degree of matching between the second processing result and the known processing conclusion. The result processing module is used to determine that the first processing result is reliable if the processing confidence level meets a preset first condition.

Images