Data quality inspection method and device, electronic equipment and storage medium

Abstract

The invention provides a data quality inspection method and device, electronic equipment and a storage medium, and relates to the technical field of computers. Comprising the steps of obtaining first data in a to-be-delivered data set, randomly extracting a first preset number of second data from the first data, and obtaining a second score corresponding to each piece of second data; determining third data of which the corresponding second score is greater than or equal to a preset first score threshold value from the second data, and when the first proportion of the third data in the second data is smaller than a preset first proportion threshold value, determining that the second score of the third data is greater than or equal to the preset first score threshold value; and determining first target data from the current remaining data of the data set according to the second proportion and a preset confidence interval width, and delivering the first target data and the third data as qualified data. According to the method, manual quality inspection with a low proportion is carried out, and the machine quality inspection data is extracted from the current residual data through the evaluation model as the delivery data, so that secondary or multiple quality inspection is avoided, and the efficiency of data quality inspection is effectively improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology, and particularly relates to a data quality inspection method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the gradual popularization of the application of artificial intelligence models, data quality inspection of artificial intelligence models is a key step to guarantee data quality and a key means to improve data confidence. Different types of data quality inspection methods can be various, for example, corpus data required by a large language model (LLM) will be scored from multiple dimensions such as language logic, fluency, application scenario, etc.; for the data of the image-text of artificial intelligence generated content (AIGC), scoring will be performed from multiple dimensions such as image-text consistency, image fineness, artistic level, etc.; and a conventional target detection model can be scored according to whether the data is correctly framed.

[0003] The common quality inspection methods in the related art are various: (1) manual quality inspection, an annotator determines all annotation results, and complex samples often rely on expert experience or multi-person cooperation; (2) machine quality inspection, a model (for example, a large language model) scores the results, and according to the comparison of the scoring results and a preset threshold, it is determined whether the data quality inspection result is passed; (3) mixed quality inspection, which is a mixed quality inspection of manual scoring and machine scoring, uses different combinations to output the evaluation results of samples.

[0004] The above-mentioned method (3) is a reliable direction, which is a mixed quality inspection of manual quality inspection and machine quality inspection. The solution for the quality inspection not passing after manual quality inspection is imperfect, there is a risk of secondary or multiple times, and the efficiency of data quality inspection is low. SUMMARY

[0005] The purpose of the embodiments of the present disclosure is to provide a data quality inspection method and device, electronic equipment and storage medium.

[0006] To solve the above technical problems, the embodiments of the present disclosure are implemented through the following aspects.

[0007] According to a first aspect of embodiments of the present disclosure, a method for data quality inspection is provided. The method comprises: obtaining first data in a data set to be delivered, randomly sampling a first preset number of second data from the first data, obtaining a second score corresponding to each of the second data, the second score being obtained after artificial quality inspection of the second data; determining third data corresponding to the second score greater than or equal to a preset first score threshold from the second data; in a case where a first proportion of the third data in the second data is less than a preset first proportion threshold, determining first target data from current remaining data of the data set according to a second proportion and a preset confidence interval width, and delivering the first target data and the third data as qualified data; wherein the current remaining data is other data in the first data except the third data, and the second proportion is determined by a pre-trained evaluation model.

[0008] According to a second aspect of embodiments of the present disclosure, a device for data quality inspection is provided. The device comprises: an obtaining module configured to obtain first data in a data set to be delivered, randomly sample a first preset number of second data from the first data, and obtain a second score corresponding to each of the second data, the second score being obtained after artificial quality inspection of the second data; a processing module configured to determine third data corresponding to the second score greater than or equal to a preset first score threshold from the second data; the processing module is configured to, in a case where a first proportion of the third data in the second data is less than a preset first proportion threshold, determine first target data from current remaining data of the data set according to a second proportion and a preset confidence interval width, and deliver the first target data and the third data as qualified data; wherein the current remaining data is other data in the first data except the third data, and the second proportion is determined by a pre-trained evaluation model.

[0009] According to a third aspect of embodiments of the present disclosure, an electronic device is provided. The electronic device comprises: a processor; a memory for storing instructions executable by the processor; and wherein the processor is configured to perform the steps of the method for data quality inspection according to the first aspect.

[0010] According to a fourth aspect of embodiments of the present disclosure, a computer readable storage medium is provided. The computer readable storage medium stores one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the steps of the method for data quality inspection according to the first aspect.

[0011] According to a fifth aspect of the embodiments of the present disclosure, the computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, perform the steps of the method for data quality inspection according to the first aspect.

[0012] One of the above technical solutions has the following advantages or beneficial effects: by combining manual quality inspection and machine quality inspection, a lower proportion of manual quality inspection is performed, and by using a pre-trained evaluation model to extract machine quality inspection data from the current remaining data as delivery data, secondary or multiple quality inspections are avoided, and the efficiency of data quality inspection is effectively improved.

[0013] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure.

[0014] Other features and advantages of the present disclosure will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 A flowchart of a method for data quality inspection provided by an embodiment of the present disclosure is shown; Figure 2 Another flowchart of a method for data quality inspection provided by an embodiment of the present disclosure is shown; Figure 3 Another flowchart of a method for data quality inspection provided by an embodiment of the present disclosure is shown; Figure 4 Another flowchart of a method for data quality inspection provided by an embodiment of the present disclosure is shown; Figure 5 A block diagram of a data quality inspection device provided by an embodiment of the present disclosure is shown; Figure 6 Another block diagram of a data quality inspection device provided by an embodiment of the present disclosure is shown; Figure 7 A hardware structure schematic diagram of an electronic device for performing the method for data quality inspection provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0017] In order for those skilled in the art to better understand the technical solutions in the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in the following with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present disclosure.

[0018] The technical solutions of the present application will be described below in conjunction with specific embodiments.

[0019] Figure 1 A flowchart of a method for data quality inspection provided by an embodiment of the present disclosure is shown in FIG. 1, which can include the following steps: Figure 1 In step S101, a first data in a data set to be delivered is obtained, a first preset number of second data is randomly sampled from the first data, and a second score corresponding to each second data is obtained.

[0020] The second score is obtained after manual quality inspection of the second data.

[0021] By way of example, the first data in the data set to be delivered is the data to be inspected, and in some embodiments, machine quality inspection can be performed by a model (such as a large language model), so that a first score corresponding to each first data can be obtained. For example, the data set to be delivered includes M first data, and a first score corresponding to each of the M first data can be obtained by a model (such as a large language model).

[0022] A first preset number (e.g., n) of data can be randomly sampled from the M first data, and in some embodiments, the first preset number is determined according to a quantile of a standard normal distribution corresponding to a preset confidence interval, a confidence interval width, and a preset estimate.

[0023] Assuming that the probability of each data passing quality inspection is p , the case of n data passing quality inspection conforms to a binomial distribution B n , p and combining the fact that the binomial distribution is close to a normal distribution when the number is large enough (usually greater than or equal to 30), in some embodiments, a confidence interval can be generated by means of the Wald method of the binomial distribution, and the first preset number can be determined according to the quantile of the standard normal distribution corresponding to the confidence interval, the confidence interval width, and the preset estimate by the following formula one.

[0024] (Formula one) ​​Wherein, n is the first preset quantity, Z is the quantile of the standard normal distribution corresponding to the confidence interval, for example, the quantile of the standard normal distribution corresponding to the 95% confidence interval is 1.96, W is the preset confidence interval width, and p is the preset estimate, for example, which can be 0.5.

[0025] In step S102, third data corresponding to the second score greater than or equal to the preset first score threshold is determined from the second data.

[0026] Wherein, the third data whose second score is greater than or equal to the preset first score threshold is the data whose artificial scoring result is passed.

[0027] In step S103, in the case that the first proportion of the third data in the second data is less than the preset first proportion threshold, the first target data is determined from the current remaining data of the data set according to the second proportion and the preset confidence interval width, and the first target data and the third data are delivered as qualified data.

[0028] Wherein, the current remaining data is the other data in the first data except the third data, and the second proportion is determined by the pre-trained evaluation model.

[0029] In some embodiments, the first proportion q of the third data in the second data can be counted, and the relationship between the first proportion q and the preset first proportion threshold Q is compared to determine the corresponding delivery data. Wherein, in the case that the first proportion q is greater than or equal to the first proportion threshold Q, it can be considered that the first data as a whole meets the delivery requirements, and the first data is delivered as qualified data. And in the case that the first proportion q is less than the first proportion threshold Q, the third data can be taken as a part of qualified data, and the first target data is extracted from the current remaining data as qualified data for delivery.

[0030] By using the above technical solution, low proportion artificial quality inspection is carried out by combining artificial quality inspection and machine quality inspection, and the data for delivery is extracted from the current remaining data by the pre-trained evaluation model, so as to avoid secondary or multiple quality inspection, effectively improve the efficiency of data quality inspection, on the other hand, a complete data quality inspection theoretical basis is put forward, and a usable quality inspection scheme is put forward relying on the related theory, so that the number of artificial quality inspection can be accurately determined, and the amount of artificial quality inspection can replace the full sample confidence, greatly reducing the sampling error introduced in the sampling stage, and avoiding unreasonable quality inspection process caused by distribution difference.

[0031] Figure 2 Another flowchart of the method of data quality inspection provided by the embodiments of the present disclosure is shown as follows. Figure 2 As shown in the figure, step S103 can specifically include the following steps.

[0032] In step S1031, the third proportion is determined according to the second proportion, the confidence interval width, and the accuracy of the evaluation model.

[0033] In some embodiments, the accuracy of the evaluation model is obtained by verifying the evaluation model according to a verification set of the evaluation model, and the second proportion is determined by the evaluation model according to a proportion of positive samples in the training data, where the positive samples represent samples whose scoring results of the large language model and manual scoring results are both passed.

[0034] In some possible implementations, the third proportion can be determined by Formula Two as follows.

[0035] (Formula Two) wherein, is the third proportion, is the second proportion, which can be determined by the evaluation model according to a proportion of positive samples in the training data, for example, can be a product of the proportion of positive samples and a preset coefficient, J is the accuracy of the evaluation model determined by the evaluation model, which can be determined by a verification result of the evaluation model on the verification set, and W is a preset confidence interval width.

[0036] In another possible implementation, the industry experience bias can be further considered, and the third proportion can be determined by Formula Three as follows.

[0037] (Formula Three) wherein, represents the industry experience bias, .

[0038] In step S1032, the second preset number of fourth data is extracted from the current remaining data sorted according to the first score, and the second preset number is determined according to a product of the third proportion and the current remaining data.

[0039] wherein, the first score is obtained by scoring the first data by the large language model.

[0040] In some embodiments, the fourth data can be data with a high first score and a passed evaluation model result.

[0041] In step S1033, the current remaining data other than the fourth data is taken as new current remaining data, and the step of extracting the second preset number of fourth data from the current remaining data sorted according to the first score is repeatedly executed until the total amount of the obtained fourth data reaches a preset number, or the number of repeated execution reaches a preset number threshold.

[0042] In some possible implementations, after the (i-1)th extraction of data, the total amount of the current remaining data can be determined by Formula Four as follows.

[0043] (Formula Four) wherein, is the current remaining data after the i-1th extraction of data, M is the total amount of the first data, represents the total amount of data extracted the jth time, which can be determined by It should be noted that in the embodiments of the present application, the total amount of the first data can be constantly changing, for example, new data can be generated and added, which is not limited in the present application.

[0044] The data extracted through step S1033 is the machine quality inspection result selected according to the evaluation model within the confidence interval, which can be passed data and delivered together with the third data as qualified data.

[0045] In step S1034, the collection of the fourth data obtained by each execution of the step of extracting the second preset number of fourth data from the current remaining data sorted according to the first score is taken as the first target data.

[0046] Figure 3 Another flowchart of the method for data quality inspection provided by the embodiments of the present application is shown, as Figure 3 The method can include the following steps.

[0047] In step 100, the first preset number of data is subjected to manual quality inspection.

[0048] In step 101, the third data corresponding to the second score greater than or equal to the preset first score threshold is determined from the second data, and in the case that the first proportion of the third data in the second data is less than the preset first proportion threshold, the second preset number of fourth data can be re-extracted from the current remaining data.

[0049] The specific process can be referred to steps S1032 and S1033, which will not be expanded here.

[0050] In step 102, the data after quality inspection is delivered.

[0051] Specifically, it can include any one of the following: 1) In the case that the first proportion is greater than or equal to the first proportion threshold, the first data is delivered as qualified data; 2) In the case that the extraction number reaches the preset number threshold, the current accumulated fourth data can be delivered as the first target data (which can be delivered together with the third data or batch-delivered, which is not limited in the present application).

[0052] It can be understood that, for example, the remaining data cannot meet the requirements (for example, the first score of the large language model is lower than the preset second score threshold, or the result of the evaluation model is not passed), and the current accumulated fourth data can be delivered as qualified data when the extraction number l reaches the preset number threshold L.

[0053] 3) When the total amount of the fourth data reaches the preset number, the current accumulated fourth data can be delivered as the first target data (which can be delivered together with the third data or in batches, which is not limited by the present application).

[0054] For example, the total delivery data is 2000, and the number of third data is 600, then the current accumulated fourth data can be delivered as the first target data when the total amount of the fourth data reaches the preset number (2000-600=1400).

[0055] The above technical solutions are adopted, the artificial quality inspection and machine quality inspection are combined, the artificial quality inspection with a lower proportion is performed, and the data for machine quality inspection is extracted from the current remaining data as delivery data through the pre-trained evaluation model, so that secondary or multiple quality inspections are avoided, and the efficiency of data quality inspection is effectively improved.

[0056] Figure 4 Another flowchart of a method for data quality inspection provided by an embodiment of the present disclosure is shown in FIG. 6. Figure 4 As shown in FIG. 6, the evaluation model can be obtained based on the following method, and in some implementations, the training of the evaluation model can be completed before step S101.

[0057] In step S201, the third score corresponding to each training data is obtained by scoring the training data through the large language model.

[0058] In step S202, the evaluation label corresponding to each training data is obtained by scoring the training data through the artificial quality inspection.

[0059] For example, assuming that the total amount of training data is k, and the number of training data that passes the artificial scoring is m (m is less than or equal to k), then the number of training data that does not pass is k-m.

[0060] The third scores can be arranged in descending order as The evaluation label corresponding to each training data can be determined by the following formula five.

[0061] (Formula five) Wherein, is the evaluation label corresponding to the i-th training data. is the third score of the i-th training data.

[0062] In step S203, the evaluation model to be trained is trained according to the training data, the third score corresponding to each training data respectively, and the evaluation label corresponding to each training data respectively, to obtain the trained evaluation model.

[0063] The evaluation model can be a binary classification model, for example, can be a large language model, and the model type of the specific binary classification model is not limited in the present application. The training data can be text, image or a combination of the two. Taking the combination of image and text as the training data and the large language model as the evaluation model as an example, before inputting the evaluation model to be trained, each image can be converted into a corresponding image token sequence, and the text description of the image corresponding Question and Answer can also be converted into corresponding text tokens. The input language token of the evaluation model can be “Please score and rank”, and the output corresponding evaluation result.

[0064] In some embodiments, the loss function of the evaluation model can be shown in Formula Six as follows.

[0065] (Formula Six) Wherein, represents the loss function, represents the evaluation label corresponding to the i th training data, is the evaluation model, represents the parameters of the evaluation model, represents the third score corresponding to the i th training data, is the i th training data.

[0066] The above loss function aims to minimize the deviation between the evaluation result of the evaluation model and the evaluation label, and continuously optimizes the parameters of the evaluation model to be trained, thereby obtaining the trained evaluation model.

[0067] It can be understood that the data executing step S101 can also be input into the evaluation model to continuously optimize the evaluation model, thereby further improving the applicability of the evaluation model.

[0068] By using the above technical solution, the evaluation model can be trained, and the data extracted by the machine quality inspection from the current remaining data as the delivery data through the pre-trained evaluation model can be facilitated, so as to avoid secondary or multiple quality inspection, and effectively improve the efficiency of data quality inspection.

[0069] Figure 5 A block diagram of a data quality inspection device provided by an embodiment of the present application is shown in FIG. 2. Figure 5 As shown in FIG. 2, the data quality inspection device 200 comprises: The acquisition module 210 is configured to acquire first data in a data set to be delivered. The processing module 220 is configured to randomly extract a first preset number of second data from the first data, and acquire a second score corresponding to each second data, wherein the second score is obtained by performing artificial quality inspection on the second data. The processing module 220 is configured to determine third data corresponding to a second score greater than or equal to a preset first score threshold from the second data. The processing module 220 is configured to, in a case where a first proportion of the third data in the second data is less than a preset first proportion threshold, determine a first target data from current remaining data of the data set according to a second proportion and a preset confidence interval width, and deliver the first target data and the third data as qualified data. The current remaining data is other data in the first data except the third data, and the second proportion is determined by a pre-trained evaluation model.

[0070] Optionally, the first preset number is determined according to a quantile of a standard normal distribution corresponding to a preset confidence interval, a confidence interval width, and a preset estimation quantity.

[0071] Optionally, the processing module 220 is further configured to, in a case where the first proportion is greater than or equal to the first proportion threshold, deliver the first data as qualified data.

[0072] Optionally, the processing module 220 is further configured to determine the third proportion according to the second proportion, the confidence interval width, and an accuracy rate of the evaluation model. extract a second preset number of fourth data from the current remaining data sorted according to the first score, wherein the second preset number is determined according to a product of the third proportion and the current remaining data, and the first score is obtained by scoring the first data by a large language model; treat other data of the current remaining data except the fourth data as new current remaining data, and repeatedly execute the step of extracting the second preset number of fourth data from the current remaining data sorted according to the first score until a total amount of the obtained fourth data reaches a preset number, or a number of repeated executions reaches a preset number threshold; combine the fourth data obtained by each execution of the step of extracting the second preset number of fourth data from the current remaining data sorted according to the first score as the first target data.

[0073] Optionally, the accuracy rate of the evaluation model is obtained by verifying the evaluation model according to a verification set of the evaluation model, and the second proportion is determined by the evaluation model according to a proportion of positive samples in training data, wherein the positive samples represent samples in which the scoring result of the large language model and the artificial scoring result are both passed.

[0074] The device 200 provided by the embodiments of the present application can execute the methods in the foregoing method embodiments, realize the functions of the methods in the foregoing method embodiments, and achieve the corresponding beneficial effects, which will not be described here again.

[0075] Figure 6 A block diagram of another device for data quality inspection provided by the embodiments of the present application is shown in FIG. 2B. Figure 6 As shown in FIG. 2B, the device 200 for data quality inspection further includes a training module 230, configured to: a third score corresponding to each training data respectively obtained by scoring the training data by the large language model; an evaluation label corresponding to each training data respectively obtained by scoring the training data by the human; training an evaluation model to be trained according to the training data, the third score corresponding to each training data respectively, and the evaluation label corresponding to each training data respectively, to obtain a trained evaluation model.

[0076] The device 200 provided by the embodiments of the present application can execute the methods in the foregoing method embodiments, realize the functions of the methods in the foregoing method embodiments, and achieve the corresponding beneficial effects, which will not be described here again.

[0077] Figure 7 A schematic diagram of a hardware structure of an electronic device for executing the embodiments of the present disclosure is shown in FIG. 3. Figure 7 As shown in FIG. 3, at the hardware level, the electronic device includes at least one processor, and optionally includes an internal bus, a network interface, and a memory. The memory can include an internal memory such as a high-speed random access memory (RAM), and can also include a non-volatile memory such as at least one disk memory. Of course, the electronic device can also include other hardware required by the business.

[0078] The processor, the network interface, and the memory can be connected to each other through the internal bus, which can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0079] The memory stores programs. Specifically, the programs can include program codes including at least one computer operation instruction. The memory can include an internal memory and a nonvolatile memory, and provide instructions and data for the processor.

[0080] The at least one processor reads the corresponding computer program from the nonvolatile memory into the internal memory and then runs, and forms the device for locating the target user at a logical level. The at least one processor executes the programs stored in the memory, and specifically executes the method disclosed in the embodiment of the first aspect and realizes the functions and beneficial effects of the methods disclosed in the foregoing method embodiments, which will not be repeated here.

[0081] The method disclosed in the embodiment of the first aspect of the present disclosure can be applied to at least one processor or implemented by at least one processor. The processor can be an integrated circuit chip with a processing capability of signals. In the implementation process, each step of the above method can be completed by integrated logic circuits or instructions in the form of software in the at least one processor. The processor mentioned above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiments of the present disclosure can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0082] The electronic device can also execute the methods disclosed in the foregoing method embodiments, and realize the functions and beneficial effects of the methods disclosed in the foregoing method embodiments, which will not be repeated here.

[0083] Of course, in addition to the software implementation, the electronic device of the present disclosure does not exclude other implementation manners, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or a logic device.

[0084] The embodiment of the present disclosure further provides a computer readable storage medium, which stores one or more programs, and the one or more programs, when executed by at least one processor, implement the method disclosed in the embodiment of the first aspect and achieve the functions and beneficial effects of the methods described in the foregoing method embodiments, which are not repeated here.

[0085] The computer readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disc or an optical disc, and the like.

[0086] Further, the embodiment of the present disclosure further provides a computer program product, which includes a computer program stored on a non-transitory computer readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a computer, the following processes are implemented: the method disclosed in the embodiment of the first aspect and the functions and beneficial effects of the methods described in the foregoing method embodiments, which are not repeated here.

[0087] In summary, the above only describes the preferred embodiments of the present disclosure, and does not limit the protection scope of the present disclosure. Any modification, equivalent replacement, improvement, and the like made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0088] The system, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer may, for example, be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0089] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0090] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0091] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

Claims

1. A method for data quality inspection, characterized in that, The method comprises: obtaining first data in a data set to be delivered, randomly sampling a first preset number of second data from the first data, obtaining a second score corresponding to each second data, the second score being obtained after artificial quality inspection of the second data; determining third data corresponding to the second score greater than or equal to a preset first score threshold from the second data; in the case that the first proportion of the third data in the second data is less than a preset first proportion threshold, determining a first target data from the current remaining data of the data set according to a second proportion and a preset confidence interval width, and delivering the first target data and the third data as qualified data; wherein the current remaining data is other data in the first data except the third data, and the second proportion is determined by a pre-trained evaluation model.

2. The method of claim 1, wherein, The first preset number is determined according to a quantile of a standard normal distribution corresponding to a preset confidence interval, the confidence interval width, and a preset estimate.

3. The method of claim 1, wherein, The method further comprises: in the case that the first proportion is greater than or equal to the first proportion threshold, delivering the first data as qualified data.

4. The method of claim 1, wherein, The determination of the first target data from the current remaining data of the data set according to the second proportion and the preset confidence interval width comprises: determining a third proportion according to the second proportion, the confidence interval width, and the accuracy of the evaluation model; sampling a second preset number of fourth data from the current remaining data sorted according to the first score, the second preset number being determined according to the product of the third proportion and the current remaining data, and the first score being obtained by scoring the first data by a large language model; regarding other data in the current remaining data except the fourth data as new current remaining data, repeating the step of sampling a second preset number of fourth data from the current remaining data sorted according to the first score until the total amount of the obtained fourth data reaches a preset amount, or the number of repetitions reaches a preset number threshold; regarding the union of the fourth data obtained by each execution of the step of sampling a second preset number of fourth data from the current remaining data sorted according to the first score as the first target data.

5. The method of claim 4, wherein, The accuracy of the evaluation model is obtained by verifying the evaluation model according to a verification set of the evaluation model, and the second proportion is determined by the evaluation model according to the proportion of positive samples in training data, wherein the positive samples represent samples whose scoring results by the large language model and artificial scoring results are both passed.

6. The method according to any one of claims 1 to 5, characterized in that, The evaluation model is trained based on the following method: a third score corresponding to each training data obtained by scoring the training data by a large language model; obtaining an evaluation label corresponding to each training data obtained by artificially scoring the training data; The training data, the third score corresponding to each training data, and the evaluation label corresponding to each training data are used to train an evaluation model to be trained to obtain the trained evaluation model.

7. A device for data quality inspection, characterized in that, The device comprises: The acquisition module is configured to acquire first data in a data set to be delivered, randomly extract a first preset number of second data from the first data, and acquire a second score corresponding to each second data, wherein the second score is obtained after manual quality inspection of the second data. The processing module is configured to determine third data corresponding to the second data from the second data, wherein the second score of the third data is greater than or equal to a preset first score threshold. The processing module is configured to determine first target data from the current remaining data in the data set according to a second proportion and a preset confidence interval width when the first proportion of the third data in the second data is less than a preset first proportion threshold, and deliver the first target data and the third data as qualified data. The current remaining data is other data in the first data except the third data, and the second proportion is determined by a pre-trained evaluation model.

8. An electronic device, comprising: The computer program is stored in the memory and executable on the processor, and when the computer program is executed by the processor, the method for data quality inspection according to any one of claims 1 to 6 is implemented. The computer program is stored in the memory and executable on the processor, and when the computer program is executed by the processor, the method for data quality inspection according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that, The computer program product comprises a computer program stored in a non-transitory computer readable storage medium, and the computer program comprises program instructions, which, when executed by a computer, implement the method for data quality inspection according to any one of claims 1 to 6.

10. A computer program product, characterised in that, ​

Images