Repeated record detection method and device and computer readable storage medium

Abstract

The invention provides a duplicate record detection method and device and a computer readable storage medium, and the method comprises the steps: obtaining a record pair, inputting the record pair into a first duplicate record detection model, and obtaining first prediction information (including the probability that the first duplicate record detection model predicts that two records in the record pair are duplicate or not duplicate); outputting a determination result when it can be determined whether the two records in the record pair are duplicated based on the first prediction information, and inputting the record pair into a second duplicated record detection model having higher accuracy than the first duplicated record detection model when it cannot be determined whether the two records in the record pair are duplicated based on the first prediction information; and obtaining second prediction information (including a probability that the second duplicate record detection model predicts that the two records in the record pair are duplicated or not), determining whether the two records in the record pair are duplicated based on the second prediction information, and outputting a determination result. According to the method, repeated record detection can be realized, the detection precision is ensured, and the cost is saved.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on October 18, 2023, with application number 202311355965.8, and priority to the Chinese patent application with the invention name “Data processing method, device and computer-readable storage medium”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of data processing technology, and in particular to a duplicate record detection method, device and computer-readable storage medium. Background Art

[0003] In the fields of population data management, health care data management, data preparation for machine learning, etc., there are usually many duplicate records in the acquired raw data. The so-called duplicate records refer to multiple records of the same entity in the real world. For example, in Table 1 and Table 2, it can be seen that the content of the row to which Zhang San belongs in Table 1 and the content of the row to which Zhang San belongs in Table 2 are duplicate records, and the content of the row to which Li Si belongs in Table 1 and the content of the row to which Li Si belongs in Table 2 are duplicate records.

[0004] Table 1

[0005] Name age gender Date of Birth Zhang San 20 male 1992.03.25 Li Si 21 male 1985.02.21

[0006] Table 2

[0007] Name age gender Date of Birth Zhang San 20 years old Male 1992 / 03 / 25 Li Si 21 years old Male 1985 / 02 / 21

[0008] It is understandable that the presence of duplicate records in the original data will affect the data quality and the efficiency of artificial intelligence (AI) model training. For example, when conducting population census, if the population of city A includes Zhang San and the population of city B also includes Zhang San, the total population count will be wrong. If there is duplicate data in the training data obtained during artificial intelligence model training, the model will be redundantly trained using the duplicate data, resulting in low model training efficiency. Summary of the invention

[0009] The present application provides a duplicate record detection method, device and computer-readable storage medium, which can realize duplicate record detection to improve data quality, ensure detection accuracy and save costs.

[0010] In a first aspect, a duplicate record detection method is provided, which can be applied to a duplicate record detection system, the system comprising a first duplicate record detection model and a second duplicate record detection model, the accuracy of the first duplicate record detection model being lower than the accuracy of the second duplicate record detection model, the method comprising the following steps: obtaining a first record pair; inputting the first record pair into the first duplicate record detection model to obtain first prediction information, the first prediction information comprising the probability that the first duplicate record detection model predicts that the two records in the first record pair are repeated or not; judging whether it is possible to determine whether the two records in the first record pair are repeated based on the first prediction information, and in the case that it is possible to determine whether the two records in the first record pair are repeated based on the first prediction information, outputting a first determination result, the first determination result being whether the two records in the first record pair are repeated or not; in the case that it is not possible to determine whether the two records in the first record pair are repeated based on the first prediction information, inputting the first record pair into the second duplicate record detection model to obtain second prediction information, the second prediction information comprising the probability that the second duplicate record detection model predicts that the two records in the first record pair are repeated or not; determining whether the two records in the first record pair are repeated based on the second prediction information, and outputting the first determination result.

[0011] The accuracy of the above-mentioned first duplicate record detection model is lower than that of the second duplicate record detection model. It can be understood that, under normal circumstances, since the cost of a low-precision model is lower than the cost of a high-precision model, including training cost, inference cost, etc., the cost of the first duplicate record detection model is lower than the cost of the second duplicate record detection model.

[0012] The above scheme first uses a low-precision (low-cost) model for detection. When it cannot be determined whether two records in a record pair are duplicates based on the prediction information of the low-precision model, a high-precision (high-cost) model is used for detection. Compared with the current scheme of only using a low-precision model for duplicate record detection, the detection accuracy can be improved.

[0013] Moreover, under normal circumstances, a low-precision model will output a large number of record pairs that cannot be determined whether they are repeated. In the current solutions that only use low-precision models for detection, most of the record pairs output by the low-precision model that cannot be determined whether they are repeated are manually detected, which is time-consuming and labor-intensive and will consume a lot of costs. The present application uses a high-precision model to detect record pairs output by the low-precision model that cannot be determined whether they are repeated, thereby reducing costs and improving detection efficiency.

[0014] Furthermore, compared with the current solution of only using a high-precision model for duplicate record detection, the above solution does not need to use a high-precision model for detection when a low-precision model can already identify whether two records are duplicates. This can reduce the workload of the high-precision model. It can be understood that, generally, since a high-precision model is more complex than a low-precision model, the high-precision model will occupy more computing resources than the low-precision model each time it works, and the high-precision model will also take longer to detect each time than the low-precision model. Therefore, the use of the above solution can improve detection efficiency, reduce the consumption of computing resources, and further reduce costs.

[0015] In some possible implementations, the computing resources occupied by the second duplicate record detection model for detection are greater than the computing resources occupied by the first duplicate record detection model for detection.

[0016] In some possible implementations, the time consumed by the second duplicate record detection model for detection is longer than the time consumed by the first duplicate record detection model for detection.

[0017] In some possible implementations, the first duplicate record detection model is a model that uses multiple duplicate record detection rules to perform duplicate record detection, the second duplicate record detection model is an artificial intelligence model, or both the first duplicate record detection model and the second duplicate record detection model are artificial intelligence models.

[0018] In some possible implementations, when the second duplicate record detection model is a deep learning model, before determining whether it is possible to determine whether two records in the first record pair are duplicates based on the second prediction information, the method further includes the following step: calibrating the second prediction information.

[0019] Since deep learning models usually have the problem of overconfidence, which means that the model is too confident in its prediction results when predicting, and ignores the uncertainty and error rate of the prediction results, by implementing the above implementation method, when the second duplicate record detection model is a deep learning model, calibrating the second prediction information can make the second prediction information more accurate and reliable, thereby improving the accuracy of subsequent judgment based on the second prediction information on whether the two records in the first record pair are duplicates.

[0020] In some possible implementations, the method further includes the following steps: obtaining a second record pair; inputting the second record pair into a first duplicate record detection model to obtain third prediction information, the third prediction information including the probability that the first duplicate record detection model predicts that the two records in the second record pair are repeated or not; determining whether the two records in the second record pair are repeated based on the third prediction information, and using the second determination result as a classification label for the second record pair, and adding it to a training sample set of the second duplicate record detection model, the second determination result being whether the two records in the second record pair are repeated or not.

[0021] When the second duplicate record detection model requires supervised training of the artificial intelligence model, the training samples of the second duplicate record detection model need to carry corresponding classification labels. If a large number of training samples are manually labeled, it will be very time-consuming and labor-intensive, and the cost is high. By implementing the above-mentioned implementation method, the unlabeled training samples of the second duplicate record detection model are detected through the first duplicate record detection model to obtain the corresponding classification labels, which can improve the efficiency of obtaining classification labels, thereby improving the efficiency of model training and reducing costs.

[0022] In some possible implementations, the first duplicate record detection model and the second duplicate record detection model are deployed on the same computing device or on different computing devices.

[0023] It can be understood that when two models are deployed on the same computing device, compared with being deployed on different computing devices, in the former deployment mode, the communication distance between the two models is shorter, the communication efficiency is higher, and the duplicate detection efficiency is higher.

[0024] In a second aspect, a duplicate record detection device is provided, which can be applied to a duplicate record detection system, the system comprising a first duplicate record detection model and a second duplicate record detection model, the accuracy of the second duplicate record detection model is higher than the accuracy of the first duplicate record detection model, the device comprising:

[0025] An acquisition module, used for acquiring a first record pair;

[0026] a detection module, configured to input the first record pair into the first duplicate record detection model to obtain first prediction information, wherein the first prediction information includes a probability that the first duplicate record detection model predicts that two records in the first record pair are duplicate or non-duplicate;

[0027] The detection module is further configured to determine whether it is possible to determine whether the two records in the first record pair are repeated based on the first prediction information, and output a first determination result if it is possible to determine whether the two records in the first record pair are repeated based on the first prediction information, wherein the first determination result is whether the two records in the first record pair are repeated or not;

[0028] The detection module is further configured to input the first record pair into the second duplicate record detection model to obtain second prediction information when it cannot be determined whether the two records in the first record pair are duplicates based on the first prediction information, wherein the second prediction information includes a probability that the second duplicate record detection model predicts that the two records in the first record pair are duplicates or not;

[0029] The detection module is further used to determine whether two records in the first record pair are repeated based on the second prediction information, and output the first determination result.

[0030] In some possible implementations, the computing resources occupied by the second duplicate record detection model for detection are greater than the computing resources occupied by the first duplicate record detection model for detection.

[0031] In some possible implementations, the time consumed by the second duplicate record detection model for detection is longer than the time consumed by the first duplicate record detection model for detection.

[0032] In some possible implementations, the first duplicate record detection model is a model that uses multiple duplicate record detection rules to perform duplicate record detection, the second duplicate record detection model is an artificial intelligence model, or both the first duplicate record detection model and the second duplicate record detection model are artificial intelligence models.

[0033] In some possible implementations, when the second duplicate record detection model is a deep learning model, before the detection module is used to determine whether it can be determined whether the two records in the first record pair are duplicates based on the second prediction information, the device also includes: a calibration module for calibrating the second prediction information.

[0034] In some possible implementations, the acquisition module is also used to acquire a second record pair; the detection module is also used to input the second record pair into the first duplicate record detection model to obtain third prediction information, the third prediction information including the probability that the first duplicate record detection model predicts that the two records in the second record pair are repeated or not; the detection module is also used to determine whether the two records in the second record pair are repeated based on the third prediction information, and use the second determination result as the classification label of the second record pair, and add it to the training sample set of the second duplicate record detection model, the second determination result being whether the two records in the second record pair are repeated or not.

[0035] In some possible implementations, the first duplicate record detection model and the second duplicate record detection model are deployed on the same computing device or on different computing devices.

[0036] In a third aspect, a computing device is provided, comprising a processor and a memory, wherein the memory is used to store instructions and the processor is used to execute instructions, so that the computing device implements the method described in the first aspect and any implementation manner of the first aspect.

[0037] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed by a computing device or a computing device cluster, the method described in the first aspect and any implementation manner of the first aspect is implemented.

[0038] In a fifth aspect, a computing device cluster is provided, which includes at least one computing device, each of the at least one computing device includes a processor and a memory, and the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster implements the method described in the first aspect and any implementation method of the first aspect.

[0039] In a sixth aspect, a computer program product comprising instructions is provided. The computer program product includes instructions that can run on a computing device or be software or a program product stored in any available medium. When the computer program product runs on a computing device or a computing device cluster, the computing device or the computing device cluster executes the method described in the first aspect and any implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a schematic diagram of the architecture of a duplicate record detection system provided in an embodiment of the present application;

[0041] Figure 2 is a schematic diagram of the architecture of another duplicate record detection system provided in an embodiment of the present application;

[0042] Figure 3 is a schematic diagram of the architecture of another duplicate record detection system provided in an embodiment of the present application;

[0043] Figure 4 is an interactive schematic diagram of a duplicate record detection method provided in an embodiment of the present application;

[0044] Figure 5 is an interactive schematic diagram of another duplicate record detection method provided in an embodiment of the present application;

[0045] Figure 6is a flowchart of obtaining training samples of a second repeated record model provided by an embodiment of the present application;

[0046] Figure 7 is a schematic diagram of a duplicate record detection pipeline provided in an embodiment of the present application;

[0047] Figure 8 is a schematic diagram of another duplicate record detection pipeline provided in an embodiment of the present application;

[0048] Fig. 9 is a schematic diagram of another duplicate record detection pipeline provided in an embodiment of the present application;

[0049] Fig.10 is a structural schematic diagram of a duplicate record detection device provided in an embodiment of the present application;

[0050] Fig.11 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0051] Fig.12 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0052] Fig.13 It is a structural diagram of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solution provided by this application will be described below in conjunction with the accompanying drawings.

[0054] In order to facilitate a clear understanding of the technical solution provided by this application, some nouns and terms involved in the technical solution provided by this application are first explained.

[0055] (1) A rule-based model is a model based on predefined rules and logic, used to solve problems or make decisions. The rule-based model infers and reasons through a series of rules and conditions, matches the rules based on the input features and conditions, and obtains the corresponding output or decision results.

[0056] (2) Artificial intelligence models, also known as machine learning models, or machine learning algorithms or artificial intelligence algorithms, can be divided into traditional machine learning (machine learning) and deep learning (DL) models. Traditional machine learning models include but are not limited to support vector machines (SVMs), decision trees, random forests, etc. Deep learning models include but are not limited to models for generating word vectors (word to vector, Word2vec), generative pre-trained transformers (GPT), bidirectional encoder representations from transformers (BERT), etc.

[0057] (3) Confidence refers to the measure of the credibility of the model prediction results. The higher the confidence, the more credible the model prediction results are. Conversely, the lower the confidence, the less credible the model prediction results are.

[0058] The application scenarios of the embodiments of the present application are introduced below.

[0059] The present application involves scenarios where it is necessary to detect duplicate records in a data source, such as data deduplication scenarios and data repetition rate assessment scenarios. By identifying duplicate records, it is possible to reduce redundancy and duplicate information and improve data quality. Another example is a data detection scenario based on data on a black / white list. By identifying whether there are records on the black / white list that are duplicates of the information to be checked, data detection is achieved. Another example is an artificial intelligence model training data preparation scenario. By identifying duplicate records in the training data, it is possible to avoid redundant training of the artificial intelligence model and improve training efficiency.

[0060] The present application provides systems, methods, devices, etc. that can implement duplicate record detection. Before introducing the systems, methods, and devices for duplicate record detection provided by the present application, the record pairs (such as the first record pair and the second record pair described below), duplicate record detection models (such as the first duplicate record detection model and the second duplicate record detection model described below), and prediction information of duplicate record detection models (such as the first prediction information corresponding to the first duplicate record detection model and the second prediction information corresponding to the second duplicate record detection model described below) involved in the embodiments of the present application are first introduced.

[0061] (1) A record pair includes two records. For example, in a data table, any two rows of data can be used as a record pair. For example, in different data tables (Table 1 and Table 2), any row of data in Table 1 and any row of data in Table 2 can be used as a record pair. For example, in different documents (Document A and Document B), any continuous segment of characters in Document A (assuming 20 continuous characters) and any continuous segment of characters in Document B (20 continuous characters) can be used as a record pair. For example, in a black / white list detection scenario, if the black / white list contains information of multiple users, the user information to be detected and each user information recorded in the black / white list can be used as a record pair.

[0062] (2) Duplicate record detection model refers to a model used to detect whether two records in a record pair are duplicates. It can give a prediction category, such as duplicate or non-duplicate, such as duplicate, non-duplicate or uncertain, and can also give a prediction probability corresponding to each prediction category.

[0063] (3) The prediction information of the duplicate record detection model refers to the probability distribution information that can reflect the duplicate record detection model's prediction of whether two records in a record pair are duplicate and / or non-duplicate. The form in which the probability is expressed depends on the specific model.

[0064] For example, if the duplicate record detection model is a rule model, the model will give a prediction category, and give the proportion of rules in the model rule base that predict the two records in the record pair to be duplicated, and / or give the proportion of rules that predict the two records in the record pair to be non-duplicates. Here, the proportion is the prediction probability corresponding to the prediction category. Taking the probability of the model predicting the two records in the record pair to be duplicated as an example, it can be understood that a larger proportion (i.e., probability) indicates that the model is relatively certain that the two records in the record pair are duplicated, a medium proportion indicates that the model is not quite sure whether the two records in the record pair are duplicated, and a smaller proportion indicates that the model is relatively uncertain that the two records in the record pair are duplicated. In other words, it indicates that the model is relatively certain that the two records in the record pair are not duplicated. Similarly, taking the probability of the model predicting the two records in the record pair to be non-duplicates as an example, it can be understood that a larger proportion indicates that the model is relatively certain that the two records in the record pair are not duplicated, a medium proportion indicates that the model is not quite sure whether the two records in the record pair are duplicated, and a smaller proportion indicates that the model is relatively uncertain that the two records in the record pair are not duplicated. In other words, it indicates that the model is relatively certain that the two records in the record pair are duplicated.

[0065] For example 2, if the duplicate record detection model is a support vector machine, the model will give a predicted category and the distance between the feature vector corresponding to the record pair and the classification hyperplane in the support vector machine. Here, the distance is the predicted probability corresponding to the predicted category. Taking the probability that the model predicts that two records in a record pair are duplicated as an example, it can be understood that the larger the distance (i.e., probability), the more certain the model is that the two records in the record pair are duplicated, and the smaller the distance, the less certain the model is that the two records in the record pair are duplicated. In other words, it means that the model is less certain whether the two records in the record pair are duplicated. Similarly, taking the probability that the model predicts that two records in a record pair are not duplicated as an example, it can be understood that the larger the distance, the more certain the model is that the two records in the record pair are not duplicated, and the smaller the distance, the less certain the model is that the two records in the record pair are not duplicated. In other words, it means that the model is less certain whether the two records in the record pair are duplicated.

[0066] For example three, if the duplicate record detection model is a random forest, the model will give a predicted category, and give the proportion of decision trees in the total number of decision trees in the decision trees included in the model that predict that the two records in the record pair are repeated, and / or give the proportion of decision trees in the total number of decision trees in the decision trees included in the model that predict that the two records in the record pair are not repeated. Here, the proportion is the predicted probability corresponding to the predicted category. Taking the probability of the model predicting that the two records in the record pair are repeated as an example, it can be understood that a larger proportion (i.e., probability) indicates that the model is relatively certain that the two records in the record pair are repeated, a medium proportion indicates that the model is not sure whether the two records in the record pair are repeated, and a smaller proportion indicates that the model is relatively uncertain that the two records in the record pair are repeated. In other words, it indicates that the model is relatively certain that the two records in the record pair are not repeated. Similarly, taking the probability of the model predicting that the two records in the record pair are not repeated as an example, it can be understood that a larger proportion indicates that the model is relatively certain that the two records in the record pair are not repeated, a medium proportion indicates that the model is not sure whether the two records in the record pair are repeated, and a smaller proportion indicates that the model is relatively uncertain that the two records in the record pair are not repeated. In other words, it indicates that the model is relatively certain that the two records in the record pair are repeated.

[0067] Based on the above description of the prediction information of the duplicate record detection model, it can be seen that whether the model determines whether two records in a record pair are duplicates can be determined based on the prediction information of the duplicate record detection model.

[0068] Taking the duplicate record detection model as a rule model / random forest as an example, when the probability (here is the ratio) that the two records in a record pair are predicted to be repeated / non-repeated by the duplicate record detection model included in the prediction information is large enough (such as greater than a certain threshold) or small enough (such as less than a certain threshold), the judgment model determines whether the two records in the record pair are repeated / non-repeated; when the probability that the duplicate record detection model included in the prediction information predicts that the two records in the record pair are repeated / non-repeated is medium, the judgment model is not sure whether the two records in the record pair are repeated.

[0069] Taking the duplicate record detection model as a support vector machine as an example, when the probability (here is the distance) that the two records in the record pair are repeated / non-repeated is predicted by the duplicate record detection model included in the prediction information is large enough (such as greater than a certain distance threshold), the judgment model determines that the two records in the record pair are repeated / non-repeated; when the probability that the two records in the record pair are repeated / non-repeated is not large enough (such as less than or equal to the aforementioned distance threshold), the judgment model is not sure whether the two records in the record pair are repeated.

[0070] Next, the duplicate record detection system, method and device provided by the present application are described in detail with reference to the corresponding drawings.

[0071] See also Figure 1 , Figure 1 is a schematic diagram of the architecture of a duplicate record detection system provided in an embodiment of the present application, such as Figure 1 As shown, the architecture includes a client 100 and a duplicate record detection system 200, and the duplicate record detection system 200 is deployed with a first duplicate record detection model and a second duplicate record detection model. There is a communication connection between the client 100 and the duplicate record detection system 200, which can be a wired connection or a wireless connection, which is not specifically limited in this application. The number of clients 100 that establish a communication connection with the duplicate record detection system 200 can be one or more, which is not specifically limited in this application.

[0072] The client 100 is used to realize human-computer interaction and can be deployed on terminal devices, including personal computers (PCs), smart phones, wearable devices, handheld processing devices, tablet computers, mobile notebooks, augmented reality (AR) devices, virtual reality (VR) devices, integrated handheld devices, wearable devices, vehicle-mounted devices, smart conference devices, smart advertising devices, smart home appliances, etc. Smart home appliances can be sweeping robots, mopping robots, etc., which are not specifically limited here.

[0073] In a specific implementation, the client 100 may be software or an application running on a terminal device or a computing device controlled by a user, such as a personal computer client, or a World Wide Web client accessed through a browser, or an application (APP) client running on a mobile terminal, or a console of a cloud platform, which is not specifically limited in this application.

[0074] Optionally, the client 100 may be a separate client specifically used to implement duplicate record detection, such as a duplicate record detection tool, a duplicate record detection application, etc. Alternatively, the client 100 may also be a duplicate record detection function module or plug-in in comprehensive software, such as a duplicate record detection module in data mining and analysis software commonly used by enterprises, etc., which is not specifically limited in this application.

[0075] Optionally, the client 100 may also be a client of the cloud platform, such as a console of the cloud platform, which may be a web-based console or an application programming interface (API)-based console, which is not specifically limited in this application. The console may provide users with duplicate record detection cloud services, and users may obtain the right to use the duplicate record detection system 200 provided in this application by purchasing cloud services.

[0076] The duplicate record detection system 200 can be deployed on a computing device or a computing device cluster, wherein the computing device includes a bare metal server (BMS), a virtual machine (VM), a container (container) or an edge computing device. BMS refers to a general physical server, such as an ARM server or an X86 server; a virtual machine refers to a complete computer system with complete hardware system functions and running in a completely isolated environment simulated by software. All work that can be done in a physical computer can be done in a virtual machine. When creating a virtual machine in a computing device, part of the hard disk and memory capacity of the physical machine needs to be used as the hard disk and memory capacity of the virtual machine. Each virtual machine has an independent basic input / output system (BIOS), hard disk and operating system, and can operate the virtual machine like a physical machine; a container is a portable software unit that can merge an application and all its dependencies into a software package that is not limited by the underlying host operating system, so there is no need to build a complex environment, simplifying the process from application development to deployment; an edge computing device refers to a device that is closer to the data source and end users and has low latency and high bandwidth characteristics, such as smart routing, edge servers, etc. The computing device cluster may include multiple computing devices as described above, such as a data center, which is not specifically limited in this application.

[0077] Optionally, the duplicate record detection system 200 and the client 100 can be deployed in the same or different computing device clusters, for example, the client 100 is deployed on a computing device in a first computing device cluster, and the duplicate record detection system 200 is deployed on a computing device in a second computing device cluster; or, the client 100 and the duplicate record detection system 200 are deployed in the same computing device cluster. It should be understood that the above examples are for illustration only and are not specifically limited in this application.

[0078] The accuracy of the above-mentioned first duplicate record detection model is lower than that of the second duplicate record detection model. It can be understood that, under normal circumstances, since the cost of a low-precision model is lower than the cost of a high-precision model, including training cost, inference cost, etc., the cost of the first duplicate record detection model is lower than the cost of the second duplicate record detection model.

[0079] In a specific implementation, the same test data set may be used to test the accuracy of two duplicate record detection models, and the model with lower accuracy among the two duplicate record detection models may be determined as the first duplicate record detection model, and the model with higher accuracy may be determined as the second duplicate record detection model.

[0080] Since the accuracy and cost of rule models are usually lower than those of artificial intelligence models, in a possible embodiment, the first duplicate record detection model is a model that integrates multiple duplicate record detection rules, and the second duplicate record detection model is an artificial intelligence model, which can be a traditional machine learning model, such as a support vector machine, decision tree, random forest, etc., or a deep learning model, such as Word2vec, GPT, BERT, etc.

[0081] Since the accuracy and cost of traditional machine learning models are generally lower than those of deep learning models, in one possible embodiment, the first duplicate record detection model is a traditional machine learning model, and the second duplicate record detection model is a deep learning model.

[0082] Since the accuracy and cost of a rule model with fewer integrated rules are usually lower than those of a rule model with more integrated rules, in a possible embodiment, the first duplicate record detection model is a model that integrates m duplicate record detection rules, and the second duplicate record detection model is a model that integrates n duplicate record detection rules, where m and n are both positive numbers, and m is less than n.

[0083] Optionally, the first duplicate record detection model and the second duplicate record detection model can both be traditional machine learning models / deep learning models. For example, the first duplicate record detection model is a support vector machine, and the second duplicate record detection model is a random forest. For another example, the first duplicate record detection model is Word2vec, and the second duplicate record detection model is GPT.

[0084] Optionally, the first duplicate record detection model and the second duplicate record detection model may be the same type of artificial intelligence models, such as both are random forests, or both are GPTs. This application does not specifically limit the first duplicate record detection model and the second duplicate record detection model.

[0085] In the present application, the first duplicate record detection model and the second duplicate record detection model in the duplicate record detection system 200 can be deployed in the same computing device, or in different computing devices in the same computing device cluster, or in different computing device clusters, which is not specifically limited in the present application. The following will be described in conjunction with the accompanying drawings and embodiments.

[0086] In the first method, the duplicate record detection system 200 is deployed on a single computing device as an example. Figure 2 The schematic diagram of the architecture of another duplicate record detection system 200 provided in the embodiment of the present application is shown in FIG. Figure 2 In the example, the duplicate record detection system 200 is deployed on the first computing device 10 .

[0087] exist Figure 2 In the illustrated architecture, the process of duplicate record detection by the duplicate record detection system 200 may be as follows: the first computing device 10 may receive a data source (such as different data tables, different documents) input by a user through the client 100, and then obtain at least one record pair from the user's data source, and input the at least one record pair into the first duplicate record detection model in sequence for detection, and obtain prediction information of the first duplicate record detection model, the prediction information including the probability that the first duplicate record detection model predicts that two records in a record pair are duplicated and / or non-duplicate, and then judges whether it can be determined whether the two records in the record pair are duplicates based on the prediction information.

[0088] In the case where it can be determined whether two records in a record pair are repeated based on the prediction information, a determination result is output, which is whether the two records in the record pair are repeated or not. In the case where it cannot be determined whether the two records in the record pair are repeated based on the prediction information, the record pair is input into a second duplicate record detection model for detection to obtain prediction information of the second duplicate record detection model, the prediction information including the probability that the second duplicate record detection model predicts that the two records in the record pair are repeated and / or not repeated. Thereafter, based on the prediction information, it is determined whether the two records in the record pair can be determined whether they are repeated. In the case where it is determined that it can be determined whether the two records in the record pair are repeated based on the prediction information, a determination result is output.

[0089] After obtaining the result of whether two records in at least one record pair are duplicated, the first computing device 10 may send the detection result to the client 100 , which is presented to the user.

[0090] It should be understood that Figure 2 The duplicate record detection system 200 shown is only an example and is not specifically limited in this application. Figure 2 In the illustrated architecture, a third duplicate record detection model with higher accuracy than the second duplicate record detection model may also be deployed on the first computing device 10 ( Figure 2 (not shown), when the first computing device 10 cannot determine whether the two records in the record pair are duplicates based on the prediction information of the second duplicate record detection model, the record pair is input into the third duplicate record detection model for detection to obtain the prediction information of the third duplicate record detection model, the prediction information including the probability that the third duplicate record detection model predicts that the two records in the record pair are duplicated and / or non-duplicate, then based on the prediction information, it is determined whether it is possible to determine whether the two records in the record pair are duplicates, and when it is determined that it is possible to determine whether the two records in the record pair are duplicates based on the prediction information, a determination result is output.

[0091] For example, Figure 2The duplicate record detection system 200 shown includes, in addition to the third duplicate record detection model mentioned above, a fourth duplicate record detection model with higher accuracy than the third duplicate record detection model. When the first computing device 10 cannot determine whether two records in a record pair are duplicates based on the prediction information of the third duplicate record detection model, the record pair is input into the fourth duplicate record detection model for detection to obtain the prediction information of the fourth duplicate record detection model, which includes the probability that the fourth duplicate record detection model predicts that the two records in the record pair are duplicated and / or non-duplicate. Then, based on the prediction information, it is determined whether it is possible to determine whether the two records in the record pair are duplicated. When it is determined that it is possible to determine whether the two records in the record pair are duplicated based on the prediction information, a determination result is output.

[0092] In a specific implementation, the same test data set can be used to test the accuracy of multiple duplicate record detection models, and the model with the lowest accuracy among the multiple duplicate record detection models is determined as the first duplicate record detection model, the model with the second lowest accuracy is determined as the second duplicate record detection model, and so on.

[0093] In the second method, the duplicate record detection system 200 is deployed on two computing devices as an example. Figure 3 The structural diagram of another duplicate record detection system 200 provided in the embodiment of the present application is shown in FIG. Figure 3 In the duplicate record detection system 200 , the first duplicate record detection model is deployed on the first computing device 10 , and the second duplicate record detection model is deployed on the second computing device 20 .

[0094] exist Figure 3 In the illustrated architecture, the process of duplicate record detection by the duplicate record detection system 200 may be as follows: the first computing device 10 may receive a data source (such as different data tables, different documents) input by a user, and then obtain at least one record pair from the user's data source, and input the at least one record pair into the first duplicate record detection model in sequence for detection, and obtain prediction information of the first duplicate record detection model, the prediction information including the probability that the first duplicate record detection model predicts that two records in a record pair are duplicated and / or non-duplicate, and then judges whether it can be determined whether the two records in the record pair are duplicates based on the prediction information.

[0095] In the case where it can be determined whether two records in a record pair are duplicates based on the prediction information, a determination result is output, which is the two records in the record pair being duplicated or not. In the case where it cannot be determined whether two records in a record pair are duplicates based on the prediction information, the record pair is input into a second duplicate record detection model for detection to obtain prediction information of the second duplicate record detection model, the prediction information including the probability of the second duplicate record detection model predicting that the two records in the record pair are duplicated and / or not. Thereafter, it is determined whether it can be determined whether the two records in the record pair are duplicated based on the prediction information, and in the case where it is determined whether the two records in the record pair can be determined whether they are duplicated based on the prediction information, a determination result is output.

[0096] After obtaining the result of whether two records in at least one record pair are duplicated, the second computing device 20 may send the detection result to the client 100 , which is presented to the user by the client 100 .

[0097] It should be understood that Figure 3 The duplicate record detection system 200 shown is only an example and is not specifically limited in this application. Figure 3 The duplicate record detection system 200 shown in the figure further includes a third duplicate record detection model ( Figure 3 The third duplicate record detection model is deployed on the first computing device 10 or the second computing device 20, or is deployed on a third computing device 30 different from the first computing device 10 and the second computing device 20 (not shown). Figure 3 (not shown), taking the deployment on the third computing device 30 as an example, when the second computing device 20 cannot determine whether the two records in the record pair are duplicates based on the prediction information of the second duplicate record detection model, the record pair is sent to the third computing device 30, and the third computing device 30 inputs the record pair into the third duplicate record detection model for detection, and obtains the prediction information of the third duplicate record detection model, which includes the probability that the third duplicate record detection model predicts that the two records in the record pair are duplicated and / or non-duplicate, and then determines whether to determine whether the two records in the record pair are duplicated based on the prediction information, and outputs the determination result when it is determined that whether the two records in the record pair are duplicated based on the prediction information.

[0098] For example, Figure 3 In the duplicate record detection system 200 shown in the figure, in addition to the third duplicate record detection model, a fourth duplicate record detection model ( Figure 3 (not shown), the fourth duplicate record detection model is deployed on the first computing device 10, the second computing device 20 or the third computing device 30 ( Figure 3), or deployed on a fourth computing device 40 (not shown) which is different from the first computing device 10, the second computing device 20, and the third computing device 30. Figure 3 (not shown), taking the deployment on the third computing device 30 as an example, when the third computing device 30 cannot determine whether the two records in the record pair are duplicates based on the prediction information of the third duplicate record detection model, the record pair is input into the fourth duplicate record detection model for detection to obtain the prediction information of the fourth duplicate record detection model, the prediction information including the probability that the fourth duplicate record detection model predicts that the two records in the record pair are duplicated and / or non-duplicate, and then determines whether it is possible to determine whether the two records in the record pair are duplicated based on the prediction information, and when it is determined that it is possible to determine whether the two records in the record pair are duplicated based on the prediction information, the determination result is output.

[0099] In a specific implementation, the same test data set can be used to test the accuracy of multiple duplicate record detection models, and the model with the lowest accuracy among the multiple duplicate record detection models is determined as the first duplicate record detection model, the model with the second lowest accuracy is determined as the second duplicate record detection model, and so on.

[0100] It can be seen that Figure 1 , Figure 2 , Figure 3 The duplicate record detection system 200 shown can implement duplicate detection of record pairs through two or more duplicate record detection models.

[0101] In addition, since the duplicate record detection system 200 first uses a low-precision (low-cost) model for detection, when it cannot determine whether two records in a record pair are duplicates based on the prediction information of the low-precision model, it then uses a high-precision (high-cost) model for detection. Compared with the current solution of only using a low-precision model for duplicate record detection, the detection accuracy can be improved.

[0102] Moreover, under normal circumstances, a low-precision model will output a large number of record pairs that cannot be determined whether they are repeated. In the current solutions that only use low-precision models for detection, most of the record pairs output by the low-precision model that cannot be determined whether they are repeated are manually detected, which is time-consuming and labor-intensive and will consume a lot of costs. The present application uses a high-precision model to detect record pairs output by the low-precision model that cannot be determined whether they are repeated, thereby reducing costs and improving detection efficiency.

[0103] Furthermore, compared to the current solution of only using high-precision models for duplicate record detection, since when the low-precision model can already identify whether two records are duplicates, there is no need to use the high-precision model for detection, which can reduce the workload of the high-precision model. It can be understood that, usually, since the high-precision model is more complex than the low-precision model, the high-precision model will occupy more computing resources than the low-precision model each time it works, and the high-precision model will also take longer to detect each time than the low-precision model. Therefore, the use of the above solution can improve detection efficiency, reduce the consumption of computing resources, and further reduce costs.

[0104] It should be understood that the above-mentioned duplicate record detection system 200 includes the first duplicate record detection model, the second duplicate record detection model, the third duplicate record detection model, and the fourth duplicate record model only as examples. In specific implementation, more duplicate record detection models may be included, and this application does not make specific limitations. In other words, in the specific implementation process, the model selection and combination in the duplicate record detection system 200 can be flexibly configured. However, in order to reduce the cost of the overall task, the duplicate record detection system 200 needs to follow the principle of low-precision (low-cost) models first and high-precision (high-cost) models later.

[0105] In order to more clearly understand the specific process of duplicate record detection by the duplicate record detection system 200, Figure 4 The flowchart shown in the figure is described in more detail. Figure 4 In Figure 3 The duplicate record detection system 200 shown is used as an example for description. Figure 2 The duplicate record detection system 200 shown in the figure is a specific process for executing a duplicate record detection by an execution subject. Figure 4 The process is similar to that shown in Figure 4 For the sake of brevity of the specification, the related description will not be elaborated any more.

[0106] like Figure 4 As shown, the method comprises the following steps:

[0107] S401: The first computing device 10 obtains a data source.

[0108] The data source may be data in any form, such as data in the form of tables, documents, or web pages, and this application does not make any specific limitations.

[0109] Specifically, if the duplicate record detection scenario is a data deduplication scenario / data duplication rate assessment scenario, the data source may be uploaded by the user to the first computing device 10, or created by the user on the first computing device 10, and this application does not make any specific limitations on this.

[0110] If the duplicate record detection scenario is a data detection scenario based on data on a black / white list, the data source may be a black / white list and information to be inspected, wherein the black / white list may be uploaded by a user to the first computing device 10, or may be created by a user on the first computing device 10; if the information to be inspected is generated or obtained by other devices, the first computing device 10 may receive the information to be inspected sent by other devices; optionally, the information to be inspected may also be information generated or obtained by the first computing device 10 itself, such as business information running on the first computing device 10, operation status information of components of the first computing device 10 itself, visitor information received by the first computing device 10, etc. It can be understood that the type of information to be inspected depends on the specific detection scenario.

[0111] In a possible embodiment, Figure 5 As shown in S501, the first computing device 10 acquires the data source in the following manner: the first computing device 10 receives a duplicate record detection request carrying the data source sent by the client 100, and acquires the data source. It can be seen that Figure 5 S502 to S510 shown are similar to Figure 4 S402 to S410 shown correspond to each other, please refer to the relevant description of S402 to S410 below, and for the sake of brevity of the specification, they will not be elaborated again.

[0112] S402: The first computing device 10 obtains a first record pair from a data source.

[0113] The first record pair can be any record pair in the data source. Taking the data source as a single table as an example, the first record pair can include any two rows of data in the single table. Taking the data source as two tables as an example, the first record pair can include any row of data in one of the tables and any row of data in the other table. Taking the data source as two documents as an example, the first record pair can include any continuous segment of characters in one of the documents and any continuous segment of characters in the other document, usually with the same number of characters. For example, in a black / white list detection scenario, the information to be inspected and each information recorded in the black / white list are taken as a record pair.

[0114] In the case where the data source includes multiple record pairs, the first computing device 10 may take each record pair in the data source as a first record pair, and then perform Figure 4 The method shown can detect duplicate records for each record pair in the data source.

[0115] In a possible embodiment, the first computing device 10 may divide the data source into multiple blocks by using a blocking technology, where each block contains a group of related records. Then, for each block, a heuristic method may be used within the block to remove record pairs that are unlikely to be repeated, and then duplicate record detection may be performed on the remaining record pairs that may be repeated within the block without matching in the entire data source, thereby reducing the number of record pairs that need to be detected for duplicate records and improving duplicate record detection efficiency.

[0116] Block segmentation techniques include but are not limited to the following:

[0117] (1) Rule-based chunking refers to assigning records in a data source to different chunks based on predefined rules. For example, chunking can be based on a certain attribute value range, first letter, etc. of the record.

[0118] (2) Index-based blocking refers to using the index structure to allocate records in the data source into different blocks. For example, records can be blocked using hash indexes, B+ tree indexes, etc.

[0119] (3) Clustering-based chunking refers to using a clustering algorithm to cluster similar records in a data source into the same chunk. For example, you can use the K-means clustering algorithm (K-means clustering algorithm), density-based spatial clustering of applications with noise (DBSCAN), etc. for chunking.

[0120] (4) Sample-based chunking refers to randomly selecting a portion of records from the data source as samples, and then allocating other records to different chunks based on the characteristics of the samples. For example, the K-nearest neighbor (KNN) algorithm can be used to perform chunking based on the characteristics of the samples.

[0121] S403: The first computing device 10 inputs the first record pair into the first duplicate record detection model to obtain first prediction information, where the first prediction information includes the probability that the first duplicate record detection model predicts that the two records in the first record pair are duplicated and / or non-duplicate.

[0122] For the specific content of the first prediction information, please refer to the above description of the prediction information of the duplicate record detection model. For the sake of brevity of the specification, it will not be elaborated here.

[0123] Among them, the first duplicate record detection model can be expressed as:

[0124] y1=f1(x1)

[0125] Among them, y1 is the prediction information of the first duplicate record detection model, x1 is the record pair, and f1() is the mapping relationship between the record pair and the prediction information of the first duplicate record detection model.

[0126] The second duplicate record detection model can be expressed as:

[0127] y2=f2(x2)

[0128] Among them, y2 is the prediction information of the second duplicate record detection model, x2 is the record pair, and f2() is the mapping relationship between the record pair and the prediction information of the second duplicate record detection model.

[0129] The accuracy of the first duplicate detection model is lower than the accuracy of the second duplicate record detection model described below.

[0130] In a possible embodiment, the first duplicate record detection model is a model that integrates multiple duplicate record detection rules, and the second duplicate record detection model is an artificial intelligence model, which can be a traditional machine learning model, such as a support vector machine, a decision tree, a random forest, etc., or a deep learning model, such as Word2vec, GPT, BERT, etc.

[0131] Optionally, the first duplicate record detection model is a traditional machine learning model, and the second duplicate record detection model is a deep learning model. Optionally, the first duplicate record detection model and the second duplicate record detection model may also both be traditional machine learning models / deep learning models. Optionally, the first duplicate record detection model and the second duplicate record detection model may also be the same or different artificial intelligence models, and this application does not specifically limit the first duplicate record detection model and the second duplicate record detection model.

[0132] Regarding the first duplicate record detection model and the second duplicate record detection model, the related description of the duplicate record detection system 200 has been described in detail above, and will not be elaborated here.

[0133] S404: The first computing device 10 determines whether it can be determined whether the two records in the first record pair are repeated based on the first prediction information. Specifically, when it can be determined whether the two records in the first record pair are repeated based on the first prediction information, S405 is executed; when it cannot be determined whether the two records in the first record pair are repeated based on the first prediction information, S406 is executed.

[0134] The first computing device 10 may predict the probability of duplication and / or non-duplication of the two records in the first record pair based on the first duplicate record detection model included in the first prediction information, and determine whether it is possible to determine whether the two records in the first record pair are duplicates. For ease of distinction and description, the probability that the first duplicate record detection model predicts that the two records in the first record pair are duplicated is referred to as the first probability 11, and the probability that the first duplicate record detection model predicts that the two records in the first record pair are non-duplicates is referred to as the first probability 12.

[0135] The following takes the first duplicate record detection model as a rule model as an example to illustrate the process of the first computing device 10 determining whether the two records in the first record pair can be determined to be duplicated based on the first prediction information. As can be seen from the above, the first probability 11 is the proportion of the rules in the first duplicate record detection model rule base that predict the two records in the first record pair to be duplicated to the total rules, and the first probability 12 is the proportion of the rules in the first duplicate record detection model rule base that predict the two records in the first record pair to be non-duplicate to the total rules.

[0136] In a first implementation manner, the first prediction information includes a first probability 11.

[0137] The first computing device 10 may determine that the two records in the first record pair are duplicates when it is determined that the first probability 11 is greater than the first threshold value, determine that the two records in the first record pair are not duplicates when it is determined that the first probability 11 is less than the second threshold value, and determine that it cannot be determined whether the two records in the first record pair are duplicates when it is determined that the first probability 11 is less than or equal to the first threshold value and greater than or equal to the second threshold value.

[0138] In a second implementation manner, the first prediction information includes a first probability 12.

[0139] The first computing device 10 may determine that the two records in the first record pair are not repeated when it is determined that the first probability 12 is greater than the first threshold, determine that the two records in the first record pair are repeated when it is determined that the first probability 12 is less than the second threshold, and determine that it cannot be determined whether the two records in the first record pair are repeated when it is determined that the first probability 12 is less than or equal to the first threshold and greater than or equal to the second threshold.

[0140] In a third implementation manner, the first prediction information includes a first probability 11 and a first probability 12.

[0141] The first computing device 10 may determine that the two records in the first record pair are duplicates when it is determined that the first probability 11 is greater than the first threshold, determine that the two records in the first record pair are not duplicates when it is determined that the first probability 12 is greater than the first threshold, and determine that it cannot be determined whether the two records in the first record pair are duplicates when it is determined that the first probability 11 / the first probability 12 is less than or equal to the first threshold and greater than or equal to the second threshold.

[0142] The first threshold and the second threshold can be customized based on the actual situation of the first duplicate record detection model, and are not specifically limited in this application. For example, the first threshold is set to 0.6, and the second threshold is set to 0.4.

[0143] Taking the first duplicate record detection model as a support vector machine as an example, the process of the first computing device 10 judging whether it is possible to determine whether the two records in the first record pair are duplicates based on the first prediction information is described. As can be seen from the above, the first probability 11 is the distance between the feature vector corresponding to the first record pair and the classification hyperplane of the support vector machine when the feature vector corresponding to the first record pair falls on both sides of the classification hyperplane of the support vector machine to represent the "duplicate" category, and the first probability 12 is the distance between the feature vector corresponding to the first record pair and the classification hyperplane of the support vector machine when the feature vector corresponding to the first record pair falls on both sides of the classification hyperplane of the support vector machine to represent the "non-duplicate" category.

[0144] The first computing device 10 may determine that the two records in the first record pair are repeated when it is determined that the first probability 11 is greater than the first distance threshold, determine that it cannot be determined whether the two records in the first record pair are repeated when it is determined that the first probability 11 is less than or equal to the first distance threshold, determine that it can be determined that the two records in the first record pair are not repeated when it is determined that the first probability 12 is greater than the first distance threshold, and determine that it cannot be determined whether the two records in the first record pair are repeated when it is determined that the first probability 12 is less than or equal to the first distance threshold.

[0145] The first distance threshold may be customized based on the actual situation of the first duplicate record detection model, and is not specifically limited in this application.

[0146] It should be understood that when the types of the first duplicate record detection models are different, the first prediction information is different. Therefore, the present application does not specifically limit the process of determining whether it is possible to determine whether two records in the first record pair are duplicates based on the first prediction information.

[0147] S405: The first computing device 10 outputs a first determination result, where the first determination result is whether the two records in the first record pair are repeated or not.

[0148] S406 : The first computing device 10 sends the first record pair to the second computing device 20 , and correspondingly, the second computing device 20 receives the first record pair sent by the first computing device 10 .

[0149] S407: The second computing device 20 inputs the first record pair into the second duplicate record detection model to obtain second prediction information, where the second duplicate record detection model predicts the probability that the two records in the first record pair are duplicated and / or non-duplicate.

[0150] For the specific content of the second prediction information, please refer to the above description of the prediction information of the duplicate record detection model. For the sake of brevity of the specification, it will not be elaborated here.

[0151] S408: The second computing device 20 determines whether it can be determined whether the two records in the first record pair are repeated based on the second prediction information. Specifically, when it can be determined whether the two records in the first record pair are repeated based on the second prediction information, S409 is executed; when it cannot be determined whether the two records in the first record pair are repeated based on the second prediction information, S410 is executed.

[0152] The second computing device 20 may predict the probability of duplication and / or non-duplication of the two records in the first record pair based on the second duplicate record detection model included in the second prediction information, and determine whether it is possible to determine whether the two records in the first record pair are duplicates. For ease of distinction and description, the probability that the second duplicate record detection model predicts that the two records in the first record pair are duplicated is referred to as the second probability 21, and the probability that the second duplicate record detection model predicts that the two records in the first record pair are non-duplicates is referred to as the second probability 22.

[0153] The following takes the second duplicate record detection model as a random forest as an example to illustrate the process of the second computing device 20 judging whether it is possible to determine whether the two records in the second record pair are duplicated based on the second prediction information. As can be seen from the above, the second probability 11 here is the proportion of the decision trees included in the second duplicate record detection model that predict the two records in the second record pair to be duplicated, accounting for the total number of decision trees, and the first probability 12 is the proportion of the decision trees included in the second duplicate record detection model that predict the two records in the second record pair to be non-duplicate, accounting for the total number of decision trees.

[0154] In a first implementation manner, the second prediction information includes a second probability 21.

[0155] The second computing device 20 may determine that the two records in the first record pair are repeated when it is determined that the second probability 21 is greater than the third threshold, determine that the two records in the first record pair are not repeated when it is determined that the second probability 21 is less than the fourth threshold, and determine that it cannot be determined whether the two records in the first record pair are repeated when it is determined that the second probability 21 is less than or equal to the third threshold and greater than or equal to the fourth threshold.

[0156] In a second implementation manner, the second prediction information includes a second probability 22.

[0157] The second computing device 20 may determine that the two records in the first record pair are not repeated when it is determined that the second probability 22 is greater than the third threshold, determine that the two records in the first record pair are repeated when it is determined that the second probability 22 is less than the fourth threshold, and determine that it cannot be determined whether the two records in the first record pair are repeated when it is determined that the second probability 22 is less than or equal to the third threshold and greater than or equal to the fourth threshold.

[0158] In a third implementation manner, the second prediction information includes a second probability 21 and a second probability 22.

[0159] The second computing device 20 may determine that the two records in the first record pair are duplicates when it is determined that the second probability 21 is greater than the third threshold, determine that the two records in the first record pair are not duplicates when it is determined that the second probability 22 is greater than the third threshold, and determine that it cannot be determined whether the two records in the first record pair are duplicates when it is determined that the second probability 21 / the second probability 22 is less than or equal to the third threshold and greater than or equal to the fourth threshold.

[0160] The third threshold and the fourth threshold can be customized based on the actual situation of the second duplicate record detection model, and are not specifically limited in this application. For example, the third threshold is set to 0.7, and the fourth threshold is set to 0.3.

[0161] Taking the second duplicate record detection model as a support vector machine as an example, the second computing device 20 determines whether the two records in the first record pair are duplicated based on the second prediction information. As can be seen from the above, the second probability 21 is the distance from the classification hyperplane of the support vector machine when the feature vector corresponding to the first record pair falls on both sides of the classification hyperplane of the support vector machine to represent the "duplicate" category, and the second probability 22 is the distance from the classification hyperplane of the support vector machine when the feature vector corresponding to the first record pair falls on both sides of the classification hyperplane of the support vector machine to represent the "non-duplicate" category.

[0162] The second computing device 20 may determine that the two records in the first record pair are repeated when it is determined that the second probability 21 is greater than the second distance threshold, determine that it cannot be determined whether the two records in the first record pair are repeated when it is determined that the second probability 21 is less than or equal to the second distance threshold, determine that it can be determined that the two records in the first record pair are not repeated when it is determined that the second probability 22 is greater than the second distance threshold, and determine that it cannot be determined whether the two records in the first record pair are repeated when it is determined that the second probability 22 is less than or equal to the second distance threshold.

[0163] The second distance threshold may be customized based on the actual situation of the second duplicate record detection model, and is not specifically limited in this application.

[0164] It should be understood that when the types of the second duplicate record detection models are different, the second prediction information is different. Therefore, the present application does not specifically limit the process of determining whether it is possible to determine whether two records in the first record pair are duplicates based on the second prediction information.

[0165] S409: The second computing device 20 outputs the first determination result.

[0166] S410: The second computing device 20 provides the first record pair to a marking person, and the marking person determines whether two records in the first record pair are repeated.

[0167] In one possible embodiment, the second computing device 20 provides the first record pair to a labeler when the duplicate record detection system 200 does not include a third duplicate record detection model with higher accuracy than the second duplicate record detection model, and the labeler determines whether two records in the first record pair are duplicates.

[0168] In another possible embodiment, the above-mentioned duplicate record detection system 200 includes a third duplicate record detection model with higher accuracy than the second duplicate record detection model. In this case, S410 can be: the second computing device 20 inputs the first record pair into the third duplicate record detection model to obtain prediction information of the third duplicate record detection model, and the prediction information includes the probability that the third duplicate record detection model predicts that the two records in the first record pair are repeated and / or non-repeated, and then determines whether it can be determined whether the two records in the first record pair are repeated based on the prediction information.

[0169] Optionally, in addition to including a third duplicate record detection model with higher accuracy than the second duplicate record detection model, the above-mentioned duplicate record detection system 200 also includes a fourth duplicate record detection model with higher accuracy than the third duplicate record detection model. In this case, S410 can be: the second computing device 20 inputs the first record pair into the third duplicate record detection model to obtain prediction information of the third duplicate record detection model, the prediction information including the probability that the third duplicate record detection model predicts that the two records in the first record pair are repeated and / or non-repeated, and then determines whether it is possible to determine whether the two records in the first record pair are repeated based on the prediction information. In the case that it is possible to determine whether the two records in the first record pair are repeated based on the prediction information, a first determination result is output. In the case that it is not possible to determine whether the two records in the first record pair are repeated based on the prediction information, the first record pair is input into the fourth duplicate record detection model to obtain prediction information of the fourth duplicate record detection model, the prediction information including the probability that the fourth duplicate record detection model predicts that the two records in the first record pair are repeated and / or non-repeated, and then determines whether it is possible to determine whether the two records in the first record pair are repeated based on the prediction information.

[0170] The specific process of judging whether the two records in the first record pair can be determined to be repeated based on the prediction information of the third / fourth duplicate record detection model is similar to the specific process of judging whether the two records in the first record pair can be determined to be repeated based on the second prediction information. Please refer to the relevant description above and will not be elaborated for the sake of brevity.

[0171] Optionally, the third duplicate record detection model and the fourth duplicate record detection model may be a rule model, a traditional machine learning model, or a deep learning model, which is not specifically limited in this application. The third duplicate record detection model and the fourth duplicate record detection model may be deployed on the second computing device 20 or the first computing device 10, or deployed on a third computing device 30 independent of the second computing device 20 and the first computing device 10, or the third duplicate record detection model is deployed on the third computing device 30, and the fourth duplicate record detection model is deployed on the fourth computing device 40, which is not specifically limited in this application.

[0172] It can be known from the relevant description in S403 that the second duplicate record detection model can be a deep learning model. Since deep learning models usually have the problem of overconfidence, the overconfidence of deep learning models means that the model is too confident in its own prediction results when predicting, and ignores the uncertainty and error rate of the prediction results. Therefore, in a specific embodiment of the present application, when the second duplicate record detection model is a deep learning model, the second prediction information of the second duplicate record detection model can be calibrated to make the second prediction information more accurate and reliable, thereby improving the accuracy of subsequent judgment based on the second prediction information whether the two records in the first record pair are duplicated.

[0173] Similarly, when the above-mentioned first / third / fourth duplicate record detection model is a deep learning model, the prediction information of the first / third / fourth duplicate record detection model can also be calibrated to make the prediction information more accurate and reliable, thereby improving the accuracy of judging whether the two records in the first record pair are duplicates based on the prediction information.

[0174] In the specific implementation, the methods for calibrating the prediction information include but are not limited to the following: (1) Temperature scaling: By adjusting the temperature parameters of the model, the distribution of the prediction probability can be stretched or compressed, so that the prediction information of the model is more accurate. This method can be achieved by normalizing the probability output by the model; (2) Calibration regression: By training an additional calibration model, the prediction probability of the model is mapped to a more accurate probability distribution. This method can use some classic regression algorithms, such as smoothing average, Bayesian logistic regression, etc.; (3) Confidence estimation: By training an additional confidence estimation model, the accuracy of the model's prediction probability is estimated. This method can use some classic classifiers, such as support vector machines, random forests, etc.; (4) Histogram binning: The prediction probability is divided into multiple intervals, and then the average accuracy of each interval is calculated. The confidence of the model is calibrated by adjusting the prediction probability of each interval.

[0175] Since the training methods for artificial intelligence models include supervised training and unsupervised training, when the training method is supervised training, the training samples need to carry corresponding classification labels. If a large number of training samples are manually labeled, it will be very time-consuming and labor-intensive, and the cost is high. In order to further reduce costs and improve model training efficiency, when the second duplicate record detection model is a model obtained by supervised training of the artificial intelligence model, the classification labels corresponding to the training samples in the training sample set of the second duplicate record detection model can be obtained by detecting the training samples with the first duplicate record detection model. It can be understood that the training samples in the training sample set are a large number of record pairs.

[0176] The second record pair represents any unlabeled training sample in the training sample set of the second duplicate record detection model, such as Figure 6 As shown, the unlabeled second record pair can be input into the first duplicate record detection model to obtain third prediction information, which includes the probability that the first duplicate record detection model predicts that the two records in the second record pair are repeated and / or non-repeated. Then, based on the third prediction information, it is determined whether the two records in the second record pair are repeated, and the second determination result is used as the classification label of the second record pair and added to the training sample set of the second duplicate record detection model. The second determination result is whether the two records in the second record pair are repeated or not.

[0177] Similarly, when the third / fourth duplicate record detection model is a model obtained by supervised training of an artificial intelligence model, the classification labels corresponding to the training samples in the training sample set of the third / fourth duplicate record detection model can be obtained by detecting the training samples using the first duplicate record detection model, or by detecting the training samples using the second duplicate record detection model, or by detecting different training samples using a combination of the two models.

[0178] The following is an introduction to the duplicate record detection method provided in the embodiment of the present application with reference to a more detailed example.

[0179] See also Figure 7 , Figure 7 is a schematic diagram of a duplicate record detection pipeline provided in an embodiment of the present application, Figure 7In the figure, table1 and table2 represent the data sources for duplicate record detection, the low-precision model M1 represents the first duplicate record detection model, and the low-precision model M1 is a rule model, the medium-precision model M2 represents the second duplicate record detection model, and the medium-precision model M2 is a random forest, the high-precision model M3 represents the third duplicate record detection model, and the high-precision model M3 is GPT, the diverter 1 is used to divert the record pairs detected by the low-precision model M1, the M2 classifier is used to divert the record pairs detected by the medium-precision model M2, and the diverter 3 is used to divert the record pairs detected by the high-precision model M3. Uncertain record pairs (also called low-confidence record pairs) represent record pairs that the model is not sure whether they are duplicates, and high-confidence record pairs represent record pairs that the model is relatively sure whether they are duplicates.

[0180] Figure 7 The process from left to right is:

[0181] First, through the block technology, filter out the record pairs that need to be detected for duplicate records from table1 and table2.

[0182] Then, the record pair is input into the low-precision model M1 for detection to obtain first prediction information, which includes a first probability 11 that the two records in the record pair are repeated, and a first probability 12 that the two records in the record pair are not repeated, predicted by the low-precision model M1.

[0183] Afterwards, the diverter 1 diverts the record pair based on the first prediction information. Specifically, when the diverter 1 determines that the first probability 11 is greater than the first threshold, or when the diverter 1 determines that the first probability 12 is less than the second threshold, it determines that the confidence of the low-precision model M1 predicting that the two records in the record pair are repeated is high, and the record pair is output to the "high confidence" record pair set (also called the "repeated + non-repeated" record pair set). When the diverter 1 determines that the first probability 12 is greater than the first threshold, or when the diverter 1 determines that the first probability 11 is less than the second threshold, it determines that the confidence of the low-precision model M1 predicting that the two records in the record pair are not repeated is high, and the record pair is output to the "high confidence" record pair set. When the diverter 1 determines that the first probability 11 is less than or equal to the first threshold and greater than or equal to the second threshold, or when the diverter 1 determines that the first probability 12 is less than or equal to the first threshold and greater than or equal to the second threshold, it determines that the confidence of the low-precision model M1 predicting whether the two records in the record pair are repeated is low, that is, it is determined that the low-precision model M1 is not very sure whether the two records in the record pair are repeated, and then the record pair is input into the medium-precision model M2.

[0184] Afterwards, the medium precision model M2 detects the record pair to obtain second prediction information, which includes a second probability 21 predicted by the medium precision model M2 that the two records in the record pair are repeated, and a second probability 22 predicted by the medium precision model M2 that the two records in the record pair are not repeated.

[0185] Afterwards, the diverter 2 diverts the record pair based on the second prediction information. Specifically, when the diverter 2 determines that the second probability 21 is greater than the third threshold, or when the diverter 2 determines that the second probability 22 is less than the fourth threshold, it determines that the confidence of the medium-precision model M2 predicting that the two records in the record pair are repeated is high, and the record pair is output to the "high confidence" record pair set. When the diverter 2 determines that the second probability 22 is greater than the third threshold, or when the second probability 21 is less than the fourth threshold, it determines that the confidence of the medium-precision model M2 predicting that the two records in the record pair are not repeated is high, and the record pair is output to the "high confidence" record pair set. When the diverter 2 determines that the second probability 21 is less than or equal to the third threshold and greater than or equal to the fourth threshold, or when the second probability 22 is less than or equal to the third threshold and greater than or equal to the fourth threshold, it determines that the confidence of the medium-precision model M2 predicting whether the two records in the record pair are repeated is low, that is, it is determined that the medium-precision model M2 is not very sure whether the two records in the record pair are repeated, and then the record pair is input into the high-precision model M3.

[0186] Afterwards, the high-precision model M3 detects the record pair to obtain fourth prediction information, which includes a third probability 31 predicted by the high-precision model M3 that the two records in the record pair are repeated, and a third probability 32 predicted by the high-precision model M3 that the two records in the record pair are not repeated.

[0187] Afterwards, the diverter 3 diverts the record pair based on the fourth prediction information. Specifically, when the diverter 3 determines that the third probability 31 is greater than the fifth threshold, or when the diverter 3 determines that the third probability 32 is less than the sixth threshold, it determines that the confidence of the high-precision model M3 predicting that the two records in the record pair are repeated is high, and the record pair is output to the "high confidence" record pair set. When the diverter 3 determines that the third probability 32 is greater than the fifth threshold, or when the third probability 31 is less than the sixth threshold, it determines that the confidence of the high-precision model M3 predicting that the two records in the record pair are not repeated is high, and the record pair is output to the "high confidence" record pair set. When the diverter 3 determines that the third probability 31 is less than or equal to the fifth threshold and greater than or equal to the sixth threshold, or when the third probability 32 is less than or equal to the fifth threshold and greater than or equal to the sixth threshold, it determines that the confidence of the high-precision model M3 predicting whether the two records in the record pair are repeated is low, that is, it is determined that the high-precision model M3 is not very sure whether the two records in the record pair are repeated, and then the record pair is input into the manual inspection and annotation module, and the two records in the record pair are manually inspected for duplication.

[0188] The fifth threshold and the sixth threshold can be customized based on the actual situation of the third duplicate record detection model, and this application does not make any specific limitation. For example, the fifth threshold is set to 0.6, and the sixth threshold is set to 0.4.

[0189] Since the cost of a low-precision model is usually lower than that of a high-precision model, including training cost, inference cost, etc., the cost of the above-mentioned low-precision model M1 is lower than that of the medium-precision model M2, and the cost of the medium-precision model M2 is lower than that of the high-precision model M3.

[0190] It can be seen that through Figure 7 Example, duplicate record detection can be implemented.

[0191] In addition, due to Figure 7 An example is to first use a low-precision (low-cost) model for detection. When it is not certain whether two records in a record pair are duplicates based on the prediction information output by the low-precision model, a medium-precision (medium-cost) model is used for detection. When it is not certain whether two records in a record pair are duplicates based on the prediction information output by the medium-precision model, a high-precision (high-cost) model is used for detection. Compared with the current solution of only using low-precision / medium-precision models for detection, the detection accuracy can be improved.

[0192] Moreover, under normal circumstances, models with low / medium precision will output more record pairs that cannot be determined whether they are repeated. In the current solutions that only use low / medium precision models for detection, most of the record pairs output by low / medium precision models that cannot be determined whether they are repeated are manually detected, which is time-consuming, labor-intensive, and consumes a lot of costs. The present application uses a high-precision model to detect record pairs output by low / medium precision models that cannot be determined whether they are repeated, thereby reducing costs and improving detection efficiency.

[0193] Combining the above analysis, we can also see that Figure 7 Compared with the current solution of using a combination of low-precision models and medium-precision models for detection, the example can also improve detection accuracy and reduce costs.

[0194] Further, Figure 7Compared with the current solution of only using high-precision models for duplicate record detection, since the low-precision model / medium-precision model can already identify whether two records are duplicates, there is no need to use the high-precision model for detection. This can reduce the workload of the high-precision model. It can be understood that, generally, since the high-precision model is more complex than the low-precision model and the medium-precision model, the high-precision model will occupy more computing resources than the low-precision model / medium-precision model each time it works. Moreover, the detection time of the high-precision model each time will also be longer than the detection time of the low-precision model / medium-precision model. Therefore, the use of the above solution can improve detection efficiency, reduce the consumption of computing resources, and further reduce costs.

[0195] exist Figure 7 In the example, the overall process of the duplicate record detection method from the low-precision (low-cost) model to the high-precision (high-cost) model serial execution pipeline is mainly described. In the specific implementation process, the model selection and combination in the pipeline can be flexibly configured. For example, the pipeline can include multiple low-precision models M1 / medium-precision models M2 / high-precision models M3, but the principle of low-precision models first and high-precision models later should be followed to reduce the cost of the overall task. Figure 8 and Fig. 9 , Figure 8 and Fig. 9 Schematic diagrams of two other duplicate record detection pipelines provided in the embodiments of the present application, such as Figure 8 As shown, the pipeline includes two medium-precision models M2 and one high-precision model M3, where the two medium-precision models M2 are support vector machines and random forests, and the high-precision model M3 is BERT, as shown in Fig. 9 As shown, the pipeline includes 1 low-precision model M1 and 1 high-precision model M3, wherein the low-precision model M1 is a rule model and the high-precision model M3 is GPT.

[0196] Figure 8 as well as Fig. 9 The specific process of the duplicate record detection method shown is similar to Figure 7 Similarly, see Figure 7 For the sake of brevity of the specification, the related description will not be elaborated any more.

[0197] In a specific implementation, each duplicate record detection model and a diverter for diverting the record pairs output by the duplicate record detection model may be deployed and implemented by the same computing device, or deployed and implemented by different computing devices of the same or different computing device clusters, and this application does not make any specific limitations.

[0198] In a specific implementation, the above-mentioned splitters 1, 2, 3, etc. can be implemented by software or by hardware. For more specific implementation methods, please refer to Fig.10 For the sake of brevity of the specification, the implementation method of the detection model 1020 described in the embodiment will not be elaborated here.

[0199] It should be noted that the above Figure 1-Figure 9 The embodiments are all introduced by taking the example that the classification labels of each duplicate record detection model include two labels of "duplicate" and "non-duplicate". In the specific implementation, the classification labels of each duplicate record detection model can be subdivided into more labels, such as subdivided into three labels of "duplicate", "non-duplicate", and "uncertain", such as subdivided into five labels of "very certain to be repeated", "relatively certain to be repeated", "not sure whether to be repeated", "relatively certain not to be repeated", and "very certain not to be repeated". However, no matter how the classification labels of the duplicate record detection model change, the idea of ​​implementing duplicate record detection is the same as the above Figure 1-Figure 9 The ideas shown in the embodiments are similar and will not be described in detail for the sake of brevity.

[0200] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0201] The above describes in detail the duplicate record detection system 200 and the duplicate record detection method provided in the embodiments of the present application. Based on the same inventive concept, the following further describes the duplicate record detection apparatus and computing device cluster provided in the embodiments of the present application.

[0202] The duplicate recording detection device can be applied to the duplicate recording detection system 200 to realize various functions of the duplicate recording detection system 200. The duplicate recording detection device can be divided into one or more unit modules. For example, see Fig.10 , Fig.10 It is a structural schematic diagram of a duplicate record detection device provided in an embodiment of the present application.

[0203] like Fig.10 As shown, the duplicate record detection device 1000 includes: an acquisition module 1010 and a detection module 1020, and the detection module 1020 includes a first duplicate record detection model and a second duplicate record detection model. It should be understood that Fig.10This is an exemplary division method. The duplicate record detection device 1000 may also include more or fewer modules. For example, the duplicate record detection device 1000 may also include a sending module 1040 for sending the result of the duplicate record detection performed by the detection module 1020 on the record pair to the client 100, or to other storage systems for storage. For example, the detection module 1020 can be split into a first detection module and a second detection module, the first detection module includes a first duplicate record detection model, and the second detection module includes a second duplicate record detection model. The division method of the modules in the duplicate record detection device 1000 can be flexibly adjusted based on the actual business scenario, and this application does not make any specific limitation.

[0204] The accuracy of the first duplicate record detection model included in the detection module 1020 is lower than the accuracy of the second duplicate record detection model.

[0205] The following is an exemplary introduction to the functions of each module of the duplicate recording detection device 1000. It should be understood that the functions of each module described below are only functions that the duplicate recording detection device 1000 may have in some embodiments of the present application, and the present application does not limit the functions of each module.

[0206] The acquisition module 1010 is used to acquire the first record pair. Specifically, the acquisition module 1010 can be used to receive a duplicate record detection request carrying a data source sent by the client 100, and then extract at least one record pair from the data source, and use the at least one record pair as the first record pair in turn.

[0207] The detection module 1020 is used to input the first record pair into the first duplicate record detection model to obtain first prediction information, wherein the first prediction information includes the probability that the first duplicate record detection model predicts that the two records in the first record pair are duplicated and / or non-duplicate.

[0208] The detection module 1020 is also used to determine whether it is possible to determine whether the two records in the first record pair are repeated based on the first prediction information. If it is possible to determine whether the two records in the first record pair are repeated based on the first prediction information, a first determination result is output, and the first determination result is whether the two records in the first record pair are repeated or not.

[0209] The detection module 1020 is also used to input the first record pair into a second duplicate record detection model to obtain second prediction information when it cannot be determined whether the two records in the first record pair are duplicates based on the first prediction information. The second prediction information includes the probability that the two records in the first record pair are duplicated and / or non-duplicates predicted by the second duplicate record detection model.

[0210] The detection module 1020 is further configured to determine whether two records in the first record pair are repeated based on the second prediction information, and output a first determination result.

[0211] In some possible embodiments, the computing resources occupied by the second duplicate record detection model for detection are greater than the computing resources occupied by the first duplicate record detection model for detection.

[0212] In some possible embodiments, the time consumed by the second duplicate record detection model for detection is longer than the time consumed by the first duplicate record detection model for detection.

[0213] In some possible embodiments, the first duplicate record detection model is a model that uses multiple duplicate record detection rules to perform duplicate record detection, the second duplicate record detection model is an artificial intelligence model, or both the first duplicate record detection model and the second duplicate record detection model are artificial intelligence models.

[0214] In some possible embodiments, when the second duplicate record detection model is a deep learning model, the duplicate record detection device 1000 further includes: a calibration module 1030 (such as Fig.10 As shown), the calibration module 1030 is used to calibrate the second prediction information before the detection module 1020 determines whether it can determine whether the two records in the first record pair are repeated based on the second prediction information, so that the detection module 1020 can determine whether it can determine whether the two records in the first record pair are repeated based on the calibrated second prediction information, thereby improving accuracy.

[0215] In some possible embodiments, the acquisition module 1010 is also used to acquire a second record pair; the detection module 1020 is also used to input the second record pair into the first duplicate record detection model to obtain third prediction information, the third prediction information including the probability that the first duplicate record detection model predicts that the two records in the second record pair are repeated and / or non-repeated; the detection module 1020 is also used to determine whether the two records in the second record pair are repeated based on the third prediction information, and use the second determination result as the classification label of the second record pair, and add it to the training sample set of the second duplicate record detection model, the second determination result being whether the two records in the second record pair are repeated or not.

[0216] In some possible embodiments, Fig.10 As shown, the apparatus 1000 may further include a sending module 1040 for sending the first determination result / the second determination result to the client 100 .

[0217] In some possible embodiments, the device 1000 may further include a first flow splitting module 1050 (also referred to as a flow splitter 1050, Fig.10 (not shown), for diverting the record pairs detected by the detection module 1020 through the first duplicate record detection model. For example, when the detection module 1020 can determine whether two records in the first record pair are duplicates based on the first prediction information corresponding to the first duplicate record detection model, the first record pair is output to the "high confidence" record pair set ( Fig.10 (not shown), when the detection module 1020 cannot determine whether the two records in the first record pair are repeated based on the first prediction information, the first record pair is input into the second duplicate record detection model. Among them, the record pair with high confidence represents the record pair that the model compares to determine whether it is repeated.

[0218] In some possible embodiments, the device 1000 may further include a second flow splitting module 1060 (also referred to as a flow splitter 1060, Fig.10 (not shown), used to divert the record pairs detected by the detection module 1020 through the second duplicate record detection model. For example, when the detection module 1020 can determine whether the two records in the first record pair are duplicates based on the second prediction information corresponding to the second duplicate record detection model, the first record pair is output to the "high confidence" record pair set. When the detection module 1020 cannot determine whether the two records in the first record pair are duplicates based on the second prediction information, the first record pair is input into a third duplicate record detection model with higher accuracy than the second duplicate record detection model, or the first record pair is input into the manual inspection and annotation module ( Fig.10 (not shown), manually check and mark whether the two records in the first record pair are duplicates.

[0219] In some possible embodiments, after obtaining a record pair (such as the first record pair and the second record pair described above), the obtaining module 1010 may store the obtained record pair in the same data bucket (such as Fig.10 The subsequent detection module 1020 can obtain the record pairs from the first data bucket for duplicate record detection, and store the prediction information obtained by the detection (such as the first prediction information and the second prediction information) in another data bucket (such as Fig.10 The second data bucket shown in FIG. 10 ) and the result of detecting whether the record pair is repeated (such as the first determination result and the second determination result) are stored in another data bucket (such as FIG. 10 ). Fig.10The subsequent sending module 1040 can obtain the duplicate record detection results of multiple record pairs from the third data bucket and send them to the client 100. Optionally, the aforementioned record pairs, prediction information, and determination results can also be stored in the same data bucket, or the record pairs and prediction information are stored in the same data bucket, and the determination results are stored in another data bucket, or the record pairs and determination results are stored in the same data bucket, and the prediction information is stored in another data bucket, which is not specifically limited in this application.

[0220] In a specific implementation, the acquisition module 1010, the detection module 1020, the calibration module 1030 and the sending module 1040 can all be implemented by software, or can be implemented by hardware. Exemplarily, the detection module 1020 is taken as an example to introduce the implementation of the detection module 1020. Similarly, the implementation of the acquisition module 1010, the calibration module 1030 and the sending module 1040 can refer to the implementation of the detection module 1020.

[0221] As an example of a software functional unit, the detection module 1020 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the detection module 1020 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0222] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0223] As an example of a hardware functional unit, the detection module 1020 may include at least one computing device, such as a server, etc. Alternatively, the detection module 1020 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0224] When the detection module 1020 includes multiple computing devices, the multiple computing devices included can be distributed in the same region or in different regions. The multiple computing devices included in the detection module 1020 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the detection module 1020 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0225] It should be noted that, in other embodiments, the detection module 1020 may be used to execute Figure 4-Figure 9 In any step of the duplicate record detection method shown in FIG. Figure 4-Figure 9 The calibration module 1030 may be used to perform any step performed in the duplicate record detection method shown in FIG. Figure 4-Figure 9 In any step of the duplicate record detection method shown in FIG. 1 , the sending module 1040 can be used to perform Figure 4-Figure 9 In any step performed in the duplicate record detection method shown in FIG. 1 , the steps implemented by the acquisition module 1010, the detection module 1020, the calibration module 1030 and the sending module 1040 can be specified as needed. Figure 4-Figure 9 The different steps in the duplicate recording detection method shown are used to implement all functions of the duplicate recording detection device 1000.

[0226] The present application embodiment also provides a computing device 1100, which can be deployed Fig.10The duplicate record detection device 1000 shown in the figure may also be the duplicate record detection system 200 described above. The operations and / or functions of the various modules in the computing device 1100 are respectively to implement Figure 4-Figure 9 The corresponding steps in the detection method are repeatedly recorded as shown.

[0227] like Fig.11 As shown, the computing device 1100 includes: a processor 1110 , a memory 1120 , and a communication interface 1130 , wherein the processor 1110 , the memory 1120 , and the communication interface 1130 may be interconnected via a bus 1140 .

[0228] The processor 1110 can read the program code (including instructions) stored in the memory 1120, execute the program code stored in the memory 1120, and enable the computing device 1100 to execute Figure 4-Figure 9 The duplicate record detection method shown in the figure may enable the computing device 1100 to deploy the duplicate record detection apparatus 1000 .

[0229] The processor 1110 may have a variety of specific implementation forms, such as a central processing unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip may be an ASIC, a PLD, or a combination thereof. The PLD may be a CPLD, an FPGA, a GAL, or any combination thereof. The processor 1110 executes various types of digital storage instructions, such as software or firmware programs stored in the memory 1120, which enables the computing device 1100 to provide a wide variety of services.

[0230] In a specific implementation, as an embodiment, the processor 1110 includes one or more CPUs.

[0231] In a specific implementation, as an embodiment, the computing device 1100 also includes multiple processors, each of which can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here refers to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0232] The memory 1120 is used to store program codes and is controlled by the processor 1110 to execute the above Figure 4-Figure 9 In any embodiment, the processing steps of the detection method are repeatedly recorded. The program code may include one or more software modules, which may be Fig.10 The software modules provided in the embodiment include, for example, an acquisition module 1010 , a detection module 1020 , a calibration module 1030 and a sending module 1040 .

[0233] The memory 1120 may include a volatile memory, such as a random access memory (RAM); the memory 1120 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 1120 may also include a combination of the above types.

[0234] The communication interface 1130 may be a wired interface (e.g., an Ethernet interface, a fiber optic interface, or other types of interfaces (e.g., an infiniBand interface)) or a wireless interface (e.g., a cellular network interface or a wireless local area network interface) for communicating with other computing devices or apparatuses. The communication interface 1130 may use a protocol family above the transmission control protocol / internet protocol (TCP / IP), such as a remote function call (RFC) protocol, a simple object access protocol (SOAP) protocol, a simple network management protocol (SNMP) protocol, a common object request broker architecture (CORBA) protocol, and a distributed protocol, etc.

[0235] The bus 1140 may be a peripheral component interconnect express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus 1140 may be divided into an address bus, a data bus, a control bus, etc.

[0236] In addition to the data bus, the bus 1140 may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus 1140 in the figure. Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0237] The computing device 1100 is used to execute Figure 4-Figure 9 The specific implementation process of the duplicate record detection method shown is detailed in the above method embodiment, which will not be repeated here.

[0238] It should be understood that the computing device 1100 is only an example provided in the embodiment of the present application, and the computing device 1100 may have more Fig.11 The components shown may be more or less than the components shown, may be combined with two or more components, or may have different configurations of components. Figure 1-Figure 10 The relevant explanations in the embodiments will not be repeated here.

[0239] The present application also provides a computing device cluster 1200, which can be deployed Fig.10 The duplicate record detection device 1000 shown in the figure may also be the duplicate record detection system 200 described above. The operations and / or functions of the modules in the computing device cluster 1200 are respectively to realize Figure 4-Figure 9 The corresponding steps in the detection method are repeatedly recorded as shown.

[0240] like Fig.12 As shown, the computing device cluster 1200 includes at least one computing device 1100. The memory 1120 in one or more computing devices 1100 in the computing device cluster may store the same Figure 4-Figure 9 The computing device 1100 may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device 1100 may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0241] In some possible implementations, the memory 1120 of one or more computing devices 1100 in the computing device cluster 1200 may also store a program for executing Figure 4-Figure 9 In other words, the combination of one or more computing devices 1100 can jointly execute the method for executing Figure 4-Figure 9 The instructions for the repeating recording detection method are shown.

[0242] It should be noted that the memory 1120 in different computing devices 1100 in the computing device cluster 1200 may store different instructions, which are respectively used to execute part of the functions of the duplicate record detection apparatus 1000. That is, the instructions stored in the memory 1120 in different computing devices 1100 may implement the functions of one or more modules among the acquisition module 1010, the detection module 1020, the calibration module 1030 and the sending module 1040.

[0243] In some possible implementations, one or more computing devices 1100 in the computing device cluster 1200 may be connected via a network, which may be a wide area network or a local area network. Fig.13 A possible implementation is shown, such as Fig.13 As shown, two computing devices 1100A and 1100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 1120 in the computing device 1100A stores instructions for executing the functions of the acquisition module 1010 and the sending module 1040. At the same time, the memory 1120 in the computing device 1100B stores instructions for executing the functions of the detection module 1020 and the calibration module 1030.

[0244] It should be understood that Fig.13 The functions of the computing device 1100A shown in FIG. 1100A may also be completed by multiple computing devices 1100. Similarly, the functions of the computing device 1100B may also be completed by multiple computing devices 1100.

[0245] The present application also provides a computer program product containing instructions, which may be software or a program product containing instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes Figure 4-Figure 9 The duplicate record detection method shown.

[0246] The present application also provides a computer-readable storage medium, which can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a high-density digital video disc (DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute Figure 4-Figure 9 The duplicate record detection method shown.

[0247] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0248] In the above embodiments, it can be implemented in whole or in part by software, hardware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium, etc.

[0249] The above is only a specific implementation of the present application. Those skilled in the art may conceive of changes or substitutions based on the specific implementation provided by the present application, which should all be included in the protection scope of the present application.

Claims

1. A duplicate record detection method, characterized in that: Applied to a duplicate record detection system, the system comprising a first duplicate record detection model and a second duplicate record detection model, the accuracy of the second duplicate record detection model being higher than the accuracy of the first duplicate record detection model, the method comprising: Get the first record pair; Inputting the first record pair into the first duplicate record detection model to obtain first prediction information, wherein the first prediction information includes a probability that the first duplicate record detection model predicts that two records in the first record pair are duplicate or non-duplicate; determining whether it is possible to determine whether the two records in the first record pair are repeated based on the first prediction information, and outputting a first determination result if it is possible to determine whether the two records in the first record pair are repeated based on the first prediction information, the first determination result being whether the two records in the first record pair are repeated or not; In a case where it cannot be determined whether the two records in the first record pair are duplicates based on the first prediction information, inputting the first record pair into the second duplicate record detection model to obtain second prediction information, wherein the second prediction information includes a probability that the second duplicate record detection model predicts that the two records in the first record pair are duplicates or not; It is determined whether two records in the first record pair are duplicated based on the second prediction information, and the first determination result is output.

2. The method according to claim 1, characterized in that: The computing resources occupied by the second duplicate record detection model for detection are greater than the computing resources occupied by the first duplicate record detection model for detection.

3. The method according to claim 1 or 2, characterized in that: The time consumed by the second duplicate record detection model for detection is longer than the time consumed by the first duplicate record detection model for detection.

4. The method according to any one of claims 1 to 3, characterized in that: The first duplicate record detection model is a model that uses multiple duplicate record detection rules to perform duplicate record detection, the second duplicate record detection model is an artificial intelligence model, or both the first duplicate record detection model and the second duplicate record detection model are artificial intelligence models.

5. The method according to claim 4, characterized in that In the case where the second duplicate record detection model is a deep learning model, before judging whether it is possible to determine whether the two records in the first record pair are duplicates based on the second prediction information, the method further includes: The second prediction information is calibrated.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Get the second record pair; Inputting the second record pair into the first duplicate record detection model to obtain third prediction information, wherein the third prediction information includes the probability that the first duplicate record detection model predicts that two records in the second record pair are duplicated or non-duplicate; Based on the third prediction information, determine whether the two records in the second record pair are duplicates, and use the second determination result as the classification label of the second record pair, and add it to the training sample set of the second duplicate record detection model, the second determination result being whether the two records in the second record pair are duplicates or not.

7. The method according to any one of claims 1 to 6, characterized in that: The first duplicate record detection model and the second duplicate record detection model are deployed on the same computing device or on different computing devices.

8. A duplicate record detection device, characterized in that: Applied to a duplicate record detection system, the system comprising a first duplicate record detection model and a second duplicate record detection model, the accuracy of the second duplicate record detection model being higher than the accuracy of the first duplicate record detection model, the device comprising: An acquisition module, used for acquiring a first record pair; a detection module, configured to input the first record pair into the first duplicate record detection model to obtain first prediction information, wherein the first prediction information includes a probability that the first duplicate record detection model predicts that two records in the first record pair are duplicate or non-duplicate; The detection module is further configured to determine whether it is possible to determine whether the two records in the first record pair are repeated based on the first prediction information, and output a first determination result if it is possible to determine whether the two records in the first record pair are repeated based on the first prediction information, wherein the first determination result is whether the two records in the first record pair are repeated or not; The detection module is further configured to input the first record pair into the second duplicate record detection model to obtain second prediction information when it cannot be determined whether the two records in the first record pair are duplicates based on the first prediction information, wherein the second prediction information includes a probability that the second duplicate record detection model predicts that the two records in the first record pair are duplicates or not; The detection module is further used to determine whether two records in the first record pair are repeated based on the second prediction information, and output the first determination result.

9. The device according to claim 8, characterized in that The computing resources occupied by the second duplicate record detection model for detection are greater than the computing resources occupied by the first duplicate record detection model for detection.

10. The device according to claim 8 or 9, characterized in that The time consumed by the second duplicate record detection model for detection is longer than the time consumed by the first duplicate record detection model for detection.

11. The device according to any one of claims 8 to 10, characterized in that The first duplicate record detection model is a model that uses multiple duplicate record detection rules to perform duplicate record detection, the second duplicate record detection model is an artificial intelligence model, or both the first duplicate record detection model and the second duplicate record detection model are artificial intelligence models.

12. The device according to claim 11, characterized in that In the case where the second duplicate record detection model is a deep learning model, before the detection module is used to determine whether it is possible to determine whether the two records in the first record pair are duplicates based on the second prediction information, the device further includes: A calibration module is used to calibrate the second prediction information.

13. The device according to any one of claims 8 to 12, characterized in that The acquisition module is further used to acquire a second record pair; The detection module is further used to input the second record pair into the first duplicate record detection model to obtain third prediction information, wherein the third prediction information includes the probability that the first duplicate record detection model predicts that two records in the second record pair are duplicated or non-duplicate; The detection module is also used to determine whether the two records in the second record pair are duplicates based on the third prediction information, and use the second determination result as the classification label of the second record pair, and add the second determination result to the training sample set of the second duplicate record detection model, and the second determination result is whether the two records in the second record pair are duplicated or not.

14. The device according to any one of claims 8 to 13, characterized in that The first duplicate record detection model and the second duplicate record detection model are deployed on the same computing device or on different computing devices.

15. A computing device cluster, characterized in that: It includes at least one computing device, each of which includes a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 7.

Images