Data insertion method and device, computer device and storage medium

Abstract

The application provides a data insertion method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining to-be-inserted data, a target data table to be inserted and a corresponding deduplication rule; in the case that the deduplication rule is not empty, determining the coincidence degree between the to-be-inserted data and each first data in the target data table according to the deduplication rule; and in the case that the coincidence degree between the to-be-inserted data and each first data is less than a threshold value, inserting the to-be-inserted data into the target data table. The method can deduplicate the to-be-inserted data by using the deduplication rule of the data table to which the to-be-inserted data is inserted, without any coding development and code maintenance work, thereby reducing the cost and improving the data insertion efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, and storage medium for inserting data. Background Technology

[0002] In various data import scenarios, a common approach is to implement deduplication logic through coding. However, this coding-based deduplication solution requires developers to write the deduplication logic themselves. Whenever a new feature is added or an existing feature is changed, the original deduplication code needs to be added or modified again, resulting in high costs. Summary of the Invention

[0003] This application discloses a method, apparatus, computer device, and storage medium for inserting data.

[0004] One embodiment of this application proposes a data insertion method, including:

[0005] Get the data to be inserted, the target data table to be inserted into, and the corresponding deduplication rules;

[0006] If the deduplication rule is not empty, determine the degree of overlap between the data to be inserted and the first data in each data in the target data table according to the deduplication rule;

[0007] If the overlap between the data to be inserted and each of the first data is less than the threshold, the data to be inserted will be inserted into the target data table.

[0008] Another embodiment of this application proposes a data insertion device, comprising:

[0009] The acquisition module is used to acquire the data to be inserted, the target data table to be inserted, and its corresponding deduplication rules.

[0010] The determination module is used to determine the degree of overlap between the data to be inserted and each first data in the target data table, based on the deduplication rules, provided that the deduplication rules are not empty.

[0011] The insertion module is used to insert the data to be inserted into the target data table when the overlap between the data to be inserted and each of the first data is less than a threshold.

[0012] Another embodiment of this application provides a computer device including a processor and a memory;

[0013] The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the data insertion method as described in one aspect of the above embodiment.

[0014] Another embodiment of this application proposes a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data insertion method as described in the above-described embodiment.

[0015] Another embodiment of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the data insertion method according to the above-described embodiment.

[0016] The data insertion method, apparatus, computer device, and storage medium of this application embodiment obtain the data to be inserted, the target data table to be inserted, and their corresponding deduplication rules. If the deduplication rules are not empty, the overlap between the data to be inserted and each first piece of data in the target data table is determined according to the deduplication rules. If the overlap between the data to be inserted and each first piece of data is less than a threshold, the data to be inserted is inserted into the target data table. Therefore, by utilizing the deduplication rules of the data table to which the data to be inserted is inserted, deduplication of the data to be inserted can be performed without any coding development or code maintenance work, reducing costs and improving data insertion efficiency.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0019] Figure 1 A flowchart illustrating a data insertion method provided in an embodiment of this application;

[0020] Figure 2 A flowchart illustrating another data insertion method provided in an embodiment of this application;

[0021] Figure 3 A flowchart illustrating another data insertion method provided in an embodiment of this application;

[0022] Figure 4 A flowchart illustrating another data insertion method provided in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram illustrating a data insertion method provided in an embodiment of this application.

[0024] Figure 6 This is a schematic diagram of a data insertion device provided in an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0026] The following description, with reference to the accompanying drawings, outlines a data insertion method, apparatus, computer device, and storage medium according to embodiments of this application.

[0027] In various data import scenarios, a common approach is to implement deduplication logic through coding. However, this coding-based deduplication solution requires developers to write the deduplication logic themselves. Whenever a new feature is added or an existing feature is changed, the original deduplication code needs to be added or modified again, resulting in high costs.

[0028] Based on this, this application proposes a data insertion method. By deduplicating the data to be inserted according to the deduplication rules corresponding to the target data table, the developers only need to maintain the deduplication rules of the data table, without any coding development and code maintenance work, which reduces costs and improves data insertion efficiency.

[0029] Figure 1 This is a flowchart illustrating a data insertion method provided in an embodiment of this application.

[0030] The data insertion method of this application embodiment can be executed by the data insertion device provided in this application embodiment. The device can be configured in a computer device to deduplicate the data to be inserted according to the deduplication rules corresponding to the target data table to be inserted, thereby achieving the goal of eliminating the need for any coding development and code maintenance work, reducing costs and improving data insertion efficiency.

[0031] like Figure 1 As shown, the data insertion methods include:

[0032] Step 101: Obtain the data to be inserted, the target data table to be inserted, and its corresponding deduplication rules.

[0033] In practical applications, a database can include multiple tables, and data can be imported into the database for storage. In this application, deduplication rules can be pre-configured for the tables in the database as needed. These deduplication rules can be understood as the logic for removing duplicate data from the tables.

[0034] During the data import process, before inserting data into a specific table, you can obtain the data to be inserted, the target table, and the deduplication rules for the target table. The data to be inserted can be one or more pieces.

[0035] Step 102: If the deduplication rule is not empty, determine the overlap between the data to be inserted and the first data in the target data table according to the deduplication rule.

[0036] In this application, some data tables may have deduplication rules configured, while others may not. After obtaining the target data table, it can be determined whether the deduplication rules of the target data table are empty. If empty, it means that the target data table does not have corresponding deduplication rules, and the data to be inserted can be directly inserted into the target data table. If the target data table has deduplication rules, that is, the deduplication rules are not empty, it means that deduplication judgment is required when inserting data into the target data table. The degree of overlap between the data to be inserted and the first data in the target data table can be determined according to the deduplication rules.

[0037] The overlap here can be used to indicate the degree of duplication between the data to be inserted and the existing data in the target data table.

[0038] In this application, the target data table may contain multiple fields, and each set of values ​​from these fields constitutes a first set of data; that is, each first set of data contains the values ​​of each field. Different data tables may use different deduplication rules, i.e., different deduplication types.

[0039] For example, deduplication rules can include fields to consider during deduplication, and based on these fields, the overlap between the data to be inserted and the first data in the target table can be determined. Alternatively, deduplication rules can include an extended interface, which can be used to obtain the index data corresponding to the data to be inserted and the first data, and determine the overlap between the index data.

[0040] For example, if there is one field used for deduplication, and the value of this field in the data to be inserted is the same as the value of this field in a first piece of data, then the overlap between the data to be inserted and the first piece of data can be considered to be 100%.

[0041] If multiple fields are used for deduplication, the overlap can be determined by the number of fields in the first set of data that have the same value as the data to be inserted. For example, if there are two fields used for deduplication, and one of these fields has the same value, the overlap can be considered to be 50%.

[0042] Alternatively, the overlap can be determined by whether the values ​​of the same fields used for deduplication in the data to be inserted are all the same as those in the first data. For example, if there are three fields A, B, and C used for deduplication, if the values ​​of the data to be inserted are the same as those of the first data in fields A, B, and C, then the overlap between the data to be inserted and the first data can be determined to be 100%; otherwise, the overlap can be considered to be 0%.

[0043] It should be noted that the above method for determining the degree of overlap is merely an example and should not be regarded as a limitation of this application.

[0044] Step 103: If the overlap between the data to be inserted and each of the first data is less than the threshold, insert the data to be inserted into the target data table.

[0045] In this application, the overlap between the data to be inserted and each first data in the target data table can be compared with a threshold. If the overlap between the data to be inserted and each first data is less than the threshold, it can be considered that there is no duplication between the data to be inserted and the data in the target data table, and the data to be inserted can be inserted into the target data table.

[0046] For example, if the threshold is 1 and there are 3 fields used for deduplication, and the data to be inserted has the same values ​​in two fields as the first data, the overlap is two-thirds, which is less than the threshold of 1, so it can be considered that the statement to be inserted does not repeat the first data.

[0047] It should be noted that the threshold can be determined according to actual needs, and this application does not limit it.

[0048] Understandably, if there are multiple data to be inserted, each data to be inserted can be traversed. Based on the degree of overlap between each data to be inserted and each first data, it can be determined whether each data to be inserted is duplicated with existing data in the target data table. If there is no duplication, it is inserted into the target data table.

[0049] Alternatively, each piece of data to be inserted can be iterated over, and based on the overlap between each piece of data and each set of initial data, it can be determined whether each piece of data to be inserted is duplicated with existing data in the target data table. If duplicates are found, the data to be inserted can be removed. If, after iterating over all the data to be inserted, there are still valid pieces of data to be inserted, that is, there are pieces of data to be inserted that are not duplicated with existing data in the target data table, these valid pieces of data to be inserted can be inserted into the target data table.

[0050] In this embodiment, by obtaining the data to be inserted, the target data table to be inserted, and their corresponding deduplication rules, and assuming the deduplication rules are not empty, the overlap between the data to be inserted and each first data in the target data table is determined according to the deduplication rules. If the overlap between the data to be inserted and each first data is less than a threshold, the data to be inserted is inserted into the target data table. Therefore, by utilizing the deduplication rules of the data table to which the data to be inserted is inserted, deduplication of the data to be inserted can be performed without any coding development or code maintenance work, reducing costs and improving data insertion efficiency.

[0051] Figure 2This is a flowchart illustrating another data insertion method provided in an embodiment of this application.

[0052] like Figure 2 As shown, the data insertion methods include:

[0053] Step 201: Obtain the data to be inserted, the target data table to be inserted, and its corresponding deduplication rules. The deduplication rules include multiple deduplication fields.

[0054] In this application, the deduplication rule may include multiple deduplication fields. These deduplication fields can be understood as fields used for deduplication judgment or fields used to determine the index data in the target data table.

[0055] For example, the values ​​of fields A, B, and C in the target data table are combined to form a unique index, which cannot be duplicated.

[0056] Step 202: Determine the first key value corresponding to the data to be inserted based on the value of each deduplication field in the data to be inserted.

[0057] In this application, the data to be inserted may include values ​​from multiple fields, some of which are deduplication fields. The first key value corresponding to the data to be inserted can be determined based on the value corresponding to each deduplication field in the data to be inserted. The first key value may include the value corresponding to each deduplication field in the data to be inserted.

[0058] For example, the data to be inserted includes values ​​from 10 fields, where fields A, B, and C are deduplication fields. The values ​​from fields A, B, and C in the data to be inserted can be combined to obtain the first key value.

[0059] It should be noted that there can be one or more deduplication fields, and this application does not limit this.

[0060] Step 203: Determine the second key value corresponding to each first data based on the value of each deduplication field in each first data.

[0061] In this application, for each first data in the target data table, the second key value corresponding to each first data can be determined based on the value of each deduplication field in each first data.

[0062] For example, if the deduplication fields are field A, field B, and field C, the values ​​of field A, field B, and field C in the first set of data can form the second key value.

[0063] Step 204: Determine the degree of overlap between the first key value and each second key value.

[0064] After obtaining the first key value corresponding to the data to be inserted and the second key value corresponding to each first data point, the overlap between the first and second key values ​​can be determined based on the vectors corresponding to the first and second key values, respectively. The vectors corresponding to the first and second key values ​​can be determined based on the vectors corresponding to the values ​​of multiple deduplication fields.

[0065] For example, if the fields to be deduplicated are fields A, B, and C, and the first key value corresponding to the data to be inserted is a1-b1-c1, and the second key value corresponding to a certain first data is a1-b1-c2, it can be determined that the overlap between the first key value and the second key value is 0. If the second key value is a1-b1-c1, it can be considered that the overlap between the first key value and the second key value is 100%.

[0066] Step 205: If the overlap between the first key value and each second key value is less than the threshold, insert the data to be inserted into the target data table.

[0067] In this application, if the overlap between the first key value and each second key value is less than a threshold, it can be considered that the data to be inserted does not duplicate any of the second key values, that is, the data to be inserted does not duplicate existing data in the target data table, and thus the data to be inserted can be inserted into the target data table. If the overlap between the first key value and any second key value is greater than or equal to the threshold, it indicates that the data to be inserted duplicates the first key value corresponding to that second key value, and thus the data to be inserted is not inserted into the target data table.

[0068] If there are multiple data to be inserted, each data to be inserted can be traversed. Based on the overlap between the first key value and each second key value corresponding to each data to be inserted, it can be determined whether each data to be inserted is duplicated with existing data in the target data table. If they are not duplicated, they can be inserted into the target data table; if they are duplicated, they are not inserted.

[0069] Alternatively, each piece of data to be inserted can be iterated over, and based on the overlap between the first key value and each second key value corresponding to each piece of data to be inserted, it can be determined whether each piece of data to be inserted is duplicated with existing data in the target data table. If duplicates are found, the data to be inserted can be removed. If, after iterating over all the data to be inserted, there are still valid pieces of data to be inserted, that is, there are pieces of data to be inserted that are not duplicated with existing data in the target data table, these valid pieces of data to be inserted can be inserted into the target data table.

[0070] In this embodiment, the deduplication rule corresponding to the target data table may include multiple deduplication fields. When determining the overlap between the data to be inserted and each first data in the target data table according to the deduplication rule, a first key value corresponding to the data to be inserted can be determined based on the value corresponding to each deduplication field in the data to be inserted. A second key value corresponding to each first data can be determined based on the value corresponding to each deduplication field in each first data. The overlap between the first key value and each second key value is then determined. Based on the overlap between the first key value and each second key value, it is determined whether to insert the data to be inserted into the target data table. Thus, the overlap between the data to be inserted and each first data in the target data table can be determined based on the multiple deduplication fields included in the deduplication rule, thereby determining whether to insert the data to be inserted into the target data table. This eliminates the need to write and maintain code, reducing costs and improving data insertion efficiency.

[0071] Figure 3 This is a flowchart illustrating another data insertion method provided in an embodiment of this application.

[0072] like Figure 3 As shown, the data insertion methods include:

[0073] Step 301: Obtain the data to be inserted, the target data table to be inserted, and its corresponding deduplication rules, wherein the deduplication rules include the first index field.

[0074] In this application, the deduplication rule corresponding to the target data table may include a first index field, which can be a dynamic field in the target data table that determines the index construction method. That is to say, the construction of unique index data in the data table is based on the dynamic changes of the first index field; different values ​​of this field result in different construction methods for the unique index data.

[0075] For example, in a data table, when field A = 1, the values ​​of field B and field C are combined to form a unique index; when field A = 2, the values ​​of field B, field C, and field D are combined to form a unique index. Here, field A is the first index field.

[0076] Step 302: Obtain the first value of the first index field in the data to be inserted, and the second value of the first index field in each piece of first data in the target data table.

[0077] In this application, the data to be inserted and each piece of first data may include values ​​for multiple fields, including the value of the first index field. Therefore, it is possible to obtain the first value of the first index field in the data to be inserted and the second value of the first index field in each piece of first data in the target data table.

[0078] Step 303: If the first value is the same as any second value, determine the second index field.

[0079] Because the values ​​of the first index field differ, the second index field used to construct the unique index data may also differ. For example, in the example above, when the first index field A = 1, the second index fields are fields B and C; when the first index field A = 2, the second index fields are fields B, C, and D. Therefore, after obtaining the first value corresponding to the data to be inserted and the second value corresponding to each first data point, the first value can be compared with each second value. If the first value is different from each second value, it can be considered that the data to be inserted does not duplicate existing data in the target data table, and the data to be inserted can be inserted into the target data table.

[0080] If the first value is the same as any second value, it is necessary to compare whether the values ​​of the second index field are the same. The second index field corresponding to the first value can be determined based on the correspondence between the values ​​of the first index field and the second index field.

[0081] Understandably, there may be no second value that is the same as the first value, or there may be one or more of them. In other words, there may be no first data in the target data table that has the same value as the first index field in the data to be inserted, or there may be one or more of them.

[0082] Step 304: Obtain the third value of the second index field in the data to be inserted, and the fourth value of the second index field in the first data to which any second value belongs.

[0083] After determining the second index field, we can obtain the third value of the second index field in the data to be inserted, and the fourth value of the second index field in the first data to which any second value belongs. It is understood that "any second value" here refers to the same second value as the first value.

[0084] Step 305: Determine the degree of overlap between the third and fourth values.

[0085] In this application, the method for determining the overlap between the third and fourth values ​​can be found in the method for determining the overlap between the first and second key values, as described above, and will not be repeated here.

[0086] If there are multiple second values ​​that are the same as the first value, then the fourth value of the second index field in the first data to which each second value that is the same as the first value belongs can be obtained, and the degree of overlap between the third value and each fourth value can be determined.

[0087] Step 306: If the overlap between the third value and each of the fourth values ​​is less than the threshold, insert the data to be inserted into the target data table.

[0088] In this application, if the overlap between the third value and each of the fourth values ​​is less than the threshold, it means that the data to be inserted does not overlap with the existing data in the target data table, and the data to be inserted can be inserted into the target data table.

[0089] If there are multiple data to be inserted, each data to be inserted can be iterated through. The method described above can be used to determine whether each data to be inserted is duplicated with existing data in the target data table. If they are not duplicated, they can be inserted into the target data table. If they are duplicated, they are not inserted.

[0090] Alternatively, each piece of data to be inserted can be iterated through, and each piece of data can be determined using the method described above to see if it is a duplicate of existing data in the target table. If it is a duplicate, the data to be inserted can be removed. If, after iterating through all the data to be inserted, there are still valid pieces of data to be inserted, that is, there are pieces of data to be inserted that are not duplicates of existing data in the target table, these valid pieces of data can be inserted into the target table.

[0091] In this embodiment, the deduplication rule corresponding to the target data table includes a first index field. When determining the overlap between the data to be inserted and each first data in the target data table according to the deduplication rule, the first value of the first index field in the data to be inserted and the second value of the first index field in each first data in the target data table can be obtained. If the first value is the same as any second value, the second index field is determined. Then, the third value of the second index field in the data to be inserted and the fourth value of the second index field in the first data to which any second value belongs are obtained. The overlap between the third value and the fourth value is determined. Based on the overlap between the third value and each fourth value, it is determined whether to insert the data to be inserted into the target data table. Thus, when the construction of the unique index data in the target data table is based on the dynamic change of a certain field, the value of the index determined by the deduplication rule can be used for deduplication, avoiding the insertion of duplicate data into the target data table.

[0092] Figure 4 This is a flowchart illustrating another data insertion method provided in an embodiment of this application.

[0093] like Figure 4 As shown, the data insertion methods include:

[0094] Step 401: Obtain the data to be inserted, the target data table to be inserted, and its corresponding deduplication rules, wherein the deduplication rules include an extended interface.

[0095] In practical applications, the unique index data in some data tables may change dynamically, have complex combination methods, or depend on external data. To address this, the deduplication rules for the target data table in this application may include an extension interface. This extension interface allows an external implementation to output the combination relationships of the unique index data.

[0096] Step 402: Call the extended interface corresponding to the data to be inserted and each first data respectively to obtain the first index data corresponding to the data to be inserted and the second index data corresponding to each first data.

[0097] In this application, the extended interfaces corresponding to the data to be inserted and each first data can be called separately to obtain the first index data corresponding to the data to be inserted and the second index data corresponding to each first data from the outside. Thus, the first index data corresponding to the data to be inserted and the second index data corresponding to each first data can be obtained through the extended interfaces. The first index data and the second index data may include fields and their corresponding values.

[0098] For example, the first index data consists of the value a1 of field A and the value e1 of field E, and the second index data consists of the value a2 of field A and the value f2 of field F.

[0099] Step 403: Determine the overlap between the first index data and each second index data.

[0100] When determining the overlap between the first index data and each second index data, we can first determine whether the field combinations corresponding to the first index data and each second index data are the same. If they are all different, it means that the data to be inserted does not duplicate the existing data in the target data table, and the data to be inserted can be inserted into the target data table. If the field combinations corresponding to the first index data and the second index data are the same, we can further determine the overlap based on the values ​​of the corresponding fields.

[0101] For example, the first index data consists of the values ​​a1 of field A and e1 of field E, and the second index data consists of the values ​​a2 of field A and f2 of field F. Since the first index data is composed of the values ​​of field A and field E, which is different from the corresponding field combination in the second index data, the overlap between the first and second index data can be determined to be 0%. If the second index data is composed of the values ​​a1 of field A and e2 of field E, since the values ​​of field E in the first and second index data are different, the overlap between the first and second index data can be determined to be 50%.

[0102] Step 404: If the overlap between the first index data and each second index data is less than the threshold, insert the data to be inserted into the target data table.

[0103] If the overlap between the first index data and each second index data is less than the threshold, it means that the data to be inserted does not duplicate existing data in the target data table, and the data to be inserted can be inserted into the target data table. If the overlap between the first index data and any second index data is greater than or equal to the threshold, it means that the data to be inserted duplicates the first data corresponding to that second index data, and the data to be inserted will not be inserted into the target data table.

[0104] If there are multiple data to be inserted, each data to be inserted can be traversed. Based on the overlap between the first index data and each second index data corresponding to each data to be inserted, it can be determined whether each data to be inserted is duplicated with existing data in the target data table. If they are not duplicated, they can be inserted into the target data table; if they are duplicated, they are not inserted.

[0105] Alternatively, each piece of data to be inserted can be iterated over, and the overlap between the first index data and each second index data corresponding to each piece of data can be used to determine whether each piece of data to be inserted is duplicated with existing data in the target data table. If any piece of data to be inserted is duplicated with existing data in the target data table, that piece of data can be removed; if after iterating over all the pieces of data to be inserted, there are still valid pieces of data to be inserted, that is, there are pieces of data to be inserted that are not duplicated with existing data in the target data table, these valid pieces of data to be inserted can be inserted into the target data table.

[0106] In this embodiment, the deduplication rule may include an extended interface. When determining whether the data to be inserted is duplicated with each first piece of data in the target data table according to the deduplication rule, the extended interface corresponding to the data to be inserted and each first piece of data can be called respectively to obtain the first index data corresponding to the data to be inserted and the second index data corresponding to each first piece of data, and to determine the overlap between the first index data and each second index data. Based on the overlap between the first index data and each second index data, it can be determined whether to insert the data to be inserted into the target data table. Therefore, when the deduplication rule corresponding to the target data table includes an extended interface, the extended interface can be called one by one for the data to be inserted and each first piece of data to obtain the corresponding unique index data and compare it to determine whether to insert the data to be inserted into the target data table.

[0107] To further improve data insertion efficiency, in one embodiment of this application, before determining the overlap between the data to be inserted and each first data in the target data table according to the deduplication rules, the target data table can be searched according to the search conditions corresponding to the deduplication rules, and it can be determined whether the search results are empty.

[0108] In this application, the search criteria can be configured when configuring the deduplication rules of the target data table, or determined according to the deduplication rules. The search criteria can be fields included in the deduplication rules, and the field can be searched in the target data table to obtain search results.

[0109] If the search results are not empty, it indicates that the target data table is not empty and contains the values ​​of these fields. Based on deduplication rules, the overlap between the data to be inserted and the first data entries in the target data table can be determined. Then, based on the overlap, it can be determined whether to insert the data to be inserted into the target data table. If the search results are empty, the data to be inserted can be directly inserted into the target data table. Therefore, based on the search conditions and the determination that the search results are not empty, deduplication rules can be used to perform data deduplication and insertion.

[0110] In one embodiment of this application, when obtaining the data to be inserted, the target data table to be inserted, and the corresponding deduplication rules, the data to be inserted and the target data table to be inserted can be obtained, and the deduplication rules corresponding to the target data table can be determined according to the configuration information corresponding to the target data table.

[0111] In this application, a data model can be pre-built, which can define multiple data tables, the relationships between data tables, and deduplication rules for individual data tables. When inserting data into a data table, the deduplication rules for that data table can be determined based on the configuration information corresponding to that data table, which may include deduplication rules.

[0112] In this embodiment, the deduplication rules corresponding to the target data table can be determined based on the configuration information of the target data table, and data insertion can be performed based on the deduplication rules. Therefore, by configuring the deduplication rules of the data table, data deduplication can be achieved when inserting data into the data table. Developers only need to maintain the data deduplication configuration; no code writing or maintenance is required, reducing costs and improving data insertion efficiency.

[0113] Figure 5 This is a schematic diagram illustrating a data insertion method provided in an embodiment of this application.

[0114] Figure 5In step 501, assuming the existing data model defines the relationships between data tables, analyze the deduplication rules for individual data tables and configure them accordingly for different deduplication types. In step 502, during the data import process, in the data insertion step, determine if the target data table for which data is to be inserted has deduplication rules. If not, proceed to step 503 to begin the data insertion process. If rules exist, proceed to step 504 to retrieve data from the current database based on the search conditions defined in the data model. Then, proceed to step 505 to determine if duplicate data exists based on the different deduplication types. If duplicates exist, remove the data to be inserted; otherwise, begin the data insertion process.

[0115] To implement the above embodiments, this application also proposes a data insertion device. Figure 6 This is a schematic diagram of a data insertion device provided in an embodiment of this application.

[0116] like Figure 6 As shown, the data insertion device 600 includes: an acquisition module 610, a determination module 620, and an insertion module 630.

[0117] The acquisition module 610 is used to acquire the data to be inserted, the target data table to be inserted, and the corresponding deduplication rules;

[0118] The determination module 620 is used to determine the degree of overlap between the data to be inserted and each first data in the target data table, based on the deduplication rules, when the deduplication rules are not empty.

[0119] The insertion module 630 is used to insert the data to be inserted into the target data table when the overlap between the data to be inserted and each of the first data is less than a threshold.

[0120] In one implementation of this application, the deduplication rule includes multiple deduplication fields, and the determining module 620 is used for:

[0121] Based on the value of each deduplication field in the data to be inserted, determine the first key value corresponding to the data to be inserted;

[0122] Based on the value of each deduplication field in each first data, determine the second key value corresponding to each first data;

[0123] Determine the degree of overlap between the first key value and each second key value.

[0124] In one implementation of this application, the deduplication rule includes a first index field, and the determining module 620 is used for:

[0125] Get the first value of the first index field in the data to be inserted, and the second value of the first index field in each piece of data in the target data table;

[0126] If the first value is the same as any second value, determine the second index field;

[0127] Get the third value of the second index field in the data to be inserted, and the fourth value of the second index field in the first data to which any second value belongs;

[0128] Determine the degree of overlap between the third and fourth values.

[0129] In one implementation of this application, the deduplication rule includes an extended interface, and a determining module 620, based on the deduplication rule, is used for:

[0130] Call the extended interface corresponding to the data to be inserted and each first data respectively to obtain the first index data corresponding to the data to be inserted and the second index data corresponding to each first data;

[0131] Determine the degree of overlap between the first index data and each second index data.

[0132] In one implementation of this application, the apparatus may further include:

[0133] The retrieval module is used to retrieve data from the target data table based on the retrieval conditions corresponding to the deduplication rules, and to ensure that the retrieval results are not empty.

[0134] In one implementation of this application embodiment, the acquisition module 610 is used for:

[0135] Retrieve the data to be inserted and the target data table to be inserted;

[0136] Based on the configuration information corresponding to the target data table, the deduplication rules corresponding to the target data table are determined. It should be noted that the above explanation of the data insertion method embodiment also applies to the data insertion device of this embodiment, and therefore will not be repeated here.

[0137] In this embodiment, by obtaining the data to be inserted, the target data table to be inserted, and their corresponding deduplication rules, and assuming the deduplication rules are not empty, the overlap between the data to be inserted and each first data in the target data table is determined according to the deduplication rules. If the overlap between the data to be inserted and each first data is less than a threshold, the data to be inserted is inserted into the target data table. Therefore, by utilizing the deduplication rules of the data table to which the data to be inserted is inserted, deduplication of the data to be inserted can be performed without any coding development or code maintenance work, reducing costs and improving data insertion efficiency.

[0138] To implement the above embodiments, this application also proposes a computer device, including a processor and a memory;

[0139] The processor reads executable program code stored in memory to run a program corresponding to the executable program code, so as to implement the data insertion method as described in the above embodiments.

[0140] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data insertion method as described in the above embodiments.

[0141] To implement the above embodiments, this application also proposes a computer program product, characterized in that it includes a computer program, which, when executed by a processor, implements the data insertion method according to the above embodiments.

[0142] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0143] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for inserting data, characterized in that, include: Obtain the data to be inserted, the target data table to be inserted, and the deduplication rules corresponding to the target data table; If the deduplication rule is not empty, the overlap between the data to be inserted and each first data in the target data table is determined according to the deduplication rule. If the overlap between the data to be inserted and each of the first data is less than the threshold, the data to be inserted is inserted into the target data table. The deduplication rule includes a first index field, which is a dynamic field in the target data table that determines the index construction method. Determining the overlap between the data to be inserted and each first piece of data in the target data table according to the deduplication rule includes: Obtain the first value of the first index field in the data to be inserted, and the second value of the first index field in each piece of first data in the target data table; If the first value is the same as any second value, then the second index field is determined. Obtain the third value of the second index field in the data to be inserted, and the fourth value of the second index field in the first data to which any of the second values ​​belong; Determine the degree of overlap between the third value and the fourth value.

2. The method as described in claim 1, characterized in that, The deduplication rule includes multiple deduplication fields. Determining the overlap between the data to be inserted and each piece of first data in the target data table according to the deduplication rule includes: Based on the value of each deduplication field in the data to be inserted, determine the first key value corresponding to the data to be inserted; Based on the value of each deduplication field in each of the first data, determine the second key value corresponding to each of the first data; Determine the degree of overlap between the first key value and each of the second key values.

3. The method as described in claim 1, characterized in that, The deduplication rule includes an extended interface. The step of determining whether the data to be inserted is duplicated with each of the first data items in the target data table, based on the deduplication rule, includes: Call the extended interface corresponding to the data to be inserted and each of the first data respectively to obtain the first index data corresponding to the data to be inserted and the second index data corresponding to each of the first data; Determine the degree of overlap between the first index data and each of the second index data.

4. The method according to any one of claims 1-3, characterized in that, Before determining the overlap between the data to be inserted and each first piece of data in the target data table according to the deduplication rule, the method further includes: The target data table is searched according to the search conditions corresponding to the deduplication rule, and the search results are determined to be non-empty.

5. The method as described in any one of claims 1-3, characterized in that, The process of obtaining the data to be inserted, the target data table to be inserted, and the corresponding deduplication rules includes: Retrieve the data to be inserted and the target data table to be inserted; Based on the configuration information corresponding to the target data table, determine the deduplication rule corresponding to the target data table.

6. A data insertion device, characterized in that, include: The acquisition module is used to acquire the data to be inserted, the target data table to be inserted, and the deduplication rules corresponding to the target data table. The determination module is used to determine the overlap between the data to be inserted and each first data in the target data table, based on the deduplication rule, when the deduplication rule is not empty. An insertion module is used to insert the data to be inserted into the target data table when the overlap between the data to be inserted and each of the first data is less than a threshold. The deduplication rule includes a first index field, which is a dynamic field in the target data table that determines the index construction method. The determining module is used to: Obtain the first value of the first index field in the data to be inserted, and the second value of the first index field in each piece of first data in the target data table; If the first value is the same as any second value, then the second index field is determined. Obtain the third value of the second index field in the data to be inserted, and the fourth value of the second index field in the first data to which any of the second values ​​belong; Determine the degree of overlap between the third value and the fourth value.

7. The apparatus as claimed in claim 6, characterized in that, The deduplication rule includes multiple deduplication fields, and the determining module is used for: Based on the value of each deduplication field in the data to be inserted, determine the first key value corresponding to the data to be inserted; Based on the value of each deduplication field in each of the first data, determine the second key value corresponding to each of the first data; Determine the degree of overlap between the first key value and each of the second key values.

8. The apparatus as claimed in claim 6, characterized in that, The deduplication rule includes an extended interface, and the determining module, based on the deduplication rule, is used for: Call the extended interface corresponding to the data to be inserted and each of the first data respectively to obtain the first index data corresponding to the data to be inserted and the second index data corresponding to each of the first data; Determine the degree of overlap between the first index data and each of the second index data.

9. The apparatus according to any one of claims 6-8, characterized in that, Also includes: The retrieval module is used to retrieve the target data table according to the retrieval conditions corresponding to the deduplication rules, and to determine that the retrieval results are not empty.

10. The apparatus according to any one of claims 6-8, characterized in that, The acquisition module is used for: Retrieve the data to be inserted and the target data table to be inserted; Based on the configuration information corresponding to the target data table, determine the deduplication rule corresponding to the target data table.

11. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the data insertion method as described in any one of claims 1-5.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data insertion method as described in any one of claims 1-5.

Images