Data management method and device, electronic equipment and storage medium

Abstract

The invention provides a data management method and device, electronic equipment and a storage medium, and the data management method comprises the steps: receiving an instruction used for indicating to write a plurality of data rows into a data table, if the data table and an index of the data table have uniqueness constraints, determining whether each data row in the plurality of data rows has uniqueness conflict with the data table or not, and if the data table and the index of the data table have uniqueness constraints, executing the data management; judging whether the index line corresponding to each data line in the plurality of data lines has a uniqueness conflict with the index of the data table or not; writing all conflict-free data lines in the plurality of data lines into a data table in the persistent storage device, and writing index lines corresponding to all the conflict-free data lines in the plurality of data lines into indexes of the data table in the persistent storage device; and writing all conflict data lines in the plurality of data lines into a data table in the memory, and writing index lines corresponding to all conflict data in the plurality of data lines into indexes of the data table in the memory.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to a data management method and device, an electronic device, and a storage medium. Background Art

[0002] In the field of databases, data tables can generate indexes to speed up queries, provide sorting functions, and achieve data integrity. The unique key in the data table has a uniqueness constraint, and the uniqueness of the unique key must be guaranteed when inserting data. Some indexes in the data table also have a uniqueness constraint, that is, the uniqueness of the generated index field must be guaranteed when inserting data.

[0003] In the related art, when the database has a uniqueness constraint on the data table but the index has no uniqueness constraint, the database can ensure efficiency and quality when inserting data into the data table. However, when the database inserts data into the data table when both the data table and its index have uniqueness constraints, it is easy for the inserted data to have uniqueness conflicts or problems such as low insertion efficiency. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a data management method and device, an electronic device, and a storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a data management method is proposed, the method comprising: receiving an instruction for instructing to write a plurality of data rows into a data table, and if the data table and the index of the data table both have a uniqueness constraint, determining whether each of the plurality of data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the plurality of data rows has a uniqueness conflict with the index of the data table; Writing all conflict-free data rows among the multiple data rows into the data table in the persistent storage device, and writing index rows corresponding to all conflict-free data rows among the multiple data rows into the index of the data table in the persistent storage device, wherein if any data row does not have a uniqueness conflict with the data table, and the index row corresponding to the data row does not have a uniqueness conflict with the index of the data table, then the data row is a conflict-free data row; All conflicting data rows among the multiple data rows are written into the data table in the memory, and index rows corresponding to all conflicting data among the multiple data rows are written into the index of the data table in the memory, wherein if any data row has a uniqueness conflict with the data table, or if the index row corresponding to the data row has a uniqueness conflict with the index of the data table, then the data row is a conflicting data row.

[0006] In a possible embodiment of the present specification, determining whether each of the multiple data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the multiple data rows has a uniqueness conflict with an index of the data table, includes: Writing the multiple data rows into a temporary file, and generating at least one index temporary file based on the multiple data rows, wherein the index temporary file contains index rows of the multiple data rows; Determine, based on the temporary file and the data table, whether each of the multiple data rows has a uniqueness conflict with a data row already included in the data table; For each temporary index file, according to the corresponding indexes of the temporary index file and the data table, it is determined whether each index row in the temporary index file has a uniqueness conflict with an index row in the corresponding index of the data table.

[0007] In a possible embodiment of the present specification, writing the multiple data rows into a temporary file and generating at least one index temporary file based on the multiple data rows includes: The multiple data rows and the serial numbers of the data rows are written into a temporary file, and at least one index temporary file is generated based on all the data rows in the temporary file, wherein the index rows in the index temporary file have the serial numbers of the corresponding data rows.

[0008] In a possible embodiment of this specification, the method further includes: Generate a conflict record based on the serial number of the data row that has a uniqueness conflict with the data table and the serial number in the index row that has a uniqueness conflict with the index of the data table, wherein the conflict record records the serial number that has a uniqueness conflict with the index of the data table and / or the index table; Based on the temporary file and the conflict record, at least one conflict-free data row and / or at least one conflicting data row is determined in the temporary file.

[0009] In a possible embodiment of the present specification, before determining, based on the temporary file and the data table, whether each of the multiple data rows has a uniqueness conflict with a data row already included in the data table, the method further includes: Sort the data rows in the temporary file based on the unique key of the data table; Before determining, based on the corresponding indexes of the temporary index file and the data table, whether each index row in the temporary index file has a uniqueness conflict with an index row in a corresponding index of the data table, the method further includes: The index rows in the index temporary file are sorted based on the index fields of the corresponding indexes of the data table.

[0010] In a possible embodiment of the present specification, before writing all conflicting data rows in the multiple data rows into the data table in the memory, and writing index rows corresponding to all conflicting data in the multiple data rows into the index of the data table in the memory, the method further includes: All conflicting data rows in the temporary file are sorted according to the sequence numbers.

[0011] In a possible embodiment of this specification, the method further includes: Uniqueness conflict handling is performed during the writing of all conflicting data rows and index rows.

[0012] In a possible embodiment of the present specification, the step of writing all conflicting data rows among the multiple data rows into the data table in the memory, and writing index rows corresponding to all conflicting data among the multiple data rows into the index of the data table in the memory, and performing uniqueness conflict processing in the process of writing all conflicting data rows and index rows, includes: All conflicting data rows in the multiple data rows are written into the data table in the memory according to the replacement insertion mode or the ignore insertion mode, and index rows corresponding to all conflicting data rows in the multiple data rows are written into the index of the data table in the memory.

[0013] In a possible embodiment of the present specification, the receiving of an instruction for instructing to write a plurality of data rows into a data table, if the data table and an index of the data table both have a uniqueness constraint, determining whether each of the plurality of data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the plurality of data rows has a uniqueness conflict with the index of the data table, includes: An instruction for instructing to write multiple data rows into a data table is received. If the data table and the index of the data table both have uniqueness constraints, and the number of the multiple data rows is greater than a preset threshold, it is determined whether each of the multiple data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the multiple data rows has a uniqueness conflict with the index of the data table.

[0014] According to a second aspect of one or more embodiments of this specification, a data management device is provided, the device comprising: a determination module, configured to receive an instruction for instructing to write a plurality of data rows into a data table, and if the data table and the index of the data table both have a uniqueness constraint, determine whether each of the plurality of data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the plurality of data rows has a uniqueness conflict with the index of the data table; A first writing module is used to write all conflict-free data rows among the multiple data rows into the data table in the persistent storage device, and write index rows corresponding to all conflict-free data rows among the multiple data rows into the index of the data table in the persistent storage device, wherein if any data row does not have a uniqueness conflict with the data table, and the index row corresponding to the data row does not have a uniqueness conflict with the index of the data table, then the data row is a conflict-free data row; The second writing module is used to write all conflicting data rows among the multiple data rows into the data table in the memory, and write index rows corresponding to all conflicting data among the multiple data rows into the index of the data table in the memory, wherein if any data row has a uniqueness conflict with the data table, or the index row corresponding to the data row has a uniqueness conflict with the index of the data table, then the data row is a conflicting data row.

[0015] According to a third aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0016] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor implements the method as described in the first aspect by running the executable instructions.

[0017] According to a fifth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] The technical solutions provided by the embodiments of this specification may have the following beneficial effects: The data management method provided in the embodiment of the present specification receives an instruction for instructing to write multiple data rows into a data table, and if the data table and the index of the data table both have a uniqueness constraint, determines whether each of the multiple data rows has a uniqueness conflict with the data table, and whether the index row corresponding to each of the multiple data rows has a uniqueness conflict with the index of the data table; writes all conflict-free data rows in the multiple data rows into the data table in a persistent storage device, and writes the index rows corresponding to all conflict-free data rows in the multiple data rows into the persistent storage device. In the index of the data table, if any data row has no uniqueness conflict with the data table, and the index row corresponding to the data row has no uniqueness conflict with the index of the data table, then the data row is a non-conflicting data row; all conflicting data rows in the multiple data rows are written into the data table in memory, and the index rows corresponding to all conflicting data in the multiple data rows are written into the index of the data table in memory, wherein if any data row has a uniqueness conflict with the data table, or the index row corresponding to the data row has a uniqueness conflict with the index of the data table, then the data row is a conflicting data row. The method adopts a method of directly writing data rows that have no uniqueness conflict with the data table and its index into a persistent storage device, thereby ensuring data insertion efficiency, and adopts a method of writing data in memory for data that has a uniqueness conflict with the data table or its index, thereby resolving uniqueness conflicts in the method of writing in memory to ensure that the written data has no uniqueness conflicts, so that when the method inserts data into the data table when both the data table and its index have uniqueness constraints, both insertion efficiency and quality are taken into account. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a data management method provided by an exemplary embodiment.

[0020] Figure 2 It is a schematic diagram of inserting a small amount of data rows into a database provided by an exemplary embodiment.

[0021] Figure 3 It is a schematic diagram of generating a temporary file provided by an exemplary embodiment.

[0022] Figure 4 It is a schematic diagram of uniqueness conflict detection provided by an exemplary embodiment.

[0023] Figure 5 It is a schematic diagram of writing conflict-free data rows and their corresponding index rows into a persistent storage device provided by an exemplary embodiment.

[0024] Figure 6 It is a schematic diagram of writing conflict-free data rows into a persistent storage device provided by an exemplary embodiment.

[0025] Figure 7 It is a schematic diagram of writing index rows corresponding to conflict-free data rows into a persistent storage device provided by an exemplary embodiment.

[0026] Figure 8 It is a schematic diagram of writing conflicting data rows into a memory provided by an exemplary embodiment.

[0027] Figures 9 to 17 It is a schematic diagram of a process of writing batch data into a database provided by an exemplary embodiment.

[0028] Fig.18 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0029] Fig.19 is a block diagram of a data management device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0031] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0032] Based on the above technical problems mentioned in the background technology, at least one embodiment of this specification provides a data management method, which can improve the quality and efficiency of data insertion when inserting data into a data table when both the data table and its index have unique constraints. The method can be applied to databases, such as distributed databases based on LSM-Tree storage engines, etc.

[0033] Please follow the attached Figure 1 , which exemplarily shows the process of the method, including step S201 and step S203.

[0034] In step S101, an instruction for instructing to write multiple data rows into a data table is received. If the data table and the index of the data table both have uniqueness constraints, it is determined whether each of the multiple data rows has a uniqueness conflict with the data table, and whether the index row corresponding to each of the multiple data rows has a uniqueness conflict with the index of the data table.

[0035] It should be understood that the problem of low insertion efficiency will only occur when a large number of data rows are inserted in batches; if a small number of data rows are inserted, even if the insertion method is poor, it will not cause too much efficiency problem. Therefore, this step can be performed according to the following conditions: receiving an instruction for instructing to write multiple data rows into a data table, if the data table and the index of the data table both have unique constraints, and the number of the multiple data rows is greater than a preset threshold, then determining whether each of the multiple data rows has a uniqueness conflict with the data table, and whether the index row corresponding to each of the multiple data rows has a uniqueness conflict with the index of the data table. The preset threshold is used to measure whether the inserted data rows are considered batch insertions, that is, if the number of inserted data rows is greater than the preset threshold, it is considered to be a batch insertion, and if it is not greater than the preset threshold, it is not considered to be a batch insertion.

[0036] Correspondingly, if a small number of data rows are inserted, you can follow the Figure 2 Write to the database in the manner shown. The data rows to be inserted are written to the Active Memory Table in the memory by means of write-ahead logging (WAL). When the Active Memory Table meets the freezing conditions, it is frozen into the Immutable Memory Table. The Immutable Memory Table is further dumped into the lowest level 0 ordered string table SSTable in the persistent storage device Disk by flushing. The lowest level 0 ordered string table SSTable will be gradually merged and compacted into higher layers, such as the ordered string table SSTable of level 1, level 2, etc.

[0037] Exemplarily, whether each of the multiple data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the multiple data rows has a uniqueness conflict with the index of the data table is determined in the following manner: First, please refer to the attached Figure 3 , writing the multiple data rows into a temporary file, and generating at least one index temporary file based on the multiple data rows, wherein the index temporary file contains index rows of the multiple data rows.

[0038] For example, multiple data rows may have sequence numbers. In this step, the multiple data rows and the sequence numbers of the data rows may be written into a temporary file, and at least one index temporary file may be generated based on all the data rows in the temporary file, wherein the index rows in the index temporary file have the sequence numbers of the corresponding data rows.

[0039] For example, after parsing multiple data rows into a format for internal storage in a database, the data rows are written into a temporary file together with row sequence numbers to obtain a temporary file having a sequence number field.

[0040] For example, an index temporary file can be generated as a temporary file with a serial number field according to the index definition of the data table, and the index temporary file also has a serial number field. If the data table has one index, an index temporary file is generated according to the definition of the index; if the data table has multiple indexes, a corresponding index temporary file is generated according to the definition of each index, that is, multiple index temporary files are generated in total.

[0041] Next, please refer to the attached Figure 4 , determining, based on the temporary file and the data table, whether each of the multiple data rows has a uniqueness conflict with a data row already included in the data table.

[0042] For example, before determining whether each of the multiple data rows has a uniqueness conflict with a data row already included in the data table, the data rows in the temporary file may be sorted based on the unique key of the data table to improve the efficiency of uniqueness conflict detection.

[0043] Finally, please refer to the attached Figure 4 For each temporary index file, according to the corresponding index of the temporary index file and the data table, determine whether each index row in the temporary index file has a uniqueness conflict with the index row in the corresponding index of the data table.

[0044] For example, before determining whether each index row in the index temporary file has a uniqueness conflict with the index row in the corresponding index of the data table, the index rows in the index temporary file can be sorted based on the index field of the corresponding index of the data table to improve the efficiency of uniqueness conflict detection.

[0045] Preferably, based on the above example, after determining whether each of the multiple data rows has a uniqueness conflict with the data table, and whether the index row corresponding to each of the multiple data rows has a uniqueness conflict with the index of the data table, a conflict record can be generated based on the serial number of the data row that has a uniqueness conflict with the data table, and the serial number in the index row that has a uniqueness conflict with the index of the data table, wherein the conflict record records the serial number that has a uniqueness conflict with the index of the data table and / or the index table; and based on the temporary file and the conflict record, at least one non-conflict data row and / or at least one conflicting data row is determined in the temporary file.

[0046] Among them, if any data row has no uniqueness conflict with the data table, and the index row corresponding to the data row has no uniqueness conflict with the index of the data table, then the data row is a non-conflicting data row; if any data row has a uniqueness conflict with the data table, or the index row corresponding to the data row has a uniqueness conflict with the index of the data table, then the data row is a conflicting data row.

[0047] For example, if a data row has a uniqueness conflict with the data table, the serial number of the data row is added to the conflict record of the data table; for each index temporary file, if an index row in the index temporary file has a uniqueness conflict with the corresponding index, the serial number of the index row is added to the conflict record of the index temporary file; finally, the conflict record of the data table and the conflict record of each index temporary file are unioned, and the union is used as the conflict record in the above steps.

[0048] For another example, the sequence number of at least one conflicting data row may be written into the conflict query table to facilitate querying which data rows are conflicting data rows.

[0049] In step S102, all conflict-free data rows among the multiple data rows are written into the data table in the persistent storage device, and index rows corresponding to all conflict-free data rows among the multiple data rows are written into the index of the data table in the persistent storage device, wherein, if any data row does not have a uniqueness conflict with the data table, and the index row corresponding to the data row does not have a uniqueness conflict with the index of the data table, then the data row is a conflict-free data row.

[0050] Please refer to the attached Figure 5In this step, all conflict-free data rows and their corresponding index rows are directly written to the persistent storage device Disk to form the ordered string table SSTable in the bottom level 0; the ordered string table SSTable of the bottom level 0 will be gradually merged and compacted into higher layers, such as the ordered string table SSTable of level 1, level 2, etc. Writing directly to the persistent storage device can avoid the time-consuming dumping of memory to the persistent storage device, thereby improving the writing efficiency of large batches of data.

[0051] Please refer to the attached Figure 6 When writing all conflict-free data rows, the conflict query table can be queried to determine whether each data row in the temporary file is a conflict-free data row or a conflicting data row, and the data row determined to be a conflict-free data row is written into the persistent storage device.

[0052] Please refer to the attached Figure 7 When writing the index rows corresponding to all conflict-free data rows, the conflict query table can be queried to determine whether each index row in the temporary file corresponds to an index row corresponding to a conflict-free data row or an index row corresponding to a conflicting data row, and the index row determined to be an index row corresponding to a conflict-free data row is written to the persistent storage device.

[0053] In step S103, all conflicting data rows among the multiple data rows are written into the data table in the memory, and index rows corresponding to all conflicting data among the multiple data rows are written into the index of the data table in the memory, wherein, if any data row has a uniqueness conflict with the data table, or the index row corresponding to the data row has a uniqueness conflict with the index of the data table, then the data row is a conflicting data row.

[0054] Please refer to the attached Figure 8 In this step, when writing the conflicting data row to the data table in memory, the corresponding index row is generated and written into the index of the data table. For example, the conflicting data row and its corresponding index row are written into the memory in a normal writing method of DML (Data Manipulation Language).

[0055] Since the inserted conflicting data row has a uniqueness conflict with the data table or its corresponding index row has a uniqueness conflict with the index of the data table, if it is directly inserted, the data table or its index will not satisfy the uniqueness constraint. Therefore, uniqueness conflict processing is performed in the process of writing all conflicting data rows and index rows. For example, all conflicting data rows in the multiple data rows are written into the data table in the memory according to the replace into mode or the ignore into mode, and the index rows corresponding to all conflicting data in the multiple data rows are written into the index of the data table in the memory.

[0056] When the conflicting data row is written in replace into mode, the corresponding original data row in the data table is deleted and the conflicting data row is written, and the corresponding index in the index of the data table is deleted and the index row generated for the conflicting data row is written.

[0057] If the conflicting data row is written using the ignore into method, the writing is abandoned.

[0058] There may also be uniqueness conflicts between conflicting data rows, for example, there is a uniqueness conflict between the unique keys of two or more conflicting data rows and / or there is a uniqueness conflict between the index fields of two or more conflicting data rows. Therefore, the writing order of the conflicting data rows will affect the result of the uniqueness conflict processing to a certain extent. Based on this, all conflicting data rows in the temporary file can be sorted according to the sequence number before executing this step. Because the sequence number represents the writing order between multiple data rows written in batches, writing the conflicting data rows according to the sequence number can ensure that the initial writing requirements are met, thereby avoiding the execution of this method disrupting the writing order of the conflicting data rows, thereby ensuring the accuracy of the writing.

[0059] Next, a specific example is taken to introduce in more detail the process of writing batch data into a database obtained in combination with the above embodiments.

[0060] Data sheet as attached Fig. 9 As shown in the figure, c1 is the primary key and unique key; c2 is the index field. The data table and its index in the database are shown in the attached figure. Fig.10 The data that users need to import is shown in the attached Fig.11 As shown, the row number is the serial number of the data row.

[0061] First, generate the attached Fig.12 The data temporary file and index temporary file shown, the data rows in the data temporary file are arranged according to row numbers and the row numbers are recorded in the data rows, and the index rows in the index temporary file are arranged according to row numbers and the row numbers are recorded in the index rows.

[0062] Next, sort the data temporary files and index temporary files according to the unique key and get the attached Fig.13 Results shown.

[0063] Next, please refer to the attached Fig.14 Based on the data table, a uniqueness check is performed on the data rows in the temporary data file to obtain a conflict record of the temporary data file, which records the row number of the data row in the temporary data file that has a uniqueness conflict with the data table. Based on the index table, a uniqueness check is performed on the index rows in the temporary index file to obtain a conflict record of the temporary index file, which records the row number of the index row in the temporary index file that has a uniqueness conflict with the index table.

[0064] Next, please refer to the attached Fig.15 , take the union of the conflict records of the data temporary file and the conflict records of the index temporary file to obtain the conflict query table.

[0065] Next, please refer to the attached Fig.16 Based on the conflict query table, the data rows in the data temporary file are divided into non-conflict data rows and conflict data rows. The non-conflict data rows are written to the persistent storage device to form SSTable, and the conflict data rows are recorded in the conflict data.

[0066] Next, please refer to the attached Fig.17 , based on the conflict query table, query the index row corresponding to the conflict-free data row in the index temporary file, and write the query result to the persistent storage device to form an SSTable.

[0067] Finally, the conflicting data rows in the conflicting data are sorted according to the row numbers and written into the memory in a replacement insertion or ignored insertion manner.

[0068] The data management method provided in the embodiments of the present specification adopts a method of directly writing data rows that have no uniqueness conflicts with the data table and its index into a persistent storage device, thereby ensuring data insertion efficiency, and adopts a method of memory writing for data that has uniqueness conflicts with the data table or its index, thereby resolving uniqueness conflicts in the memory writing method to ensure that the written data has no uniqueness conflicts, so that when the method inserts data into a data table when the data table and its index both have uniqueness constraints, both insertion efficiency and quality are taken into account.

[0069] Fig.18 is a schematic structural diagram of a device provided by an exemplary embodiment. Fig.18At the hardware level, the device includes a processor 1802, an internal bus 1804, a network interface 1806, a memory 1808, and a non-volatile memory 1810, and may also include hardware required for other tasks. One or more embodiments of this specification may be implemented based on software, such as the processor 1802 reading the corresponding computer program from the non-volatile memory 1810 into the memory 1808 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0070] Please refer to Fig.19 , the data management device can be used for Fig.18 The data management device may include: A determination module 1901 is used to receive an instruction for instructing to write multiple data rows into a data table, and if the data table and the index of the data table both have uniqueness constraints, determine whether each of the multiple data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the multiple data rows has a uniqueness conflict with the index of the data table; The first writing module 1902 is used to write all conflict-free data rows among the multiple data rows into the data table in the persistent storage device, and write index rows corresponding to all conflict-free data rows among the multiple data rows into the index of the data table in the persistent storage device, wherein if any data row does not have a uniqueness conflict with the data table, and the index row corresponding to the data row does not have a uniqueness conflict with the index of the data table, then the data row is a conflict-free data row; The second writing module 1903 is used to write all conflicting data rows among the multiple data rows into the data table in the memory, and write index rows corresponding to all conflicting data among the multiple data rows into the index of the data table in the memory, wherein if any data row has a uniqueness conflict with the data table, or the index row corresponding to the data row has a uniqueness conflict with the index of the data table, then the data row is a conflicting data row.

[0071] In a possible embodiment of this specification, the determining module is used to: Writing the multiple data rows into a temporary file, and generating at least one index temporary file based on the multiple data rows, wherein the index temporary file contains index rows of the multiple data rows; Determine, based on the temporary file and the data table, whether each of the multiple data rows has a uniqueness conflict with a data row already included in the data table; For each temporary index file, according to the corresponding indexes of the temporary index file and the data table, it is determined whether each index row in the temporary index file has a uniqueness conflict with an index row in the corresponding index of the data table.

[0072] In a possible embodiment of the present specification, when the determining module is used to write the multiple data rows into a temporary file and generate at least one index temporary file based on the multiple data rows, it is used to: The multiple data rows and the serial numbers of the data rows are written into a temporary file, and at least one index temporary file is generated based on all the data rows in the temporary file, wherein the index rows in the index temporary file have the serial numbers of the corresponding data rows.

[0073] In a possible embodiment of the present specification, the device further includes a partitioning module, which is used to: Generate a conflict record based on the serial number of the data row that has a uniqueness conflict with the data table and the serial number in the index row that has a uniqueness conflict with the index of the data table, wherein the conflict record records the serial number that has a uniqueness conflict with the index of the data table and / or the index table; Based on the temporary file and the conflict record, at least one conflict-free data row and / or at least one conflicting data row is determined in the temporary file.

[0074] In a possible embodiment of the present specification, the device further includes a sorting module, which is used to: Before determining, based on the temporary file and the data table, whether each of the plurality of data rows has a uniqueness conflict with a data row already included in the data table, sorting the data rows in the temporary file based on the unique key of the data table; Before determining whether each index row in the index temporary file has a uniqueness conflict with an index row in a corresponding index of the data table based on the corresponding index of the index temporary file and the data table, the index rows in the index temporary file are sorted based on the index field of the corresponding index of the data table.

[0075] In a possible embodiment of this specification, the sorting module is further used for: Before writing all conflicting data rows in the multiple data rows into the data table in the memory and writing index rows corresponding to all conflicting data in the multiple data rows into the index of the data table in the memory, all conflicting data rows in the temporary file are sorted according to serial numbers.

[0076] In a possible embodiment of the present specification, the device further includes a conflict handling module, which is used to: Uniqueness conflict handling is performed during the writing of all conflicting data rows and index rows.

[0077] In a possible embodiment of this specification, the second writing module is used for: All conflicting data rows in the multiple data rows are written into the data table in the memory according to the replacement insertion mode or the ignore insertion mode, and index rows corresponding to all conflicting data rows in the multiple data rows are written into the index of the data table in the memory.

[0078] In a possible embodiment of this specification, the determining module is used to: An instruction for instructing to write multiple data rows into a data table is received. If the data table and the index of the data table both have uniqueness constraints, and the number of the multiple data rows is greater than a preset threshold, it is determined whether each of the multiple data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the multiple data rows has a uniqueness conflict with the index of the data table.

[0079] One or more embodiments of the present specification also propose a computer program product, including a computer program / instruction, which implements the steps of the method provided in the first aspect when the computer program / instruction is executed by a processor.

[0080] One or more embodiments of the present specification also propose a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0081] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0082] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0083] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0084] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0085] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0086] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0087] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0088] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0089] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0090] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A data management method, the method comprising: receiving an instruction for instructing to write a plurality of data rows into a data table, and if the data table and the index of the data table both have a uniqueness constraint, determining whether each of the plurality of data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the plurality of data rows has a uniqueness conflict with the index of the data table; Writing all conflict-free data rows among the multiple data rows into the data table in the persistent storage device, and writing index rows corresponding to all conflict-free data rows among the multiple data rows into the index of the data table in the persistent storage device, wherein if any data row does not have a uniqueness conflict with the data table, and the index row corresponding to the data row does not have a uniqueness conflict with the index of the data table, then the data row is a conflict-free data row; All conflicting data rows among the multiple data rows are written into the data table in the memory, and index rows corresponding to all conflicting data among the multiple data rows are written into the index of the data table in the memory, wherein if any data row has a uniqueness conflict with the data table, or if the index row corresponding to the data row has a uniqueness conflict with the index of the data table, then the data row is a conflicting data row.

2. The data management method according to claim 1, wherein determining whether each of the plurality of data rows has a uniqueness conflict with the data table, and determining whether an index row corresponding to each of the plurality of data rows has a uniqueness conflict with an index of the data table, comprises: Writing the multiple data rows into a temporary file, and generating at least one index temporary file based on the multiple data rows, wherein the index temporary file contains index rows of the multiple data rows; Determine, based on the temporary file and the data table, whether each of the multiple data rows has a uniqueness conflict with a data row already included in the data table; For each temporary index file, according to the corresponding indexes of the temporary index file and the data table, it is determined whether each index row in the temporary index file has a uniqueness conflict with an index row in the corresponding index of the data table.

3. The data management method according to claim 2, wherein the step of writing the plurality of data rows into a temporary file and generating at least one index temporary file based on the plurality of data rows comprises: The multiple data rows and the serial numbers of the data rows are written into a temporary file, and at least one index temporary file is generated based on all the data rows in the temporary file, wherein the index rows in the index temporary file have the serial numbers of the corresponding data rows.

4. The data management method according to claim 3, further comprising: Generate a conflict record based on the serial number of the data row that has a uniqueness conflict with the data table and the serial number in the index row that has a uniqueness conflict with the index of the data table, wherein the conflict record records the serial number that has a uniqueness conflict with the index of the data table and / or the index table; Based on the temporary file and the conflict record, at least one conflict-free data row and / or at least one conflicting data row is determined in the temporary file.

5. The data management method according to claim 3, before determining, based on the temporary file and the data table, whether each of the plurality of data rows has a uniqueness conflict with a data row already included in the data table, the method further comprises: Sort the data rows in the temporary file based on the unique key of the data table; Before determining, based on the corresponding indexes of the temporary index file and the data table, whether each index row in the temporary index file has a uniqueness conflict with an index row in a corresponding index of the data table, the method further includes: The index rows in the index temporary file are sorted based on the index fields of the corresponding indexes of the data table.

6. The data management method according to claim 5, before writing all conflicting data rows in the plurality of data rows into the data table in the memory and writing index rows corresponding to all conflicting data rows in the plurality of data rows into the index of the data table in the memory, the method further comprises: All conflicting data rows in the temporary file are sorted according to the sequence numbers.

7. The data management method according to claim 1, further comprising: Uniqueness conflict handling is performed during the writing of all conflicting data rows and index rows.

8. The data management method according to claim 7, wherein the step of writing all conflicting data rows in the plurality of data rows into the data table in the memory, and writing index rows corresponding to all conflicting data rows in the plurality of data rows into the index of the data table in the memory, and performing uniqueness conflict processing in the process of writing all conflicting data rows and index rows, comprises: All conflicting data rows in the multiple data rows are written into the data table in the memory according to the replacement insertion mode or the ignore insertion mode, and index rows corresponding to all conflicting data rows in the multiple data rows are written into the index of the data table in the memory.

9. The data management method according to claim 1, wherein receiving an instruction for instructing to write a plurality of data rows into a data table, and if both the data table and the index of the data table have a uniqueness constraint, determining whether each of the plurality of data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the plurality of data rows has a uniqueness conflict with the index of the data table, comprises: An instruction for instructing to write multiple data rows into a data table is received. If the data table and the index of the data table both have uniqueness constraints, and the number of the multiple data rows is greater than a preset threshold, it is determined whether each of the multiple data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the multiple data rows has a uniqueness conflict with the index of the data table.

10. A data management device, comprising: a determination module, configured to receive an instruction for instructing to write a plurality of data rows into a data table, and if the data table and the index of the data table both have a uniqueness constraint, determine whether each of the plurality of data rows has a uniqueness conflict with the data table, and whether an index row corresponding to each of the plurality of data rows has a uniqueness conflict with the index of the data table; A first writing module is used to write all conflict-free data rows among the multiple data rows into the data table in the persistent storage device, and write index rows corresponding to all conflict-free data rows among the multiple data rows into the index of the data table in the persistent storage device, wherein if any data row does not have a uniqueness conflict with the data table, and the index row corresponding to the data row does not have a uniqueness conflict with the index of the data table, then the data row is a conflict-free data row; The second writing module is used to write all conflicting data rows among the multiple data rows into the data table in the memory, and write index rows corresponding to all conflicting data among the multiple data rows into the index of the data table in the memory, wherein if any data row has a uniqueness conflict with the data table, or the index row corresponding to the data row has a uniqueness conflict with the index of the data table, then the data row is a conflicting data row.

11. The data management device according to claim 10, wherein the determining module is used for: The plurality of data rows are written into a temporary file, and at least one index temporary file is generated based on the plurality of data rows, wherein: The index temporary file contains index rows of the multiple data rows; Determine, based on the temporary file and the data table, whether each of the multiple data rows has a uniqueness conflict with a data row already included in the data table; For each temporary index file, according to the corresponding indexes of the temporary index file and the data table, it is determined whether each index row in the temporary index file has a uniqueness conflict with an index row in the corresponding index of the data table.

12. The data management device according to claim 11, wherein when the determining module is used to write the plurality of data rows into a temporary file and generate at least one index temporary file based on the plurality of data rows, it is used to: The plurality of data rows and the sequence numbers of the data rows are written into a temporary file, and at least one index temporary file is generated based on all the data rows in the temporary file, wherein: The index rows in the index temporary file have the serial numbers of the corresponding data rows.

13. The data management device according to claim 12, further comprising a partitioning module, configured to: Based on the serial number of the data row that has a uniqueness conflict with the data table and the serial number in the index row that has a uniqueness conflict with the index of the data table, a conflict record is generated, wherein: The conflict record records the serial numbers that have uniqueness conflicts with the indexes of the data table and / or the index table; Based on the temporary file and the conflict record, at least one conflict-free data row and / or at least one conflicting data row is determined in the temporary file.

14. The data management device according to claim 10, wherein the second writing module is used for: All conflicting data rows in the multiple data rows are written into the data table in the memory according to the replacement insertion mode or the ignore insertion mode, and index rows corresponding to all conflicting data rows in the multiple data rows are written into the index of the data table in the memory.

15. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

16. An electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

17. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images