Multi-source data storage method and system

Abstract

The application provides a multi-source data storage method and system, and relates to the technical field of data processing. The method comprises the following steps: pre-processing multi-source data based on a checking system to obtain first data, and storing the multi-source data in a to-be-checked area of the checking system; storing the first data in each table in the checking area of the checking system, and merging repeated data in the first data based on the checking system to obtain second data; and storing the second data in a target database. The application uses a unified data format to quickly store data. A multi-batch virtual temporary table rotation mechanism is used to give all virtual space users the ability to efficiently use the database, which greatly improves the speed of database construction. A multi-level and multi-layer virtual temporary table screening and checking mechanism ensures the reliability, consistency and security of the data. The safe data obtained through multi-level screening is further divided and stored according to categories, which improves the overall utilization efficiency of the database and ensures the convenience of data access.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multi-source data storage method and system. Background Technology

[0002] Traditional database construction involves a dedicated database administrator performing CRUD operations (creating, deleting, modifying, and querying) on ​​database tables using limited data. The administrator first analyzes and processes the data to be stored, creating appropriate databases and tables based on the data structure and scale. Data is then stored in the database either manually or through batch imports. The data stored in the database must be maintained and managed by a dedicated database administrator.

[0003] Traditional database construction methods are limited by manual storage methods, resulting in many problems such as slow database construction, low database construction efficiency, high error rate of manual construction, and high management costs. Summary of the Invention

[0004] The multi-source data storage method and system provided by this invention are used to solve the problem of high database construction error rate caused by the limitation of manual storage methods in the prior art.

[0005] This invention provides a multi-source data storage method, comprising:

[0006] The verification system preprocesses the multi-source data to obtain the first data, and the multi-source data is stored in the verification area of ​​the verification system.

[0007] The first data is stored in various tables within the verification area of ​​the verification system, and duplicate data in the first data is merged based on the verification system to obtain the second data;

[0008] The second data is stored in the target database.

[0009] According to a multi-source data storage method provided by the present invention, the first data is obtained by preprocessing the multi-source data based on a verification system, including:

[0010] Based on the verification system, any target data from the multi-source data stored in the area to be verified is preprocessed as follows to obtain the first data:

[0011] Determine whether the data format in the source data type declaration field of the target data is consistent with the data format in the source data field of the target data;

[0012] If there is a discrepancy, the target data shall be discarded.

[0013] If they match, then determine whether the target data is complete based on the source data fields of the target data;

[0014] If incomplete, the target data is discarded;

[0015] If complete, an identifier is added to the target data to obtain the first data.

[0016] According to a multi-source data storage method provided by the present invention, the step of merging duplicate data in the first data based on the verification system to obtain the second data includes:

[0017] If the sum of the number of tables in the verification area and the number of new tables is greater than or equal to two, the tables in the verification area are paired up in pairs based on the verification system to obtain each paired table;

[0018] For any pairing table, perform the following merge procedure:

[0019] If duplicate data exists in any of the tables included in the pairing table, merge the source field of the duplicate data and modify the timestamp field of the duplicate data to obtain third data. The duplicate data is the first data whose source data type declaration field and source data field are consistent.

[0020] The third data is stored in the pairing table, which includes the duplicate data, and the duplicate data is deleted.

[0021] If there is no duplicate data in any of the tables included in the pairing table, but there is duplicate data between the two tables included in the pairing table, the source field of the duplicate data is merged, and the timestamp field of the duplicate data is modified to obtain the fourth data.

[0022] The fourth data is stored in any of the tables included in the pairing table, and the duplicate data is deleted.

[0023] Store the remaining data from the two tables included in the pairing table into the new table. The remaining data is the data from the two tables included in the pairing table, excluding the duplicate data.

[0024] If the sum of the number of tables in the verification area and the number of new tables is less than two, the second data is obtained based on the data stored in the new table after the last merge process.

[0025] According to a multi-source data storage method provided by the present invention, the method for obtaining the multi-source data stored in the area to be verified includes:

[0026] According to a unified data format, the multi-source data is stored in one or more virtual temporary tables in a rotation area, where the rotation area is a virtual storage space used to store the multi-source data;

[0027] Store the virtual temporary table in the area to be verified.

[0028] According to a multi-source data storage method provided by the present invention, the unified data format includes at least the following fields:

[0029] Record length field, source data type declaration field, traceability field, timestamp field, and source data field.

[0030] According to a multi-source data storage method provided by the present invention, when the data format in the source data type declaration field of the target data is inconsistent with the data format in the source data field of the target data, after removing the target data, the method further includes:

[0031] The user is given a first feedback prompt, which includes information about data inconsistency and storage failure.

[0032] Accordingly, if the target data is incomplete, after removing the target data, the method further includes:

[0033] A second prompt is sent to the user, which includes information about incomplete data and storage failure.

[0034] The user is determined based on the traceability field of the target data.

[0035] According to a multi-source data storage method provided by the present invention, storing the second data in a target database includes:

[0036] The data type of the second data is determined based on the fields declared in the source data type declaration of the second data.

[0037] According to the data type, the second data is stored in the target database, and the data type stored in the target database is the same as the data type of the second data.

[0038] The present invention also provides a multi-source data storage system, comprising: a data processing module, a data acquisition module, and a data storage module;

[0039] The data processing module is used to preprocess multi-source data based on the verification system to obtain first data, and the multi-source data is stored in the verification area of ​​the verification system;

[0040] The data acquisition module is used to store the first data into various tables in the verification area of ​​the verification system, and to obtain the second data by merging duplicate data in the first data based on the verification system.

[0041] The data storage module is used to store the second data into the target database.

[0042] The present invention also provides an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the multi-source data storage method described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multi-source data storage method described above.

[0044] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the multi-source data storage methods described above.

[0045] The multi-source data storage method and system provided by this invention utilizes a verification system to preprocess the data to be stored (i.e., multi-source data), removing abnormal data to obtain first data. By storing the first data in the verification area of ​​the verification system, duplicate data in the first data is merged to obtain second data. After multiple verifications of the multi-source data by the verification system, abnormal and duplicate data in the multi-source data are cleaned up, and finally, safe and correct second data is obtained. By storing the second data in the target database, the accuracy of the constructed database is improved. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the multi-source data storage method provided by the present invention;

[0048] Figure 2 This is a schematic diagram of the data verification process within the area to be verified provided by the present invention;

[0049] Figure 3 This is an example diagram of the unified data format provided by the present invention;

[0050] Figure 4This is a schematic diagram of the data merging process in the verification area of ​​the verification system provided by the present invention;

[0051] Figure 5 This is a schematic diagram of the database construction process provided by the present invention;

[0052] Figure 6 This is a schematic diagram of the structure of the multi-source data storage system provided by the present invention;

[0053] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] The multi-source data storage method provided by this invention standardizes all user data (such as industry-specific digital software repair data and repair scenario environment data) in a unified data format within a virtual space. Users in the virtual space can quickly store their data into one or more dedicated virtual temporary tables in the current rotation area. A batch of stored virtual temporary tables undergoes detailed screening to remove all duplicate and corrupted data, and the storage user is identified and notified of data storage failure based on the traceability field in the data. The remaining data forms physical tables, which are then sharded and stored according to data type and relevant rules, completing the construction of industry-specific digital software repair databases and repair scenario environment databases. The specific implementation is as follows:

[0056] Figure 1 This is a flowchart illustrating the multi-source data storage method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0057] Step 110: Preprocess the multi-source data based on the verification system to obtain the first data, wherein the multi-source data is stored in the verification area of ​​the verification system;

[0058] Step 120: Store the first data in various tables within the verification area of ​​the verification system, and merge duplicate data in the first data based on the verification system to obtain the second data;

[0059] Step 130: Store the second data in the target database.

[0060] It should be noted that the above method can be implemented by computer equipment.

[0061] Optionally, the verification system may specifically include a verification area and a verification area. The verification area stores multi-source data to be stored. The verification system creates multiple tables in the verification area. The tables in the verification area are used to store the first data obtained after preprocessing the multi-source data. The multi-source data may specifically include digital software maintenance data from various industries and maintenance scenario environment data, which may be provided by users.

[0062] This verification system can be specifically used to preprocess multi-source data provided by users, removing abnormal / bad data from the multi-source data to obtain preprocessed data, i.e., the first data.

[0063] The verification system can also be used to verify the first data in each table stored in the verification area, merge the duplicate data in the first data, and obtain the merged data, i.e., the second data.

[0064] Based on the verification system, the multi-source data provided by the user is preprocessed to obtain the first data, and the obtained first data is stored in various tables in the verification area of ​​the verification system. Based on the verification system, the duplicate data in the first data stored in the various tables in the verification area are merged to obtain the second data.

[0065] The second data is stored in the target database, thus completing the construction of the target database.

[0066] For example, when the multi-source data consists of digital software maintenance data and maintenance scenario environment data from various industries, the corresponding second data is obtained after the verification system verifies the digital software maintenance data and maintenance scenario environment data of each industry. The second data is then stored in the target database to complete the construction of the digital software maintenance database and maintenance scenario environment database for each industry.

[0067] The multi-source data storage method provided by this invention utilizes a verification system to preprocess the data to be stored (i.e., multi-source data), removing abnormal data to obtain first data. This first data is then stored in the verification area of ​​the verification system to merge duplicate data, resulting in second data. After multiple verifications of the multi-source data by the verification system, abnormal and duplicate data are removed, ultimately yielding safe and correct second data. This second data is then stored in the target database, improving the accuracy of the constructed database.

[0068] Furthermore, in one embodiment, the preprocessing of multi-source data based on the verification system to obtain the first data may specifically include:

[0069] Based on the verification system, any target data from the multi-source data stored in the area to be verified is preprocessed as follows to obtain the first data:

[0070] Determine whether the data format in the source data type declaration field of the target data is consistent with the data format in the source data field of the target data;

[0071] If there is a discrepancy, the target data shall be discarded.

[0072] If they match, then determine whether the target data is complete based on the source data fields of the target data;

[0073] If incomplete, the target data is discarded;

[0074] If complete, an identifier is added to the target data to obtain the first data.

[0075] Optionally, Figure 2 This is a schematic diagram of the data verification process within the area to be verified provided by the present invention, such as... Figure 2 As shown, the multi-source data sent to the verification system is not directly stored in the verification system, but is first transferred to the verification area of ​​the verification system for unified verification. This operation can, on the one hand, utilize the large space of the verification system to increase the data storage capacity of the entire database system, and on the other hand, avoid the slow data verification process from occupying valuable working time and space in the rotation area, thereby increasing the instantaneous receiving capacity of the entire database system.

[0076] The verification process will perform the following preprocessing on each piece of data (i.e. target data) from the multi-source data stored in the area to be verified, to obtain the first data.

[0077] A rigorous check is performed based on two conditions: whether the source data declaration field of the target data is consistent with the actual data format stored in the source data field, and whether the target data stored in the source data field maintains integrity. If either condition is not met, the data will not be stored in the verification area of ​​the verification system. Specifically:

[0078] Determine whether the data format in the source data type declaration field of the target data is consistent with the data format in the source data field of the target data. If they are inconsistent, it means that the target data is incorrect and the target data is removed.

[0079] If they match, determine whether the target data is complete based on the source data fields. If it is incomplete, it means that the target data is incorrect and the target data is removed.

[0080] The system identifies the user who stored the target data based on the traceability field in the target data, notifies them that the target data storage has failed, and ensures the consistency, stability, security, and reliability of database storage.

[0081] If the data format in the source data type declaration field of the target data is consistent with the data format in the source data field of the target data, and the target data is complete, then a number will be added to the target data record length field to obtain the first data, and this first data will be stored in the verification area to await the next step of merging processing. The verification process will stop when all the first data in the verification area of ​​the verification system has been verified.

[0082] Furthermore, in one embodiment, the method for obtaining the multi-source data stored in the area to be verified may specifically include:

[0083] According to a unified data format, the multi-source data is stored in one or more virtual temporary tables in a rotation area, where the rotation area is a virtual storage space used to store the multi-source data;

[0084] Store the virtual temporary table in the area to be verified.

[0085] Optionally, the multi-source data sent by the user to the database is quickly converted into a unified data format and stored in one or more virtual temporary tables in the current rotation area. If the current rotation area is exhausted by the multi-source data sent by the user to the database, each virtual temporary table in the current rotation area is sent to the verification area of ​​the verification system for storage. Then, an empty rotation area is switched as the new current rotation area to continue receiving and storing multi-source data.

[0086] Furthermore, in one embodiment, the unified data format includes at least the following fields:

[0087] Record length field, source data type declaration field, traceability field, timestamp field, and source data field.

[0088] Optionally, Figure 3 This is an example diagram of the unified data format provided by the present invention, such as... Figure 3 As shown, this unified data format includes at least a record length field, a source data type declaration field, a traceability field, a timestamp field, and a source data field. Specifically, the record length field (1 byte) stores the memory space occupied by the current record; the source data type declaration field (1 byte) stores the data format declared by the data source when sending multi-source data; the traceability field (32 bytes) stores the client credentials sent for the current record; the timestamp field (8 bytes) records the database system time; and the source data field (1~10 bytes)... 8-1 byte) specifically stores multi-source data sent by the data source.

[0089] The multi-source data storage method provided by this invention standardizes multi-source data to unify the data format, uses the unified data format to achieve fast storage of multi-source data, creates a virtual temporary table rotation mechanism, efficiently utilizes the high-performance storage space of the storage service server, improves the overall data receiving capability of the database, gives all users the ability to access the database at high frequency, improves the overall concurrency performance of the database system, and increases the speed of database construction.

[0090] Furthermore, in one embodiment, the step of merging duplicate data in the first data based on the verification system to obtain the second data may specifically include:

[0091] If the sum of the number of tables in the verification area and the number of new tables is greater than or equal to two, the tables in the verification area are paired up in pairs based on the verification system to obtain each paired table;

[0092] For any pairing table, perform the following merge procedure:

[0093] If duplicate data exists in any of the tables included in the pairing table, merge the source field of the duplicate data and modify the timestamp field of the duplicate data to obtain third data. The duplicate data is the first data whose source data type declaration field and source data field are consistent.

[0094] The third data is stored in the pairing table, which includes the duplicate data, and the duplicate data is deleted.

[0095] If there is no duplicate data in any of the tables included in the pairing table, but there is duplicate data between the two tables included in the pairing table, the source field of the duplicate data is merged, and the timestamp field of the duplicate data is modified to obtain the fourth data.

[0096] The fourth data is stored in any of the tables included in the pairing table, and the duplicate data is deleted.

[0097] Store the remaining data from the two tables included in the pairing table into the new table. The remaining data is the data from the two tables included in the pairing table, excluding the duplicate data.

[0098] If the sum of the number of tables in the verification area and the number of new tables is less than two, the second data is obtained based on the data stored in the new table after the last merge process.

[0099] Optionally, Figure 4 This is a schematic diagram of the data merging process in the verification area of ​​the verification system provided by the present invention, as shown below. Figure 4 As shown, after the merging process begins, it is determined whether the sum of the number of tables in the verification area of ​​the verification system and the number of new tables is greater than or equal to two. If not, there are no matching tables, and the merging ends. If so, there are matching tables, and the tables in the verification area are paired up. When the total number of tables is odd during pairing, the remaining tables directly flow into the next round of pairing. The process iterates through each successfully paired table, treating it as the pairing table. First, it checks for duplicate data in both tables within the pairing table. If duplicate data (the first data where both the source data type declaration fields and the source data fields are identical) is found in either table, the source fields of two or more duplicate data entries from both tables are merged using bitwise operations to create the source field for the new data (the third data). The data with the smaller timestamp value from the duplicate data is used as the timestamp value for the third data, while other fields remain unchanged. The merged third data is then stored back in the original table within the pairing table, and duplicate data in the original table is deleted. Next, it checks for duplicate data between the paired tables. If no duplicate data exists within either table within the pairing table, but duplicate data exists between the paired tables, the same duplicate data merging operation is performed to obtain the fourth data. However, only the duplicate data is stored back in one of the tables, and the duplicate data between the paired tables is deleted. Once the duplicate data checks within and between paired tables are complete, all remaining non-duplicate data (the remaining data) is merged into a new table, which is then used in the next round of pairing. The space of the original table is then released. Then return to iterate through the next successfully paired table until the merge is complete.

[0100] The merging process continues until the sum of the number of tables in the verification area and the new table is less than two. At this point, there are no matching tables, and the merging ends. Based on the data stored in the new table after the last merging process, the final second data is obtained. The second data obtained after the end of the duplicate data merging process is all safe data that conforms to the rules. This part of the data will be stably stored in the database system to facilitate a series of subsequent database operations such as automatic database management and reconstruction.

[0101] The multi-source data storage method provided by this invention, based on the multi-level verification methods of the verification system, removes all bad data that does not meet the storage rules while the database system efficiently receives data, and merges duplicate data, thus ensuring the reliability, consistency and security of the data, and thereby ensuring that all data entering the system is safe and correct.

[0102] Furthermore, in one embodiment, when the data format in the source data type declaration field of the target data is inconsistent with the data format in the source data field of the target data, after removing the target data, the method may further include:

[0103] The user is given a first feedback prompt, which includes information about data inconsistency and storage failure.

[0104] Accordingly, if the target data is incomplete, after removing the target data, the method may further include:

[0105] A second prompt is sent to the user, which includes information about incomplete data and storage failure.

[0106] The user is determined based on the traceability field of the target data.

[0107] Optionally, if the data format in the source data type declaration field of the target data is inconsistent with the data format in the source data field of the target data, the target data is removed, and a first prompt is sent to the user who provided the target data. The first prompt may specifically include information about data inconsistency and storage failure.

[0108] This user can specifically determine the client credentials recorded in the traceability field of the target data.

[0109] When the target data is incomplete, the target data is removed, and a second prompt is sent to the user who provided the target data. The second prompt may specifically include information about incomplete data and storage failure.

[0110] Furthermore, in one embodiment, storing the second data in the target database may specifically include:

[0111] The data type of the second data is determined based on the fields declared in the source data type declaration of the second data.

[0112] According to the data type, the second data is stored in the target database, and the data type stored in the target database is the same as the data type of the second data.

[0113] Optionally, the second data can be classified according to the source data type declaration field of the second data according to the established rules under actual conditions to obtain the data type of the second data. For example, the second data can be divided into audio data, video data, model data, text data, etc. According to the data type of the second data, the second data of the same category can be stored in the same target database to facilitate the expansion, iteration, optimization and other work of the database. The data type stored in the target database is the same as the data type of the second data.

[0114] Figure 5 This is a schematic diagram of the database construction process provided by the present invention, such as... Figure 5As shown, after users in the virtual space send the multi-source data they want to store to the database, the first step of rapid database construction is completed, standardizing the multi-source data into a unified data format (specifically as follows). Figure 3 (As shown). The multi-source data sent by users to the database is quickly converted into a unified data format and then stored in one or more virtual temporary tables in the current rotation area. If the current rotation area is exhausted by the data sent by users to the database, the rotation area is sent to the verification system, and then an empty rotation area is switched to continue receiving and storing data. The data obtained after the verification system preprocesses the multi-source data and merges the data is all safe data that conforms to the rules. This part of the data will be stably stored in the database system (e.g., database 1, database 2, etc.).

[0115] Compared with existing technologies, this invention establishes a virtual temporary table area for rapid storage and reception of data from all users in the virtual space, quickly forming a one-to-one data storage structure. At the software level, this simulates the fast-response environment of using a dedicated database. A dedicated virtual temporary table rotation area utilizes high-speed communication storage space, fully leveraging the rapid advantages of multi-user construction. The rotated virtual temporary table area undergoes virtual temporary table verification, cleaning up duplicate and non-compliant bad data to ultimately form a secure and reliable physical table. Finally, the database is partitioned and sharded according to the data type and size of the stored data in the physical table, thus completing the rapid construction, detailed screening, and stable storage of the entire database.

[0116] The multi-source data storage method provided by this invention classifies and stores all secure and reliable data received by the database according to predetermined transfer rules, resulting in a well-structured and logically clear database structure, which facilitates database expansion, iteration, optimization, and other tasks.

[0117] The multi-source data storage system provided by the present invention is described below. The multi-source data storage system described below can be referred to in correspondence with the multi-source data storage method described above.

[0118] Figure 6 This is a schematic diagram of the structure of the multi-source data storage system provided by the present invention, as shown below. Figure 6 As shown, it includes:

[0119] Data processing module 610, data acquisition module 611, and data storage module 612;

[0120] The data processing module 610 is used to preprocess multi-source data based on the verification system to obtain first data, wherein the multi-source data is stored in the verification area of ​​the verification system;

[0121] The data acquisition module 611 is used to store the first data into various tables in the verification area of ​​the verification system, and to obtain the second data by merging duplicate data in the first data based on the verification system.

[0122] The data storage module 612 is used to store the second data into the target database.

[0123] The multi-source data storage system provided by this invention utilizes a verification system to preprocess the data to be stored (i.e., multi-source data), removing abnormal data to obtain first data. By storing the first data in the verification area of ​​the verification system, duplicate data in the first data is merged to obtain second data. After multiple verifications of the multi-source data by the verification system, abnormal and duplicate data in the multi-source data are cleaned up, and finally, safe and correct second data is obtained. By storing the second data in the target database, the accuracy of the constructed database is improved.

[0124] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a communication interface 711, a memory 712, and a bus 713. The processor 710, communication interface 711, and memory 712 communicate with each other via the bus 713. The processor 710 can call logical instructions from the memory 712 to execute the following methods:

[0125] The verification system preprocesses the multi-source data to obtain the first data, and the multi-source data is stored in the verification area of ​​the verification system.

[0126] The first data is stored in various tables within the verification area of ​​the verification system, and duplicate data in the first data is merged based on the verification system to obtain the second data;

[0127] The second data is stored in the target database.

[0128] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer power supply (which may be a personal computer, server, or network power supply, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] Furthermore, this invention discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when these instructions are executed by a computer, the computer can execute the multi-source data storage method provided in the above-described method embodiments, for example, including:

[0130] The verification system preprocesses the multi-source data to obtain the first data, and the multi-source data is stored in the verification area of ​​the verification system.

[0131] The first data is stored in various tables within the verification area of ​​the verification system, and duplicate data in the first data is merged based on the verification system to obtain the second data;

[0132] The second data is stored in the target database.

[0133] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the multi-source data storage methods provided in the above embodiments, for example including:

[0134] The verification system preprocesses the multi-source data to obtain the first data, and the multi-source data is stored in the verification area of ​​the verification system.

[0135] The first data is stored in various tables within the verification area of ​​the verification system, and duplicate data in the first data is merged based on the verification system to obtain the second data;

[0136] The second data is stored in the target database.

[0137] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer power supply (which may be a personal computer, server, or network power supply, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source data storage method, characterized in that, include: The verification system preprocesses the multi-source data to obtain the first data, and the multi-source data is stored in the verification area of ​​the verification system. The first data is stored in various tables within the verification area of ​​the verification system, and duplicate data in the first data is merged based on the verification system to obtain the second data; Store the second data in the target database; The step of merging duplicate data in the first data based on the verification system to obtain the second data includes: If the sum of the number of tables in the verification area and the number of new tables is greater than or equal to two, the tables in the verification area are paired up in pairs based on the verification system to obtain each paired table; For any pairing table, perform the following merge procedure: If duplicate data exists in any of the tables included in the pairing table, merge the source field of the duplicate data and modify the timestamp field of the duplicate data to obtain third data. The duplicate data is the first data whose source data type declaration field and source data field are consistent. The third data is stored in the pairing table, which includes the duplicate data, and the duplicate data is deleted. If there is no duplicate data in any of the tables included in the pairing table, but there is duplicate data between the two tables included in the pairing table, the source field of the duplicate data is merged, and the timestamp field of the duplicate data is modified to obtain the fourth data. The fourth data is stored in any of the tables included in the pairing table, and the duplicate data is deleted. Store the remaining data from the two tables included in the pairing table into the new table. The remaining data is the data from the two tables included in the pairing table, excluding the duplicate data. If the sum of the number of tables in the verification area and the number of new tables is less than two, the second data is obtained based on the data stored in the new table after the last merge process.

2. The multi-source data storage method according to claim 1, characterized in that, The first data, obtained by preprocessing multi-source data based on the verification system, includes: Based on the verification system, any target data from the multi-source data stored in the area to be verified is preprocessed as follows to obtain the first data: Determine whether the data format in the source data type declaration field of the target data is consistent with the data format in the source data field of the target data; If there is a discrepancy, the target data shall be discarded. If they match, then determine whether the target data is complete based on the source data fields of the target data; If incomplete, the target data is discarded; If complete, an identifier is added to the target data to obtain the first data.

3. The multi-source data storage method according to claim 2, characterized in that, The methods for obtaining the multi-source data stored in the area to be verified include: According to a unified data format, the multi-source data is stored in one or more virtual temporary tables in a rotation area, where the rotation area is a virtual storage space used to store the multi-source data; Store the virtual temporary table in the area to be verified.

4. The multi-source data storage method according to claim 3, characterized in that, The unified data format includes at least the following fields: Record length field, source data type declaration field, traceability field, timestamp field, and source data field.

5. The multi-source data storage method according to claim 2, characterized in that, If the data format in the source data type declaration field of the target data is inconsistent with the data format in the source data field of the target data, after removing the target data, the method further includes: The user is given a first feedback prompt, which includes information about data inconsistency and storage failure. Accordingly, if the target data is incomplete, after removing the target data, the method further includes: A second prompt is sent to the user, which includes information about incomplete data and storage failure. The user is determined based on the traceability field of the target data.

6. The multi-source data storage method according to any one of claims 1-5, characterized in that, The step of storing the second data in the target database includes: The data type of the second data is determined based on the fields declared in the source data type declaration of the second data. According to the data type, the second data is stored in the target database, and the data type stored in the target database is the same as the data type of the second data.

7. A multi-source data storage system, characterized in that, include: Data processing module, data acquisition module, and data storage module; The data processing module is used to preprocess multi-source data based on the verification system to obtain first data, and the multi-source data is stored in the verification area of ​​the verification system; The data acquisition module is used to store the first data into various tables in the verification area of ​​the verification system, and to obtain the second data by merging duplicate data in the first data based on the verification system. The data storage module is used to store the second data into the target database; Specifically, the data acquisition module is used for: If the sum of the number of tables in the verification area and the number of new tables is greater than or equal to two, the tables in the verification area are paired up in pairs based on the verification system to obtain each paired table; For any pairing table, perform the following merge procedure: If duplicate data exists in any of the tables included in the pairing table, merge the source field of the duplicate data and modify the timestamp field of the duplicate data to obtain third data. The duplicate data is the first data whose source data type declaration field and source data field are consistent. The third data is stored in the pairing table, which includes the duplicate data, and the duplicate data is deleted. If there is no duplicate data in any of the tables included in the pairing table, but there is duplicate data between the two tables included in the pairing table, the source field of the duplicate data is merged, and the timestamp field of the duplicate data is modified to obtain the fourth data. The fourth data is stored in any of the tables included in the pairing table, and the duplicate data is deleted. Store the remaining data from the two tables included in the pairing table into the new table. The remaining data is the data from the two tables included in the pairing table, excluding the duplicate data. If the sum of the number of tables in the verification area and the number of new tables is less than two, the second data is obtained based on the data stored in the new table after the last merge process.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the multi-source data storage method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multi-source data storage method as described in any one of claims 1 to 6.

Images