Data synchronization method, apparatus, device, and storage medium

Abstract

The present disclosure provides a data synchronization method, device and equipment and storage medium, which can be applied to the field of information security and financial technology. The method comprises: in response to a received data file, performing security detection on the data file based on a preset configuration table to obtain a security detection result; in the case that the security detection result indicates that the data file is safe, comparing a file hash value of the data file with a historical file hash value stored in an information table to generate a comparison result; in the case that the comparison result indicates that the data file is inconsistent with a historical data file stored in a target path, parsing the data file to determine an attribute hash value of an attribute sub-file and data hash values corresponding to N pieces of data in a data sub-file, N>2; and processing the historical data file based on the target hash value to complete synchronization of the data file.

Description

Technical Field

[0001] This disclosure relates to the fields of information security and financial technology, and in particular to a data synchronization method, apparatus, device, medium, and program product. Background Technology

[0002] With the rapid development of computer technology, computers are capable of processing increasingly larger amounts of data, and the amount of data that needs to be stored on computers is also increasing. Related technologies monitor data files as a whole, and when a data file changes, the entire file is synchronized to ensure the real-time performance of the data.

[0003] In the process of implementing this disclosure, it was found that when synchronizing data files, the synchronization efficiency is low because the data volume in the data files is large and the entire file is synchronized. Summary of the Invention

[0004] In view of the above problems, this disclosure provides data synchronization methods, apparatus, devices, media and program products.

[0005] According to a first aspect of this disclosure, a data synchronization method is provided, comprising: responding to a received data file, performing a security check on the data file based on a preset configuration table to obtain a security check result, wherein the data file includes attribute subfiles and data subfiles; if the security check result indicates that the data file is secure, comparing the file hash value of the data file with the historical file hash values ​​stored in an information table to generate a comparison result; if the comparison result indicates that the data file is inconsistent with historical data files stored in a target path, parsing the data file to determine the attribute hash value of the attribute subfile and the data hash value corresponding one-to-one with N data entries in the data subfile, where N>2; and processing the historical data file based on the target hash value to complete the synchronization of the data file, wherein the target hash value represents a hash value among multiple comparison hash values ​​that differs from the historical hash value corresponding to the comparison hash value in the information table, and the multiple comparison hash values ​​include attribute hash values ​​and N data hash values.

[0006] According to embodiments of this disclosure, after comparing the file hash value of the data file with the historical file hash values ​​stored in the information table to generate a comparison result, the method further includes:

[0007] If the comparison result determines that the file hash value is inconsistent with the historical file hash value, it is determined that the data file represented by the comparison result is inconsistent with the historical data file stored in the target path, and the data file is parsed; if the file comparison result determines that the file hash value is consistent with the historical file hash value, a prompt message is output to indicate that there is no need to synchronize the data file.

[0008] According to embodiments of this disclosure, parsing a data file to determine the attribute hash value of an attribute subfile and the data hash value corresponding one-to-one with N data entries in the data subfile, where N>2, includes:

[0009] Determine the attribute hash value of the attribute subfile; determine the target parsing method based on the data format of the data subfile; parse the data subfile using the target parsing method to obtain N data items, where N>2; determine the data hash value corresponding to each of the N data items.

[0010] According to embodiments of this disclosure, historical data files are processed based on a target hash value to complete the synchronization of the data files, including:

[0011] Given that the target hash value is the attribute hash value, the attribute subfile is parsed to obtain M attribute categories, where M>2; the attribute categories of the temporary and formal tables in the historical data file are updated to obtain the first temporary table and the first formal table, which each contain M attribute categories; the first temporary table and the first formal table are stored in the target path to complete the synchronization of the attribute subfile.

[0012] According to embodiments of this disclosure, processing historical data files based on target hash values ​​to complete data file synchronization further includes:

[0013] Perform hash value detection on the information table to obtain the hash detection result; if the hash detection result indicates that there is no historical attribute hash value in the information table, generate a second temporary table and a second formal table based on M attribute categories; store the second temporary table and the second formal table in the target path to complete the synchronization of the attribute sub-files.

[0014] According to embodiments of this disclosure, processing historical data files based on target hash values ​​to complete data file synchronization further includes:

[0015] Given that the target hash value is the data hash value, the data in the temporary table stored in the target path is formatted; N data entries are stored in the formatted temporary table to obtain a third temporary table; the target data corresponding to the target hash value is determined in the third temporary table; the historical data corresponding to the target data in the official table stored in the target path is updated to obtain a third official table including the target data, so as to complete the synchronization of the data sub-file.

[0016] According to embodiments of this disclosure, before processing the historical data file based on the target hash value, the method further includes:

[0017] If the target hash value is the data hash value, determine the number of target hash values; if the number is greater than the preset number, output a warning message to terminate the synchronization of the data file; if the number is less than the preset number, process the historical data file based on the target hash value to complete the synchronization of the data file.

[0018] A second aspect of this disclosure provides a data synchronization apparatus, comprising:

[0019] The data detection module is used to respond to the received data file, perform security detection on the data file based on a preset configuration table, and obtain the security detection result. The data file includes attribute sub-files and data sub-files.

[0020] The hash comparison module is used to compare the file hash value of the data file with the historical file hash values ​​stored in the information table, when the security detection result indicates that the data file is secure, and generate the comparison result.

[0021] The data parsing module is used to parse the data file when the comparison result characterization data file is inconsistent with the historical data file stored in the target path, and to determine the attribute hash value of the attribute sub-file and the data hash value corresponding one-to-one with N data in the data sub-file, where N>2;

[0022] The data synchronization module is used to process historical data files based on a target hash value to complete the synchronization of the data files. The target hash value represents a hash value among multiple comparison hash values ​​that is different from the historical hash value corresponding to the comparison hash value in the information table. The multiple comparison hash values ​​include attribute hash values ​​and N data hash values.

[0023] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0024] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0025] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0026] According to embodiments of this disclosure, a security check is performed upon receiving the data file, and data synchronization only proceeds after the security check is passed. This effectively enhances data security and filters out invalid data. By determining the attribute hash value and data hash value of the data file, it is possible to identify the data that has changed within the data file, thus enabling monitoring of the data within the data file. During data synchronization, historical data files are processed based on the target hash value, meaning only changed data is processed, effectively reducing the amount of data that needs to be processed and thereby improving the efficiency of data synchronization. Attached Figure Description

[0027] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0028] Figure 1 The illustration schematically depicts application scenarios of data synchronization methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure;

[0029] Figure 2 A flowchart illustrating a data synchronization method according to an embodiment of the present disclosure is shown schematically;

[0030] Figure 3 A flowchart illustrating the data source identification process according to an embodiment of this disclosure is shown schematically.

[0031] Figure 4 A flowchart illustrating the data file synchronization determination process according to an embodiment of the present disclosure is shown schematically.

[0032] Figure 5 This schematically illustrates an attribute synchronization flowchart for data file synchronization according to an embodiment of the present disclosure;

[0033] Figure 6 This illustration schematically shows a data synchronization flowchart according to an embodiment of the present disclosure;

[0034] Figure 7 A schematic block diagram of a data synchronization apparatus according to an embodiment of the present disclosure is shown; and

[0035] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a data synchronization method according to an embodiment of the present disclosure. Detailed Implementation

[0036] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0037] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0038] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0039] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0040] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0041] In the process of implementing this disclosure, it was found that existing data synchronization technologies cannot cope with and meet the needs of multi-format data storage in distributed systems, and are difficult to remedy when erroneous operations occur, which easily leads to data errors.

[0042] The embodiments of this disclosure provide a data synchronization method, including: responding to a received data file, performing a security check on the data file based on a preset configuration table to obtain a security check result, wherein the data file includes attribute subfiles and data subfiles; if the security check result indicates that the data file is secure, comparing the file hash value of the data file with the historical file hash values ​​stored in an information table to generate a comparison result; if the comparison result indicates that the data file is inconsistent with historical data files stored in a target path, parsing the data file to determine the attribute hash value of the attribute subfile and the data hash value corresponding one-to-one with N data entries in the data subfile, where N>2; processing the historical data file based on the target hash value to complete the synchronization of the data file, wherein the target hash value represents a hash value among multiple comparison hash values ​​that is different from the historical hash value corresponding to the comparison hash value in the information table, and the multiple comparison hash values ​​include attribute hash values ​​and N data hash values.

[0043] Figure 1 The diagram illustrates an application scenario of data synchronization according to an embodiment of the present disclosure.

[0044] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0045] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0046] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0047] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0048] It should be noted that the data synchronization method provided in this embodiment can generally be executed by server 105. Correspondingly, the data synchronization device provided in this embodiment can generally be located in server 105. The data synchronization method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data synchronization device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0049] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0050] The following will be based on Figure 1 The described scene, through Figures 2-6 The data synchronization method of the disclosed embodiments will be described in detail.

[0051] Figure 2 A flowchart illustrating a data synchronization method according to an embodiment of the present disclosure is shown schematically.

[0052] like Figure 2 As shown, the data synchronization in this embodiment includes operations S210 to S240.

[0053] In operation S210, in response to the received data file, a security check is performed on the data file based on a preset configuration table to obtain the security check result. The data file includes an attribute subfile and a data subfile.

[0054] In operation S220, if the security detection result indicates that the data file is secure, the file hash value of the data file is compared with the historical file hash values ​​stored in the information table to generate a comparison result.

[0055] In operation S230, if the comparison result characterization data file is inconsistent with the historical data file stored in the target path, the data file is parsed to determine the attribute hash value of the attribute sub-file and the data hash value corresponding one-to-one with N data in the data sub-file, where N>2.

[0056] In operation S240, the historical data file is processed based on the target hash value to complete the synchronization of the data file. The target hash value represents a hash value among multiple comparison hash values ​​that is different from the historical hash value corresponding to the comparison hash value in the information table. The multiple comparison hash values ​​include attribute hash values ​​and N data hash values.

[0057] According to embodiments of this disclosure, the preset configuration table can be pre-configured within the server and may include information such as data sources, the storage paths of data files from each data source on the server, the storage format of the data file content, and the data format of data sub-files. The data sources included in the preset configuration table are trusted by the server, therefore they are secure data sources. The data files can be files from different data sources.

[0058] Figure 3 A flowchart illustrating the data source identification process according to an embodiment of this disclosure is shown schematically.

[0059] like Figure 3 As shown, the data source identification method of this embodiment includes operations S301 to S303.

[0060] When operating S301, a data synchronization task is automatically triggered according to a preset time.

[0061] In operation S302, it is determined whether the data source transmitted by the scheduled task exists in the preset configuration table. If the data source exists in the preset configuration table, operation S303 is executed; if the data source does not exist in the preset configuration table, the data source identification process ends.

[0062] When operating S303, the parameters in the preset configuration table are returned.

[0063] According to embodiments of this disclosure, the system automatically initiates a data synchronization task every preset time interval. After the scheduled task is triggered, the data source corresponding to the data file received between the completion of the previous scheduled task and the triggering of the current scheduled task is passed to the data source identification process. If the data source exists in the preset configuration table, it indicates that the data source is secure, and the data source passed by the scheduled task exists in the preset configuration table. Other parameters of the corresponding data source in the preset configuration table are returned. If it does not exist, it indicates that the data source is insecure, the data source identification process ends, and the corresponding data file is not saved.

[0064] According to embodiments of this disclosure, security checks on data files are performed based on a preset configuration table. This can be achieved by searching for the data source corresponding to the data file among the data sources stored in the preset configuration table, thereby verifying whether the source of the data file is secure. If the data source corresponding to the data file is found among the data sources stored in the preset configuration table, the security check result indicates that the data file is secure; otherwise, the security check result indicates that the data file is insecure.

[0065] According to embodiments of this disclosure, a data file includes an attribute subfile and a data subfile. The attribute subfile may include attributes such as data attribute encoding, length, whether it is a primary key, and column names in the data file table. The data subfile may contain the values ​​corresponding to the attributes in the data file table; for example, the data in the data subfile corresponding to the "date" in the attribute subfile may be "2023-04-09".

[0066] According to embodiments of this disclosure, the historical file can be a data file previously sent by the data source corresponding to the data file received this time, which is stored in the storage path corresponding to the data source after data synchronization. The hash value of the historical file can be the hash value corresponding to the historical file.

[0067] According to embodiments of this disclosure, by comparing the hash value of a historical file with the hash value of a received data file, it can be determined whether the data file has changed. When the hash value of the historical file matches the hash value of the received data file, the comparison result indicates that the data file matches the historical data file stored in the target path; when the hash value of the historical file does not match the hash value of the received data file, the comparison result indicates that the data file does not match the historical data file stored in the target path.

[0068] According to embodiments of this disclosure, hash values ​​can be determined using algorithms such as MD5 and SHA.

[0069] According to embodiments of this disclosure, the historical data file is a data file previously stored in the target path. The received data file needs to be synchronized and stored in the target path. Therefore, the comparison result indicates that the received data file is inconsistent with the historical data file stored in the target path, meaning that the received data file has changed compared to the historical file. At this point, it is necessary to determine the changed data. By parsing the data file, the attribute hash value of the attribute sub-file and the data hash value corresponding one-to-one with N data entries in the data sub-file are determined. Each data entry is then compared to determine the changed data. Here, the attribute hash value of the attribute sub-file and the data hash value corresponding one-to-one with N data entries in the data sub-file are comparison hash values.

[0070] According to embodiments of this disclosure, when the comparison result indicates that the data file is inconsistent with the historical data file stored in the target path, the data file is parsed, and after determining the comparison hash value, the comparison hash value is compared with the hash value of the corresponding historical file in the information table. The comparison hash value with different comparison results is the target hash value. The data corresponding to the target hash value is the data that has changed, therefore, the data corresponding to the target hash value needs to be synchronized.

[0071] According to embodiments of this disclosure, a security check is performed upon receiving the data file. Only after passing the security check does the subsequent judgment proceed, preventing data from insecure sources from entering and effectively enhancing data security. By determining the attribute hash value and data hash value of the data file, it is possible to identify the data that has changed within the data file, enabling monitoring of the data within the data file. During data synchronization, historical data files are processed based on the target hash value, meaning only changed data is processed, effectively reducing the amount of data that needs to be processed and thus improving the efficiency of data synchronization.

[0072] Figure 4 A flowchart illustrating the data file synchronization determination process according to an embodiment of the present disclosure is shown.

[0073] like Figure 4 As shown, the data file synchronization determination method in this embodiment includes operations S401 to S405.

[0074] In operation S401, retrieve the historical file hash value from the information table.

[0075] In operation S402, the file hash value of the data file is generated.

[0076] In operation S403, it is determined whether the hash value of the historical file is consistent with the hash value of the data file. If the hash value of the historical file is inconsistent with the hash value of the data file, operation S404 is executed; if the hash value of the historical file is consistent with the hash value of the data file, operation S405 is executed.

[0077] In operation S404, the data file is parsed and then proceeded to the subsequent comparison.

[0078] When operating the S405, a prompt message is output indicating that there is no need to synchronize the data files.

[0079] According to embodiments of this disclosure, historical file hash values ​​are obtained from an information table, and file hash values ​​for data files are generated. It is determined whether the historical file hash values ​​and the data file hash values ​​are consistent. If they are inconsistent, the comparison result indicates that the data file is inconsistent with historical data files stored in the target path. Attribute hash values ​​and N data hash values ​​are determined, and subsequent comparisons are performed based on the corresponding data in the information table. If they are consistent, the comparison result indicates that the data file is consistent with historical data files stored in the target path, meaning the content of the data file has not changed. Therefore, data synchronization is unnecessary, a prompt message is output, and the data synchronization process ends.

[0080] According to embodiments of this disclosure, by comparing the file hash value of the data file with the hash values ​​of historical files, if the content of the data file has not changed, a prompt message is output and the data synchronization process ends. This filters out unnecessary data synchronization operations, further reducing the amount of data that needs to be processed and improving the efficiency of data synchronization.

[0081] According to embodiments of this disclosure, parsing a data file to determine the attribute hash value of an attribute subfile and the data hash value corresponding one-to-one with N data entries in the data subfile, where N>2, includes:

[0082] Determine the attribute hash value of the attribute subfile; determine the target parsing method based on the data format of the data subfile; parse the data subfile using the target parsing method to obtain N data items, where N>2; determine the data hash value corresponding to each of the N data items.

[0083] According to embodiments of this disclosure, the attribute sub-file is parsed to determine the attribute hash value.

[0084] According to the embodiments of this disclosure, the data format of the data sub-file is determined in the preset configuration table, and the parsing method corresponding to the data format of the data sub-file is determined as the target parsing method. The data sub-file is parsed line by line. In the parsing result, the order of each data in the data sub-file is consistent with the order of the data attributes in the data attribute sub-file, and the data hash value corresponding to each data is determined.

[0085] For example, parsing the attribute subfile of data file A yields its attribute hash values. The preset configuration table indicates that the data in the data subfile of data file A is stored in JSON format. Therefore, the target parsing method for the data subfile of data file A is the JSON parsing method. Following the target parsing method, each line of content in the data subfile of A is converted to JSON format, and the corresponding data hash value for each data line is determined.

[0086] According to embodiments of this disclosure, a corresponding parsing method is selected based on the data format of the data file, and the attribute subfiles and data subfiles of the data file are parsed separately. This solves the problem that common data synchronization methods cannot handle the complex data format requirements between services in existing distributed systems. After parsing, it is ensured that the data values ​​in the parsing result are consistent with the order of the corresponding attributes, so that changed data can be accurately identified in subsequent comparisons and judgments.

[0087] According to embodiments of this disclosure, historical data files are processed based on a target hash value to complete the synchronization of the data files, including:

[0088] Given that the target hash value is the attribute hash value, the attribute subfile is parsed to obtain M attribute categories, where M>2; the attribute categories of the temporary and formal tables in the historical data file are updated to obtain the first temporary table and the first formal table, which each contain M attribute categories; the first temporary table and the first formal table are stored in the target path to complete the synchronization of the attribute subfile.

[0089] According to embodiments of this disclosure, a temporary table can be used to determine the changed data in a data file by comparing the file hash value of the received data file with the hash values ​​of historical files. A formal table can be used to modify the changed data in the formal table after determining it in the temporary table, and then save the modified data in the storage path to complete the data synchronization operation.

[0090] According to embodiments of this disclosure, when the target hash value is an attribute hash value, it indicates that the attributes of the data file have changed, requiring an update to the database data file table structure. During the update process, the attribute sub-file is parsed to obtain M attribute categories. Based on these M attribute categories, each category is compared with each attribute category in the corresponding table structure in the database to identify the changed attribute categories. A corresponding Structured Query Language (SQL) is then generated to update the attribute categories of the temporary and formal tables. After the update is complete, the first temporary table and the first formal table are stored in the target path.

[0091] According to embodiments of this disclosure, adding a new data attribute requires generating an SQL statement to append a column to the table. The database table stores logical operators to indicate the validity of the attribute. When deleting a data attribute, instead of using an SQL statement to delete the column, the logical operator is changed to invalid. When the primary key attribute of a database table changes, the primary key must first be deleted, and then regenerated based on the latest primary key attribute. This logical deletion ensures that data can be correctly recovered after accidental deletion, thus guaranteeing data security.

[0092] According to embodiments of this disclosure, processing historical data files based on target hash values ​​to complete data file synchronization further includes:

[0093] Perform hash value detection on the information table to obtain the hash detection result; if the hash detection result indicates that there is no historical attribute hash value in the information table, generate a second temporary table and a second formal table based on M attribute categories; store the second temporary table and the second formal table in the target path to complete the synchronization of the attribute sub-files.

[0094] Figure 5 An attribute synchronization flowchart for data file synchronization according to an embodiment of the present disclosure is illustrated schematically.

[0095] like Figure 5 As shown, the attribute synchronization method for data file synchronization in this embodiment includes operations S501 to S508.

[0096] In operation S501, the attribute subfile hash value is generated.

[0097] In operation S502, determine whether a corresponding historical attribute hash value exists in the information table. If it does not exist, proceed to operation S503; if it does exist, proceed to operation S505.

[0098] When operating S503, parse the data attribute sub-file to obtain attribute information.

[0099] In operation S504, the corresponding SQL statement for creating a new database table structure is generated and executed.

[0100] In operation S505, determine whether the target hash value is an attribute hash value. If it is, execute operation S506; otherwise, execute operation S508.

[0101] When operating S506, parse the data attribute sub-file to obtain attribute information.

[0102] When operating S507, identify the attribute categories that have changed, generate the corresponding SQL statements, and execute them.

[0103] When operating S508, the data sub-file parsing and processing flow is entered.

[0104] According to embodiments of this disclosure, during the synchronization of attribute categories of data files, an attribute subfile hash value is generated, and it is determined whether a historical attribute hash value corresponding to the attribute subfile hash value exists in the information table. If it does not exist, it means that the current data file is not stored in the database. The data attribute subfile is parsed to obtain attribute information, and based on the attribute information of the attribute subfile, a corresponding SQL statement for creating a new database table structure is generated to create a new database table. If it exists, it means that the database table corresponding to the current data file is already stored in the database, and it is determined whether the target hash value is an attribute hash value. If it is, it means that the attributes of the data file have changed compared to historical files. The data attribute subfile is parsed to obtain attribute information, and the changed attribute categories are determined. A corresponding SQL statement is generated to update the attribute categories of the database table. If it is not, it means that the attributes of the data file have not changed compared to historical files.

[0105] According to embodiments of this disclosure, when it is necessary to create a new database table or update the attribute categories of a database table, a second temporary table and a second formal table are generated using SQL statements. The second temporary table and the second formal table are then stored in a target path to synchronize the attribute sub-files and initiate the data sub-file parsing process. When the attribute categories of the database table do not need to be updated, the process directly proceeds to the data sub-file parsing process.

[0106] According to embodiments of this disclosure, processing historical data files based on target hash values ​​to complete data file synchronization further includes:

[0107] Given that the target hash value is the data hash value, the data in the temporary table stored in the target path is formatted; N data entries are stored in the formatted temporary table to obtain a third temporary table; the target data corresponding to the target hash value is determined in the third temporary table; the historical data corresponding to the target data in the official table stored in the target path is updated to obtain a third official table including the target data, so as to complete the synchronization of the data sub-file.

[0108] According to embodiments of this disclosure, when the target hash value is the data hash value, it indicates that the data in the data file has changed, and the data in the database data file table needs to be updated. During the update process, the data in the temporary table in the storage path corresponding to the data source of the data file is cleared, the temporary table is formatted, and N records from the data sub-file are stored in the formatted temporary table to obtain a third temporary table.

[0109] According to embodiments of this disclosure, after determining the target data corresponding to the target hash value, a database update statement is used to change the data that needs to be modified in the formal table to the corresponding data in the temporary table; a database insert statement is used to insert and save the data that is less in the formal table than in the temporary table into the formal table; by changing the logical operators in the database to invalid, the data that is more in the formal table than in the temporary table is logically deleted, thus completing the update of the formal table, obtaining a third formal table, and saving the third formal table to the storage path corresponding to the data source, thus completing the synchronization of the data sub-files.

[0110] According to embodiments of this disclosure, when data in a data file changes and requires updating data in a database data file table, the target data to be modified is first determined in a temporary table. Then, the corresponding data in the formal table is modified to obtain the formal table, thus completing the synchronization of the data sub-file. By using a temporary table, only the changed data is modified in the formal table, reducing direct modifications to the formal table, minimizing the possibility of errors, and improving data security.

[0111] According to embodiments of this disclosure, before processing the historical data file based on the target hash value, the method further includes:

[0112] If the target hash value is the data hash value, determine the number of target hash values; if the number is greater than the preset number, output a warning message to terminate the synchronization of the data file; if the number is less than the preset number, process the historical data file based on the target hash value to complete the synchronization of the data file.

[0113] According to embodiments of this disclosure, the preset number can be the maximum allowed number of operations within the server, obtained by analyzing historical data synchronization operations from different data sources to determine the typical number of operations for each source. During data synchronization, the number of target hash values ​​represents the number of operations required for data file synchronization. When the number of target hash values ​​(i.e., the number of operations) exceeds the preset number, it indicates excessive operations and potential security vulnerabilities in the current data synchronization. The data synchronization operation is terminated, and a warning message is output to the corresponding data source. If the data file is a secure data file sent normally by the data source, the data source can manually continue the data file synchronization operation. When the number of target hash values ​​is less than the preset number, data file synchronization continues.

[0114] According to embodiments of this disclosure, by setting a preset number to control the maximum allowed number of operations, data synchronization operations are stopped when the number of data operations exceeds the preset number, thus preventing data tampering due to malicious operations and further improving data security. For different data sources, historical data is analyzed, and preset numbers are set for different data sources, enabling personalized management of the data synchronization process for different data sources.

[0115] Figure 6 A schematic diagram illustrating a data synchronization flowchart according to an embodiment of the present disclosure is provided.

[0116] like Figure 6 As shown, the data synchronization method for data file synchronization in this embodiment includes operations S601 to S607.

[0117] In operation S601, format the temporary table in the database to store data.

[0118] In operation S602, the target parsing method is determined based on the data format of the data subfile, and parsing is performed line by line according to the target parsing method.

[0119] When operating S603, SQL statements are generated based on the data subfile and stored in a third temporary table in the database.

[0120] In operation S604, the target data that has been modified is determined in the third temporary table based on the target hash value.

[0121] In operation S605, determine whether the number of target hash values ​​is greater than the preset number. If it is not greater, execute operation S606; if it is greater, execute operation S607.

[0122] In operation S606, the historical data corresponding to the target data in the formal table is updated.

[0123] When operating S607, a warning message is output, and the data synchronization ends.

[0124] According to embodiments of this disclosure, in the data synchronization process, the data stored in the temporary table is cleared, and the temporary table is formatted. The target parsing method is determined based on the data format of the data subfile, and parsing is performed row by row according to the target parsing method. SQL statements are generated based on the data subfile and stored in a third temporary table in the database. The target data that has been modified is determined in the third temporary table based on the target hash value. It is then determined whether the number of target hash values ​​is greater than a preset number. If the number of target hash values ​​is not greater than the preset number, the corresponding historical data is replaced with the target data, and the replaced formal table is stored in the storage path corresponding to the data source; if the number of target hash values ​​is greater than the preset number, it indicates that the number of operations is too large, and the current data synchronization may involve unsafe behavior, so a warning message is output, and the current data synchronization ends.

[0125] Based on the above data synchronization method, this disclosure also provides a data synchronization device. The following will be combined with... Figure 7 The device is described in detail.

[0126] Figure 7 A schematic block diagram of a data synchronization apparatus according to an embodiment of the present disclosure is shown.

[0127] like Figure 7 As shown, the data synchronization device 700 of this embodiment includes a data detection module 710, a hash comparison module 720, a data parsing module 730, and a data synchronization module 740.

[0128] The data detection module 710 is used to perform security checks on the received data file based on a preset configuration table in response to the data file, and obtain a security detection result. The data file includes attribute subfiles and data subfiles. In one embodiment, the data detection module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0129] The hash comparison module 720 is used to compare the file hash value of the data file with the historical file hash values ​​stored in the information table when the security detection result indicates that the data file is secure, and generate a comparison result. In one embodiment, the hash comparison module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0130] The data parsing module 730 is used to parse the data file when the comparison result characterization data file is inconsistent with the historical data file stored in the target path, and to determine the attribute hash value of the attribute sub-file and the data hash value corresponding one-to-one with N data in the data sub-file, where N>2. In one embodiment, the data parsing module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0131] The data synchronization module 740 is used to process historical data files based on a target hash value to complete the synchronization of the data files. The target hash value represents a hash value among multiple comparison hash values ​​that differs from the historical hash value corresponding to the comparison hash value in the information table. The multiple comparison hash values ​​include attribute hash values ​​and N data hash values. In one embodiment, the data synchronization module 740 can be used to perform the operation S240 described above, which will not be repeated here.

[0132] According to embodiments of this disclosure, the data synchronization device 700 further includes a file parsing module and a prompt output module.

[0133] The file parsing module is used to determine that the data file represented by the comparison result is inconsistent with the historical data file stored in the target path when the comparison result shows that the file hash value is inconsistent with the historical file hash value, and then parses the data file.

[0134] The prompt output module is used to output a prompt message when the file comparison result determines that the file hash value is consistent with the historical file hash value, so as to indicate that there is no need to synchronize the data file.

[0135] According to embodiments of this disclosure, the data parsing module 730 includes an attribute determination submodule, a method determination submodule, a data file parsing submodule, and a hash determination submodule.

[0136] The attribute determination submodule is used to determine the attribute hash values ​​of the attribute subfile.

[0137] The method determination submodule is used to determine the target parsing method based on the data format of the data subfile.

[0138] The data file parsing submodule is used to parse the data subfile using the target parsing method to obtain N data entries, where N>2.

[0139] The hash determination submodule is used to determine the hash value of each data item among N data items.

[0140] According to embodiments of this disclosure, the data synchronization module 740 includes an attribute file parsing submodule, an attribute update submodule, and a first data storage submodule.

[0141] The attribute file parsing submodule is used to parse the attribute subfile when the target hash value is the attribute hash value, and obtain M attribute categories, where M>2.

[0142] The attribute update submodule is used to update the attribute categories of temporary tables and formal tables in historical data files, resulting in a first temporary table and a first formal table, each containing M attribute categories.

[0143] The first data storage submodule is used to store the first temporary table and the first formal table in the target path to complete the synchronization of the attribute subfile.

[0144] According to embodiments of this disclosure, the data synchronization module 740 further includes a hash detection submodule, a data generation module, and a second data storage submodule.

[0145] The hash detection submodule is used to perform hash value detection on the information table and obtain the hash detection result.

[0146] The data generation module is used to generate a second temporary table and a second formal table based on M attribute categories when there are no historical attribute hash values ​​in the hash detection result characterization information table.

[0147] The second data storage submodule is used to store the second temporary table and the second formal table in the target path to complete the synchronization of the attribute subfile.

[0148] According to embodiments of this disclosure, the data synchronization module 740 further includes a data formatting submodule, a third data storage submodule, a data determination submodule, and a data update submodule.

[0149] The data formatting submodule is used to format the data in a temporary table stored in the target path, provided that the target hash value is the data hash value.

[0150] The third data storage submodule is used to store N data records into a formatted temporary table to obtain the third temporary table.

[0151] The data determination submodule is used to determine the target data corresponding to the target hash value in the third temporary table.

[0152] The data update submodule is used to update the historical data corresponding to the target data in the official table stored in the target path, so as to obtain a third official table including the target data, and complete the synchronization of the data sub-file.

[0153] According to embodiments of this disclosure, the data synchronization device 700 further includes a quantity determination module, an alert output module, and a file processing module.

[0154] The quantity determination module is used to determine the quantity of target hash values ​​given that the target hash value is the data hash value.

[0155] The warning output module is used to output a warning message when the number of data files exceeds a preset number, thereby terminating the synchronization of the data files.

[0156] The file processing module is used to process historical data files based on the target hash value when the number is less than the preset number, so as to complete the synchronization of data files.

[0157] According to embodiments of this disclosure, any plurality of modules among the data detection module 710, hash comparison module 720, data parsing module 730, and data synchronization module 740 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the data detection module 710, hash comparison module 720, data parsing module 730, and data synchronization module 740 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the data detection module 710, hash comparison module 720, data parsing module 730, and data synchronization module 740 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0158] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a data synchronization method according to an embodiment of the present disclosure.

[0159] like Figure 8 As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0160] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0161] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0162] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0163] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0164] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data synchronization method provided in the embodiments of this disclosure.

[0165] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0166] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0167] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0168] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0171] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data synchronization method, comprising: performing security detection on a received data file based on a preset configuration table to obtain a security detection result, wherein the data file comprises an attribute sub-file and a data sub-file; in a case where the security detection result indicates that the data file is safe, comparing a file hash value of the data file with historical file hash values stored in an information table to generate a comparison result; in a case where the comparison result indicates that the data file is inconsistent with a historical data file stored in a target path, parsing the data file to determine an attribute hash value of the attribute sub-file and data hash values corresponding to N pieces of data in the data sub-file, N>2; processing the historical data file based on a target hash value to complete synchronization of the data file, comprising: in a case where the target hash value is the attribute hash value, parsing the attribute sub-file to obtain M attribute categories, M>2; updating attribute categories of temporary tables and formal tables in the historical data file to obtain first temporary tables and first formal tables respectively comprising the M attribute categories; and storing the first temporary tables and the first formal tables in the target path to complete synchronization of the attribute sub-file, wherein the target hash value represents a hash value in a plurality of comparison hash values that is different from a historical hash value corresponding to the comparison hash value in the information table, and the plurality of comparison hash values comprise the attribute hash value and N data hash values; updating attribute categories of temporary tables and formal tables in the historical data file to obtain first temporary tables and first formal tables respectively comprising the M attribute categories, comprising: in a case where it is determined to delete a data attribute category according to the M attribute categories, changing a logical symbol of the data attribute category that needs to be deleted in the temporary tables and the formal tables to invalid, wherein the logical symbol is used to represent whether a data attribute category corresponding to the logical symbol is valid.

2. The method of claim 1, wherein, after the comparing the file hash value of the data file with the historical file hash values stored in the information table to generate the comparison result, further comprising: in a case where the comparison result determines that the file hash value is inconsistent with the historical file hash value, determining that the comparison result indicates that the data file is inconsistent with the historical data file stored in the target path, and parsing the data file; in a case where the file comparison result determines that the file hash value is consistent with the historical file hash value, outputting prompt information to prompt that the data file does not need to be synchronized.

3. The method of claim 1, wherein, the parsing the data file to determine the attribute hash value of the attribute sub-file and the data hash values corresponding to the N pieces of data in the data sub-file, N>2, comprising: determining the attribute hash value of the attribute sub-file; determining a target parsing mode based on a data format of the data sub-file; parsing the data sub-file using the target parsing mode to obtain N pieces of data, N>2; determining a data hash value corresponding to each piece of data in the N pieces of data.

4. The method of claim 1, wherein, The processing of the historical data file based on the target hash value to complete synchronization of the data file further includes: hash value detection on the information table to obtain a hash detection result; in a case where the hash detection result indicates that the historical attribute hash value does not exist in the information table, generating a second temporary table and a second formal table based on the M attribute categories; storing the second temporary table and the second formal table in the target path to complete synchronization of the attribute sub-file.

5. The method of claim 1, wherein, The processing of the historical data file based on the target hash value to complete synchronization of the data file further includes: in a case where the target hash value is the data hash value, formatting data of a temporary table stored in the target path; storing the N pieces of data into the formatted temporary table to obtain a third temporary table; determining target data corresponding to the target hash value in the third temporary table; updating historical data corresponding to the target data in a formal table stored in the target path to obtain a third formal table including the target data to complete synchronization of the data sub-file.

6. The method of claim 1, wherein, Before the processing of the historical data file based on the target hash value, further includes: in a case where the target hash value is the data hash value, determining a quantity of the target hash values; in a case where the quantity is greater than a preset quantity, outputting warning information to terminate synchronization of the data file; in a case where the quantity is less than the preset quantity, processing the historical data file based on the target hash value to complete synchronization of the data file.

7. A data synchronization apparatus, comprising: a data detection module configured to, in response to a received data file, perform security detection on the data file based on a preset configuration table to obtain a security detection result, wherein the data file includes an attribute sub-file and a data sub-file; a hash comparison module configured to, in a case where the security detection result indicates that the data file is secure, compare a file hash value of the data file with historical file hash values stored in an information table to generate a comparison result; a data analysis module configured to, in a case where the comparison result indicates that the data file is inconsistent with a historical data file stored in a target path, analyze the data file to determine an attribute hash value of the attribute sub-file and data hash values corresponding one-to-one to N pieces of data in the data sub-file, N>2; a data synchronization module configured to process the historical data file based on a target hash value to complete synchronization of the data file, wherein the target hash value represents a hash value in a plurality of comparison hash values that is different from a historical hash value corresponding to the comparison hash value in the information table, and the plurality of comparison hash values include the attribute hash value and the N data hash values; wherein the data synchronization module includes: an attribute file analysis submodule configured to, in a case where the target hash value is the attribute hash value, analyze the attribute sub-file to obtain M attribute categories, M>2; The attribute updating submodule is configured to update attribute categories of the temporary table and the formal table in the historical data file to obtain a first temporary table and a first formal table respectively including the M attribute categories. The first data storage submodule is configured to store the first temporary table and the first formal table in the target path to complete synchronization of the attribute sub-file. The attribute updating submodule is configured to update attribute categories of the temporary table and the formal table in the historical data file to obtain a first temporary table and a first formal table respectively including the M attribute categories. In a case where it is determined to delete a data attribute category according to the M attribute categories, a logic symbol of the data attribute category to be deleted in the temporary table and the formal table is changed to invalid, wherein the logic symbol is used to represent whether a data attribute category corresponding to the logic symbol is valid. 8.An electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, enable the one or more processors to perform the method according to any one of claims 1-6. 9.A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-6. 10.A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-6.

Images