Data analysis method and device

Abstract

The invention discloses a data analysis method and device, and relates to the technical field of big data. A specific embodiment of the method comprises the following steps: analyzing a log file of a master table cluster, generating first log overview information, and synchronizing the first log overview information to a slave table cluster to obtain second log overview information; synchronizing the preview log information of the master table cluster to a slave table cluster, analyzing the preview log information of the master table cluster by using the slave table cluster, and updating the second log overview information based on an analysis result; and determining a synchronization integrity analysis result of the slave table cluster based on the updating result. The synchronization progress of the master table and the slave table can be determined through the log overview information, so that the completeness of the slave table data is judged, and the accuracy of a data processing result is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data analysis method and device. Background Art

[0002] In the existing HBase (open source distributed column-based database) active-active solution, the Replication mechanism is a preferred solution that can implement hot standby of data at the table level of one master and multiple slaves among multiple HBase clusters. Under this mechanism, when the cluster where the HBase master table is located is abnormal, the HBase slave table can only take over the read requests of the business.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

[0004] It is impossible to determine the integrity of the slave table data relative to the master table data, resulting in poor accuracy of the data processing results. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data analysis method and device, which can determine the progress of master-slave table synchronization through log overview information, thereby judging the integrity of slave table data and improving the accuracy of data processing results.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a data analysis method is provided, comprising:

[0007] Parse the log files of the master table cluster to generate first log summary information, and synchronize the first log summary information to the slave table cluster to obtain second log summary information;

[0008] Synchronize the preview log information of the master table cluster to the slave table cluster, use the slave table cluster to parse the preview log information of the master table cluster, and update the second log summary information based on the parsing result;

[0009] The synchronization integrity analysis result of the slave table cluster is determined based on the update result.

[0010] Optionally, updating the second log summary information based on the parsing result includes:

[0011] Determine that the preview log information corresponding to the second log overview information has been parsed, and delete the second log overview information to obtain an update result;

[0012] Determine the synchronization integrity analysis results of the slave table cluster based on the update results, including:

[0013] In response to the existence of the second log overview information in the slave table cluster, determining that the synchronization integrity analysis result of the slave table cluster is not synchronized;

[0014] In response to the second log summary information not existing in the slave table cluster, it is determined that the synchronization integrity analysis result of the slave table cluster is synchronization completed.

[0015] Optionally, updating the second log summary information based on the parsing result further includes:

[0016] It is determined that the preview log information corresponding to the second log overview information has been parsed, and the synchronization state of the second log overview information is marked as synchronized to obtain an update result.

[0017] Optionally, after determining the synchronization integrity analysis result of the slave table cluster based on the updated second log overview information, the method further includes:

[0018] An analysis result acquisition request sent by a user is received, and in response to a failure of the main table cluster, a synchronization integrity analysis result corresponding to the analysis result acquisition request is fed back to the user through a preset table synchronization status interface.

[0019] Optionally, the method further comprises:

[0020] The first log summary information is synchronized to the slave table cluster using the endpoint coprocessor.

[0021] Optionally, before parsing the log files of the main table cluster, the following steps are also included:

[0022] In response to a user operating the main table cluster, a write-ahead log corresponding to the operation is generated;

[0023] Persist the write-ahead log and generate a log file;

[0024] The log files are stored in the distributed coordination service of the master table cluster to parse the log files of the master table cluster retrieved from the distributed coordination service.

[0025] According to a second aspect of an embodiment of the present invention, there is provided a data analysis device, comprising:

[0026] A parsing module, used to parse the log files of the master table cluster, generate first log summary information, and synchronize the first log summary information to the slave table cluster to obtain second log summary information;

[0027] An update module, used to synchronize the preview log information of the master table cluster to the slave table cluster, parse the preview log information of the master table cluster using the slave table cluster, and update the second log overview information based on the parsing result;

[0028] The determination module is used to determine the synchronization integrity analysis result of the slave table cluster based on the update result.

[0029] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0030] one or more processors;

[0031] a memory for storing one or more programs,

[0032] When one or more programs are executed by one or more processors, the one or more processors are enabled to implement the method of any of the above embodiments.

[0033] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method of any of the above embodiments is implemented.

[0034] According to a fifth aspect of an embodiment of the present invention, there is provided a computer program product, including a computer program, which implements the method of any of the above embodiments when the computer program is executed by a processor.

[0035] An embodiment of the above invention has the following advantages or beneficial effects: by parsing the log files of the master table cluster, generating first log overview information, and synchronizing the first log overview information to the slave table cluster to obtain second log overview information; synchronizing the preview log information of the master table cluster to the slave table cluster, using the slave table cluster to parse the preview log information of the master table cluster, and updating the second log overview information based on the parsing result; determining the synchronization integrity analysis result of the slave table cluster based on the update result; thereby, the progress of the master-slave table synchronization can be determined through the log overview information, thereby judging the integrity of the slave table data and improving the accuracy of the data processing results.

[0036] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0038] Figure 1 is a schematic diagram of the main process of the data analysis method according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of HBase cluster active-active according to a preferred embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of a flow chart for judging the integrity of table data according to a preferred embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of main modules of a data analysis device according to an embodiment of the present invention;

[0042] Figure 5is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0043] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0045] It should be noted that the acquisition, storage and application of personal information etc. involved in the embodiments of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0046] In the existing HBase (open source distributed column-based database) active-active solution, the Replication mechanism is a preferred solution that can achieve hot standby of data at the table level of one master and multiple slaves between multiple HBase clusters. Under this mechanism, when the cluster where the HBase master table is located is abnormal, the HBase slave table can only accept the read request of the business. It is impossible to determine the integrity of the slave table data relative to the master table data, resulting in poor accuracy of data processing results.

[0047] In view of this, according to one aspect of an embodiment of the present invention, a data analysis method is provided.

[0048] Figure 1 FIG. 1 is a schematic diagram of the main process of the data analysis method according to an embodiment of the present invention. Figure 1 As shown, the data analysis method according to the embodiment of the present invention includes the following steps S101 to S103.

[0049] Step S101: parse the log file of the master table cluster to generate first log summary information, and synchronize the first log summary information to the slave table cluster to obtain second log summary information.

[0050] The master table cluster refers to the cluster that handles core data storage and management tasks in the database system. It can be used to process write operations and manage transaction logs to achieve disaster recovery backup and consistency maintenance of data. The log files of the master table cluster can be parsed by reading WAL (Write-Ahead Log) or hlog files, and the operations recorded therein can be processed in chronological order. The log files contain detailed records of operations such as addition, deletion, and modification. When parsing these log files, useful field information such as log file name, offset, write time, etc. can be identified, so that all operations are correctly parsed and will not be missed or repeated. The first log overview information refers to the log summary generated in the master table cluster. The first log overview information is the result of summarizing and simplifying all parsed log records, including key indicators related to data synchronization, such as log file name, synchronization progress, offset, and synchronization status. The log overview information can help the system monitor and record the overall processing progress of the log, so that abnormal situations can be detected during synchronization or backup, and recovery operations can be performed. In addition, generating log overview information helps to simplify the processing of the master table data by the slave table cluster and reduce unnecessary redundant information.

[0051] The synchronization of the first log overview information to the slave table cluster can be performed through a specific synchronization mechanism, such as using a network communication interface (such as an API or an endpoint) to pass the first log overview information to a distributed coordination system such as Zookeeper (an open source distributed coordination service) of the slave table cluster. During the data synchronization process, the master table cluster can push the latest processing status of the log to the slave table cluster in real time or regularly, so that the slave table can update the synchronization progress in real time. In order to prevent data loss or incomplete processing, the master table cluster can record the status of log synchronization on Zookeeper and continue synchronization from the breakpoint when abnormal recovery occurs to avoid rollback or data overwriting. The slave table cluster is a backup node or replica node of the master table cluster, which is used for redundant data storage, disaster recovery, and read operation diversion. The log information synchronized from the master table cluster maintains consistency with the master table data. The second log overview information refers to the log summary generated by the slave table cluster based on the synchronized first log overview information, which reflects the synchronization and processing progress of the slave table cluster and ensures data consistency between the master table and the slave table. Similar to the first log overview information of the master table, the second log overview information records key information such as the name of the currently processed log file, synchronization progress, offset, etc., to help the system determine whether the slave table cluster has completed synchronization with the master table cluster.

[0052] Step S102: synchronize the preview log information of the master table cluster to the slave table cluster, parse the preview log information of the master table cluster using the slave table cluster, and update the second log summary information based on the parsing result.

[0053] When the user operates the master table cluster to perform write operations, the master table cluster can first generate preview log information, which includes key information related to data synchronization, such as the name of the log file, synchronization progress, offset, timestamp, etc. In the synchronization operation, the master table cluster sends the preview log information to the slave table cluster through a network communication channel (such as an API or a specific endpoint). To ensure the accuracy and integrity of the data, the master table cluster can record and manage the synchronization status through a distributed coordination service (such as Zookeeper) to prevent data loss or irrecoverable synchronization interruptions.

[0054] After receiving the preview log information synchronized with the master table cluster, the slave table cluster can parse the preview log information. The parsing process includes checking the content, offset and synchronization status of the log file, and confirming the integrity and correctness of the preview log information. During parsing, the slave table cluster can focus on analyzing which log files have been fully synchronized and which need further processing to ensure that the data of the slave table is consistent with the master table. The parsing process involves a detailed inspection of each log entry or block to ensure that no data operations are missed, and to identify potential problems in the synchronization process, such as damaged or incomplete logs.

[0055] Based on the parsing results, the slave table cluster updates the second log overview information in real time. The second log overview information is the localized processing result of the slave table cluster on the master table cluster log, reflecting the status of the slave table in the synchronization operation. The second log overview information may include which log files have been processed, which are still being processed, and the specific progress of the processing. The slave table continuously updates the second log overview information by parsing the preview log information of the master table, so that it keeps in sync with the master table.

[0056] Step S103: determining the synchronization integrity analysis result of the slave table cluster based on the update result.

[0057] After the slave cluster updates the second log overview information, you can check the processing status of each synchronization task one by one. If the log files have been completely parsed and successfully processed, the slave cluster can mark these log files as completed on its Zookeeper node and remove them from the synchronization list, so that all write operations of the master cluster are correctly transmitted to the slave table and no update operations are missed. If a log file is still in an unfinished state or an exception occurs during the parsing process, the slave cluster will mark its status as unfinished, and the system can continue with subsequent synchronization processing. The synchronization integrity analysis result is generated based on the inspection of all synchronization tasks of the slave table cluster. By checking the second log overview information, the processing status of each log file can be evaluated to ensure that the offset of each file is consistent with the master table cluster. If the synchronization progress of all log files is displayed as 100% and all offsets match, it means that the slave table cluster has successfully completed the synchronization task with the master table cluster. At this time, the synchronization integrity analysis result will be displayed as "Synchronization Completed", which indicates that the data of the slave table cluster is consistent with the master table cluster, and there is no loss or error in the data transmission and parsing process; if the analysis result shows that some log files are still being processed or the synchronization is incomplete, the system will mark the synchronization task as "Incomplete". At this time, the slave table cluster can be resynchronized, or the lost log data can be re-obtained from the master table as needed. In this case, the exception handling mechanism can also be triggered, such as rolling back to the previous synchronization point, or re-parsing and synchronizing the problematic log files.

[0058] Optionally, updating the second log overview information based on the parsing result includes: determining that the preview log information corresponding to the second log overview information has been parsed, deleting the second log overview information to obtain an updated result; determining the synchronization integrity analysis result of the slave table cluster based on the update result, including: in response to the existence of the second log overview information in the slave table cluster, determining that the synchronization integrity analysis result of the slave table cluster is not synchronized; in response to the absence of the second log overview information in the slave table cluster, determining that the synchronization integrity analysis result of the slave table cluster is synchronized.

[0059] After the slave table cluster receives the preview log information of the master table cluster and completes the parsing, the second log overview information is updated according to the parsing result. The updating process includes checking whether the slave table cluster has completed the parsing of the log file, etc. By parsing the preview log information, the processing progress and synchronization status of each log file can be determined. When all operations in the preview log information have been successfully parsed and applied to the slave table cluster, the log file is marked as processed. At this time, as part of the update, the second log overview information is deleted, indicating that the slave table cluster has completely processed and synchronized the log file, ensuring that the data synchronization task is successfully completed. The deletion operation not only releases resources in Zookeeper, but also avoids subsequent repeated processing of the same log file.

[0060] Based on the update result, the synchronization integrity analysis result of the slave table cluster can be further determined. If there is still undeleted second log overview information in the Zookeeper node of the slave table cluster, this indicates that some log files have not been completely parsed or synchronized. In response to this status, it can be determined that the synchronization has not been completed, and the synchronization integrity analysis result is marked as "not synchronized". In this case, you can continue to wait for the parsing progress, or re-execute the synchronization operation as needed to ensure the complete transmission and parsing of the log file. In response to the absence of the second log overview information in the slave table cluster, that is, all log files have been processed and the synchronization status has been updated, the synchronization integrity analysis result is determined to be "synchronized", which means that the data between the slave table cluster and the master table cluster has been completely synchronized, and all operation records have been correctly processed, ensuring data consistency between the two clusters.

[0061] The embodiment of the present invention can monitor and evaluate the synchronization status between the master and slave clusters in real time through this parsing, updating and analyzing process, ensuring that there are no omissions or errors in data transmission and processing between different nodes.

[0062] Optionally, updating the second log overview information based on the parsing result further includes: determining that the preview log information corresponding to the second log overview information has been parsed, marking the synchronization status of the second log overview information as synchronized, to obtain an update result.

[0063] If all the preview log information corresponding to the second log overview information has been successfully parsed, the synchronization status of the second log overview information can be updated to "synchronized and processed". This status mark not only reflects the progress achieved by the table cluster during the processing, but also provides a basis for subsequent business logic, which can accurately understand the synchronization status of each table and improve the consistency and integrity of data in the system. When executing this mark, relevant business logic can also be considered, such as displaying the synchronization status on the user interface, triggering subsequent data processing tasks, or providing it to the monitoring system for real-time monitoring.

[0064] The embodiment of the present invention realizes transparent management of the log synchronization process through this marking mechanism, so that the slave table cluster can process log information more efficiently and respond to business needs in a timely manner. At the same time, this update result will also become the basis for subsequent data integrity analysis. According to this state, data consistency checks can be continued to ensure that the states of the master and slave tables are always synchronized, avoiding potential data inconsistency problems.

[0065] Optionally, after determining the synchronization integrity analysis result of the slave table cluster based on the updated second log overview information, it also includes: receiving an analysis result acquisition request sent by a user, and in response to a failure of the main table cluster, feeding back the synchronization integrity analysis result corresponding to the analysis result acquisition request to the user through a preset table synchronization status interface.

[0066] After determining the synchronization integrity analysis result of the slave table cluster based on the updated second log overview information, it can be used for user interaction and receive the analysis result acquisition request sent by the user, especially in the case of a failure in the master table cluster. When the user sends an analysis result acquisition request through the preset table synchronization status interface, the validity of the request is first verified to determine that its source is legitimate and has the authority to obtain relevant information, and then the previously performed synchronization integrity analysis result is retrieved to determine that the result is consistent with the content of the user's request. In response to the failure of the master table cluster, the previously recorded synchronization integrity analysis result is fed back to the user. This feedback includes not only the synchronization status information, but also the relevant log overview information and any potential abnormalities.

[0067] In this way, users can quickly get feedback on the latest synchronization status of the slave table cluster and understand the synchronization status of data after the master table cluster fails. This timely information transmission mechanism helps users make corresponding business decisions, such as deciding whether to perform data recovery, resynchronization, or other necessary operations. This seamless combination of user interaction and data processing can effectively improve the overall performance of the system and ensure the reliability and timeliness of data synchronization when encountering failures.

[0068] Optionally, the method further includes: synchronizing the first log summary information to the slave table cluster using an endpoint coprocessor.

[0069] The endpoint coprocessor receives the first log overview information from the master table cluster, and can perform necessary verification and analysis on the information to confirm its validity and accuracy, including checking the data format, verifying important fields, and ensuring that the information is not damaged during the transmission process. Through such verification, potential problems can be discovered and handled in a timely manner to ensure the smooth progress of the synchronization process. After completing the verification, the endpoint coprocessor can encapsulate the first log overview information into a format suitable for transmission, and send it to the slave table cluster using the data transmission mechanism. A suitable communication protocol can be selected to minimize delays and improve throughput, such as using message queues or other suitable transmission means. During the data transmission process, the endpoint coprocessor can also continuously monitor the transmission status to ensure that the information can reach the target cluster smoothly, and implement a retry mechanism when necessary to cope with network fluctuations or other emergencies. After receiving the first log overview information from the slave table cluster, it can be saved to a designated storage node through a suitable storage mechanism, such as Zookeeper, etc. After completing this series of operations, the endpoint coprocessor can record the synchronization status information and feedback it to the master table cluster to ensure that the status between the two remains consistent. This efficient synchronization through the endpoint coprocessor not only improves the reliability of data transmission, but also lays a solid foundation for subsequent data processing and consistency maintenance.

[0070] Optionally, before parsing the log file of the main table cluster, it also includes: in response to a user operating the main table cluster, generating a pre-write log corresponding to the operation; persisting the write-ahead log to generate a log file; storing the log file in the distributed coordination service of the main table cluster to parse the log file of the main table cluster called from the distributed coordination service.

[0071] When a user initiates an operation, relevant information about the operation can be captured, including the type of operation, the data records involved and their changes, and a write-ahead log corresponding to the operation is generated. The generated write-ahead log is persisted, that is, the log information is written to a persistent storage medium to prevent data loss, ensuring the reliability and integrity of the log file. Even when a system failure occurs, the rollback or replay of the operation can be carried out normally. After persistence, the write-ahead log is converted into a log file, which contains all necessary operation details and status information. The generated log file is stored in the distributed coordination service of the master table cluster, that is, the log file is distributed on multiple nodes to improve data availability and access speed. The use of distributed coordination services such as Zookeeper not only provides a centralized management and coordination mechanism for log files, but also ensures consistency and reliability between different nodes, and can efficiently retrieve the required log files for subsequent parsing and processing. Once the log file is successfully stored, the log file of the master table cluster can be retrieved from the distributed coordination service for parsing.

[0072] Figure 2 FIG. 1 is a schematic diagram of an HBase cluster active-active configuration according to a preferred embodiment of the present invention. Figure 2 As shown in the figure, table t1 in HBase cluster A is used as the main table for upper-layer business reading and writing, and WAL is synchronized to table t1 in cluster B. Table t1 in cluster B only provides read services for upper-layer businesses; the same is true for table t2 in HBase cluster B.

[0073] Taking table t1 as an example, the principle of data synchronization and integrity judgment of the slave table in the active-active cluster is as follows: When the user operates the HBase in the cluster where the master table is located to perform write operations, the pre-write log WAL will be written first. The file where WAL is persisted on HDFS is the hlog file. The EndPoint coprocessor service on the RS node (that is, the RegionServer service node of HBase), for the master table t1, HBase cluster A is the client node of EndPoint, and HBase cluster B is the server node of endpoint. The EndPoint client node of the cluster where the master table is located will need to be synchronized to the node where the EndPoint server of the cluster where the slave table is located; at the same time, the client node records the synchronization status information (file name, offset, etc.) of the hlog log file to the zookeeper of this cluster. At the same time, when an hlog file is generated in the cluster where the master table is located, EndPoint analyzes the hlog file, parses it into synchronization overview information such as table name: number of operations, and records it on the zk of the cluster where the slave table is located. The EndPoint of cluster B where the slave table is located receives the WAL log information for processing, and updates the synchronization overview information of the modified table on the zk of this cluster according to the table information of the operation. When cluster A where the t1 master table is located fails or the business actively switches to cluster B where the t1 slave table is located, the integrity of the synchronization information can be obtained through the t1 table synchronization status interface on cluster B. The integrity is judged by the synchronization overview information of the t1 table on cluster B.

[0074] Figure 3 FIG. 1 is a schematic diagram of a flow chart for judging the integrity of table data according to a preferred embodiment of the present invention. Figure 3As shown in the figure, under the HBase Replication mechanism, the cluster where the master table is located records the hlog information (file name, synchronized offset, etc.) to the zookeeper of this cluster to prevent the rollback of the hlog that has not been synchronized and the breakpoint synchronization after abnormal recovery. Each RS node in the cluster where the master table is located will parse the hlog log overview information and synchronize it to the zookeeper of the cluster where the slave table is located through EndPoint. WAL (hlog) information is synchronized through EndPoint. After the cluster where the slave table is located processes the synchronized WAL information, it updates the corresponding log overview information on zookeeper. If the hlog file corresponding to the overview information has been processed, it can be deleted. The business party can obtain the synchronization integrity information of the slave table through the table synchronization status interface. The integrity information is judged by the log overview information on zookeeper. When the table still has hlog information of the log overview information, the table is not synchronized; if the table does not have the corresponding log overview information on zk, the synchronization of the slave table is completed at this moment and is consistent with the information of the master table.

[0075] In the embodiment of the present invention, under the Replication mechanism of HBase, by extending the Endpoint coprocessor (HBase endpoint coprocessor), the write-ahead log WAL (Write-Ahead Logging) of HBase is monitored, the write-ahead log is scanned and analyzed, the pre-operation information overview of each table is extracted, and the pre-operation information overview is synchronized to the Zookeeper (open source distributed coordination service) of the slave cluster. The slave cluster can know the progress of the master-slave table synchronization through the pre-write overview information synchronized to Zookeeper, and can judge the integrity of the slave table data.

[0076] According to a second aspect of an embodiment of the present invention, a data analysis device is provided.

[0077] Figure 4 Schematic diagram of the main modules of the data analysis device according to an embodiment of the present invention. Figure 4 As shown, a data analysis device 400 includes:

[0078] The parsing module 401 is used to parse the log files of the master table cluster, generate first log summary information, and synchronize the first log summary information to the slave table cluster to obtain second log summary information;

[0079] An updating module 402 is used to synchronize the preview log information of the master table cluster to the slave table cluster, parse the preview log information of the master table cluster using the slave table cluster, and update the second log summary information based on the parsing result;

[0080] The determination module 403 is used to determine the synchronization integrity analysis result of the slave table cluster based on the update result.

[0081] Optionally, the updating module 402 is further configured to:

[0082] Determine that the preview log information corresponding to the second log overview information has been parsed, and delete the second log overview information to obtain an update result;

[0083] Determine the synchronization integrity analysis results of the slave table cluster based on the update results, including:

[0084] In response to the existence of the second log overview information in the slave table cluster, determining that the synchronization integrity analysis result of the slave table cluster is not synchronized;

[0085] In response to the second log summary information not existing in the slave table cluster, it is determined that the synchronization integrity analysis result of the slave table cluster is synchronization completed.

[0086] Optionally, the updating module 402 is further configured to:

[0087] It is determined that the preview log information corresponding to the second log overview information has been parsed, and the synchronization state of the second log overview information is marked as synchronized to obtain an update result.

[0088] Optionally, the data analysis device 400 further includes an acquisition module, which is used to:

[0089] An analysis result acquisition request sent by a user is received, and in response to a failure of the main table cluster, a synchronization integrity analysis result corresponding to the analysis result acquisition request is fed back to the user through a preset table synchronization status interface.

[0090] Optionally, the data analysis device 400 further includes a processing module, and the processing module is used to:

[0091] The first log summary information is synchronized to the slave table cluster using the endpoint coprocessor.

[0092] Optionally, the data analysis device 400 further includes a storage module, and the storage module is used to:

[0093] In response to a user operating the main table cluster, a write-ahead log corresponding to the operation is generated;

[0094] Persist the write-ahead log and generate a log file;

[0095] The log files are stored in the distributed coordination service of the master table cluster to parse the log files of the master table cluster retrieved from the distributed coordination service.

[0096] It should be noted that the specific implementation content of the data analysis device of the present invention has been described in detail in the above data analysis method, so the repeated content will not be described again here.

[0097] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the first aspect of the embodiment of the present invention.

[0098] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0099] According to a fifth aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0100] Figure 5 An exemplary system architecture 500 is shown to which the data analysis method or data analysis device according to the embodiments of the present application can be applied.

[0101] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0102] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0103] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0104] The server 505 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The backend management server may analyze and process the received data such as the data analysis request, and feed back the processing result (such as the analysis result - only an example) to the terminal device.

[0105] It should be noted that the data analysis method provided in the embodiment of the present application is generally executed by the server 505 , and accordingly, the data analysis device is generally set in the server 505 .

[0106] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0107] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present application. Figure 6 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0108] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0109] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0110] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present application are executed.

[0111] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0112] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a parsing module, an update module and a determination module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the determination module may be described as "a module for determining the synchronization integrity analysis result of the slave table cluster based on the update result".

[0114] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: parsing the log file of the master table cluster, generating first log overview information, and synchronizing the first log overview information to the slave table cluster to obtain second log overview information; synchronizing the preview log information of the master table cluster to the slave table cluster, parsing the preview log information of the master table cluster using the slave table cluster, and updating the second log overview information based on the parsing result; determining the synchronization integrity analysis result of the slave table cluster based on the update result.

[0115] The computer program product provided by the embodiment of the present invention includes a computer program, and when the computer program is executed by a processor, the data analysis method in the embodiment of the present invention is implemented.

[0116] The technical solution according to the embodiment of the present invention has the following advantages or beneficial effects: by parsing the log file of the master table cluster, generating the first log overview information, and synchronizing the first log overview information to the slave table cluster, obtaining the second log overview information; synchronizing the preview log information of the master table cluster to the slave table cluster, using the slave table cluster to parse the preview log information of the master table cluster, and updating the second log overview information based on the parsing result; determining the synchronization integrity analysis result of the slave table cluster based on the update result; thereby, the progress of the master-slave table synchronization can be determined through the log overview information, thereby judging the integrity of the slave table data and improving the accuracy of the data processing result.

[0117] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

[0118] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

Claims

1. A data analysis method, characterized in that: include: Parse the log file of the master table cluster to generate first log summary information, and synchronize the first log summary information to the slave table cluster to obtain second log summary information; Synchronize the preview log information of the master table cluster to the slave table cluster, parse the preview log information of the master table cluster using the slave table cluster, and update the second log summary information based on the parsing result; A synchronization integrity analysis result of the slave table cluster is determined based on the update result.

2. The method according to claim 1, characterized in that Updating the second log summary information based on the parsing result includes: Determining that the preview log information corresponding to the second log overview information has been parsed, and deleting the second log overview information to obtain an update result; Determining the synchronization integrity analysis result of the slave table cluster based on the update result includes: In response to the existence of the second log overview information in the slave table cluster, determining that a synchronization integrity analysis result of the slave table cluster is not completed; In response to the second log overview information not existing in the slave table cluster, it is determined that the synchronization integrity analysis result of the slave table cluster is synchronization completed.

3. The method according to claim 1, characterized in that Updating the second log summary information based on the parsing result also includes: It is determined that the preview log information corresponding to the second log overview information has been parsed, and the synchronization status of the second log overview information is marked as synchronized to obtain an update result.

4. The method according to claim 1, characterized in that: After determining the synchronization integrity analysis result of the slave table cluster based on the updated second log overview information, the method further includes: An analysis result acquisition request sent by a user is received, and in response to a failure of the main table cluster, a synchronization integrity analysis result corresponding to the analysis result acquisition request is fed back to the user through a preset table synchronization status interface.

5. The method according to claim 1, characterized in that: The method further comprises: The first log summary information is synchronized to the slave table cluster using an endpoint coprocessor.

6. The method according to claim 1, characterized in that Before parsing the log files of the main table cluster, it also includes: In response to a user operating the main table cluster, generating a write-ahead log corresponding to the operation; Persistently process the write-ahead log to generate a log file; The log file is stored in the distributed coordination service of the master table cluster to parse the log file of the master table cluster retrieved from the distributed coordination service.

7. A data analysis device, characterized in that: include: A parsing module, used to parse the log files of the master table cluster, generate first log summary information, and synchronize the first log summary information to the slave table cluster to obtain second log summary information; An update module, configured to synchronize the preview log information of the master table cluster to the slave table cluster, parse the preview log information of the master table cluster using the slave table cluster, and update the second log summary information based on the parsing result; A determination module is used to determine a synchronization integrity analysis result of the slave table cluster based on the update result.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images