A data platform data synchronization method and system

Abstract

The application provides a kind of detection data platform data synchronization method and system, comprising: in response to the received data synchronization request, determine the type of data synchronization request based on data synchronization request;Based on the type of data synchronization request, select the preset audit condition, judge whether the data synchronization request meets the preset audit condition;If not, return or cancel the data synchronization request;If it is correct, determine the source data corresponding to the data synchronization request and the target data;Through the mode of data transmission, obtain the source data as the data to be synchronized, and perform data verification on the data to be synchronized;The data to be synchronized that passes data verification is synchronized to the target data in the target database, and when data synchronization fails, abnormal handling is performed according to the preset abnormal handling procedure.The application optimizes the data synchronization method of detection data platform, realizes the fine management of data synchronization process, improves the integrity, correctness and security of synchronized data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for synchronizing data on a detection data platform. Background Technology

[0002] With the continuous development of sensing technology, information technology, communication technology, and computer technology, various new types of testing equipment, instruments, and sensors are constantly emerging, which has promoted the improvement of the technical level and product performance of the testing industry.

[0003] With the accelerated advancement of industrialization and intelligentization, the demand for testing in areas such as product quality, safety, and environmental protection is constantly increasing. Industries such as biopharmaceuticals, food, and environmental protection are also expanding their demand for testing technologies. Testing companies' testing data platforms refer to comprehensive software systems specifically designed to manage and process various types of data generated during the testing process. However, these testing data platforms have numerous data sources, large data volumes, and are growing rapidly, making it difficult to effectively synchronize and process the data. This often results in low efficiency, frequent errors, and high maintenance difficulty, which not only affects the normal progress of testing and reduces testing speed but also easily leads to data inconsistencies, resulting in a decrease in the reliability of test results.

[0004] Therefore, a solution is needed to address at least one of the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application proposes a data synchronization method and system for a detection data platform. To solve at least one of the above technical problems, the technical solution of this application is:

[0006] A data synchronization method for a detection data platform, comprising: In response to a received data synchronization request, the type of the data synchronization request is determined based on the data synchronization request. The type of the data synchronization request includes a first data synchronization request involving laboratory data and a second data synchronization request involving testing standard data. Based on the type of data synchronization request, select preset review conditions and determine whether the data synchronization request meets the preset review conditions; If the conditions are not met, the data synchronization request is rejected or cancelled; if the conditions are met, the source data and target data corresponding to the data synchronization request are determined. The source data is obtained as the data to be synchronized through data transmission, and the data to be synchronized is verified. The data to be synchronized that has passed the data verification is synchronized to the target data in the target database. The target database includes one or more databases, and different target databases are related to each other through key fields. When data synchronization fails, exception handling is performed according to the preset exception handling steps.

[0007] In one specific embodiment, the process of performing data verification on the data to be synchronized includes: a primary verification involving the data format and a secondary verification involving the data content; If the first verification fails, the data to be synchronized is preprocessed one or more times until the first verification passes. The preprocessing includes any one or more of data cleaning, data conversion, and data formatting; When the first verification passes, a second verification is performed on the data to be synchronized. The second verification includes one or more of the following: data integrity verification, data accuracy verification, data consistency verification, and data quality verification.

[0008] In one specific embodiment, the process of determining the type of the data synchronization request based on the data synchronization request includes: Parse the data synchronization request and extract key data synchronization information, which includes one or more of the following: source data identifier, target data identifier, synchronization frequency information, and synchronization strategy information. The source data identifier is analyzed to obtain first information including data source, nature and structure; the target data identifier is analyzed to obtain second information including data purpose, structure and storage method; and the synchronization frequency information and the synchronization strategy information are analyzed to obtain third information including synchronization type, synchronization priority and synchronization time. By comprehensively analyzing any and multiple of the first information, the second information, and the third information, the type of the data synchronization request is determined.

[0009] In one specific embodiment, the method further includes generating a data synchronization request. The monitoring involves a primary data source of laboratory data and a secondary data source of industry standard data. If changes are detected in the data from the first data source and the second data source, the changed data is collected to obtain the source data. Data synchronization information is determined based on the source data, and the data synchronization information includes target data, synchronization method, and synchronization frequency. Verify whether the data synchronization request generation permission is met. If it is met, generate a data synchronization request corresponding to the source data.

[0010] In one specific embodiment, the monitoring of the first data source is achieved by a preset trigger event. When the data of the first data source changes, the insertion, update, and deletion operations of the data are captured. Monitoring of the second data source is achieved by periodically polling the second data source and detecting data changes by querying and comparing the data before and after the second data source.

[0011] In one specific embodiment, the source data includes static data and dynamic data; The static data includes: industry testing standard data and basic laboratory data; The dynamic data includes: detection item data; The data synchronization methods for static data include a full synchronization method, and the data synchronization methods for dynamic data include an incremental synchronization method and a real-time synchronization method.

[0012] In one specific embodiment, the process of handling exceptions according to preset exception handling steps includes: By setting up scheduled monitoring tasks, we can continuously monitor for anomalies during the data synchronization process and scan and capture any possible error messages in real time. When an anomaly is detected in the data synchronization, a synchronization anomaly log including the error information is generated; An exception is handled, and the data to be synchronized is synchronized one or more times according to the preset number of synchronizations and time intervals. If the first exception handling fails, a second exception handling is performed to process the error corresponding to the error message. After the processing is completed, the data to be synchronized is synchronized one or more times again. If the secondary anomaly handling fails, a synchronous anomaly alarm message will be triggered.

[0013] In one specific embodiment, before synchronizing the data to be synchronized that has passed data verification to the target data in the target database, the method further includes: Simulated data synchronization involves synchronizing the data to be synchronized to the intermediate target data in an intermediate database, wherein the intermediate database is generated based on the target database mapping. Check if there are any problems with the intermediate target data after synchronization. If there are no problems, then synchronize the data to be synchronized through the check to the target data in the target database.

[0014] In one specific embodiment, the method further includes: When multiple data synchronization requests are received simultaneously, the system first checks whether the data synchronization resources of the detection system are sufficient. If the data synchronization resources are sufficient, all the data synchronization requests are responded to simultaneously for data synchronization processing. If data synchronization resources are insufficient, check whether there is a priority field in each of the data synchronization requests. If there is a priority field in each of the data synchronization requests, process the data synchronization requests with priority fields first. If none of the data synchronization requests contain a priority field, then the synchronization will be performed in the order in which the data synchronization requests were generated. When the data synchronization request is a second data synchronization request, data synchronization of the first data source is also required.

[0015] A system for implementing detection data management and synchronization, used to implement the detection data platform data synchronization method described in any of the first parts, the system comprising: A receiving module is used to respond to a received data synchronization request and determine the type of the data synchronization request based on the data synchronization request. The type of the data synchronization request includes a first data synchronization request involving laboratory data and a second data synchronization request involving testing standard data. The synchronization request review module is used to select different preset review conditions based on different types of data synchronization requests, and determine whether the data synchronization request meets the preset review conditions. If the conditions are not met, the data synchronization request is rejected or cancelled; if the conditions are met, the data source and target data corresponding to the data synchronization request are determined. The data synchronization module is used to acquire the source data as the data to be synchronized through data transmission, and to perform data verification on the data to be synchronized; and to synchronize the data to be synchronized that has passed the data verification to the target data in the target database. The target database includes one or more databases, and different target databases are related to each other through key fields. The exception handling module is used to handle exceptions according to preset exception handling steps when data synchronization fails.

[0016] Beneficial effects: This invention provides a data synchronization method and system for a testing data platform, optimizing the data synchronization method for testing enterprises' testing data platforms. Specifically, it categorizes received data synchronization requests and performs differentiated audits for different types of data synchronization requests, enabling refined management of the data synchronization process. Through data verification and anomaly alerts, it improves the integrity, accuracy, and security of synchronized data. Generating synchronization anomaly logs helps track and analyze problems that occur during the synchronization process, providing a basis for subsequent problem investigation and handling, thereby enhancing the manageability and traceability of the data synchronization process. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a data synchronization method for a detection data platform according to one embodiment. Figure 2 This is a schematic diagram of an exception handling process according to another embodiment; Figure 3 This is a diagram showing the module composition of the detection data platform data synchronization system in an embodiment.

[0019] Figure label: 1-Receiving module; 2-Synchronization request review module; 3-Data synchronization module; 4-Exception handling module. Detailed Implementation

[0020] Various embodiments of this disclosure will be described more fully below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0021] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of features, numbers, steps, operations, elements, components, or combinations of the foregoing.

[0022] In various embodiments of this disclosure, the expression "or" or "at least one of A and / or B" includes any combination or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A and / or B" may include A, may include B, or may include both A and B.

[0023] The terms used in the various embodiments of this disclosure (such as "first," "second," etc.) may modify various components in the various embodiments, but do not limit the corresponding components. For example, the above terms do not limit the order and / or importance of the components. The above terms are only used for the purpose of distinguishing one component from others. For example, a first user device and a second user device refer to different user devices, although both are user devices. For example, a first component may be referred to as a second component without departing from the scope of the various embodiments of this disclosure, and similarly, a second component may also be referred to as a first component.

[0024] It should be noted that if a description is made of "connecting" one component to another, then the first component can be directly connected to the second component, and a third component can be "connected" between the first and second components. Conversely, when a component is "directly connected" to another component, it can be understood that there is no third component between the first and second components.

[0025] The term "user" as used in various embodiments of this disclosure may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0026] The terminology used in the various embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this disclosure pertain. Terms (such as those defined in a generally used dictionary) are to be interpreted as having the same meaning as in the context of the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this disclosure.

[0027] Example 1 This application proposes a data synchronization method for a detection data platform, such as... Figure 1 As shown, it includes: S100, respond to the received data synchronization request, determine the type of data synchronization request based on the data synchronization request, the type of data synchronization request includes a first data synchronization request involving laboratory data and a second data synchronization request involving test standard data; S200: Based on the type of data synchronization request, select preset review conditions and determine whether the data synchronization request meets the preset review conditions; If S30A is not met, the data synchronization request will be returned or cancelled. If S30B is met, the source and target data corresponding to the data synchronization request are determined. S400 acquires source data as data to be synchronized via data transmission and performs data verification on the data to be synchronized; synchronizes the data to be synchronized that has passed the data verification to the target data in the target database. The target database may include one or more databases, and different target databases are related to each other through key fields. S500: When data synchronization fails, exception handling is performed according to the preset exception handling steps.

[0028] In the data synchronization process, source data refers to the original data that needs to be copied, transmitted, or synchronized to the target location. This original data is typically stored in one or more data sources, such as databases, file systems, application outputs, sensor devices, etc.

[0029] Target data refers to the data at the target location where the source data is synchronized. When performing data synchronization, the source data is usually copied, transmitted, or synchronized to one or more target locations. These locations are where the target data is stored. After the data synchronization is completed, the content of the target data is replaced by the content of the source data.

[0030] Understandably, performing pre-set audit conditions and data verification during data synchronization can help identify and correct data errors, thereby reducing the risks caused by data inconsistency or inaccuracy and improving the accuracy and reliability of data before, during, and after data synchronization.

[0031] By further refining the classification of data synchronization requests and selecting different review conditions for different requests, it is possible to better meet the requirements of regulatory authorities and laws and regulations regarding the data synchronization process, thereby providing compliance and legal protection. By setting up a data synchronization anomaly handling mechanism, abnormal situations that occur during the data synchronization process can be detected in a timely manner, reducing the impact of data synchronization anomalies; it can also reduce the possibility of system crashes or shutdowns, ensuring the normal operation of the system.

[0032] For example, preset audit conditions may include any one or more of the following: data security, data integrity, data accuracy, data permissions, business compliance, and system load. The data source and target database of a data synchronization request will vary, and the preset review conditions will naturally differ. For example, for the first data synchronization request involving laboratory data, the sensitivity of the experimental data often needs to be considered, and higher review requirements will be set for data security and confidentiality. The accuracy of laboratory data is crucial to the credibility of scientific research and experimental results, so the preset review conditions will also focus more on the accuracy and completeness of the data. In addition, laboratory data may involve information such as patents and trade secrets, so the preset review conditions may focus more on data access control to ensure that only authorized personnel can access and operate the data.

[0033] The second data synchronization request involves testing standard data. The testing standard documents may involve legal regulations, industry standards, etc. The preset review conditions need to include compliance review to ensure that the synchronized data meets the requirements of relevant laws, regulations and industry standards. At the same time, the testing standard data may need to be updated regularly to maintain consistency with the latest standards. The review process may focus on the frequency and timeliness of document updates in the data synchronization request.

[0034] Of course, in some embodiments of this application, the type of data synchronization request also includes a third data synchronization request involving user requirements, such as customer requirements, customer limits, etc., to timely and clearly define customer needs and improve customer satisfaction.

[0035] Key fields refer to important fields or attributes used when matching, comparing, or mapping between source and target data. Key fields include, but are not limited to, primary keys, unique identifiers, timestamps, and composite fields.

[0036] Furthermore, the process of performing data verification on the data to be synchronized includes: a primary verification involving the data format and a secondary verification involving the data content; If a check fails once, the data to be synchronized will be preprocessed one or more times until a check passes once. The preprocessing includes any one or more of data cleaning, data conversion, and data formatting; When the first verification passes, a second verification is performed on the data to be synchronized. The second verification includes one or more of the following: data integrity verification, data accuracy verification, data consistency verification, and data quality verification.

[0037] Understandably, multiple verifications and preprocessing steps can improve data quality and accuracy, reduce data errors, ensure consistency in format and content between the data to be synchronized and the target data, and reduce data mismatch or inconsistency issues. Automated verification and preprocessing processes can reduce manual intervention, improve the efficiency and automation of data synchronization, and reduce the cost of subsequent data processing and adjustment by identifying and fixing data problems in advance, avoiding additional expenses caused by data quality issues.

[0038] For example, data format verification can use: Cyclic Redundancy Check (CRC), used to detect errors during data transmission or storage, mainly focusing on the formal structure of data blocks; Parity Check, used to detect single-bit errors during transmission, also falls under the category of data format verification.

[0039] For further examples, data content verification can use: hash values, which generate unique hash values ​​by hashing the data content to verify the integrity of the data and whether the content has changed; digital signatures, a data verification method based on asymmetric encryption algorithms, used to verify the source and integrity of the data, falling under the category of data content verification; comparing mirror data, comparing the content differences between the source data and the target data to ensure consistency in content; and checksums, which sum the bytes in a data block to detect errors during transmission and indirectly verify whether the data content is complete.

[0040] Of course, there are no restrictions on the specific data verification methods.

[0041] Furthermore, the process of determining the type of data synchronization request based on the data synchronization request includes: Parse the data synchronization request and extract key data synchronization information, which includes one or more of the source data identifier, target data identifier, synchronization frequency information, and synchronization strategy information. Analyze the source data identifier to obtain the first information, including the data source, nature and structure; analyze the target data identifier to obtain the second information, including the data purpose, structure and storage method; analyze the synchronization frequency information and synchronization strategy information to obtain the third information, including the synchronization type, synchronization priority and synchronization time. By comprehensively analyzing any multiple of the first, second, and third pieces of information, the type of data synchronization request can be determined.

[0042] Understandably, by parsing data synchronization requests, multiple aspects of data synchronization information can be obtained, and by combining any number of these data synchronization information, the accuracy of data synchronization request classification can be improved. Different processing methods are adopted for different types of data, and customized processing solutions are provided for different types of data synchronization requests to improve processing efficiency and accuracy. For example, pre-set audit conditions are set for different types of data synchronization requests. Different types of data synchronization requests may also have different requirements for system resources. By classifying them, resource allocation can be better optimized, so that critical data can be processed first, improving the overall performance of the system. If different types of data may conflict or be incompatible, classifying data synchronization requests makes it easier to identify and handle potential conflicts, reducing the risks in the synchronization process. Classified data synchronization requests are easier to manage and monitor, allowing the system to have a clearer understanding of the synchronization status of different types of data, facilitating the tracking and troubleshooting of problems.

[0043] Furthermore, the method also includes the generation of data synchronization requests. The monitoring involves a primary data source of laboratory data and a secondary data source of industry standard data. If changes are detected in the data from the first data source and the second data source, the changed data is collected to obtain the source data; Data synchronization information is determined based on the source data. This information includes the target data, synchronization method, and synchronization frequency. Verify whether the data synchronization request generation permission is met. If it is met, generate a data synchronization request corresponding to the source data.

[0044] Understandably, by monitoring changes in the data source and collecting source data in real time, the generated data synchronization requests can be matched with the latest data status, ensuring the real-time nature and accuracy of data synchronization. Automatically monitoring changes in the data source and generating data synchronization requests reduces the need for manual intervention and processing, improves the automation level of data synchronization, and lowers the possibility of errors. Determining key data synchronization information based on the source data and verifying compliance with data synchronization request generation permissions allows for the generation of customized data synchronization requests based on specific data conditions and permission requirements, meeting the needs of different data types and permission levels. Verifying data synchronization request generation permissions effectively controls the generation of data synchronization requests; only data changes that meet permission requirements will generate corresponding data synchronization requests, thereby improving data security and controllability. Generating data synchronization requests only when the data source changes avoids unnecessary data synchronization request generation, saving system resources and processing costs.

[0045] The primary and secondary data sources can exist in various forms. For example, they can be various types of databases, such as relational databases (e.g., MySQL, Oracle, SQL Server) and NoSQL databases (e.g., MongoDB, Cassandra). They can also be files or folders in a file system, including text files, log files, and configuration files. Data may also be exposed as an API interface, allowing users to retrieve and synchronize data by calling the API. Alternatively, applications may generate data output that can serve as a data source for synchronization. Examples include log data and report data from web applications. Sometimes, data synchronization requires interaction with third-party service platforms, and the data provided by these platforms can also serve as a data source.

[0046] Furthermore, the monitoring of the first data source is achieved through preset trigger events. When the data in the first data source changes, the insertion, update, and deletion operations of the data are captured. Monitoring of the second data source is achieved by periodically polling the second data source and detecting data changes by querying and comparing the data before and after the second data source.

[0047] Understandably, for the first data source, by using preset trigger events, it is possible to capture data insertion, update, and deletion operations in real time, thus achieving timely response to data changes. For the second data source, although it is not real-time, by using timed polling, it is possible to detect data changes within a certain time interval, thereby achieving a certain degree of data synchronization.

[0048] Preset-triggered events are relatively lightweight, enabling real-time monitoring without consuming excessive system resources. While timed polling consumes some system resources, it can meet the monitoring needs of infrequently changing data sources, thus balancing resource utilization. Preset-triggered events have a relatively small impact on the data source, monitoring only when the data changes, without placing an excessive burden on the data source itself. Although timed polling queries the data source periodically, its impact on the data source can be reduced by setting the query frequency appropriately.

[0049] Specifically, in some embodiments of this application, the first data source involves laboratory data, particularly data from testing projects. This type of data updates quickly and frequently, so monitoring of the first data source needs to be achieved through preset trigger events to ensure timely and effective data updates. The second data source involves data information from testing industry standards. The customization and updating of this type of data information requires a long process, the update speed is slow, and it is generally not easily changed. Therefore, it does not require such strong real-time performance. Periodic polling helps reduce the occupation of system resources. For example, the polling time can be set to once a day, once a week, or once a month. Of course, there are no restrictions on the specific time setting. The shorter the interval, the more conducive it is to timely updates. The specific time interval setting can also refer to the frequency of frequent updates of the second data source in the past.

[0050] Furthermore, the source data includes both static and dynamic data; Static data includes: industry testing standard data and basic laboratory data; Dynamic data includes: data from the testing items; Specifically, industry testing standard data includes: version validity, effective date, expiration date, applicable products, classification, and applicable standards; Basic laboratory data includes: equipment, accreditation, cycle time, price, sample quantity, detection limit, classification, description, laboratory capabilities, etc. The testing data includes: test items, test plans, basic testing clauses, testing method clauses, clause names, version validity, effective date, category, clause, name, English date, expiration date, test results, test data, etc.

[0051] Static data synchronization methods include full synchronization, while dynamic data synchronization methods include incremental synchronization and real-time synchronization.

[0052] The full synchronization method is a data synchronization strategy used to completely synchronize all data in the source data system to the target database, rather than just synchronizing the parts that have changed.

[0053] Incremental synchronization is a data synchronization strategy used to synchronize changes (additions, updates, deletions) occurring in the source data system to the target data system, rather than synchronizing all data completely each time. Real-time synchronization, on the other hand, is a data synchronization strategy designed to ensure that changes (additions, updates, deletions) in the source data system are synchronized to the target data system immediately, maintaining data real-time performance and consistency.

[0054] Static data refers to data that remains largely unchanged or changes very slowly over a period of time. This type of data is typically not frequently updated or modified. Static data can usually be acquired once and accessed and used multiple times within a certain period without requiring frequent updates. Dynamic data, on the other hand, refers to data that changes frequently within a certain timeframe. This type of data is usually real-time or near real-time and is constantly updated over time. It needs to be acquired in real-time or periodically and processed and updated promptly to maintain its accuracy and timeliness.

[0055] The full synchronization method can ensure that all static data is completely synchronized to the target system, reducing the possibility of missing any information. Moreover, the full synchronization method is relatively simple and easy to implement. It only requires one-time data synchronization at the appropriate time, without considering incremental changes in data. Since static data changes little or not at all, the full synchronization method is more stable and reliable while ensuring data consistency.

[0056] Incremental and real-time synchronization methods can save data transmission and processing resources by synchronizing only the changed data instead of transmitting all data each time, thus maintaining data real-time performance and keeping the data in the target system synchronized with the data in the source system. Incremental and real-time synchronization methods can also avoid a large amount of repetitive data transmission, reducing redundant data storage. Moreover, incremental and real-time synchronization methods are more flexible than full synchronization methods and can be dynamically adjusted according to business needs and data characteristics.

[0057] Furthermore, such as Figure 2 As shown, the process of handling exceptions according to the preset exception handling steps includes: By setting up scheduled monitoring tasks, we can continuously monitor for anomalies during the data synchronization process and scan and capture any possible error messages in real time. By conducting scheduled monitoring tasks, abnormal situations that occur during data synchronization can be captured in a timely manner, allowing for immediate responses when problems arise and reducing the impact of data synchronization anomalies.

[0058] When an anomaly is detected in the data synchronization, a synchronization anomaly log including error information is generated. Perform an exception handling operation, and synchronize the data to be synchronized one or more times according to the preset number of synchronizations and time intervals; For example, in some embodiments of this application, the preset number of synchronizations is 3 times and the time interval is 30 minutes. After the automatic exception handling has been performed 3 times, it indicates that the exception cannot be handled by repeated operations or the handling method is incorrect, and the exception handling is stopped immediately.

[0059] Generating a synchronous exception log can record exception information, which helps in subsequent analysis of the cause of the exception and problem solving, thereby improving the maintainability of the system and the efficiency of troubleshooting.

[0060] If one exception handling fails, a second exception handling is performed to handle the error corresponding to the error message. After the handling is completed, the data to be synchronized is synchronized one or more times again. In an exception handling process, performing one or more data synchronizations on the data to be synchronized can resolve some exceptions caused by network problems, increase the chances of successful data synchronization, and improve the success rate of data synchronization.

[0061] If the secondary exception handling fails, a synchronous exception alarm message will be triggered.

[0062] In secondary anomaly handling, processing the errors corresponding to the error messages helps to solve the root problem and thus prevent similar anomalies from recurring. When secondary anomaly handling fails, triggering synchronous anomaly alarm information can promptly notify relevant personnel, prompting them to take further action and prevent the problem from escalating further.

[0063] By using preset synchronization counts and time intervals to incrementally process anomalies, the processing intensity can be gradually increased, improving the efficiency of anomaly handling and avoiding waste of processing resources.

[0064] The first instance of exception handling primarily addresses issues that can be resolved by repeating the process, such as: network problems (including network latency, disconnections, instability, etc., affecting data transmission); data format mismatch (inconsistent or incompatible data formats between the source and target systems, leading to incorrect data parsing or import); excessive data volume (data volume exceeding system processing capacity, causing some data to fail to synchronize or synchronizing slowly); data conflicts (multiple synchronization sources simultaneously modifying the same data, causing conflicts that need to be resolved); data latency (delays during data synchronization, causing data to not be immediately synchronized to the target system after an update in the source system); inconsistent environment configurations (inconsistent environment configurations between the source and target systems, such as database versions, operating system versions, etc., affecting normal data synchronization); concurrency issues (in high-concurrency situations, multiple synchronization tasks running simultaneously may lead to resource contention, locking, and other problems, causing data synchronization anomalies); and data loss or corruption (data loss or corruption may occur during data synchronization, such as network transmission errors or packet loss).

[0065] Secondary anomaly handling primarily addresses issues that the system can handle automatically, such as: Permission issues: users lack sufficient permissions to perform data synchronization operations; Software failure: malfunctions in the target system or synchronization tool software lead to data synchronization anomalies; Data quality issues: poor source data quality, including missing, incomplete, or inconsistent data, causes synchronization errors; Hardware failure: target system hardware failures, such as disk damage or memory failure, affect normal data synchronization; Configuration errors: incorrect synchronization task configurations, such as incorrect mapping relationships or improper scheduled task settings, lead to synchronization failure; Human error: human error, such as accidental data deletion or misconfiguration, causes data synchronization anomalies; System upgrades / updates: upgrades or updates to the source or target system may cause interface changes, configuration changes, or software version incompatibility, leading to data synchronization anomalies; Data security requirements: target systems with high data security requirements require data encryption or data anonymization, which may increase the complexity and risk of anomalies in data synchronization; Changes in external system interfaces: data synchronization involves external systems, and changes in external system interfaces or malfunctions in the external system itself may cause data synchronization anomalies.

[0066] Of course, there are no restrictions on the specific problems that can be solved by single-stage or double-stage exception handling.

[0067] Furthermore, before synchronizing the data to be synchronized that has passed data verification to the target data in the target database, the method also includes: Simulated data synchronization synchronizes the data to be synchronized to the intermediate target data in the intermediate database, which is generated based on the target database mapping. Check if there are any problems with the intermediate target data after synchronization. If there are no problems, synchronize the data to be synchronized through the check to the target data in the target database.

[0068] Specifically, in some embodiments of this application, the synchronization of the data to be synchronized to the target data in the target database by verifying the data to be synchronized can be achieved by mapping the intermediate target data in the intermediate database to the target data in the target database. The specific implementation of the mapping can be achieved through foreign keys.

[0069] Understandably, by simulating synchronization and checking on an intermediate database, the impact on the target database can be minimized, reducing the risks associated with synchronization errors. Simulated synchronization and data checks on the intermediate database ensure the integrity and consistency of the data to be synchronized, preventing data corruption or loss and preemptively eliminating potential errors. The existence of the intermediate database also reduces the need for direct synchronization operations on the target database, effectively mitigating resource consumption. Even if errors occur during data synchronization on the intermediate database, they will not affect the target data, and data synchronization is reversible. Simulated synchronization and data checks on the intermediate database make it easier to identify and repair problems that arise during the synchronization process, thereby improving the stability and reliability of data synchronization.

[0070] In computer programming, mapping is often used to describe the process of mapping elements or attributes in one data structure to another data structure.

[0071] Specifically, in some embodiments of this application, the process of creating an intermediate database through mapping includes: analyzing the target database structure: First, the target database needs to be structurally analyzed, including information such as tables, columns, indexes, and constraints. This can be done using database management tools or SQL queries to obtain database structure information. Creating an intermediate database: A new database is created in the target system to store the structure and data copied from the target database. Mapping the database structure: The table structure, column definitions, indexes, constraints, and other information in the target database are mapped to the intermediate database. This can be done by executing DDL statements to create the same structure in the intermediate database. Data mapping: Data in the target database is mapped to the intermediate database. This can be achieved using ETL tools or by writing SQL scripts to accurately copy data from the source database to the target database. Verification and testing: After mapping is completed, verification and testing are required to ensure that the structure and data in the intermediate database are completely consistent with the target database.

[0072] Furthermore, the method also includes: When multiple data synchronization requests are received at the same time, the system first checks whether the data synchronization resources of the detection data platform are sufficient. If the data synchronization resources are sufficient, all data synchronization requests are responded to at the same time for data synchronization processing. If data synchronization resources are insufficient, check if there is a priority field in each data synchronization request. If there is a priority field in each data synchronization request, prioritize processing the data synchronization requests with priority fields. If none of the data synchronization requests contain a priority field, then the synchronization will be processed in the order in which the data synchronization requests were generated. When the data synchronization request is the second data synchronization request, data synchronization of the first data source is also required.

[0073] Understandably, by determining whether the data synchronization resources of the detection data platform are sufficient, it can be ensured that the data synchronization tasks are fully utilized and that multiple data synchronization requests can be responded to simultaneously, which helps to improve the efficiency and response speed of data synchronization. When data synchronization resources are insufficient, the processing priority is determined according to the priority field in each data synchronization request, which can more flexibly meet the priority requirements of different data synchronization tasks and ensure that important data is processed in a timely manner. If there is no priority field in any data synchronization request, the synchronization is performed in the order of generation, which ensures the sequentiality of data synchronization and avoids possible data errors or chaos.

[0074] For the second data synchronization request, the first data source also needs to be synchronized. This ensures the integrity and consistency of the data and avoids omissions or errors during the data synchronization process. Testing standards play an important role in the formulation of experimental testing methods and standards. Through secondary data synchronization, timely updates to laboratory testing standards can be achieved, which helps the testing projects proceed smoothly.

[0075] The embodiments of this application have at least the following beneficial effects: This application provides a data synchronization method for a testing data platform, which optimizes the data synchronization method for testing enterprises' testing data platforms. Specifically, it classifies received data synchronization requests and performs differentiated audits for different types of data synchronization requests, enabling refined management of the data synchronization process. Through data verification and anomaly alerts, it improves the integrity, correctness, and security of synchronized data. Generating synchronization anomaly logs helps track and analyze problems that occur during the synchronization process, providing a basis for subsequent problem investigation and handling, thereby improving the manageability and traceability of the data synchronization process.

[0076] Example 2 This application provides a system for implementing detection data management and synchronization, such as... Figure 3 As shown, the system includes: Receiving module 1 is used to respond to received data synchronization requests and determine the type of data synchronization request based on the data synchronization request. The types of data synchronization requests include a first data synchronization request involving laboratory data and a second data synchronization request involving testing standard data. Synchronization request review module 2 is used to select different preset review conditions based on different types of data synchronization requests and determine whether the data synchronization request meets the preset review conditions. If the data synchronization request does not meet the requirements, it will be rejected or canceled; if it does meet the requirements, the data source and target data corresponding to the data synchronization request will be determined. Data synchronization module 3 is used to acquire source data as data to be synchronized through data transmission and to perform data verification on the data to be synchronized; the data to be synchronized that has passed the data verification is synchronized to the target data in the target database. The target database may include one or more databases, and different target databases are related to each other through key fields. The exception handling module 4 is used to handle exceptions according to preset exception handling steps when data synchronization fails.

[0077] The rest is the same as in Example 1, and will not be repeated here.

[0078] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0079] The serial numbers of the present invention mentioned above are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenarios.

[0080] The above-disclosed examples are only a few specific implementation scenarios of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A data synchronization method for a detection data platform, characterized in that, include: In response to a received data synchronization request, the type of the data synchronization request is determined based on the data synchronization request. The type of the data synchronization request includes a first data synchronization request involving laboratory data and a second data synchronization request involving testing standard data. Based on the type of data synchronization request, select preset review conditions and determine whether the data synchronization request meets the preset review conditions; If it does not meet the requirements, the data synchronization request will be rejected or cancelled. If they match, then the source data and target data corresponding to the data synchronization request are determined. The source data is obtained as the data to be synchronized through data transmission, and the data to be synchronized is verified. The data to be synchronized that has passed the data verification is synchronized to the target data in the target database. The target database includes one or more databases, and different target databases are related to each other through key fields. When data synchronization fails, exception handling is performed according to the preset exception handling steps; The process of performing data verification on the data to be synchronized includes: a primary verification involving the data format and a secondary verification involving the data content; If the first verification fails, the data to be synchronized is preprocessed one or more times until the first verification passes. The preprocessing includes any one or more of data cleaning, data conversion, and data formatting; When the first verification passes, a second verification is performed on the data to be synchronized. The second verification includes one or more of the following: data integrity verification, data accuracy verification, data consistency verification, and data quality verification. The process of handling exceptions according to preset exception handling steps includes: By setting up scheduled monitoring tasks, we can continuously monitor for anomalies during the data synchronization process and scan and capture any possible error messages in real time. When an anomaly is detected in the data synchronization, a synchronization anomaly log including the error information is generated; An exception is handled, and the data to be synchronized is synchronized one or more times according to the preset number of synchronizations and time intervals. If the first exception handling fails, a second exception handling is performed to process the error corresponding to the error message. After the processing is completed, the data to be synchronized is synchronized one or more times again. If the secondary anomaly handling fails, a synchronous anomaly alarm message will be triggered. Before synchronizing the data to be synchronized that has passed data verification into the target data in the target database, the method further includes: Simulated data synchronization involves synchronizing the data to be synchronized to the intermediate target data in an intermediate database, wherein the intermediate database is generated based on the target database mapping. Check if there are any problems with the intermediate target data after synchronization. If there are no problems, then synchronize the data to be synchronized through the check to the target data in the target database.

2. The data synchronization method for a detection data platform according to claim 1, characterized in that, The process of determining the type of the data synchronization request based on the data synchronization request includes: Parse the data synchronization request and extract key data synchronization information, which includes one or more of the following: source data identifier, target data identifier, synchronization frequency information, and synchronization strategy information. The source data identifier is analyzed to obtain first information including data source, nature and structure; the target data identifier is analyzed to obtain second information including data purpose, structure and storage method; and the synchronization frequency information and the synchronization strategy information are analyzed to obtain third information including synchronization type, synchronization priority and synchronization time. By comprehensively analyzing any and multiple of the first information, the second information, and the third information, the type of the data synchronization request is determined.

3. The data synchronization method for a detection data platform according to claim 1, characterized in that, The method also includes generating a data synchronization request. The monitoring involves a primary data source of laboratory data and a secondary data source of industry standard data. If changes are detected in the data from the first data source and the second data source, the changed data is collected to obtain the source data. Data synchronization information is determined based on the source data, and the data synchronization information includes target data, synchronization method, and synchronization frequency. Verify whether the data synchronization request generation permission is met. If it is met, generate a data synchronization request corresponding to the source data.

4. The data synchronization method for a detection data platform according to claim 3, characterized in that, Monitoring of the first data source is achieved through preset trigger events. When the data in the first data source changes, the insertion, update, and deletion operations of the data are captured. Monitoring of the second data source is achieved by periodically polling the second data source and detecting data changes by querying and comparing the data before and after the second data source.

5. The data synchronization method for a detection data platform according to claim 1, characterized in that, The source data includes static data and dynamic data; The static data includes: industry testing standard data and basic laboratory data; The dynamic data includes: detection item data; The data synchronization methods for static data include a full synchronization method, and the data synchronization methods for dynamic data include an incremental synchronization method and a real-time synchronization method.

6. The data synchronization method for a detection data platform according to claim 1, characterized in that, The method also includes: When multiple data synchronization requests are received simultaneously, the system first checks whether the data synchronization resources of the detection data platform are sufficient. If the data synchronization resources are sufficient, all the data synchronization requests are responded to simultaneously for data synchronization processing. If data synchronization resources are insufficient, check whether there is a priority field in each of the data synchronization requests. If there is a priority field in each of the data synchronization requests, process the data synchronization requests with priority fields first. If none of the data synchronization requests contain a priority field, then the synchronization will be performed in the order in which the data synchronization requests were generated. When the data synchronization request is a second data synchronization request, data synchronization of the first data source is also required.

7. A system for implementing detection data management and synchronization, used to implement the detection data platform data synchronization method according to any one of claims 1 to 6, characterized in that, The implementation system includes: A receiving module is used to respond to a received data synchronization request and determine the type of the data synchronization request based on the data synchronization request. The type of the data synchronization request includes a first data synchronization request involving laboratory data and a second data synchronization request involving testing standard data. The synchronization request review module is used to select different preset review conditions based on different types of data synchronization requests, and determine whether the data synchronization request meets the preset review conditions. If the conditions are not met, the data synchronization request is rejected or cancelled; if the conditions are met, the source data and target data corresponding to the data synchronization request are determined. The process of performing data verification on the data to be synchronized includes: a primary verification involving the data format and a secondary verification involving the data content; If the first verification fails, the data to be synchronized is preprocessed one or more times until the first verification passes. The preprocessing includes any one or more of data cleaning, data conversion, and data formatting; When the first verification passes, a second verification is performed on the data to be synchronized. The second verification includes one or more of the following: data integrity verification, data accuracy verification, data consistency verification, and data quality verification. The data synchronization module is used to acquire the source data as data to be synchronized via data transmission and to perform data verification on the data to be synchronized; to synchronize the data to be synchronized that has passed the data verification to the target data in the target database, wherein the target database includes one or more databases and different target databases are related to each other through key fields; before synchronizing the data to be synchronized that has passed the data verification to the target data in the target database, a simulated data synchronization is performed to synchronize the data to be synchronized to the intermediate target data in the intermediate database, wherein the intermediate database is generated based on the target database mapping; Check if there are any problems with the intermediate target data after synchronization. If there are no problems, then synchronize the data to be synchronized that has passed the check to the target data in the target database. An exception handling module is used to handle exceptions according to preset exception handling steps when data synchronization fails. The process of handling exceptions according to preset exception handling steps includes: continuously monitoring abnormal situations during data synchronization by setting a timed monitoring task, scanning and capturing possible error information in real time; when an exception is detected in data synchronization, generating a synchronization exception log including the error information; performing one exception handling operation, synchronizing the data to be synchronized one or more times according to a preset number of synchronization attempts and time intervals; if the first exception handling operation fails, performing a second exception handling operation, processing the error corresponding to the error information, and then synchronizing the data to be synchronized one or more times again after processing; if the second exception handling operation fails, triggering a synchronization exception alarm.

Images