Data synchronization method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly discloses a data synchronization method and device, computer equipment and a storage medium. The target transmission channel and the target transmission mode can be dynamically determined according to the actual scale, the transmission requirements and the real-time transmission environment of the to-be-synchronized data, the data transmission channel and the transmission mode are reasonably utilized, the optimal transmission channel and the optimal transmission mode can be adopted for the to-be-synchronized data with different scales and transmission requirements, and the transmission efficiency is improved. And the data synchronization efficiency is improved. When the method is applied to financial management in the field of financial services, the optimal transmission channel and transmission mode can be determined according to the scale, the transmission requirement and the real-time transmission environment of the financial data, and the synchronization efficiency of the data among different financial systems is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data synchronization method, apparatus, computer equipment, and storage medium. Background Art

[0002] Financial management in the financial sector requires data synchronization across multiple financial systems, departments, and distributed storage environments to ensure the consistency and timeliness of financial information. However, as financial enterprises expand in size and their businesses continue to expand, the volume of financial data is exploding, including massive transaction records, complex financial statements, numerous accounting entries, and business-related financial information. Traditional data synchronization methods suffer from slow data transmission speeds and often experience long transmission delays when synchronizing large amounts of financial data. This results in low data synchronization efficiency and an inability to promptly respond to enterprises' needs for real-time financial information. Therefore, improving data synchronization efficiency has become a pressing issue. Summary of the Invention

[0003] The present application provides a data synchronization method, apparatus, computer equipment and storage medium to improve data synchronization efficiency.

[0004] In a first aspect, the present application provides a data synchronization method, the method comprising:

[0005] Acquire data to be synchronized and the data size and transmission requirements of the data to be synchronized;

[0006] detecting a real-time transmission environment, and determining a target transmission channel and a target transmission mode based on the real-time transmission environment, the data scale, and the transmission requirements;

[0007] Based on the target transmission channel and the target transmission mode, the data to be synchronized is transmitted to synchronize the data to a target storage location.

[0008] In a second aspect, the present application further provides a data synchronization device, the device comprising:

[0009] A data acquisition module is used to acquire the data to be synchronized and the data size and transmission requirements of the data to be synchronized;

[0010] a configuration determination module, configured to detect a real-time transmission environment and determine a target transmission channel and a target transmission mode based on the real-time transmission environment, the data scale, and the transmission requirements;

[0011] The data transmission module is configured to transmit the data to be synchronized based on the target transmission channel and the target transmission mode, so as to synchronize the data to be synchronized to a target storage location.

[0012] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the data synchronization method as described above when executing the computer program.

[0013] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the data synchronization method as described above.

[0014] The present application discloses a data synchronization method, apparatus, computer equipment, and storage medium, which obtain the data to be synchronized and the data scale and transmission requirements of the data to be synchronized; detect the real-time transmission environment, and determine the target transmission channel and target transmission mode based on the real-time transmission environment, the data scale, and the transmission requirements; based on the target transmission channel and the target transmission mode, transmit the data to be synchronized to the target storage location. The present application can dynamically determine the target transmission channel and target transmission mode based on the actual scale, transmission requirements, and real-time transmission environment of the data to be synchronized, reasonably utilize the data transmission channel and transmission mode, and can adopt the optimal transmission channel and transmission mode for data to be synchronized of different scales and transmission requirements, thereby improving the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 is a schematic flow chart of a data synchronization method provided in the first embodiment of the present application;

[0017] Figure 2 is a schematic flow chart of a data synchronization method provided in the second embodiment of the present application;

[0018] Figure 3 is a schematic flow chart of a data synchronization method provided in the third embodiment of the present application;

[0019] Figure 4 A schematic block diagram of a data synchronization device provided in an embodiment of the present application;

[0020] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0023] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] It should be further understood that the term “and / or” used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0025] Embodiments of the present application provide a data synchronization method, apparatus, computer device, and storage medium. The data synchronization method can be applied to servers and data synchronization systems. Based on the actual size of the data to be synchronized, transmission requirements, and real-time transmission environment, the target transmission channel and target transmission mode are dynamically determined. The data transmission channel and transmission mode are rationally utilized, and the optimal transmission channel and transmission mode can be adopted for data to be synchronized of different sizes and transmission requirements, thereby improving the efficiency of data synchronization. The server can be a standalone server or a server cluster. The data synchronization system includes a data perception and acquisition module, which is used to automatically identify and connect various financial data sources and collect data using different acquisition methods based on the characteristics and importance of the data; a data preprocessing module is used to clean, convert and compress the collected data to obtain the data to be synchronized; a data transmission module is used to select appropriate transmission channels and transmission modes based on the scale, real-time requirements and network environment of the data to be synchronized, and to use multi-layer encryption and authentication technologies to ensure the security of the data to be synchronized; a data storage and synchronization module is used to store data using a distributed storage system, and through data sharding and replication mechanisms, the data is dispersed and stored on multiple storage nodes, and a distributed consistency algorithm is used to ensure the consistency of data between multiple storage nodes and different systems; a monitoring and feedback module is used to monitor various performance indicators in the data synchronization process in real time, comprehensively monitor the quality of the synchronized financial data, check the accuracy, completeness, consistency and timeliness of the data, and automatically generate system optimization suggestions and provide feedback based on the monitoring results.

[0026] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0027] See also Figure 1 , Figure 1 This is a schematic flow chart of a data synchronization method provided in an embodiment of the present application. This data synchronization method can be applied to servers and data synchronization systems. It is used to dynamically determine the target transmission channel and target transmission mode based on the actual scale of the data to be synchronized, the transmission requirements, and the real-time transmission environment. It rationally utilizes the data transmission channel and transmission mode, and can adopt the optimal transmission channel and transmission mode for data to be synchronized with different scales and transmission requirements, thereby improving the efficiency of data synchronization.

[0028] like Figure 1 As shown, the data synchronization method specifically includes steps S101 to S103.

[0029] S101, obtaining data to be synchronized and the data size and transmission requirements of the data to be synchronized;

[0030] In one embodiment, various data sources are automatically identified and connected through the data perception and collection module, including but not limited to the company's internal financial systems (such as general ledger systems, detailed ledger systems, financial analysis systems), business-related systems (such as sales systems, procurement systems, inventory systems) and external financial institution interfaces.

[0031] Acquire the data to be synchronized from the identified data sources, and calculate and record the scale of this data, including data volume, number of data files, number of data records, etc. Also determine data transmission requirements, such as real-time requirements (whether real-time transmission is required, maximum allowable delay time, etc.), reliability requirements (whether data must not be lost, erroneous, or duplicated during transmission, etc.), security requirements (whether encrypted transmission is required, etc.), and cost constraints (whether bandwidth costs and cloud computing service costs during transmission are within the budget).

[0032] S102: Detecting a real-time transmission environment, and determining a target transmission channel and a target transmission mode based on the real-time transmission environment, the data scale, and the transmission requirements;

[0033] In one embodiment, a detailed analysis of the current real-time transmission environment is performed, including key indicators such as network bandwidth, delay, and packet loss rate, to evaluate the stability and transmission capacity of the network.

[0034] Comprehensively analyze the acquired real-time transmission environment information, data size, and transmission requirements, and compare and match them with the preset transmission channel and transmission mode mapping table. Based on the mapping table and analysis results, select the transmission channel and transmission mode that best suits the current situation. For example, for batch transmission of large amounts of data with sufficient and stable network bandwidth, an internal high-speed fiber-optic network channel can be selected. For scenarios with high real-time requirements, small data volumes, and average network conditions, a high-speed network connection based on cloud computing can be selected in combination with appropriate optimization techniques. For transaction data with extremely high real-time requirements, a real-time transmission mode can be used. For non-critical, periodically generated data, a batch transmission mode can be used to transmit during low-peak business periods.

[0035] Furthermore, determining the target transmission channel and the target transmission mode based on the real-time transmission environment, the data scale and the transmission requirements includes: obtaining a preset transmission channel and transmission mode mapping table; based on the real-time transmission environment, the data scale and the transmission requirements, searching the transmission channel and the target transmission mode corresponding to the real-time transmission environment, the data scale and the transmission requirements from the transmission channel and transmission mode mapping table.

[0036] In one embodiment, a transmission channel and transmission mode mapping table is established based on historical data synchronization records, covering different transmission environments, data sizes, transmission requirements, etc. The mapping table provides the system with a basis for selecting the appropriate transmission channel and mode in various situations.

[0037] The complexity and resources required for data transmission are determined based on the scale of the data to be synchronized (e.g., data volume, number of data files, etc.). Furthermore, the data transmission requirements are considered, including but not limited to real-time requirements (e.g., whether real-time transmission is required, the permissible latency, etc.) and reliability requirements (e.g., whether data integrity and lossless transmission must be ensured). The analyzed real-time transmission environment, data scale, and transmission requirements are then compared with the entries in the transmission channel and transmission mode mapping table to find the transmission channel and mode combination that best suits the current situation.

[0038] For example, appropriate transmission channels are selected based on the scale of financial data, real-time requirements, and network environment. For synchronizing large amounts of historical financial data, high-bandwidth channels within the enterprise network (such as fiber optic networks) can be used for batch transmission. For financial data with extremely high real-time requirements (such as online payment data), low-latency dedicated network channels or high-speed cloud-based network connections are preferred. Furthermore, network traffic monitoring and adaptive adjustment technologies can be used to optimize bandwidth allocation within transmission channels and improve transmission efficiency.

[0039] S103: Transmit the data to be synchronized based on the target transmission channel and the target transmission mode, so as to synchronize the data to be synchronized to a target storage location.

[0040] In one embodiment, the data to be synchronized is transmitted to the target storage location via the target transmission channel and target transmission mode. During the transmission process, the system monitors the transmission status and performance indicators in real time, such as transmission speed, bandwidth utilization, and data integrity, to ensure the stability and reliability of data transmission. Based on the monitoring results and adaptive adjustment technology, the bandwidth allocation of the transmission channel is optimized to improve transmission efficiency.

[0041] In another embodiment, to ensure data security during transmission, multi-layer encryption and authentication technologies are employed. At the transport layer, the SSL (Secure Sockets Layer) / TLS (Transport Layer Security) protocol is used to encrypt and protect data integrity, preventing theft or tampering during transmission. Digital certificates and two-way authentication mechanisms are combined to rigorously verify the identities of both parties involved in the transmission, ensuring that only authorized systems and users can participate in data transmission. For particularly sensitive financial data, additional encryption algorithms can be employed at the application layer for secondary encryption.

[0042] The above embodiments provide a data synchronization method, apparatus, computer equipment, and storage medium, which obtain the data to be synchronized and the data scale and transmission requirements of the data to be synchronized; detect the real-time transmission environment, and determine the target transmission channel and target transmission mode based on the real-time transmission environment, the data scale, and the transmission requirements; based on the target transmission channel and the target transmission mode, transmit the data to be synchronized to the target storage location. The present application can dynamically determine the target transmission channel and target transmission mode based on the actual scale, transmission requirements, and real-time transmission environment of the data to be synchronized, reasonably utilize the data transmission channel and transmission mode, and can use the optimal transmission channel and transmission mode for data to be synchronized of different scales and transmission requirements, thereby improving the efficiency of data synchronization.

[0043] See also Figure 2 , Figure 2 This is a schematic flow chart of a data synchronization method provided in an embodiment of the present application. This data synchronization method can be applied to servers and data synchronization systems. Through technical means such as asynchronous transmission, real-time monitoring, and policy adjustment, it effectively addresses the shortcomings of traditional data synchronization methods in terms of transmission efficiency, adaptability, and reliability. It can better meet the needs of enterprises when processing large-scale financial data synchronization and improve the efficiency and accuracy of data synchronization.

[0044] like Figure 2 As shown, the data synchronization method specifically includes steps S201 to S203.

[0045] S201: When the target transmission mode is an asynchronous transmission mode, transmitting the data to be synchronized to a local cache area of ​​a data source device based on the target transmission channel;

[0046] S202: Obtain a preset data transmission strategy, call a data transmission thread to transmit the data in the local cache area according to the data transmission strategy, and monitor various performances during the transmission process to obtain performance indicators;

[0047] S203: Based on a preset performance threshold and the performance indicator, the data transmission strategy is adjusted in real time until the data to be synchronized is transmitted to the target storage location.

[0048] In one embodiment, when the data transmission mode is the asynchronous transmission mode, it means that the data can be first stored in the cache areas corresponding to the data source device and the data receiving device, and then subsequently transmitted according to the data transmission strategy.

[0049] The data to be synchronized is sent to the local cache area of ​​the data source device via the selected target transmission channel. The local cache area is typically located in the data source device's memory or local disk and is used to temporarily store data to be transmitted. For example, in an enterprise financial system, transaction data to be synchronized is first written to the local cache area, awaiting subsequent asynchronous transmission processing.

[0050] Get the preset data transmission policy from the configuration file or database. The policy includes but is not limited to transmission priority rules (such as high-priority data is transmitted first), transmission interval (such as transmission every 5 minutes), data batch size (such as 100 records per transmission), etc.

[0051] Based on the acquired transmission policy, a dedicated data transmission thread is invoked. Following the data transmission policy, the thread reads data from the local cache one by one and transmits it to the data receiving device. For example, if the policy specifies transmitting 10 records at a time, the thread will retrieve 10 records from the cache at a time for transmission. The data transmission thread is an independent execution unit in the data synchronization system responsible for executing data transmission tasks.

[0052] In one embodiment, during the data transmission process, various performance indicators of the data synchronization process are monitored in real time, such as data transmission speed (the amount of data transmitted per second), transmission bandwidth utilization (the ratio of actual used bandwidth to available bandwidth), transmission delay (the time required for data to travel from the sender to the receiver), CPU and memory usage (the occupation of system resources during data transmission), etc.

[0053] The monitored performance indicators are compared with the preset performance thresholds. If the performance indicators exceed or fall below the corresponding thresholds, the data transmission strategy is adjusted accordingly and the performance indicators are continuously monitored to ensure that the adjusted strategy effectively solves the problem. This process continues until all the data to be synchronized is successfully transferred to the target storage location and the performance indicators remain within a reasonable range.

[0054] In another embodiment, the overall performance and data quality of the data synchronization system may be evaluated regularly to provide a basis for continuous optimization of the system.

[0055] In the above embodiments, through technical means such as asynchronous transmission, real-time monitoring and policy adjustment, the shortcomings of traditional data synchronization methods in transmission efficiency, adaptability and reliability are effectively solved, which can better meet the needs of enterprises in processing large-scale financial data synchronization and improve the efficiency and accuracy of data synchronization.

[0056] Furthermore, the data transmission strategy is adjusted in real time based on the preset performance threshold and the performance indicator, including: analyzing the performance indicator and the performance threshold to obtain performance optimization suggestions; generating alarm information based on the performance indicator and the performance optimization suggestions, and feeding back the alarm information to the user; receiving the transmission configuration information set by the user based on the alarm information, and adjusting the data transmission strategy based on the transmission configuration information.

[0057] In one embodiment, the monitored performance indicators are compared with preset performance thresholds. Performance thresholds are set based on system design requirements and business needs and serve as benchmarks for determining whether performance is normal. For example, the transmission delay threshold may be set at 100 milliseconds, and the bandwidth utilization threshold may be set at 80%.

[0058] By comparing performance indicators, if they exceed or fall below corresponding thresholds, possible performance bottlenecks will be identified. For example, if CPU (Central Processing Unit) utilization is consistently above the 90% threshold, it may indicate insufficient system resources, affecting data transmission efficiency. If transmission delays frequently exceed the set threshold, it may be due to poor network conditions.

[0059] Based on the identified performance bottlenecks, corresponding performance optimization suggestions are generated. For example, for high CPU usage, suggestions can be made to optimize data processing algorithms, increase server resources, or implement load balancing. For high transmission latency, suggestions can be made to check network connections, optimize transmission paths, or upgrade network equipment.

[0060] In one embodiment, based on performance indicators and optimization suggestions, alerts are generated containing details of performance indicator anomalies, optimization suggestions, and more. This is intended to clearly inform users of current system issues, possible causes, and suggested solutions. Specifically, a performance dashboard and alert platform can intuitively display the operational status of the data synchronization system. When performance indicators exceed normal performance thresholds, timely alerts are issued to prompt users to troubleshoot and optimize the system.

[0061] In a specific embodiment, the alarm information can be promptly fed back to the user through user interface display, email notification, SMS reminder, etc. In the user interface, the alarm information may be highlighted in the form of a pop-up window, a flashing icon, etc.; emails and SMS messages will send detailed alarm content to ensure that the user can receive and view it in time.

[0062] In one embodiment, the transmission configuration information submitted by the user through the configuration interface is received and verified to ensure the integrity and correctness of the configuration information. For example, the transmission time window set by the user is checked to see if it is reasonable, and whether the selected transmission channel is available.

[0063] Based on the transmission configuration information set by the user, the system will adjust the data transmission strategy accordingly. For example, if the user adjusts the transmission time window, the system will change the data transmission schedule accordingly to avoid large-scale data transmission during peak business hours, thereby reducing the impact on business systems. If the user changes the transmission channel, the system will re-establish the connection with the new transmission channel and optimize the relevant transmission parameters.

[0064] Apply the adjusted data transmission strategy to the actual data transmission process and continuously monitor performance indicators to evaluate the effectiveness of the adjustment. If performance improves, the adjustment strategy is effective. If performance issues persist or new problems arise, further analysis and optimization will be carried out.

[0065] In the above embodiment, real-time monitoring of performance indicators during data transmission can accurately identify performance bottlenecks in the data transmission process and timely adjust the data transmission strategy so that the data transmission strategy can better adapt to the actual transmission scenario, thereby improving data transmission efficiency and reliability.

[0066] Furthermore, the data to be synchronized is transmitted based on the target transmission channel and the target transmission mode to synchronize the data to be synchronized to the target storage location, including: based on the target transmission channel and the target transmission mode, the data to be synchronized is transmitted to the receiving cache area of ​​the data receiving end device; the data to be synchronized is decompressed in the receiving cache area to obtain the synchronized data; based on the preset data quality detection rules, the quality detection is performed on the synchronized data to obtain the detection result; when the detection result is normal, the synchronized data is written to the target storage location for storage.

[0067] In one embodiment, the data to be synchronized is sent to a receiving buffer area of ​​a data receiving device via a target transmission channel in a target transmission mode. The receiving buffer area is typically located in the receiving device's memory or local disk and is used to temporarily store received data. Within the receiving buffer area, the compressed data to be synchronized is decompressed using the same algorithm used for data compression, restoring the data's original size and format to obtain the synchronized data.

[0068] In one embodiment, data quality detection rules are used to identify anomalies in synchronized data, including but not limited to data accuracy rules for checking whether the data conforms to business logic and numerical ranges, for example, checking whether the amount in the financial transaction record is positive, whether the date is within a reasonable range, etc.; data integrity rules are used to verify whether the data is missing key fields or records, for example, checking whether each transaction record contains necessary information such as transaction number, transaction amount, transaction time, etc.; data consistency rules are used to ensure that data remains consistent between different systems or storage locations, for example, checking whether the same transaction records in the general ledger system and the detailed ledger system are consistent; data timeliness rules are used to verify whether the data is updated in a timely manner or within a valid time range, for example, checking whether real-time transaction data is synchronized within the specified time window.

[0069] Through data verification and comparison algorithms, logical rule checks and other methods, a comprehensive quality check is performed on the synchronized data in accordance with the preset data quality detection rules to check the accuracy, completeness, consistency and timeliness of the data, and the results of the quality test are recorded, including the data that passed the test, the data that failed the test and their specific abnormalities. Specifically, the consistency check can use a distributed consistency algorithm to ensure the consistency of financial data across multiple storage nodes and different data systems. During the data synchronization process, concurrent updates and conflicts of data are handled through version control and conflict detection mechanisms. When a data conflict is detected, it is handled according to the preset conflict resolution strategy (such as based on timestamps, based on data source priority) to ensure the ultimate consistency of financial data.

[0070] In one embodiment, if the detection result shows that there is no abnormality in the synchronized data, that is, all detection rules are passed, the system is ready to write the synchronized data to the target storage location. Specifically, the synchronized data is written from the receiving cache area to the target storage location, such as a designated storage node in a distributed storage system. During the writing process, corresponding storage operations such as data sharding, creating copies, etc. will be performed on the data according to the architecture and requirements of the target storage system to ensure high availability and reliability of the data. For example, a distributed storage system can be used to store financial ultra-large data. Through data sharding and copy mechanisms, the data is dispersed and stored on multiple storage nodes to improve the storage capacity and availability of the data. Each data shard has multiple copies distributed on different nodes to prevent data loss due to a single node failure.

[0071] In one embodiment, after the synchronized data is written to the target storage location, the storage index and metadata management system are updated to enable subsequent rapid query and access to this data. This efficient storage index and metadata management system allows for rapid location and access to financial data. The index structure can be designed based on the characteristics of the financial data. For example, for financial statement data, multi-level indexes can be established based on time, department, project, and other dimensions. The metadata management system records information such as the source, format, update time, and storage location of the financial data, facilitating data query, retrieval, and management.

[0072] In the above embodiment, decompressing data in the receive buffer area can promptly detect potential data corruption or loss during transmission. By performing integrity verification on the decompressed data, the integrity and accuracy of the data are ensured. Only complete data can enter the subsequent quality inspection process, thereby ensuring data consistency.

[0073] Furthermore, the quality detection of the synchronized data is performed based on preset data quality detection rules, and after obtaining the detection results, it also includes: when the detection result shows that there is an abnormality in the synchronized data, recording the abnormality location and the abnormality type, and analyzing the abnormality location and the abnormality type to obtain the predicted abnormality cause; based on the abnormality location, the abnormality type and the predicted abnormality cause, generating a data quality feedback report for the user to view.

[0074] In one embodiment, if a data anomaly is found, the location of the anomaly is precisely recorded, including detailed information such as the data field, record number, and the form it is located in, to accurately locate the root cause of the problem. The anomaly type is also clearly recorded, such as incorrect data format, inconsistent content, missing data, or data outside of a reasonable range.

[0075] Analyze the recorded abnormal location and type, integrate multi-dimensional background information such as historical data, data sources and business processes, make intelligent predictions on the potential causes of the abnormalities, and obtain the predicted abnormal causes.

[0076] Integrate key information such as abnormal location, type and predicted cause to generate a data quality feedback report and feed it back to the user.

[0077] See also Figure 3 , Figure 3This is a schematic flow chart of a data synchronization method provided in an embodiment of the present application. This data synchronization method can be applied to servers and data synchronization systems to dynamically adjust collection schemes and processing parameters based on the characteristics of different data sources and data changes, enabling the data synchronization system to adapt to the ever-changing business needs and complex and diverse financial data environments of enterprises. It effectively solves the problems of incomplete collection, low processing efficiency, and difficulty adapting to data changes that exist in traditional financial data synchronization, ensuring the efficiency and accuracy of data synchronization.

[0078] like Figure 3 As shown, the data synchronization method specifically includes steps S301 to S303.

[0079] S301, identifying and connecting to a data source, and scanning and monitoring data in the data source to obtain data change information of the data source;

[0080] S302: Analyze the data change information to obtain data characteristics, and determine a target acquisition scheme, preprocessing parameters, and data compression parameters based on the data characteristics;

[0081] S303 : Based on the target acquisition scheme, target data is collected from the data source, and based on the preprocessing parameters and the data compression parameters, the target data is preprocessed and compressed to obtain the data to be synchronized.

[0082] In this embodiment, the data synchronization system's data perception and acquisition module automatically identifies and connects to various data sources, including internal enterprise financial systems (such as general ledgers, subsidiary ledgers, and financial analysis systems), business-related systems (such as sales, procurement, and inventory systems), and interfaces with external financial institutions. Through intelligent scanning and monitoring technology, it detects changes in data sources in real time, such as data updates, record additions, and deletions.

[0083] Data characteristics include whether it is static or dynamic, and if it is dynamic, the frequency of data updates. Different collection plans are adopted based on the data characteristics. For example, for static basic financial data (such as the chart of accounts and financial policy parameters), a regular full-volume collection plan is adopted; for data that changes frequently and dynamically (such as real-time transaction flows and fund change information), an incremental collection plan is used. Through methods such as data log analysis and timestamp comparison, only newly generated or updated data is obtained. At the same time, the breakpoint-resume collection function is supported, and if the collection process is interrupted, collection can be resumed from the last interruption point.

[0084] In one embodiment, data preprocessing includes data cleaning and data format conversion. Preprocessing parameters include standard date formats, standard data formats, standard data types, and other parameters. Specifically, collected data is cleaned to remove noise, duplicate data, and erroneous data. For example, financial data cleaning involves filtering out abnormal transaction records caused by system failures or human errors, and correcting dates and amounts that are not in the correct format. Data purification is achieved through data pattern matching, rule checking, and outlier detection algorithms.

[0085] Convert cleaned data into a standard format for subsequent processing. Specifically, data in different formats and encodings is converted to a standard format. This includes normalizing data types (such as integers, floating-point numbers, and strings) across different financial systems, standardizing date and currency formats, and converting data in financial statements from text to a computable numeric format. Furthermore, necessary encoding conversion and data structure adjustments are performed based on the requirements of the target system (the system of the data receiving device).

[0086] In one embodiment, an efficient data compression algorithm is used to compress the cleaned and converted data to reduce data transmission volume and storage usage. During the compression process, an adaptive compression strategy is used to select optimal compression parameters based on data characteristics, ensuring both compression rate and data decompression efficiency.

[0087] In another embodiment, relevant performance indicators during the data collection process are monitored in real time, and optimization suggestions are automatically generated based on the monitoring results. For example, if the data collection speed of a certain data source is found to be too slow, it can be suggested to adjust the collection strategy or increase the collection threads.

[0088] In the above embodiment, the collection plan and processing parameters are dynamically adjusted according to the characteristics of different data sources and data changes, so that the data synchronization system can adapt to the ever-changing business needs and complex and diverse financial data environment of the enterprise, effectively solving the problems of incomplete collection, low processing efficiency, and difficulty in adapting to data changes in traditional financial data synchronization, and ensuring the efficiency and accuracy of data synchronization.

[0089] See also Figure 4 , Figure 4 1 is a schematic block diagram of a data synchronization device provided by an embodiment of the present application, wherein the data synchronization device is used to execute the aforementioned data synchronization method.

[0090] like Figure 4 As shown, the data synchronization device 400 includes:

[0091] The data acquisition module 401 is used to acquire the data to be synchronized and the data size and transmission requirements of the data to be synchronized;

[0092] A configuration determination module 402 is configured to detect a real-time transmission environment and determine a target transmission channel and a target transmission mode based on the real-time transmission environment, the data size, and the transmission requirements;

[0093] The data transmission module 403 is configured to transmit the data to be synchronized based on the target transmission channel and the target transmission mode, so as to synchronize the data to be synchronized to a target storage location.

[0094] Furthermore, the data transmission module 403 includes:

[0095] a data cache unit, configured to transmit the data to be synchronized to a local cache area of ​​a data source device based on the target transmission channel when the target transmission mode is an asynchronous transmission mode;

[0096] a performance monitoring unit, configured to obtain a preset data transmission strategy, call a data transmission thread to transmit data in the local cache area according to the data transmission strategy, and monitor various performances during the transmission process to obtain performance indicators;

[0097] The policy adjustment unit is configured to adjust the data transmission policy in real time based on a preset performance threshold and the performance indicator until the data to be synchronized is transmitted to a target storage location.

[0098] Furthermore, the policy adjustment unit includes:

[0099] An optimization suggestion obtaining subunit is used to analyze the performance indicators and the performance thresholds to obtain performance optimization suggestions;

[0100] An alarm information generating subunit, configured to generate alarm information based on the performance indicator and the performance optimization suggestion, and feed the alarm information back to the user;

[0101] The transmission policy adjustment subunit is configured to receive transmission configuration information set by a user based on the alarm information, and adjust the data transmission policy based on the transmission configuration information.

[0102] Furthermore, the data transmission module 403 includes:

[0103] A data cache unit, configured to transmit the data to be synchronized to a receiving cache area of ​​a data receiving end device based on the target transmission channel and the target transmission mode;

[0104] A data decompression unit, configured to decompress the data to be synchronized in the receiving buffer area to obtain synchronized data;

[0105] A quality detection unit, configured to perform a quality detection on the synchronized data based on a preset data quality detection rule to obtain a detection result;

[0106] The data writing unit is configured to write the synchronized data into the target storage location for storage when the detection result shows no abnormality.

[0107] Furthermore, the data transmission module 403 further includes:

[0108] an abnormality analysis unit, configured to, when the detection result indicates that the synchronized data has an abnormality, record the abnormality location and the abnormality type, and analyze the abnormality location and the abnormality type to obtain a predicted abnormality cause;

[0109] The quality feedback unit is used to generate a data quality feedback report based on the abnormal location, the abnormal type and the predicted abnormal cause for the user to view.

[0110] Furthermore, the data synchronization device 400 further includes a data acquisition module to be synchronized, and the data acquisition module to be synchronized includes:

[0111] a change information obtaining unit, configured to identify and connect to a data source, and scan and monitor data in the data source to obtain data change information of the data source;

[0112] a data characteristic obtaining unit, configured to analyze the data change information, obtain data characteristics, and determine a target acquisition scheme, preprocessing parameters, and data compression parameters based on the data characteristics;

[0113] The data acquisition unit for synchronization is used to acquire target data from the data source based on the target acquisition scheme, and to preprocess and compress the target data based on the preprocessing parameters and the data compression parameters to acquire the data for synchronization.

[0114] Furthermore, the configuration determination module 402 includes:

[0115] A mapping table acquisition unit, configured to acquire a preset transmission channel and transmission mode mapping table;

[0116] A configuration determination unit is used to search the target transmission channel and the target transmission mode corresponding to the real-time transmission environment, the data scale and the transmission requirement from the transmission channel and transmission mode mapping table based on the real-time transmission environment, the data scale and the transmission requirement.

[0117] It should be noted that those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0118] The above-mentioned device can be realized in the form of a computer program. The computer program can be used in Figure 5 Runs on the computer equipment shown.

[0119] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0120] See Figure 5 The computer device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0121] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any data synchronization method.

[0122] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0123] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any data synchronization method.

[0124] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0125] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0126] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0127] Acquire data to be synchronized and the data size and transmission requirements of the data to be synchronized;

[0128] detecting a real-time transmission environment, and determining a target transmission channel and a target transmission mode based on the real-time transmission environment, the data scale, and the transmission requirements;

[0129] Based on the target transmission channel and the target transmission mode, the data to be synchronized is transmitted to synchronize the data to a target storage location.

[0130] In one embodiment, when the processor transmits the data to be synchronized based on the target transmission channel and the target transmission mode to synchronize the data to be synchronized to the target storage location, the processor is configured to implement:

[0131] When the target transmission mode is an asynchronous transmission mode, transmitting the data to be synchronized to a local cache area of ​​a data source end device based on the target transmission channel;

[0132] Obtaining a preset data transmission strategy, calling a data transmission thread to transmit the data in the local cache area according to the data transmission strategy, and monitoring various performances during the transmission process to obtain performance indicators;

[0133] Based on a preset performance threshold and the performance indicator, the data transmission strategy is adjusted in real time until the data to be synchronized is transmitted to the target storage location.

[0134] In one embodiment, when the processor adjusts the data transmission policy in real time based on a preset performance threshold and the performance indicator, it is configured to:

[0135] Analyze the performance indicators and the performance thresholds to obtain performance optimization suggestions;

[0136] Generate warning information based on the performance indicators and the performance optimization suggestions, and feed back the warning information to the user;

[0137] Receive transmission configuration information set by the user based on the alarm information, and adjust the data transmission strategy based on the transmission configuration information.

[0138] In one embodiment, when the processor transmits the data to be synchronized based on the target transmission channel and the target transmission mode to synchronize the data to be synchronized to the target storage location, the processor is configured to implement:

[0139] Based on the target transmission channel and the target transmission mode, transmitting the data to be synchronized to a receiving buffer area of ​​a data receiving end device;

[0140] Decompressing the data to be synchronized in the receiving buffer area to obtain synchronized data;

[0141] Performing a quality test on the synchronized data based on a preset data quality test rule to obtain a test result;

[0142] When the detection result is no abnormality, the synchronized data is written into the target storage location for storage.

[0143] In one embodiment, after performing quality detection on the synchronized data based on a preset data quality detection rule and obtaining a detection result, the processor is further configured to:

[0144] When the detection result indicates that the synchronized data has an anomaly, recording the anomaly location and the anomaly type, and analyzing the anomaly location and the anomaly type to obtain a predicted anomaly cause;

[0145] Based on the abnormal location, the abnormal type and the predicted abnormal cause, a data quality feedback report is generated for the user to review.

[0146] In one embodiment, before obtaining the data to be synchronized and the data size and transmission requirements of the data to be synchronized, the processor is further configured to:

[0147] Identify and connect to a data source, and scan and monitor the data in the data source to obtain data change information of the data source;

[0148] Analyzing the data change information to obtain data characteristics, and determining a target acquisition scheme, preprocessing parameters, and data compression parameters based on the data characteristics;

[0149] Based on the target acquisition scheme, target data is collected from the data source, and based on the preprocessing parameters and the data compression parameters, the target data is preprocessed and compressed to obtain the data to be synchronized.

[0150] In one embodiment, when determining the target transmission channel and the target transmission mode based on the real-time transmission environment, the data scale, and the transmission requirement, the processor is configured to implement:

[0151] Obtaining a preset transmission channel and transmission mode mapping table;

[0152] Based on the real-time transmission environment, the data scale and the transmission requirement, the target transmission channel and the target transmission mode corresponding to the real-time transmission environment, the data scale and the transmission requirement are searched from the transmission channel and transmission mode mapping table.

[0153] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the data synchronization methods provided in the embodiments of the present application.

[0154] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc., equipped on the computer device.

[0155] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data synchronization method, characterized in that: include: Acquire data to be synchronized and the data size and transmission requirements of the data to be synchronized; detecting a real-time transmission environment, and determining a target transmission channel and a target transmission mode based on the real-time transmission environment, the data scale, and the transmission requirements; Based on the target transmission channel and the target transmission mode, the data to be synchronized is transmitted to synchronize the data to a target storage location.

2. The data synchronization method according to claim 1, characterized in that: The transmitting the data to be synchronized based on the target transmission channel and the target transmission mode to synchronize the data to be synchronized to a target storage location includes: When the target transmission mode is an asynchronous transmission mode, transmitting the data to be synchronized to a local cache area of ​​a data source end device based on the target transmission channel; Obtaining a preset data transmission strategy, calling a data transmission thread to transmit the data in the local cache area according to the data transmission strategy, and monitoring various performances during the transmission process to obtain performance indicators; Based on a preset performance threshold and the performance indicator, the data transmission strategy is adjusted in real time until the data to be synchronized is transmitted to the target storage location.

3. The data synchronization method according to claim 2, characterized in that: The real-time adjustment of the data transmission strategy based on the preset performance threshold and the performance indicator includes: Analyze the performance indicators and the performance thresholds to obtain performance optimization suggestions; Generate warning information based on the performance indicators and the performance optimization suggestions, and feed back the warning information to the user; Receive transmission configuration information set by the user based on the alarm information, and adjust the data transmission strategy based on the transmission configuration information.

4. The data synchronization method according to claim 2, wherein: The transmitting the data to be synchronized based on the target transmission channel and the target transmission mode to synchronize the data to be synchronized to a target storage location includes: Based on the target transmission channel and the target transmission mode, transmitting the data to be synchronized to a receiving buffer area of ​​a data receiving end device; Decompressing the data to be synchronized in the receiving buffer area to obtain synchronized data; Performing a quality test on the synchronized data based on a preset data quality test rule to obtain a test result; When the detection result is no abnormality, the synchronized data is written into the target storage location for storage.

5. The data synchronization method according to claim 4, characterized in that: After performing quality detection on the synchronized data based on the preset data quality detection rules and obtaining the detection result, the method further includes: When the detection result indicates that the synchronized data has an anomaly, recording the anomaly location and the anomaly type, and analyzing the anomaly location and the anomaly type to obtain a predicted anomaly cause; Based on the abnormal location, the abnormal type and the predicted abnormal cause, a data quality feedback report is generated for the user to review.

6. The data synchronization method according to claim 1, wherein: Before acquiring the data to be synchronized and the data size and transmission requirements of the data to be synchronized, the method further includes: Identify and connect to a data source, and scan and monitor the data in the data source to obtain data change information of the data source; Analyzing the data change information to obtain data characteristics, and determining a target acquisition scheme, preprocessing parameters, and data compression parameters based on the data characteristics; Based on the target acquisition scheme, target data is collected from the data source, and based on the preprocessing parameters and the data compression parameters, the target data is preprocessed and compressed to obtain the data to be synchronized.

7. The data synchronization method according to any one of claims 1 to 6, characterized in that: The determining of a target transmission channel and a target transmission mode based on the real-time transmission environment, the data scale, and the transmission requirement includes: Obtaining a preset transmission channel and transmission mode mapping table; Based on the real-time transmission environment, the data scale and the transmission requirement, the target transmission channel and the target transmission mode corresponding to the real-time transmission environment, the data scale and the transmission requirement are searched from the transmission channel and transmission mode mapping table.

8. A data synchronization device, characterized in that: include: A data acquisition module is used to acquire the data to be synchronized and the data size and transmission requirements of the data to be synchronized; a configuration determination module, configured to detect a real-time transmission environment and determine a target transmission channel and a target transmission mode based on the real-time transmission environment, the data scale, and the transmission requirements; The data transmission module is configured to transmit the data to be synchronized based on the target transmission channel and the target transmission mode, so as to synchronize the data to be synchronized to a target storage location.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the data synchronization method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the data synchronization method according to any one of claims 1 to 7.

Images