A data synchronization method, device, system, electronic device, and storage medium

Abstract

The application provides a data synchronization method, device and system, electronic equipment and a storage medium. The method comprises: obtaining data synchronization statistical data of a working node, and parallelism and thread numbers of an extraction and transformation link set by the working node; determining the execution efficiency of the working node based on the parallelism, the thread numbers of the extraction and transformation link and the data synchronization statistical data; simulating and calculating the synchronization data volume in the data synchronization statistical data by using an artificial intelligence (AI) trial model to obtain the optimal parallelism and the optimal thread numbers of the extraction and transformation link of the working node; adjusting the parallelism and the thread numbers of the extraction and transformation link of the working node based on the execution efficiency, the optimal parallelism and the optimal thread numbers of the extraction and transformation link of the working node; and synchronizing the synchronization data in the data synchronization statistical data based on the adjusted parallelism and the thread numbers of the extraction and transformation link. The problem of synchronization delay caused by sudden increase in data flow is solved by simulation and calculation.

Description

Technical Field

[0001] This invention relates to the field of computer data synchronization technology, and in particular to a data synchronization method, apparatus, system, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of the information age, data has become one of the main means of production for enterprises to create business value. More and more enterprises require the synchronization of various internal business data across multiple data centers and different heterogeneous storage media. Among these, real-time data synchronization from relational databases to various heterogeneous storage media is one of the most common application scenarios. Data synchronization refers to the process of data transmission and processing to maintain consistency between the data source and the destination.

[0003] In related technologies, data synchronization transmission typically employs a staged event-driven architecture (SEDA) scheduling model for network data transmission.

[0004] To better support the system's scalability and flexibility, the entire synchronous transmission process is abstracted into several stages: Select / Extract / Transform / Load (abbreviated as S / E / T / L). During synchronous transmission, the S / E / T / L stages are unaware of each other. Data transfer between these stages uses a set window parallelism. The S / L stage uses a serial mechanism to ensure the sequential processing of database logs, while the E / T stage uses a parallel mechanism to accelerate transmission efficiency. This technology can guarantee the real-time performance of data synchronization in most cases. However, since the parallelism needs to be manually set and cannot be adaptively adjusted, it can cause synchronization delays and reduce synchronization efficiency when data traffic surges. Summary of the Invention

[0005] This invention provides a data synchronization method, apparatus, system, electronic device, and computer-readable storage medium to at least solve the problem in related technologies where increased synchronization latency occurs due to the inability of data synchronization transmission to adaptively adjust under sudden increases in data traffic. The technical solution of this invention is as follows:

[0006] According to a first aspect of the present invention, a data synchronization method is provided, comprising:

[0007] Obtain data synchronization statistics for the worker nodes, as well as the parallelism and number of threads for the extraction and transformation process set for the worker nodes;

[0008] The execution efficiency of the work node is determined based on the parallelism of the work node, the number of threads in the extraction and transformation process, and the data synchronization statistics.

[0009] The amount of synchronized data in the data synchronization statistics is simulated and calculated using an artificial intelligence (AI) trial calculation model to obtain the optimal parallelism of the working node and the optimal number of threads in the extraction and transformation process. The AI ​​trial calculation model is a model obtained after simulation and training based on the support vector machine algorithm.

[0010] Based on the execution efficiency of the work node, the optimal parallelism of the work node, and the optimal number of threads in the extraction and transformation stage, adjust the parallelism of the work node and the number of threads in the extraction and transformation stage.

[0011] Based on the adjusted parallelism of the working nodes and the number of threads in the extraction and transformation process, the synchronized data in the data synchronization statistics are synchronized.

[0012] Optionally, the step of using an artificial intelligence (AI) simulation model to calculate the amount of synchronized data in the data synchronization statistics to obtain the optimal parallelism of the working node and the optimal number of threads in the extraction and transformation stage includes:

[0013] Obtain the hardware parameters of the working node, the synchronization medium information and trial parameters of the working node, the trial parameters including: synchronization delay sample dataset and expected synchronization delay value;

[0014] The hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node and the number of threads in the extraction and transformation stage, the trial calculation parameters, and the amount of synchronized data in the data synchronization statistics are input into the artificial intelligence (AI) trial calculation model.

[0015] The AI ​​simulation model is used to simulate the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and conversion stage based on the synchronization delay sample dataset. AI simulation is performed, and learning simulation is conducted based on the expected value of the synchronization delay. The optimal parallelism of the working node and the optimal number of threads in the extraction and conversion stage are output when transmitting the amount of synchronization data. The learning simulation uses a support vector machine regression model for learning simulation.

[0016] Optionally, the AI ​​simulation model simulates the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and transformation stage for AI simulation, including:

[0017] The AI ​​trial model, based on the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, the number of threads in the extraction and transformation stage, and the amount of synchronized data in the data synchronization statistics, selects a corresponding AI decision strategy and simulates the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and transformation stage to perform AI trial calculations. The AI ​​decision strategy must satisfy the following: the latency of the working node is greater than the expected synchronization latency.

[0018] Optionally, adjusting the parallelism of the worker node and the number of threads in the extraction and transformation stage based on the execution efficiency of the worker node, the optimal parallelism of the worker node, and the optimal number of threads in the extraction and transformation stage includes:

[0019] The optimal synchronization efficiency of the working node is determined based on the optimal parallelism of the working node, the optimal number of threads in the extraction and transformation stage, and the data synchronization statistics.

[0020] The optimal synchronization efficiency of the working node is compared with the execution efficiency of the working node;

[0021] Adjust the parallelism of the working nodes and the number of threads in the extraction and transformation process based on the comparison results.

[0022] Optionally, the method further includes: pre-training an AI trial model, the training process of which includes:

[0023] Obtain a training dataset, which includes: synchronous data related to AI decision-making strategies, historical synchronous data, parallelism, and the number of threads in the extraction and transformation process;

[0024] The training set is preprocessed;

[0025] The preprocessed dataset is used to build an AI trial model based on the support vector machine algorithm;

[0026] The AI ​​trial model is iteratively trained using the acquired historical synchronous sample data, and the parallelism of the sample nodes and the number of threads in the extraction and transformation process are output.

[0027] When the parallelism of the sample node and the number of threads in the extraction and transformation process meet the corresponding set thresholds, a trained AI trial model is obtained.

[0028] Optionally, the method further includes:

[0029] During iterative training, based on the need for real-time synchronization and training objectives, AI decision-making strategies for different situations are formulated for the AI ​​trial model. These AI decision-making strategies are used to guide the output and decision selection of the AI ​​trial model.

[0030] The AI ​​trial model is iteratively trained based on the AI ​​decision-making strategies for the different situations, and the AI ​​decision-making strategies are continuously optimized based on the iterative training results.

[0031] According to a second aspect of the present invention, a data synchronization apparatus is provided, comprising:

[0032] The acquisition module is used to acquire data synchronization statistics of the worker nodes, as well as the parallelism and number of threads in the extraction and transformation process set by the worker nodes;

[0033] The determination module is used to determine the execution efficiency of the work node based on the parallelism of the work node, the number of threads in the extraction and transformation stage, and the data synchronization statistics.

[0034] The simulation calculation module is used to simulate and calculate the amount of synchronized data in the data synchronization statistics using an artificial intelligence (AI) trial calculation model, so as to obtain the optimal parallelism of the working node and the optimal number of threads in the extraction and transformation stage. The AI ​​trial calculation model is a model obtained after simulation calculation training based on the support vector machine algorithm.

[0035] An adjustment module is used to adjust the parallelism of the work node and the number of threads in the extraction and transformation stage based on the execution efficiency of the work node, the optimal parallelism of the work node, and the optimal number of threads in the extraction and transformation stage.

[0036] The synchronization module is used to synchronize the synchronization data in the data synchronization statistics based on the adjusted parallelism of the working nodes and the number of threads in the extraction and transformation process.

[0037] Optionally, the simulation calculation module includes:

[0038] The parameter acquisition module is used to acquire the hardware parameters of the working node, the synchronization medium information of the working node, and the trial parameters, the trial parameters including: the synchronization delay sample dataset and the expected synchronization delay value;

[0039] The input module is used to input the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node and the number of threads in the extraction and transformation stage, the trial parameters, and the amount of synchronized data in the data synchronization statistics into the artificial intelligence AI trial model.

[0040] The AI ​​simulation module is used to simulate the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and transformation stage based on the synchronization delay sample set using the AI ​​simulation model. It performs AI simulation based on the expected value of the synchronization delay and outputs the optimal parallelism and the optimal number of threads in the extraction and transformation stage of the working node when transmitting the amount of synchronized data. The learning simulation is performed using a support vector machine regression model.

[0041] Optionally, the AI ​​trial calculation module is specifically used to utilize the AI ​​trial calculation model based on the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node and the number of threads in the extraction and transformation stage, and the amount of synchronized data in the data synchronization statistics, to select a corresponding AI decision strategy, simulate the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node and the number of threads in the extraction and transformation stage to perform AI trial calculations, wherein the AI ​​decision strategy must satisfy: the delay duration of the working node is greater than the expected value of the synchronization delay.

[0042] Optionally, the adjustment module includes:

[0043] An efficiency determination module is used to determine the optimal synchronization efficiency of the working node based on the optimal parallelism of the working node, the optimal number of threads in the extraction and transformation stage, and the data synchronization statistics.

[0044] The comparison module is used to compare the optimal synchronization efficiency of the working node with the execution efficiency of the working node.

[0045] The parameter adjustment module is used to adjust the parallelism of the working node and the number of threads in the extraction and transformation process based on the comparison results.

[0046] Optionally, the device further includes a training module for pre-training an AI trial model.

[0047] The training module includes:

[0048] The dataset acquisition module is used to acquire the training dataset, which includes: synchronous data related to the AI ​​decision-making strategy, historical synchronous data, parallelism, and the number of threads in the extraction and transformation process.

[0049] The preprocessing module is used to preprocess the training set;

[0050] The building module is used to construct an AI trial model based on the support vector machine algorithm from the preprocessed dataset;

[0051] The iterative training module is used to iteratively train the AI ​​trial model using the acquired historical synchronous sample data, and outputs the parallelism of the sample nodes and the number of threads in the extraction and transformation process.

[0052] The model determination module is used to obtain a trained AI trial model when the parallelism of the sample node and the number of threads in the extraction and transformation process meet the corresponding set thresholds.

[0053] Optionally, the device further includes:

[0054] The strategy setting module is used to formulate AI decision-making strategies for the AI ​​trial model under different situations during iterative training, based on the real-time synchronization requirements and training objectives. The AI ​​decision-making strategies are used to guide the output and decision selection of the AI ​​trial model.

[0055] The strategy optimization module is used to iteratively train the AI ​​trial model based on the AI ​​decision-making strategies for the different situations, and continuously optimize the AI ​​decision-making strategies based on the iterative training results.

[0056] According to a third aspect of the present invention, a data synchronization system is provided, comprising: a data synchronization module, an AI decision center module, and an AI trial calculation module, wherein,

[0057] The data synchronization module is used to collect data synchronization statistics from each node in real time or at regular intervals, set the parallelism of the synchronization data in the data synchronization statistics according to the deployment environment, and sequentially capture, extract, transform, and load the synchronization data. The processed data synchronization statistics of each node, the set parallelism of each node, and the number of threads in the extraction and transformation stages are sent to the AI ​​decision center module. The capture and loading are a serial mechanism to ensure the timeliness of data processing, and the extraction and transformation are parallel processes to accelerate transmission based on the set parallelism.

[0058] The AI ​​decision center module is used to determine the execution efficiency of the working node based on the data synchronization statistics received from each node, as well as the parallelism and number of threads in the extraction and transformation process set for each node, and to send the data synchronization statistics of nodes whose operation conforms to the AI ​​trial calculation strategy rules to the AI ​​trial calculation module.

[0059] The AI ​​trial calculation module is used to simulate and calculate the amount of synchronized data in the received data synchronization statistics using an artificial intelligence AI trial calculation model, to obtain the optimal parallelism of the node and the optimal number of threads in the extraction and transformation process. The AI ​​trial calculation model is a model obtained after simulation and training based on the support vector machine algorithm, and the optimal parallelism of the node and the optimal number of threads in the extraction and transformation process are sent to the AI ​​decision center module.

[0060] The AI ​​decision center module is also used to determine the optimal synchronization efficiency of the node based on the optimal parallelism of the node and the optimal number of threads in the extraction and transformation stage, as well as the data synchronization statistics, and to compare the optimal synchronization efficiency of the node with the execution efficiency of the working node; and to send the comparison result to the data synchronization module for corresponding adjustments.

[0061] The data synchronization module is also used to adjust the parallelism of the node and the number of threads in the extraction and transformation process based on the received comparison results, and to synchronize the synchronization data in the data synchronization statistics based on the adjusted parallelism of the node and the number of threads in the extraction and transformation process.

[0062] According to a fourth aspect of the present invention, an electronic device is provided, comprising:

[0063] processor;

[0064] Memory used to store the processor's executable instructions;

[0065] The processor is configured to execute the instructions to implement the synchronization method described above.

[0066] According to a fifth aspect of the present invention, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the data synchronization method described above.

[0067] According to a sixth aspect of the present invention, a computer program product is provided, comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor of an electronic device, constitute the data synchronization method described above.

[0068] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects:

[0069] In this embodiment of the invention, data synchronization statistics of a working node are obtained, along with the parallelism and the number of threads in the extraction and transformation stage set for the working node. Based on the parallelism of the working node, the number of threads in the extraction and transformation stage, and the data synchronization statistics, the execution efficiency of the working node is determined. An artificial intelligence (AI) simulation model is used to calculate the amount of synchronized data in the data synchronization statistics to obtain the optimal parallelism and the optimal number of threads in the extraction and transformation stage for the working node. The AI ​​simulation model is a model obtained after simulation and training based on a support vector machine algorithm. Based on the execution efficiency of the working node, the optimal parallelism, and the optimal number of threads in the extraction and transformation stage, the parallelism and the number of threads in the extraction and transformation stage are adjusted. Based on the adjusted parallelism and the number of threads in the extraction and transformation stage, the synchronized data in the data synchronization statistics is synchronized. In other words, in this embodiment of the invention, the optimal parallelism and the number of threads in the extraction and transformation stage are dynamically simulated and calculated based on the support vector machine algorithm. The parallelism and the number of threads in the extraction and transformation stage of the current working node are adjusted based on the optimal parallelism and the number of threads in the extraction and transformation stage, so as to realize intelligent adaptive adjustment of data synchronization and solve the problem of synchronization delay caused by sudden increase in data traffic.

[0070] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0071] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the invention. They do not constitute an undue limitation of the invention. To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0072] Figure 1 This is a flowchart of a data synchronization method provided in an embodiment of the present invention.

[0073] Figure 2 This is a schematic diagram of an AI calculation provided in an embodiment of the present invention.

[0074] Figure 3 This is a schematic diagram illustrating the interaction between an AI decision center and AI trial calculation and data synchronization work nodes, as provided in an embodiment of the present invention.

[0075] Figure 4This is an interactive flowchart of an AI decision center provided by an embodiment of the present invention.

[0076] Figure 5 This is a block diagram of a data synchronization device provided in an embodiment of the present invention.

[0077] Figure 6 This is a block diagram of a data synchronization system provided in an embodiment of the present invention.

[0078] Figure 7 This is an application block diagram of a data synchronization system provided in an embodiment of the present invention.

[0079] Figure 8 This is a block diagram of an electronic device provided in an embodiment of the present invention.

[0080] Figure 9 This is a block diagram of a data synchronization device provided in an embodiment of the present invention. Detailed Implementation

[0081] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0082] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0083] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0084] Technical terms:

[0085] Support Vector Machine (SVM) is a generalized linear classifier that performs binary classification of data using supervised learning. Its decision boundary is the maximum margin hyperplane obtained by solving for the learning samples. SVM uses the hinge loss function to calculate empirical risk and adds a regularization term to the solution system to optimize structural risk, making it a classifier with sparsity and robustness.

[0086] In addition to understanding the technical terms, please also refer to the following examples.

[0087] Figure 1 This is a flowchart of a data synchronization method provided in an embodiment of the present invention, such as... Figure 1 As shown, this data synchronization method includes the following steps:

[0088] Step 101: Obtain the data synchronization statistics of the worker node, as well as the parallelism and number of threads in the extraction and transformation process set for the worker node.

[0089] Step 102: Determine the execution efficiency of the work node based on the parallelism of the work node, the number of threads in the extraction and transformation process, and the data synchronization statistics.

[0090] Step 103: Use an artificial intelligence (AI) trial calculation model to simulate and calculate the amount of synchronized data in the data synchronization statistics, and obtain the optimal parallelism of the working node and the optimal number of threads in the extraction and transformation stage. The AI ​​trial calculation model is a model obtained after simulation and training based on the support vector machine algorithm.

[0091] Step 104: Based on the execution efficiency of the worker node, the optimal parallelism of the worker node, and the optimal number of threads in the extraction and transformation stage, adjust the parallelism of the worker node and the number of threads in the extraction and transformation stage.

[0092] Step 105: Synchronize the synchronization data in the data synchronization statistics based on the adjusted parallelism of the working node and the number of threads in the extraction and transformation stage.

[0093] The data synchronization method described in this invention can be applied to terminals, servers, etc., without limitation. The terminal implementation device can be an electronic device such as a smartphone, laptop, tablet, desktop computer, personal digital assistant (PDA), or wearable device, or an electronic device integrated with a data synchronization system. The server can be a standalone server, a server cluster, or a server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, or a big data and artificial intelligence platform, etc., without limitation.

[0094] The following is combined Figure 1 The specific implementation steps of a data synchronization method provided in this embodiment of the invention will be described in detail.

[0095] In step 101, the data synchronization statistics of the working node are obtained, as well as the parallelism and the number of threads in the extraction and transformation process set by the working node.

[0096] In this step, the number of threads in the extraction and transformation process includes the number of extraction threads and the number of transformation threads. The data synchronization statistics, the parallelism set for the worker nodes, and the number of threads in the extraction and transformation process can all be referred to as parameters of the worker nodes.

[0097] The data synchronization statistics for the working nodes are obtained based on real-time incremental data acquisition technology from database logs. Specifically, data synchronization statistics for each node are collected in real-time or periodically. The parallelism of the synchronization data in the data synchronization statistics is set according to the deployment environment. The synchronization data is then sequentially captured, extracted, transformed, and loaded. The processed data synchronization statistics for each node, the set parallelism for each node, and the number of threads in the extraction and transformation stages are sent to the AI ​​decision center. The capture and loading are performed serially to ensure the timeliness of data processing, while the extraction and transformation are performed in parallel to accelerate transmission processing according to the set parallelism.

[0098] In step 102, the execution efficiency of the working node is determined based on the parallelism of the working node, the number of threads in the extraction and transformation process, and the data synchronization statistics.

[0099] In this embodiment, the AI ​​decision center determines the execution efficiency of the work node based on the parallelism of the work node, the number of threads in the extraction and transformation process, and the data synchronization statistics. The execution efficiency is the latency during data synchronization. The method for determining the execution efficiency of the work node is well-known to those skilled in the art and will not be elaborated upon here.

[0100] For example, if the data volume (generated per second) is 10,000, the parallelism is 6, the number of extraction threads is 20, and the number of transformation threads is 20, then the calculated latency (in seconds) is 2. As another example, if the data volume (generated per second) is 10,000, the parallelism is 12, the number of extraction threads is 30, and the number of transformation threads is 30, then the calculated latency (in seconds) is 1.4.

[0101] In step 103, the amount of synchronized data in the data synchronization statistics is simulated and calculated using an artificial intelligence (AI) trial calculation model to obtain the optimal parallelism of the working node and the optimal number of threads in the extraction and transformation process. The AI ​​trial calculation model is a model obtained after simulation and training based on the support vector machine algorithm.

[0102] In this step, the AI ​​trial calculation first obtains the hardware parameters of the working node, the synchronization medium information of the working node, and the trial calculation parameters, including: the synchronization delay sample dataset and the expected synchronization delay value. Then, the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, the number of threads in the extraction and conversion stage, the trial calculation parameters, and the amount of synchronized data in the data synchronization statistics are input into the AI ​​trial calculation model. Finally, the AI ​​trial calculation model simulates the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and conversion stage based on the synchronization delay sample dataset. It then performs learning simulation based on the expected synchronization delay value, outputting the optimal parallelism and the optimal number of threads in the extraction and conversion stage for the working node when transmitting the synchronized data amount. The learning simulation uses a support vector machine regression model for learning simulation.

[0103] The AI ​​trial calculation model simulates the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and transformation stage. This includes: using the AI ​​trial calculation model based on the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, the number of threads in the extraction and transformation stage, and the amount of synchronized data in the data synchronization statistics, selecting a corresponding AI decision strategy, and simulating the hardware parameters of the working node, the synchronization medium information of the working node, and the parallelism of the working node and the number of threads in the extraction and transformation stage to perform AI trial calculations. The AI ​​decision strategy must satisfy the following: the latency of the working node is greater than the expected synchronization latency value.

[0104] In other words, the AI ​​decision center pushes the parameters of the work node to be simulated to the AI ​​trial calculation. These parameters include the synchronization delay sample dataset and the expected synchronization delay value, such as 0.5. Upon receiving these parameters, the AI ​​trial calculation simulates the work node's hardware parameters, parallelism, and the number of threads in the extraction and transformation stage. It then performs self-learning training based on the expected synchronization delay value and outputs the optimal parallelism and number of threads in the extraction and transformation stage for that work node under the current synchronization data volume. Specifically... Figure 2 As shown, Figure 2 This is a schematic diagram illustrating an AI calculation method provided in an embodiment of the present invention. Specifically, it includes:

[0105] First, the AI ​​trial calculation receives the work node parameters pushed by the AI ​​decision center and starts the AI ​​trial calculation by simulating the work node parameters. The work node parameters may include: work node hardware parameters, work node synchronization statistics parameters, work node parallelism, number of threads, and work node synchronization medium information, etc. The work node hardware parameters include CPU parameters, memory parameters, etc.; the work node synchronization medium information includes: the address of the source medium connected to the synchronization source end, the address of the target medium connected to the synchronization target end, etc.

[0106] Secondly, the AI ​​Decision Center prepares a set of sample datasets of hardware operating environments. These sample datasets are obtained through testing and stored in the AI ​​Decision Center. At the same time, the AI ​​Decision Center enriches the sample data with daily production synchronization statistical parameters. The AI ​​Decision Center pushes the sample datasets to the AI ​​Trial Calculation. The simplified format of the sample datasets is shown in Table 1, but in actual applications, it is not limited to this.

[0107] Table 1

[0108] 10000 6 20 20 2 10000 12 30 30 1.4 10000 18 40 40 1 10000 24 50 50 0.7 10000 30 60 60 0.5 10000 36 70 70 0.4 10000 42 80 80 0.35

[0109] Next, after the AI ​​trial obtains the sample dataset (i.e., the synchronization delay sample dataset) and the current synchronization data volume, it will perform self-learning training based on the expected synchronization delay value. The training and calculation model uses a support vector machine regression model to obtain the optimal parallelism of the worker node and the number of threads in the extraction and transformation stage. The optimal parallelism of the worker node and the number of threads in the extraction and transformation stage are then pushed to the AI ​​decision center. Specifically, the AI ​​trial involves, after receiving the worker node parameters, it iteratively processing the sample dataset obtained from the database using capture, extraction, transformation, and loading steps. This simulates the hardware parameters, parallelism, and number of threads in the extraction and transformation stage of the worker node. The AI ​​trial then performs self-learning training based on the expected synchronization delay value, stores the training results in the target storage database, and outputs the optimal parallelism and number of threads in the extraction and transformation stage for the worker node under the current synchronization data volume.

[0110] In step 104, based on the execution efficiency of the worker node, the optimal parallelism of the worker node, and the optimal number of threads in the extraction and transformation stage, the parallelism of the worker node and the number of threads in the extraction and transformation stage are adjusted.

[0111] In this step, the AI ​​decision center determines the optimal synchronization efficiency of the work node based on the optimal parallelism of the work node, the optimal number of threads in the extraction and transformation stage, and the data synchronization statistics; compares the optimal synchronization efficiency of the work node with the execution efficiency of the work node; and adjusts the parallelism of the work node and the number of threads in the extraction and transformation stage based on the comparison results.

[0112] In other words, the AI ​​decision center collects real-time data synchronization statistics and hardware parameters of the data synchronization work nodes. Based on the AI ​​trial calculation strategy, when the data synchronization work node's performance conforms to the strategy rules (e.g., work node latency exceeding expected values), the AI ​​decision center initiates relevant information push AI trial calculations. Based on the AI ​​trial calculation results and AI decision adjustments, the center initiates adaptive adjustments to the data synchronization work nodes (parallelism, number of extraction and loading threads), such as... Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the interaction between an AI decision center and AI trial calculation and data synchronization work nodes, as provided in an embodiment of the present invention.

[0113] like Figure 3 As shown, the AI ​​decision center interacts with both the AI ​​trial calculation and data synchronization work nodes (hereinafter referred to as work nodes). The AI ​​decision center stores AI decision adjustment strategies and AI trial calculation strategies for various situations. Simultaneously, the AI ​​decision center can also receive trial calculation results, push work node parameters, push sample datasets and expected synchronization delay values, and collect data from work nodes.

[0114] 1) The AI ​​decision center collects data synchronization statistics and parameters of the data synchronization work nodes in real time. The statistics include the amount of data synchronized per second and the delay duration (seconds). The work node parameters include CPU, memory, parallelism, and the number of threads in each stage.

[0115] 2) Based on the AI ​​trial calculation strategy (e.g., the current reported work node delay is greater than the expected value), the AI ​​decision center will push the work node parameters to the AI ​​trial calculation and initiate the AI ​​trial calculation.

[0116] 3) The AI ​​Decision Center prepares a set of sample datasets with the same hardware operating environment. This sample data is obtained through testing and stored in the AI ​​Decision Center. At the same time, the AI ​​Decision Center enriches the sample data with daily production synchronous statistical parameters and pushes it to the AI ​​Trial Calculation.

[0117] 4) The AI ​​decision center receives the AI ​​trial calculation results, combines them with the AI ​​decision adjustment strategy, and initiates adaptive adjustment of the work nodes, adjusting the parallelism of the work nodes and the number of extraction and loading threads.

[0118] 5) The AI ​​decision-making center improves the sample dataset based on the received AI trial calculation results.

[0119] Specifically, the AI ​​decision center interaction process includes: data collection and preprocessing, model building and training, AI decision strategy formulation, real-time decision support and decision feedback and adjustment, and iterative optimization. Specific steps are as follows: Figure 4 As shown, Figure 4An AI decision center interaction flowchart provided in this embodiment of the invention specifically includes:

[0120] Step 401: The AI ​​decision center collects and organizes decision-related data and preprocesses the data.

[0121] The relevant data may include: real-time synchronized data, historical synchronized data, parallelism, number of extraction and loading threads, etc.; the preprocessing may include cleaning, transforming and normalizing the data to make it suitable for subsequent analysis and decision-making.

[0122] Step 402: Based on the preprocessed data, the AI ​​Decision Center constructs a suitable deep learning model based on the support vector machine algorithm and uses historical data for training and optimization so that the model can accurately infer the optimal parallelism and the number of extraction and loading threads.

[0123] Step 403: The AI ​​Decision Center formulates decision-making strategies for different situations based on the needs and goals of real-time synchronization.

[0124] These decision-making strategies can be based on specific conditions, thresholds, or priorities to guide the model's output and decision selection.

[0125] Step 404: When the AI ​​Decision Center needs to make a decision, it inputs the current data into the trained model, obtains the model's output, and outputs it to the real-time synchronization work node.

[0126] Step 405: The AI ​​Decision Center provides feedback and evaluation based on the decision results and effects, and continuously iterates and optimizes each link of the AI ​​Decision Center.

[0127] This step may involve data updates, model updates, and adjustments to decision-making rules, in order to gradually improve the performance and adaptability of the AI ​​decision-making center.

[0128] In step 105, the synchronization data in the data synchronization statistics is synchronized based on the adjusted parallelism of the working node and the number of threads in the extraction and transformation stage.

[0129] In this embodiment of the invention, data synchronization statistics of a working node are obtained, along with the parallelism and the number of threads in the extraction and transformation stage set for the working node. Based on the parallelism of the working node, the number of threads in the extraction and transformation stage, and the data synchronization statistics, the execution efficiency of the working node is determined. An artificial intelligence (AI) simulation model is used to calculate the amount of synchronized data in the data synchronization statistics to obtain the optimal parallelism and the optimal number of threads in the extraction and transformation stage for the working node. The AI ​​simulation model is a model obtained after simulation and training based on a support vector machine algorithm. Based on the execution efficiency of the working node, the optimal parallelism, and the optimal number of threads in the extraction and transformation stage, the parallelism and the number of threads in the extraction and transformation stage are adjusted. Based on the adjusted parallelism and the number of threads in the extraction and transformation stage, the synchronized data in the data synchronization statistics is synchronized. In other words, in this embodiment of the invention, the optimal parallelism and the number of threads in the extraction and transformation stage are dynamically simulated and calculated based on the support vector machine algorithm. The parallelism and the number of threads in the extraction and transformation stage of the current working node are adjusted based on the optimal parallelism and the number of threads in the extraction and transformation stage, so as to realize intelligent adaptive adjustment of data synchronization and solve the problem of synchronization delay caused by sudden increase in data traffic.

[0130] Optionally, in another embodiment, based on the above embodiments, the method may further include: pre-training an AI trial model, the training process of which includes:

[0131] A training dataset is obtained, comprising: synchronous data related to the AI ​​decision-making strategy, historical synchronous data, parallelism, and the number of threads in the extraction and transformation stage; the training dataset is preprocessed; an AI trial model based on the support vector machine algorithm is constructed from the preprocessed dataset; the AI ​​trial model is iteratively trained using the obtained historical synchronous sample data, and the parallelism and the number of threads in the extraction and transformation stage corresponding to the sample nodes are output; when the parallelism and the number of threads in the extraction and transformation stage of the sample nodes meet the corresponding set thresholds, the trained AI trial model is obtained.

[0132] Optionally, in another embodiment, based on the above embodiment, during the iterative training process, AI decision-making strategies for different situations are formulated for the AI ​​trial model based on the need for real-time synchronization and the training objectives. The AI ​​decision-making strategies are used to guide the output and decision selection of the AI ​​trial model. The AI ​​trial model is iteratively trained based on the AI ​​decision-making strategies for different situations, and the AI ​​decision-making strategies are continuously optimized according to the iterative training results.

[0133] In this embodiment of the invention, an intelligent data synchronization method based on the support vector machine algorithm is provided. To overcome the inability of existing data synchronization technologies to adaptively adjust, the machine learning support vector machine algorithm is introduced to achieve intelligent adaptive adjustment of data synchronization. This solves the problem of synchronization delay caused by a sudden increase in data traffic. The optimal synchronization parallelism of the current node and the number of threads in the extraction and transformation process are obtained through the support vector machine algorithm. The parallelism is automatically adjusted according to the change in the amount of synchronized data, thereby avoiding synchronization delay.

[0134] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the present invention.

[0135] Figure 5 This is a block diagram of a data synchronization device provided in an embodiment of the present invention. The device includes: an acquisition module 501, a determination module 502, a simulation calculation module 503, an adjustment module 504, and a synchronization module 505, wherein...

[0136] The acquisition module 501 is used to acquire data synchronization statistics of the working node, as well as the parallelism and number of threads in the extraction and transformation process set by the working node;

[0137] The determination module 502 is used to determine the execution efficiency of the working node based on the parallelism of the working node, the number of threads in the extraction and transformation stage, and the data synchronization statistics.

[0138] The simulation calculation module 503 is used to simulate and calculate the amount of synchronized data in the data synchronization statistics using an artificial intelligence (AI) trial calculation model, so as to obtain the optimal parallelism of the working node and the optimal number of threads in the extraction and transformation stage. The AI ​​trial calculation model is a model obtained after simulation calculation training based on the support vector machine algorithm.

[0139] The adjustment module 504 is used to adjust the parallelism of the work node and the number of threads in the extraction and transformation stage based on the execution efficiency of the work node, the optimal parallelism of the work node, and the optimal number of threads in the extraction and transformation stage.

[0140] The synchronization module 505 is used to synchronize the synchronization data in the data synchronization statistics based on the adjusted parallelism of the working node and the number of threads in the extraction and transformation stage.

[0141] Optionally, in another embodiment, based on the above embodiment, the simulation calculation module 3 includes: a parameter acquisition module, an input module, and an AI trial calculation module, wherein,

[0142] The parameter acquisition module is used to acquire the hardware parameters of the working node, the synchronization medium information of the working node, and the trial parameters, the trial parameters including: the synchronization delay sample dataset and the expected synchronization delay value;

[0143] The input module is used to input the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node and the number of threads in the extraction and transformation stage, the trial parameters, and the amount of synchronized data in the data synchronization statistics into the artificial intelligence AI trial model.

[0144] The AI ​​simulation module is used to simulate the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and transformation stage based on the synchronization delay sample dataset using the AI ​​simulation model. It performs AI simulation based on the expected value of the synchronization delay and outputs the optimal parallelism and the optimal number of threads in the extraction and transformation stage of the working node when transmitting the amount of synchronization data. The learning simulation is performed using a support vector machine regression model.

[0145] Optionally, in another embodiment, based on the above embodiment, the AI ​​trial calculation module is specifically used to utilize the AI ​​trial calculation model to select a corresponding AI decision strategy based on the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node and the number of threads in the extraction and transformation stage, and the amount of synchronized data in the data synchronization statistics, and to simulate the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node and the number of threads in the extraction and transformation stage for AI trial calculation, wherein the AI ​​decision strategy needs to satisfy: the delay duration of the working node is greater than the expected value of the synchronization delay.

[0146] Optionally, in another embodiment, based on the above embodiments, the adjustment module includes:

[0147] An efficiency determination module is used to determine the optimal synchronization efficiency of the working node based on the optimal parallelism of the working node, the optimal number of threads in the extraction and transformation stage, and the data synchronization statistics.

[0148] The comparison module is used to compare the optimal synchronization efficiency of the working node with the execution efficiency of the working node.

[0149] The parameter adjustment module is used to adjust the parallelism of the working node and the number of threads in the extraction and transformation process based on the comparison results.

[0150] Optionally, in another embodiment, based on the above embodiments, the apparatus further includes: a training module for pre-training an AI trial model.

[0151] The training module includes:

[0152] The dataset acquisition module is used to acquire the training dataset, which includes: synchronous data related to the AI ​​decision-making strategy, historical synchronous data, parallelism, and the number of threads in the extraction and transformation process.

[0153] The preprocessing module is used to preprocess the training set;

[0154] The building module is used to construct an AI trial model based on the support vector machine algorithm from the preprocessed dataset;

[0155] The iterative training module is used to iteratively train the AI ​​trial model using the acquired historical synchronous sample data, and outputs the parallelism of the sample nodes and the number of threads in the extraction and transformation process.

[0156] The model determination module is used to obtain a trained AI trial model when the parallelism of the sample node and the number of threads in the extraction and transformation process meet the corresponding set thresholds.

[0157] Optionally, in another embodiment, based on the above embodiments, the apparatus further includes:

[0158] The strategy setting module is used to formulate AI decision-making strategies for the AI ​​trial model under different situations during iterative training, based on the real-time synchronization requirements and training objectives. The AI ​​decision-making strategies are used to guide the output and decision selection of the AI ​​trial model.

[0159] The strategy optimization module is used to iteratively train the AI ​​trial model based on the AI ​​decision-making strategies for the different situations, and continuously optimize the AI ​​decision-making strategies based on the iterative training results.

[0160] Please also see Figure 6 This is a block diagram of a data synchronization system provided in an embodiment of the present invention. The system includes: a data synchronization module 601, an AI decision center module 602, and an AI trial calculation module 603, wherein...

[0161] The data synchronization module 601 is used to collect data synchronization statistics from each node in real time or at regular intervals, set the parallelism of the synchronization data in the data synchronization statistics according to the deployment environment, and sequentially capture, extract, transform, and load the synchronization data. The processed data synchronization statistics of each node, the set parallelism of each node, and the number of threads in the extraction and transformation stages are sent to the AI ​​decision center module 602. The capture and loading are a serial mechanism to ensure the timeliness of data processing, and the extraction and transformation are parallel accelerated transmission processes based on the set parallelism.

[0162] The AI ​​decision center module 602 is used to determine the execution efficiency of the working node based on the data synchronization statistics received from each node, as well as the parallelism and the number of threads in the extraction and transformation process set by each node, and to send the data synchronization statistics of the nodes whose operation conforms to the AI ​​trial calculation strategy rules to the AI ​​trial calculation module 603.

[0163] The AI ​​trial calculation module 603 is used to simulate and calculate the amount of synchronized data in the received data synchronization statistics using an artificial intelligence AI trial calculation model, to obtain the optimal parallelism of the node and the optimal number of threads in the extraction and transformation process. The AI ​​trial calculation model is a model obtained after simulation and training based on the support vector machine algorithm, and the optimal parallelism of the node and the optimal number of threads in the extraction and transformation process are sent to the AI ​​decision center module 602.

[0164] The AI ​​decision center module 603 is also used to determine the optimal synchronization efficiency of the node based on the optimal parallelism of the node and the optimal number of threads in the extraction and transformation stage, as well as the data synchronization statistics, and to compare the optimal synchronization efficiency of the node with the execution efficiency of the working node; and to send the comparison result to the data synchronization module 601 for corresponding adjustments.

[0165] The data synchronization module 601 is further configured to adjust the parallelism of the node and the number of threads in the extraction and transformation process based on the received comparison results, and synchronize the synchronization data in the data synchronization statistics based on the adjusted parallelism of the node and the number of threads in the extraction and transformation process.

[0166] Please also see Figure 7 This diagram illustrates an application example of a data synchronization system provided in an embodiment of the present invention. The system includes: a data synchronization module, an AI decision center module, and an AI trial calculation module.

[0167] The data synchronization module comprises two parts: a synchronization data processing part and an AI adjustment part. The synchronization data processing part collects log data from database 2, specifically using existing technology based on real-time incremental data acquisition of database logs, and synchronizes the acquired data to the target storage database (i.e., the target storage in the diagram). The synchronization process includes: selection, extraction, transformation, and loading. Selection / loading is a serial mechanism to ensure processing timeliness, while extraction / transformation is parallel to accelerate transmission efficiency. The degree of parallelism in the synchronization is set according to the deployment environment.

[0168] The AI ​​adjustment (AI intelligent adjustment) part is used to achieve self-adjustment and addition. It reports the statistical data of synchronous operation to the AI ​​decision center in real time through the Dubbo protocol, and receives the optimal parallelism and the number of threads in the extraction and transformation process from the AI ​​decision center. Based on the optimal parallelism and the number of threads in the extraction and transformation process, it automatically adjusts the received data and modifies the parameters of the parameter resource pool.

[0169] The AI ​​Decision Center module is used to push the working node parameters that need to be simulated to the AI ​​Trial Calculation module. The trial calculation parameters include the sample dataset of synchronization delay and the expected value of synchronization delay, such as 0.5.

[0170] The AI ​​trial calculation module, upon receiving the worker node parameters, simulates the worker node's hardware parameters, parallelism, and the number of threads in the extraction and transformation stages based on a sample dataset with synchronization latency obtained from Database 1. It then initiates AI trials (including capture, extraction, transformation, and loading processes), stores the processing results in the target storage database (i.e., the target storage in the diagram), performs self-learning training based on the expected synchronization latency value, and outputs the optimal parallelism and number of threads in the extraction and transformation stages for the worker node under the current synchronization data volume. It should be noted that the data obtained from Database 1 and Database 2 can be the same or related, and this embodiment does not impose any restrictions.

[0171] The AI ​​Decision Center module is used to collect real-time data synchronization statistics and hardware parameters of the data synchronization work nodes. Based on the AI ​​trial calculation strategy, when the operation of the data synchronization work nodes conforms to the AI ​​trial calculation strategy rules (generally, the work node latency exceeds the expected value), the AI ​​Decision Center initiates relevant information push AI trial calculations. Based on the AI ​​trial calculation results and AI decision adjustments, it initiates adaptive adjustments to the data synchronization work nodes (parallelism, number of extraction and loading threads). The AI ​​Decision Center module stores AI decision adjustment strategies and AI trial calculation strategies for various scenarios. Simultaneously, the AI ​​Decision Center module can also perform operations such as receiving trial calculation results, pushing work node parameters, pushing sample datasets and expected synchronization latency values, and collecting work node data.

[0172] The specific implementation process of each module in this embodiment is detailed in the corresponding parts of the above embodiments, and will not be repeated here.

[0173] In this embodiment of the invention, not only is a machine learning support vector machine algorithm introduced, but deep learning is also combined with factors such as the parallelism of synchronization, the number of extraction threads, and the number of transformation threads to achieve intelligent adaptive adjustment of data synchronization. It can also overcome the performance bottlenecks caused by the surge in the amount of synchronized data in traditional data synchronization technologies, solving the problem of increased data synchronization latency.

[0174] This invention achieves trial sample data push and storage configuration through dynamic configuration, improving the device's flexibility and adapting to various scenarios. By introducing machine learning algorithms, it solves the synchronization delay problem that occurs when the amount of data to be synchronized suddenly increases, greatly improving synchronization efficiency and reducing synchronization latency by 80%-20% compared to industry data synchronization technologies.

[0175] The embodiments of the present invention dynamically match the optimal parallelism and extract the number of threads in the transformation process based on the support vector machine algorithm, and correct the synchronization efficiency of the nodes, which can significantly optimize and improve the stability and real-time performance of data synchronization.

[0176] Optionally, embodiments of the present invention also provide an electronic device, comprising:

[0177] processor;

[0178] Memory used to store the processor's executable instructions;

[0179] The processor is configured to execute the instructions to implement the synchronization method described above.

[0180] Optionally, embodiments of the present invention also provide a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the data synchronization method described above.

[0181] Optionally, embodiments of the present invention also provide a computer program product, including a computer program or instructions, which, when executed by a processor of an electronic device, implement the data synchronization method described above.

[0182] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0183] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0184] Figure 8 This is a block diagram of an electronic device 800 provided in an embodiment of the present invention. For example, the electronic device 800 can be a mobile terminal or a server; in this embodiment, a mobile terminal is used as an example for explanation. For example, the electronic device 800 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0185] Reference Figure 8 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0186] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0187] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0188] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0189] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0190] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0191] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0192] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0193] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0194] In an embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the data synchronization method described above.

[0195] In this embodiment, a computer-readable storage medium is also provided, which, when executed by a processor of an electronic device, enables the electronic device 800 to perform the data synchronization method described above. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0196] In one embodiment, a computer program product is also provided, including a computer program or instructions that, when executed by the processor 820 of the electronic device 800, cause the electronic device 800 to perform the data synchronization method described above.

[0197] Figure 9 This is a block diagram of a data synchronization device 900 provided in an embodiment of the present invention. For example, device 900 can be provided as a server. (See also...) Figure 9 The apparatus 900 includes a processing component 922, which further includes one or more processors, and memory resources represented by memory 932 for storing instructions, such as application programs, that can be executed by the processing component 922. The application programs stored in memory 932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 922 is configured to execute instructions to perform the methods described above.

[0198] The device 900 may also include a power supply component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 can operate on an operating system stored in memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0199] In this embodiment of the invention, the term "module" refers to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as a processor or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules. Furthermore, each module can be part of a larger module that includes the functionality of that module.

[0200] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0201] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data synchronization method, characterized in that, include: Obtain data synchronization statistics for the worker nodes, as well as the parallelism and number of threads for the extraction and transformation process set for the worker nodes; The execution efficiency of the work node is determined based on the parallelism of the work node, the number of threads in the extraction and transformation process, and the data synchronization statistics. The amount of synchronized data in the data synchronization statistics is simulated and calculated using an artificial intelligence (AI) trial calculation model to obtain the optimal parallelism of the working node and the optimal number of threads in the extraction and transformation process. The AI ​​trial calculation model is a model obtained after simulation and training based on the support vector machine algorithm. Based on the execution efficiency of the work node, the optimal parallelism of the work node, and the optimal number of threads in the extraction and transformation stage, adjust the parallelism of the work node and the number of threads in the extraction and transformation stage. Based on the adjusted parallelism of the working nodes and the number of threads in the extraction and transformation process, the synchronized data in the data synchronization statistics are synchronized. The method of using an artificial intelligence (AI) simulation model to calculate the amount of synchronized data in the data synchronization statistics to obtain the optimal parallelism of the working node and the optimal number of threads in the extraction and transformation process includes: Obtain the hardware parameters of the working node, the synchronization medium information and trial parameters of the working node, the trial parameters including: synchronization delay sample dataset and expected synchronization delay value; The hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node and the number of threads in the extraction and transformation stage, the trial calculation parameters, and the amount of synchronized data in the data synchronization statistics are input into the artificial intelligence (AI) trial calculation model. When the latency of the working node exceeds the expected synchronization latency, the AI ​​simulation model is used to simulate the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and conversion stage based on the synchronization latency sample dataset. AI simulation is performed, and learning simulation is conducted based on the expected synchronization latency. The optimal parallelism and the optimal number of threads in the extraction and conversion stage of the working node are output when transmitting the amount of synchronization data. The learning simulation uses a support vector machine regression model for learning simulation.

2. The data synchronization method according to claim 1, characterized in that, The AI ​​simulation model is used to calculate the hardware parameters of the worker node, the synchronization medium information of the worker node, the parallelism of the worker node, and the number of threads in the extraction and transformation process. The calculations include: Using the AI ​​trial model based on the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, the number of threads in the extraction and transformation stage, and the amount of synchronized data in the data synchronization statistics, a corresponding AI decision strategy is selected to simulate the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and transformation stage for AI trial calculation. The AI ​​decision strategy must satisfy the following: the latency of the working node is greater than the expected synchronization latency.

3. The data synchronization method according to claim 1, characterized in that, The adjustment of the parallelism of the worker node and the number of threads in the extraction and transformation stage based on the execution efficiency of the worker node, the optimal parallelism of the worker node, and the optimal number of threads in the extraction and transformation stage includes: The optimal synchronization efficiency of the working node is determined based on the optimal parallelism of the working node, the optimal number of threads in the extraction and transformation stage, and the data synchronization statistics. The optimal synchronization efficiency of the working node is compared with the execution efficiency of the working node; Adjust the parallelism of the working nodes and the number of threads in the extraction and transformation process based on the comparison results.

4. The data synchronization method according to any one of claims 1 to 3, characterized in that, The method further includes: pre-training an AI trial model, the training process of which includes: Obtain a training dataset, which includes: synchronous data related to AI decision-making strategies, historical synchronous data, parallelism, and the number of threads in the extraction and transformation process; The training dataset is preprocessed; The preprocessed dataset is used to build an AI trial model based on the support vector machine algorithm; The AI ​​trial model is iteratively trained using the acquired historical synchronous sample data, and the parallelism of the sample nodes and the number of threads in the extraction and transformation process are output. When the parallelism of the sample node and the number of threads in the extraction and transformation process meet the corresponding set thresholds, a trained AI trial model is obtained.

5. The data synchronization method according to claim 4, characterized in that, The method further includes: During iterative training, based on the need for real-time synchronization and training objectives, AI decision-making strategies for different situations are formulated for the AI ​​trial model. These AI decision-making strategies are used to guide the output and decision selection of the AI ​​trial model. The AI ​​trial model is iteratively trained based on the AI ​​decision-making strategies for the different situations, and the AI ​​decision-making strategies are continuously optimized based on the iterative training results.

6. A data synchronization device, characterized in that, include: The acquisition module is used to acquire data synchronization statistics of the worker nodes, as well as the parallelism and number of threads in the extraction and transformation process set by the worker nodes; The determination module is used to determine the execution efficiency of the work node based on the parallelism of the work node, the number of threads in the extraction and transformation stage, and the data synchronization statistics. The simulation calculation module is used to simulate and calculate the amount of synchronized data in the data synchronization statistics using an artificial intelligence (AI) trial calculation model, so as to obtain the optimal parallelism of the working node and the optimal number of threads in the extraction and transformation stage. The AI ​​trial calculation model is a model obtained after simulation calculation training based on the support vector machine algorithm. An adjustment module is used to adjust the parallelism of the work node and the number of threads in the extraction and transformation stage based on the execution efficiency of the work node, the optimal parallelism of the work node, and the optimal number of threads in the extraction and transformation stage. The synchronization module is used to synchronize the synchronization data in the data synchronization statistics based on the adjusted parallelism of the working nodes and the number of threads in the extraction and transformation process. The simulation and calculation module is also used to obtain the hardware parameters of the working node, the synchronization medium information of the working node, and the trial calculation parameters, which include: the synchronization delay sample dataset and the expected synchronization delay value; The hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node and the number of threads in the extraction and transformation stage, the trial calculation parameters, and the amount of synchronized data in the data synchronization statistics are input into the artificial intelligence (AI) trial calculation model. When the latency of the working node exceeds the expected synchronization latency, the AI ​​simulation model is used to simulate the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and conversion stage based on the synchronization latency sample dataset. AI simulation is performed, and learning simulation is conducted based on the expected synchronization latency. The optimal parallelism and the optimal number of threads in the extraction and conversion stage of the working node are output when transmitting the amount of synchronization data. The learning simulation uses a support vector machine regression model for learning simulation.

7. A data synchronization system, characterized in that, include: The module includes a data synchronization module, an AI decision center module, and an AI trial calculation module. The data synchronization module is used to collect data synchronization statistics from each node in real time or at regular intervals, set the parallelism of the synchronization data in the data synchronization statistics according to the deployment environment, and sequentially capture, extract, transform, and load the synchronization data. It then sends the processed data synchronization statistics of each node, the set parallelism of each node, and the number of threads in the extraction and transformation stages to the AI ​​decision center module. The capture and loading are performed serially, while the extraction and transformation are performed in parallel to accelerate transmission based on the set parallelism. Based on the comparison results received from the AI ​​decision center module, the parallelism of the nodes and the number of threads in the extraction and transformation stages are adjusted, and data is synchronized based on the adjusted parallelism and number of threads in the extraction and transformation stages. The AI ​​decision center module is used to determine the execution efficiency of worker nodes based on the received data synchronization statistics from each node, as well as the parallelism and number of threads in the extraction and transformation stages set for each node; and to send the data synchronization statistics of nodes whose operation conforms to the AI ​​trial calculation strategy rules to the AI ​​trial calculation module; to determine the optimal synchronization efficiency of a node based on the optimal parallelism and optimal number of threads in the extraction and transformation stages, as well as the data synchronization statistics; to compare the optimal synchronization efficiency of a node with the execution efficiency of the worker nodes; and to send the comparison result to the data synchronization module for corresponding adjustments. The AI ​​trial calculation module is used to simulate and calculate the amount of received synchronous data, obtain the optimal parallelism of the node and the optimal number of threads in the extraction and transformation process, and send them to the AI ​​decision center module. The AI ​​trial calculation module is a model obtained after simulation and training based on the support vector machine algorithm. It also obtains the hardware parameters of the working node, the synchronization medium information of the working node, and the trial calculation parameters. The hardware parameters, synchronization medium information, parallelism, number of threads in the extraction and transformation process, trial parameters, and synchronization data volume of the working node are input into the artificial intelligence (AI) trial model; the trial parameters include: the synchronization delay sample dataset and the expected synchronization delay value; When the latency of the working node exceeds the expected synchronization latency, the AI ​​simulation module simulates the hardware parameters of the working node, the synchronization medium information of the working node, the parallelism of the working node, and the number of threads in the extraction and conversion stage based on the synchronization latency sample dataset. Based on the expected synchronization latency, the module performs a learning simulation to output the optimal parallelism and the optimal number of threads in the extraction and conversion stage for the working node when transmitting the amount of synchronization data. The learning simulation uses a support vector machine regression model for learning simulation.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the synchronization method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the data synchronization method as described in any one of claims 1 to 5.

Images