Data timing synchronization processing method and device based on timing scheduling platform, and terminal
By separating timing task management from business logic and deploying independently using the xxl-job platform, the problem of limited scalability and stability risks in the timing task processing mode is solved, and flexible timing task management and efficient system operation are achieved.
Patent Information
- Application Number
- CN202510274907.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-10
- Publication Date
- 2025-07-04
AI Technical Summary
In the prior art, the scalability of the timing task processing mode is limited, and the business code and timing task logic are overcoupled, making it difficult to flexibly and efficiently respond to new timing tasks or task changes, and the operation stability is high, which can easily affect the normal operation of the system.
The timed scheduling platform xxl-job is adopted to separate the timing task management from the main business logic, and deploy it from the business services on independent servers. New synchronization tasks are created through xxl-job, task scripts are written using lightweight scripting language, and task execution is controlled by the scheduled scheduling platform, and periodic scheduling is supported by minutes, hours, days, etc., and execution parameters are adjusted according to environment variables.
It realizes effective decoupling of timing tasks and business logic, improves the clarity and maintainability of the system architecture, has good scalability and stability, avoids competition for CPU resources, and ensures continuous operation of the business.
Smart Images
Figure CN120256045A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data timing synchronization, and particularly relates to a data timing synchronization processing method, device, intelligent terminal and storage medium based on a timing scheduling platform. Background Art
[0002] In the early stage of software development, there were no such rich and mature timing task scheduling frameworks and platforms as there are now. Developers had relatively few tools and technologies to choose from. In the traditional timing task processing mode, a common practice was to embed the timing synchronization-related logic into the business code inside the service program. For example, when the service was built and supported by JAVA, the timing synchronization logic was often directly written in the business code of the JAVA service.
[0003] The disadvantages of the existing technology for timing task processing are as follows: limited scalability: the business code is overly coupled with the timing task logic, severely restricting the system's expansion ability. When facing new timing tasks or changes to existing tasks, it is difficult to respond flexibly and efficiently, hindering business development and innovation.
[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a data timing synchronization processing method, device, intelligent terminal and storage medium based on a timing scheduling platform in view of the problems and defects of the above-mentioned existing technology. The present invention realizes the effective decoupling of timing tasks and conventional business logic by means of the timing scheduling platform xxl-job, and solves the problems such as limited scalability and operation stability risks existing in the traditional timing task processing mode.
[0006] The technical solution adopted by the present invention to solve the problem is as follows: A data timing synchronization processing method based on a timing scheduling platform, which includes: Pre-deploy the timing scheduling platform xxl-job, separate the timing task management from the main business logic, and use the deployed timing scheduling platform xxl-job as a dedicated scheduling platform; and deploy the timing scheduling platform and the business service on separate servers; Create a synchronization task in the deployed timing scheduling platform xxl-job; Obtain the written task script that needs to be synchronized regularly; Add the task script to the task management process of the timing scheduling platform xxl-job, and obtain the set synchronization time; According to the set synchronization time and the newly created synchronization task, the timing scheduling platform xxl-job controls the execution of the corresponding task script to control the data timing synchronization.
[0007] The data timing synchronization processing method based on the timing scheduling platform, wherein, for the step of the timing scheduling platform xxl-job controlling the execution of the corresponding task script according to the set synchronization time and the newly created synchronization task to control the data timing synchronization, it further includes: Controlling the timing scheduling platform xxl-job to adjust the time interval of task execution according to actual requirements for periodic scheduling in time units of minutes, hours or days.
[0008] The data timing synchronization processing method based on the timing scheduling platform, wherein, for the step of the timing scheduling platform xxl-job controlling the execution of the corresponding task script according to the set synchronization time and the newly created synchronization task to control the data timing synchronization, it further includes: The task script automatically adjusts the execution parameters according to the environmental variables or system status during runtime to control the data timing synchronization.
[0009] The data timing synchronization processing method based on the timing scheduling platform, wherein, for the step of the timing scheduling platform xxl-job controlling the execution of the corresponding task script according to the set synchronization time and the newly created synchronization task to control the data timing synchronization, it further includes: When a new timing task requirement is detected, it expands on the basis of the task script to add new task modules or modify the scheduling configuration.
[0010] The data timing synchronization processing method based on the timing scheduling platform, wherein the task script in the step of obtaining the written task script that needs to be synchronized regularly implements the timing task using the lightweight scripting language Python or NodeJS.
[0011] The data timing synchronization processing method based on the timing scheduling platform, wherein the steps of the timing scheduling platform xxl-job controlling the execution of the corresponding task script according to the set synchronization time and the newly created synchronization task to control the data timing synchronization include: When detecting that a user accesses the system page, collecting the user's page usage information; Based on the collected page usage information, saving the brief page information; The timing scheduling platform xxl-job regularly analyzes the data types that need to be synchronized according to the set synchronization time and the newly created synchronization task, and controls the insertion of the synchronized data into the data table to control the data timing synchronization.
[0012] The data timing synchronization processing method based on a timing scheduling platform, wherein the steps of the timing scheduling platform xxl-job controlling the execution of corresponding task scripts according to the set synchronization time and the newly created synchronization task to control data timing synchronization include: When it detects an access page operation, it collects access information through logging and saves the brief page information to the registry; The timing scheduling platform xxl-job obtains the brief information required for synchronizing data from the registry; The timing scheduling platform xxl-job controls the timing synchronization of data to the database according to the set synchronization time and the newly created synchronization task.
[0013] A data timing synchronization processing device based on a timing scheduling platform, wherein the device includes: A deployment and task separation module, which is used to pre-deploy the timing scheduling platform xxl-job, separate the timing task management from the main business logic, and use the deployed timing scheduling platform xxl-job as a dedicated scheduling platform; and deploy the timing scheduling platform and the business service on separate servers respectively; A synchronization task creation module, which is used to create a synchronization task on the deployed timing scheduling platform xxl-job; A task script extraction module, which is used to obtain the written task scripts that need to be synchronized regularly; A task script addition and time setting module, which is used to add the task script to the task management process of the timing scheduling platform xxl-job and obtain the set synchronization time; A data synchronization control module, which is used to control the execution of corresponding task scripts by the timing scheduling platform xxl-job according to the set synchronization time and the newly created synchronization task to control data timing synchronization.
[0014] An intelligent terminal, which includes a memory, and one or more programs, where one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include those for executing the method described in any one of the above.
[0015] A computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of an electronic device, the electronic device can execute the method described in any one of the above.
[0016] Advantages of the present invention: The present invention provides a method, apparatus, intelligent terminal, and storage medium for data timing synchronization processing based on a timing scheduling platform; by means of the timing scheduling platform xxl-job, the present invention effectively decouples timing tasks from normal business logic. In this way, the management and maintenance of timing tasks no longer tightly depend on the main business logic code, significantly improving the clarity and maintainability of the system architecture. And the present invention also has the following advantages: 1). The time interval of task execution can be flexibly adjusted, supporting periodic scheduling in conventional time units such as minutes, hours, days, etc.
[0017] 2). For various parameters during task execution, such as data source address, release environment, statistical time, etc., a dynamic adaptation mechanism can be selected to be implemented inside the script. In this way, the script can automatically adjust execution parameters according to runtime environment variables or system status, so as to more efficiently adapt to different business scenarios and environmental changes.
[0018] 3). It has good scalability. When there are new timing task requirements, new task modules can be easily added or scheduling configurations can be modified on the basis of existing scripts without interfering with the existing task logic.
[0019] 4). In the system architecture of the present invention, the timing scheduling platform and business services are placed on separate servers, fundamentally eliminating CPU resource competition and effectively ensuring the stable and efficient operation of the system. Due to physical isolation, the timing scheduling platform can schedule independently according to its own task plan and load without affecting the CPU resources of business services. When maintenance operations such as version upgrade, configuration change, or restart are performed on business services, the timing scheduling platform can run independently, maintaining the continuity of timing tasks and system stability, enhancing the anti-interference ability of the architecture, and ensuring continuous business operation. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0021] Figure 1 It is a schematic diagram of an application terminal of the data timing synchronization processing method based on a timing scheduling platform provided in Embodiment 1 of the present invention.
[0022] Figure 2 It is a schematic flowchart of the data timing synchronization processing method based on a timing scheduling platform provided in Embodiment 1 of the present invention.
[0023] Figure 3 It is a schematic diagram of the overall service timing structure of the data timing synchronization processing method provided in Embodiment 1 of the present invention based on a timing scheduling platform.
[0024] Figure 4 It is a schematic diagram of the overall service process of a data timing synchronization processing method based on a timing scheduling platform according to an embodiment of the present invention.
[0025] Figure 5 It is a schematic diagram of the usage process of the timing scheduling platform of a data timing synchronization processing method based on a timing scheduling platform according to an embodiment of the present invention.
[0026] Figure 6 The principle block diagram of the data timing synchronization processing device embodiment provided by the present invention based on a timing scheduling platform.
[0027] Figure 7 It is the internal structure principle block diagram of the intelligent terminal provided by an embodiment of the present invention. Detailed implementation manners
[0028] To make the purpose, technical solutions and advantages of the present invention clearer and more definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.
[0029] In the early stage of technology development, due to limited technology choices, small system scale, low degree of operation and maintenance automation, and insufficient awareness of separation of concerns, etc., developers tend to directly implement the timing task logic in the service code to quickly meet the requirements.
[0030] With the development of technology and the expansion of the application scale, the drawbacks of this approach in the prior art gradually emerge. The disadvantages of the prior art are mainly reflected in the following aspects: 1. Limited scalability: The business code is overly coupled with the timing task logic, seriously restricting the system's expansion ability. When facing new timing tasks or changes to existing tasks, it is difficult to respond flexibly and efficiently, hindering business development and innovation.
[0031] 2. Risk of running stability: The later maintenance and update operations of the timing service logic are extremely likely to interfere with the normal operation of the service, and may cause problems such as service interruption and performance degradation, reducing the reliability and availability of the system.
[0032] To solve the above technical problems, as Figure 1 shown, it is the application environment of the data timing synchronization processing method based on a timing scheduling platform in an embodiment. Refer to Figure 1, The data timing synchronization processing method based on the timing scheduling platform is applied to the user terminal 10. Among them, the user terminal 10 can be an independent user terminal, such as a mobile phone, a tablet computer, a laptop computer, a wearable device or a desktop computer, etc.; the user terminal 10 is communicatively connected to the server 20 through a network, and the server processes the relevant access services of the user terminal. The server 20 can be implemented by an independent server 20 or a server group composed of multiple servers 20, and can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In this embodiment, the user terminal 10 can be used as the front end for running the data timing synchronization processing of the present invention, and the server 20 can be used as the back end for running the data timing synchronization processing of the present invention. In this way, when the server 20 obtains a request from a user in the user terminal 10 to access the system page through the client application, it can quickly establish a long connection with the user terminal 110.
[0033] The present invention is based on the timing scheduling platform xxl-job to achieve an effective decoupling of timing tasks and normal business logic. In this way, the management and maintenance of timing tasks no longer tightly depend on the main business logic code, significantly improving the clarity and maintainability of the system architecture.
[0034] Among them, the timing scheduling platform xxl-job adopted in the embodiment of the present invention is an open-source distributed task scheduling platform, which can solve the timing task scheduling problem in a distributed system, and has characteristics such as lightweight, easy to expand, and highly available, and is widely used in enterprise-level timing task management scenarios.
[0035] Next, the data timing synchronization processing method based on the timing scheduling platform provided by the embodiments of the present application will be introduced and described in detail through several specific embodiments.
[0036] As Figure 2 shown, a data timing synchronization processing method based on the timing scheduling platform in Embodiment 1 of the present invention includes the following steps: Step S100, pre-deploy the timing scheduling platform xxl-job, separate the timing task management from the main business logic, and use the deployed timing scheduling platform xxl-job as a dedicated scheduling platform; and deploy the timing scheduling platform and the business service on independent servers respectively; In the embodiments of the present invention, specifically during implementation, first, an independent timed task scheduling platform needs to be built. The timed scheduling platform xxl-job is selected in the present invention, and all timed tasks are uniformly managed and scheduled. And the timed task code originally embedded in the main business system is stripped out, and the business system is no longer directly responsible for the execution of timed tasks. Finally, xxl-job and business services are deployed on different servers to achieve physical isolation.
[0037] That is, in the system architecture adopted in the embodiments of the present invention, the timed scheduling platform and business services are placed on separate servers, which fundamentally eliminates CPU resource competition and effectively guarantees the stable and efficient operation of the system. Due to physical isolation, the timed scheduling platform can independently schedule according to its own task plan and load without affecting the CPU resources of business services. When maintenance operations such as version upgrade, configuration change or restart are performed on business services, the timed scheduling platform can run independently, maintaining the continuity of timed tasks and system stability, enhancing the anti-interference ability of the architecture, and ensuring the continuous operation of the business.
[0038] Among them, the timed scheduling platform xxl-job in the embodiments of the present invention is also called the distributed task scheduling platform XXL-JOB. It is an open-source distributed task scheduling platform that can provide a simple, easy-to-use, highly available, and high-performance task scheduling solution. It is similar to an automated "timed task manager", but with more powerful functions and can handle more complex distributed environments.
[0039] The main components of the timed scheduling platform xxl-job deployed in the embodiments of the present invention include: Scheduling Center (Admin): Responsible for the scheduling and management of tasks, including task creation, modification, deletion, start, stop, etc.
[0040] Executor: Responsible for executing tasks, receiving instructions from the scheduling center and executing corresponding tasks.
[0041] Through the deployed timed scheduling platform xxl-job, the present invention is like an intelligent alarm clock that can automatically execute specified tasks according to the set time and rules; it is suitable for use in distributed systems and can more conveniently manage and schedule various timed tasks.
[0042] In the embodiments of the present invention, the management of the system's timed tasks is separated from the main business logic, and the deployed timed scheduling platform xxl-job is used as a dedicated scheduling platform.
[0043] In this way, separating the timed tasks from the main business logic reduces the coupling degree between systems, and modifying or upgrading timed tasks will not affect the core business.
[0044] In the embodiments of the present invention, centralized management of timed tasks is adopted, which improves maintainability and facilitates viewing, modification, monitoring, and maintenance. Moreover, when the number of timed tasks increases, the demand can be met by expanding the xxl-job cluster without modifying the business system code, thus having scalability.
[0045] Furthermore, resource isolation is adopted, so the execution of timed tasks does not occupy the resources of the business server, avoiding impact on business performance; and it has fault tolerance, even if xxl-job fails, it will not affect the normal operation of the main business.
[0046] Step S200: Create a synchronization task in the deployed timed scheduling platform xxl-job. In the embodiments of this step, on the management interface of the deployed timed scheduling platform xxl-job, a new task is created. Information such as a unique ID, task description, and person in charge needs to be specified for this task.
[0047] For example, when a simple task is required: every 5 minutes, synchronize the data in MySQL database A to MySQL database B.
[0048] Prerequisites: In the previous steps, the XXL-JOB scheduling center (Admin) and at least one executor (Executor) have been successfully deployed. And MySQL databases A and B have been created and can access each other. There is a script or program for executing data synchronization. For example, a simple synchronization script can be written in Python.
[0049] Steps: S101: Log in to the deployed timed scheduling platform xxl-job scheduling center: Enter the address of the XXL-JOB scheduling center in the browser, such as http: / / localhost:8080 / xxl-job-admin (the specific port number and path may vary depending on the deployment). Log in with the default account password admin / 123456.
[0050] S102: Create an executor. If an executor has not been created yet, an executor needs to be created on the "Executor Management" page. Fill in the following information: Appname: The application name of the executor, such as data-sync-executor (needs to be consistent with xxl.job.executor.appname in the deployed executor configuration). Title: The description information of the executor, such as data synchronization executor. Registration method: Select "Automatic registration". Address list: Leave it blank because we use automatic registration. Click the "Save" button.
[0051] S103. Create a task: Click the "Task Management" menu. Click the "New" button.
[0052] Fill in the following information: JobId: (Automatically generated, no need to fill in); Executor selection: Select the executor data-sync-executor created just now.
[0053] Task description: For example, synchronize database A to database B every 5 minutes.
[0054] Cron: Fill in the Cron expression, such as 0 / 5 * * * * (indicating that it is executed once every 5 seconds. Here, for demonstration purposes, it is recommended to set at least 5 minutes in the actual production environment). If you want to execute once every 5 minutes, you can set it to 0 0 / 5 * * *.
[0055] Running mode: Select "BEAN mode".
[0056] JobHandler: Fill in the corresponding method name in the executor, such as dataSyncTask (this requires defining a method named dataSyncTask in the executor code and adding the @XxlJob annotation).
[0057] Parameters: Fill in the parameters required to execute the task, such as dbA_host=192.168.1.100&dbA_port=3306&dbA_user=root&dbA_password=password&dbB_host=192.168.1.101&dbB_port=3306&dbB_user=root&dbB_password=password (in this example, we directly pass the database connection information as parameters to the executor. In the actual production environment, it is recommended to use a more secure method, such as reading from a configuration file or configuration center).
[0058] Blocking handling strategy: Select "Single machine serial" (select the appropriate strategy according to the requirements).
[0059] Timeout: Set the timeout of the task, such as 300 (unit: seconds).
[0060] Failure retry times: Set the number of times to retry after the task fails, such as 3.
[0061] Responsible person: Fill in the name of the responsible person.
[0062] Alarm email: Fill in the alarm email for receiving alarm emails when the task execution fails.
[0063] Click the "Save" button.
[0064] In this way, through xxl-job, the present invention can clearly manage all scheduled synchronization tasks, including task names, descriptions, statuses, execution histories, etc. And different task management permissions can be assigned to different users to ensure security.
[0065] Step S300: Obtain the written task script that needs to be synchronized regularly; In the embodiment of the present invention, a script (which can be Shell, Python, Java, etc.) is written, and this script is responsible for performing specific data synchronization operations. For example, reading data from one database and then writing it to another database.
[0066] In the embodiment of the present invention, the technology stack used by the scheduling platform script is flexible, and the timing scheduling platform can be independently selected. For example, lightweight Python / NodeJS scripts. When introducing common dependencies such as Redis and databases, the code volume is small, the installation is convenient, and the technology stack can be updated independently easily. This reduces the technology expansion cost, reduces the impact on the business system, maintains the technological advancement of the platform, and flexibly adapts to business requirements and technological development.
[0067] In the embodiment of the present invention, any programming language can be used to write the synchronization script to meet different synchronization requirements. And the synchronization logic can be customized according to specific business scenarios, such as data conversion, cleaning, etc.
[0068] Step S400: Add the task script to the task management process of the timing scheduling platform xxl-job and obtain the set synchronization time; In the embodiment of the present invention, since XXL-JOB is a task scheduling platform, it does not directly store and manage task scripts itself. Its core lies in managing task configurations (such as executors, Cron expressions, parameters, etc.), and then the executor is responsible for running the script or code.
[0069] Therefore, adding the task script to the task management process in this step actually means: deploying the task script (or the code containing task logic) to the executor of XXL-JOB. Create a task in the XXL-JOB scheduling center (Admin), configure the executor and scheduling time (Cron expression) so that the scheduling center can trigger the task script on the executor.
[0070] Among them, configuring the written synchronization script into the task of xxl-job includes specifying the path of the script, execution parameters, etc. At the same time, set the trigger time of the task, for example, execute at 2 am every day; In the embodiments of the present invention, there is no need to manually execute the synchronization script, and xxl-job will automatically trigger tasks according to the set time. Moreover, the timed execution in the embodiments of the present invention can precisely control the execution time of tasks to meet different synchronization requirements.
[0071] Step S500: According to the set synchronization time and the newly created synchronization task, the timed scheduling platform xxl-job controls the execution of the corresponding task script to control the timed synchronization of data.
[0072] In the embodiments of the present invention, the timed scheduling platform xxl-job will automatically execute the corresponding synchronization script according to the set time. After the script execution is completed, xxl-job will record the execution log of the task, which is convenient for monitoring and troubleshooting.
[0073] Specifically, the core of the timed scheduling platform xxl-job lies in its precise scheduling mechanism. When a data synchronization task is created and the synchronization time is set (through a Cron expression), the scheduling center (Admin) of the timed scheduling platform xxl-job assumes the role of a "commander". It will trigger the execution of the task at a specific time point according to the set Cron expression.
[0074] This trigger does not directly run the script, but sends an instruction to the selected executor, telling it that it is time to execute this task. After receiving the instruction, the executor will take corresponding actions according to the running mode defined in the task configuration (for example, GLUE(Shell) or BEAN): GLUE(Shell) mode: The executor will run the Shell script filled in the task configuration. This Shell script usually takes care of calling the data synchronization script (such as a Python script or a compiled Java program).
[0075] BEAN mode: The executor will directly call the Bean method defined in the code. This usually requires encapsulating the data synchronization logic in a Spring Bean and marking it with the @XxlJob annotation.
[0076] For example, taking the order data synchronization of an e-commerce platform as an example, when an e-commerce platform needs to synchronize order data from the transaction database to the data warehouse for report analysis and BI.
[0077] If it is a traditional solution: A Python script may be written and executed using cron or other timed task tools. Each server needs to configure cron tasks, which is relatively scattered to manage. There is a lack of unified monitoring and management means.
[0078] The solution of the present invention that adopts the timed scheduling platform xxl-job will perform the following steps: 1) Define the data synchronization script: Write a Java program (or Python script) to implement the logic of reading order data from the transaction database and writing it to the data warehouse.
[0079] 2) Deploy the executor: Deploy the XXL-JOB executor on several servers.
[0080] 3) Configure the XXL-JOB task: Create a task in the XXL-JOB scheduling center and select the deployed executor. Set the Cron expression to 0 0 2 * *? (Execute at 2 am every day). Configure the task parameters, such as the connection information of the transaction database and the data warehouse. Specify the running mode as BEAN and fill in the Bean method that implements data synchronization in the written Java program.
[0081] 4) Monitoring and management: Through the Web interface of XXL-JOB, the execution status of the task can be monitored and the execution log can be viewed.
[0082] If the task execution fails, XXL-JOB will automatically retry and send an alarm email notification.
[0083] The Cron expression or parameters of the task can be modified at any time without restarting any services.
[0084] It can be seen that by adopting a timed scheduling platform such as xxl-job, the present invention can centrally manage data synchronization tasks, achieving automation, reliability, and scalability. The benefits it brings are far more than just timed execution. More importantly, it provides a powerful task management and operation and maintenance platform, allowing one to focus on the implementation of business logic without spending a lot of energy on task scheduling and monitoring.
[0085] Furthermore, a specific application embodiment is used to further describe the method embodiment of the present invention in detail. When there is an e-commerce website that needs to synchronize the order data of the previous day from the MySQL database to the Hadoop cluster every day at 2 am for data analysis, the following steps are executed by the method of the present invention: Corresponding to step S100: First, deploy the timed scheduling platform xxl-job and deploy it separately from the business server of the e-commerce website.
[0086] Corresponding to step S200: Create a task named order data synchronization on the timed scheduling platform xxl-job.
[0087] Corresponding to step S300: Write a Python script to connect to the MySQL database, read the order data of the previous day, and then write the data to the HDFS of the Hadoop cluster.
[0088] Corresponding to step S400: Configure the Python script into the order data synchronization task and set the trigger time to 2 am every day.
[0089] Corresponding to step S500: At 2 am every day, the timed scheduling platform xxl-job will automatically execute the Python script to synchronize the order data to the Hadoop cluster.
[0090] It can be seen that the embodiment of the present invention can achieve automatic timed synchronization of data without manual intervention. And the execution status and logs of the task can be viewed through the timed scheduling platform xxl-job to discover and solve problems in a timely manner.
[0091] Through the above steps, the present invention can achieve automatic timed synchronization of order data without manual intervention. At the same time, since xxl-job is deployed separately from the e-commerce website, even if there are problems with the data synchronization task, it will not affect the normal operation of the website. In addition, the progress and status of data synchronization can be understood in real time through the monitoring function of xxl-job to discover and solve problems in a timely manner.
[0092] Based on the above embodiment, further, the method of the present invention further includes the step of controlling the timed scheduling platform xxl-job to adjust the time interval of task execution according to actual needs for periodic scheduling in time units of minutes, hours, or days. That is, in the embodiment of the present invention, the time interval of task execution can be flexibly adjusted to support periodic scheduling in conventional time units such as minutes, hours, and days.
[0093] In a further embodiment, the method of the present invention further includes the step of automatically adjusting the execution parameters by the task script according to the environment variables or system status during runtime to control the timed data synchronization.
[0094] That is, in the embodiment of the present invention, for various parameters during the task execution process, such as data source address, release environment, statistical time, etc., a dynamic adaptation mechanism will be implemented inside the script. This way enables the script to automatically adjust the execution parameters according to the environment variables or system status during runtime, so as to more efficiently adapt to different business scenarios and environmental changes.
[0095] In a further embodiment, the method of the present invention further includes the steps of: when a new timing task requirement is detected, expanding on the basis of the task script, adding a new task module or modifying the scheduling configuration. That is, in the embodiment of the present invention, when there is a new timing task requirement, it is possible to easily expand on the basis of the existing script, add a new task module or modify the scheduling configuration without interfering with the existing task logic, thus having good scalability.
[0096] In addition, the present invention can also achieve free and precise setting of the synchronization time, and the execution mode combines timing and manual operation, meeting the requirements of various business scenarios, ensuring the timely and accurate execution of tasks, and supporting the efficient operation of the business.
[0097] The following further describes the present invention in detail through additional specific application embodiments, as Figure 3 shown, is the overall business timing diagram of a data timing synchronization processing method based on a timing scheduling platform according to an embodiment of the present invention. Among them, steps S11 - S15 are the steps for users to use synchronous data logging, and steps S21 - S26 are the steps for users to use data acquisition. Specifically, the implementation includes the following steps: S11. Detect that the user accesses the system page; and enter step S12; S12. When it is detected that the user accesses the system page, the server can collect user page usage information from the browser page; and enter step S13; S13. Based on the collected page usage information, save the brief page information. For example, the brief information can be saved to the registry (Redis); S14. The timing scheduling platform xxl-job parses the data types to be synchronized at regular intervals according to the set synchronization time and the newly created synchronization task; and enter step S15; Of course, the timing scheduling platform xxl-job can be automatically executed according to the set timing time, or can be manually executed by receiving user operation instructions.
[0098] S15. Control the insertion of synchronous data into the data table to control the data timing synchronization. Then end the process of the user using synchronous data logging.
[0099] When the user uses data acquisition, it includes the following steps: S21. The browser obtains the user's browsing statistics dashboard page instruction and enters S22; S22. The browser requests business statistics data from the server and enters S23; S23. The server queries the statistical result data from the (SQL) database and enters S24; S24. The database returns the query data and enters S25; S25. The server assembles and feeds back the statistical data to the browser, and proceeds to step S26; S26. The browser renders the page and displays the result; then the user data acquisition process ends.
[0100] In a further embodiment, such as Figure 4 shown, is the overall business process schematic diagram of a data timing synchronization processing method based on a timing scheduling platform according to an embodiment of the present invention, where S31 - S34 are the steps for executing buried point statistical data, and S41 - S44 are the steps for querying statistical data. As Figure 4 shown, this embodiment includes the steps: S31. When it is detected that there is an operation of accessing a page, the server buries points to collect access information and proceeds to S32; S32. The server controls to save the brief page information to the registry (Redis); and proceeds to S33; S33. The timing scheduling platform xxl - job obtains the brief information required for synchronizing data from the registry; and proceeds to S34; S34. The timing scheduling platform xxl - job controls to synchronize data to the database according to the set synchronization time and the newly created synchronization task, and then ends the buried point statistical data process.
[0101] In the embodiment of the present invention, when querying statistical data, the following steps are executed: S41. The server obtains the query request for the query statistical page and proceeds to S42; S42. The server sends a query request to the database, and then proceeds to S43; S43. The database returns the query data to the server and proceeds to S44; S44. The server returns the rendered interface data to the user terminal browser, controls to display the query result; then the process ends.
[0102] In a further embodiment, such as Figure 5 shown, is the schematic diagram of the usage process of the timing scheduling platform for a data timing synchronization processing method based on a timing scheduling platform according to an embodiment of the present invention. As Figure 5 shown, the usage process of the timing scheduling platform for the data timing synchronization processing method of this specific embodiment includes the steps: S51. Set the cron expression corn, and set the scripting language, generate / update the task script according to the task management requirements, and proceed to S52; S52. According to the set timing, execute the task script at regular intervals and proceed to S54; S53. Install the business-dependent program library in the task execution environment. The task script in the embodiment of the present invention can use lightweight scripting languages such as Python or NodeJS to implement timed tasks.
[0103] S54. Execute the task and detect whether the task execution is successful. If successful, proceed to S55; if not, proceed to S56. S55. When the task execution is successful, update the database and feedback to the task management terminal. S56. When the task execution fails, list the error information and feedback to the task management terminal until the usage process of the timed scheduling platform ends and the process stops. Save the page brief information of the usage page of the code collection. Exemplary Device As Figure 6 shown, the embodiment of the present invention provides a data timed synchronization processing device based on a timed scheduling platform. The device includes: A deployment and task separation module 310, configured to pre-deploy the timed scheduling platform xxl-job, separate the timed task management from the main business logic, and use the deployed timed scheduling platform xxl-job as a dedicated scheduling platform; and deploy the timed scheduling platform and the business service on separate servers respectively. A synchronization task creation module 320, configured to create a synchronization task in the deployed timed scheduling platform xxl-job. A task script extraction module 330, configured to obtain the written task script that needs to be synchronized regularly. A task script addition and time setting module 340, configured to add the task script to the task management process of the timed scheduling platform xxl-job and obtain the set synchronization time. A data synchronization control module 350, configured to control the execution of the corresponding task script according to the set synchronization time and the newly created synchronization task, and control the data timed synchronization, as described above.
[0104] Based on the above embodiments, the present invention also provides an intelligent terminal, and its principle block diagram can be as Figure 7As shown. The intelligent terminal includes a processor, a memory, a network interface, a display screen, and a database connected via a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with external terminals via a network connection. The computer program, when executed by the processor, implements a data timing synchronization processing method based on a timing scheduling platform. The database of the intelligent terminal is used to store a data timing synchronization processing program based on the timing scheduling platform.
[0105] Those skilled in the art can understand that Figure 7 the block diagram of the principle shown in it is only a block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the intelligent terminal to which the solution of the present invention is applied. The specific intelligent terminal may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.
[0106] In one embodiment, an intelligent terminal is provided, including a memory, and one or more programs, where one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: Pre-deploy the timing scheduling platform xxl-job, and separate the timing task management from the main business logic, and use the deployed timing scheduling platform xxl-job as a dedicated scheduling platform; and deploy the timing scheduling platform and the business service on separate servers respectively; Create a synchronization task in the deployed timing scheduling platform xxl-job; Obtain the written task script that needs to be synchronized regularly; Add the task script to the task management process of the timing scheduling platform xxl-job, and obtain the set synchronization time; According to the set synchronization time and the newly created synchronization task, the timing scheduling platform xxl-job controls the execution of the corresponding task script to control data timing synchronization, as specifically described above.
[0107] Among them, the step of the timing scheduling platform xxl-job controlling the execution of the corresponding task script according to the set synchronization time and the newly created synchronization task to control data timing synchronization further includes: Control the timing scheduling platform xxl-job to adjust the time interval of task execution to be periodically scheduled in time units of minutes, hours or days according to actual needs.
[0108] Among them, for the step of controlling data timing synchronization by the timing scheduling platform xxl-job to control the execution of the corresponding task script according to the set synchronization time and the newly created synchronization task, it further includes: The task script automatically adjusts the execution parameters according to the environment variables or system status during runtime to control data timing synchronization.
[0109] Among them, for the step of controlling data timing synchronization by the timing scheduling platform xxl-job to control the execution of the corresponding task script according to the set synchronization time and the newly created synchronization task, it further includes: When a new timing task requirement is detected, it expands on the basis of the task script to add new task modules or modify the scheduling configuration.
[0110] Among them, the task script in the step of obtaining the written task script that needs to be synchronized regularly uses the lightweight scripting languages Python or NodeJS to implement the timing task.
[0111] Among them, the steps of controlling data timing synchronization by the timing scheduling platform xxl-job to control the execution of the corresponding task script according to the set synchronization time and the newly created synchronization task include: When it detects that a user accesses the system page, it collects the user's page usage information; Based on the collected page usage information, it saves the brief page information; The timing scheduling platform xxl-job regularly parses the data types that need to be synchronized according to the set synchronization time and the newly created synchronization task, and controls the insertion of the synchronized data into the data table to control data timing synchronization.
[0112] Among them, the steps of controlling data timing synchronization by the timing scheduling platform xxl-job to control the execution of the corresponding task script according to the set synchronization time and the newly created synchronization task include: When it detects an access page operation, it performs data logging to collect access information and saves the brief page information to the registry; The timing scheduling platform xxl-job obtains the brief information required for synchronizing data from the registry; The timing scheduling platform xxl-job controls the timing synchronization of data to the database according to the set synchronization time and the newly created synchronization task, as described above.
[0113] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0114] In summary, the present invention provides a data timing synchronization processing method, device, intelligent terminal and storage medium based on a timing scheduling platform. The present invention uses the timing scheduling platform xxl-job to achieve effective decoupling of timing tasks from ordinary business logic. In this way, the management and maintenance of timing tasks are no longer closely dependent on the main business logic code, which significantly improves the clarity and maintainability of the system architecture. And the present invention also has the following advantages: 1). The time interval for task execution can be flexibly adjusted, and periodic scheduling based on regular time units such as minutes, hours, and days is supported.
[0115] 2). For various parameters during task execution, such as data source address, publishing environment, statistical time, etc., you can choose to implement a dynamic adaptation mechanism within the script. This method allows the script to automatically adjust the execution parameters according to the runtime environment variables or system status, so as to adapt to different business scenarios and environmental changes more efficiently.
[0116] 3). Good scalability. When there is a new scheduled task requirement, it can be easily expanded on the basis of the existing script, adding new task modules or modifying the scheduling configuration without interfering with the existing task logic.
[0117] 4). In the system architecture of the present invention, the timing scheduling platform and the business service are respectively placed on independent servers, fundamentally eliminating CPU resource competition and effectively ensuring the stable and efficient operation of the system. Due to physical isolation, the timing scheduling platform can be independently scheduled according to its own task plan and load, without affecting the CPU resources of the business service. When maintenance operations such as version upgrade, configuration change or restart are performed on the business service, the timing scheduling platform can operate independently, maintaining the continuity of timing tasks and the stability of the system, enhancing the anti-interference ability of the architecture and ensuring the continuous operation of the business.
Claims
1. A data timing synchronization processing method based on a timing scheduling platform, characterized in that Including: Pre-deploy the timing scheduling platform xxl-job, separate the timing task management from the main business logic, and use the deployed timing scheduling platform xxl-job as a dedicated scheduling platform; and deploy the timing scheduling platform and the business service on separate servers respectively; Create a synchronization task in the deployed timing scheduling platform xxl-job; Obtain the task script that has been written and needs to be synchronized regularly; Add the task script to the task management process of the timing scheduling platform xxl-job and obtain the set synchronization time; According to the set synchronization time and the newly created synchronization task, the timing scheduling platform xxl-job controls the execution of the corresponding task script to control the timing synchronization of data.
2. The data timing synchronization processing method based on a timing scheduling platform according to claim 1, wherein The step of the timing scheduling platform xxl-job controlling the execution of the corresponding task script to control the timing synchronization of data according to the set synchronization time and the newly created synchronization task further includes: Controlling the timing scheduling platform xxl-job to adjust the time interval of task execution to perform periodic scheduling in time units of minutes, hours, or days according to actual needs.
3. The data timing synchronization processing method based on a timing scheduling platform according to claim 1, wherein The step of the timing scheduling platform xxl-job controlling the execution of the corresponding task script to control the timing synchronization of data according to the set synchronization time and the newly created synchronization task further includes: The task script automatically adjusts the execution parameters according to the environment variables or system status during runtime to control the timing synchronization of data.
4. The data timing synchronization processing method based on a timing scheduling platform according to claim 1, wherein The step of the timing scheduling platform xxl-job controlling the execution of the corresponding task script to control the timing synchronization of data according to the set synchronization time and the newly created synchronization task further includes: When a new timing task requirement is detected, expand on the basis of the task script, add new task modules or modify the scheduling configuration.
5. The data timing synchronization processing method based on a timing scheduling platform according to claim 1, wherein The task script in the step of obtaining the task script that has been written and needs to be synchronized regularly implements the timing task using the lightweight scripting language Python or NodeJS.
6. The data timing synchronization processing method based on a timing scheduling platform according to claim 1, characterized in that The step of the timing scheduling platform xxl-job controlling the execution of the corresponding task script to control the timing synchronization of data according to the set synchronization time and the newly created synchronization task includes: When detecting that a user accesses the system page, collect the user's page usage information; Based on the collected page usage information, save the brief page information; The timing scheduling platform xxl-job regularly parses the data types that need to be synchronized according to the set synchronization time and the newly created synchronization task, and controls the insertion of the synchronized data into the data table to control the timing synchronization of data.
7. The data timing synchronization processing method based on a timing scheduling platform according to claim 1, wherein The step of the timing scheduling platform xxl-job controlling the execution of the corresponding task script to control the timing synchronization of data according to the set synchronization time and the newly created synchronization task includes: When detecting an access page operation, perform buried point collection of access information and save the brief page information to the registry; The timing scheduling platform xxl-job obtains the brief information required for synchronizing data from the registry; The timing scheduling platform xxl-job controls the timing synchronization of data to the database according to the set synchronization time and the newly created synchronization task.
8. A data timing synchronization processing device based on a timing scheduling platform, characterized in that, The device includes: A deployment and task separation module is used to pre-deploy the timed scheduling platform xxl-job, separate the timed task management from the main business logic, and use the deployed timed scheduling platform xxl-job as a dedicated scheduling platform; and deploy the timed scheduling platform and business services on separate servers respectively; A synchronous task creation module is used to create synchronous tasks in the deployed timed scheduling platform xxl-job; A task script extraction module is used to obtain the written task scripts that need to be synchronized regularly; A task script addition and time setting module is used to add the task scripts to the task management process of the timed scheduling platform xxl-job and obtain the set synchronization time; A data synchronization control module is used to control the execution of the corresponding task scripts by the timed scheduling platform xxl-job according to the set synchronization time and the newly created synchronous tasks, and control the timed synchronization of data.
9. An intelligent terminal, characterized in that, It includes a memory and one or more programs, where one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include those for executing the method described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the method described in any one of claims 1-7.