A method and device for regularly uploading local data based on cloud object storage
By configuring parameter files and timing strategies, and using the Quartz scheduler to achieve scheduled upload of local data, the problems of manual oversight and network congestion in existing technologies are solved, and automated, low-cost cloud object storage upload is achieved.
Patent Information
- Application Number
- CN202411490037.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2044-10-24
AI Technical Summary
In existing technologies, uploading local data to cloud object storage requires manual supervision, which increases operation and maintenance management costs and easily causes network congestion during peak hours.
By configuring parameter files and timing strategies, the upload time and frequency are automatically set, and the Quartz scheduler is used to trigger tasks to achieve scheduled upload of local data, reduce manual intervention, and utilize bandwidth during off-peak hours to avoid network congestion during peak hours.
It reduces management costs, saves network resources, realizes scheduled uploading without manual intervention, and avoids network congestion during peak hours.
Smart Images
Figure CN119449873B_ABST
Abstract
Description
Technical Field
[0001] The present invention discloses a method, device, computer equipment and storage medium for regularly uploading local data based on cloud object storage, and relates to the technical field of cloud computing. Background Art
[0002] Compared to traditional storage methods, cloud vendors' object storage has gained widespread adoption due to its pay-as-you-go, reliable, affordable, and reduced operational costs. Currently, uploading local data to cloud object storage primarily involves using the cloud vendor's console interface or SDK. However, both methods require real-time uploads and manual oversight, increasing operational and management costs. Summary of the Invention
[0003] The present invention addresses the problems of the prior art by providing a method, device, computer equipment, and storage medium for scheduled local data upload based on cloud object storage. The method allows for setting the time and frequency of object storage uploads based on demand, eliminating the need for manual intervention and reducing management costs. Furthermore, the method utilizes bandwidth during off-peak hours, avoiding network congestion during peak hours and conserving network resources.
[0004] The specific scheme proposed by the present invention is:
[0005] The present invention provides a method for regularly uploading local data based on cloud object storage, comprising:
[0006] Step 1: Configure the parameter configuration file for local storage. Record the configuration parameters in the parameter configuration file. The configuration parameters include cloud object storage information, local data directory, timing parameters, and environment variables. Record the endpoint, bucket name, key, bucket path, permissions, and storage type in the cloud object storage information. Define the rules and frequency for executing automatic upload tasks through timing parameters.
[0007] Step 2: Create multiple task instances based on the configuration parameters. Group the multiple configuration parameters into list data. Each list item contains the cloud object storage information, the local data directory, timing parameters and environment variables. Create a task instance based on the list item.
[0008] Step 3: Create a timing policy based on the timing parameters. Based on the timing policy, register the task instance: according to the timing policy, set the execution cycle, execution date, execution time, and interval hours.
[0009] Step 4: Generate a cron expression based on the timing strategy, use Quartz to create a Quartz job, Quartz trigger, and Quartz jobListener, and register multiple task instances based on the Quartz job and Quartz trigger.
[0010] Step 5: According to the timing strategy, when the specified time is reached, the task execution is triggered, multiple sub-threads are started to upload local data, and the total upload amount, number of successes, number of failures, start time, and end time are recorded in the log file.
[0011] Furthermore, in step 1 of the method for scheduled uploading of local data based on cloud object storage, in response to a startup command, a configured parameter configuration file is read, wherein the startup command is sent in the client command line tool in the form of java-jar, and the startup command is used to query whether the parameter configuration file ending with .conf exists. If it exists, the parameter configuration file is read.
[0012] Furthermore, in step 1 of the method for scheduled local data upload based on cloud object storage, the automatic upload tasks defined according to the timing parameters are divided into automatic upload tasks executed at time intervals and automatic upload tasks executed at specified time points. The automatic upload tasks executed at time intervals are set to be executed every x hours, where x is a positive integer with a minimum granularity of 1. The time points of the automatic upload tasks executed at specified time points are divided into daily, weekly, and monthly. If the task is executed daily, the time point of execution is set. If the task is executed weekly, the time point of execution is set. If the task is executed monthly, the time point of execution is set.
[0013] Furthermore, in step 2 of the method for scheduled uploading of local data based on cloud object storage, each list item Item contains a unique ID, the name of the task instance is the Item ID, and the cloud object storage information, the directory where the local data is located, the timing parameters and the environment variables are detected based on the task instance to see whether they are correct. If so, the task instance creation is completed; otherwise, a parameter error is prompted and the task creation fails.
[0014] Furthermore, in step 3 of the method for timing uploading local data based on cloud object storage, the execution cycle, execution date, execution time, and interval hours are set according to the timing strategy. The execution cycle includes an execution cycle based on time intervals and an execution cycle based on time points. The execution cycle based on time points is further divided into a daily execution cycle, a monthly execution cycle, and a weekly execution cycle. If the execution cycle is an execution cycle based on time intervals, the generated timing strategy includes the interval hours; if the execution cycle is a daily execution cycle, the generated timing strategy includes the execution time; if the execution cycle is a monthly execution cycle, the generated timing strategy includes the execution date and execution time; if the execution cycle is a weekly execution cycle, the generated timing strategy includes the execution date and execution time.
[0015] Furthermore, in step 5 of the method for timing uploading local data based on cloud object storage, according to the timing strategy, when the specified time point is reached, Quartz is responsible for triggering the task execution and reading the environment variables stored in the Item. The environment variables include the number of concurrent threads and whether to upload incrementally.
[0016] The number of concurrent threads is the number of threads that upload data in parallel.
[0017] The incremental upload refers to whether to upload incrementally when it is true, and the files uploaded last time will not be uploaded again this time, and only the files added after the last time will be uploaded; when it is false, it is a full upload, and all files will be uploaded.
[0018] Furthermore, in step 5 of the method for scheduled uploading of local data based on cloud object storage, after the task is triggered, the data entry is added to the consumption queue, the data block size is recorded, and the data is accumulated until the current batch task is completed to obtain the total upload amount. When the consumption queue consumes data entries, the number of successes and failures is recorded, the start time of the task is recorded, and the end time of the task is recorded. The record is first added to the cache, and the cache is persisted to the log file at regular intervals.
[0019] Furthermore, in step 5 of the method for scheduled uploading of local data based on cloud object storage, a detection program is used to detect the status of the upload program. The detection program is a program that exists independently of the upload program and is used to ensure the high availability of the upload program. The upload program is a program that performs scheduled tasks to upload data. The ps-ef detection command is executed by the detection program. If the upload program process is not detected, the java-jar command is executed to restart the upload program.
[0020] Add the detection program to the Linux system's scheduled task cronjob and set the detection cycle.
[0021] The present invention also provides a local data timing upload device based on cloud object storage, including a parameter configuration unit, a task loading unit, a timing strategy unit, a data upload unit and a log reporting unit.
[0022] The parameter configuration unit configures the parameter configuration file for local storage and records the configuration parameters through the parameter configuration file. The configuration parameters include cloud object storage information, local data directory, timing parameters and environment variables. The endpoint, bucket name, key, bucket path, permissions and storage type are recorded through the cloud object storage information. The timing parameters define the rules and frequency for executing automatic upload tasks.
[0023] The task loading unit creates multiple task instances based on the configuration parameters, and organizes multiple sets of configuration parameters into list data. Each list item contains information about cloud object storage, the directory where local data is located, timing parameters, and environment variables. Based on the list item, a task instance is created.
[0024] The timing strategy unit creates a timing strategy according to the timing parameters, and registers the task instance based on the timing strategy: according to the timing strategy, sets the execution cycle, execution date, execution time, and interval hours.
[0025] The task loading unit generates cron expressions based on the timing strategy, uses Quartz to create Quartz jobs, Quartz triggers, and Quartz jobListeners, and registers multiple task instances based on Quartz jobs and Quartz triggers.
[0026] The data upload unit triggers the task execution according to the timing strategy when the specified time is reached, and starts multiple sub-threads to upload local data. The log reporting unit records the total upload amount, number of successes, number of failures, start time, and end time in the log file.
[0027] Furthermore, the local data scheduled upload device based on cloud object storage further includes a detection unit and an execution unit. The detection unit uses a detection program to detect the status of the upload program. The detection program is a program that exists independently of the upload program and is used to ensure the high availability of the upload program. The upload program is a program that executes scheduled tasks to upload data. The ps-ef detection command is executed by the detection program. If the upload program process cannot be detected, the java-jar command is executed to restart the upload program.
[0028] The execution unit adds the detection program to the cronjob of the Linux system and sets the detection cycle.
[0029] The benefits of the present invention are:
[0030] The time and frequency of object storage uploads can be set based on demand, eliminating the need for manual intervention and reducing management costs. Bandwidth can also be utilized during off-peak hours, avoiding network congestion during peak hours and conserving network resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a schematic flow chart of the method of the present invention.
[0032] Figure 2 It is a schematic diagram of the interaction of the device of the present invention. DETAILED DESCRIPTION
[0033] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.
[0034] Example 1
[0035] The present invention provides a method for regularly uploading local data based on cloud object storage, comprising:
[0036] Step 1: Configure a parameter configuration file for local storage. Record the configuration parameters in the parameter configuration file. The configuration parameters include cloud object storage information, the directory where local data is located, timing parameters, and environment variables. Record the endpoint, bucket name, key, bucket path, permissions, and storage type in the cloud object storage information. Define the rules and frequency for executing automatic upload tasks through timing parameters.
[0037] In step 1, in response to a startup command, a configured parameter configuration file is read. The startup command is sent in a client command line tool using a java-jar format. A query is made based on the startup command to determine whether a parameter configuration file ending with .conf exists. If so, the parameter configuration file is read. The client can be Windows or Linux, and the command line tool is not limited to cmd, shell, etc.
[0038] In addition, automatic upload tasks defined by timing parameters are divided into time-based automatic upload tasks and time-based automatic upload tasks. Time-based automatic upload tasks are set to execute every x hours, where x is a positive integer with a minimum granularity of 1. Time-based automatic upload tasks are set to execute daily, weekly, or monthly. For daily execution, set the time of execution, such as 12:00 PM every day; for weekly execution, set the time of execution, such as 12:00 PM every Monday, Wednesday, and Friday; and for monthly execution, set the time of execution, such as 12:00 PM on the 3rd, 14th, and 20th of each month.
[0039] Step 2: Create multiple task instances based on the configuration parameters. Group the multiple sets of configuration parameters into list data. Each list item contains information about cloud object storage, the directory where local data is located, timing parameters, and environment variables. Create a task instance based on the list item.
[0040] Each list item contains a unique ID. The name of the task instance is the ID of the item. The system checks whether the cloud object storage information, local data directory, timing parameters, and environment variables are correct based on the task instance. If they are correct, the task instance creation is completed. Otherwise, a parameter error is prompted and the task creation fails.
[0041] Step 3: Create a timing policy according to the timing parameters, and register a task instance based on the timing policy: set the execution cycle, execution date, execution time, and interval hours according to the timing policy.
[0042] According to the timing strategy, the execution cycle, execution date, execution time, and interval hours are set. The execution cycle includes an execution cycle based on time intervals and an execution cycle based on time points. The execution cycle based on time points is further divided into a daily execution cycle, a monthly execution cycle, and a weekly execution cycle. If the execution cycle is an execution cycle based on time intervals, the generated timing strategy includes the interval hours. If the execution cycle is a daily execution cycle, the generated timing strategy includes the execution time, for example, the execution time is 12. If the execution cycle is a monthly execution cycle, the generated timing strategy includes the execution date and execution time. For example, if the execution date is 2 and the execution time is 12, it will be executed at 12 o'clock on the 2nd of each month. If the execution cycle is a weekly execution cycle, the generated timing strategy includes the execution date and execution time. For example, if the execution date is 2 and the execution time is 12, it will be executed at 12 o'clock every Tuesday.
[0043] Step 4: Generate a cron expression based on the timing strategy. Using Quartz, create a Quartz job, a Quartz trigger, and a Quartz jobListener. Register multiple task instances based on the Quartz job and the Quartz trigger. The Quartz job defines a Job interface, indicating the task to be completed, and is responsible for receiving and passing items to Quartz. The Quartz trigger is responsible for passing the cron expression to Quartz for use in triggering task instances. The Quartz jobListener is responsible for monitoring the task execution time, reception time, and next execution time.
[0044] Step 5: According to the timing strategy, when the specified time is reached, the task execution is triggered, multiple sub-threads are started to upload local data, and the total upload amount, number of successes, number of failures, start time, and end time are recorded in the log file.
[0045] According to the timing strategy, when the specified time point is reached, Quartz is responsible for triggering the task execution and reading the environment variables stored in the Item. The environment variables include the number of concurrent threads and whether to upload incrementally.
[0046] The number of concurrent threads is the number of threads that upload data in parallel.
[0047] The incremental upload refers to whether to upload incrementally when it is true, and the files uploaded last time will not be uploaded again this time, and only the files added after the last time will be uploaded; when it is false, it is a full upload, and all files will be uploaded.
[0048] After the task is triggered, the data entry is added to the consumption queue, the data block size is recorded, and the data is accumulated until the current batch task is completed to obtain the total upload amount. When the consumption queue consumes data entries, the number of successes and failures is recorded, the start time of the task is recorded, and the end time of the task is recorded. The record is first added to the cache, and the cache is persisted to the log file at regular intervals.
[0049] Example 2
[0050] Based on Example 1, in step 5, a detection program is used to detect the status of the upload program. The detection program is a program that exists independently of the upload program and is used to ensure the high availability of the upload program. The upload program is a program that performs scheduled tasks to upload data. The ps-ef detection command is executed by the detection program. If the upload program process is not detected, the java-jar command is executed to restart the upload program.
[0051] Add the detection program to the Linux system's scheduled task cronjob and set the detection cycle.
[0052] Example 3
[0053] The present invention also provides a local data timing upload device based on cloud object storage, including a parameter configuration unit, a task loading unit, a timing strategy unit, a data upload unit and a log reporting unit.
[0054] The parameter configuration unit configures the parameter configuration file for local storage and records the configuration parameters through the parameter configuration file. The configuration parameters include cloud object storage information, local data directory, timing parameters and environment variables. The endpoint, bucket name, key, bucket path, permissions and storage type are recorded through the cloud object storage information. The timing parameters define the rules and frequency for executing automatic upload tasks.
[0055] The task loading unit creates multiple task instances based on the configuration parameters, and organizes multiple sets of configuration parameters into list data. Each list item contains information about cloud object storage, the directory where local data is located, timing parameters, and environment variables. Based on the list item, a task instance is created.
[0056] The timing strategy unit creates a timing strategy according to the timing parameters, and registers the task instance based on the timing strategy: according to the timing strategy, sets the execution cycle, execution date, execution time, and interval hours.
[0057] The task loading unit generates cron expressions based on the timing strategy, uses Quartz to create Quartz jobs, Quartz triggers, and Quartz jobListeners, and registers multiple task instances based on Quartz jobs and Quartz triggers.
[0058] The data upload unit triggers the task execution according to the timing strategy when the specified time is reached, and starts multiple sub-threads to upload local data. The log reporting unit records the total upload amount, number of successes, number of failures, start time, and end time in the log file.
[0059] The information interaction, execution process, etc. between the units of the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention and will not be repeated here.
[0060] Similarly, the device of the present invention sets the time and frequency of uploading object storage according to demand, without manual intervention, reducing management costs. At the same time, it can utilize bandwidth during off-peak hours, avoiding network congestion during peak hours and saving network resources.
[0061] It should be noted that not all steps and units in the above-mentioned processes and device structures are required, and certain steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or may be implemented by certain components in multiple independent devices.
[0062] The above embodiments are merely preferred embodiments for the purpose of fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the claims.
Claims
1. A method for regularly uploading local data based on cloud object storage, characterized by: include: Step 1: Configure the parameter configuration file for local storage. Record the configuration parameters in the parameter configuration file. The configuration parameters include cloud object storage information, local data directory, timing parameters, and environment variables. Record the endpoint, bucket name, key, bucket path, permissions, and storage type in the cloud object storage information. Define the rules and frequency for executing automatic upload tasks through timing parameters. Step 2: Create multiple task instances based on the configuration parameters. Group the multiple configuration parameters into list data. Each list item contains the cloud object storage information, the local data directory, timing parameters and environment variables. Create a task instance based on the list item. Step 3: Create a timing policy based on the timing parameters. Based on the timing policy, register the task instance: according to the timing policy, set the execution cycle, execution date, execution time, and interval hours. Step 4: Generate a cron expression based on the timing strategy, use Quartz to create a Quartz job, Quartz trigger, and Quartz jobListener, and register multiple task instances based on the Quartz job and Quartz trigger. Step 5: According to the timing strategy, when the specified time is reached, the task execution is triggered, multiple sub-threads are started to upload local data, and the total upload amount, number of successes, number of failures, start time, and end time are recorded in the log file.
2. A method for regularly uploading local data based on cloud object storage according to claim 1, characterized in that In step 1, in response to the startup command, the configured parameter configuration file is read, wherein the startup command is sent in the java-jar mode in the client command line tool, and the parameter configuration file ending with .conf is checked according to the startup command. If it exists, the parameter configuration file is read.
3. The method for regularly uploading local data based on cloud object storage according to claim 1, characterized in that The automatic upload tasks defined according to the timing parameters in step 1 are divided into automatic upload tasks executed at time intervals and automatic upload tasks executed at specified time points. The automatic upload tasks executed at time intervals are set to be executed every x hours, where x is a positive integer and the minimum granularity is 1. The time points of the automatic upload tasks executed at specified time points are divided into daily, weekly, and monthly. If it is executed daily, set the time of execution every day; if it is executed weekly, set the time of execution every week; if it is executed monthly, set the date and time of execution every month.
4. A method for scheduled uploading of local data based on cloud object storage according to claim 1, characterized in that each list item Item in step 2 contains a unique ID, the name of the task instance is the ID of the Item, and the cloud object storage information, the directory where the local data is located, the timing parameters and the environment variables are detected according to the task instance. If so, the task instance creation is completed; otherwise, a parameter error is prompted and the task creation fails.
5. The method for regularly uploading local data based on cloud object storage according to claim 1 is characterized in that In step 3, according to the timing strategy, the execution cycle, execution date, execution time, and interval hours are set. The execution cycle includes an execution cycle based on time intervals and an execution cycle based on time points. The execution cycle based on time points is further divided into a daily execution cycle, a monthly execution cycle, and a weekly execution cycle. If the execution cycle is an execution cycle based on time intervals, the generated timing strategy includes the interval hours; if the execution cycle is a daily execution cycle, the generated timing strategy includes the execution time; if the execution cycle is a monthly execution cycle, the generated timing strategy includes the execution date and execution time; if the execution cycle is a weekly execution cycle, the generated timing strategy includes the execution date and execution time.
6. The method for regularly uploading local data based on cloud object storage according to claim 1, characterized in that In step 5, according to the timing strategy, when the specified time point is reached, Quartz is responsible for triggering the task execution and reading the environment variables stored in the Item. The environment variables include the number of concurrent threads and whether to upload incrementally. The number of concurrent threads is the number of threads that upload data in parallel. The incremental upload refers to whether to upload incrementally when it is true, and the files uploaded last time will not be uploaded again this time, and only the files added after the last time will be uploaded; when it is false, it is a full upload, and all files will be uploaded.
7. The method for regularly uploading local data based on cloud object storage according to claim 1, characterized in that In step 5, after the task is triggered, the data entry is added to the consumption queue, the data block size is recorded, and the data is accumulated until the current batch task is completed to obtain the total upload amount. When the consumption queue consumes data entries, the number of successes and failures is recorded, the start time of this task is recorded, and the end time of this task is recorded. The record is first added to the cache, and the cache is persisted to the log file at regular intervals.
8. The method for regularly uploading local data based on cloud object storage according to claim 1, characterized in that In step 5, a detection program is used to detect the status of the upload program. The detection program is a separate program that exists independently of the upload program and is used to ensure the high availability of the upload program. The upload program is a program that performs scheduled tasks to upload data. The ps-ef detection command is executed through the detection program. If the upload program process is not detected, the java-jar command is executed to restart the upload program. Add the detection program to the Linux system's scheduled task cronjob and set the detection cycle.
9. A local data scheduled upload device based on cloud object storage, characterized by It includes parameter configuration unit, task loading unit, timing strategy unit, data upload unit and log reporting unit. The parameter configuration unit configures the parameter configuration file for local storage and records the configuration parameters through the parameter configuration file. The configuration parameters include cloud object storage information, local data directory, timing parameters and environment variables. The endpoint, bucket name, key, bucket path, permissions and storage type are recorded through the cloud object storage information. The timing parameters define the rules and frequency for executing automatic upload tasks. The task loading unit creates multiple task instances based on the configuration parameters, and organizes multiple sets of configuration parameters into list data. Each list item contains information about cloud object storage, the directory where local data is located, timing parameters, and environment variables. Based on the list item, a task instance is created. The timing strategy unit creates a timing strategy according to the timing parameters, and registers the task instance based on the timing strategy: according to the timing strategy, sets the execution cycle, execution date, execution time, and interval hours. The task loading unit generates cron expressions based on the timing strategy, uses Quartz to create Quartz jobs, Quartz triggers, and Quartz job Listeners, and registers multiple task instances based on Quartz jobs and Quartz triggers. The data upload unit triggers the task execution according to the timing strategy when the specified time is reached, and starts multiple sub-threads to upload local data. The log reporting unit records the total upload amount, number of successes, number of failures, start time, and end time in the log file.
10. The local data scheduled uploading device based on cloud object storage according to claim 9 is characterized in that The system also includes a detection unit and an execution unit. The detection unit uses a detection program to detect the status of the upload program. The detection program is a program that exists independently of the upload program and is used to ensure the high availability of the upload program. The upload program is a program that performs scheduled tasks to upload data. The ps-ef detection command is executed by the detection program. If the upload program process is not detected, the java-jar command is executed to restart the upload program. The execution unit adds the detection program to the cronjob of the Linux system and sets the detection cycle.