Local data timing uploading method and device based on cloud object storage

Abstract

The invention discloses a local data timing uploading method and device based on cloud object storage, computer equipment and a storage medium, and relates to the technical field of cloud computing. Comprising the steps of (1) configuring a locally stored parameter configuration file and recording configuration parameters through the parameter configuration file, (2) creating a plurality of task instances according to the configuration parameters, (3) creating a timing strategy according to timing parameters and registering the task instances based on the timing strategy, (4) generating a cron expression based on the timing strategy and registering the task instances based on a quartz job and a quartz trigger. And 5, according to a timing strategy, when appointed time is reached, triggering task execution, starting a plurality of sub-threads to upload local data, and recording the total uploading amount, the success number, the failure number, the starting time and the ending time into a log file.

Description

Technical Field

[0001] The present invention discloses a method, device, computer equipment and storage medium for regularly uploading local data based on cloud object storage, and relates to the technical field of cloud computing. Background Art

[0002] Compared with traditional storage methods, cloud vendors' object storage has been widely used due to its advantages such as pay-as-you-go, reliability, low price, and reduced operation and maintenance costs. Currently, local data is uploaded to cloud object storage mainly through the console interface and SDK provided by cloud vendors. However, both methods require real-time upload and manual supervision, which increases operation and maintenance management costs. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a method, device, computer equipment and storage medium for uploading local data based on cloud object storage, which can set the time point and frequency of uploading object storage according to the needs, without manual intervention, and reduce the management cost. At the same time, it can utilize bandwidth during off-peak hours, avoid network congestion during peak hours, and save network resources.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a method for regularly uploading local data based on cloud object storage, comprising:

[0006] Step 1: Configure the parameter configuration file for local storage. Record the configuration parameters through the parameter configuration file. The configuration parameters include cloud object storage information, local data directory, timing parameters and environment variables. Record the endpoint, bucket name, key, bucket path, permissions and storage type through the cloud object storage information. Define the rules and frequency for executing automatic upload tasks through timing parameters.

[0007] Step 2: Create multiple task instances based on the configuration parameters. Group multiple groups of configuration parameters into list data. Each list item contains information about cloud object storage, the directory where local data is located, timing parameters and environment variables. Create a task instance based on the list item.

[0008] Step 3: Create a timing policy based on the timing parameters, and register the task instance based on the timing policy: set the execution cycle, execution date, execution time, and interval hours according to the timing policy.

[0009] Step 4: Generate a cron expression based on the timing strategy, use Quartz to create a Quartz job, Quartz trigger, Quartz jobListener, and register multiple task instances based on the Quartz job and Quartz trigger.

[0010] Step 5: According to the timing strategy, when the specified time is reached, the task execution is triggered, multiple sub-threads are started to upload local data, and the total upload amount, number of successes, number of failures, start time, and end time are recorded in the log file.

[0011] Furthermore, in step 1 of the method for scheduled uploading of local data based on cloud object storage, in response to a startup command, a configured parameter configuration file is read, wherein the startup command is sent in the client command line tool in the form of java-jar, and the parameter configuration file ending with .conf is queried according to the startup command, and if it exists, the parameter configuration file is read.

[0012] Furthermore, in step 1 of the method for timing uploading local data based on cloud object storage, the automatic upload tasks defined according to the timing parameters are divided into automatic upload tasks executed at time intervals and automatic upload tasks executed at specified time points. The automatic upload tasks executed at time intervals are set to be executed every x hours, x is a positive integer, and the minimum granularity is 1. The time points of the automatic upload tasks executed at specified time points are divided into daily, weekly, and monthly. If it is executed daily, the time point is set to be executed every day, if it is executed weekly, the time point is set to be executed every week, and if it is executed monthly, the time point is set to be executed every month.

[0013] Furthermore, in step 2 of the method for scheduled uploading of local data based on cloud object storage, each list item Item contains a unique ID, the name of the task instance is the Item ID, and the cloud object storage information, the directory where the local data is located, the timing parameters and the environment variables are detected based on the task instance to see if they are correct. If so, the task instance creation is completed; otherwise, a parameter error is prompted and the task creation fails.

[0014] Furthermore, in step 3 of the method for timing uploading local data based on cloud object storage, the execution cycle, execution date, execution time, and interval hours are set according to the timing strategy, and the execution cycle includes an execution cycle based on time intervals and an execution cycle based on time points, and the execution cycle based on time points is further divided into a daily execution cycle, a monthly execution cycle, and a weekly execution cycle; if the execution cycle is an execution cycle based on time intervals, the generated timing strategy includes the interval hours; if the execution cycle is a daily execution cycle, the generated timing strategy includes the execution time; if the execution cycle is a monthly execution cycle, the generated timing strategy includes the execution date and execution time; if the execution cycle is a weekly execution cycle, the generated timing strategy includes the execution date and execution time.

[0015] Furthermore, in step 5 of the method for timing uploading local data based on cloud object storage, according to the timing strategy, when the specified time point is reached, Quartz is responsible for triggering the task execution and reading the environment variables stored in the Item, wherein the environment variables include the number of concurrent threads and whether to upload incrementally.

[0016] The number of concurrent threads is the number of threads that upload data in parallel.

[0017] Whether to upload incrementally means that when it is true, it is an incremental upload, and the files uploaded last time will not be uploaded again this time, and only the files newly added after the last time will be uploaded; when it is false, it is a full upload, and all files will be uploaded.

[0018] Furthermore, in step 5 of the method for scheduled uploading of local data based on cloud object storage, after the task is triggered, the data entry is added to the consumption queue, the data block size is recorded, the data is accumulated until the current batch task is completed, and the total upload amount is obtained. When the consumption queue consumes data entries, the number of successes and failures is recorded, the start time of the task is recorded, and the end time of the task is recorded. The record is first added to the cache, and the cache is persisted to the log file at regular intervals.

[0019] Furthermore, in step 5 of the method for scheduled uploading of local data based on cloud object storage, a detection program is used to detect the status of the upload program. The detection program is a program that exists independently of the upload program and is used to ensure the high availability of the upload program. The upload program is a program that performs scheduled tasks to upload data. The ps-ef detection command is executed by the detection program. If the upload program process cannot be detected, the java-jar command is executed to restart the upload program.

[0020] Add the detection program to the Linux system's scheduled task cronjob and set the detection cycle.

[0021] The present invention also provides a local data timing upload device based on cloud object storage, comprising a parameter configuration unit, a task loading unit, a timing strategy unit, a data upload unit and a log reporting unit.

[0022] The parameter configuration unit configures the parameter configuration file stored locally, and records the configuration parameters through the parameter configuration file. The configuration parameters include the information of cloud object storage, the directory where the local data is located, the timing parameters and the environment variables. The endpoint, bucket name, key, bucket path, permission and storage type are recorded through the information of cloud object storage. The timing parameters define the rules and frequency for executing the automatic upload task.

[0023] The task loading unit creates multiple task instances according to the configuration parameters, and organizes multiple groups of configuration parameters into list data. Each list item Item contains information about cloud object storage, the directory where local data is located, timing parameters and environment variables. Based on the list item Item, a task instance is created.

[0024] The timing strategy unit creates a timing strategy according to the timing parameters, and based on the timing strategy, registers the task instance: according to the timing strategy, sets the execution cycle, execution date, execution time, and interval hours.

[0025] The task loading unit generates cron expressions based on the timing strategy, uses Quartz to create Quartz jobs, Quartz triggers, and Quartz jobListeners, and registers multiple task instances based on Quartz jobs and Quartz triggers.

[0026] According to the timing strategy, the data upload unit triggers the task execution when the specified time is reached, and starts multiple sub-threads to upload local data. The log reporting unit records the total upload amount, number of successes, number of failures, start time, and end time in the log file.

[0027] Furthermore, the local data scheduled upload device based on cloud object storage also includes a detection unit and an execution unit. The detection unit uses a detection program to detect the status of the upload program. The detection program is a program that exists independently of the upload program and is used to ensure the high availability of the upload program. The upload program is a program that executes scheduled tasks to upload data. The ps-ef detection command is executed by the detection program. If the upload program process cannot be detected, the java-jar command is executed to restart the upload program.

[0028] The execution unit adds the detection program to the cronjob of the Linux system and sets the detection cycle.

[0029] The benefits of the present invention are:

[0030] The time and frequency of uploading object storage can be set according to demand, without manual intervention, reducing management costs. At the same time, bandwidth can be used during off-peak hours to avoid network congestion during peak hours and save network resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic flow chart of the method of the present invention.

[0032] Figure 2 It is a schematic diagram of the interaction of the device of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0034] Example 1

[0035] The present invention provides a method for regularly uploading local data based on cloud object storage, comprising:

[0036] Step 1: Configure the parameter configuration file for local storage. Record the configuration parameters through the parameter configuration file. The configuration parameters include cloud object storage information, the directory where the local data is located, timing parameters and environment variables. Record the endpoint, bucket name, key, bucket path, permissions and storage type through the cloud object storage information. Define the rules and frequency for executing automatic upload tasks through timing parameters.

[0037] In step 1, in response to the startup command, the configured parameter configuration file is read, wherein the startup command is sent in the client command line tool in the form of java-jar, and the parameter configuration file ending with .conf is queried according to the startup command, and if it exists, the parameter configuration file is read. The client can be Windows or Linux, and the command line tool is not limited to cmd, shell, etc.

[0038] In addition, the automatic upload tasks defined according to the timing parameters are divided into automatic upload tasks executed at time intervals and automatic upload tasks executed at specified time points. The automatic upload tasks executed at time intervals are set to be executed every x hours, x is a positive integer, and the minimum granularity is 1. The time points of the automatic upload tasks executed at specified time points are divided into daily, weekly, and monthly. If it is executed daily, set the time to execute every day, such as 12:00 every day; if it is executed weekly, set the time to execute every week, such as 12:00 every Monday, Wednesday, and Friday; if it is executed monthly, set the time to execute every month, such as 12:00 on the 3rd, 14th, and 20th of each month.

[0039] Step 2: Create multiple task instances based on the configuration parameters. Group the multiple groups of configuration parameters into list data. Each list item Item contains information about cloud object storage, the directory where local data is located, timing parameters and environment variables. Create a task instance based on the list item Item.

[0040] Each list item Item contains a unique ID. The name of the task instance is the Item ID. The cloud object storage information, local data directory, timing parameters, and environment variables are detected based on the task instance to see if they are correct. If they are correct, the task instance creation is completed. Otherwise, a parameter error is prompted and the task creation fails.

[0041] Step 3: Create a timing policy according to the timing parameters, and register a task instance based on the timing policy: set the execution cycle, execution date, execution time, and interval hours according to the timing policy.

[0042] According to the timing strategy, the execution cycle, execution date, execution time, and interval hours are set. The execution cycle includes an execution cycle based on time intervals and an execution cycle based on time points. The execution cycle based on time points is further divided into a daily execution cycle, a monthly execution cycle, and a weekly execution cycle. If the execution cycle is an execution cycle based on time intervals, the generated timing strategy includes the interval hours. If the execution cycle is a daily execution cycle, the generated timing strategy includes the execution time, for example, the execution time is 12. If the execution cycle is a monthly execution cycle, the generated timing strategy includes the execution date and execution time. For example, if the execution date is 2 and the execution time is 12, it will be executed at 12 o'clock on the 2nd of each month. If the execution cycle is a weekly execution cycle, the generated timing strategy includes the execution date and execution time. For example, if the execution date is 2 and the execution time is 12, it will be executed at 12 o'clock every Tuesday.

[0043] Step 4: Generate cron expressions based on the timing strategy, use quartz to create quartz job, quartztrigger, quartz jobListener, and register multiple task instances based on quartz job and quartz trigger. The quartz job defines the Job interface, indicating what kind of tasks need to be completed, is responsible for receiving items, and passing items to quartz. The quartz trigger is responsible for passing the cron expression to quartz for task instance triggering. The quartz jobListener is responsible for monitoring the task execution time, reception time, and next execution time.

[0044] Step 5: According to the timing strategy, when the specified time is reached, the task execution is triggered, multiple sub-threads are started to upload local data, and the total upload amount, number of successes, number of failures, start time, and end time are recorded in the log file.

[0045] According to the timing strategy, when the specified time point is reached, Quartz is responsible for triggering the task execution and reading the environment variables stored in the Item. The environment variables include the number of concurrent threads and whether to upload incrementally.

[0046] The number of concurrent threads is the number of threads that upload data in parallel.

[0047] Whether to upload incrementally means that when it is true, it is an incremental upload, and the files uploaded last time will not be uploaded again this time, and only the files newly added after the last time will be uploaded; when it is false, it is a full upload, and all files will be uploaded.

[0048] After the task is triggered, the data entry is added to the consumption queue, the data block size is recorded, and the data is accumulated until the current batch task is completed to obtain the total upload amount. When the consumption queue consumes data entries, the number of successes and failures is recorded, the start time of the task is recorded, and the end time of the task is recorded. The record is first added to the cache, and the cache is persisted to the log file at regular intervals.

[0049] Example 2

[0050] Based on Example 1, in step 5, a detection program is used to detect the status of the upload program. The detection program is a program that exists independently of the upload program and is used to ensure the high availability of the upload program. The upload program is a program that performs scheduled tasks to upload data. The ps-ef detection command is executed by the detection program. If the upload program process cannot be detected, the java-jar command is executed to restart the upload program.

[0051] Add the detection program to the Linux system's scheduled task cronjob and set the detection cycle.

[0052] Example 3

[0053] The present invention also provides a local data timing upload device based on cloud object storage, comprising a parameter configuration unit, a task loading unit, a timing strategy unit, a data upload unit and a log reporting unit.

[0054] The parameter configuration unit configures the parameter configuration file stored locally, and records the configuration parameters through the parameter configuration file. The configuration parameters include the information of cloud object storage, the directory where the local data is located, the timing parameters and the environment variables. The endpoint, bucket name, key, bucket path, permission and storage type are recorded through the information of cloud object storage. The timing parameters define the rules and frequency for executing the automatic upload task.

[0055] The task loading unit creates multiple task instances according to the configuration parameters, and organizes multiple groups of configuration parameters into list data. Each list item Item contains information about cloud object storage, the directory where local data is located, timing parameters and environment variables. Based on the list item Item, a task instance is created.

[0056] The timing strategy unit creates a timing strategy according to the timing parameters, and based on the timing strategy, registers the task instance: according to the timing strategy, sets the execution cycle, execution date, execution time, and interval hours.

[0057] The task loading unit generates cron expressions based on the timing strategy, uses Quartz to create Quartz jobs, Quartz triggers, and Quartz jobListeners, and registers multiple task instances based on Quartz jobs and Quartz triggers.

[0058] According to the timing strategy, the data upload unit triggers the task execution when the specified time is reached, and starts multiple sub-threads to upload local data. The log reporting unit records the total upload amount, number of successes, number of failures, start time, and end time in the log file.

[0059] As the information interaction and execution process between the units of the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0060] Similarly, the device of the present invention sets the time point and frequency of uploading object storage according to demand, without manual intervention, reducing management costs. At the same time, it can use bandwidth during off-peak hours, avoid network congestion during peak hours, and save network resources.

[0061] It should be noted that not all steps and units in the above-mentioned processes and device structures are necessary, and some steps or units can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0062] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for regularly uploading local data based on cloud object storage, characterized in that include: Step 1: Configure the parameter configuration file for local storage. Record the configuration parameters through the parameter configuration file. The configuration parameters include cloud object storage information, local data directory, timing parameters and environment variables. Record the endpoint, bucket name, key, bucket path, permissions and storage type through the cloud object storage information. Define the rules and frequency for executing automatic upload tasks through timing parameters. Step 2: Create multiple task instances based on the configuration parameters. Group multiple groups of configuration parameters into list data. Each list item contains information about cloud object storage, the directory where local data is located, timing parameters and environment variables. Create a task instance based on the list item. Step 3: Create a timing policy based on the timing parameters, and register the task instance based on the timing policy: set the execution cycle, execution date, execution time, and interval hours according to the timing policy. Step 4: Generate a cron expression based on the timing strategy, use Quartz to create a Quartz job, Quartz trigger, and Quartz jobListener, and register multiple task instances based on the Quartz job and Quartz trigger. Step 5: According to the timing strategy, when the specified time is reached, the task execution is triggered, multiple sub-threads are started to upload local data, and the total upload amount, number of successes, number of failures, start time, and end time are recorded in the log file.

2. According to the method of claim 1, the local data scheduled upload method based on cloud object storage is characterized in that In step 1, in response to the startup command, the configured parameter configuration file is read, wherein the startup command is sent in the client command line tool in the form of java-jar, and the parameter configuration file ending with .conf is queried according to the startup command. If it exists, the parameter configuration file is read.

3. According to the method of claim 1, the local data scheduled upload method based on cloud object storage is characterized in that The automatic upload tasks defined according to the timing parameters in step 1 are divided into automatic upload tasks executed at time intervals and automatic upload tasks executed at specified time points. The automatic upload tasks executed at time intervals are set to be executed every x hours, x is a positive integer, and the minimum granularity is 1. The time points of the automatic upload tasks executed at specified time points are divided into daily, weekly, and monthly. If it is executed daily, set the time of execution every day; if it is executed weekly, set the time of execution every week; if it is executed monthly, set the date and time of execution every month.

4. According to the method for scheduled uploading of local data based on cloud object storage in claim 1, it is characterized in that each list item Item in step 2 contains a unique ID, the name of the task instance is the ID of the Item, and the cloud object storage information, the directory where the local data is located, the timing parameters and the environment variables are detected according to the task instance. If so, the task instance creation is completed, otherwise it is prompted that the parameters are wrong and the task creation fails.

5. The method for regularly uploading local data based on cloud object storage according to claim 1, characterized in that In step 3, according to the timing strategy, the execution cycle, execution date, execution time, and interval hours are set. The execution cycle includes an execution cycle based on time intervals and an execution cycle based on time points. The execution cycle based on time points is further divided into a daily execution cycle, a monthly execution cycle, and a weekly execution cycle. If the execution cycle is an execution cycle based on time intervals, the generated timing strategy includes the interval hours. If the execution cycle is a daily execution cycle, the generated timing strategy includes the execution time. If the execution cycle is a monthly execution cycle, the generated timing strategy includes the execution date and execution time. If the execution cycle is a weekly execution cycle, the generated timing strategy includes the execution date and execution time.

6. The method for regularly uploading local data based on cloud object storage according to claim 1, characterized in that In step 5, according to the timing strategy, when the specified time point is reached, Quartz is responsible for triggering the task execution and reading the environment variables stored in the Item. The environment variables include the number of concurrent threads and whether to upload incrementally. The number of concurrent threads is the number of threads that upload data in parallel. Whether to upload incrementally means that when it is true, it is an incremental upload, and the files uploaded last time will not be uploaded again this time, and only the files newly added after the last time will be uploaded; when it is false, it is a full upload, and all files will be uploaded.

7. The method for regularly uploading local data based on cloud object storage according to claim 1, characterized in that In step 5, after the task is triggered, the data entry is added to the consumption queue, the data block size is recorded, and the data is accumulated until the current batch task is completed to obtain the total upload amount. When the data entry is consumed in the consumption queue, the number of successes and failures is recorded, the start time of this task is recorded, and the end time of this task is recorded. The record is first added to the cache, and the cache is persisted to the log file at regular intervals.

8. The method for regularly uploading local data based on cloud object storage according to claim 1, characterized in that In step 5, a detection program is used to detect the status of the upload program. The detection program is a program that exists independently of the upload program and is used to ensure the high availability of the upload program. The upload program is a program that performs scheduled tasks to upload data. The ps-ef detection command is executed through the detection program. If the upload program process cannot be detected, the java-jar command is executed to restart the upload program. Add the detection program to the Linux system's scheduled task cronjob and set the detection cycle.

9. A local data scheduled upload device based on cloud object storage, characterized in that It includes parameter configuration unit, task loading unit, timing strategy unit, data upload unit and log reporting unit. The parameter configuration unit configures the parameter configuration file stored locally, and records the configuration parameters through the parameter configuration file. The configuration parameters include the information of cloud object storage, the directory where the local data is located, the timing parameters and the environment variables. The endpoint, bucket name, key, bucket path, permission and storage type are recorded through the information of cloud object storage. The timing parameters define the rules and frequency for executing the automatic upload task. The task loading unit creates multiple task instances according to the configuration parameters, and organizes multiple groups of configuration parameters into list data. Each list item Item contains information about cloud object storage, the directory where local data is located, timing parameters and environment variables. Based on the list item Item, a task instance is created. The timing strategy unit creates a timing strategy according to the timing parameters, and based on the timing strategy, registers the task instance: according to the timing strategy, sets the execution cycle, execution date, execution time, and interval hours. The task loading unit generates cron expressions based on the timing strategy, uses quartz to create quartz jobs, quartz triggers, and quartz jobListeners, and registers multiple task instances based on quartz jobs and quartz triggers. According to the timing strategy, the data upload unit triggers the task execution when the specified time is reached, and starts multiple sub-threads to upload local data. The log reporting unit records the total upload amount, number of successes, number of failures, start time, and end time in the log file.

10. The local data scheduled uploading device based on cloud object storage according to claim 9, characterized in that It also includes a detection unit and an execution unit. The detection unit uses a detection program to detect the status of the upload program. The detection program is a program that exists independently of the upload program and is used to ensure the high availability of the upload program. The upload program is a program that executes scheduled tasks to upload data. The ps-ef detection command is executed through the detection program. If the upload program process cannot be detected, the java-jar command is executed to restart the upload program. The execution unit adds the detection program to the cronjob of the Linux system and sets the detection cycle.

Images