A method of serial operation based on data acquisition

Through the combination of multi-threading technology and controller, the problem of data acquisition sequence and frequency limitation is solved, the orderly acquisition and integrity of data is achieved, resource waste is avoided, and the efficiency and reliability of data acquisition are improved.

CN115685729BActive Publication Date: 2024-11-05四川启睿克科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211372293.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-03
Publication Date
2024-11-05
Estimated Expiration
2042-11-03

AI Technical Summary

Technical Problem

During the data acquisition process, when the order of data is required and the frequency of acquisition is limited, the traditional parallel data acquisition method can easily lead to data loss or repeated acquisition, resulting in the problem of resource waste.

Method used

Multithreading technology and three controllers (STATUS_MAP, NOT_DONE_MAP, TIME_MAP) are used to combine timing tasks and log tracking to ensure the orderliness and integrity of data collection and avoid data loss and repeated acquisition.

Benefits of technology

Through multi-threaded timing task configuration and real-time management of controllers, the orderliness and integrity of data acquisition is achieved, resource waste is reduced, time and server costs are saved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115685729B_ABST
    Figure CN115685729B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for serial operation based on data acquisition. Through multi-threaded timed task configuration, three controllers are used to control the data type, unfinished data progress and current data progress time. At the same time, real-time tracking of data logs is performed, so that the situation that either data loss or repeated loading when data is re-running and causing waste of server resources can be avoided during serial acquisition, which saves time cost and server cost and achieves data integrity.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and in particular to a serial operation method based on data acquisition. Background Art

[0002] Data is of paramount importance in every industry. Completing data acquisition and data circulation is the most important component before analyzing the data. Data acquisition and data circulation are completed through data collection.

[0003] At present, data collection is completed in parallel, but this method is based on the premise that the data is not related, and the data before and after the data collection will not affect the data itself; however, when it is necessary to collect data in a certain order, if the data collection order is different, it will cause errors in the data or even completely different data. This will easily lead to errors when using parallel methods. In this case, serial data collection is usually used to ensure the order of the data and limit the frequency of calls.

[0004] In the field of data acquisition, if you want to complete the implementation of serial data acquisition technology, you usually use a direct flow to complete data acquisition, that is, complete the collection of data from the first to the last type in one process to achieve the purpose of serialization. This method is the simplest way to implement serial data acquisition. The code is simple and easy to implement, but it also has a serious defect. Once an error occurs in the collection of some data in the middle, it is easy to lose data or the system crashes, making it impossible to complete the subsequent data collection. If a log is set here and a data problem is found, you can only start the collection from the beginning when you want to re-collect data. Even if there is no previous data duplication problem, there will be a problem of wasting resources for repeated collection. Because when implementing serialization in a process, after a problem occurs, either the data is lost and the process continues to run, or the data is re-run and the previous resources are wasted. Summary of the invention

[0005] The present invention is aimed at the traditional data collection requirements. When there are requirements for the order of data collection in actual production, and there are also restrictions on the frequency of the collected data, the original commonly used parallel data collection cannot be carried out, and data collection can only be completed in a serial manner. The current single serial data collection is prone to either data loss or repeated data collection that wastes server resources. A method for serial operation based on data collection is provided. The present invention uses the multi-threading technology commonly used in parallel data collection as the underlying support, and adopts a method of controlling the data type, the unfinished data progress and the current data progress time through three controllers, and simultaneously tracking the data log in real time to complete the serial data collection.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] A method for serial operation based on data acquisition comprises the following steps:

[0008] Step 1: Initialize the controller, scheduled tasks and log table;

[0009] Step 2, determine and sort the data types;

[0010] Step 3: Call multi-threaded scheduled tasks by setting the multi-threaded method in the code;

[0011] Step 4: After the program enters all current threads through multiple threads every 60 seconds, each thread first checks the current status in STATUS_MAP;

[0012] Step 5, after entering the thread, go to NOT_DONE_MAP to find the current number of unfinished tasks of this type;

[0013] If the number is 0, it means there is no unfinished number of this type before, and the next judgment starts directly; if it is greater than 0, it means there are unfinished collection tasks of this type before, and then the difference between the task start time and the current time is obtained from TIME_MAP as the interval time. If the interval time exceeds the set maximum threshold, the log is captured to track the current position, the breakpoint position and data attributes of the collection are recorded, and the collection task is forced to be paused. The current type in STATUS_MAP is set to the waiting state, and the value of the next type is changed to the starting state, and step 4 is restarted; if the interval time does not exceed the set maximum threshold, it means that the previous collection task has not been completed, so the thread is terminated and step 4 is restarted;

[0014] Step 6, when NOT_DONE_MAP finds that the number of unfinished items of this type is 0, check the log to see if there is a collection breakpoint. If yes, get the data attribute and set it to the start parameter of the current collection. If no, check the log to see the data attribute of the last collection end and set it to the start parameter of the current collection. This ensures that the data is not lost and not repeated.

[0015] Step 7, store the number of acquisition modules of the current type in NOT_DONE_MAP, assign the current time to the value of this type in TIME_MAP, and start the acquisition function;

[0016] Step 8, when a module completes the acquisition function, the number in NOT_DONE_MAP is reduced by 1 until all modules have completed the acquisition. After the value of NOT_DONE_MAP is 0, the current type in STATUS_MAP is set to the waiting state, and the value of the next type is changed to the starting state. Start step 4 again to prepare to start the next type of data acquisition.

[0017] A further solution is that in step 1, three controllers are initialized, namely, STATUS_MAP data type sequence controller, NOT_DONE_MAP data unfinished progress controller, and TIME_MAP data progress time controller; the timer task is initialized, and it is set to perform a timer task every 60 seconds; the log tracking table and the final data acquisition receiving table are initialized.

[0018] A further solution is that in step 2, the data types are placed in STATUS_MAP in sequence, the data types are placed in NOT_DONE_MAP in sequence, and finally the data types are placed in TIME_MAP.

[0019] For example, it is necessary to collect data with the data type and order of "Orders, Finances, Reports, Sales, Service", that is, to complete multiple collections starting from Orders to the end of Service. First, place the data types in STATUS_MAP in order. The function of this controller is to control the order and opening type of the current data collection. It can control which types are opened and which types can be temporarily closed to avoid data redundancy and data order confusion. Then place the data types in NOT_DONE_MAP in order. The function of this controller is to track the number of unfinished data collection and data progress of the current data collection to ensure data integrity and data completion position record. Finally, place the data type in TIME_MAP. The function of this controller is to control the time when the data has been collected. If the collection time is too long, the data location judgment and data log tracking judgment will be performed to ensure the accuracy and integrity of the data.

[0020] In step 3, by setting the multi-threading method in the code, five different capture methods and codes are placed in the corresponding thread's scheduled tasks. The purpose of this step is to use the multi-threading non-interference effect. When collecting a certain type of data, the data collection will not be suspended or lost due to problems with other types of data. Subsequent re-runs of other types of data will not affect this type of data, reducing the re-run rate of the entire data and greatly reducing the waste of resources.

[0021] A further solution is that in step 4, if the state is a waiting state, the thread is ended and waits for the next scheduled task to start; if the state is a progress state, the thread is started; in this way, the state controller can control the order and start type of the current data collection to ensure the orderliness of data collection.

[0022] After the scheduled task is started, the program enters its own thread through multiple threads every 60 seconds, and each thread first checks the current status in STATUS_MAP. If the status is waiting, the thread ends and waits for the next scheduled task to start. If the status is in progress, the thread starts. In this way, the state controller can control the order and start type of current data collection to ensure the orderliness of data collection.

[0023] A further solution is that before starting data collection, the number of collection modules of the current type can be stored in NOT_DONE_MAP, indicating that the number of unfinished modules of this type includes these modules; at the same time, the current time is assigned to the value of this type in TIME_MAP and configured as the start time of this collection.

[0024] The beneficial effects of the present invention are:

[0025] The present invention provides a serial operation method based on data acquisition. Through multi-threaded timed task configuration, three controllers are used to control the data type, unfinished data progress and current data progress time. At the same time, real-time tracking of data logs is performed. This can avoid the situation that either data loss or repeated loading of data during re-running leads to waste of server resources, which is easy to occur during serial acquisition. This saves time and server costs and achieves data integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0028] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0029] In any embodiment, if Figure 1 As shown, a serial operation method based on data acquisition of the present invention comprises the following steps:

[0030] Step 1, initialize the controller, scheduled task and log table; initialize three controllers: STATUS_MAP data type sequence controller, NOT_DONE_MAP data unfinished progress controller, TIME_MAP data progress time controller; initialize the scheduled tasker, set it to perform a scheduled task every 60 seconds; initialize the log tracking table and the final data collection receiving table;

[0031] Step 2, determine and sort the data types;

[0032] For example, if you need to collect data of the data type and order "Orders, Finances, Reports, Sales, Service", that is, you need to complete multiple collections starting from Orders to Service. First, place the data types in STATUS_MAP in order, set the status value of Orders to ready to start, and the others to waiting to start; then place the data types in NOT_DONE_MAP in order, initialize the unfinished progress of each to 0, and finally place the five data types in TIME_MAP, and set the initial data progress time to the current time LocalDateTime.now();

[0033] Step 3, call multi-threaded scheduled tasks by setting the built-in multi-threaded method in the code; call multi-threaded scheduled tasks of 5 types respectively through the GlobalThreadPool.SCHEDULED_POOL.scheduleAtFixedRate method that comes with JAVA;

[0034] Step 4: After the program enters all current threads through multiple threads every 60 seconds, each thread first checks the current status in STATUS_MAP;

[0035] After the program enters all current threads through multiple threads every 60 seconds, each thread first checks the current status in STATUS_MAP. If the status is waiting, the thread is terminated, waiting for the next scheduled task to start, and restarting step 4. If the status is in progress, start the next step;

[0036] Step 5, after entering the thread, go to NOT_DONE_MAP to find the current number of unfinished tasks of this type;

[0037] If the number is 0, it means there is no unfinished number of this type before, and the next judgment starts directly; if it is greater than 0, it means there are unfinished collection tasks of this type before, and then the difference between the task start time and the current time is obtained from TIME_MAP as the interval time. If the interval time exceeds the set maximum threshold, the log is captured to track the current position, the breakpoint position and data attributes of the collection are recorded, and the collection task is forced to be paused. The current type in STATUS_MAP is set to the waiting state, and the value of the next type is changed to the starting state, and step 4 is restarted; if the interval time does not exceed the set maximum threshold, it means that the previous collection task has not been completed, so the thread is terminated and step 4 is restarted;

[0038] Step 6: When NOT_DONE_MAP finds that the number of unfinished items of this type is 0, check the log to see if there is a collection breakpoint. If so, get the data attribute and set it to the start parameter of the current collection. If not, check the log for the data attribute of the last collection end and set it to the start parameter of the current collection.

[0039] Step 7, store the number of acquisition modules of the current type in NOT_DONE_MAP, assign the current time to the value of this type in TIME_MAP, and start the acquisition function;

[0040] Step 8, when a module completes the acquisition function, the number in NOT_DONE_MAP is reduced by 1 until all modules have completed the acquisition. After the value of NOT_DONE_MAP is 0, the current type in STATUS_MAP is set to the waiting state, and the value of the next type is changed to the starting state. Start step 4 again to prepare to start the next type of data acquisition.

[0041] Before starting data collection, the number of collection modules of the current type is stored in NOT_DONE_MAP, indicating that the number of unfinished modules of this type is these numbers; at the same time, the current time is assigned to the value of this type in TIME_MAP and configured as the start time of this collection.

[0042] The present application sets up three controllers (data type sequence controller, data unfinished progress controller, data progress time controller). The system enters the parallel system through the timing task (such as starting once every 60 seconds) at the beginning, and generates multiple sub-threads at the same time. Each sub-thread is only allowed to collect one type of data, and then controls the parallel system through the data type sequence controller and determines the data type that needs to be collected at present (for example, only order data collection is currently allowed). Only the data of the allowed data type in the controller is collected, and multiple sub-threads of other types are directly terminated, waiting for the timing task to wake up the multiple sub-threads again. The technical support principle of this distinguishing feature is to achieve the result by using multiple threads and timing tasks at the same time, plus the continuous configuration of the states of the three controller sets, which can achieve the unique technical effect of using parallel technology to meet the requirements of a system that can only perform serial data collection and will not cause the entire data collection to be interrupted due to errors in a certain type of data.

[0043] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further explain various possible combinations. In addition, the various different embodiments of the present invention can also be combined arbitrarily, as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A method for serial operation based on data acquisition, characterized in that: The following steps are involved: Step 1: Initialize the controller, scheduled tasks and log table; Step 2, determine and sort the data types; Step 3: Call multi-threaded scheduled tasks by setting the multi-threaded method in the code; Step 4: After the program enters all current threads through multiple threads every 60 seconds, each thread first checks the current status in STATUS_MAP; Step 5, after entering the thread, go to NOT_DONE_MAP to find the current number of unfinished tasks of this type; If the number is 0, it means there is no unfinished number of this type before, and the next judgment starts directly; if it is greater than 0, it means there are unfinished collection tasks of this type before, and then the difference between the task start time and the current time is obtained from TIME_MAP as the interval time. If the interval time exceeds the set maximum threshold, the log is captured to track the current position, the breakpoint position and data attributes of the collection are recorded, and the collection task is forced to be paused. The current type in STATUS_MAP is set to the waiting state, and the value of the next type is changed to the starting state, and step 4 is restarted; if the interval time does not exceed the set maximum threshold, it means that the previous collection task has not been completed, so the thread is terminated and step 4 is restarted; Step 6: When NOT_DONE_MAP finds that the number of unfinished items of this type is 0, check the log to see if there is a collection breakpoint. If so, get the data attribute and set it to the start parameter of the current collection. If not, check the log for the data attribute of the last collection end and set it to the start parameter of the current collection. Step 7, store the number of acquisition modules of the current type in NOT_DONE_MAP, assign the current time to the value of this type in TIME_MAP, and start the acquisition function; Step 8, when a module completes the acquisition function, the number in NOT_DONE_MAP is reduced by 1 until all modules have completed the acquisition. After the value of NOT_DONE_MAP is 0, the current type in STATUS_MAP is set to the waiting state, and the value of the next type is changed to the starting state. Start step 4 again to prepare to start the next type of data acquisition.

2. A method for serial operation based on data acquisition as claimed in claim 1, characterized in that: In the step 1, three controllers are initialized, namely, the STATUS_MAP data type sequence controller, the NOT_DONE_MAP data unfinished progress controller, and the TIME_MAP data progress time controller; the timer task is initialized, and it is set to perform a timer task every 60 seconds; the log tracking table and the final data collection receiving table are initialized.

3. A method for serial operation based on data acquisition as claimed in claim 1, characterized in that: In the step 2, the data types are placed in STATUS_MAP in sequence, the data types are placed in NOT_DONE_MAP in sequence, and finally the data types are placed in TIME_MAP.

4. A method for serial operation based on data acquisition as claimed in claim 1, characterized in that: In step 4, if the state is in the waiting state, the thread is ended and the next scheduled task is waited for to be started; if the state is in the progress state, the thread is started; in this way, the state controller can control the order and start type of the current data collection to ensure the orderliness of data collection.

5. A method for serial operation based on data acquisition as claimed in claim 1, characterized in that: Before starting data collection, the number of collection modules of the current type is stored in NOT_DONE_MAP, indicating that the number of unfinished modules of this type is these numbers; at the same time, the current time is assigned to the value of this type in TIME_MAP and configured as the start time of this collection.