Data acquisition and warehousing method and device

Abstract

The embodiment of the present application discloses a kind of data acquisition warehousing method and device, it can be used in financial field or other technical field, the method comprises: setting N acquisition tasks to target data index carries out data acquisition, wherein, N is the integer greater than 1;According to the data time of the latest data of each acquisition task in the N acquisition tasks, determine the target acquisition task from the N acquisition tasks, and then execute data warehousing operation by the target acquisition task according to the data collected by itself.The present application helps to improve the effect of data acquisition warehousing.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and more specifically, to a data acquisition and storage method and apparatus. Background Technology

[0002] With the continuous development of information technology, the financial industry has gradually shifted towards digitalization and intelligence. Innovative financial products and services are constantly being launched, and transaction methods are constantly evolving. Therefore, real-time monitoring of business transaction status is crucial, making data collection a vital step. However, existing data collection methods carry the risk of data entry conflicts. If two or more collection tasks are started simultaneously and data is written to the database at the same time, primary key conflicts can occur due to the large amount of data being written repeatedly, leading to database security vulnerabilities. Summary of the Invention

[0003] In order to solve at least one of the technical problems in the background art, the present invention proposes a data acquisition and storage method and apparatus.

[0004] To achieve the above objectives, according to one aspect of the present invention, a data acquisition and storage method is provided, the method comprising:

[0005] Set up N data collection tasks to collect data on the target data indicators, where N is an integer greater than 1;

[0006] Based on the data time of the latest data collected by each of the N collection tasks, a target collection task is determined from the N collection tasks, and then the target collection task performs a data entry operation based on the data it has collected.

[0007] Optionally, the data collection and storage method further includes:

[0008] Read the field value of the forced data entry task corresponding to the target data indicator from the task status table;

[0009] If the field value contains the first collection task identifier information, then the collection task corresponding to the first collection task identifier information in the N collection tasks is determined, and then the corresponding collection task performs the data entry operation based on the data it collects.

[0010] Based on the data time of the latest data collected by each of the N collection tasks, a target collection task is determined from the N collection tasks. Then, the target collection task performs a data entry operation based on the data it has collected, specifically including:

[0011] If the field value does not contain the first collection task identifier information, then the target collection task is determined from the N collection tasks based on the data time of the latest data collected by each of the N collection tasks, and then the target collection task performs a data entry operation based on the data it has collected.

[0012] Optionally, based on the data time of the latest data collected by each of the N collection tasks, a target collection task is determined from the N collection tasks, and then the target collection task performs a data entry operation based on the data it has collected, specifically including:

[0013] Calculate the time difference between the latest data time collected by each collection task and the current time to obtain the time delay corresponding to each collection task.

[0014] Based on the time delay and the acceptable data delay time corresponding to the target data indicator, the target acquisition task is determined from the N acquisition tasks.

[0015] Optionally, based on the time delay and the acceptable data delay time corresponding to the target data indicator, a target data acquisition task is determined from the N data acquisition tasks, specifically including:

[0016] Read the second data collection task identifier information recorded in the default data entry task field corresponding to the target data indicator from the task status table;

[0017] If the time delay of the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is less than the acceptable data delay time, then the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is determined as the target acquisition task.

[0018] If the time delay of the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is greater than or equal to the acceptable data delay time, then any acquisition task in the N acquisition tasks whose time delay is less than the acceptable data delay time is determined as the target acquisition task.

[0019] If the time delay of each of the N acquisition tasks is greater than or equal to the acceptable data delay time, then the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is determined as the target acquisition task.

[0020] Optionally, based on the time delay and the acceptable data delay time corresponding to the target data indicator, a target data acquisition task is determined from the N data acquisition tasks, specifically including:

[0021] The acceptable data delay time is obtained by reading the field value of the acceptable data delay time field corresponding to the target data indicator from the task status table.

[0022] Optionally, the data collection and storage method further includes:

[0023] The data collected by each of the N collection tasks is stored in its corresponding message queue;

[0024] The step of determining the target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each of the N acquisition tasks specifically includes:

[0025] Retrieve the latest data and time of each data collection task from its respective message queue.

[0026] Optionally, the data collection and storage method further includes:

[0027] Store the message queue in a cache.

[0028] To achieve the above objectives, according to another aspect of the present invention, a data acquisition and storage device is provided, the device comprising:

[0029] The data acquisition unit is used to set up N acquisition tasks to collect data from the target data indicators, where N is an integer greater than 1;

[0030] The data entry unit is used to determine the target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each acquisition task, and then the target acquisition task performs the data entry operation based on the data it has collected.

[0031] To achieve the above objectives, according to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data acquisition and storage method.

[0032] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program / instructions are stored, which, when executed by a processor, implement the steps of the above-described data acquisition and storage method.

[0033] To achieve the above objectives, according to another aspect of the present invention, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data acquisition and storage method.

[0034] The beneficial effects of this invention are as follows:

[0035] This invention, in its embodiments, sets up N collection tasks to collect data on target data indicators. Then, based on the data time of the latest data collected by each of the N collection tasks, a target collection task is determined from the N collection tasks. The target collection task then performs a data entry operation based on the data it has collected. This invention simultaneously starts multiple collection tasks and selects one target collection task from among the multiple collection tasks to perform the entry operation. This avoids the problem of primary key conflicts caused by the simultaneous and repeated writing of a large amount of data when two or more collection tasks perform data entry at the same time, which leads to database security risks. This improves the efficiency of data collection and entry. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0037] Figure 1 This is a flowchart of the data acquisition and storage method according to an embodiment of the present invention;

[0038] Figure 2 This is a flowchart illustrating the data entry operation performed according to an embodiment of the present invention;

[0039] Figure 3 This is the first flowchart of the embodiment of the present invention for determining the target acquisition task;

[0040] Figure 4 This is the second flowchart of the present invention for determining the target acquisition task;

[0041] Figure 5 This is a flowchart illustrating the process of obtaining the latest data time according to an embodiment of the present invention;

[0042] Figure 6 This is a structural block diagram of the data acquisition and storage device according to an embodiment of the present invention;

[0043] Figure 7 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0044] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0045] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0046] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0047] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0049] It should be noted that the data acquisition and storage method and apparatus of the present invention can be used in the financial field, or in any field other than the financial field. The application field of the data acquisition and storage method and apparatus of the present invention is not limited.

[0050] This invention provides a method for real-time data acquisition and storage. This method can perform batch storage operations through caching, improving storage efficiency and reducing data latency; it can simultaneously start multiple identical data acquisition tasks, and ensure that only one task performs the data storage operation based on the task status, guaranteeing high availability and avoiding conflicts; it can flexibly handle and update the acquisition configuration table at any time. Compared to traditional data acquisition methods, it offers better high availability, security, scalability, and efficiency.

[0051] Figure 1 This is a flowchart of the data acquisition and storage method according to an embodiment of the present invention, such as... Figure 1 As shown, in one embodiment of the present invention, the data acquisition and storage method of the present invention includes steps S101 and S102.

[0052] Step S101: Set up N data collection tasks to collect data from the target data indicators, where N is an integer greater than 1.

[0053] In one embodiment of the present invention, the present invention first configures the data collection rules, configuring the basic information of the data source to be collected, including the collection type, involved IP, involved topic, data interface URL, data sample, data structure, data frequency, data volume, and other related information. It also configures the data collection indicators, configuring the basic information of the data indicators to be collected, including the data source type, data indicator ID, indicator filtering conditions, and indicator entry tables (including database tables, which can be multiple). Then, based on the data collection rule configuration and the data collection indicator configuration, a data collection task is established to collect and store data for a specific data indicator. The present invention deploys N copies of the task, corresponding to Task 1, Task 2, ..., Task N.

[0054] In this invention, N data acquisition tasks independently collect data on the target data indicators and store the collected data separately.

[0055] In one embodiment of the present invention, each of the N acquisition tasks filters data that meets the conditions and stores it in the cache according to acquisition rules and acquisition indicators. Multiple acquisition tasks can be stored in the same cache or different caches. At the same time as acquisition, the data is generated into a message queue in the cache according to time. The message queues of different acquisition tasks exist independently and are used for subsequent acquisition and synchronization into the database.

[0056] Step S102: Based on the data time of the latest data collected by each of the N collection tasks, a target collection task is determined from the N collection tasks, and then the target collection task performs a data entry operation based on the data it has collected.

[0057] This invention simultaneously initiates multiple data collection tasks and selects a target data collection task from among the multiple tasks to perform the data entry operation during the data entry process. This avoids the problem of database security risks caused by primary key conflicts due to the simultaneous and repeated writing of a large amount of data when two or more data collection tasks are performing data entry at the same time, thus improving the efficiency of data collection and data entry.

[0058] Figure 2 This is a flowchart of the data entry operation performed according to an embodiment of the present invention, such as... Figure 2As shown, in one embodiment of the present invention, the process of performing data entry operation includes steps S201 to S203.

[0059] Step S201: Read the field value of the forced data entry task field corresponding to the target data indicator from the task status table.

[0060] In one embodiment of the present invention, the present invention configures a task execution status table, including basic information such as task name, task ID, task exception threshold, and default task to be stored in the database. The synchronization task will update the maximum data time of the task in real time. If the data time delay of the task exceeds the threshold for the first time, the task exception is detected.

[0061] In one embodiment of the present invention, if a task abnormality is detected, data loss may occur during the intermediate switching process. This step replenishes the lost data to ensure data integrity.

[0062] In one embodiment of the present invention, a sample task status table is shown in Table 1 below:

[0063]

[0064] Table 1

[0065] As shown in Table 1 above, in the task status table, each data collection task corresponds to one data entry. Each data collection task is used to collect data for a specific data metric. Each data collection task deploys N data collection tasks, namely Task 1, Task 2, ..., Task N. The maximum data time for each task corresponds to a field. The acceptable data delay time is the delay threshold. When the difference between the maximum data time and the current time of a task does not exceed this threshold, it is considered normal; otherwise, it is considered abnormal. The default data insertion task is the task that performs the data insertion operation when all other tasks are normal. The forced data insertion task, when this field is configured, will no longer perform any checks and will directly force the data insertion task to perform the data insertion operation.

[0066] Step S202: If the field value contains the first collection task identifier information, then determine the collection task corresponding to the first collection task identifier information among the N collection tasks, and then the corresponding collection task performs the data entry operation based on the data it collects.

[0067] In this invention, if the forced data entry task field in the task status table is configured with data collection task identification information (such as data collection task ID), no further judgment will be made, and the data entry operation will be directly performed by the data collection task configured in the forced data entry task field.

[0068] Step S203: If the field value does not contain the first collection task identifier information, then the target collection task is determined from the N collection tasks based on the data time of the latest data collected by each of the N collection tasks, and then the target collection task performs a data entry operation based on the data it has collected.

[0069] In this invention, if the forced data entry task field in the task status table is not configured, the target data collection task needs to be determined first, and then the target data collection task performs the data entry operation.

[0070] In one embodiment of the present invention, step S102, which determines the target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each of the N acquisition tasks, and then the target acquisition task performs a data entry operation based on the data it has collected, specifically involves: if the field value does not contain the first acquisition task identification information, then the target acquisition task is determined from the N acquisition tasks based on the data time of the latest data collected by each of the N acquisition tasks, and then the target acquisition task performs a data entry operation based on the data it has collected.

[0071] like Figure 3 As shown, in one embodiment of the present invention, the steps S102 and S203 above determine the target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each of the N acquisition tasks, and then the target acquisition task performs a data entry operation based on the data it has collected, specifically including steps S301 and S302.

[0072] Step S301: Calculate the time difference between the latest data time collected by each acquisition task and the current time to obtain the time delay corresponding to each acquisition task.

[0073] In one embodiment of the present invention, this step specifically involves reading the field value of the acceptable data delay time field corresponding to the target data indicator from the task status table to obtain the acceptable data delay time.

[0074] Step S302: Based on the time delay and the acceptable data delay time corresponding to the target data indicator, determine the target acquisition task from the N acquisition tasks.

[0075] In optional embodiments of the present invention, this step can be implemented in various ways. For example, one can be randomly selected from all acquisition tasks with a time delay less than the acceptable data delay time as the target acquisition task, or the acquisition task with the smallest time delay can be determined as the target acquisition task.

[0076] like Figure 4 As shown, in one embodiment of the present invention, step S302, which determines the target acquisition task from the N acquisition tasks based on the time delay and the acceptable data delay time corresponding to the target data indicator, specifically includes steps S401 to S404.

[0077] Step S401: Read the second collection task identifier information recorded in the default data entry task field corresponding to the target data indicator from the task status table.

[0078] In this embodiment of the invention, the second acquisition task identification information can specifically be an acquisition task ID, such as acquisition task 1, acquisition task 2, etc.

[0079] Step S402: If the time delay of the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is less than the acceptable data delay time, then the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is determined as the target acquisition task.

[0080] Step S403: If the time delay of the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is greater than or equal to the acceptable data delay time, then any acquisition task in the N acquisition tasks whose time delay is less than the acceptable data delay time is determined as the target acquisition task.

[0081] Step S404: If the time delay of each of the N acquisition tasks is greater than or equal to the acceptable data delay time, then the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is determined as the target acquisition task.

[0082] In one embodiment of the present invention, the step S302 above, which determines the target acquisition task from the N acquisition tasks based on the time delay and the acceptable data delay time corresponding to the target data indicator, specifically includes:

[0083] The acceptable data delay time is obtained by reading the field value of the acceptable data delay time field corresponding to the target data indicator from the task status table.

[0084] Figure 5 This is a flowchart illustrating the process of obtaining the latest data time according to an embodiment of the present invention, such as... Figure 5 As shown, in one embodiment of the present invention, the process of obtaining the latest data time includes steps S501 and S502.

[0085] Step S501: Store the data collected by each of the N collection tasks into its corresponding message queue.

[0086] Step S502: Obtain the data time of the latest data collected by each collection task from the message queue corresponding to each collection task.

[0087] In one embodiment of the present invention, step S102, which determines the target acquisition task from the N acquisition tasks based on the data time of the latest data acquired by each of the N acquisition tasks, specifically includes:

[0088] Retrieve the latest data and time of each data collection task from its respective message queue.

[0089] In one embodiment of the present invention, the data acquisition and storage method of the present invention further includes:

[0090] Store the message queue in a cache.

[0091] In this invention, each acquisition task stores the acquired data in a cache. The cache forms message queues according to the data time. Each acquisition task corresponds to a set of message queues, and each data metric corresponds to a message queue. The message queues store the time series of newly acquired but unconsumed data.

[0092] In a specific embodiment of the present invention, the data acquisition and storage method of the present invention is as follows:

[0093] Step S001: Obtain the configurations to be synchronized from the data collection metric configuration, including data source type, data metrics, and data entry table. Each type corresponds to one data collection task, and multiple copies of each data collection task can be deployed.

[0094] In step S002, each data collection task retrieves the corresponding message queue from the cache according to its configuration. Based on the maximum data time recorded in the message queue, it determines whether the task needs to perform an input operation according to the method for avoiding data entry conflicts. If so, only one data collection task will perform the input operation. Each task corresponds to one set of message queues, and each metric corresponds to one message queue. Each message queue stores the time series of newly collected but unconsumed data.

[0095] Step S003: At regular intervals, compare the maximum time of the data collected by this acquisition task and update the maximum time of the data collected by this task in the task status table.

[0096] Step S004: Each data collection task queries the task status table at regular intervals to determine whether the task should perform an input operation, obtains the result of whether input is required, and decides whether the data within the current interval (i.e., all data before the next execution of the operation) should be input.

[0097] The judgment rules are as follows:

[0098] 0) If the Forced Input Task field is not empty, the input operation is performed through the collection task configured in the Forced Input Task field, and no further checks are needed. Otherwise, the following checks are performed.

[0099] 1) Calculate the difference between the maximum data time and the current time for each data acquisition task, and determine whether it exceeds the maximum acceptable delay time, i.e., the delay threshold.

[0100] 2) If all data collection tasks have not exceeded the delay threshold, determine if this data collection task is the default data insertion task. If so, execute the data insertion; otherwise, do not insert the data. If all data collection tasks have exceeded the delay threshold, determine if this data collection task is the default data insertion task. If so, execute the data insertion; otherwise, do not insert the data. If the default data insertion task has not exceeded the delay threshold, determine if this data collection task is the default data insertion task. If so, execute the data insertion; otherwise, do not insert the data. If the default data insertion task has exceeded the delay threshold, determine if this data collection task is the first task found in sequence that has not exceeded the delay threshold. If so, execute the data insertion; otherwise, do not insert the data.

[0101] Step S005: Based on the above judgment result, perform the data entry operation or consume only the data.

[0102] As can be seen from the above embodiments, the real-time data acquisition and storage method of the present invention can improve storage efficiency and reduce data latency by caching intermediate data and performing storage operations in batches; it can simultaneously start multiple identical data acquisition tasks and ensure that only one task performs data storage operation based on the status of the acquisition task; it can automatically switch to other healthy tasks to perform data storage operation when an acquisition task is abnormal or interrupted, ensuring high availability of acquisition tasks while avoiding conflicts; and it can flexibly handle and update the acquisition configuration table at any time. Compared with traditional data acquisition methods, it has better high availability, security, scalability, and efficiency. Ultimately, it ensures stable and reliable operation of data acquisition and reduces the risk of data loss.

[0103] This invention not only avoids the problems and risks of existing data acquisition methods, such as high data latency, single point of failure, and data entry conflict risks, but also improves data entry efficiency, reduces data latency, and ensures high availability of the acquisition task while avoiding conflict risks. Compared with traditional data acquisition methods, it has better high availability, security, scalability, and efficiency. Ultimately, it ensures stable and reliable data acquisition operation and reduces the risk of data loss.

[0104] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0105] Based on the same inventive concept, embodiments of the present invention also provide a data acquisition and storage device, which can be used to implement the data acquisition and storage method described in the above embodiments, as described in the following embodiments. Since the principle of the data acquisition and storage device in solving the problem is similar to that of the data acquisition and storage method, embodiments of the data acquisition and storage device can refer to embodiments of the data acquisition and storage method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0106] Figure 6 This is a structural block diagram of the data acquisition and storage device according to an embodiment of the present invention, as shown below. Figure 6 As shown, in one embodiment of the present invention, the data acquisition and storage device of the present invention includes:

[0107] Data acquisition unit 1 is used to set up N acquisition tasks to acquire data from the target data indicators, where N is an integer greater than 1;

[0108] Data entry unit 2 is used to determine the target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each acquisition task, and then the target acquisition task performs data entry operation based on the data it has collected.

[0109] In one embodiment of the present invention, the data acquisition and storage device of the present invention further includes:

[0110] The field reading unit is used to read the field value of the forced data entry task field corresponding to the target data indicator from the task status table;

[0111] The data forced entry operation unit is used to determine the collection task corresponding to the first collection task identification information among the N collection tasks if the field value contains the first collection task identification information, and then the corresponding collection task performs the data entry operation according to the data it has collected.

[0112] In one embodiment of the present invention, the data entry unit 2 is specifically used to determine the target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each of the N acquisition tasks if the field value does not contain the first acquisition task identifier information, and then the target acquisition task performs the data entry operation based on the data it has collected.

[0113] In one embodiment of the present invention, the data entry unit 2 specifically includes:

[0114] The time delay calculation module is used to calculate the time difference between the data time of the latest data collected by each acquisition task and the current time, so as to obtain the time delay corresponding to each acquisition task.

[0115] The target acquisition task determination module is used to determine the target acquisition task from the N acquisition tasks based on the time delay and the acceptable data delay time corresponding to the target data indicator.

[0116] In one embodiment of the present invention, the target acquisition task determination module specifically includes:

[0117] The default data entry task determination submodule is used to read the second data collection task identifier information recorded in the default data entry task field corresponding to the target data indicator from the task status table.

[0118] The first processing submodule is configured to determine the collection task corresponding to the second collection task identification information in the N collection tasks as the target collection task if the time delay of the collection task corresponding to the second collection task identification information in the N collection tasks is less than the acceptable data delay time.

[0119] The second processing submodule is used to determine any one of the N collection tasks whose time delay is less than the acceptable data delay time as the target collection task if the time delay of the collection task corresponding to the second collection task identification information in the N collection tasks is greater than or equal to the acceptable data delay time.

[0120] The third processing submodule is used to determine the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks as the target acquisition task if the time delay of each acquisition task in the N acquisition tasks is greater than or equal to the acceptable data delay time.

[0121] In one embodiment of the present invention, the target acquisition task determination module specifically includes:

[0122] The acceptable data delay time acquisition submodule is used to read the field value of the acceptable data delay time field corresponding to the target data indicator from the task status table to obtain the acceptable data delay time.

[0123] In one embodiment of the present invention, the data acquisition and storage device of the present invention further includes:

[0124] The message queue storage unit is used to store the data collected by each of the N collection tasks into its respective message queue;

[0125] In one embodiment of the present invention, the data acquisition unit 1 is specifically used to obtain the data time of the latest data acquired by each acquisition task from the message queue corresponding to each acquisition task.

[0126] In one embodiment of the present invention, the data acquisition and storage device of the present invention further includes:

[0127] A data caching unit is used to store the message queue in a cache.

[0128] To achieve the above objectives, according to another aspect of this application, a computer device is also provided. For example... Figure 7 As shown, the computer device includes a memory, a processor, a communication interface, and a communication bus. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps in the method of the above embodiments.

[0129] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0130] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and units, such as the program units corresponding to the above-described method embodiments of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods described in the above-described method embodiments.

[0131] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0132] The one or more units are stored in the memory and, when executed by the processor, perform the methods described in the above embodiments.

[0133] The specific details of the aforementioned computer equipment can be understood by referring to the relevant descriptions and effects in the above embodiments, and will not be repeated here.

[0134] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed in a computer processor, implements the steps in the data acquisition and storage method described above. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0135] To achieve the above objectives, according to another aspect of this application, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data acquisition and storage method.

[0136] Obviously, those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.

[0137] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for data acquisition and storage, characterized in that, include: Set up N data collection tasks to collect data on the target data indicators, where N is an integer greater than 1; The data collected by each of the N collection tasks is stored in its corresponding message queue, and the message queue is stored in a cache. The message queue stores the time series of newly collected but unconsumed data. Based on the data time of the latest data collected by each of the N collection tasks, a target collection task is determined from the N collection tasks, and then the target collection task performs a data entry operation based on the data it has collected. Read the field value of the forced data entry task corresponding to the target data indicator from the task status table; If the field value contains the first collection task identifier information, then the collection task corresponding to the first collection task identifier information in the N collection tasks is determined, and then the corresponding collection task performs the data entry operation based on the data it collects. The step involves determining a target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each of the N acquisition tasks, and then having the target acquisition task perform a data storage operation based on its own collected data. Specifically, this includes: If the field value does not contain the first collection task identifier information, then the target collection task is determined from the N collection tasks based on the data time of the latest data collected by each of the N collection tasks, and then the target collection task performs a data entry operation based on the data it has collected. The step of determining the target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each of the N acquisition tasks specifically includes: Retrieve the data time of the latest data collected by each collection task from the message queue corresponding to each collection task.

2. The data acquisition and storage method according to claim 1, characterized in that, Based on the data time of the latest data collected by each of the N collection tasks, a target collection task is determined from the N collection tasks. Then, the target collection task performs a data entry operation based on the data it has collected, specifically including: Calculate the time difference between the latest data time collected by each collection task and the current time to obtain the time delay corresponding to each collection task. Based on the time delay and the acceptable data delay time corresponding to the target data indicator, the target acquisition task is determined from the N acquisition tasks.

3. The data acquisition and storage method according to claim 2, characterized in that, Based on the time delay and the acceptable data delay time corresponding to the target data indicator, a target data acquisition task is determined from the N data acquisition tasks, specifically including: Read the second data collection task identifier information recorded in the default data entry task field corresponding to the target data indicator from the task status table; If the time delay of the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is less than the acceptable data delay time, then the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is determined as the target acquisition task. If the time delay of the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is greater than or equal to the acceptable data delay time, then any acquisition task in the N acquisition tasks whose time delay is less than the acceptable data delay time is determined as the target acquisition task. If the time delay of each of the N acquisition tasks is greater than or equal to the acceptable data delay time, then the acquisition task corresponding to the second acquisition task identification information in the N acquisition tasks is determined as the target acquisition task.

4. The data acquisition and storage method according to claim 2, characterized in that, Based on the time delay and the acceptable data delay time corresponding to the target data indicator, a target data acquisition task is determined from the N data acquisition tasks, specifically including: The acceptable data delay time is obtained by reading the field value of the acceptable data delay time field corresponding to the target data indicator from the task status table.

5. A data acquisition and storage device, characterized in that, include: The data acquisition unit is used to set up N acquisition tasks to collect data from the target data indicators, where N is an integer greater than 1; The data collected by each of the N collection tasks is stored in its corresponding message queue, and the message queue is stored in a cache. The message queue stores the time series of newly collected but unconsumed data. The data entry unit is used to determine the target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each acquisition task, and then the target acquisition task performs the data entry operation based on the data it has collected. Read the field value of the forced data entry task corresponding to the target data indicator from the task status table; If the field value contains the first collection task identifier information, then the collection task corresponding to the first collection task identifier information in the N collection tasks is determined, and then the corresponding collection task performs the data entry operation based on the data it collects. The step involves determining a target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each of the N acquisition tasks, and then having the target acquisition task perform a data storage operation based on its own collected data. Specifically, this includes: If the field value does not contain the first collection task identifier information, then the target collection task is determined from the N collection tasks based on the data time of the latest data collected by each of the N collection tasks, and then the target collection task performs a data entry operation based on the data it has collected. The step of determining the target acquisition task from the N acquisition tasks based on the data time of the latest data collected by each of the N acquisition tasks specifically includes: Retrieve the data time of the latest data collected by each collection task from the message queue corresponding to each collection task.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 4.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 4.

Images