A data acquisition method, device and storage medium

Abstract

The embodiment of the present invention provides a data acquisition method, comprising: acquiring a node identifier corresponding to data in a form; determining target data in the form according to the node identifier; and acquiring the target data. The data acquisition method provided by the embodiment of the present invention can realize that a form can be loaded in parallel by multiple nodes.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a data acquisition method, device and storage medium. Background Art

[0002] When a traditional relational database management system (RDBMS) loads data into a distributed cache, it can only load different forms on each node for parallel processing, but cannot load a form in parallel on multiple nodes. Summary of the invention

[0003] The embodiment of the present invention provides a data acquisition method, device and storage medium, which can realize parallel loading of a form by multiple nodes.

[0004] An embodiment of the present invention provides a data acquisition method, comprising: acquiring a node identifier corresponding to data in a form; determining target data in the form according to the node identifier; and acquiring the target data.

[0005] Optionally, before obtaining the node identifier corresponding to the data in the form, the method further includes: configuring the node identifier corresponding to the data in the form.

[0006] Optionally, configuring the node identifier corresponding to the data in the form includes: configuring the node identifier corresponding to each data in the form according to a primary key or a shard key in the form.

[0007] Optionally, acquiring the target data includes: loading the target data into a distributed cache, and processing the target data in the distributed cache.

[0008] An embodiment of the present invention also provides a data acquisition device, including a first acquisition module, a determination module and a second acquisition module, wherein the first acquisition module is used to acquire a node identifier corresponding to the data in a form; the determination module is used to determine the target data in the form according to the node identifier; and the second acquisition module is used to acquire the target data.

[0009] Optionally, the device further includes a configuration module, wherein the configuration module is used to configure the node identifier corresponding to the data in the form.

[0010] Optionally, the configuration module is used to configure the node identifier corresponding to each data in the form according to the primary key or shard key in the form.

[0011] Optionally, the second acquisition module is used to load the target data into a distributed cache and process the target data in the distributed cache.

[0012] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the methods in the embodiments of the present invention is implemented.

[0013] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the processor implements any one of the methods in the embodiments of the present invention by executing a program in the memory.

[0014] Through the data acquisition method provided by the embodiment of the present invention, a form can be loaded in parallel by multiple nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of a data acquisition method provided by an embodiment of the present invention;

[0016] Figure 2 is a framework diagram of a data acquisition device according to an embodiment of the present invention;

[0017] Figure 3 is a schematic diagram of a data processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

[0019] In addition, in the embodiments of the present invention, words such as "optional" or "exemplary" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "optional" or "exemplary" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "optional" or "exemplary" is intended to present related concepts in a specific way.

[0020] An embodiment of the present invention provides a data acquisition method. Figure 1 is a flow chart of a data acquisition method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include the following steps:

[0021] S102, obtaining the node identifier corresponding to the data in the form;

[0022] S104, determining target data in the form according to the node identifier;

[0023] S106, obtaining the target data

[0024] Optionally, before obtaining the node identifier corresponding to the data in the form, the method further includes: configuring the node identifier corresponding to the data in the form.

[0025] Optionally, configuring the node identifier corresponding to the data in the form includes: configuring the node identifier corresponding to each piece of data in the form according to the primary key or shard key in the form.

[0026] When all nodes of the in-memory database access the RDBMS relational database, they mark the target data to be loaded according to the application requirements. The specific method can be to calculate the node identification ID mapped to each data in each table according to the primary key or shard key of the record value in the table according to the in-memory database sharding algorithm, and mark the node ID in the last column of the table.

[0027] The node queries the target data that needs to be loaded by the node according to the marked node ID and loads it. Multiple nodes can query the data of the same table. Since the data loaded by each node is different, the same table can be loaded in parallel on multiple nodes.

[0028] It should be noted that the embodiment of the present invention is not only applicable to loading of RDBMS relational database into distributed cache, but also applicable to loading of other databases into cache.

[0029] Optionally, obtaining the target data includes: loading the target data into a distributed cache, and processing the target data in the distributed cache.

[0030] The node loads the target data into the distributed cache and performs distributed parallel computing in the cache to achieve batch processing of the data.

[0031] The embodiment of the present invention also provides a data acquisition device. Figure 2 is a framework diagram of a data acquisition device according to an embodiment of the present invention. Figure 2 As shown, the device includes a first acquisition module 22, a determination module 24 and a second acquisition module 26, wherein:

[0032] The first acquisition module 22 is used to acquire the node identifier corresponding to the data in the form;

[0033] The determination module 24 is used to determine the target data in the form according to the node identifier;

[0034] The second acquisition module 26 is used to acquire the target data.

[0035] Optionally, the device further includes a configuration module, wherein the configuration module is used to configure the node identifier corresponding to the data in the form.

[0036] Optionally, the configuration module is used to configure the node identifier corresponding to each data in the form according to the primary key or shard key in the form.

[0037] Optionally, the second acquisition module 26 is used to load the target data into a distributed cache and process the target data in the distributed cache.

[0038] The embodiment of the present invention also provides a data processing method. Figure 3 is a schematic diagram of a data processing method according to an embodiment of the present invention, such as Figure 3 As shown, the method includes the following process:

[0039] 1. Data preprocessing

[0040] All nodes of the in-memory database access the RDBMS relational database and mark the data to be loaded according to application requirements. According to the in-memory database sharding algorithm, the node ID mapped to each piece of data in each table is calculated based on the primary key or shard key of the record value in the table, and marked in the last column of the table.

[0041] 2. Loading data

[0042] The distributed cache node queries the data that needs to be loaded by the node according to the marked node ID and loads it. Multiple nodes can query the data of the same table. Since the data loaded by each node is different, the same table can be loaded in parallel on multiple nodes.

[0043] 3. Batch logical calculation

[0044] Data is loaded into the distributed cache, and distributed parallel computing is performed in the cache to implement batch processing logic.

[0045] 4. Data Post-Processing

[0046] After data processing, each node of the distributed cache cleans up the data.

[0047] Among them, the above steps can be processed in parallel by multiple nodes at the same time, which greatly improves the processing efficiency.

[0048] Compared with the single-table node loading method, the data and node mapping algorithm provided by the embodiment of the present invention enables multiple threads to mark each piece of data in a single table in parallel, and subsequent nodes load the data that needs to be mapped to their own nodes according to the marked node ID. The entire process is multi-threaded parallel processing, which greatly speeds up the efficiency of data loading.

[0049] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory; the number of processors in the electronic device may be one or more, and the memory, as a computer-readable storage medium, may be used to store computer executable programs. The processor executes various functional applications and data processing of the electronic device by running software programs and instructions stored in the memory, that is, implementing the method in any of the above embodiments.

[0050] An embodiment of the present application also provides a storage medium containing computer executable instructions, which implement the method in any of the above embodiments when executed by a computer processor.

[0051] Optionally, the processor implements a data acquisition method by executing instructions, the method comprising:

[0052] S1, obtain the node identifier corresponding to the data in the form;

[0053] S2, determining the target data in the form according to the node identifier;

[0054] S3, obtaining the target data.

[0055] The above are merely exemplary embodiments of the present application and are not intended to limit the protection scope of the present application.

[0056] In general, various embodiments of the present application can be implemented in hardware or dedicated circuits, software, logic or any combination thereof. For example, some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, microprocessor or other computing device, although the present application is not limited thereto.

[0057] Embodiments of the present application may be implemented by executing computer program instructions by a data processor of a mobile device, for example in a processor entity, or by hardware, or by a combination of software and hardware. The computer program instructions may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages.

[0058] The block diagram of any logic flow in the accompanying drawings of the present application can represent program steps, or can represent interconnected logic circuits, modules and functions, or can represent a combination of program steps and logic circuits, modules and functions. The computer program can be stored on a memory. The memory can have any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as but not limited to read-only memory (ROM), random access memory (RAM), optical memory device and system (digital versatile disc DVD or CD disc), etc. Computer-readable media may include non-transient storage media. The data processor can be any type suitable for the local technical environment, such as but not limited to a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (FGPA) and a processor based on a multi-core processor architecture.

[0059] By way of exemplary and non-limiting examples, a detailed description of exemplary embodiments of the present application has been provided above. However, various modifications and adjustments to the above embodiments will be apparent to those skilled in the art when considered in conjunction with the accompanying drawings and claims, but will not depart from the scope of the present invention. Therefore, the proper scope of the present invention will be determined according to the claims.

Claims

1. A data acquisition method, characterized in that: include: Get the node ID corresponding to the data in the form; Determine the target data in the form according to the node identifier; Acquiring the target data includes: loading the target data into a distributed cache, and processing the target data in the distributed cache, wherein a node of the distributed cache queries the data that the node needs to load according to a marked node identifier, and loads the data, multiple nodes query the data of the same form, each node loads different data, and the same form is loaded in parallel at multiple nodes; Before obtaining the node identifier corresponding to the data in the form, it also includes: Configure the node identifier corresponding to the data in the form; The node identifier corresponding to the data in the configuration form includes: According to the sharding algorithm, the node identifier mapped to each piece of data in the form is calculated according to the primary key or sharding key in the form, and the node identifier is marked in the form.

2. A data acquisition device, characterized in that: It includes a first acquisition module, a determination module and a second acquisition module, wherein: The first acquisition module is used to acquire the node identifier corresponding to the data in the form; The determination module is used to determine the target data in the form according to the node identifier; The second acquisition module is used to acquire the target data, including: loading the target data into a distributed cache, and processing the target data in the distributed cache, wherein the node of the distributed cache queries the data that the node needs to load according to the marked node identifier, and loads the data, multiple nodes query the data of the same form, each node loads different data, and the same form is loaded in parallel at multiple nodes; The device also includes a configuration module, wherein: The configuration module is used to configure the node identifier corresponding to the data in the form; The configuration module is used to calculate the node identifier mapped to each data in the form according to the sharding algorithm and the primary key or sharding key in the form, and mark the node identifier in the form.

3. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to claim 1 is implemented.

4. An electronic device comprising a memory and a processor, characterized in that: The processor implements the method according to claim 1 by executing the computer program in the memory.

Images