Data processing method and device

Abstract

The present invention discloses a data processing method and device, and relates to the field of computer technology. The method comprises: receiving a first instruction; based on the first instruction, filtering calculation data in a measurement database, filtering out the data to be calculated according to the configuration; while filtering the data, receiving a second instruction; based on the second instruction, starting an asynchronous thread to process parameters in a configuration database; after the data filtering is completed, receiving a third instruction; and based on the third instruction, executing a subtask and returning the calculation result to the measurement database.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] Multi-node parallel computing can be implemented on MPP (massively parallel processing) databases (for example, green plum databases). MPP databases generally use scripting languages ​​such as PERL to implement batch processing of massive data. The processing logic is encapsulated into SQL statements, which are exposed in plain text when the program is deployed and cannot be encrypted. In addition, some complex financial instrument calculation processes and logics contain a large number of complex mathematical models, curve processing, etc., which are difficult to implement with SQL, resulting in a poor user experience.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: traditional relational databases are not suitable for complex logical processing of massive data, and their core codes (stored procedures) are directly exposed to users in plain text and cannot be encrypted, which is not conducive to the protection of core technologies and intellectual property rights when outputting products. Summary of the invention

[0004] In view of this, the present invention is implemented. The present invention uses the JAVA framework to implement data computing and processing logic, and combines the memory database to perform peer-to-peer distributed deployment on multiple application servers. Massive data is segmented and processed, distributed to each computing node for parallel computing, and computing nodes can be expanded according to computing performance requirements to improve computing efficiency, solving the problem of complex logical batch processing of massive data, and the code can be encrypted, which can effectively protect intellectual property rights.

[0005] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a data processing method is provided, which is characterized in that it includes: receiving a first instruction; based on the first instruction, filtering the calculation data in the measurement database, and filtering out the data to be calculated according to the configuration; while the data is being filtered, receiving a second instruction; based on the second instruction, starting an asynchronous thread to perform parameter processing in the configuration database; after the data filtering is completed, receiving a third instruction; and based on the third instruction, executing a subtask and returning the calculation result to the measurement database.

[0006] Optionally, the data processing method is characterized in that the parameter processing includes: processing the parameters required for calculation into a form that can be directly used, and then writing it into a data file, and the computing node installs the parameters into a memory database.

[0007] Optionally, the data processing method is characterized by slicing the data and forming one or more subtasks, and distributing the one or more subtasks to idle computing processes.

[0008] Optionally, the data processing method is characterized in that the calculation logics of different subtasks are the same and independent of each other.

[0009] Optionally, the data processing method is characterized by further comprising: executing subtasks through a computing process, and moving the data to be calculated to a memory database for calculation.

[0010] Optionally, the data processing method is characterized by further comprising: after the calculation is completed, waiting for the next subtask.

[0011] Optionally, the data processing method is characterized by further comprising: after all calculation results are returned to the measurement database, the calculation results are summarized.

[0012] Optionally, the data processing method is characterized in that the calculation of the slice subtask is performed in an in-memory database.

[0013] Optionally, the data processing method is characterized in that the subtasks are distributed to each computing node in a distributed peer-to-peer computing manner.

[0014] To achieve the above-mentioned purpose, according to the second aspect of an embodiment of the present invention, a data processing device is provided, characterized in that it includes: a main database module, the main database module includes a configuration database and a calculation database; a memory database module, the memory database module includes a memory database; and a controller, the controller is operably connected to the main database module and the memory database module, and the controller is configured to: receive a first instruction; based on the first instruction, filter the calculation data in the calculation database, and filter out the data to be calculated according to the configuration; while filtering the data, receive a second instruction; based on the second instruction, start an asynchronous thread to perform parameter processing in the configuration database; after the data screening is completed, receive a third instruction; and based on the third instruction, execute a subtask and return the calculation result to the calculation database.

[0015] Optionally, the data processing device is characterized in that the controller is further configured to: slice the data and form one or more subtasks, and distribute the one or more subtasks to idle computing processes.

[0016] Optionally, the data processing device is characterized in that the parameter processing includes: processing the parameters required for calculation into a form that can be directly used, and then writing it into a data file, and the computing node installs the parameters into the memory database.

[0017] Optionally, the data processing device is characterized in that the calculation logics of different subtasks are the same and independent of each other.

[0018] Optionally, the data processing device is characterized in that it executes subtasks through a computing process and moves data to be calculated to a memory database for calculation.

[0019] Optionally, the data processing device is characterized in that the controller is further configured to: after the calculation is completed, wait for the next subtask.

[0020] Optionally, the data processing device is characterized in that the controller is configured such that: after all calculation results are transmitted back to the measurement database, the calculation results are summarized.

[0021] Optionally, the data processing device is characterized in that the calculation of the sliced ​​subtask is performed in the memory database.

[0022] Optionally, the data processing device is characterized in that the controller is further configured to distribute the subtask to each computing node in a distributed peer-to-peer computing manner.

[0023] To achieve the above-mentioned purpose, according to the third aspect of an embodiment of the present invention, there is provided an electronic device for data processing, characterized in that it comprises: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement any method described in the data processing device method as described above.

[0024] To achieve the above-mentioned purpose, according to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, a method as described in any one of the above-mentioned data processing device methods is implemented.

[0025] One embodiment of the above invention has the following advantages or beneficial effects: it has the following three characteristics: it is scalable, easy to transplant and deploy, and can be encrypted, as described in detail as follows:

[0026] (1) Distributed peer-to-peer computing is used to evenly distribute massive amounts of data to each computing node, and the number of nodes can be easily expanded as needed, effectively improving computing efficiency;

[0027] (2) It is implemented using the general JAVA language and in-memory database, has no special requirements for the operating system and hardware configuration, and can be easily ported; and

[0028] (3) The core processing logic of the program is implemented using JAVA code, and the resulting product installation package is compiled into binary code using code obfuscation technology and technically encrypted to facilitate product output.

[0029] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0031] Figure 1 is an architecture diagram of a system to which an embodiment of the present invention is applied;

[0032] Figure 2 is a schematic diagram of a parallel computing process provided by an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of a data processing method according to an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of a data processing device according to an embodiment of the present invention;

[0035] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0036] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] Figure 1 It is an architecture diagram of a system to which an embodiment of the present invention is applied.

[0039] like Figure 1 As shown in the figure, the main database consists of two databases: the configuration database and the calculation database. The configuration database processes and stores the calculation parameters. The calculation database stores the full amount of calculation source data and calculation results.

[0040] Perform computation of slice subtasks in the memory database.

[0041] The master control node is deployed on a separate application server (AP) and connected to the configuration database and the measurement database. It has the functions of source data screening and slicing, calculation parameter processing, calculation task scheduling, and calculation result aggregation. Among them, the database is located on another server, and the AP can access the database. Among them, the master control node can access the configuration database, and then execute SQL statements on the database for parameter processing, and the master control node can access the measurement database and execute SQL in the database to summarize the results.

[0042] Computing nodes are deployed on all remaining APs, one per AP, connecting the configuration database, the measurement database, and the memory database, thus having the functions of transferring computing parameters and returning computing results. The memory database is directly deployed on the AP, and the data is stored in the memory, which is equivalent to a local database.

[0043] The computing process is deployed on all remaining APs, with multiple processes deployed on each AP, connected to the measurement database and memory database. It has the functions of slice data transfer and local computing. A node and multiple computing processes are deployed on an AP. The node can restart the suspended computing process and return the results of the computing process to the measurement database.

[0044] The present invention has the advantages of linear scalability, convenient portability and deployment, and encryption. Specifically, it adopts a distributed peer-to-peer computing method to evenly distribute massive data to each computing node, and can easily expand the number of nodes as needed, effectively improving computing efficiency; it adopts the general JAVA language and in-memory database, has no special requirements for the operating system and hardware configuration, and can be easily ported; and (3) the core processing logic of the program is implemented in JAVA code, and the resulting product installation package is compiled into binary code using code obfuscation technology, and technical encryption is performed to facilitate product output.

[0045] By adopting the distributed peer-to-peer computing method, massive data is evenly distributed to each computing node (that is, subtasks are distributed to each computing node), and the number of nodes can be easily expanded as needed, effectively improving computing efficiency, and with the expansion of the number of machines, the improvement of computing efficiency is almost linear. Currently, 8 APs are used, each with 4 computing processes, and 50 million data can be calculated within 7 hours. The traditional method uses 8 MPP databases for batch processing, which takes 20 hours. In addition, the operation and maintenance of MPP databases are complex and the machines are expensive.

[0046] Traditional MPP databases directly use SQL statements for processing, while the Java framework can call SQL statements and combine with Java itself to implement logic as well as more flexible and complex functions.

[0047] It is implemented using the general JAVA language and in-memory database, has no special requirements for the operating system and hardware configuration, and can be easily transplanted. According to the present invention, the computing logic of various different businesses can be used to perform distributed computing. For a main task, each subtask of the main task executes the same logic. If different business logics need to be executed, the corresponding code can be added according to the needs. In addition, different computing nodes all require the JAVA framework.

[0048] The core processing logic of the program is implemented with JAVA code, and the resulting product installation package is compiled into binary code using code obfuscation technology and technically encrypted to facilitate product output.

[0049] Figure 2 It is a schematic diagram of a parallel computing process provided by an embodiment of the present invention.

[0050] like Figure 2 As shown, after receiving the calculation instruction, the master control node will filter the calculation data in the measurement database and filter out the data that needs to be calculated according to the configuration.

[0051] While filtering the data, the master node will start an asynchronous thread to process parameters in the configuration database, process the parameters needed for calculation into a form that can be directly used, and then write it into a data file. The computing node will load the parameters into the memory database.

[0052] After the data screening is completed, the master node will slice the data, each slice is a subtask, and then find the idle computing process to distribute the subtasks. The computing logic of different subtasks is the same, independent of each other, and there is no dependency. For example, inside the database, each piece of data has its own serial number, and the data can be sliced ​​according to the serial number, such as 1-1000, 1001-2000, and so on. A slice is a subtask.

[0053] After receiving the subtask, the calculation process will move the data to be calculated to the memory database for calculation. After the calculation is completed, it will wait for the arrival of the next subtask. The calculation results are returned to the measurement database by the calculation node.

[0054] When all calculation results are sent back to the calculation database, the master control node summarizes the calculation results and ends the calculation task.

[0055] Figure 3 It is a schematic diagram of a data processing method according to an embodiment of the present invention.

[0056] like Figure 3 As shown, the data processing method includes: receiving a first instruction (S301); based on the first instruction, filtering the calculation data in the measurement database, and filtering out the data to be calculated according to the configuration (S302); while filtering the data, receiving a second instruction (S303); based on the second instruction, starting an asynchronous thread to perform parameter processing in the configuration database (S304); after the data filtering is completed, receiving a third instruction (S305); and based on the third instruction, executing a subtask and returning the calculation result to the measurement database (S306).

[0057] Optionally, parameter processing includes: processing parameters required for calculation into a form that can be directly used, and then writing the form into a data file, and the computing node loads the parameters into a memory database.

[0058] Optionally, the data is sliced ​​and formed into one or more subtasks, and the one or more subtasks are distributed to idle computing processes.

[0059] Optionally, the calculation logics of different subtasks are the same and independent of each other.

[0060] Optionally, the subtask is executed by the computing process, and the data to be calculated is moved to the memory database for calculation.

[0061] Optionally, after the calculation is completed, wait for the next subtask.

[0062] Optionally, after all calculation results are transmitted back to the measurement database, the calculation results are summarized.

[0063] Optionally, computation of the slice subtasks is performed in an in-memory database.

[0064] Optionally, the subtask is distributed to each computing node in a distributed peer-to-peer computing manner.

[0065] Figure 4 It is a schematic diagram of a data processing device according to an embodiment of the present invention.

[0066] The data processing device 400 includes: a main database module 410, which includes a configuration database 411 and a measurement database 412; a memory database module 420, which includes a memory database 421; and a controller 430, which is operably connected to the main database module 410 and the memory database module 420, and the controller 430 is configured to:

[0067] receiving a first instruction;

[0068] Based on the first instruction, the calculation data is screened in the measurement database 412, and the data to be calculated is screened according to the configuration;

[0069] While the data is being screened, a second instruction is received;

[0070] Based on the second instruction, an asynchronous thread is started to perform parameter processing in the configuration database 410;

[0071] After the data screening is completed, a third instruction is received; and

[0072] Based on the third instruction, the subtask is executed and the calculation result is returned to the calculation database 410 .

[0073] Optionally, parameter processing includes: processing the parameters required for calculation into a form that can be directly used, and then writing it into a data file, and the computing node loads the parameters into the memory database 420.

[0074] Optionally, the controller 430 is further configured to: slice the data and form one or more subtasks, and distribute the one or more subtasks to idle computing processes.

[0075] Optionally, the calculation logics of different subtasks are the same and independent of each other.

[0076] Optionally, the subtask is executed by the computing process, and the data to be calculated is moved to the memory database 420 for calculation.

[0077] Optionally, the controller 430 is further configured to: after the calculation is completed, wait for the next subtask.

[0078] Optionally, the controller 430 is configured such that after all calculation results are transmitted back to the measurement database 412 , the calculation results are summarized.

[0079] Optionally, calculation of the slice subtask is performed in the memory database 421.

[0080] Optionally, the controller 430 is further configured to distribute the subtask to each computing node in a distributed peer-to-peer computing manner.

[0081] An embodiment of the present invention also provides an electronic device for data processing, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in any of the above embodiments.

[0082] An embodiment of the present invention further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in any of the above embodiments is implemented.

[0083] Figure 5 An exemplary system architecture 500 is shown in which a method or an apparatus for data processing according to an embodiment of the present invention can be applied.

[0084] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0085] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Terminal devices 501, 502, 503 can be installed with various communication client applications, such as web browser applications, search applications, instant messaging tools, and email clients.

[0086] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0087] The server 505 may be a server that provides various services. It should be noted that the method for data processing provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the device for data processing is generally set in the server 505 .

[0088] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0089] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0090] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0091] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0092] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0093] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0094] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The modules and / or units involved in the embodiments of the present invention may be implemented by software or hardware. The modules and / or units described may also be set in a processor, for example, may be described as: a data processing device, characterized in that it comprises: a main database module, the main database module comprises a configuration database and a measurement database; a memory database module, the memory database module comprises a memory database; and a controller, the controller is operably connected to the main database module and the memory database module, and the controller is configured to: receive a first instruction; based on the first instruction, filter the calculation data in the measurement database, and filter out the data to be calculated according to the configuration; while filtering the data, receive a second instruction; based on the second instruction, start an asynchronous thread to perform parameter processing in the configuration database; after the data filtering is completed, receive a third instruction; and based on the third instruction, execute a subtask and return the calculation result to the measurement database.

[0096] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: receiving a first instruction; based on the first instruction, filtering the calculation data in the measurement database, and filtering out the data to be calculated according to the configuration; while filtering the data, receiving a second instruction; based on the second instruction, starting an asynchronous thread to perform parameter processing in the configuration database; after the data filtering is completed, receiving a third instruction; and based on the third instruction, executing a subtask and returning the calculation result to the measurement database.

[0097] The business process processing method and device provided by the embodiment of the present invention can have the following three characteristics: scalability, convenient transplantation and deployment, and encryption, as described in detail below:

[0098] (1) Distributed peer-to-peer computing is used to evenly distribute massive amounts of data to each computing node, and the number of nodes can be easily expanded as needed, effectively improving computing efficiency;

[0099] (2) It is implemented using the general JAVA language and in-memory database, has no special requirements for the operating system and hardware configuration, and can be easily ported; and

[0100] (3) The core processing logic of the program is implemented using JAVA code, and the resulting product installation package is compiled into binary code using code obfuscation technology and technically encrypted to facilitate product output.

[0101] The present invention uses the JAVA framework to implement data computing and processing logic, and combines the memory database to perform peer-to-peer distributed deployment on multiple application servers. Massive data is segmented and processed, distributed to each computing node for parallel computing, and computing nodes can be expanded according to computing performance requirements to improve computing efficiency, solving the problem of complex logical batch processing of massive data, and the code can be encrypted to effectively protect intellectual property rights.

[0102] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0103] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: receiving a first instruction; Based on the first instruction, the calculation data is screened in the calculation database, and the data to be calculated is screened out according to the configuration, wherein the calculation database stores the full amount of calculation source data and calculation results; While the data is being screened, a second instruction is received; Based on the second instruction, an asynchronous thread is started to process parameters in the configuration database, wherein the parameter processing includes: processing the parameters required for calculation into a form for direct use, and then writing the parameters into a data file and loading it into the memory database, wherein the configuration database processes and stores the calculation parameters; After the data screening is completed, a third instruction is received; and Based on the third instruction, the data is sliced ​​in the memory database according to the sequence number of each data and one or more subtasks are formed, and the one or more subtasks are distributed to idle computing processes. The subtasks are executed by the computing processes, and the data to be calculated is moved to the memory database for calculation and the calculation results are returned to the measurement database.

2. The method according to claim 1, characterized in that The calculation logic of different subtasks is the same and independent of each other.

3. The method according to claim 1, characterized in that Further including: After the calculation is completed, wait for the next subtask.

4. The method according to claim 1, characterized in that: Further including: After all calculation results are transmitted back to the measurement database, the calculation results are summarized.

5. The method according to claim 1, characterized in that The subtasks are distributed to each computing node in a distributed peer-to-peer computing manner.

6. A data processing device, characterized in that: include: A main database module, wherein the main database module includes a configuration database and a measurement database; an in-memory database module, the in-memory database module comprising an in-memory database; and a controller operably connected to the main database module and the in-memory database module, and configured to: receiving a first instruction; Based on the first instruction, the calculation data is screened in the calculation database, and the data to be calculated is screened out according to the configuration, wherein the calculation database stores the full amount of calculation source data and calculation results; While the data is being screened, a second instruction is received; Starting an asynchronous thread based on the second instruction to process parameters in the configuration database includes: processing the parameters required for calculation into a form that can be used directly, and then writing the parameters into a data file and loading it into the memory database, wherein the configuration database processes and stores the calculation parameters; After the data screening is completed, a third instruction is received; and Based on the third instruction, the data is sliced ​​in the memory database according to the sequence number of each data and one or more subtasks are formed, and the one or more subtasks are distributed to idle computing processes. The subtasks are executed by the computing process, and the data to be calculated is moved to the memory database for calculation and the calculation results are returned to the measurement database.

7. The device according to claim 6, characterized in that The calculation logic of different subtasks is the same and independent of each other.

8. The device according to claim 6, characterized in that The controller is further configured to: after the calculation is completed, wait for the next subtask.

9. The device according to claim 6, characterized in that The calculation of the sliced ​​subtask is performed in the in-memory database.

10. The device according to claim 6, characterized in that The controller is further configured to distribute the subtasks to each computing node in a distributed peer-to-peer computing manner.

11. An electronic device, characterized in that: include: one or more processors; A storage device, used for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-5.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images