Data processing method and device, equipment and storage medium

Abstract

The application provides a data processing method and device, equipment and a storage medium, which can be applied to any node equipment in a data processing system. The node equipment obtains a plurality of to-be-processed tasks from a message queue of a message server, wherein each to-be-processed task comprises operation data of a user for a target product and a data operation mode corresponding to the operation data. The node equipment allocates threads to each to-be-processed task according to the data operation mode, executes the plurality of to-be-processed tasks in parallel through the plurality of threads, and obtains a data processing result of each to-be-processed task. In the first aspect, the message server distributes to-be-processed tasks to each node equipment in the data processing system, realizes data decoupling between the node equipments, and reduces the dependency relationship between the node equipments. On the other hand, each node equipment can concurrently process to-be-processed tasks of different data operation modes, thereby improving the data processing efficiency of the system.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of information technology, and in particular, to a data processing method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of Internet finance and the improvement of people's financial awareness, various financial application programs APP have become very important traffic portals for personal financial finance. People purchase various financial products such as stocks and funds through financial APP.

[0003] For service enterprises providing various financial products, a perfect data processing system needs to be established to obtain user transaction data, calculate various parameters and indicators of various financial products, and facilitate user query of related information. The existing data processing system provides a synchronous Remote Procedure Call Protocol (RPC) interface scheme, that is, the service ends of the data processing system call each other through the RPC interface to ensure data consistency.

[0004] The above scheme has strong dependency between service ends, and the data processing efficiency of the system is low. SUMMARY

[0005] Embodiments of the present application provide a data processing method, device, equipment and storage medium to improve the data processing efficiency of the system.

[0006] A first aspect of an embodiment of the present application provides a data processing method, comprising:

[0007] obtaining a plurality of to-be-processed tasks from a message queue of a message server, each of the to-be-processed tasks including operation data of a user for a target product and a data operation mode corresponding to the operation data;

[0008] allocating a thread to each of the to-be-processed tasks according to the data operation mode;

[0009] executing the plurality of to-be-processed tasks in parallel through a plurality of threads to obtain a data processing result of each of the to-be-processed tasks.

[0010] In an embodiment of the present application, the step of allocating a thread to each of the to-be-processed tasks according to the data operation mode comprises:

[0011] performing a hash transformation on the data operation mode in each of the to-be-processed tasks to obtain a hash value corresponding to the data operation mode;

[0012] determining a thread number corresponding to each of the to-be-processed tasks according to the hash value and a preset thread number of the node device.

[0013] In an embodiment of the present application, the determining of the thread number corresponding to each of the to-be-processed tasks according to the hash value and the preset thread number of the node device comprises:

[0014] obtaining the thread number corresponding to each of the to-be-processed tasks by taking the modulus of the hash value on the thread number.

[0015] In an embodiment of the present application, if the operation data corresponding to at least two of the to-be-processed tasks have the same data operation mode, the method further comprises:

[0016] processing the at least two to-be-processed tasks in the thread corresponding to the same data operation mode in turn according to the queue order of the at least two to-be-processed tasks in the message queue.

[0017] In an embodiment of the present application, for each of the threads, before processing each of the to-be-processed tasks in the thread, the method further comprises:

[0018] executing a first command in the thread, the first command being used for requesting to lock the data operation mode of the to-be-processed task;

[0019] determining whether to execute the to-be-processed task in the thread according to the return result of the first command.

[0020] In an embodiment of the present application, the determining of whether to execute the to-be-processed task in the thread according to the return result of the first command comprises:

[0021] if the return result is a first value, executing the to-be-processed task in the thread; or

[0022] if the return result is a second value, waiting for other threads to release the lock of the data operation mode.

[0023] In an embodiment of the present application, after the to-be-processed task is processed in the thread, or the execution of the to-be-processed task in the thread is timed out, the method further comprises:

[0024] releasing the lock of the data operation mode of the to-be-processed task in the thread.

[0025] In an embodiment of the present application, the executing of the to-be-processed task in the thread comprises:

[0026] obtaining a historical data processing result corresponding to the data operation mode of the to-be-processed task from a database server;

[0027] According to the historical processing result and the operation data of the to-be-processed task, the historical processing result is updated.

[0028] A second aspect of the embodiment of the application provides a data processing apparatus, comprising:

[0029] An acquisition module is configured to acquire a plurality of to-be-processed tasks from a message queue of a message server, wherein each to-be-processed task comprises operation data of a user for a target product and a data operation mode corresponding to the operation data.

[0030] A processing module is configured to assign a thread to each to-be-processed task according to the data operation mode.

[0031] The plurality of to-be-processed tasks are executed in parallel through the plurality of threads, and a data processing result of each to-be-processed task is obtained.

[0032] In an embodiment of the application, the processing module is specifically configured to perform a hash transformation on the data operation mode in each to-be-processed task, and obtain a hash value corresponding to the data operation mode.

[0033] According to the hash value and a preset thread number of the node device, a thread number corresponding to each to-be-processed task is determined.

[0034] In an embodiment of the application, the processing module is specifically configured to take a modulus of the thread number through the hash value, and obtain a thread number corresponding to each to-be-processed task.

[0035] In an embodiment of the application, if the operation data of at least two to-be-processed tasks in the plurality of to-be-processed tasks correspond to the same data operation mode, the processing module is further configured to:

[0036] According to a queue order of the at least two to-be-processed tasks in the message queue, the at least two to-be-processed tasks are processed in a thread corresponding to the same data operation mode in sequence.

[0037] In an embodiment of the application, for each thread, before the processing module processes each to-be-processed task in the thread, the processing module is further configured to:

[0038] A first command is executed in the thread, and the first command is used to request to lock the data operation mode of the to-be-processed task.

[0039] According to a return result of the first command, it is determined whether the to-be-processed task is executed in the thread.

[0040] In an embodiment of the application, the processing module is specifically configured to:

[0041] if the return result is the first value, executing the to-be-processed task in the thread; or

[0042] if the return result is the second value, waiting for other threads to release the lock of the data operation mode.

[0043] In an embodiment of the present application, after the to-be-processed task is processed in the thread, or the thread executes the to-be-processed task for a timeout, the processing module is further configured to:

[0044] release the lock of the data operation mode of the to-be-processed task in the thread.

[0045] In an embodiment of the present application, the obtaining module is further configured to obtain a historical data processing result corresponding to the data operation mode of the to-be-processed task from a database server;

[0046] The processing module is further configured to update the historical processing result according to the historical processing result and the operation data of the to-be-processed task.

[0047] A third aspect of an embodiment of the present application provides an electronic device, comprising:

[0048] a memory;

[0049] a processor; and

[0050] a computer program;

[0051] The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of the first aspect.

[0052] A fourth aspect of an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method according to any one of the first aspect.

[0053] A fifth aspect of an embodiment of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the method according to any one of the first aspect.

[0054] The embodiment of the present application provides a data processing method, device and equipment and a storage medium, which can be applied to any node device in a data processing system. The node device obtains a plurality of to-be-processed tasks from a message queue of a message server, wherein each to-be-processed task includes operation data of a user for a target product and a data operation mode corresponding to the operation data. The node device allocates threads for each to-be-processed task according to the data operation mode, executes the plurality of to-be-processed tasks in parallel through the plurality of threads, and obtains a data processing result of each to-be-processed task. In the first aspect, the message server distributes to-be-processed tasks to each node device in the data processing system, realizes data decoupling between the node devices, and reduces the dependency relationship between the node devices. In the second aspect, each node device can concurrently process to-be-processed tasks of different data operation modes, and the data processing efficiency of the system is improved. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A system architecture diagram of the data processing method provided by the embodiment of the present application is provided.

[0056] Figure 2 A scene interaction diagram of the data processing method provided by the embodiment of the present application is provided.

[0057] Figure 3 A flowchart of the data processing method provided by the embodiment of the present application is provided.

[0058] Figure 4 A flowchart of the data processing method provided by the embodiment of the present application is provided.

[0059] Figure 5 A structure diagram of the data processing device provided by the embodiment of the present application is provided.

[0060] Figure 6 A hardware structure diagram of the electronic device provided by the embodiment of the present application is provided. DETAILED DESCRIPTION

[0061] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0062] It is to be understood that the terms "including" and "having" and any variation thereof used herein are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or units need not be limited to those clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products or devices.

[0063] First, the professional terms involved in the embodiments of the application are briefly described.

[0064] Asset management business is a new type of business developed on the basis of traditional business of securities operating institutions, mainly refers to securities, futures, fund and other financial investment companies as asset managers, according to the way, condition, requirement and limitation agreed in the asset management contract, operating and operating the customer's assets, providing investment management services for customers of securities and other financial products.

[0065] Fast message queue (FMQ) is a system or platform, also known as message middleware server or message server, used for communication between application programs or service nodes, mainly through message passing to complete interaction.

[0066] Distributed lock is a kind of distributed coordination technology, in order to prevent mutual interference between multiple processes in distributed system, through the technology to realize the reasonable scheduling of multiple processes. In distributed system, a method can only be executed by a thread of a service node at the same time.

[0067] Position refers to the total amount of investment products in the stock account of the investment user. There is no product called empty position, and there is product called position.

[0068] In the financial service industry, in order to improve the service quality of asset management business, service enterprises need to establish a perfect data processing system. The data processing system needs to undertake the tasks of position calculation, transaction deduction, etc.; according to the transaction information, calculate the portfolio position change, portfolio value, portfolio net value, product size and other parameter indexes; according to the valuation table information after daily settlement, deduce the portfolio net value, transaction change and other parameter indexes.

[0069] In the current data processing system, various service providers provide synchronous remote procedure call (RPC) interface, in which single thread serial calculation of portfolio position change, portfolio value, portfolio net value, product size and other parameter indexes, deduction of portfolio net value, transaction change and other parameter indexes. In practical application, java synchronization lock synchronized is used to realize the synchronization of code block.

[0070] The synchronous RPC interface scheme is interdependent between service nodes, which increases the difficulty of service management. Taking a data processing system of a fund management business as an example, the system obtains various business data initiated by different users on the client side. If the service nodes in the system are interdependent and not decoupled enough, too many RPC synchronous dependencies between services will be caused, which will inevitably affect the efficiency and accuracy of data processing.

[0071] In addition, the implementation scheme of the single-threaded synchronization lock is that the service cannot be horizontally expanded. The single-threaded synchronization lock can ensure the consistency of data between services, but the service cannot be distributed and expanded. Once the performance bottleneck is reached, the service node cannot be horizontally expanded, and the overall throughput of the system cannot be improved.

[0072] To solve the above problems, the embodiments of the present application provide a data processing method to solve the problem of high difficulty of service interdependence and horizontal expansion of the current system, and improve the overall throughput of the system. The overall idea of the present scheme is as follows: since the message is directly stored and distributed by the message middleware, the business party only needs to focus on how to send the message to the message middleware server, and each service node only needs to focus on how to obtain the message from the message middleware server, so as to realize the decoupling between service nodes and reduce the dependence between service nodes. Each service node in the system can be configured with multiple threads, each thread listens to the tasks in the memory queue, and executes each task according to the order of the memory queue. When each thread processes a task, it needs to obtain the corresponding distributed lock according to the data operation mode of each task to ensure the consistency of data processing. In order to further improve the overall throughput of the system, multiple service nodes can be deployed in the system to realize the horizontal expansion of the system.

[0073] The system architecture of the technical scheme provided by the embodiments of the present application will be briefly introduced below in combination with the drawings. Exemplarily, Figure 1 The system architecture diagram of the data processing method provided by the embodiments of the present application is shown in Figure 1 The data processing system provided by the present embodiment includes:

[0074] A plurality of clients (for example, the clients 11, 12 and 13 shown in Figure 1 The message server 14, a plurality of node devices (for example, the node devices 15, 16 and 17 shown in Figure 1 The database server 18. The plurality of clients are connected with the message server, the message server is connected with each node device, and each node device is connected with the database server.

[0075] Exemplarily, Figure 2 The scene interaction diagram of the data processing method provided by the embodiments of the present application, which includes a business party (i.e. any client in Figure 1 The fast message queue server (i.e. Figure 1a message server (14) in the system architecture shown in Figure 1 Unlike the system architecture shown in Figure 2 The database server in this embodiment has multiple databases, such as database 1 and database 2, which are connected to each other to form a distributed database system, and the data in each database in the distributed database system is shared with each other.

[0076] For ease of understanding, the following takes a fund management business as an example to briefly describe the interaction process between devices in the foregoing data processing system:

[0077] (1) The user initiates an investment transaction through different clients, and the generated operation data (or transaction data) is pushed to the message server. According to the user operation time, the message server sequentially adds the operation data to the message queue of the message server, and each node device obtains a to-be-processed task from the message server, which includes the user's operation data and the data operation mode corresponding to the operation data.

[0078] (2) Each node device can execute multiple to-be-processed tasks in parallel at the same time, but the to-be-processed tasks of the same data operation mode can only be executed in one thread of one node device at the same time to ensure the consistency requirement of data processing.

[0079] (3) After any node device completes a task, it stores the data processing result to the database server so that the subsequent node device or other node device can obtain the data processing result.

[0080] The technical solutions of the present application will be described in detail in the following specific embodiments. The following several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0081] Figure 3 The flowchart of the data processing method provided by the embodiment of the present application. The data processing method provided by the embodiment of the present application can be applied to Figure 1 or Figure 2 The node device shown in the system architecture shown in Figure 3 The data processing method comprises the following steps:

[0082] Step 201, obtaining multiple to-be-processed tasks from the message queue of the message server.

[0083] Each to-be-processed task includes the operation data of the user for the target product and the data operation mode corresponding to the operation data.

[0084] In the embodiment, the message server receives operation data initiated by different users through clients, adds the operation data into a message queue of the message server according to a time sequence of the operation data, so that each node device in the data processing system obtains a to-be-processed task from the message queue.

[0085] The operation data refers to transaction data of a user for a target product, the target product can be a stock, a fund, a futures, and the like, and of course can be an entity product or a service product in other fields, and the embodiment of the application does not limit this. For example, user A purchases a fund product of 10,000 yuan for one year, and user B purchases 200 shares of stock code 600xxx.

[0086] The data operation mode refers to an investment portfolio mode adopted by the operation data. For example, user A creates two investment portfolio modes, for example, investment portfolio mode 1 is 20% of a stock market, 30% of a futures market, and 50% of a fund market, and investment portfolio mode 2 is 30% of a real estate market, 40% of a stock market, and 30% of a gold market. If user A purchases stock 600xx1 and associates investment portfolio mode 1, the data operation mode corresponding to the transaction data is investment portfolio mode 1, and the number is code-001. If user A purchases stock 600xx2 and associates investment portfolio mode 2, the data operation mode corresponding to the transaction data is investment portfolio mode 2, and the number is code-002.

[0087] In step 202, a thread is allocated for each to-be-processed task according to the data operation mode.

[0088] The node device obtains a plurality of to-be-processed tasks from the message queue of the message server, and allocates a thread for each to-be-processed task according to the data operation mode corresponding to the operation data in the to-be-processed task. Specifically, the node device performs a hash transformation on the data operation mode corresponding to the operation data in the to-be-processed task to obtain a hash value corresponding to the data operation mode, and determines a thread number corresponding to each to-be-processed task according to the hash value and a preset thread number of the node device. Specifically, the thread number corresponding to each to-be-processed task can be obtained by taking the modulus of the hash value with the thread number.

[0089] For example, it is assumed that the preset thread number of the node device is 4, that is, the node device can execute a maximum of 4 to-be-processed tasks in parallel at the same time. The data operation mode corresponding to the operation data in a to-be-processed task received by the node device can be represented by an investment portfolio number, for example, code-001, a hash value is calculated according to the investment portfolio number, for example, 101, and then the modulus of the above thread number is taken, for example, 101%4=1, so that the to-be-processed task enters the first memory queue and is processed by the first thread.

[0090] In one embodiment of the present application, if the operation data of at least two of the plurality of to-be-processed tasks correspond to the same data operation mode, the data processing method further comprises: the node device sequentially processes the at least two to-be-processed tasks in the thread corresponding to the same data operation mode according to the queue order of the at least two to-be-processed tasks in the message queue.

[0091] Specifically, the node device sequentially adds the operation data of the same data operation mode into the memory queue of the thread corresponding to the same data operation mode according to the queue order in the message queue, so that the operation data of the same data operation mode is sequentially processed, and the consistency of the data processing result is ensured.

[0092] Step 203: executing the plurality of to-be-processed tasks in parallel through the plurality of threads to obtain the data processing result of each to-be-processed task.

[0093] In the embodiment, each node device in the data processing system can start a plurality of threads, for example, 4 threads, to process the operation data of each user in the system. Since there are a plurality of node devices in the data processing system, there are a plurality of parallel threads in the system, for example, if the system includes 5 node devices, the total number of parallel threads in the system is 5x4=20. In actual application, the node devices can be expanded horizontally according to actual needs to improve the overall throughput of the system.

[0094] As can be seen from the above example, the data processing method provided in the embodiment can ensure that the operation data of the same investment portfolio in the same node device always enters the same memory queue in sequence and is sequentially processed in the thread corresponding to the memory queue. In this way, each node device can concurrently process the to-be-processed tasks in the plurality of threads, that is, each node device can simultaneously process the operation data of a plurality of investment portfolios. The operation data of the same investment portfolio is sequentially processed, thereby ensuring the consistency of the data processing result in the node device.

[0095] The data processing method provided in the embodiment can be applied to any node device in the data processing system. The node device obtains a plurality of to-be-processed tasks from the message queue of the message server, wherein each to-be-processed task includes operation data of a user for a target product and a data operation mode corresponding to the operation data. The node device allocates a thread for each to-be-processed task according to the data operation mode, executes the plurality of to-be-processed tasks in parallel through the plurality of threads, and obtains the data processing result of each to-be-processed task. In the first aspect, the message server distributes the to-be-processed tasks to each node device in the data processing system in the embodiment, thereby realizing data decoupling between the node devices and reducing the dependency relationship between the node devices. On the other hand, each node device can concurrently process to-be-processed tasks of different data operation modes, thereby improving the data processing efficiency of the system.

[0096] On the basis of the above embodiments, the following embodiment shows the process of data processing of the node device of the data processing system based on the distributed lock.

[0097] Figure 4 The flowchart of the data processing method provided in the embodiments of the present application can also be applied to the node device shown in Figure 1 or Figure 2 , as shown in Figure 4 , the data processing method comprises the following steps:

[0098] Step 301, executing the first command in the thread.

[0099] The first command is used to request the lock of the data operation mode of the to-be-processed task.

[0100] Step 302, determining whether to execute the to-be-processed task in the thread according to the return result of the first command.

[0101] If the return result is the first value, the to-be-processed task is executed in the thread.

[0102] If the return result is the second value, the lock of the data operation mode is waited to be released by other threads.

[0103] In an embodiment of the present application, the first command is setnx command. The return result of the setnx command comprises the first value and the second value. Exemplarily, the first value is 1 and the second value is 0. If the return result is 1, it indicates that the data operation mode of the to-be-processed task is not locked, and the thread successfully obtains the lock. If the return result is 0, it indicates that the data operation mode of the to-be-processed task is locked by other threads, and the lock needs to be waited to be released by other threads.

[0104] In an embodiment of the present application, after the node device executes the to-be-processed task in the thread, the lock of the data operation mode of the to-be-processed task in the thread needs to be released in time.

[0105] In an embodiment of the present application, when the node device executes the to-be-processed task in the thread, if an error occurs in the execution process, the current resource will be locked, causing other threads to be unable to obtain the lock. In order to solve the above problem, a preset time length needs to be set for each thread, so as to ensure that the lock in each thread can be automatically released when the preset time length is reached, avoiding other threads being in the waiting state all the time. That is, when the task of a thread of the node device is timed out, the node device can release the lock of the data operation mode of the to-be-processed task in the thread.

[0106] In one embodiment of this application, executing a task to be processed in a thread includes: obtaining historical data processing results corresponding to the data operation mode of the task to be processed from a database server; and updating the historical processing results based on the historical processing results and the operation data of the task to be processed.

[0107] It should be noted that the database server in this embodiment can be one or more, and this embodiment does not impose any restrictions on this. Multiple database servers are interconnected to achieve data sharing.

[0108] As can be seen from the above description, after the data processing is completed, each thread of each node device in the data processing system will update the data processing results to the database server to ensure the consistency of data processing.

[0109] by Figure 2 Taking the illustrated application scenario as an example, suppose node device 1 obtains user A's purchase transaction data for fund product 1 from the message server. The portfolio number corresponding to this transaction data is code-001. At the same time, node device 2 obtains user B's purchase transaction data for fund product 2 from the message server. The portfolio number corresponding to this transaction data is also code-001. Although these two transactions have the same portfolio number, because the transaction data is on different node devices and belongs to different users, node device A and node device B can process the data simultaneously.

[0110] For example, suppose node device 1 retrieves user A's purchase transaction data for fund product 1 from the message server, and the portfolio number corresponding to this transaction data is code-001. Simultaneously, node device 2 retrieves user A's purchase transaction data for stock code 600xx2 from the message server, and the portfolio number corresponding to this transaction data is also code-001. Since user A's operation on fund product 1 occurred earlier than their operation on stock code 600xx2, node device 1 executes the first command in its thread, requesting a lock on the thread for portfolio number code-001. The returned result is 1. Node device 1 processes the transaction data for portfolio number code-001 in its thread, and after processing, updates the transaction information for that portfolio number in the database. If node device 2 executes the first command in its thread while node device 1 is performing the above-mentioned pending task, the returned result is 0, and it needs to wait for node device 1 to release the lock. After node device 2 detects that node device 1 has completed its execution and released the lock, node device 2 retrieves the latest transaction information of portfolio number code-001 from the database. Based on the latest transaction information and user A's transaction data for purchasing stock 600xx2, node device 2 further updates the transaction information of portfolio number code-001 in the database.

[0111] In an embodiment of the present application, the update of the transaction information includes the update of the holding amount, the market value, the total size and the net value. Wherein, the new holding amount = the old holding amount + the transaction volume of the current transaction, the new market value = the old market value + the transaction volume of the current transaction x the transaction price of the current transaction, the total size = the initial size + the income from the fluctuation of the transaction price, and the net value = the total size / the initial size.

[0112] The data processing method provided by the embodiment of the present application is that each node device in the data processing system performs a data processing task based on a distributed lock. When processing a task, each thread first acquires a distributed lock of a data operation mode in the to-be-processed task by executing a first command. If the lock is successfully acquired, the corresponding data processing task is executed, and the obtained data processing result is updated to a database server, so that other threads can acquire the latest data processing result when executing subsequent tasks. If the lock acquisition fails, the thread needs to wait until the thread that has acquired the lock finishes processing and releases the lock. In this way, when multiple node devices concurrently process the same data operation mode, the multiple node devices can be sequentially executed, and the consistency of the data processing results on the multiple node devices is ensured.

[0113] The embodiment of the present application can divide the data processing device into functional modules according to the method embodiments described above. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or software functional module. It should be noted that the division of the modules in the embodiment of the present application is illustrative, and is only a logical function division. In actual implementation, another division mode can be used. The following will be described by taking the division of each functional module according to each function as an example.

[0114] Figure 5 The structure diagram of the data processing device provided by the embodiment of the present application is shown in FIG. 4. As shown in FIG. 4, the data processing device 400 provided by the embodiment of the present application includes: Figure 5

[0115] The acquisition module 401 is configured to acquire a plurality of to-be-processed tasks from a message queue of a message server. Each to-be-processed task includes operation data of a user for a target product and a data operation mode corresponding to the operation data.

[0116] The processing module 402 is configured to allocate a thread to each to-be-processed task according to the data operation mode.

[0117] The plurality of threads are configured to execute the plurality of to-be-processed tasks in parallel, and obtain a data processing result of each to-be-processed task.

[0118] ​In an embodiment of the present application, the processing module 402 is specifically configured to perform a hash transformation on the data operation mode in each of the to-be-processed tasks, to obtain a hash value corresponding to the data operation mode.

[0119] According to the hash value and a preset thread number of the node device, a thread number corresponding to each of the to-be-processed tasks is determined.

[0120] In an embodiment of the present application, the processing module 402 is specifically configured to obtain the thread number corresponding to each of the to-be-processed tasks by performing a modulo operation on the thread number with the hash value.

[0121] In an embodiment of the present application, if the operation data corresponding to the data operation mode of at least two of the to-be-processed tasks are the same, the processing module 402 is further configured to:

[0122] According to the queue order of the at least two to-be-processed tasks in the message queue, the at least two to-be-processed tasks are processed in turn in the thread corresponding to the same data operation mode.

[0123] In an embodiment of the present application, for each of the threads, before processing each of the to-be-processed tasks in the thread, the processing module 402 is further configured to:

[0124] execute a first command in the thread, the first command being used to request locking of the data operation mode of the to-be-processed task;

[0125] According to a return result of the first command, it is determined whether to execute the to-be-processed task in the thread.

[0126] In an embodiment of the present application, the processing module 402 is specifically configured to:

[0127] if the return result is a first value, execute the to-be-processed task in the thread; or

[0128] if the return result is a second value, wait for other threads to release the lock of the data operation mode.

[0129] In an embodiment of the present application, after the to-be-processed task is processed in the thread, or the thread executes the to-be-processed task beyond a timeout, the processing module 402 is further configured to:

[0130] release the lock of the data operation mode of the to-be-processed task in the thread.

[0131] In an embodiment of the present application, the obtaining module 401 is further configured to obtain a historical data processing result corresponding to the data operation mode of the to-be-processed task from a database server.

[0132] The processing module 402 is further configured to update the historical processing result according to the historical processing result and the operation data of the to-be-processed task.

[0133] The data processing apparatus provided in the embodiment can execute the technical solutions of any of the method embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0134] Figure 6 The hardware structure diagram of the electronic device provided in the embodiment is shown in FIG. 5, and the electronic device 500 provided in the embodiment includes: Figure 6

[0135] a memory 501;

[0136] a processor 502; and

[0137] a computer program;

[0138] The computer program is stored in the memory 501 and is configured to be executed by the processor 502 to implement the technical solutions of any of the method embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0139] Optionally, the memory 501 can be independent or integrated with the processor 502. When the memory 501 is independent of the processor 502, the electronic device 500 further includes a bus 503 for connecting the memory 501 and the processor 502.

[0140] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor 502 to implement the technical solutions of any of the preceding method embodiments.

[0141] The embodiment of the present application further provides a computer program product, which includes a computer program, and the computer program is executed by the processor 502 to implement the technical solutions of any of the preceding method embodiments.

[0142] The embodiment of the present application further provides a chip, which includes a memory, a processor and a computer program, the computer program is stored in the memory, and the processor runs the computer program to execute the technical solutions of any of the preceding method embodiments.

[0143] ​It should be appreciated that the above processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution or combined with hardware and software modules in the processor for execution.

[0144] The memory can include a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory, and can also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0145] The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0146] The above storage medium can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0147] An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in the electronic device.

[0148] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized by, A node device applied to a data processing system, the method comprising: obtaining a plurality of to-be-processed tasks from a message queue of a message server, each of the to-be-processed tasks comprising operation data of a user for a target product and a data operation mode corresponding to the operation data, the operation data being transaction data of the user for the target product, and the data operation mode being a portfolio mode adopted by the operation data; allocating a thread to each of the to-be-processed tasks according to the data operation mode; performing the plurality of to-be-processed tasks in parallel through the plurality of threads to obtain a data processing result of each of the to-be-processed tasks; the step of allocating a thread to each of the to-be-processed tasks according to the data operation mode comprises: performing a hash transformation on the data operation mode in each of the to-be-processed tasks to obtain a hash value corresponding to the data operation mode; determining a thread number corresponding to each of the to-be-processed tasks according to the hash value and a preset thread number of the node device; for each of the threads, before processing each of the to-be-processed tasks in the thread, the method further comprises: executing a first command in the thread, the first command being used to request locking of the data operation mode of the to-be-processed task; determining whether to execute the to-be-processed task in the thread according to a return result of the first command.

2. The method of claim 1, wherein, the step of determining a thread number corresponding to each of the to-be-processed tasks according to the hash value and a preset thread number of the node device comprises: taking a modulus of the thread number through the hash value to obtain the thread number corresponding to each of the to-be-processed tasks.

3. The method according to claim 1 or 2, characterized in that, if the operation data of at least two of the to-be-processed tasks correspond to the same data operation mode, the method further comprises: processing the at least two to-be-processed tasks in the thread corresponding to the same data operation mode in a queue order of the at least two to-be-processed tasks in the message queue.

4. The method of claim 1, wherein, the step of determining whether to execute the to-be-processed task in the thread according to a return result of the first command comprises: if the return result is a first value, executing the to-be-processed task in the thread; or if the return result is a second value, waiting for other threads to release the lock of the data operation mode.

5. The method of claim 4, wherein, after processing the to-be-processed task in the thread, or if the thread executing the to-be-processed task times out, the method further comprises: releasing the lock of the data operation mode of the to-be-processed task in the thread.

6. The method of claim 4, wherein, the step of executing the to-be-processed task in the thread comprises: obtaining a historical data processing result corresponding to the data operation mode of the to-be-processed task from a database server; updating the historical processing result according to the historical data processing result and the operation data of the to-be-processed task.

7. A data processing apparatus, characterized by, comprises: an obtaining module, configured to obtain a plurality of to-be-processed tasks from a message queue of a message server, each of the to-be-processed tasks comprising operation data of a user for a target product and a data operation mode corresponding to the operation data, the operation data being transaction data of the user for the target product, and the data operation mode being a portfolio mode adopted by the operation data; a processing module, configured to assign a thread to each of the to-be-processed tasks according to the data operation mode; execute the plurality of to-be-processed tasks in parallel through the plurality of threads to obtain a data processing result of each of the to-be-processed tasks; the processing module is specifically configured to: perform a hash transformation on the data operation mode in each of the to-be-processed tasks to obtain a hash value corresponding to the data operation mode; and determine a thread number corresponding to each of the to-be-processed tasks according to the hash value and a preset thread number of a node device; for each of the threads, the processing module is further configured to, before processing each of the to-be-processed tasks in the thread: execute a first command in the thread, the first command being used to request to lock the data operation mode of the to-be-processed task; determine, according to a return result of the first command, whether to execute the to-be-processed task in the thread.

8. An electronic device, comprising: comprise: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method in any one of claims 1 to 6. a computer program stored thereon, the computer program being executed by a processor to implement the method in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, comprise a computer program, the computer program being executed by a processor to implement the method in any one of claims 1 to 6.

10. A computer program product, characterised in that, ​

Images