Mass data concurrency processing method based on receiving and processing separation

Abstract

The invention discloses a mass data concurrency processing method based on receiving and processing separation. The mass data concurrency processing method includes that when server progress is started, at least one data processing thread and one data receiving thread are started, and a section of a dynamic storage area is developed simultaneously; the data receiving thread monitors and receives client data through a socket and puts the data into the dynamic memory area; the data processing thread polls the dynamic memory area, and if data exist in the dynamic memory area, the data processing thread sequentially extracts the data from the dynamic memory area and judges the data; if the data are based on the last data pack, the data are packed, and whether the data are the last data based on the last pack is judged, otherwise data are continually extracted from the dynamic memory area until the extracted data are the last data pack based on the same operation, and the packing is finished; and the complete data based on the same operation are subjected to follow-up business processing. The mass data concurrency processing method can improve the efficiency of volume data transmission, data-intensive computing and parallel computing, and improve the system stability.

Description

technical field [0001] The invention relates to a data processing method, in particular to a massive data concurrent processing method based on the separation of receiving and processing, and belongs to the field of computer technology and communication. Background technique [0002] In the development of communication systems based on TCP / IP, the development of programs based on SOCKET technology must be involved. Development based on SOCKET technology involves sending data from the client and receiving data from the server. The implementation process of the server based on SOCKET technology is as follows: [0003] 1. Load the socket [0004] 2. Create a listening socket [0005] 3. Bind the socket [0006] 4. Listening socket [0007] 5. Receive client data and process client-related requests [0008] At point 5, when the server receives the client data, it generally generates a thread and passes the received data to this new thread for processing. [0009] Since the ...

AI Technical Summary

Problems solved by technology

[0009] Since the default data of a soctet communication based on windows, linux or Unix system is 8K bytes, in a complete business process, if the data transmitted from the client to the server reaches several megabytes, tens of megabytes or more 100M, then the client will send the data multiple times in order, and the server will receive the data sent by the client based on the same business multiple times, and then group the transmitted data to perform Processing, according to the server receives the client data and then generates a thread to process the passed data, so that the server will generate a large number of threads for processing data in an instant, and at the same time, according to the multi-thread mechanism, the server will generate a large number of threads, but the execution of these threads is not executed in the order in which the sub-threads are generated by the main thread, and the sub-threads generated at the same time are not executed in sequence, but are executed in parallel according to the scheduling disorder, so that the data based on the same business is in the service There will be problems with grouping at the end

Images