Data processing method under high concurrency condition

Abstract

The invention discloses a data processing method under a high concurrency condition, and belongs to the technical field of computer data processing. In order to achieve maximum utilization of hardware performance in a high-concurrency environment, the method comprises the steps that a data generation module, a data adding module, a data processing module and a data deleting module are constructed by combining a standard library POSIX.Sockets and a multi-thread programming technology under a Linux operating system; the data generation module creates a server by using a standard library POSIX.Sockets in Linux, receives data transmitted by a client by using the server, and then transmits the data to the data addition module; the data adding module adds the received data into a two-dimensional array; the data processing module acquires the data received by the data increasing module by scanning a two-dimensional array, and then the data processing module interacts with system hardware to process the data; and after data processing is finished, the data deleting module is responsible for deleting data in the data processing module.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer data processing, and in particular relates to a data processing method under high concurrency conditions. Background Art

[0002] At present, the existing technologies mainly adopt the hardware data processing mode based on the serial processing architecture. In this architecture, data is processed by the processor or processing unit in a strict time order. Each processing task is performed independently and depends on the completion of the previous task. Specifically, the existing technology usually adopts the pipeline processing or time slice rotation mechanism. When the data blocks pass through the processor, they are processed one by one in a fixed queue order, and cannot be skipped or processed in parallel.

[0003] For example, in traditional server architectures, after a request enters the system, it queues up for processing according to the first-in-first-out principle. In addition, the scheduling mechanism often uses task priority sorting or fixed time slice allocation to allocate hardware resources, but these methods can only reduce the waiting time between users to a limited extent. In a multi-user concurrent scenario, each user's data processing task depends on the resources and order allocated by the system, so the user's request will queue up. In addition, the existing serial data processing model has several significant defects and problems.

[0004] First, hardware resources cannot be fully utilized, especially in high concurrency situations. When the processor or processing unit is processing the data of the previous user, the data of other users is in a waiting state, resulting in a large amount of idle or underutilized hardware resources. For example, during peak hours, the CPU utilization of the server may be only 50%, and the rest of the time it is idle.

[0005] Secondly, this serial processing method will result in a longer response time, especially when there are a large number of users. The accumulation of waiting time will reduce the response performance of the overall system and cannot meet the needs of real-time processing.

[0006] In addition, when processing large-scale data, the sequential nature of data transmission and processing increases the system bottleneck, delays the system throughput, and seriously affects the system processing efficiency and user experience. In practical applications, this delay may cause data backlogs, affecting business continuity and service quality. Summary of the invention

[0007] The problem to be solved by the present invention is to achieve maximum utilization of hardware performance in a high-concurrency environment, and propose a data processing method in a high-concurrency environment.

[0008] To achieve the above object, the present invention is implemented through the following technical solutions:

[0009] A data processing method under high concurrency conditions comprises the following steps:

[0010] S1. Combining the standard library POSIX.Sockets and multi-threaded programming technology under the Linux operating system, construct a data generation module, a data addition module, a data processing module and a data deletion module, wherein the data generation module is connected to the data addition module, the data addition module is connected to the data processing module, and the data processing module is connected to the data deletion module and the system hardware respectively;

[0011] S2. The data generation module uses the standard library POSIX.Sockets in Linux to create a server, uses the server to receive data transmitted by the client, and then transmits the data to the data addition module;

[0012] S3. The data adding module adds the received data to the two-dimensional array;

[0013] S4. The data processing module collects data by scanning the two-dimensional array to increase the data received by the module, and then the data processing module interacts with the system hardware to process the data;

[0014] S5. After the data processing is completed, the data deletion module is responsible for deleting the data in the data processing module.

[0015] Furthermore, the specific implementation method of step S2 includes the following steps:

[0016] S2.1. The data generation module uses the standard library POSIX.Sockets in Linux to create a server. The server uses the socket function to create a socket, and the returned file descriptor is used for subsequent network communication.

[0017] The specific function call is int server_fd = socket (AF_INET, SOCK_STREAM, 0), AF_INET means using the IPv4 protocol, SOCK_STREAM means using the TCP connection, parameter 0 means the system automatically selects the default protocol for the specified socket type and protocol family, server_fd is the returned file descriptor;

[0018] Then use the bind() function to bind the socket created by the socket function to the local IP address and port number;

[0019] The specific function call is bind(server_fd,(struct sockaddr*)&address,sizeof(address), server_fd is a file descriptor, address is a structure used to store IPv4 address information, where the four attributes of the structure are address family sin_family, port number sin_port, IP address structure sin_addr and padding field sin_zero;

[0020] S2.2. The data generation module receives the data in the server created in step S2.1;

[0021] The server created in step S2.1 uses the listen function to listen for connection requests from clients;

[0022] The specific function call is listen(server_fd,3), where server_fd is the file descriptor and 3 represents the maximum number of queued connections allowed to be processed simultaneously;

[0023] After receiving the client's connection request, use the accept function to establish a connection with the client;

[0024] The specific function call is int client_fd = accept(server_fd, (struct sockaddr*)&address, (socklen_t*)&addrlen), where server_fd is a file descriptor, and (socklen_t*)&addrlen is a pointer to the socklen_t type, which is used to store the size of the structure;

[0025] After the client connects successfully, the server allocates a new socket clientSocket for the client to communicate with the client;

[0026] The server uses a multi-threaded architecture to process client connection requests. Whenever the server receives a new connection request, it uses the pthread_create function in the pthread library to create a new thread to handle the connection.

[0027] The specific function call is (pthread_create(&clientThread,NULL,clientHandler,data), where clientThread is a pointer to a pthread_t variable, which is used to store the newly created thread identifier. ClientHandler is a judgment function, in which the switch function is used to judge the input instruction, and different system hardware is used according to the judgment result. Data is the transmitted data.

[0028] When the thread ends, use the pthread_detach function in the pthread library to automatically recycle resources

[0029] The specific function call is pthread_detach(clientThread), where clientThread is the thread identifier;

[0030] S2.3. The data generation module transfers the data to the data addition module through the pointer between functions;

[0031] The specific function call is handleClientMessage(data), where data is a pointer to the transmitted data.

[0032] Further, in step S2.2, the client uses the send function to send a TCP data packet to the server through the socket interface;

[0033] The specific function call is send(clientSocket,data,strlen(data),0), where clientSocket is the socket used to communicate with the server, data is the specific data to be sent, strlen(data) indicates the length of the data to be sent, and 0 indicates that no special flag is set and the default transmission mode is used;

[0034] Then, the server reads data from the receiving buffer of the TCP protocol stack through the recv function;

[0035] The specific function call is recv(clientSocket,data,sizeof(data),0). clientSocket is the socket used to communicate with the client, data is the specific data received, sizeof(data) indicates the length of the received data, and 0 indicates using the default blocking mode to receive data;

[0036] The data transmitted from the client to the server includes data transmitted by sensors, data from hard disks and SSD storage devices, audio, and video.

[0037] Furthermore, the specific implementation method of step S3 includes the following steps:

[0038] S3.1. In the data augmentation module, a two-dimensional array is created using C language. The two-dimensional array uses the malloc function to allocate memory for row pointers. For each row, malloc is used again to allocate memory for the columns of the row.

[0039] The specific function call is DataUnit**array=(DataUnit**)malloc(rows*sizeof(DataUnit*)), where DataUnit is a data structure, in which the pointer data_address represents the address of the data to be processed, the integer data process_flag is the data processing flag, the integer data scan_flag is the data scanning flag, array is the name of the array, rows represents the number of rows, and sizeof(DataUnit*) represents the size of the structure;

[0040] The specific function call is array[i] = (DataUnit*)malloc(cols*sizeof(DataUnit)), where cols represents the number of columns;

[0041] Each row of the two-dimensional array is used as a minimum data unit to store different data information, including the first data element is the address of the data to be processed, the second data element is the data processing flag, and the last data element is the data scanning flag;

[0042] There are three data elements in each data unit. The first data element is the address of the data to be processed, the second data element is the data processing flag, and the last data element is the data scanning flag. In the default state, the data processing flag and the data scanning flag are both 0, indicating an unprocessed state. The address element storing the data to be processed is also set to 0, indicating that no data is stored.

[0043] S3.2. The data addition module uses the initialization mutex function in the pthread library to dynamically initialize a mutex;

[0044] The specific function call is pthread_mutex_t mutex;

[0045] Then call the lock mutex function to lock the two-dimensional array constructed in step S3.1;

[0046] The specific function call is pthread_mutex_lock(&mutex), where mutex represents an instance of a mutex lock. Ensure that the array is a shared resource and there will be no problems when multiple threads use it at the same time;

[0047] Then, the address of the data received in step S2 is copied to the data address of the row data corresponding to the two-dimensional array using the memcpy function;

[0048] The specific function call is memcpy(&array[i][0].data_address,data,sizeof(void*)), where i,0 represents the 0th column in the i-th row, data_address represents the address of the data to be processed, and data represents the stored data pointer. The data processing flag and data scanning flag of the row data are 0 at this time, indicating an unprocessed state.

[0049] Furthermore, the specific implementation method of step S4 includes the following steps:

[0050] S4.1. In the data processing module, a thread for scanning the array is set. This thread will continuously scan the data address elements of each row in the two-dimensional array in the data adding module. When the data address element is not 0, it means that the corresponding address data is stored in the data address element. The data scanning flag bit of the row of data is modified to 1 to indicate the processed state, and then the memcpy function is used to obtain the data address in the row of data, create a pointer, and copy the data address to the pointer to use the data stored in the corresponding address;

[0051] The specific call is, memcpy(&pointer_to_data,array[i][0].data_address,sizeof(void*)), where pointer_to_data is the created pointer and data_address is the address of the stored data;

[0052] After the data processing module processes the data, it uses the system hardware to process the data in the pointer;

[0053] For example, in the server cipher machine, the encryption and decryption function of the hardware rsp20 cipher card is called to encrypt or decrypt the data. The specific function call is:

[0054] SDF_PublicKey_RSA(index, pointer_to_data, pointer_to_data_length, outdata, outdatalength), index represents the index of the encryption public key, pointer_to_data represents the incoming data to be encrypted, pointer_to_data_length represents the length of the incoming data to be encrypted, outdata is the encrypted data output by the password card, and outdatalength is the length of the encrypted data output by the password card; SDF_PrivatKey_RSA(index, pointer_to_data pointer_to_data_length, outdata, outdatalength), index represents the index of the decryption private key, pointer_to_data represents the incoming data to be decrypted, pointer_to_data_length represents the length of the incoming data to be decrypted, outdata is the encrypted data output by the password card, and outdatalength is the length of the encrypted data output by the password card;

[0055] When the system hardware has finished processing the data, the data processing flag element of the row of data is set to 1 to indicate that it has been processed; at this point, the operation on the array is completed, and the pthread_mutex_unlock function is used to unlock the mutex;

[0056] The specific function call is pthread_mutex_unlock(&mutex).

[0057] Furthermore, the specific implementation method of step S5 includes the following steps:

[0058] S5.1. When the data processing flag element of the row data is in the processed state, that is, the data processing flag element is equal to 1, the data deletion module will call the pthread_mutex_lock function to lock the array;

[0059] The specific function call is pthread_mutex_lock(&mutex), where mutex represents an instance of a mutex lock. This prevents the array from being called by multiple threads at the same time;

[0060] S5.2. The data deletion module sets the data address element, the data processing flag element and the data scanning flag element to 0, thereby deleting the row of data from the array in the data processing module, and then uses the pthread_mutex_unlock function to unlock the mutex;

[0061] The specific function call is pthread_mutex_unlock(&mutex);

[0062] Then continue to loop the data addition module, data processing module, hardware system and data deletion module. When all the data in the array are 0, that is, all the data processing is completed, use the pthread_mutex_destroy function to destroy the mutex lock;

[0063] The specific function call is pthread_mutex_destroy(&mutex).

[0064] Beneficial effects of the present invention:

[0065] The data processing method under high concurrency described in the present invention makes full use of hardware performance: by optimizing the data processing flow, shortening the processing time interval, maximizing the hardware performance, and improving the overall efficiency of the system.

[0066] The data processing method under high concurrency conditions described in the present invention improves the stability of the system: a reasonable data integration mechanism effectively avoids multi-threaded data competition problems, ensures data integrity and security, and is particularly suitable for application in high-concurrency network environments.

[0067] The data processing method under high concurrency conditions described in the present invention has strong scalability: the method has strong scalability, and the system hardware part can be replaced according to the needs of the user, and any hardware that can process data serially can be replaced.

[0068] The data processing method under high concurrency conditions described in the present invention is mainly used to make full use of the performance of hardware under high concurrency conditions. Specifically, this method integrates the to-be-processed data transmitted by multiple users through a reasonable scheduling algorithm to form a data processing module, so that the hardware can efficiently process these modules at a shorter time interval, thereby giving full play to its computing power. This method is applicable to multiple scenarios that need to process a large amount of concurrent data, including but not limited to: Server cryptographic machine: During the data encryption and decryption process, it quickly responds to requests from multiple users and improves the data processing speed. IoT device management: Enhance data exchange and control capabilities between devices through efficient data flow management. Communication equipment and 5G base stations: Improve signal processing processes to support larger numbers of connections and service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flow chart of a data processing method under high concurrency conditions according to the present invention;

[0070] Figure 2 This is a flow chart of the present invention;

[0071] Figure 3 A schematic diagram of a two-dimensional array in a data adding module of the present invention;

[0072] Figure 4 Run the result graph for the prior art;

[0073] Figure 5 This is a diagram of the operating results of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0075] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0076] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Figure 5 The detailed instructions are as follows:

[0077] Embodiment 1:

[0078] A data processing method under high concurrency conditions comprises the following steps:

[0079] S1. Combining the standard library POSIX.Sockets and multi-threaded programming technology under the Linux operating system, construct a data generation module, a data addition module, a data processing module and a data deletion module, wherein the data generation module is connected to the data addition module, the data addition module is connected to the data processing module, and the data processing module is connected to the data deletion module and the system hardware respectively;

[0080] S2. The data generation module uses the standard library POSIX.Sockets in Linux to create a server, uses the server to receive data transmitted by the client, and then transmits the data to the data addition module;

[0081] Furthermore, the specific implementation method of step S2 includes the following steps:

[0082] S2.1. The data generation module uses the standard library POSIX.Sockets in Linux to create a server. The server uses the socket function to create a socket, and the returned file descriptor is used for subsequent network communication.

[0083] The specific function call is int server_fd = socket (AF_INET, SOCK_STREAM, 0), AF_INET means using the IPv4 protocol, SOCK_STREAM means using the TCP connection, parameter 0 means the system automatically selects the default protocol for the specified socket type and protocol family, server_fd is the returned file descriptor;

[0084] Then use the bind() function to bind the socket created by the socket function to the local IP address and port number;

[0085] The specific function call is bind(server_fd,(struct sockaddr*)&address,sizeof(address), server_fd is a file descriptor, address is a structure used to store IPv4 address information, where the four attributes of the structure are address family sin_family, port number sin_port, IP address structure sin_addr and padding field sin_zero;

[0086] S2.2. The data generation module receives the data in the server created in step S2.1;

[0087] The server created in step S2.1 uses the listen function to listen for connection requests from clients;

[0088] The specific function call is listen(server_fd,3), where server_fd is the file descriptor and 3 represents the maximum number of queued connections allowed to be processed simultaneously;

[0089] After receiving the client's connection request, use the accept function to establish a connection with the client;

[0090] The specific function call is int client_fd = accept(server_fd, (struct sockaddr*)&address, (socklen_t*)&addrlen), where server_fd is a file descriptor, and (socklen_t*)&addrlen is a pointer to the socklen_t type, which is used to store the size of the structure;

[0091] After the client connects successfully, the server allocates a new socket clientSocket for the client to communicate with the client;

[0092] The server uses a multi-threaded architecture to process client connection requests. Whenever the server receives a new connection request, it uses the pthread_create function in the pthread library to create a new thread to handle the connection.

[0093] The specific function call is (pthread_create(&clientThread,NULL,clientHandler,data), where clientThread is a pointer to a pthread_t variable, which is used to store the newly created thread identifier. ClientHandler is a judgment function, in which the switch function is used to judge the input instruction, and different system hardware is used according to the judgment result. Data is the transmitted data.

[0094] When the thread ends, use the pthread_detach function in the pthread library to automatically recycle resources

[0095] The specific function call is pthread_detach(clientThread), where clientThread is the thread identifier;

[0096] Further, in step S2.2, the client uses the send function to send a TCP data packet to the server through the socket interface;

[0097] The specific function call is send(clientSocket,data,strlen(data),0), where clientSocket is the socket used to communicate with the server, data is the specific data to be sent, strlen(data) indicates the length of the data to be sent, and 0 indicates that no special flag is set and the default transmission mode is used;

[0098] Then, the server reads data from the receiving buffer of the TCP protocol stack through the recv function;

[0099] The specific function call is recv(clientSocket,data,sizeof(data),0). clientSocket is the socket used to communicate with the client, data is the specific data received, sizeof(data) indicates the length of the received data, and 0 indicates using the default blocking mode to receive data;

[0100] The data transmitted from the client to the server includes data transmitted by sensors, data from hard disks and SSD storage devices, audio, and video.

[0101] S2.3. The data generation module transfers the data to the data addition module through the pointer between functions;

[0102] The specific function call is handleClientMessage(data), where data is a pointer to the transmitted data.

[0103] S3. The data adding module adds the received data to the two-dimensional array;

[0104] Furthermore, the specific implementation method of step S3 includes the following steps:

[0105] S3.1. In the data augmentation module, a two-dimensional array is created using C language. The two-dimensional array uses the malloc function to allocate memory for row pointers. For each row, malloc is used again to allocate memory for the columns of the row.

[0106] The specific function call is DataUnit**array=(DataUnit**)malloc(rows*sizeof(DataUnit*)), where DataUnit is a data structure, in which the pointer data_address represents the address of the data to be processed, the integer data process_flag is the data processing flag, the integer data scan_flag is the data scanning flag, array is the name of the array, rows represents the number of rows, and sizeof(DataUnit*) represents the size of the structure;

[0107] The specific function call is array[i] = (DataUnit*)malloc(cols*sizeof(DataUnit)), where cols represents the number of columns;

[0108] Each row of the two-dimensional array is used as a minimum data unit to store different data information, including the first data element is the address of the data to be processed, the second data element is the data processing flag, and the last data element is the data scanning flag;

[0109] like Figure 3 As shown. Each data unit contains three data elements. The first data element is the address of the data to be processed, the second data element is the data processing flag, and the last data element is the data scanning flag. In the default state, the data processing flag and the data scanning flag are both 0, indicating an unprocessed state. The address element storing the data to be processed is also set to 0, indicating that no data is stored.

[0110] S3.2. The data addition module uses the initialization mutex function in the pthread library to dynamically initialize a mutex;

[0111] The specific function call is pthread_mutex_t mutex;

[0112] Then call the lock mutex function to lock the two-dimensional array constructed in step S3.1;

[0113] The specific function call is pthread_mutex_lock(&mutex), where mutex represents an instance of a mutex lock. Ensure that the array is a shared resource and there will be no problems when multiple threads use it at the same time;

[0114] Then, the address of the data received in step S2 is copied to the data address of the row data corresponding to the two-dimensional array using the memcpy function;

[0115] The specific function call is memcpy(&array[i][0].data_address,data,sizeof(void*)), where i,0 represents the 0th column in the i-th row, data_address represents the address of the data to be processed, and data represents the stored data pointer. The data processing flag and data scanning flag of the row data are 0 at this time, indicating an unprocessed state.

[0116] S4. The data processing module collects data by scanning the two-dimensional array to increase the data received by the module, and then the data processing module interacts with the system hardware to process the data;

[0117] Furthermore, the specific implementation method of step S4 includes the following steps:

[0118] S4.1. In the data processing module, a thread for scanning the array is set. This thread will continuously scan the data address elements of each row in the two-dimensional array in the data adding module. When the data address element is not 0, it means that the corresponding address data is stored in the data address element. The data scanning flag bit of the row of data is modified to 1 to indicate the processed state, and then the memcpy function is used to obtain the data address in the row of data, create a pointer, and copy the data address to the pointer to use the data stored in the corresponding address;

[0119] The specific call is, memcpy(&pointer_to_data,array[i][0].data_address,sizeof(void*)), where pointer_to_data is the created pointer and data_address is the address of the stored data;

[0120] After the data processing module processes the data, it uses the system hardware to process the data in the pointer;

[0121] For example, in the server cipher machine, the encryption and decryption function of the hardware rsp20 cipher card is called to encrypt or decrypt the data. The specific function call is:

[0122] SDF_PublicKey_RSA(index, pointer_to_data, pointer_to_data_length, outdata, outdatalength), index represents the index of the encryption public key, pointer_to_data represents the incoming data to be encrypted, pointer_to_data_length represents the length of the incoming data to be encrypted, outdata is the encrypted data output by the password card, and outdatalength is the length of the encrypted data output by the password card; SDF_PrivatKey_RSA(index, pointer_to_data pointer_to_data_length, outdata, outdatalength), index represents the index of the decryption private key, pointer_to_data represents the incoming data to be decrypted, pointer_to_data_length represents the length of the incoming data to be decrypted, outdata is the encrypted data output by the password card, and outdatalength is the length of the encrypted data output by the password card;

[0123] When the system hardware has finished processing the data, the data processing flag element of the row of data is set to 1 to indicate that it has been processed; at this point, the operation on the array is completed, and the pthread_mutex_unlock function is used to unlock the mutex;

[0124] The specific function call is pthread_mutex_unlock(&mutex).

[0125] S5. After the data processing is completed, the data deletion module is responsible for deleting the data in the data processing module;

[0126] Furthermore, the specific implementation method of step S5 includes the following steps:

[0127] S5.1. When the data processing flag element of the row data is in the processed state, that is, the data processing flag element is equal to 1, the data deletion module will call the pthread_mutex_lock function to lock the array;

[0128] The specific function call is pthread_mutex_lock(&mutex), where mutex represents an instance of a mutex lock. This prevents the array from being called by multiple threads at the same time;

[0129] S5.2. The data deletion module sets the data address element, the data processing flag element, and the data scan flag element to 0, thereby deleting the row of data from the array in the data processing module, and then uses the pthread_mutex_unlock function to unlock the mutex.

[0130] The specific function call is pthread_mutex_unlock(&mutex);

[0131] Then continue to loop the data addition module, data processing module, hardware system and data deletion module. When all the data in the array are 0, that is, all the data processing is completed, use the pthread_mutex_destroy function to destroy the mutex lock;

[0132] The specific function call is pthread_mutex_destroy(&mutex).

[0133] A 32-thread concurrent test was conducted on the same cryptographic card hardware, with each thread performing 10,000 operations to test the performance of the following three tasks: generating a 256-bit SM2 key, signing 256-bit data with SM2, and verifying SM2 signatures. Figure 4 and Figure 5 As shown in Table 1:

[0134] Table 1

[0135] Generate Keys SM2 Signature SM2 signature verification Prior art 110501 pairs / second 140642 times / second 106452 times / second This embodiment 163568 pairs / second 199091 times / second 108095 times / second

[0136] As can be seen from Table 1, the experimental data shows that the SM2 key generation rate increased from 110501 pairs / second to 163568 pairs / second, the SM2 signature rate increased from 140642 times / second to 199091 times / second, and the SM2 signature verification rate increased slightly from 106452 times / second to 108095 times / second (the SM2 signature verification rate of this password card is close to its hardware peak).

[0137] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0138] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A data processing method under high concurrency conditions, characterized in that: The steps include: S1. Combining the standard library POSIX.Sockets and multi-threaded programming technology under the Linux operating system, construct a data generation module, a data addition module, a data processing module and a data deletion module, wherein the data generation module is connected to the data addition module, the data addition module is connected to the data processing module, and the data processing module is connected to the data deletion module and the system hardware respectively; S2. The data generation module uses the standard library POSIX.Sockets in Linux to create a server, uses the server to receive data transmitted by the client, and then transmits the data to the data addition module; S3. The data adding module adds the received data to the two-dimensional array; S4. The data processing module collects data by scanning the two-dimensional array to increase the data received by the module, and then the data processing module interacts with the system hardware to process the data; S5. After the data processing is completed, the data deletion module is responsible for deleting the data in the data processing module.

2. A data processing method under high concurrency conditions according to claim 1, characterized in that: The specific implementation method of step S2 includes the following steps: S2.

1. The data generation module uses the standard library POSIX.Sockets in Linux to create a server. The server uses the socket function to create a socket, and the returned file descriptor is used for subsequent network communication. Then use the bind() function to bind the socket created by the socket function to the local IP address and port number; S2.

2. The data generation module receives the data in the server created in step S2.1; The server created in step S2.1 uses the listen function to listen for connection requests from clients; After receiving the client's connection request, use the accept function to establish a connection with the client; After the client connects successfully, the server allocates a new socket clientSocket for the client to communicate with the client; The server uses a multi-threaded architecture to process client connection requests. Whenever the server receives a new connection request, it uses the pthread_create function in the pthread library to create a new thread to handle the connection. When the thread ends, use the pthread_detach function in the pthread library to automatically recycle resources S2.

3. The data generation module transfers the data to the data addition module through pointers between functions.

3. A data processing method under high concurrency conditions according to claim 2, characterized in that: In step S2.2, the client uses the send function to send a TCP data packet to the server through the socket interface; Then, the server reads data from the receiving buffer of the TCP protocol stack through the recv function; The data transmitted from the client to the server includes data transmitted by sensors, data from hard disks and SSD storage devices, audio, and video.

4. A data processing method under high concurrency conditions according to claim 3, characterized in that: The specific implementation method of step S3 includes the following steps: S3.

1. In the data augmentation module, a two-dimensional array is created using C language. The two-dimensional array uses the malloc function to allocate memory for row pointers. For each row, malloc is used again to allocate memory for the columns of the row. Each row of the two-dimensional array is used as a minimum data unit to store different data information, including the first data element is the address of the data to be processed, the second data element is the data processing flag, and the last data element is the data scanning flag; S3.

2. The data addition module uses the initialization mutex function in the pthread library to dynamically initialize a mutex; Then call the lock mutex function to lock the two-dimensional array constructed in step S3.1; Then, the address of the data received in step S2 is copied to the data address of the row data corresponding to the two-dimensional array using the memcpy function.

5. A data processing method under high concurrency conditions according to claim 4, characterized in that: The specific implementation method of step S4 includes the following steps: S4.

1. In the data processing module, a thread for scanning the array is set. This thread will continuously scan the data address elements of each row in the two-dimensional array in the data adding module. When the data address element is not 0, it means that the corresponding address data is stored in the data address element. The data scanning flag bit of the row of data is modified to 1 to indicate the processed state, and then the memcpy function is used to obtain the data address in the row of data, create a pointer, and copy the data address to the pointer to use the data stored in the corresponding address; After the data processing module processes the data, it uses the system hardware to process the data in the pointer; When the system hardware finishes processing the data, the data processing flag element of the row of data is set to 1 to indicate that it has been processed; at this point, the operation on the array is completed, and the pthread_mutex_unlock function is used to unlock the mutex.

6. A data processing method under high concurrency conditions according to claim 5, characterized in that: The specific implementation method of step S5 includes the following steps: S5.

1. When the data processing flag element of the row data is in the processed state, that is, the data processing flag element is equal to 1, the data deletion module will call the pthread_mutex_lock function to lock the array; S5.

2. The data deletion module sets the data address element, the data processing flag element, and the data scan flag element to 0, thereby deleting the row of data from the array in the data processing module, and then uses the pthread_mutex_unlock function to unlock the mutex. Then continue to loop the data addition module, data processing module, hardware system and data deletion module. When all the data in the array are 0, that is, all the data processing is completed, use the pthread_mutex_destroy function to destroy the mutex lock.

Images