A data processing method, an electronic device, a storage medium and a program product

Abstract

The embodiment of the application provides a data processing method and device, electronic equipment and storage medium, the method comprises the following steps: receiving the service message sent by the client terminal; determining the target hash value corresponding to the target transmission information according to the target transmission information in the service message; determining the scheduling strategy corresponding to the target hash value according to the target hash value and the pre-stored shared hash strategy table; and distributing the service message according to the scheduling strategy. In the embodiment of the application, the messages of different service types correspond to different processing cores respectively, the hash value corresponding to the target transmission information in the service message is calculated, the scheduling strategy corresponding to the target hash value is determined according to the pre-stored shared hash strategy table, and the service message is distributed according to the scheduling strategy. In this way, different service types are processed by different processes, and when an exception occurs in any process, other processes are not affected, thereby improving the stability of the collaborative operation of various different types of services.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a data processing method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] With the rapid development of network technology, the requirement for data security is also higher and higher. A user terminal can access data of an nginx server on each private cloud through an nginx server set in a public cloud. In the process of data access, messages of various business types are processed as a process. If an abnormality occurs in a certain business type during processing of the process by the nginx server on the public cloud, the nginx server on the public cloud causes the entire business to crash, and the stability of collaborative operation of nginx multi-protocol type business modules cannot be ensured. How to improve the stability of collaborative operation of businesses is a problem that needs to be solved at present. SUMMARY

[0003] Some embodiments of the present application aim to provide a data processing method and device, an electronic device and a storage medium. Through the technical solutions of the embodiments of the present application, a business message sent by a client terminal is received, wherein the business message at least includes target transmission information. A target hash value corresponding to the target transmission information is determined according to the target transmission information. A scheduling strategy corresponding to the target hash value is determined according to the target hash value and a pre-stored shared hash strategy table. The pre-stored shared hash strategy table at least includes preset transmission information, a preset hash value corresponding to the preset transmission information, and a scheduling strategy corresponding to the preset hash value. The business message is distributed according to the scheduling strategy. In the embodiments of the present application, messages of different business types correspond to different processing cores respectively. After receiving the business message sent by the client terminal, a hash value corresponding to target transmission information in the business message is calculated, and a scheduling strategy corresponding to the target hash value is determined according to a pre-stored shared hash strategy table. The business message is distributed according to the scheduling strategy. In this way, different processes of different business types are processed. When an abnormality occurs in any process, other processes are not affected, and the stability of collaborative operation of various types of businesses is improved.

[0004] In a first aspect, some embodiments of the present application provide a data processing method, comprising:

[0005] receiving a business message sent by a client terminal, wherein the business message at least includes target transmission information;

[0006] determining a target hash value corresponding to the target transmission information according to the target transmission information;

[0007] determine a scheduling policy corresponding to the target hash value according to the target hash value and a pre-stored shared hash policy table, wherein the pre-stored shared hash policy table at least comprises preset transmission information, a preset hash value corresponding to the preset transmission information, and a scheduling policy corresponding to the preset hash value;

[0008] distribute the service packet according to the scheduling policy.

[0009] Some embodiments of the present application correspond different service types of packets to different processing cores, calculate a hash value corresponding to target transmission information in a service packet after receiving a service packet sent by a client terminal, determine a scheduling policy corresponding to the target hash value according to a pre-stored shared hash policy table, and distribute the service packet according to the scheduling policy. In this way, different service types are processed by different processes, and when an exception occurs in any process, other processes are not affected, thereby improving the stability of the collaborative operation of various different types of services.

[0010] Optionally, the scheduling policy at least comprises the hash value, a processing core identifier corresponding to the hash value, a communication protocol, an instance process, and a number of processing cores, and the processing core identifier, the communication protocol, the instance process, and the number of processing cores are associated.

[0011] The determining of the scheduling policy corresponding to the target hash value according to the target hash value and the pre-stored shared hash policy table comprises:

[0012] comparing the target hash value with a preset hash value in the shared hash policy table;

[0013] if the target hash value matches the preset hash value, determining a target processing core identifier corresponding to the target hash value;

[0014] determining a target instance process corresponding to the target processing core identifier according to the target processing core identifier.

[0015] Some embodiments of the present application divide nginx into different service instances, isolate each service instance from each other, establish a corresponding scheduling policy for each service instance, and effectively improve the stability of the collaborative operation of multiple services of nginx.

[0016] Optionally, the method further comprises:

[0017] configuring a preset instance path for each instance process, and saving the instance process under the preset instance path.

[0018] Some embodiments of the present application configure a preset instance path for each instance process and save the instance process under the preset instance path, thereby saving storage space and call time.

[0019] Optionally, the method further comprises:

[0020] One or more of the processing core identifier in the scheduling policy, the instance process, the number of processing cores, and the association relationship are modified.

[0021] Some embodiments of the present application can deploy multiple nginx service instances on one device according to different service requirements, each nginx service instance is configured independently, and if a new service type needs to be modified or added, the service online and offline does not need to change the networking configuration of other services, the deployment is simple and flexible, and in a small high-availability server scenario where the proxy performance requirement is not very high, the operating cost can be effectively reduced.

[0022] Optionally, the method further comprises:

[0023] If the target hash value and the preset hash value do not match, the target hash value is added to the shared hash policy table.

[0024] Some embodiments of the present application divide the nginx worker process into multiple instances according to the service type, load independent nginx configuration for each service instance, an exception of one service instance does not affect the normal operation of other service instances and basic forwarding processes, each service instance can be restarted independently according to the service type, and the stability of the collaborative operation of multiple types of proxy services is improved.

[0025] Optionally, the method further comprises:

[0026] If the scheduling policy includes at least multiple processing core identifiers, a polling manner is used to sequentially schedule each instance process corresponding to the processing core identifier.

[0027] Some embodiments of the present application, in the case that the scheduling policy includes at least multiple processing core identifiers, a polling manner is used to sequentially schedule each instance process corresponding to the processing core identifier, and resources are reasonably configured.

[0028] Optionally, the method further comprises:

[0029] According to the scheduling policy, a target communication protocol corresponding to the processing core identifier is determined.

[0030] The service message is processed to obtain a processed service message.

[0031] The processed service message is stored in a lock-free queue corresponding to the target communication protocol.

[0032] The nginx worker process in some embodiment processes of the application receives a message of a corresponding protocol type, such as an http message, and processes the message, and then re-stores the processed message in a dpdk lock-free queue of the basic forwarding process for sending.

[0033] In a second aspect, some embodiments of the application provide a data processing apparatus, comprising:

[0034] The receiving module is configured to receive a service message sent by a client terminal, wherein the service message comprises at least target transmission information;

[0035] The computing module is configured to determine a target hash value corresponding to the target transmission information according to the target transmission information;

[0036] The matching module is configured to determine a scheduling strategy corresponding to the target hash value according to the target hash value and a pre-stored shared hash strategy table, wherein the pre-stored shared hash strategy table comprises at least preset transmission information, a preset hash value corresponding to the preset transmission information, and a scheduling strategy corresponding to the preset hash value;

[0037] The processing module is configured to distribute the service message according to the scheduling strategy.

[0038] Some embodiments of the application correspond different service types of messages to different processing cores, and after receiving a service message sent by a client terminal, a hash value corresponding to target transmission information in the service message is calculated, a scheduling strategy corresponding to the target hash value is determined according to a pre-stored shared hash strategy table, and the service message is distributed according to the scheduling strategy. In this way, different service types are processed by different processes, and when an exception occurs in any process, other processes are not affected, thereby improving the stability of the cooperative operation of various different types of services.

[0039] Optionally, the scheduling strategy comprises at least the hash value, a processing core identifier corresponding to the hash value, a communication protocol, an instance process, and a number of processing cores, and the processing core identifier, the communication protocol, the instance process, and the number of processing cores are associated.

[0040] The matching module is configured to:

[0041] compare the target hash value with a preset hash value in the shared hash strategy table;

[0042] If the target hash value matches the preset hash value, a target processing core identifier corresponding to the target hash value is determined.

[0043] According to the target processing core identifier, a target instance process corresponding to the target processing core identifier is determined.

[0044] Some embodiments of the present application can effectively improve the stability of the collaborative operation of multiple nginx services by dividing nginx into different service instances, isolating each service instance from each other, establishing a corresponding scheduling strategy for each service instance, and the like.

[0045] Optionally, the processing module is further configured to:

[0046] A preset instance path is configured for each instance process, and the instance process is saved under the preset instance path.

[0047] Some embodiments of the present application configure a preset instance path for each instance process and save the instance process under the preset instance path, thereby saving storage space and call time.

[0048] Optionally, the processing module is further configured to:

[0049] One or more of the processing core identifiers in the scheduling strategy, the instance processes, the number of processing cores, and the association relationship are modified.

[0050] Some embodiments of the present application can deploy multiple nginx service instances on one device according to different service requirements, each nginx service instance is configured independently, if a new service type needs to be modified or added, the service online and offline does not need to change the network configuration of other services, the deployment is simple and flexible, in a small high-availability server scenario where the proxy performance requirement is not very high, the operating cost can be effectively reduced.

[0051] Optionally, the matching module is further configured to:

[0052] If the target hash value and the preset hash value do not match, the target hash value is added to the shared hash strategy table.

[0053] Some embodiments of the present application divide nginx worker processes into multiple instances according to service types, each service instance loads an independent nginx configuration, an abnormal service instance does not affect the normal operation of other service instances and basic forwarding processes, each service instance can be restarted independently according to service types, and the stability of the collaborative operation of multiple types of proxy services is improved.

[0054] Optionally, the matching module is further configured to:

[0055] If the scheduling strategy includes at least multiple processing core identifiers, a polling manner is used to sequentially schedule each instance process corresponding to the processing core identifier.

[0056] Some embodiments of the present application adopt a polling manner to sequentially schedule each instance process corresponding to the processing core identification in the case that the scheduling strategy at least includes the plurality of processing core identifications, and reasonably configure resources.

[0057] Optionally, the processing module is configured to:

[0058] According to the scheduling strategy, a target communication protocol corresponding to the processing core identification is determined.

[0059] The service message is processed to obtain a processed service message.

[0060] The processed service message is stored in a lock-free queue corresponding to the target communication protocol.

[0061] After the nginx worker process in the instance process of some embodiments of the present application receives a message of a corresponding protocol type, the service is processed, for example, an http message, and the processed message is re-stored in the dpdk lock-free queue of the basic forwarding process for sending.

[0062] In a third aspect, some embodiments of the present application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method according to any one of the embodiments of the first aspect when executing the program.

[0063] In a fourth aspect, some embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the program is executable on a processor to implement the data processing method according to any one of the embodiments of the first aspect.

[0064] In a fifth aspect, some embodiments of the present application provide a computer program product, which includes a computer program, and the computer program is executable on a processor to implement the data processing method according to any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the following will briefly introduce the drawings needed to be used in some embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0066] Figure 1 A flowchart of a data processing method provided by some embodiments of the present application is shown in the figure.

[0067] Figure 2 A structural schematic diagram of a data processing system provided by an embodiment of the present application is shown in FIG. 1.

[0068] Figure 3 A schematic diagram of resource configuration provided by an embodiment of the present application is shown in FIG. 2.

[0069] Figure 4 A schematic diagram of generating a configuration file provided by an embodiment of the present application is shown in FIG. 3.

[0070] Figure 5 A schematic diagram of loading a configuration file provided by an embodiment of the present application is shown in FIG. 4.

[0071] Figure 6 A schematic diagram of an instance configuration provided by an embodiment of the present application is shown in FIG. 5.

[0072] Figure 7 A flowchart of policy issuing provided by an embodiment of the present application is shown in FIG. 6.

[0073] Figure 8 A flowchart of policy matching provided by an embodiment of the present application is shown in FIG. 7.

[0074] Figure 9 A flowchart of exception handling provided by an embodiment of the present application is shown in FIG. 8.

[0075] Figure 10 A structural schematic diagram of a data processing apparatus provided by an embodiment of the present application is shown in FIG. 9.

[0076] Figure 11 An electronic device provided by an embodiment of the present application is shown in FIG. 10. DETAILED DESCRIPTION

[0077] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.

[0078] It should be noted that similar reference numerals and letters refer to similar items in the following drawings, and therefore, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms “first”, “second”, etc. are only used for distinguishing description, and cannot be understood as indicating or implying relative importance.

[0079] With the rapid development of network technology, the requirement for data security is also higher and higher, a user terminal can access data of nginx servers on various private clouds through nginx servers set in a public cloud, in a data access process, various different service type messages are processed as a process, if an abnormality occurs in a certain service type during processing of the nginx servers on the public cloud, the nginx servers on the public cloud cause the whole service to crash, and the stability of collaborative operation of nginx multi-protocol type service modules cannot be ensured, how to improve the stability of collaborative operation of services is a problem that needs to be solved at present, in view of this, some embodiments of the present application provide a data processing method, the method comprises: receiving a service message sent by a client terminal, wherein the service message at least comprises target transmission information; determining a target hash value corresponding to the target transmission information according to the target transmission information; determining a scheduling strategy corresponding to the target hash value according to the target hash value and a pre-stored shared hash strategy table; wherein the pre-stored shared hash strategy table at least comprises preset transmission information, a preset hash value corresponding to the preset transmission information, and a scheduling strategy corresponding to the preset hash value; and distributing the service message according to the scheduling strategy, in the embodiments of the present application, different service type messages correspond to different processing cores respectively, after receiving the service message sent by the client terminal, a hash value corresponding to target transmission information in the service message is calculated, and a scheduling strategy corresponding to the target hash value is determined according to a pre-stored shared hash strategy table, and the service message is distributed according to the scheduling strategy, in this way, different service types are processed by different processes, when an abnormality occurs in any process, other processes are not affected, and the stability of collaborative operation of various different types of services is improved.

[0080] As Figure 1 indicated, the embodiments of the present application provide a data processing method, the method comprises:

[0081] S101, receiving a service message sent by a client terminal, wherein the service message at least comprises target transmission information;

[0082] Specifically, the server receives a service message sent by a client terminal, the service message at least comprises target transmission information, wherein the target transmission information comprises an IP address and a port number.

[0083] S102, determining a target hash value corresponding to the target transmission information according to the target transmission information;

[0084] Specifically, the server performs hash operation on the target transmission information, i.e. the IP address and the port number, to calculate a target hash value corresponding to the target transmission information.

[0085] S103, determining a scheduling strategy corresponding to the target hash value according to the target hash value and a pre-stored shared hash strategy table; wherein the pre-stored shared hash strategy table at least includes preset transmission information, a preset hash value corresponding to the preset transmission information, and a scheduling strategy corresponding to the preset hash value;

[0086] The server pre-stores a shared hash strategy table, which at least includes preset transmission information, a preset hash value corresponding to the preset transmission information, and a scheduling strategy corresponding to the preset hash value. After receiving the target hash value sent by the client terminal, the server finds the scheduling strategy corresponding to the target hash value according to the pre-stored shared hash strategy table. The scheduling strategy is which instance to use to process, specifically which CPU in which instance to use to process the service message.

[0087] S104, distributing the service message according to the scheduling strategy.

[0088] Some embodiments of the application correspond different processing cores to different service types of messages. After receiving the service message sent by the client terminal, the hash value corresponding to the target transmission information in the service message is calculated, and the scheduling strategy corresponding to the target hash value is determined according to the pre-stored shared hash strategy table. The service message is distributed according to the scheduling strategy. In this way, different processes are used to process different service types, and when an exception occurs in any process, other processes are not affected, thereby improving the stability of the cooperative operation of various different types of services.

[0089] Another embodiment of the application further supplements the data processing method provided in the above embodiment.

[0090] As shown in Figure 2 The data processing system provided by the embodiment of the application includes a client terminal, a public cloud, one or more private clouds, and an Nginx server (Nginx is a high-performance HTTP and reverse proxy web server) disposed on the public cloud side and communicating with the private cloud side. Based on the configurability of the Nginx server itself, the multi-machine room access strategy of the private cloud is configured in the Nginx server, so that the public cloud can access based on the multi-machine room access strategy of the private cloud when communicating with the private cloud. The nginx server is deployed in the public cloud and is configured with the multi-machine room access strategy in the private cloud. The nginx server on the public cloud uses different CPUs to process different service types of service messages. Multiple nginx worker processes can be established on each CPU to process multiple messages simultaneously.

[0091] Specifically, each nginx worker process is independently bound to a CPU running in user mode, different nginx worker processes can handle different protocol types, and nginx worker processes handling the same protocol type are collectively referred to as an instance. Thus, nginx worker processes handling different protocol types form a multi-instance model. Processes in different instances are isolated from each other and from memory, and an exception in a process in a certain instance does not affect the normal operation of processes in other instances. Figure 3

[0092] The specific content is as follows:

[0093] In the embodiments of the present application, a multi-core device is applied, that is, the server has multiple cores, one core (CPU) can execute a management process, configure management and process monitoring, and the other cores execute basic forwarding processes. Each core can execute different nginx worker instances, and the configurations are split and isolated according to different nginx worker instances. Instance information is saved in an instance configuration file, and the format of the instance configuration file is as follows:

[0094] [instance 1]

[0095] workers=2

[0096] core_list=2、3

[0097] [instance 2]

[0098] workers=3

[0099] core_list=4、5、6 ...

[0101] [instance n]

[0102] Workers=n

[0103] core_list=n、n+1...

[0104] Wherein: [] represents the instance name, workers represents the number of nginx worker processes, and core_list represents the CPU affinity setting of the nginx worker process.

[0105] Optionally, the scheduling strategy at least includes a hash value, a processing core identifier corresponding to the hash value, a communication protocol, an instance process, and a number of processing cores, and the processing core identifier, the communication protocol, the instance process, and the number of processing cores are associated.

[0106] ​According to the target hash value and the pre-stored shared hash strategy table, a scheduling strategy corresponding to the target hash value is determined, including:

[0107] The target hash value is compared with preset hash values in the shared hash strategy table;

[0108] If the target hash value matches the preset hash value, a target processing core identifier corresponding to the target hash value is determined;

[0109] According to the target processing core identifier, a target instance process corresponding to the target processing core identifier is determined.

[0110] As shown in the above data processing mode is executed, the policy needs to be issued: Figure 7

[0111] Step 71, the Nginx worker process is started;

[0112] Step 72, the nginx configuration file corresponding to the path is parsed;

[0113] Step 73, start listening;

[0114] Step 74, get the ip, port, process bound cpu id core identifier of the listening;

[0115] Step 75, issue to the management process shared hash strategy table, hash processing is performed on the ip, port process;

[0116] Step 76, judge whether there is a member in the hash table, that is, the target hash value;

[0117] Step 77, if there is a member in the hash table, conflict check, compare ip, port;

[0118] Step 79, if ip, port are the same, update the member, add cpu id; if ip, port are not the same, insert the hash conflict chain.

[0119] Step 78, if there is no member in the hash table, save ip, port, cpu_id and insert the hash table;

[0120] After the policy is issued, the message parsing can be performed, as shown in the following figure: Figure 8

[0121] Specifically, it includes:

[0122] Step 81, the forwarding process receives the message;

[0123] Step 82, message parsing;

[0124] ​​Step 83, hash processing is performed according to the destination ip, port, and matching is performed with the hash value in the shared hash strategy table;

[0125] Step 83, hit;

[0126] Step 84, exact matching, comparing ip, port;

[0127] Step 85, if the ip and port are the same, it is judged that multiple cpu ids exist; if multiple cpu ids exist, the Nginx worker process corresponding to the cpu id is scheduled by polling;

[0128] Step 86, if there is no multiple cpu ids, the Nginx worker process corresponding to the epu id is scheduled;

[0129] Step 87, if the ip and port are not the same or not hit, packet forwarding is performed.

[0130] Some embodiments of the application can effectively improve the stability of the cooperative operation of multiple services of nginx by dividing nginx into different service instances, isolating each service instance from each other, establishing a corresponding scheduling strategy for each service instance.

[0131] Optionally, the method further comprises:

[0132] A preset instance path is configured for each instance process, and the instance process is saved under the preset instance path.

[0133] Specifically, as shown in Figure 4 , the configuration management process parses the instance configuration to create different nginx configuration file paths for each instance, specifically:

[0134] Step 41, management process;

[0135] Step 42, parse instance configuration;

[0136] Step 43, get the number of instances and sequentially number them from 1, i.e. instance ID;

[0137] Step 44, create different nginx configuration file paths according to the number of instances and name them with instance ID, such as: / usr / local / nginx / conf / example{s instance ID);

[0138] Step 45, different instance nginx configuration files are generated according to the user configuration and installed in the corresponding path, such as creating a / usr / local / nginx / conf / examplel / directory for instance 1, and the nginx configuration is independently issued by the user according to different instances, and the configuration management process parses the user policy to generate the nginx configuration file and install it in the specified instance path.

[0139] As shown in Figure 5 , the server needs to pre-load the configuration file, specifically including:

[0140] Step 51, the Nginx worker process is started;

[0141] Step 52, parse the instance configuration to obtain the instance where the current process is bound;

[0142] Step 53, the configuration sequence in the instance configuration obtains the corresponding instance ID;

[0143] Step 54, according to the instance ID index, the corresponding nginx configuration file path is obtained,

[0144] such as: / usr / local / nginx / conf / examlel / ;

[0145] Step 55, load the nginx configuration file in the corresponding path;

[0146] When the nginx worker process is started, the nginx configuration file in the specified path is loaded according to the instance where the current process is located. Thus, each process instance independently loads the corresponding configuration file and processes different protocols.

[0147] Some embodiments of the application configure a preset instance path for each instance process, and save the instance process in the preset instance path, thereby saving storage space and call time.

[0148] Optionally, the method further comprises:

[0149] modifying one or more of the processing core identifier, the instance process, the number of processing cores, and the association relationship in the scheduling policy.

[0150] As shown in Figure 9As shown, each instance supports independent restart without affecting the nginx worker process and forwarding process in other instances, the nginx worker process in the instance reports process information to the management process when it starts, and the management process monitors the state of the nginx worker process in each instance through the heartbeat mechanism, and the abnormal exit and dead process will be restarted. Compared with the traditional nginx multi-module serial execution mode and the mode in which all workers are restarted after process exception, this method can realize flexible deployment and upgrade of different nginx business types.

[0151] Some embodiments of the present application can deploy multiple nginx business instances on one device according to different business needs, each nginx business instance is configured independently, and if a new business type needs to be modified or added, the business online and offline does not need to change the networking configuration of other business, the deployment is simple and flexible, and in a small high-availability server scenario where the proxy performance requirement is not very high, the operating cost can be effectively reduced.

[0152] Optionally, the method further comprises:

[0153] If the target hash value and the preset hash value do not match, the target hash value is added to the shared hash strategy table.

[0154] Some embodiments of the present application divide the nginx worker process into multiple instances according to the business type, each business instance loads an independent nginx configuration, and an abnormal business instance does not affect the normal operation of other business instances and the basic forwarding process, which can be independently restarted according to the business type, thereby improving the stability of the cooperative operation of multiple type proxy businesses.

[0155] Optionally, the method further comprises:

[0156] If the scheduling strategy includes at least a plurality of processing core identifiers, a polling method is used to sequentially schedule each instance process corresponding to the processing core identifier.

[0157] Specifically, the user can issue different configurations for different instances as needed, and can also adjust the number of nginx worker processes in each instance. Different nginx process instances can process different businesses, and each instance can ensure high-performance collaboration of multiple processes. For example Figure 6 As shown, the nginx worker process is configured into three different instances, for example, instance 1 processes http / https business messages, instance 2 processes ftp / tftp business messages, and instance 3 processes mail protocol business messages, and a specified number of nginx worker processes in each instance can be independently bound to a cpu for running, for example, instance 1 is processed by 4 cores, instance 2 is processed by 2 cores, and instance 3 is processed by 3 cores.

[0158] Some embodiments of the application adopt a polling manner to sequentially schedule each instance process corresponding to the processing core identifier in the case that the scheduling strategy at least includes a plurality of processing core identifiers, and reasonably configure resources.

[0159] Optionally, the service message is distributed according to the scheduling strategy, comprising:

[0160] According to the scheduling strategy, a target communication protocol corresponding to the processing core identifier is determined;

[0161] The service message is processed to obtain a processed service message;

[0162] The processed service message is stored in a lock-free queue corresponding to the target communication protocol.

[0163] Specifically, each nginx process instance on the server issues a matching strategy according to a protocol type to a shared hash table of the management process in advance, and when the nginx worker process in each instance starts socket listening, the ip, port port, and cpu id (processing core identifier) bound by the process of the target terminal to be listened to are issued to the management process, and the issued information is saved to the hash table through ip+port hashing. After the basic forwarding process receives the message, it will perform hash matching according to the destination ip and the destination port, and the message according to the saved cpu id will be distributed according to the hit strategy. If there are multiple cpu ids in the hit strategy, a polling manner will be adopted to sequentially schedule each nginx worker process. The message scheduling channel is realized based on the dpdk lock-free queue, and each process on the cpu will create a lock-free queue of its own process according to the cpu id.

[0164] After the nginx worker process in some embodiments of the application receives the message corresponding to the protocol type, the service processing is performed, such as the http message, and the processed message is re-placed into the dpdk lock-free queue of the basic forwarding process for sending.

[0165] The embodiments of the application provide a high-availability data processing method and deployment mode based on a full-autonomous data security audit product. In the full-autonomous data security audit product, the security service process and the basic forwarding process are separated, and different security service processes run in different instances, thereby improving the stability of the collaborative operation of the multiple types of security service processes and the flexibility of network deployment.

[0166] The specific processing method is as follows:

[0167] 1. On the server, instance configuration is performed, the number of nginx worker processes and cpu affinity of each instance are configured, 3 instances are configured, 2 nginx worker processes are configured for each instance, and 3 instances are used to process http, ftp and mail proxy services as an example.

[0168] 2. The management process on the server issues different nginx configuration files for each service instance, the nginx worker process in the instance starts running, and the ip, port, core id where the process is located are inserted into the shared policy hash table in the form of hash.

[0169] 3. The basic forwarding process on the server receives the service message sent by the client terminal, and matches the shared hash policy table according to the destination ip and port of the service message. If the scheduling strategy is matched, the message is put into the dpdk lock-free queue corresponding to the core id, and the corresponding nginx worker process processes it, if the scheduling strategy is not matched, the message is normally forwarded.

[0170] 4. After the nginx worker process in the instance receives the message of the corresponding protocol type, it processes the service, such as http message, and then puts the processed message into the dpdk lock-free queue of the basic forwarding process for sending.

[0171] 5. If an exception occurs in the nginx worker process during security service processing, such as the nginx worker process that processes ftp messages exits, the services that process http and mail will not be affected. After the management process detects the exited nginx worker process, it updates the shared policy matching table and restarts the nginx worker process that processes ftp services.

[0172] 6. If the service deployment needs to be changed, the corresponding type of service process instance can be started, stopped or restarted, without the need for a whole machine restart, without affecting the normal operation of other security services and basic forwarding services.

[0173] According to different service requirements, the embodiments of the application can deploy multiple nginx service instances on one device, each nginx service instance is configured independently, and service online and offline do not need to change other service networking configuration, the deployment is simple and flexible, in a small high-availability server scenario where the proxy performance requirement is not very high, the operation cost can be effectively reduced, the nginx is divided into different service instances, each service instance is isolated from each other, and the stability of the coordinated operation of multiple services of nginx can be effectively improved.

[0174] The multi-core device is used in the embodiments of the present application to replace the distributed networking, so that the complexity of network deployment and the maintenance cost can be effectively reduced.

[0175] In the embodiments of the present application, different services of nginx can be independently online and offline without changing the original service network configuration, so that the flexibility of deploying multiple types of proxy services is improved. The nginx worker process is divided into multiple instances according to service types, each service instance loads an independent nginx configuration, and an abnormal service instance does not affect other service instances and the normal operation of the basic forwarding process. The multiple types of proxy services can be independently restarted according to service types, so that the stability of the collaborative operation of the multiple types of proxy services is improved.

[0176] It should be noted that each implementable manner in the embodiments can be implemented alone or in any combination without conflict.

[0177] Another embodiment of the present application provides a data processing device for executing the data processing method provided by the above embodiments.

[0178] As shown in Figure 10 FIG. 1 is a structural schematic diagram of a data processing device provided by an embodiment of the present application. The data processing device comprises a receiving module 1001, a computing module 1002, a matching module 1003 and a processing module 1004, wherein:

[0179] The receiving module 1001 is configured to receive a service message sent by a client terminal, wherein the service message at least comprises target transmission information;

[0180] The computing module 1002 is configured to determine a target hash value corresponding to the target transmission information according to the target transmission information;

[0181] The matching module 1003 is configured to determine a scheduling strategy corresponding to the target hash value according to the target hash value and a pre-stored shared hash strategy table; wherein the pre-stored shared hash strategy table at least comprises preset transmission information, a preset hash value corresponding to the preset transmission information, and a scheduling strategy corresponding to the preset hash value;

[0182] The processing module 1004 is configured to distribute the service message according to the scheduling strategy.

[0183] As for the device in the embodiments, the specific manner in which each module performs an operation has been described in detail in the embodiments related to the method, and will not be described in detail here.

[0184] Some embodiments of the present application correspond different processing cores to different service types of packets respectively, after receiving a service packet sent by a client terminal, calculate a hash value corresponding to target transmission information in the service packet, and determine a scheduling strategy corresponding to the target hash value according to a pre-stored shared hash strategy table, and distribute the service packet according to the scheduling strategy, so that different service types are processed by different processes, and when an exception occurs in any process, other processes are not affected, thereby improving the stability of the collaborative operation of various different types of services.

[0185] Another embodiment of the present application further supplements the data processing device provided in the above embodiment.

[0186] Optionally, the scheduling strategy at least includes a hash value, a processing core identifier corresponding to the hash value, a communication protocol, an instance process, and a number of processing cores, and the processing core identifier, the communication protocol, the instance process, and the number of processing cores are associated;

[0187] The matching module is configured to:

[0188] Compare the target hash value with a preset hash value in the shared hash strategy table;

[0189] If the target hash value matches the preset hash value, a target processing core identifier corresponding to the target hash value is determined;

[0190] According to the target processing core identifier, a target instance process corresponding to the target processing core identifier is determined.

[0191] Some embodiments of the present application divide nginx into different service instances, each service instance is isolated from each other, for each service instance, a corresponding scheduling strategy is established, which can effectively improve the stability of the collaborative operation of nginx multi-service.

[0192] Optionally, the processing module is further configured to:

[0193] Configure a preset instance path for each instance process, and save the instance process under the preset instance path.

[0194] Some embodiments of the present application configure a preset instance path for each instance process, and save the instance process under the preset instance path, thereby saving storage space and call time.

[0195] Optionally, the processing module is further configured to:

[0196] Modify one or more of the processing core identifier, the instance process, the number of processing cores, and the association relationship in the scheduling strategy.

[0197] Some embodiments of the present application can deploy multiple nginx service instances on one device according to different service requirements, each nginx service instance is configured independently, if it is required to modify or add new service types, the service online and offline does not need to change the networking configuration of other services, the deployment is simple and flexible, in a small high-availability server scenario where the proxy performance requirement is not very high, the operation cost can be effectively reduced.

[0198] Optionally, the matching module is further configured to:

[0199] If the target hash value and the preset hash value do not match, the target hash value is added to the shared hash strategy table.

[0200] Some embodiments of the present application divide the nginx worker process according to the service type, each service instance loads an independent nginx configuration, an abnormal service instance does not affect other service instances and the normal operation of the basic forwarding process, each service instance can be restarted independently according to the service type, and the stability of the cooperative operation of multiple types of proxy services is improved.

[0201] Optionally, the matching module is further configured to:

[0202] If the scheduling strategy includes at least a plurality of processing core identifiers, a polling manner is used to sequentially schedule to each instance process corresponding to the processing core identifier.

[0203] Some embodiments of the present application, in the case that the scheduling strategy includes at least a plurality of processing core identifiers, a polling manner is used to sequentially schedule to each instance process corresponding to the processing core identifier, and the resources are reasonably configured.

[0204] Optionally, the processing module is configured to:

[0205] According to the scheduling strategy, a target communication protocol corresponding to the processing core identifier is determined;

[0206] The service message is processed to obtain a processed service message;

[0207] The processed service message is stored in a lock-free queue corresponding to the target communication protocol.

[0208] Some embodiments of the present application, after the nginx worker process in the process instance receives a message corresponding to a protocol type, the service is processed, such as an http message, and the processed message is re-placed in the dpdk lock-free queue of the basic forwarding process for sending.

[0209] As to the device in the embodiment, specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and will not be described in detail here.

[0210] It should be noted that each implementable manner in the present embodiment can be implemented alone or in any combination without conflict.

[0211] The present embodiment further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the operation of the method corresponding to any of the embodiments of the data processing method provided in the above embodiments.

[0212] The present embodiment further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the operation of the method corresponding to any of the embodiments of the data processing method provided in the above embodiments.

[0213] As shown in the above Figure 11 Some embodiments of the present application provide an electronic device 1100, which includes a memory 1110, a processor 1120, and a computer program stored in the memory 1110 and executable on the processor 1120, wherein the processor 1120 reads the program from the memory 1110 through a bus 1130 and executes the program to implement the method of any of the embodiments of the data processing method provided in the above embodiments.

[0214] The processor 1120 can process digital signals and can include various computing structures, such as a complex instruction set computer structure, a reduced instruction set computer structure, or a structure implementing a combination of multiple instruction sets. In some examples, the processor 1120 can be a microprocessor.

[0215] The memory 1110 can be used to store instructions executed by the processor 1120 or data related to the execution of the instructions. These instructions and / or data can include code for implementing some or all of the functions of one or more modules described in the embodiments of the present application. The processor 1120 of the embodiments of the present disclosure can be used to execute the instructions in the memory 1110 to implement the method shown in the above. The memory 1110 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memories well known to those skilled in the art.

[0216] The above is only an embodiment of the present application and does not limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0217] The above descriptions are merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0218] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus.

Claims

1. A data processing method, characterized in that, The method includes: Receive a service message sent by a client terminal, wherein the service message includes at least target transmission information; Based on the target transmission information, determine the target hash value corresponding to the target transmission information; Based on the target hash value and a pre-stored shared hash policy table, a scheduling policy corresponding to the target hash value is determined; wherein, the pre-stored shared hash policy table includes at least preset transmission information, a preset hash value corresponding to the preset transmission information, and a scheduling policy corresponding to the preset hash value. According to the scheduling strategy, the service packets are distributed. Different CPUs are used to process service packets of different service types. Multiple nginx worker processes are established on each CPU to process multiple packets simultaneously. The scheduling strategy includes at least the hash value, a processing core identifier, a communication protocol, an instance process, and a number of processing cores corresponding to the hash value, wherein the processing core identifier, the communication protocol, the instance process, and the number of processing cores are associated. The step of determining the scheduling policy corresponding to the target hash value based on the target hash value and a pre-stored shared hash policy table includes: The target hash value is compared with the preset hash value in the shared hash strategy table; If the target hash value matches the preset hash value, then the target processing core identifier corresponding to the target hash value is determined; Based on the target processing core identifier, determine the target instance process corresponding to the target processing core identifier; Configure a preset instance path for each instance process and save the instance process under the preset instance path.

2. The data processing method according to claim 1, characterized in that, The method further includes: Modify one or more of the following in the scheduling policy: the processing core identifier, the instance process, the number of processing cores, and the association relationship.

3. The data processing method according to claim 1, characterized in that, The method further includes: If the target hash value does not match the preset hash value, the target hash value is added to the shared hash strategy table.

4. The data processing method according to claim 1, characterized in that, The method further includes: If the scheduling strategy includes at least multiple processing core identifiers, then a round-robin approach is used to schedule each instance process corresponding to a processing core identifier in turn.

5. The data processing method according to claim 1, characterized in that, The step of distributing the service packets according to the scheduling policy includes: Based on the scheduling strategy, determine the target communication protocol corresponding to the processing core identifier; The service message is processed to obtain a processed service message; The processed service message is stored in a lock-free queue corresponding to the target communication protocol.

6. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the data processing method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, characterized in that, when the program is executed by a processor, it can implement the data processing method according to any one of claims 1-5.

8. A computer program product, said computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it can implement the data processing method described in any one of claims 1-5.

Images