Data processing method and system

Abstract

The embodiments of the present specification provide a data processing method and system, wherein the data processing method is applied to a multi-link programming device configured on a server, including: receiving a business data stream pushed by a business data source, and parsing target user data from the business data stream; processing the target user data according to preset data screening rules to obtain data to be distributed; obtaining load information of a data processing cluster corresponding to the server, and a distribution strategy matching the data to be distributed; selecting a data processing node in the data processing cluster based on the distribution strategy and the load information, and transmitting the data to be distributed to the data processing node for processing via a communication link matching the data processing node.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of communication technology, and in particular, to a data processing method and system. Background Art

[0002] With the development of Internet technology, more and more businesses are becoming online, and with the gradual development of online businesses, the business scope they cover and the user groups that need to be served will gradually increase. In order to provide users with stable services, the server can customize the data distribution strategy by configuring the FPGA device to enable the FPGA device to transfer all business-related data to the CPU for processing, so that the software running on the CPU can analyze the data according to the business setting requirements and allocate the relevant contract data to a specific CPU core for subsequent processing. However, when distributing data to the CPU core, most of the existing technologies use a single DMA link to achieve data transmission. This processing method has a serious data processing bottleneck, especially in service scenarios with large data volumes, which will limit the CPU processing speed and affect business operations. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, an embodiment of this specification provides a data processing method. One or more embodiments of this specification also relate to a data processing system, a data processing device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, there is provided a data processing method, which is applied to a multi-link programming device configured with a server, comprising:

[0005] Receive the business data stream pushed by the business data source, and parse the target user data from the business data stream;

[0006] Processing the target user data according to preset data screening rules to obtain data to be distributed;

[0007] Obtaining load information of a data processing cluster corresponding to the server and a distribution strategy matching the data to be distributed;

[0008] A data processing node is selected in the data processing cluster based on the distribution strategy and the load information, and the data to be distributed is transmitted to the data processing node for processing via a communication link matched by the data processing node.

[0009] According to a second aspect of an embodiment of this specification, a data processing system is provided, the data processing system comprising a multi-link programming device and a data processing cluster, wherein the data processing cluster comprises a plurality of initial data processing nodes, including:

[0010] The multi-link programming device is used to receive the business data stream pushed by the business data source, and parse the target user data from the business data stream; process the target user data according to the preset data screening rules to obtain the data to be distributed; obtain the load information of the data processing cluster and the distribution strategy matching the data to be distributed; select a data processing node in the data processing cluster based on the distribution strategy and the load information, and transmit the data to be distributed to the data processing node through the communication link matching the data processing node;

[0011] The data processing node is used to receive the data to be distributed and process the data to be distributed according to preset business rules.

[0012] According to a third aspect of the embodiments of this specification, there is provided a data processing method, which is applied to a multi-link programming device configured in a server, comprising:

[0013] A receiving module is configured to receive a business data stream pushed by a business data source, and parse the business data stream to obtain target user data;

[0014] A processing module, configured to process the target user data according to a preset data screening rule to obtain data to be distributed;

[0015] An acquisition module is configured to acquire load information of a data processing cluster corresponding to the server and a distribution strategy matching the data to be distributed;

[0016] The selection module is configured to select a data processing node in the data processing cluster based on the distribution strategy and the load information, and transmit the data to be distributed to the data processing node for processing through a communication link matched by the data processing node.

[0017] According to a fourth aspect of an embodiment of this specification, a computing device is provided, including:

[0018] Memory and processor;

[0019] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above data processing method are implemented.

[0020] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0021] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0022] The data processing method provided in this embodiment can configure a multi-link programming device on the server in order to achieve rapid response to business changes and effectively improve the overall business data processing efficiency. The data distribution node can parse the target user data from the business data stream after the multi-link programming device receives the business data stream pushed by the business data source. For the target user data, the target user data can be first processed according to the preset data screening rules to screen out the data to be distributed in the target user data that needs to be distributed to the data processing node for processing according to the processing results. On this basis, in order to improve the data processing efficiency, the load information of the data processing cluster corresponding to the server and the distribution strategy matched by the data to be distributed can be obtained to make a distribution decision before processing the data to be distributed, thereby realizing the selection of the data processing node in the data processing cluster based on the distribution strategy and load information. On this basis, the data to be distributed can be transmitted to the data processing node for processing through the communication link matched by the data processing node, so as to balance the load in a way that combines the distribution strategy and load information to avoid overloading of some data processing nodes, thereby effectively improving the processing efficiency of the overall data processing cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of a data processing method provided by an embodiment of this specification;

[0024] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;

[0025] Figure 3 It is a structural diagram of a server in a data processing method provided by an embodiment of this specification;

[0026] Figure 4 is a processing flow chart of a data processing method provided by an embodiment of this specification;

[0027] Figure 5 is a structural diagram of a data processing system provided by an embodiment of this specification;

[0028] Figure 6is a structural schematic diagram of a data processing device provided by an embodiment of this specification;

[0029] Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0030] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0031] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0033] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0034] First, the terms involved in one or more embodiments of this specification are explained.

[0035] DMA: (Direct Memory Access) is a memory access technology that allows data to be directly exchanged between peripherals (such as hard disks, graphics cards, network interface cards, etc.) and main memory (RAM) without CPU intervention. DMA technology is implemented through a dedicated DMA controller that can take over the CPU's bus control, thereby performing high-speed data transmission between the CPU and the DMA controller.

[0036] FPGA: (Field-Programmable Gate Array) is a semi-custom circuit that allows users to configure (program) it after manufacturing to implement specific digital logic functions. FPGA consists of a large number of configurable logic blocks (CLBs), input / output blocks (IOBs), and interconnection resources, which can be programmed according to user needs to create complex digital systems.

[0037] UDP: (User Datagram Protocol) is a connectionless transport layer protocol that provides a simple transaction-oriented information transmission service above the IP network layer. Unlike TCP (Transmission Control Protocol), UDP does not provide data packet grouping, assembly, and cannot guarantee the order of data packets.

[0038] TCP: (Transmission Control Protocol) is a connection-oriented, reliable, byte-stream-based transport layer communication protocol. It is part of the Internet Protocol Suite, located above the network layer and below the application layer. TCP provides a reliable data transmission service for applications by ensuring the correct sending and receiving of data packets.

[0039] In this specification, a data processing method is provided. This specification also relates to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0040] See also Figure 1As shown in the schematic diagram, the data processing method provided in this embodiment can configure a multi-link programming device on the server in order to achieve rapid response to business changes and effectively improve the overall business data processing efficiency. At the data distribution node, after the multi-link programming device receives the business data stream pushed by the business data source, the target user data can be parsed from the business data stream; for the target user data, the target user data can be first processed according to the preset data screening rules to screen out the data to be distributed in the target user data that needs to be distributed to the data processing node for processing according to the processing results; on this basis, in order to improve the data processing efficiency, the load information of the data processing cluster corresponding to the server and the distribution strategy matched by the data to be distributed can be obtained to make a distribution decision before processing the data to be distributed, thereby realizing the selection of the data processing node in the data processing cluster based on the distribution strategy and load information, and on this basis, the data to be distributed can be transmitted to the data processing node for processing through the communication link matched by the data processing node, so as to realize load balancing in a way of combining the distribution strategy and load information, and avoid overloading of some data processing nodes, thereby effectively improving the processing efficiency of the overall data processing cluster.

[0041] See also Figure 2 , Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown. The method is applied to a multi-link programming device configured in a server and specifically includes the following steps.

[0042] Step S202: receiving a service data stream pushed by a service data source, and parsing the service data stream to obtain target user data.

[0043] The data processing method provided in this embodiment can be applied to any scenario with a large amount of data distribution and processing, such as the distribution and processing of shopping data, the distribution and processing of transaction data, the distribution and processing of browsing data, the distribution and processing of live broadcast data, etc.; it is used to realize that business-related data can be dynamically allocated to reasonable data processing nodes for processing through the data processing method provided in this embodiment, thereby improving data processing efficiency, so that the data processing nodes can further aggregate and analyze the data, and execute business scenario association strategies, etc. This embodiment takes the application of the data processing method in the transaction scenario as an example for explanation, and the description of other scenarios can refer to the same or corresponding description in this embodiment, and this embodiment will not be repeated in detail here.

[0044] Specifically, the server refers to the server running the target business, and the multi-link programming device refers to the FPAG device that provides multiple DMA links, which is used to achieve that when distributing data, the DMA link matching the data processing node can be selected as needed to complete the data distribution processing. Correspondingly, the business data source refers to the data source that generates business data in the target business; accordingly, the business data flow is the data flow composed of business data submitted from the business data source to the server, which contains global business data related to the target business. Correspondingly, the target user data refers to the business data in the business data flow that meets the current needs and is distributed to the data processing node and needs to be processed by the specified software, such as transaction data within a set time interval that needs to be submitted to the data processing node for aggregation.

[0045] Based on this, in order to be able to quickly respond to business changes and effectively improve the overall business data processing efficiency, a multi-link programming device can be configured on the server to achieve data distribution nodes. After the multi-link programming device receives the business data stream pushed by the business data source, it can parse the target user data from the business data stream; for the target user data, the target user data can be first processed according to the preset data screening rules to filter out the target user data that needs to be distributed to the data processing node for processing according to the processing results; on this basis, in order to improve data processing efficiency, the load information of the data processing cluster corresponding to the server and the distribution strategy matching the data to be distributed can be obtained to make distribution decisions before processing the data to be distributed, thereby realizing the selection of data processing nodes in the data processing cluster based on the distribution strategy and load information. On this basis, the data to be distributed can be transmitted to the data processing node for processing through the communication link matching the data processing node.

[0046] Furthermore, in order to enable the multi-link programming device to parse the data to be distributed to the data processing node for processing after receiving the service data stream, it is necessary to configure the data distribution rules for the multi-link programming device in advance. In this embodiment, the specific implementation method is as follows:

[0047] During the initialization phase of the multi-link programming device, a device initialization task is executed, and the candidate communication links associated with the multi-link programming device are initialized; when the initialization task is completed and the candidate communication links are initialized, data distribution rules and initial data screening rules are determined in response to a rule configuration request; and data screening rules are configured according to the data distribution rules and the initial data screening rules.

[0048] Specifically, the initialization task specifically refers to the task of performing initialization operations on the multi-link programming device, which is used to adjust each module in the multi-link programming device to the default state, so that during the data distribution stage, each module can be supported to obtain real information for distribution and processing operations. Correspondingly, the initialization of the candidate communication link specifically refers to ensuring the usefulness of the communication link so that it can be selected and used for data transmission during the data distribution stage; wherein the candidate communication link is a plurality of DMA links provided by the multi-link programming device. Correspondingly, the data distribution rule specifically refers to the contract data distribution rule configured according to the specific needs of the business party, which is used to specify the data receiving channel, and the contract information to which the data that can be received by the channel belongs, and can also specify the contract information of the data that is not received. Correspondingly, the initial data screening rule specifically refers to the rule for screening the data that needs to be processed from the business data stream, which can be set according to actual needs, and this embodiment does not make any limitation here.

[0049] Based on this, in order to support multi-link programming devices to achieve multi-channel distribution and achieve load balancing of data processing nodes, data screening rules can be configured in advance for multi-link programming devices; specifically, the device initialization task can be executed during the initialization phase of the multi-link programming device, and the candidate communication links associated with the multi-link programming device can be initialized; on this basis, if the initialization task is completed and the candidate communication links are initialized, the data distribution rules and initial data screening rules can be determined in response to the rule configuration request; the data screening rules are configured according to the data distribution rules and the initial data screening rules to support the use of data screening rules to filter out the data to be distributed after receiving the business data stream submitted by the business data source during the business execution phase, so as to dynamically allocate the data to a reasonable data processing node for processing, so as to improve the data processing efficiency while avoiding the problem of excessive resource occupation of the data processing node.

[0050] In actual applications, when determining the data distribution rule and the initial data screening rule in response to the rule configuration request, the data distribution rule can be set as follows:

[0051] channel0]

[0052] insts=all

[0053] options=recv

[0054] [channel1]

[0055] insts=zn,ag,sc

[0056] options=norecv

[0057] …

[0058] Among them, it includes channel 0 and channel 1. Channel 0 receives all contracts and all options; channel 1 receives zn, ag, sc contracts, but does not receive options. By setting the above data distribution rules, it is possible to filter out the data that needs to be distributed for subsequent processing during the intelligent data distribution stage. When configuring the initial data screening rules, the business party can set the specified data to be received, and all other data can be discarded; during the specific implementation, it can be set according to actual needs, and this embodiment does not make any restrictions here.

[0059] In summary, by initializing the multi-link programming device and its corresponding candidate communication links, it is possible to configure data distribution rules and initial data screening rules on the multi-link programming device according to needs to ensure that at the data processing node, the multi-link programming device can reasonably distribute data to different data processing nodes for subsequent business processing.

[0060] Furthermore, after receiving the business data stream submitted by the business data source, considering that the business data stream contains many data types and not all data need to be processed later, the data type can be determined by parsing the data header, and then the user data type is selected as the target user data for subsequent distribution processing. In this embodiment, the specific implementation method is as follows:

[0061] Parse the data header of the business data stream, and determine the data type corresponding to the business data in the business data stream according to the parsing result; if it is determined according to the data type that the business data is a user data type and meets the preset distribution conditions, use the business data as target user data.

[0062] Specifically, the data header specifically refers to the message header of the service data stream, and correspondingly, the data type specifically refers to the type of service data determined after parsing the data header of the service data stream, which includes TCP data type and UDP data type, and the user data type is the UDP data type. Correspondingly, the preset distribution condition specifically refers to the condition for determining whether the service data of the user data type belongs to the software-specified UDP data after determining the service data, which can be set according to the software data processing requirements.

[0063] Based on this, after receiving the business data stream submitted by the business data source, in order to filter out the data that needs to be processed and distribute it to the data processing node for subsequent processing, the data header of the business data stream can be parsed first, and the data type corresponding to the business data in the business data stream can be determined according to the parsing result; at this time, the business data can be filtered according to the data type, and when the business data is determined to be a user data type according to the data type and meets the preset distribution conditions, the business data can be used as the target user data, so that the target user data can be intelligently distributed later.

[0064] For example, in a securities trading scenario, when the FPGA device configured by the server receives a real-time data stream from a market data source, it can parse the header information of the real-time data stream; if it is determined to be a TCP protocol and a designated port configured by the software according to the parsing result, the received TCP real-time data can be uploaded to the CPU for processing intact. If it is determined to be a UDP protocol and UDP data specified by the software according to the parsing result, it can be used as the target user data at this time, and the rest of the data can be filtered out so that the target user data can be intelligently distributed later.

[0065] In summary, by parsing the business data stream and filtering out business data of specified data types as target user data, it is possible to filter out data that meets business processing requirements from a large amount of data for processing, thereby reducing the subsequent data transmission volume and alleviating bandwidth pressure, thereby effectively improving data processing efficiency.

[0066] Step S204: Process the target user data according to preset data screening rules to obtain data to be distributed.

[0067] Specifically, after obtaining the target user data as mentioned above, further, considering that the data to be distributed by the multi-link programming device is set according to business needs, before distributing the data, it is necessary to process the target user data according to preset data screening rules, so as to realize the processing of the target user data into a specified data format, and support subsequent distribution processing, so as to obtain the data to be distributed for subsequent use.

[0068] The preset data screening rules specifically refer to rules for preprocessing, processing and converting target user data, wherein preprocessing is used to parse target user data, and processing and conversion are used to convert the parsed data into a data format / type that can be processed by the data processing node. Accordingly, the data to be distributed specifically refers to a data type / format that conforms to the data processing node processing.

[0069] Furthermore, when the target user data is processed according to the preset data screening rules, it can be completed by decompression, parsing and conversion. In this embodiment, the specific implementation is as follows:

[0070] The target user data is decompressed according to preset data screening rules to obtain decompressed user data; the decompressed user data is parsed and converted to obtain initial data to be distributed; the initial data to be distributed is processed according to the distribution rules and screening rules in the data screening rules, and the data to be distributed is generated according to the processing results.

[0071] Specifically, the decompressed user data specifically refers to the data obtained after decompressing the target user data; correspondingly, the initial data to be distributed specifically refers to the data obtained after parsing and converting the decompressed user data, wherein parsing and converting specifically refers to the processing operation of converting the data format and merging the decompressed user data. Correspondingly, the distribution rules and screening rules can refer to the description of the data distribution rules and initial data screening rules in the above embodiment, and this embodiment will not be repeated in detail here.

[0072] Based on this, after obtaining the target user data, the target user data can be decompressed through the data screening rules preset by the multi-link programming device to obtain the decompressed user data; on this basis, the decompressed user data is parsed and converted, and the initial data to be distributed can be obtained according to the parsing and conversion results; finally, the initial data to be distributed is processed according to the distribution rules and screening rules in the data screening rules, and the data to be distributed can be generated according to the processing results for subsequent distribution processing.

[0073] In actual applications, when decompressing data, the compressed data of the Vint type is actually decompressed, and the decompressed user data is combined with the data that does not need to be decompressed. After obtaining the decompressed user data, it is processed and converted to generate full data that meets the CPU processing requirements, and then distributed to the CPU core for processing.

[0074] Furthermore, when decompressing data, the incremental user data is decompressed and the initial data to be distributed is constructed by updating the data. In this embodiment, the specific implementation is as follows:

[0075] Incremental user data is selected from the target user data according to preset data screening rules, and the incremental user data is decompressed to obtain decompressed user data; wherein, parsing and converting the decompressed user data to obtain initial data to be distributed includes: determining data identification information corresponding to the decompressed user data, and querying the full user data according to the data identification information; and updating the full user data based on the decompressed user data to obtain the initial data to be distributed.

[0076] Specifically, the enhanced user data specifically refers to the data of the corresponding Vint type in the target user data. Correspondingly, the data identification information specifically refers to the contract number corresponding to the decompressed user data, and correspondingly, the full user data specifically refers to the original full data corresponding to the decompressed user data and not yet merged with the decompressed user data.

[0077] Based on this, in the data decompression stage, the incremental user data can be first selected from the target user data according to the preset data screening rules. At this time, only the incremental user data can be decompressed to obtain the decompressed user data; on this basis, the data identification information corresponding to the decompressed user data is determined. At this time, the full user data can be queried according to the data identification information, so that the full user data can be updated based on the decompressed user data, and the initial data to be distributed can be obtained, so that the data to be distributed can be subsequently constructed based on the initial data to be distributed for transmission to the data processing node for processing.

[0078] In practical applications, the MIRP message can be split into minimum business data according to the MIRP protocol rules, and then the minimum business data message type can be parsed to process only the incremental data type according to the parsing result, and the rest of the data is discarded. On this basis, the corresponding field can be Vint decompressed according to the MIRP data type, and then the full data corresponding to the storage contract table can be queried according to the contract number in the minimum business data. Finally, the original full data is updated according to the parsed incremental data, and the updated full data can be sent to the data processing node for corresponding processing operations.

[0079] Continuing with the above example, after obtaining the target user data, the Vint type compressed data in the target user data can be decompressed, and then the decompressed data and the incremental data that does not need to be decompressed are processed in depth and converted into full data suitable for CPU processing, which is sent to the CPU. This can realize the contract information and channel relationship issued by the CPU in the initial stage, and prepare for intelligent distribution after preliminary processing and screening of the data, so as to select the appropriate CPU core to process the data to be distributed.

[0080] In summary, by preprocessing the target user data according to the preset data screening rules, the data can be processed into a format that conforms to the processing of the data processing node, thereby effectively improving the data processing efficiency of subsequent data processing nodes and avoiding occupying too many computing resources.

[0081] Step S206, obtaining the load information of the data processing cluster corresponding to the server and the distribution strategy matching the data to be distributed.

[0082] Specifically, after obtaining the data to be distributed as described above, considering that the data processing cluster contains a large number of data processing nodes, and different data processing nodes correspond to different load states at the current moment, if no intelligent distribution processing is performed, it may cause some data processing nodes to be overloaded, and some data processing nodes to have idle computing resources for a long time. Therefore, in order to achieve load balancing, before distributing the data to be distributed, the load information of the data processing cluster corresponding to the server and the distribution strategy matching the data to be distributed can be obtained first, so that a reasonable data processing node can be selected for data processing in combination with the load information and the distribution strategy. Before data distribution, the load situation of the data processing cluster and the processing priority of the data to be distributed can be considered to improve data processing efficiency and be more reasonable.

[0083] Among them, the data processing cluster specifically refers to the cluster composed of data processing nodes, that is, the CPU core cluster of the CPU. Correspondingly, the load information specifically refers to the description information of the load situation corresponding to each data processing node in the data processing cluster, including but not limited to the CPU core utilization, the number and type of current processing tasks, etc. Correspondingly, the distribution strategy specifically refers to the strategy of data distribution rules when distributing and processing the data to be distributed, which is used to determine the task priority, time sensitivity, required resources, etc., and then combine the load information to select a more reasonable data processing node for the transmission and processing of the data to be distributed.

[0084] Furthermore, when obtaining the distribution strategy and complex information, in order to avoid the impact of this operation on the efficiency of the data distribution operation, the distribution strategy module and the load feedback module can provide load information and distribution strategy respectively, so that after receiving the data to be distributed, relevant information can be requested in real time to dynamically adjust the distribution operation. In this embodiment, the specific implementation method is as follows:

[0085] Submit a policy request to the distribution policy module in the multi-link programming device, and receive the initial distribution policy fed back by the distribution policy module in response to the policy request as the distribution strategy matching the data to be distributed; submit a load request to the load feedback module in the multi-link programming device, and receive the initial load information fed back by the load feedback module in response to the load request as the load information of the data processing cluster corresponding to the server; wherein, the initial distribution policy fed back by the distribution policy module is configured by the multi-link programming device through registers during the startup phase; and the initial load information fed back by the load feedback module is determined by requesting the data processing cluster.

[0086] In practical applications, the distribution strategy includes data priority division information, time limit information, and resource request information; wherein the load information includes data processing node task information, data processing node resource information, and data processing node type information.

[0087] Specifically, the distribution strategy module specifically refers to a module for storing the initial distribution strategy preset by the business party. Correspondingly, the load feedback request specifically refers to a module for obtaining the load information of the data processing cluster in real time. Correspondingly, the data priority division information specifically refers to the rule information for prioritizing the data to be distributed. The time limit information specifically refers to the rule information for time-sensitive division of the data to be distributed. The resource request information specifically refers to the information on the amount of computing resources required to calculate the data to be distributed. Correspondingly, the data processing node task information specifically refers to the information recording the number of tasks processed by the data processing node at the current moment. The data processing node resource information specifically refers to the information recording the amount of resources occupied and the amount of available resources by the data processing node at the current moment. The data processing node type information specifically refers to the information recording the type of task currently processed by the data processing node.

[0088] Based on this, when obtaining distribution strategy and load information, a policy request can be submitted to the distribution strategy module in the multi-link programming device, and the initial distribution strategy fed back by the distribution strategy module for the policy request can be received. At this time, the initial distribution strategy can be used as the distribution strategy matching the data to be distributed; at the same time, a load request can be submitted to the load feedback module in the multi-link programming device, and the initial load information fed back by the load feedback module for the load request can be received. At this time, the initial load information can be used as the load information of the corresponding data processing cluster of the server; wherein, the initial distribution strategy fed back by the distribution strategy module is configured by the multi-link programming device through registers during the startup phase; the initial load information fed back by the load feedback module is determined by requesting the data processing cluster.

[0089] In actual applications, after obtaining the data to be distributed, the distribution strategy module can be requested for the distribution strategy, and the strategy of the distribution strategy module is configured by the CPU through registers before the system starts. Among them, the distribution strategy may include data priority division information, time limit information, resource request information, etc. It is used to achieve efficient data distribution according to the preset strategy during the operation of the system. After submitting the request, detailed distribution strategy information can be obtained, and then the distribution plan for the data to be distributed can be completed.

[0090] When requesting load information from the load feedback module, the multi-link programming device can understand the usage status of the data processing node, including data processing node task information, data processing node resource information, data processing node type information, etc. This helps to dynamically adjust the distribution operation in combination with the actual load situation during data distribution, and then select a reasonable data processing node to process the data to be distributed.

[0091] Continuing with the above example, after obtaining the data to be distributed that is suitable for the CPU core to process, you can request a distribution strategy from the distribution strategy module and receive the target distribution strategy fed back by the distribution strategy module. The target distribution strategy records data priority information, time limit information, resource request information, etc. At the same time, you can request the CPU load status from the load feedback module and receive the target load information fed back by the load feedback module. The target load information records the current number of tasks, processing speed, idle time, and other information of the CPU1 core to the CPU16 core. After that, you can select a CPU core suitable for processing the data to be distributed from the CPU1 core to the CPU16 core in combination with the target load information and the target distribution strategy to complete the processing operation of the data to be distributed.

[0092] In summary, by combining the distribution strategy module and the load feedback module to record load information and distribution strategy, it is possible to quickly obtain real-time information for subsequent selection of data processing nodes after determining the strategy to be distributed, thereby effectively improving data processing efficiency.

[0093] Step S208: selecting a data processing node in the data processing cluster based on the distribution strategy and the load information, and transmitting the data to be distributed to the data processing node for processing via a communication link matched by the data processing node.

[0094] Specifically, after determining the data to be distributed, the distribution strategy and the load information as described above, further, in order to improve data processing efficiency and avoid the situation where some data processing nodes are congested and some data processing nodes are idle, the distribution strategy and the load information can be combined to select a data processing node in the data processing cluster that matches the data to be distributed. Thereafter, the data to be distributed can be transmitted to the data processing node for processing through the communication link that matches the data processing node.

[0095] The selected data processing node specifically refers to a CPU core in the data processing cluster that matches the data to be distributed. Correspondingly, the communication link is a DMA link between the data processing node and the multi-link programming device.

[0096] Using the above example, see Figure 3 As shown in the schematic diagram, after obtaining the target distribution strategy and the target load information corresponding to the CPU1 core ~ ​​CPU16 core, each CPU core can be analyzed in combination with the target distribution strategy and the target load information, and the CPU5 core is determined to match the data to be distributed according to the analysis results. Therefore, the data to be distributed can be transmitted to the CPU5 core through the DMA link corresponding to the CPU5 core, thereby optimizing resource utilization and giving priority to high-frequency trading contracts, thereby ensuring that the system can continue to run under high load conditions.

[0097] In addition, considering that the data processing cluster is evenly loaded, the data to be distributed may not be processed. In order to avoid the impact of this problem on the business, a standby data processing node can be selected to process the data processing task. In this embodiment, the specific implementation method is as follows:

[0098] When no data processing node is selected in the data processing cluster according to the distribution strategy and the load information, a standby data processing node is started according to the distribution strategy and the load information; a target key associated with the data to be distributed is allocated to the standby data processing node, and a standby communication link matching the standby data processing node is determined; and the data to be distributed is transmitted to the standby data processing node through the standby communication link, so that the standby data processing node processes the data to be distributed based on the target key.

[0099] Specifically, the standby data processing node specifically refers to a data processing node used in the disaster recovery phase, which can replace the data processing node in the data processing cluster. Correspondingly, the target key specifically refers to a key that matches the data to be distributed, which is used to ensure data security. Correspondingly, the standby communication link is a standby DMA link between the standby data processing node and the multi-link programming device.

[0100] Based on this, when no data processing node is selected in the data processing cluster according to the distribution strategy and load information, it means that the data processing stages in the data processing cluster are all under high load. At this time, you can choose to process the data to be distributed in a queue waiting manner, or you can start the standby data processing node according to the distribution strategy and load information; at this time, you can assign the target key associated with the data to be distributed to the standby data processing node, and determine the standby communication link matching the standby data processing node; thereafter, the data to be distributed can be transmitted to the standby data processing node through the standby communication link, so that the standby data processing node can process the data to be distributed based on the target key, thereby avoiding the problem of business interruption.

[0101] In summary, by using a standby data processing node to temporarily replace a data processing node to ensure normal business operation, the data processing efficiency can be effectively improved.

[0102] The data processing method provided in this embodiment can configure a multi-link programming device on the server in order to achieve rapid response to business changes and effectively improve the overall business data processing efficiency. The data distribution node can parse the target user data from the business data stream after the multi-link programming device receives the business data stream pushed by the business data source. For the target user data, the target user data can be first processed according to the preset data screening rules to screen out the data to be distributed in the target user data that needs to be distributed to the data processing node for processing according to the processing results. On this basis, in order to improve the data processing efficiency, the load information of the data processing cluster corresponding to the server and the distribution strategy matched by the data to be distributed can be obtained to make a distribution decision before processing the data to be distributed, thereby realizing the selection of the data processing node in the data processing cluster based on the distribution strategy and load information. On this basis, the data to be distributed can be transmitted to the data processing node for processing through the communication link matched by the data processing node, so as to balance the load in a way that combines the distribution strategy and load information to avoid overloading of some data processing nodes, thereby effectively improving the processing efficiency of the overall data processing cluster.

[0103] The following combination Figure 4 , taking the application of the data processing method provided in this specification in a shopping scenario as an example, the data processing method is further described. Figure 4 A processing flow chart of a data processing method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0104] Step S402, executing the device initialization task in the initialization phase of the multi-link programming device, and initializing the candidate communication links associated with the multi-link programming device.

[0105] Step S404: when the initialization task is completed and the candidate communication link is initialized, determine the data distribution rule and the initial data screening rule in response to the rule configuration request.

[0106] Step S406: configuring data screening rules according to the data distribution rules and the initial data screening rules.

[0107] Step S408, receiving the business data stream pushed by the business data source, parsing the data header of the business data stream, and determining the data type corresponding to the business data in the business data stream according to the parsing result.

[0108] Step S410: When it is determined according to the data type that the service data is a user data type and meets a preset distribution condition, the service data is used as target user data.

[0109] Step S412, decompress the target user data according to the preset data screening rule to obtain decompressed user data.

[0110] Step S414: parse and convert the decompressed user data to obtain initial data to be distributed.

[0111] Step S416, processing the initial data to be distributed according to the data distribution rule and the initial data screening rule in the data screening rule, and generating the data to be distributed according to the processing result.

[0112] Step S418, submitting a policy request to the distribution policy module in the multi-link programming device, and receiving an initial distribution policy fed back by the distribution policy module in response to the policy request as a distribution policy matching the data to be distributed.

[0113] Step S420, submitting a load request to a load feedback module in the multi-link programming device, and receiving initial load information fed back by the load feedback module in response to the load request as load information of the data processing cluster corresponding to the server.

[0114] Step S422: Select a data processing node in the data processing cluster based on the distribution strategy and load information, and transmit the data to be distributed to the data processing node for processing via a communication link matched by the data processing node.

[0115] To sum up, in order to be able to quickly respond to business changes and effectively improve the overall business data processing efficiency, a multi-link programming device can be configured on the server to achieve data distribution nodes. After the multi-link programming device receives the business data stream pushed by the business data source, it can parse the target user data from the business data stream; for the target user data, the target user data can be first processed according to the preset data screening rules to filter out the target user data that needs to be distributed to the data processing node for processing according to the processing results; on this basis, in order to improve the data processing efficiency, the load information of the data processing cluster corresponding to the server and the distribution strategy matching the data to be distributed can be obtained to make a distribution decision before processing the data to be distributed, and then select the data processing node in the data processing cluster based on the distribution strategy and load information. On this basis, the data to be distributed can be transmitted to the data processing node for processing through the communication link matching the data processing node, so as to balance the load by combining the distribution strategy and load information to avoid overloading of some data processing nodes, thereby effectively improving the processing efficiency of the overall data processing cluster.

[0116] Corresponding to the above method embodiment, this specification also provides a data processing system embodiment. Figure 5 FIG. 1 shows a schematic diagram of a data processing system provided by an embodiment of the present specification. Figure 5As shown, the data processing system 500 includes a multi-link programming device 510 and a data processing cluster 520, wherein the data processing cluster 520 includes a plurality of initial data processing nodes 522, including:

[0117] The multi-link programming device 510 is used to receive the business data stream pushed by the business data source, and parse the target user data from the business data stream; process the target user data according to the preset data screening rules to obtain the data to be distributed; obtain the load information of the data processing cluster and the distribution strategy matching the data to be distributed; select a data processing node in the data processing cluster based on the distribution strategy and the load information, and transmit the data to be distributed to the data processing node through the communication link matching the data processing node;

[0118] The data processing node 524 is used to receive the data to be distributed and process the data to be distributed according to preset business rules.

[0119] In an optional embodiment, the multi-link programming device 510 is further used for:

[0120] During the initialization phase of the multi-link programming device, a device initialization task is executed, and the candidate communication links associated with the multi-link programming device are initialized; when the initialization task is completed and the candidate communication links are initialized, data distribution rules and initial data screening rules are determined in response to a rule configuration request; and data screening rules are configured according to the data distribution rules and the initial data screening rules.

[0121] In an optional embodiment, the multi-link programming device 510 is further used for:

[0122] Parse the data header of the business data stream, and determine the data type corresponding to the business data in the business data stream according to the parsing result; if it is determined according to the data type that the business data is a user data type and meets the preset distribution conditions, use the business data as target user data.

[0123] In an optional embodiment, the multi-link programming device 510 is further used for:

[0124] The target user data is decompressed according to preset data screening rules to obtain decompressed user data; the decompressed user data is parsed and converted to obtain initial data to be distributed; the initial data to be distributed is processed according to the distribution rules and screening rules in the data screening rules, and the data to be distributed is generated according to the processing results.

[0125] In an optional embodiment, the multi-link programming device 510 is further used for:

[0126] Submit a policy request to the distribution policy module in the multi-link programming device, and receive the initial distribution policy fed back by the distribution policy module in response to the policy request as the distribution strategy matching the data to be distributed; submit a load request to the load feedback module in the multi-link programming device, and receive the initial load information fed back by the load feedback module in response to the load request as the load information of the data processing cluster corresponding to the server; wherein, the initial distribution policy fed back by the distribution policy module is configured by the multi-link programming device through registers during the startup phase; and the initial load information fed back by the load feedback module is determined by requesting the data processing cluster.

[0127] In an optional embodiment, the distribution strategy includes data priority division information, time limit information, and resource request information; wherein the load information includes data processing node task information, data processing node resource information, and data processing node type information.

[0128] In an optional embodiment, the multi-link programming device 510 is further used for:

[0129] Incremental user data is selected from the target user data according to preset data screening rules, and the incremental user data is decompressed to obtain decompressed user data; wherein, parsing and converting the decompressed user data to obtain initial data to be distributed includes: determining data identification information corresponding to the decompressed user data, and querying the full user data according to the data identification information; and updating the full user data based on the decompressed user data to obtain the initial data to be distributed.

[0130] In an optional embodiment, the multi-link programming device 510 is further used for:

[0131] When no data processing node is selected in the data processing cluster according to the distribution strategy and the load information, a standby data processing node is started according to the distribution strategy and the load information; a target key associated with the data to be distributed is allocated to the standby data processing node, and a standby communication link matching the standby data processing node is determined; and the data to be distributed is transmitted to the standby data processing node through the standby communication link, so that the standby data processing node processes the data to be distributed based on the target key.

[0132] In order to achieve rapid response to business changes and effectively improve the overall business data processing efficiency, the data processing system provided in this embodiment can be configured with a multi-link programming device on the server, so that at the data distribution node, after the multi-link programming device receives the business data stream pushed by the business data source, the target user data can be parsed from the business data stream; for the target user data, the target user data can be first processed according to the preset data screening rules, so as to screen out the data to be distributed in the target user data that needs to be distributed to the data processing node for processing according to the processing results; on this basis, in order to improve the data processing efficiency, the load information of the data processing cluster corresponding to the server and the distribution strategy matched by the data to be distributed can be obtained to make a distribution decision before processing the data to be distributed, and then select the data processing node in the data processing cluster based on the distribution strategy and load information, and on this basis, the data to be distributed can be transmitted to the data processing node for processing through the communication link matched by the data processing node, so as to balance the load in a way that combines the distribution strategy and load information, and avoid overloading of some data processing nodes, thereby effectively improving the processing efficiency of the overall data processing cluster.

[0133] The above is a schematic scheme of a data processing system of this embodiment. It should be noted that the technical scheme of the data processing system and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the data processing system can be referred to the description of the technical scheme of the above data processing method.

[0134] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 6 As shown, the device is applied to a multi-link programming device configured with a server, including:

[0135] The receiving module 602 is configured to receive a service data stream pushed by a service data source, and parse the service data stream to obtain target user data;

[0136] The processing module 604 is configured to process the target user data according to a preset data screening rule to obtain data to be distributed;

[0137] The acquisition module 606 is configured to acquire the load information of the data processing cluster corresponding to the server and the distribution strategy matched by the data to be distributed;

[0138] The selection module 608 is configured to select a data processing node in the data processing cluster based on the distribution strategy and the load information, and transmit the to-be-distributed data to the data processing node for processing via a communication link matched by the data processing node.

[0139] In an optional embodiment, the device further comprises:

[0140] An initialization module is configured to execute a device initialization task during the initialization phase of the multi-link programming device, and to initialize candidate communication links associated with the multi-link programming device; determine data distribution rules and initial data screening rules in response to a rule configuration request when the initialization task is completed and the candidate communication links are initialized; and configure data screening rules based on the data distribution rules and the initial data screening rules.

[0141] In an optional embodiment, the receiving module 602 is further configured to:

[0142] Parse the data header of the business data stream, and determine the data type corresponding to the business data in the business data stream according to the parsing result; if it is determined according to the data type that the business data is a user data type and meets the preset distribution conditions, use the business data as target user data.

[0143] In an optional embodiment, the processing module 604 is further configured to:

[0144] The target user data is decompressed according to preset data screening rules to obtain decompressed user data; the decompressed user data is parsed and converted to obtain initial data to be distributed; the initial data to be distributed is processed according to the distribution rules and screening rules in the data screening rules, and the data to be distributed is generated according to the processing results.

[0145] In an optional embodiment, the acquisition module 606 is further configured to:

[0146] Submit a policy request to the distribution policy module in the multi-link programming device, and receive the initial distribution policy fed back by the distribution policy module in response to the policy request as the distribution strategy matching the data to be distributed; submit a load request to the load feedback module in the multi-link programming device, and receive the initial load information fed back by the load feedback module in response to the load request as the load information of the data processing cluster corresponding to the server; wherein, the initial distribution policy fed back by the distribution policy module is configured by the multi-link programming device through registers during the startup phase; and the initial load information fed back by the load feedback module is determined by requesting the data processing cluster.

[0147] In an optional embodiment, the distribution strategy includes data priority division information, time limit information, and resource request information; wherein the load information includes data processing node task information, data processing node resource information, and data processing node type information.

[0148] In an optional embodiment, the processing module 604 is further configured to:

[0149] Incremental user data is selected from the target user data according to preset data screening rules, and the incremental user data is decompressed to obtain decompressed user data; wherein, parsing and converting the decompressed user data to obtain initial data to be distributed includes: determining data identification information corresponding to the decompressed user data, and querying the full user data according to the data identification information; and updating the full user data based on the decompressed user data to obtain the initial data to be distributed.

[0150] In an optional embodiment, the device further comprises:

[0151] The standby module is configured to, when no data processing node is selected in the data processing cluster according to the distribution strategy and the load information, start a standby data processing node according to the distribution strategy and the load information; assign a target key associated with the data to be distributed to the standby data processing node, and determine a standby communication link matching the standby data processing node; and transmit the data to be distributed to the standby data processing node through the standby communication link, so that the standby data processing node processes the data to be distributed based on the target key.

[0152] The data processing device provided in this embodiment can configure a multi-link programming device on the server in order to achieve rapid response to business changes and effectively improve the overall business data processing efficiency. The data distribution node can parse the target user data from the business data stream after the multi-link programming device receives the business data stream pushed by the business data source. For the target user data, the target user data can be first processed according to the preset data screening rules to screen out the data to be distributed in the target user data that needs to be distributed to the data processing node for processing according to the processing results. On this basis, in order to improve the data processing efficiency, the load information of the data processing cluster corresponding to the server and the distribution strategy matching the data to be distributed can be obtained to make a distribution decision before processing the data to be distributed, thereby realizing the selection of the data processing node in the data processing cluster based on the distribution strategy and load information. On this basis, the data to be distributed can be transmitted to the data processing node for processing through the communication link matching the data processing node, so as to balance the load in a way that combines the distribution strategy and load information to avoid overloading of some data processing nodes, thereby effectively improving the processing efficiency of the overall data processing cluster.

[0153] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0154] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0155] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0156] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0157] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.

[0158] The processor 720 is used to execute the following computer executable instructions, which implement the steps of the above data processing method when executed by the processor.

[0159] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0160] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned data processing method when executed by a processor.

[0161] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0162] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above data processing method when executed by a processor.

[0163] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above data processing method.

[0164] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0165] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0166] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0167] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0168] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A data processing method, characterized in that: Multi-link programming device for server configuration, including: Receive a business data stream pushed by a business data source, and parse the business data stream to obtain target user data; Decompress the target user data according to the preset data screening rules to obtain decompressed user data; parse and convert the decompressed user data to obtain initial data to be distributed; process the initial data to be distributed according to the distribution rules and screening rules in the data screening rules, and generate data to be distributed according to the processing results; Obtaining load information of a data processing cluster corresponding to the server and a distribution strategy matching the data to be distributed; Based on the data priority division information, time limit information and resource request information contained in the distribution strategy, and the data processing node task information, data processing node resource information and data processing node type information contained in the load information, a data processing node is selected in the data processing cluster, and the data to be distributed is transmitted to the data processing node for processing through a communication link matching the data processing node.

2. The data processing method according to claim 1, characterized in that: Before the step of receiving the business data flow pushed by the business data source is executed, it also includes: Performing a device initialization task during the initialization phase of the multi-link programming device, and initializing candidate communication links associated with the multi-link programming device; In the case where the execution of the initialization task is completed and the initialization of the candidate communication link is completed, determining a data distribution rule and an initial data screening rule in response to a rule configuration request; The data screening rule is configured according to the data distribution rule and the initial data screening rule.

3. The data processing method according to claim 1, characterized in that: The step of parsing the target user data from the service data stream includes: Parsing the data header of the service data stream, and determining the data type corresponding to the service data in the service data stream according to the parsing result; When it is determined according to the data type that the service data is a user data type and meets a preset distribution condition, the service data is used as target user data.

4. The data processing method according to claim 1, characterized in that: The obtaining of the load information of the data processing cluster corresponding to the server and the distribution strategy matching the data to be distributed includes: Submitting a policy request to a distribution policy module in the multi-link programming device, and receiving an initial distribution policy fed back by the distribution policy module in response to the policy request as a distribution policy matched by the data to be distributed; Submitting a load request to a load feedback module in the multi-link programming device, and receiving initial load information fed back by the load feedback module in response to the load request as load information of a data processing cluster corresponding to the server; The initial distribution strategy fed back by the distribution strategy module is configured by the multi-link programming device through registers during the startup phase; and the initial load information fed back by the load feedback module is determined by requesting the data processing cluster.

5. The data processing method according to claim 1, characterized in that: The step of decompressing the target user data according to a preset data screening rule to obtain the decompressed user data includes: selecting incremental user data from the target user data according to a preset data screening rule, and decompressing the incremental user data to obtain decompressed user data; The parsing and converting of the decompressed user data to obtain initial data to be distributed includes: Determine data identification information corresponding to the decompressed user data, and query the full amount of user data according to the data identification information; The full amount of user data is updated based on the decompressed user data to obtain initial data to be distributed.

6. The data processing method according to any one of claims 1 to 3, characterized in that: After the steps of obtaining the load information of the data processing cluster corresponding to the server and matching the distribution strategy of the data to be distributed are executed, the method further includes: When no data processing node is selected in the data processing cluster according to the distribution strategy and the load information, starting a standby data processing node according to the distribution strategy and the load information; allocating a target key associated with the data to be distributed to the standby data processing node, and determining a standby communication link matching the standby data processing node; The data to be distributed is transmitted to the standby data processing node via the standby communication link, so that the standby data processing node processes the data to be distributed based on the target key.

7. A data processing system, characterized in that: The data processing system includes a multi-link programming device and a data processing cluster, wherein the data processing cluster includes a plurality of initial data processing nodes, including: The multi-link programming device is used to receive a business data stream pushed by a business data source, and parse out target user data from the business data stream; decompress the target user data according to a preset data screening rule to obtain decompressed user data; parse and convert the decompressed user data to obtain initial data to be distributed; process the initial data to be distributed according to the distribution rules and screening rules in the data screening rules, and generate data to be distributed according to the processing results; obtain the load information of the data processing cluster, and the distribution strategy matching the data to be distributed; select a data processing node in the data processing cluster based on the data priority division information, time limit information and resource request information contained in the distribution strategy, and the data processing node task information, data processing node resource information and data processing node type information contained in the load information, and transmit the data to be distributed to the data processing node through the communication link matching the data processing node; The data processing node is used to receive the data to be distributed and process the data to be distributed according to preset business rules.

8. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images