Data processing method and device of data engine, equipment and medium

Abstract

The invention provides a data processing method and device for a data engine, equipment and a medium, and the method comprises the steps: determining a plurality of processing nodes based on a to-be-processed data packet, and then determining a plurality of target processing nodes according to the processing efficiency, and segmenting the to-be-processed data packet into a plurality of sub-data packets and sending the sub-data packets to each target processing node for processing. The method has the advantages that efficient blocking processing of the to-be-processed data packet is completed, multiple target processing nodes with better efficiency are selected to process the to-be-processed data packet at the same time, the processing space of a single processing node does not need to be occupied, and the processing efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method, device, equipment and medium for a data engine. Background Art

[0002] A data engine is a software system that is specifically used to store, process, and retrieve large amounts of data. It uses efficient algorithms and technologies to achieve efficient management and fast access to data. The core functions of a data engine include data storage, data retrieval, data processing, data analysis, and data visualization.

[0003] In the existing technology, when data is processed by a data engine, data packets are randomly split and assigned to different processing nodes for processing without analyzing each processing node, resulting in a low efficiency in processing data packets. Summary of the invention

[0004] The main purpose of the present invention is to provide a data processing method, device, equipment and medium for a data engine, aiming to solve the problem of low efficiency in processing data packets when the existing data engine performs data processing.

[0005] The present invention provides a data processing method of a data engine, comprising:

[0006] receiving a data packet to be processed through a data engine;

[0007] Determining a plurality of processing nodes based on the data packets to be processed;

[0008] Obtain the processing efficiency of each processing node, and determine the target processing node and the processing ratio of each target processing node according to the processing efficiency;

[0009] Dividing the data packet to be processed into corresponding sub-data packets according to the processing ratio;

[0010] Sending the sub-data packet to the corresponding target processing node;

[0011] The target processing node calculates the hash value of the corresponding processed sub-packet and forms a key-value pair with the ID;

[0012] Each of the target processing nodes sends the key-value pair to the data engine to form a key-value pair list;

[0013] Based on the key-value pair list, each target processing node and the data engine form a logical chain;

[0014] Distribute and process each of the sub-data packets based on the logical chain processing.

[0015] Furthermore, the step of obtaining the processing efficiency of each processing node and determining the target processing node and the processing ratio of each target processing node according to the processing efficiency includes:

[0016] Monitoring the number of processing flows of each processing node at preset intervals; wherein the number of processing flows is used to provide feedback on the processing efficiency of the processing node;

[0017] Compare the current processing flow number of each processing node closest to the current time with the preset relaxation flow number;

[0018] Selecting a target processing node from the processing nodes according to the comparison result, and setting conversion parameters of each of the target processing nodes;

[0019] The processing ratio of each target processing node is set based on the conversion parameter.

[0020] Furthermore, the step of setting the conversion parameters of each of the target processing nodes includes:

[0021] If the current processing flow number is greater than the preset relaxation flow number, the conversion parameter of the corresponding section is set to a fixed value of 1;

[0022] If the current processing flow number is not greater than the preset relaxation flow number, then extract multiple processing flow numbers that are closest to the current time and greater than the preset relaxation flow number, and record them as the target processing flow number set;

[0023] The data capacity set T of each target processing node is obtained by multiplying the number of each processing flow in the target processing flow set by the size of the corresponding processing data packet. i (t 1 ), T i (t 2 ),...,T i (t k ),...,T i (t l ), where T i (t k ) represents the time t k The data capacity of the i-th target processing node, and t 1 <t 2 <... <t k <... <t l , t k -t k-1 =t a , t a is the interval time;

[0024] Each element in each of the data capacity sets is input into the fitting formula T i (l) = T i(k)+A i [T i (l-1)-T i (k-1)], the conversion parameter A corresponding to each data capacity set is obtained i Among them, A i The conversion parameter A represents the data capacity set corresponding to the i-th target processing node i .

[0025] Furthermore, the step of dividing the data packet to be processed into corresponding sub-data packets according to the processing ratio includes:

[0026] Determining a plurality of pre-segmentation points in the data packet to be processed according to the processing ratio;

[0027] Based on the position of each pre-segmentation point in the data packet to be processed, finding the delimiter closest to each pre-segmentation point as the target segmentation point;

[0028] The data packet to be processed is divided into corresponding sub-data packets based on the target division point.

[0029] Furthermore, the step of forming a logical chain between each target processing node and the data engine based on the key-value pair list includes:

[0030] Based on the key-value pair list, each target processing node and the data engine form a preliminary logical chain;

[0031] Obtain data information of sub-data packets processed by each target processing node;

[0032] Set a processing logic plan for each target processing node according to each data information;

[0033] The processing logic scheme is sent to the preliminary logic chain to obtain the logic chain.

[0034] Furthermore, after the step of distributing and processing each of the sub-data packets based on the logical chain processing, the method further includes:

[0035] The Merkle tree root of the key-value pairs pre-stored at each target processing node in the data engine is updated through a preset hash algorithm to obtain a target Merkle tree root; wherein the target Merkle tree root is used to obtain the corresponding target processing node storing the sub-data packet according to the index query of the user, so as to obtain the data packet to be processed and its processing result.

[0036] Furthermore, after the step of distributing and processing each of the sub-data packets based on the logical chain processing, the method further includes:

[0037] Selecting multiple audit nodes from multiple processing nodes according to a preset selection method;

[0038] Send the hash value calculated for each sub-data packet to each audit node;

[0039] Based on the data engine, the sub-data packets in each target processing node are obtained, the target hash values ​​are calculated and sent to each audit node;

[0040] The audit node audits the target hash value based on the hash value and sends the audit result to the data engine.

[0041] The present invention also provides a data processing device of a data engine, comprising:

[0042] A receiving module, used for receiving a data packet to be processed through a data engine;

[0043] A determination module, configured to determine a plurality of processing nodes based on the data packet to be processed;

[0044] An acquisition module is used to acquire the processing efficiency of each processing node, and determine the target processing node and the processing ratio of each target processing node according to the processing efficiency;

[0045] A segmentation module, used for segmenting the data packet to be processed into corresponding sub-data packets according to the processing ratio;

[0046] A first sending module, used for sending the sub-data packet to a corresponding target processing node;

[0047] A calculation module, used to calculate the hash value of the corresponding processed sub-data packet through the target processing node, and form a key-value pair with the ID;

[0048] A second sending module, each of the target processing nodes is used to send the key-value pairs to the data engine to form a key-value pair list;

[0049] A forming module, used to form a logical chain between each target processing node and the data engine based on the key-value pair list;

[0050] A processing module is used to distribute the processing of each of the sub-data packets based on the logical chain processing.

[0051] The present invention also provides a computer device, including a processor and a processor, wherein the processor processes a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0052] The present invention also provides a computer-readable processing medium on which a computer program is processed, and when the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0053] The beneficial effects of the present invention are as follows: by determining multiple processing nodes based on the data packet to be processed, and then determining multiple target processing nodes according to the processing efficiency, the data packet to be processed is divided into multiple sub-data packets and sent to each target processing node for processing, thereby completing efficient block processing of the data packet to be processed, and selecting multiple target processing nodes with better efficiency to process the data packet to be processed at the same time, without occupying the processing space of a single processing node, thereby greatly improving the processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flowchart of a data processing method of a data engine according to an embodiment of the present invention;

[0055] Figure 2 It is a schematic block diagram of the structure of a data processing device of a data engine according to an embodiment of the present invention;

[0056] Figure 3 A schematic block diagram of the structure of a computer device according to an embodiment of the present application.

[0057] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. The connection can be a direct connection or an indirect connection.

[0060] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone.

[0061] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0062] Reference Figure 1 The present invention proposes a data processing method of a data engine, comprising:

[0063] S1: receiving data packets to be processed through the data engine;

[0064] S2: Determine multiple processing nodes based on the data packet to be processed;

[0065] S3: Obtain the processing efficiency of each processing node, and determine the target processing node and the processing ratio of each target processing node according to the processing efficiency;

[0066] S4: dividing the data packet to be processed into corresponding sub-data packets according to the processing ratio;

[0067] S5: Send the sub-data packet to the corresponding target processing node;

[0068] S6: Calculate the hash value of the corresponding processed sub-data packet through the target processing node, and form a key-value pair with the ID;

[0069] S7: Each of the target processing nodes sends the key-value pairs to the data engine to form a key-value pair list;

[0070] S8: forming a logical chain between each target processing node and the data engine based on the key-value pair list;

[0071] S9: Distribute and process each of the sub-data packets based on the logic chain processing.

[0072] As described in step S1 above, receiving the data packet to be processed through the data engine usually includes capturing, parsing, processing and storing the data packet. The data packet to be processed is a data packet uploaded manually, or it can be a storage address where the data packet to be processed is input, and then the data is acquired through a crawler or a data protocol. After the data packet to be processed is acquired, the data packet can be pre-processed, such as cleaning, formatting, deduplication, etc.

[0073] As described in the above step S2, multiple processing nodes are determined based on the data packets to be processed. That is, the total amount of data packets to be processed is determined, and the size of the data packets to be processed determines the number of processing nodes. Specifically, the corresponding relationship between the size of the data packets to be processed and the number of processing nodes can be pre-constructed. Only the size of the data packets to be processed needs to be input to determine the number of processing nodes. In some embodiments, processing nodes that can process the data to be processed can also be screened. It should be noted that here, not all processing nodes are selected for processing, but target processing nodes need to be selected from the processing nodes for processing. This is just a preliminary screening process.

[0074] As described in the above step S3, the processing efficiency of each processing node is obtained, and the target processing node and the processing ratio of each target processing node are determined according to the processing efficiency. Among them, the target processing node is determined according to the processing efficiency. Specifically, the target processing node with higher processing efficiency should have a higher corresponding processing ratio. The specific allocation method is not limited. The target processing node can be determined according to the size of the data packet to be processed, and the number of target processing nodes is determined, and then the corresponding number of target processing nodes is selected according to the processing efficiency of each processing node. The processing ratio can also be allocated according to the processing efficiency of each processing node. This application also provides a detailed allocation method later, which will not be repeated here.

[0075] As described in the above steps S4-S5, the data packet to be processed is divided into corresponding sub-data packets according to the processing ratio, and the sub-data packets are sent to the corresponding target processing node. That is, the data packet to be processed is divided into corresponding sub-data packets. It should be noted that since the sub-data packets need to ensure data integrity, a separator can be selected to divide the data packet to be processed, and then the divided sub-data packets are sent to the corresponding target processing node for processing.

[0076] As described in the above steps S6-S9, the hash value of the corresponding processed sub-data packet is calculated by the target processing node, and a key-value pair is formed with the ID. Each of the target processing nodes sends the key-value pair to the data engine to form a key-value pair list. Based on the key-value pair list, each target processing node and the data engine form a logical chain, and each of the sub-data packets is processed and distributed based on the logical chain. The key-value pair of the sub-data packet and the ID forming a data packet can facilitate subsequent query data, that is, input the corresponding index, the data engine can correspond to the corresponding data packet according to the index, and then each sub-data packet and the processing result are summarized and sent to the querying user from each target processing node. As for the formation of a logical chain between each target processing node and the data engine based on the key-value pair list, and the distribution of each sub-data packet based on the logical chain, it is because the processing process of each sub-data packet may depend on other sub-data packets, so the corresponding logical relationship can be obtained in advance, and then the logical chain processing relationship is established, and each sub-data packet is distributed, so as to complete the efficient block processing of the data packet to be processed, and multiple target processing nodes with better efficiency are selected to process the data packet to be processed at the same time, without occupying the processing space of a single processing node, which greatly improves the processing efficiency.

[0077] In one embodiment, the step S3 of obtaining the processing efficiency of each processing node and determining the target processing node and the processing ratio of each target processing node according to the processing efficiency includes:

[0078] S301: monitoring the number of processing flows of each processing node at a preset interval; wherein the number of processing flows is used to feedback the processing efficiency of the processing node;

[0079] S302: Compare the current processing flow number of each processing node closest to the current time with the preset relaxation flow number;

[0080] S303: selecting a target processing node from the processing nodes according to the comparison result, and setting conversion parameters of each of the target processing nodes;

[0081] S304: Setting the processing ratio of each target processing node based on the conversion parameter.

[0082] As described in the above steps S301-S304, the processing ratio of each target processing node is set. The number of processing flows monitored at each processing node at preset intervals is obtained, wherein the number of processing flows is the amount of data processed by the processing node per unit time, and then the current number of processing flows closest to the current time of each processing node is compared with the preset number of relaxation flows. Different conversion parameters can be set according to different comparison results. The larger the number of processing flows, the faster the processing efficiency of the corresponding target processing node, and vice versa, the slower the processing efficiency of the corresponding target processing node. The conversion parameter can further explain the processing efficiency of the target processing node, that is, due to different data, the processing efficiency of the target processing node will also be different. The conversion parameter is a balanced evaluation of the processing efficiency, which is used as an indicator of the processing efficiency, and then the processing ratio of each target processing node is set based on the conversion parameter. The setting method can set the processing ratio of each target processing node according to the ratio of the conversion parameter.

[0083] In one embodiment, the step S303 of setting the conversion parameters of each of the target processing nodes includes:

[0084] S3031: If the current processing flow number is greater than the preset relaxation flow number, the conversion parameter of the corresponding section is set to a fixed value 1;

[0085] S3032: If the current processing flow number is not greater than the preset relaxation flow number, extract multiple processing flow numbers that are closest to the current time and greater than the preset relaxation flow number, and record them as the target processing flow number set;

[0086] S3033: Multiply the number of each processing flow in the target processing flow number set by the size of the corresponding processing data packet to obtain the data capacity set T of each target processing node. i (t 1 ), T i (t 2 ),...,T i (t k ),...,T i (t l ), where T i (t k ) represents the time t k The data capacity of the i-th target processing node, and t 1 <t 2 <... <t k <... <t l , t k -t k-1 =t a , t a is the interval time;

[0087] S3034: Input each element of each data capacity set into the fitting formula T i (l) = T i (k)+A i [T i (l-1)-T i (k-1)], the conversion parameter A corresponding to each data capacity set is obtained i Among them, A i The conversion parameter A represents the data capacity set corresponding to the i-th target processing node i .

[0088] As described in the above steps S3031-S3034, for some target processing nodes with excellent processing capabilities, their conversion parameters can be set to 1 to avoid all data being processed by the same target processing node. If the current processing flow number is not greater than the preset relaxation flow number, multiple processing flow numbers that are closest to the current time and greater than the preset relaxation flow number are extracted and recorded as the target processing flow number set. Each processing flow number in the target processing flow number set is multiplied by the size of the corresponding processing data packet to obtain the data capacity set T of each target processing node. i (t 1 ), T i (t 2 ),...,T i (t k ),...,T i (t l ). Calculate the conversion parameter A corresponding to each data capacity set through the data capacity set i .

[0089] In one embodiment, the step S4 of dividing the to-be-processed data packet into corresponding sub-data packets according to the processing ratio includes:

[0090] S401: Determine a plurality of pre-segmentation points in the data packet to be processed according to the processing ratio;

[0091] S402: based on the position of each pre-segmentation point in the data packet to be processed, find the delimiter closest to each pre-segmentation point as the target segmentation point;

[0092] S403: Split the data packet to be processed into corresponding sub-data packets based on the target segmentation point.

[0093] As described in the above steps S401-S403, the segmentation of sub-data packets is achieved, that is, multiple pre-segmentation points in the data packet to be processed are determined according to the processing ratio, and then based on the position of each pre-segmentation point in the data packet to be processed, the delimiter closest to each pre-segmentation point is found as the target segmentation point. Since there will be multiple delimiters in the data packet, the position of each delimiter can be obtained in advance, and then according to the position of the pre-segmentation point, a similar delimiter is found as the target segmentation point to ensure the integrity of the data and facilitate each target processing node to analyze and process the sub-data packet.

[0094] In one embodiment, the step S9 of forming a logical chain between each target processing node and the data engine based on the key-value pair list includes:

[0095] S901: forming a preliminary logical chain between each target processing node and the data engine based on the key-value pair list;

[0096] S902: Acquire data information of sub-data packets processed by each target processing node;

[0097] S903: Setting a processing logic scheme for each target processing node according to each piece of data information;

[0098] S904: Send the processing logic solution to the preliminary logic chain to obtain the logic chain.

[0099] As described in the above steps S901-S904, the processing of the segmented sub-data packets is implemented, and the key-value pair list is used to form a preliminary logical chain between each target processing node and the data engine. The logical chain can be used in the process of reasoning, decision-making or problem solving. Different thinking steps, decision points or information sources can be regarded as nodes. Therefore, each target processing node can be used as a node of the logical chain. In some embodiments, if the user needs to view the data packet to be processed, the preliminary logical chain can be extracted and connected through the key-value pair list to obtain the data packet to be processed and the processing result. Then, a processing logic scheme is set for each target processing node according to each data information. The specific processing logic scheme needs to be determined according to the data type of the data packet to be processed and the segmentation situation. It can be designed according to the background. This application does not limit this. Then, the processing logic scheme is sent to the preliminary logical chain to obtain the logical chain.

[0100] In one embodiment, after the step S9 of distributing and processing each of the sub-data packets based on the logical chain processing, the method further includes:

[0101] S1001: The Merkle tree root of the key-value pairs pre-stored at each target processing node obtained in the data engine is updated through a preset hash algorithm to obtain a target Merkle tree root; wherein, the target Merkle tree root is used to obtain the corresponding target processing node storing the sub-data packet according to the index query of the user, so as to obtain the data packet to be processed and its processing result.

[0102] As described in step S1001 above, the Merkle tree is a binary tree that includes a group of nodes. The Merkle tree root has a large number of leaf nodes and a group of intermediate nodes. Each node is the hash of its two child nodes, and then the final root node is also formed by the hash of its two child nodes, that is, the Merkle tree root. In the subsequent retrieval process, you only need to compare the target Merkle tree root to find the corresponding key-value pair, determine the target processing node, and thus obtain the corresponding data packet to be processed and its processing result.

[0103] In one embodiment, after the step S9 of distributing and processing each of the sub-data packets based on the logical chain processing, the method further includes:

[0104] S1011: selecting multiple audit nodes from multiple processing nodes according to a preset selection method;

[0105] S1012: Sending the hash value calculated for each sub-data packet to each audit node;

[0106] S1013: Obtain sub-data packets in each target processing node based on the data engine, calculate their target hash values ​​and send them to each audit node;

[0107] S1014: The audit node audits the target hash value based on the hash value, and sends the audit result to the data engine.

[0108] As described in the above steps S1011-S1014, the processing sub-packets in each target processing node are monitored, that is, multiple audit nodes are selected, and the audit nodes can be target processing nodes or other processing nodes. The specific selection method is not limited and can be any selection method. Then, the hash value calculated for each sub-packet is sent to each audit node, and the sub-packets in each target processing node are obtained based on the data engine, and the target hash value is calculated and sent to each audit node. The audit node audits the target hash value based on the hash value and sends the audit result to the data engine.

[0109] Reference Figure 2 The present invention also provides a data processing device of a data engine, comprising:

[0110] The receiving module 10 is used to receive the data packet to be processed through the data engine;

[0111] A determination module 20, configured to determine a plurality of processing nodes based on the data packet to be processed;

[0112] An acquisition module 30 is used to acquire the processing efficiency of each processing node, and determine the target processing node and the processing ratio of each target processing node according to the processing efficiency;

[0113] A segmentation module 40, configured to segment the data packet to be processed into corresponding sub-data packets according to the processing ratio;

[0114] A first sending module 50, configured to send the sub-data packet to a corresponding target processing node;

[0115] A calculation module 60, used to calculate the hash value of the corresponding processed sub-data packet through the target processing node, and form a key-value pair with the ID;

[0116] A second sending module 70, each of the target processing nodes is used to send the key-value pairs to the data engine to form a key-value pair list;

[0117] A forming module 80, for forming a logical chain between each target processing node and the data engine based on the key-value pair list;

[0118] The processing module 90 is used to distribute the processing of each of the sub-data packets based on the logical chain processing.

[0119] In one embodiment, the acquisition module 30 includes:

[0120] A processing flow number monitoring submodule is used to monitor the processing flow number of each processing node at a preset interval; wherein the processing flow number is used to feedback the processing efficiency of the processing node;

[0121] A processing flow number comparison submodule, used for comparing the current processing flow number of each processing node closest to the current time with a preset relaxation flow number;

[0122] A target processing node selection submodule, used to select a target processing node from the processing nodes according to the comparison result, and to set conversion parameters of each of the target processing nodes;

[0123] The processing ratio setting submodule is used to set the processing ratio of each target processing node based on the conversion parameters.

[0124] In one embodiment, the target processing node selects a submodule, including:

[0125] A setting unit, used for setting the conversion parameter of the corresponding section to a fixed value 1 if the current processing flow number is greater than the preset relaxation flow number;

[0126] an extracting unit, configured to extract a plurality of processing flow numbers closest to the current time and greater than the preset relaxation flow number if the current processing flow number is not greater than the preset relaxation flow number, and record them as a target processing flow number set;

[0127] A calculation unit is used to multiply the number of each processing flow in the target processing flow number set by the size of the corresponding processing data packet to obtain a data capacity set T of each target processing node. i (t 1 ), T i (t 2 ),...,T i (t k ),...,T i (t l ), where T i (t k ) represents the time t k The data capacity of the i-th target processing node, and t 1 <t 2 <... <t k <... <t l , t k -t k-1 =t a , t a is the interval time;

[0128] An input unit is used to input each element of each data capacity set into the fitting formula T i (l) = T i (k)+A i [T i (l-1)-T i (k-1)], the conversion parameter A corresponding to each data capacity set is obtained i Among them, A i The conversion parameter A represents the data capacity set corresponding to the i-th target processing node i .

[0129] In one embodiment, the segmentation module 40 includes:

[0130] A pre-segmentation point determination submodule, used to determine a plurality of pre-segmentation points in the data packet to be processed according to the processing ratio;

[0131] A target segmentation point setting submodule is used to find the delimiter closest to each pre-segmentation point as the target segmentation point based on the position of each pre-segmentation point in the data packet to be processed;

[0132] The segmentation submodule is used to segment the data packet to be processed into corresponding sub-data packets based on the target segmentation point.

[0133] In one embodiment, the module 80 is formed, including:

[0134] A preliminary logic chain forming submodule, used for forming a preliminary logic chain between each target processing node and the data engine based on the key-value pair list;

[0135] A data information acquisition submodule, used to acquire data information of sub-data packets processed by each target processing node;

[0136] A processing logic scheme setting submodule is used to set a processing logic scheme for each target processing node according to various data information;

[0137] The processing logic scheme sending submodule is used to send the processing logic scheme to the preliminary logic chain to obtain the logic chain.

[0138] In one embodiment, the data processing device of the data engine further includes:

[0139] An update module is used to update the Merkle tree root of the key-value pairs pre-stored at each target processing node obtained in the data engine through a preset hash algorithm to obtain a target Merkle tree root; wherein the target Merkle tree root is used to obtain the corresponding target processing node storing the sub-data packet according to the index query of the user, so as to obtain the data packet to be processed and its processing result.

[0140] In one embodiment, the data processing device of the data engine further includes:

[0141] An audit node selection module is used to select multiple audit nodes from multiple processing nodes according to a preset selection method;

[0142] A hash value sending module is used to send the hash value calculated for each sub-data packet to each audit node;

[0143] The sub-data packet acquisition module is used to acquire the sub-data packets in each target processing node based on the data engine, calculate the target hash value and send it to each audit node;

[0144] An audit module is used for the audit node to audit the target hash value based on the hash value and send the audit result to the data engine.

[0145] The beneficial effects of the present invention are as follows: by determining multiple processing nodes based on the data packet to be processed, and then determining multiple target processing nodes according to the processing efficiency, the data packet to be processed is divided into multiple sub-data packets and sent to each target processing node for processing, thereby completing efficient block processing of the data packet to be processed, and selecting multiple target processing nodes with better efficiency to process the data packet to be processed at the same time, without occupying the processing space of a single processing node, thereby greatly improving the processing efficiency.

[0146] Reference Figure 3 In an embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a processor, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The processor of the computer device includes a non-volatile processing medium and an internal processor. The non-volatile processing medium processes an operating system, a computer program and a database. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile processing medium. The database of the computer device is used to process various sub-data packets, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the data processing method of the data engine described in any of the above embodiments can be implemented.

[0147] Those skilled in the art will understand that Figure 3 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0148] An embodiment of the present application also provides a computer-readable processing medium on which a computer program is processed. When the computer program is executed by a processor, the data processing method of the data engine described in any of the above embodiments can be implemented.

[0149] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be processed in a non-volatile computer-readable processing medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the processor, processing, database or other medium provided in the present application and used in the embodiments can include non-volatile and / or volatile processors. Non-volatile processors can include read-only processors (ROM), programmable ROMs (PROM), electrically programmable ROMs (EPROM), electrically erasable programmable ROMs (EEPROM) or flash memory. Volatile processors can include random access processors (RAM) or external cache processors. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), processor bus (Rambus) direct RAM (RDRAM), direct processor bus dynamic RAM (DRDRAM), and processor bus dynamic RAM (RDRAM).

[0150] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0151] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0152] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed processing, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.

Claims

1. A data processing method for a data engine, characterized in that: include: receiving a data packet to be processed through a data engine; Determining a plurality of processing nodes based on the data packets to be processed; Obtain the processing efficiency of each processing node, and determine the target processing node and the processing ratio of each target processing node according to the processing efficiency; Dividing the data packet to be processed into corresponding sub-data packets according to the processing ratio; Sending the sub-data packet to the corresponding target processing node; The target processing node calculates the hash value of the corresponding processed sub-packet and forms a key-value pair with the ID; Each of the target processing nodes sends the key-value pair to the data engine to form a key-value pair list; Based on the key-value pair list, each target processing node and the data engine form a logical chain; Distribute and process each of the sub-data packets based on the logical chain processing.

2. The data processing method of the data engine according to claim 1, characterized in that: The step of obtaining the processing efficiency of each processing node and determining the target processing node and the processing ratio of each target processing node according to the processing efficiency includes: Monitoring the number of processing flows of each processing node at preset intervals; wherein the number of processing flows is used to provide feedback on the processing efficiency of the processing node; Compare the current processing flow number of each processing node closest to the current time with the preset relaxation flow number; Selecting a target processing node from the processing nodes according to the comparison result, and setting conversion parameters of each of the target processing nodes; The processing ratio of each target processing node is set based on the conversion parameter.

3. The data processing method of the data engine according to claim 2, characterized in that: The step of setting the conversion parameters of each of the target processing nodes comprises: If the current processing flow number is greater than the preset relaxation flow number, the conversion parameter of the corresponding section is set to a fixed value of 1; If the current processing flow number is not greater than the preset relaxation flow number, then extract multiple processing flow numbers that are closest to the current time and greater than the preset relaxation flow number, and record them as the target processing flow number set; The data capacity set T of each target processing node is obtained by multiplying the number of each processing flow in the target processing flow set by the size of the corresponding processing data packet. i (t1), T i (t2), ..., T i (t k ),...,T i (t l ), where T i (t k ) represents the time t k The data capacity of the i-th target processing node, and t1 <t2<...<t k <... <t l , t k -t k-1 =t a , t a is the interval time; Each element in each of the data capacity sets is input into the fitting formula T i (l) = T i (k)+A i [T i (l-1)-T i (k-1)], the conversion parameter A corresponding to each data capacity set is obtained i Among them, A i The conversion parameter A represents the data capacity set corresponding to the i-th target processing node i .

4. The data processing method of the data engine according to claim 1, characterized in that: The step of dividing the to-be-processed data packet into corresponding sub-data packets according to the processing ratio comprises: Determining a plurality of pre-segmentation points in the data packet to be processed according to the processing ratio; Based on the position of each pre-segmentation point in the data packet to be processed, finding the delimiter closest to each pre-segmentation point as the target segmentation point; The data packet to be processed is divided into corresponding sub-data packets based on the target division point.

5. The data processing method of the data engine according to claim 1, characterized in that: The step of forming a logical chain between each target processing node and the data engine based on the key-value pair list includes: Based on the key-value pair list, each target processing node and the data engine form a preliminary logical chain; Obtain data information of sub-data packets processed by each target processing node; Set a processing logic plan for each target processing node according to each data information; The processing logic scheme is sent to the preliminary logic chain to obtain the logic chain.

6. The data processing method of the data engine according to claim 1, characterized in that: After the step of distributing and processing each of the sub-data packets based on the logic chain, the method further includes: The Merkle tree root of the key-value pairs pre-stored at each target processing node in the data engine is updated through a preset hash algorithm to obtain a target Merkle tree root; wherein the target Merkle tree root is used to obtain the corresponding target processing node storing the sub-data packet according to the index query of the user, so as to obtain the data packet to be processed and its processing result.

7. The data processing method of the data engine according to claim 1, characterized in that: After the step of distributing and processing each of the sub-data packets based on the logic chain, the method further includes: Selecting multiple audit nodes from multiple processing nodes according to a preset selection method; Send the hash value calculated for each sub-data packet to each audit node; Based on the data engine, the sub-data packets in each target processing node are obtained, the target hash values ​​are calculated and sent to each audit node; The audit node audits the target hash value based on the hash value and sends the audit result to the data engine.

8. A data processing device for a data engine, characterized in that: include: A receiving module, used for receiving a data packet to be processed through a data engine; A determination module, configured to determine a plurality of processing nodes based on the data packet to be processed; An acquisition module is used to acquire the processing efficiency of each processing node, and determine the target processing node and the processing ratio of each target processing node according to the processing efficiency; A segmentation module, used for segmenting the data packet to be processed into corresponding sub-data packets according to the processing ratio; A first sending module, used for sending the sub-data packet to a corresponding target processing node; A calculation module, used to calculate the hash value of the corresponding processed sub-data packet through the target processing node, and form a key-value pair with the ID; A second sending module, each of the target processing nodes is used to send the key-value pairs to the data engine to form a key-value pair list; A forming module, used to form a logical chain between each target processing node and the data engine based on the key-value pair list; A processing module is used to distribute the processing of each of the sub-data packets based on the logical chain processing.

9. A computer device comprising a processor and a processor, wherein the processor processes a computer program, characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable processing medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images