A method for massive secret data calculation

Abstract

The application discloses a kind of methods for massive secret data calculation, it is related to information security technical field, the method includes: obtaining multiple JSON data packets and is serialized, obtain multiple pieces of data after serialization;The multiple pieces of data after serialization are formatted, and obtain first data packet;According to first data packet, construct unidirectional graph, calculate the weight of each node in unidirectional graph, and according to the weight of each node, establish weight red-black tree;The node corresponding to the minimum weight in weight red-black tree is input to thread pool for calculation, and the calculation result is obtained;According to the calculation result, update unidirectional graph, and then recalculate the weight of each node in the updated unidirectional graph, determine the updated weight red-black tree;The node corresponding to the minimum weight in the updated weight red-black tree is input to thread pool, and the next calculation is waited, solve the problem that message fragmentation is serious, communication waiting time delay is longer, realize balanced communication and calculation overhead, and high concurrency, high throughput.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and in particular to a method for computing massive amounts of dense data. Background Technology

[0002] In recent years, machine learning has developed rapidly and has been widely applied in fields such as image processing, speech recognition, disease diagnosis, and autonomous driving. With the development of storage and network technologies, platforms such as hospitals, banks, e-commerce, and social media generate massive amounts of data daily. Data holders can collect this data to train models, which may contribute to future decision-making and provide predictive and recommendation services to customers. Training machine learning models requires collecting massive amounts of data and is time-consuming. In a typical MLaaS scenario, customers provide data to a server, which then returns the results after making a prediction. However, customers using these services may not want to disclose their sensitive data, making privacy protection increasingly important. The rise of technologies such as secure multi-party computation, homomorphic encryption, and differential privacy has driven the development of the privacy protection field. While introducing secure multi-party computation into MLaaS can alleviate security issues, it increases communication and computational overhead. Balancing the costs of communication and computation with high accuracy and efficiency has become a current research hotspot.

[0003] MP-SPDZ, an existing open-source library for secure multi-party computation, has been used by Marcel Keller et al. to develop different MPC security protocols and test their performance. It incorporates various secure multi-party computation technologies, such as obfuscated circuits, secret sharing, operational technology (OT), and homomorphism, primarily in Python. However, MP-SPDZ suffers from severe message fragmentation, resulting in numerous interaction rounds and long communication latency when the circuit depth is large. When processing concurrent tasks, the system experiences immense communication pressure, causing computational tasks to stall, leading to an imbalance between communication and computational overhead and hindering the realization of computational efficiency. Summary of the Invention

[0004] This invention provides a method for computing massive dense data, which solves the problems of severe message fragmentation, long communication latency, and delayed computing tasks in the prior art, as well as the imbalance between communication and computing overhead, making it difficult to achieve computing efficiency. It achieves a balance between communication and computing overhead and has the characteristics of high concurrency and high throughput.

[0005] This invention provides a method for computation of massive dense-state data, the method comprising:

[0006] Obtain multiple JSON data packets and serialize the multiple JSON data packets to obtain multiple serialized data items;

[0007] The serialized data is formatted to obtain the first data packet;

[0008] A one-way graph is constructed based on the first data packet, the weight of each node in the one-way graph is calculated, and a weighted red-black tree is established based on the weight of each node.

[0009] The node corresponding to the smallest weight in the weighted red-black tree is input into the thread pool for calculation to obtain the calculation result;

[0010] The one-way graph is updated based on the calculation results, and then the weights of each node in the updated one-way graph are recalculated to determine the updated weighted red-black tree.

[0011] The node corresponding to the smallest weight in the updated weighted red-black tree is input into the thread pool and awaits the next calculation.

[0012] In one possible implementation, the step of acquiring multiple JSON data packets and serializing the multiple JSON data packets to obtain multiple serialized data items specifically includes:

[0013] A timer is used to periodically package all the data in the receive buffer into multiple JSON data packets.

[0014] Using batch processing technology, the data in the multiple JSON data packets are processed to determine the business logic and operation order of each piece of data in the multiple JSON data packets, thus obtaining the first processing stage data packet;

[0015] The data packets of the first processing stage are divided into data packets of the second processing stage, which have a hierarchical structure and a coupled arrangement.

[0016] By utilizing the principle that data packets at the same layer do not depend on each other, and data packets at lower layers depend on data packets at higher layers, the data in the data packets of the second processing stage are decoupled and rearranged to obtain multiple serialized data packets.

[0017] In one possible implementation, the serialized data is formatted to obtain a first data packet, specifically including:

[0018] The serialized data is concatenated to obtain the concatenated data.

[0019] Create field names for the concatenated data, and combine the field names with the concatenated data to form a new data entry;

[0020] The new data is added to the serialized data to obtain the first data packet.

[0021] In one possible implementation, before constructing the unidirectional graph based on the first data packet, the method further includes:

[0022] Set a primary key for each piece of data in the first data packet and add a dependency field.

[0023] In one possible implementation, each node in the one-way graph is a key of the first data packet, and the edges of the one-way graph are dependencies corresponding to the nodes.

[0024] In one possible implementation, the specific formula for calculating the weight is as follows:

[0025] W = w * Q

[0026] Where w represents the initial weight of the node; Q represents the in-degree ratio of the node. I represents the in-degree of a node, and O represents the out-degree of a node.

[0027] In one possible implementation, the step of constructing a weighted red-black tree based on the weights of each node specifically includes:

[0028] The weights of each node are sorted to obtain weights in ascending order;

[0029] The weights arranged in ascending order are respectively attached to the red-black tree. Nodes with weights less than the first threshold are attached to the left tree of the weighted red-black tree, and nodes with weights greater than or equal to the first threshold are attached to the right tree of the weighted red-black tree, thus establishing the weighted red-black tree.

[0030] In one possible implementation, the weight ranges from [0,1], and the smaller the weight value, the higher its importance.

[0031] In one possible implementation, the step of inputting the node corresponding to the smallest weight in the weighted red-black tree into a thread pool for calculation to obtain the calculation result specifically includes:

[0032] Determine whether the node of the weighted red-black tree currently being calculated in the thread pool depends on its parent node, and obtain the determination result;

[0033] If the judgment result is negative, then the calculation is performed directly;

[0034] If the judgment result is yes, then it is determined whether the dependency corresponding to the parent node has reached the current calculation node. If yes, the calculation is performed to obtain the calculation result; otherwise, the thread where the current node is located is suspended in the thread pool, waiting for the dependency corresponding to the parent node to arrive.

[0035] One or more technical solutions provided in this invention have at least the following technical effects or advantages:

[0036] This invention employs a method for computing massive amounts of dense data. The method includes: acquiring multiple JSON data packets and serializing them to obtain multiple serialized data entries; using batch processing technology to process a large number of data packets with the same logic, saving data sorting time and costs; and integrating data packets within a period of time for unified transmission, reducing communication frequency and increasing communication payload; using a multi-level scheduling strategy to update scheduling weight levels in real time, always selecting the most important nodes for computation, thereby accelerating computation speed; formatting the serialized data entries to obtain a first data packet; formatting the data to obtain new, information-rich fields, integrating information, and accelerating processing speed; constructing a one-way graph based on the first data packet; calculating the weight of each node in the one-way graph; and building a weighted red-black tree based on the weights of each node; prioritizing important nodes based on their weights, and processing the upper-layer data packets... The algorithm breaks down data packets into small, hierarchical packets, decomposing and rearranging as many upper-layer packets as possible in a short time to ensure the logical order of packet processing in the lower layers. The node with the smallest weight in the weighted red-black tree is input into a thread pool for computation, yielding the result. The unidirectional graph is updated based on the result, and the weights of each node in the updated unidirectional graph are recalculated to determine the updated weighted red-black tree. The node with the smallest weight is selected for computation each time, ensuring that the most important node in the tree is chosen each time, thereby increasing the number of concurrent tasks that can be performed in the next time period and improving computation speed. The node with the smallest weight in the updated weighted red-black tree is input into the thread pool to await the next computation. This effectively solves the problems of severe message fragmentation, long communication latency, and computational task delays in existing technologies, as well as the imbalance between communication and computational overhead, which hinders computational efficiency. It achieves a balance between communication and computational overhead, resulting in high concurrency and high throughput. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the method steps for computing massive dense data provided in an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of establishing a unidirectional graph for hierarchical dependencies provided in an embodiment of the present invention;

[0040] Figure 3This is a flowchart illustrating the steps for updating the red-black diagram according to an embodiment of the present invention;

[0041] Figure 4 This is a flowchart illustrating the steps of using the method in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0043] This invention provides a method for computation of massive dense-state data, such as... Figure 1 As shown, the method includes the following steps S101 to S106.

[0044] S101: Retrieve multiple JSON data packets and serialize them to obtain multiple serialized data items, specifically including:

[0045] (1) Use a timer to package all the data in the receiving buffer into multiple JSON data packets.

[0046] (2) Using batch processing technology, the data in multiple JSON data packets are processed to determine the business logic and operation order of each data in multiple JSON data packets, and the data packets of the first processing stage are obtained.

[0047] (3) Divide the data packets of the first processing stage into data packets of the second processing stage with a hierarchical structure and coupled arrangement.

[0048] (4) Taking advantage of the principle that data packets at the same layer do not depend on each other and data packets at the lower layer depend on data packets at the upper layer, the data in the data packets of the second processing stage are decoupled and rearranged to obtain multiple serialized data.

[0049] Batch processing technology is used to process a large number of data packets with the same logic, saving time and costs in data sorting; data packets are integrated and sent uniformly within a period of time, reducing the number of communications and increasing the effective communication payload; a multi-level scheduling strategy is used to update the scheduling weight level in real time and always select the most important node for computation, thereby speeding up the calculation.

[0050] S102, the serialized data is formatted to obtain the first data packet, which specifically includes:

[0051] (1) Consolidate the serialized data to obtain the concatenated data.

[0052] (2) Create field names for the concatenated data, and combine the field names with the concatenated data to form a new data entry.

[0053] (3) Add the new data to the serialized data to obtain the first data packet.

[0054] In a specific embodiment of the present invention, the field names for creating the concatenated data include:

[0055] Construct the Taskid field to uniquely identify the task number;

[0056] Construct a Levelid field to uniquely identify the task level number;

[0057] Construct the Func_list field to identify the list of functions in the hierarchy;

[0058] Construct the Func_name field to identify the function name of the function body in the function list;

[0059] Create the Inputs field to identify the input data for the function body;

[0060] Create the Outputs field to identify the output data of the function body;

[0061] Create a Shareid field to uniquely identify the input and output data within the function body;

[0062] Construct the Is_send field to indicate whether the output data in the function body is sent to other parties;

[0063] Construct the Is_received field to indicate whether the input data in the function body needs to be received from other parties;

[0064] Construct the Is_return field to indicate whether the output data in the function body needs to be returned to the interpretation layer;

[0065] Construct a Shape field to identify the dimensions of the input and output data in the function body.

[0066] The meanings of the fields are shown in Table 1.

[0067] Table 1: Field Meanings

[0068] LEVELID Task level identifier FUNC_LIST Function List FUNC_NAME function name INPUTS Function Input OUTPUTS Function output SHAREID Data Identifier DATA Data content IS_SEND Send or not? IS_RECEIVE Receive or not? IS_RETURN Return SHAPE Data Dimensions

[0069] Before step S103, the process also includes: setting primary keys for each piece of data in the first data packet and adding dependency fields.

[0070] The upper-layer data packets are broken down into smaller data packets with a hierarchical structure. As many upper-layer data packets as possible are broken down and decoupled and rearranged in a short time to ensure the logical order of packet processing in the lower layer.

[0071] Maintain a hash table to store various data issued by the task building module, such as task ID (Taskid), task level ID (Levelid), function name (Func_name), and data ID (Shareid).

[0072] For the visible "tasks" at the upper layer, the task partitioning is decoupled and rearranged, and transformed into "data packets" (Msg). For each data packet, the task identifier and level identifier are used as the primary key, the above fields are added, and a dependency field is added to identify the task identifier (Taskid) and task level identifier (Levelid) of the data packets it depends on.

[0073] Data packets with dependency fields need to be checked for arrival; a hierarchy table is maintained. Data packets without dependencies can be processed concurrently. The hierarchy and data packets must have the following characteristics: data packets within the same layer do not depend on each other; lower-level data packets only depend on higher-level data packets.

[0074] S103. Construct a one-way graph based on the first data packet, calculate the weight of each node in the one-way graph, and build a weighted red-black tree based on the weights of each node. Each node in the one-way graph is a key of the first data packet, and the edges of the one-way graph represent the dependencies of the nodes. The specific formula for calculating the weights is as follows:

[0075] W = w * Q

[0076] Where w represents the initial weight of the node; Q represents the in-degree ratio of the node. I represents the in-degree of a node, and O represents the out-degree of a node.

[0077] In step S103, a weighted red-black tree is constructed based on the weights of each node, specifically including:

[0078] (1) Sort the weights of each node to obtain the weights in ascending order. The weights range from [0,1]. The smaller the weight value, the higher the importance.

[0079] (2) Attach the weights in ascending order to the red-black tree respectively, attach the nodes with weights less than the first threshold to the left tree of the weight red-black tree, and attach the nodes with weights greater than or equal to the first threshold to the right tree of the weight red-black tree to build the weight red-black tree.

[0080] In a specific embodiment of the present invention, step S103 is divided into a task execution phase. During the task execution phase, a multi-level scheduling strategy is used to efficiently compute the distributed tasks. The multi-level scheduling strategy is divided into an initialization phase and a computation phase.

[0081] During the initialization phase, each task in Msg is treated as a node, and its dependencies are treated as edges to create a one-way connected graph. The tasks are then placed into the task queue according to their priority.

[0082] During the computation phase, concurrent computation is performed using thread pools and callback functions, with priorities changed in real time to wake up or suspend tasks.

[0083] Initialization phase: A unidirectional graph is established based on the hierarchical dependencies of S102, such as... Figure 2 As shown, the in-degree and out-degree of a node define the node's weight.

[0084] In a specific embodiment of the present invention, a red-black tree is maintained, and nodes are mounted according to their weight values. A timer is set up, and a receive queue and a send queue are maintained. A thread starts listening to the window and waits for the other party's data packet to arrive. Nodes with a weight value of 0 are inserted into the queue and multi-threaded calculations are performed.

[0085] S104: Input the node with the smallest weight in the weighted red-black tree into the thread pool for calculation, and obtain the calculation result. Specifically, this includes:

[0086] (1) Determine whether the nodes of the weighted red-black tree currently being calculated in the thread pool are dependent on their parent nodes, and obtain the determination result.

[0087] (2) If the judgment result is negative, then perform the calculation directly.

[0088] (3) If the judgment result is yes, then determine whether the dependency corresponding to the parent node has reached the current calculation node. If yes, perform the calculation and obtain the calculation result; otherwise, suspend the thread where the current node is located in the thread pool and wait for the dependency corresponding to the parent node to arrive.

[0089] During computation, a dedicated thread maintains a weighted red-black tree. Each time a computation is performed, the node with the smallest weight in the weighted red-black tree is input into a thread pool. Threads in the thread pool retrieve the input node, perform the computation, and then retrieve the input node again to continue the computation. After completing one computation, a thread in the thread pool retrieves the input node again and continues the computation.

[0090] S105, update the one-way graph based on the calculation results, and then recalculate the weights of each node in the updated one-way graph to determine the updated weighted red-black tree.

[0091] In a specific embodiment provided by the present invention, such as Figure 3As shown, the process first receives task requests, constructs and divides tasks, and establishes a directed graph and a red-black tree. The hierarchical task queue is then placed into a thread pool, where multiple threads begin computation simultaneously. The data at the same level is first cyclically computed to obtain the results. The directed graph is then updated based on the results, i.e., the storage table is updated. Data to be sent is placed into the sending buffer. Data requiring checks on whether dependencies have arrived is retrieved from the receiving buffer, thus achieving asynchronous communication.

[0092] By selecting the node with the smallest weight for each calculation, the most important node in the tree is always chosen, thereby increasing the number of concurrent tasks that can be performed in the next time period and improving the calculation speed. The entire task execution phase is characterized by high concurrency, high computational performance, and low communication overhead.

[0093] S106: Input the node corresponding to the smallest weight in the updated weighted red-black tree into the thread pool, waiting for the next calculation.

[0094] By employing multi-level scheduling technology and updating the weights of each task node in real time, multi-threaded computation is performed, improving task concurrency and computational efficiency. Asynchronous communication is used to integrate as many processed data packets as possible within a short time, thereby reducing the number of communication calls and thus minimizing communication overhead.

[0095] In a specific embodiment provided by the present invention, such as Figure 4 As shown.

[0096] Step 1: Obtain the ADD and XOR commands sent by the interpretation layer, and parse the JSON to obtain multiple serialized data entries.

[0097] Step 2 involves the field construction module storing numerical values ​​as key-value pairs in a hash table (Map). For example, the task ADD will be decomposed into three functions: Share_a, Add, and Reveal_a, corresponding to level numbers 1, 2, and 3 respectively. Similarly, the task XOR will be decomposed into three functions: Share_b, Xor, and Reveal_b, corresponding to level numbers 1, 2, and 3 respectively.

[0098] Step 3: Use a Map to store information such as task identifier, layer identifier, function name, etc.

[0099] Step 4: Decouple the tasks. Construct a unique index from Task_id and Level_id to identify a minimum tuple. Add a dependency field; for example, tuple numbered 1-2 depends on the calculation result of tuple 1-1. Store other values. Reorder the data packets. Data packets with dependency fields need to be checked for arrival or successful calculation. For example, Share_a in ADD and Share_b in XOR can be merged into the first level; Add depends on the result of Share_a, and Xor depends on Share_b, so they can be merged and placed in the second level, and so on.

[0100] Step 5: Build a one-way graph based on the decoupled tasks.

[0101] Step 6: Construct a weighted red-black tree corresponding to the one-way graph. Update the out-degree and in-degree of nodes based on the calculation results. Recalculate the weights of adjacent nodes and add them to the task queue to await the next calculation. Send signals after each layer's calculation and receive and send data packets.

[0102] This invention constructs a three-layer scheduling structure that supports concurrent multi-task privacy-preserving computation, consisting of three layers: task construction, task partitioning, and task execution. It balances the overhead of communication and computation, thereby improving the throughput and computational efficiency of the entire system.

[0103] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0104] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this invention can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0105] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A method for computation of massive dense-state data, characterized in that, include: Obtain multiple JSON data packets and serialize the multiple JSON data packets to obtain multiple serialized data items; The serialized data is formatted to obtain the first data packet; Set a primary key for each data entry in the first data packet and add a dependency field; A one-way graph is constructed based on the first data packet. The weight of each node in the one-way graph is calculated, and a weighted red-black tree is built based on the weights of each node. Each node in the one-way graph is a key of the first data packet, and the edges of the one-way graph are the dependencies corresponding to each node. The specific formula for calculating the weights is as follows: ;in, Indicates the initial weight of the node; This represents the in-degree ratio of a node. , Indicates the in-degree of a node. Indicates the out-degree of the node; The node corresponding to the smallest weight in the weighted red-black tree is input into a thread pool for calculation to obtain the calculation result. Specifically, this process includes: determining whether the node currently being calculated in the weighted red-black tree in the thread pool has a dependency on a parent node; if the determination result is negative, calculation is performed directly; if the determination result is positive, it is determined whether the dependency corresponding to the parent node has reached the currently calculated node; if so, calculation is performed to obtain the calculation result; otherwise, the thread containing the current node is suspended in the thread pool, waiting for the dependency corresponding to the parent node to arrive. The one-way graph is updated based on the calculation results, and then the weights of each node in the updated one-way graph are recalculated to determine the updated weighted red-black tree. The node corresponding to the smallest weight in the updated weighted red-black tree is input into the thread pool and awaits the next calculation.

2. The method for computing massive dense data according to claim 1, characterized in that, The process of acquiring multiple JSON data packets and serializing them to obtain multiple serialized data items specifically includes: A timer is used to periodically package all the data in the receive buffer into multiple JSON data packets. Using batch processing technology, the data in the multiple JSON data packets are processed to determine the business logic and operation order of each piece of data in the multiple JSON data packets, thus obtaining the first processing stage data packet; The data packets of the first processing stage are divided into data packets of the second processing stage, which have a hierarchical structure and a coupled arrangement. By utilizing the principle that data packets at the same layer do not depend on each other, and data packets at lower layers depend on data packets at higher layers, the data in the data packets of the second processing stage are decoupled and rearranged to obtain multiple serialized data packets.

3. The method for computing massive dense data according to claim 1, characterized in that, The step of formatting the serialized data to obtain the first data packet specifically includes: The serialized data is concatenated to obtain the concatenated data. Create field names for the concatenated data, and combine the field names with the concatenated data to form a new data entry; The new data is added to the serialized data to obtain the first data packet.

4. The method for computing massive dense data according to claim 1, characterized in that, The step of constructing a weighted red-black tree based on the weights of each node specifically includes: The weights of each node are sorted to obtain weights in ascending order; The weights arranged in ascending order are respectively attached to the red-black tree. Nodes with weights less than the first threshold are attached to the left tree of the weighted red-black tree, and nodes with weights greater than or equal to the first threshold are attached to the right tree of the weighted red-black tree, thus establishing the weighted red-black tree.

5. The method for computing massive dense data according to claim 1, characterized in that, The range of values ​​for the weight is: The smaller the value of the weight, the higher the importance.

Images