Distributed tensor calculation method and device oriented to model training and related equipment

Abstract

The invention provides a distributed tensor calculation method and device oriented to model training and related equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining training data; preprocessing the training data to obtain a tensor corresponding to the training data; segmenting the tensor, and constructing a distributed tensor diagram based on a plurality of sub-tensors obtained after segmentation; and distributing the plurality of sub-tensors to a plurality of processing devices based on the distributed tensor diagram to perform parallel training on the to-be-trained model to obtain a target training result. The model training efficiency can be improved.

Description

Background Art

[0002] Currently, the use of large amounts of high-quality data for large-scale model training faces the following problems and challenges: Data monopoly: Data development is a field that requires huge capital investment. Currently, only companies or organizations with large amounts of data, computing power, and algorithm engineers can participate; Centralization of data, computing, and models: Currently, large-scale model training must centralize data, computing, and models, resulting in large Internet companies becoming monopoly oligarchs of large models. Moreover, these oligarchs are isolated from each other in the concentration of data, computing, and models, and they are not interoperable, let alone shared with ordinary developers.

[0003] In the existing technology, the training data is split and distributed to different central processing units (CPU) / graphics processing units (GPU) for computing and processing, and homomorphic encryption cryptography is used to ensure data security. This has problems such as high computing overhead and low model training efficiency. Traditional model training solutions based on privacy computing, such as Figure 1 As shown, cryptographic techniques such as homomorphic encryption are usually used to encrypt the training data, and then the training data and model are transmitted to each training node. The node performs calculations based on the ciphertext, and then summarizes the calculation results to obtain the training results, and then performs decryption operations. This solution has the following problems: 1. High computational overhead: Privacy computing technology involves complex cryptographic algorithms or a large number of communication interactions, which will lead to a significant increase in computing costs, and sometimes may even be several orders of magnitude slower than traditional methods; 2. High communication overhead: Especially for distributed learning frameworks such as federated learning, exchanging encrypted or partially processed data sets between multiple clients will result in large network bandwidth consumption; 3. Low model training efficiency: Due to the slower processing speed of encrypted data, the training process may become longer. In addition, some encryption technologies may not allow certain types of computing operations, which limits the types of machine learning algorithms that can be used.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a distributed tensor computing method, apparatus and related equipment for model training, which at least to a certain extent overcome the problem of low model training efficiency in related technologies.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a distributed tensor computing method for model training is provided, including: acquiring training data; preprocessing the training data to obtain a tensor corresponding to the training data; segmenting the tensor, and constructing a distributed tensor graph based on multiple sub-tensors obtained after segmentation; and allocating multiple sub-tensors to multiple processing devices based on the distributed tensor graph to perform parallel training on the model to be trained to obtain a target training result.

[0008] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, the tensor is split, and a distributed tensor graph is constructed based on the multiple sub-tensors obtained after the split, including: splitting the tensor to obtain multiple sub-tensors corresponding to the tensor; and constructing the distributed tensor graph based on the multiple sub-tensors.

[0009] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, multiple sub-tensors are distributed to multiple processing devices based on the distributed tensor graph to perform parallel training on the model to be trained to obtain a target training result, including: based on the distributed tensor graph, multiple sub-tensors are distributed to multiple processing devices to perform parallel training on the model to be trained to obtain multiple training results; and multiple training results are integrated and summarized to obtain the target training result.

[0010] In some exemplary embodiments of the present disclosure, based on the aforementioned solution, obtaining the training data to be trained includes: obtaining the training data to be trained in a pre-created distributed shared memory.

[0011] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, multiple sub-tensors are distributed to multiple processing devices based on the distributed tensor graph to perform parallel training on the model to be trained, and a target training result is obtained, including: setting a zero-copy tensor operator based on the distributed tensor graph and a pre-created distributed shared memory; and distributing multiple sub-tensors to multiple processing devices through the zero-copy tensor operator to perform parallel training on the model to be trained.

[0012] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, the training data is preprocessed to obtain a tensor corresponding to the training data, including: normalizing the training data to obtain standardized training data corresponding to the training data; converting the standardized training data into a tensor to obtain a tensor corresponding to the training data.

[0013] According to another aspect of the present disclosure, a distributed tensor computing device for model training is also provided, including: a data acquisition module for acquiring training data; a data processing module for preprocessing the training data to obtain a tensor corresponding to the training data; a tensor splitting module for splitting the tensor and constructing a distributed tensor graph based on multiple sub-tensors obtained after the splitting; a model training module for distributing multiple sub-tensors to multiple processing devices based on the distributed tensor graph to perform parallel training on the model to be trained to obtain a target training result.

[0014] According to another aspect of the present disclosure, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned distributed tensor computing methods for model training by executing the executable instructions.

[0015] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned distributed tensor computing methods for model training is implemented.

[0016] According to another aspect of the present disclosure, a computer program product is also provided, including: a computer program or instructions, which, when executed by a processor, implements any of the above-mentioned distributed tensor computing methods for model training.

[0017] A distributed tensor computing method, apparatus and related equipment for model training are provided in the embodiments of the present disclosure. The training data is converted into tensors, and then the tensors are divided to construct a distributed tensor graph. Based on the distributed tensor graph, multiple sub-tensors are distributed to multiple processing devices for parallel training of the training model to be trained, and finally the target training result is obtained. Compared with the related technology that splits the training data and distributes it to different CPU / GPU devices for calculation and processing, and adopts homomorphic encryption cryptography technology to ensure data security, there are problems such as high computing overhead and low model training efficiency. The embodiments of the present disclosure greatly improve the model training efficiency by constructing a distributed tensor graph and adopting tensor parallel computing. Furthermore, during distributed model training, the tensors are transmitted without directly revealing the training data, and the tensors can be disguised and transformed, thereby ensuring data security while ensuring efficiency.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0020] Figure 1 A schematic diagram of a model training scheme based on privacy computing in related technologies is shown;

[0021] Figure 2 A schematic diagram showing an exemplary application system architecture of a distributed tensor computing method for model training in an embodiment of the present disclosure is shown;

[0022] Figure 3 A schematic diagram of a distributed tensor computing method for model training in an embodiment of the present disclosure is shown;

[0023] Figure 4 A schematic diagram of a distributed tensor computing solution for model training in an embodiment of the present disclosure is shown;

[0024] Figure 5 A schematic diagram of a model training management contract structure in an embodiment of the present disclosure is shown;

[0025] Figure 6 A schematic diagram of a distributed model training process based on a smart contract in an embodiment of the present disclosure is shown;

[0026] Figure 7 A schematic diagram of a distributed tensor computing device for model training in an embodiment of the present disclosure is shown;

[0027] Figure 8 A schematic diagram of an electronic device that applies a distributed tensor computing method for model training in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0029] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0030] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0031] For ease of understanding, several terms involved in the present disclosure are explained below:

[0032] Graphics Processing Unit (GPU): A microprocessor specifically designed for processing images and graphics. It is similar to a computer's central processing unit (CPU), but is specifically designed to process image and graphics data. A GPU typically contains thousands of streaming processors that can process large amounts of data in parallel, enabling efficient graphics rendering and computing.

[0033] Tensor Parallelism: It is a technology used in large-scale deep learning model training. It aims to improve training efficiency and scalability by splitting the model's tensors (multidimensional arrays or matrices) and performing parallel calculations on different computing devices (such as GPUs).

[0034] Blockchain technology: is a new type of decentralized information technology. The data or information stored in the blockchain has the characteristics of "unforgeable", "leaving traces throughout the process", "traceable", "open and transparent", and "collectively maintained". Based on these characteristics, blockchain technology has laid a solid "trust" foundation, created a reliable "cooperation" mechanism, and has broad application prospects.

[0035] Smart contract: a computer protocol designed to disseminate, verify or execute contracts in an information-based manner. Smart contracts allow trusted transactions without a third party. These transactions are traceable and irreversible. The purpose of smart contracts is to provide a security method that is superior to traditional contracts and to reduce other transaction costs associated with contracts. A blockchain smart contract is a piece of code written on the blockchain. Once an event triggers a clause in the contract, the code is automatically executed. In other words, it is executed when the conditions are met without the need for human control.

[0036] Figure 2 FIG. 1 shows an exemplary application system architecture diagram to which the distributed tensor computing method for model training in the embodiment of the present disclosure can be applied. Figure 2 As shown, the system architecture may include a terminal device 201 , a network 202 and a server 203 .

[0037] The network 202 is a medium for providing a communication link between the terminal device 201 and the server 203, and may be a wired network or a wireless network.

[0038] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0039] The terminal device 201 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0040] Optionally, the client of the application installed in different terminal devices 201 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0041] The server 203 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 201. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0042] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0043] Those skilled in the art will know that Figure 2 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0044] Under the above system architecture, an embodiment of the present disclosure provides a distributed tensor computing method for model training, which can be executed by any electronic device with computing processing capabilities.

[0045] In some embodiments, the distributed tensor computing method for model training provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the distributed tensor computing method for model training provided in the embodiments of the present disclosure can be executed by a server in the above-mentioned system architecture; in other embodiments, the distributed tensor computing method for model training provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0046] First of all, in response to the above-mentioned problems, the embodiments of the present disclosure provide a distributed tensor computing that can be applied to but is not limited to model training. It can be applied to any electronic device that needs to perform distributed tensor computing. The training data is converted into tensors, and then the tensors are split to construct a distributed tensor graph. Based on the distributed tensor graph, multiple sub-tensors are distributed to multiple processing devices to perform parallel training on the training model, and finally the target training result is obtained. Compared with the related technology, the training data is split and distributed to different CPU / GPU devices for calculation and processing, and homomorphic encryption cryptography technology is used to ensure data security, which has problems such as high computing overhead and low model training efficiency. The embodiments of the present disclosure greatly improve the model training efficiency by constructing a distributed tensor graph and using tensor parallel computing. Furthermore, during distributed model training, the tensors are transmitted without directly revealing the training data, and the tensors can be disguised and transformed, thereby ensuring data security while ensuring efficiency.

[0047] Figure 3 A schematic diagram of a distributed tensor computing method for model training in an embodiment of the present disclosure is shown, and the method includes the following steps:

[0048] S302, obtaining training data.

[0049] It should be noted that the training data in the embodiments of the present disclosure can be the input data of any model to be trained. For example, the training data can be obtained from a public data set, or crawled from the web, or obtained from an application programming interface (API) interface provided by a website or platform, or obtained from a database.

[0050] S304, preprocessing the training data to obtain a tensor corresponding to the training data.

[0051] It should be noted that the tensor in the embodiment of the present disclosure is a multidimensional array used to represent and process complex data structures. The embodiment of the present disclosure can realize the operation of converting training data into tensors through a deep learning framework. In distributed computing, large-scale data can be efficiently managed and processed through tensors. It can also be distributed on multiple computing nodes. The zero-copy technology can be used to operate on tensors to reduce data transmission overhead.

[0052] S306, split the tensor, and construct a distributed tensor graph based on multiple sub-tensors obtained after the splitting.

[0053] It should be noted that in the embodiment of the present disclosure, the tensor can be split along a specific dimension according to the structure of the model to be trained and the number of available GPUs. For example, if a tensor is a 4×4 array, the tensor can be split according to the first dimension (row) to obtain four 4×1 arrays, and the distributed tensor graph of the tensor is constructed by the four 4×1 arrays.

[0054] S308, based on the distributed tensor graph, distribute multiple sub-tensors to multiple processing devices to perform parallel training on the to-be-trained model to obtain a target training result.

[0055] The distributed tensor calculation method for model training provided in the embodiments of the present disclosure first obtains training data; secondly, pre-processes the training data to obtain the tensor corresponding to the training data; then, splits the tensor, and constructs a distributed tensor graph based on the multiple sub-tensors obtained after the splitting; finally, based on the distributed tensor graph, the multiple sub-tensors are distributed to multiple processing devices for parallel training of the training model to obtain the target training result. Compared with the related art that splits the training data, distributes it to different CPU / GPU devices for calculation and processing, and uses homomorphic encryption cryptography technology to ensure data security, there are problems such as high calculation overhead and low model training efficiency. The embodiments of the present disclosure greatly improve the model training efficiency by constructing a distributed tensor graph and using tensor parallel calculation. Furthermore, during the distributed training of the model, the tensor is transmitted without directly revealing the training data, and the tensor can be disguised and transformed, which ensures the security of the data while ensuring efficiency.

[0056] In some embodiments, Figure 4 As shown, the model training scheme based on the distributed tensor computing method in the embodiment of the present disclosure includes:

[0057] S402, converting the training data into a tensor. It should be noted that the training data of the model (eg, a neural network architecture model based on a self-attention mechanism) is usually stored and processed in the form of a tensor.

[0058] S404, split the tensor, and assign different sub-tensors after splitting to different processing devices for calculation. It should be noted that the model parameter scale is very large, and a large number of processing devices (GPU, etc.) need to be used to complete the model training in a distributed manner, so the different sub-tensors after splitting are assigned to different processing devices to complete the model training in a distributed manner.

[0059] S406, use tensor parallelism to perform calculations. Tensor parallelism is to split the weight matrix of the same layer according to different dimensions so that each GPU processes a part of the weight matrix. It can be seen that this method allows different parts of the model to be calculated on multiple GPUs at the same time, thereby improving the overall calculation efficiency.

[0060] S408, integrating and summarizing the various training results to obtain the final training results and form a complete model.

[0061] In some embodiments, the tensor is split in the embodiments of the present disclosure, and a distributed tensor graph is constructed based on the multiple sub-tensors obtained after the splitting, including: splitting the tensor to obtain multiple sub-tensors corresponding to the tensor; constructing a distributed tensor graph based on the multiple sub-tensors. Specifically, the embodiments of the present disclosure can split the weight matrix along a specific dimension according to the structure of the model and the number of available GPUs through tensor splitting, and assign it to different nodes for calculation. The constructed distributed tensor graph divides the tensors (multidimensional arrays or matrices) of the model to be trained and performs parallel calculations on different computing devices (such as GPUs) to improve training efficiency and expansion capabilities.

[0062] In some embodiments, the embodiments of the present disclosure allocate multiple sub-tensors to multiple processing devices based on the distributed tensor graph to perform parallel training on the model to be trained to obtain target training results, including: based on the distributed tensor graph, allocating multiple sub-tensors to multiple processing devices to perform parallel training on the model to be trained to obtain multiple training results; integrating and summarizing the multiple training results to obtain the target training results. Specifically, the embodiments of the present disclosure construct a distributed tensor graph and use tensor parallelism to perform tensor calculations, which greatly improves the efficiency of model training.

[0063] In some embodiments, the embodiments of the present disclosure obtain training data to be trained, including: obtaining the training data to be trained in a pre-created distributed shared memory. Specifically, the distributed shared memory in the embodiments of the present disclosure generally refers to the ability to share memory space between multiple computers. This technology enables processes on multiple machines to access data in remote memory as if they were accessing local memory. The embodiments of the present disclosure create shared memory between distributed data and computing nodes through a decentralized logical bus, so that these nodes can work together like a single computer, and because it is a distributed architecture, single point failures can be avoided and the robustness of the system can be improved.

[0064] In some embodiments, the embodiment of the present disclosure distributes multiple sub-tensors to multiple processing devices based on a distributed tensor graph to perform parallel training on a model to be trained, and obtains a target training result, including: setting a zero-copy tensor operator based on a distributed tensor graph and a pre-created distributed shared memory; distributing multiple sub-tensors to multiple processing devices through a zero-copy tensor operator to perform parallel training on a model to be trained. Specifically, in a model training scenario, data needs to be transferred between different memory spaces (such as CPU memory and GPU memory). Usually, such data transfer involves copying data, which may cause additional time overhead and memory consumption. The core idea of ​​the zero-copy operation is to avoid explicit data copying as much as possible when moving data, but to directly redirect data pointers or use shared memory areas to access data, thereby improving the overall performance of the system. The embodiment of the present disclosure pre-creates a shared memory and a distributed tensor graph, designs and implements a zero-copy tensor operator, avoids the conventional practice of copying data to node memory during calculation, and does not copy tensor data during calculation, even in an encrypted format, thereby solving the problem of data control.

[0065] In some embodiments, the embodiments of the present disclosure use zero-copy tensor operators to build and train models on distributed data, such as small models such as linear regression and statistical analysis, and large models based on neural networks, without the need for additional federated learning or privacy computing platforms.

[0066] In some embodiments, the disclosed embodiments pre-build a data development software environment similar to a distributed version of NumPy (an open source numerical computing extension that can be used to store and process large matrices) or PyTorch (an open source deep learning framework for machine learning and deep learning), allowing developers to build, test and deploy models on widely distributed computing and data nodes on the Internet. In a distributed network, in order to ensure data security and privacy, the disclosed embodiments use a model training method based on smart contracts to ensure that each node conducts model training honestly, such as Figure 5As shown, the model training management contract structure in the embodiment of the present disclosure includes: node ID (NodeID): the unique identifier of the computing power node; node name (NodeName): the name of the computing power node; node information (NodeInfo): the relevant information of the computing power node, which may include location, network, computing power status and other information; training task ID (TaskID): the unique identifier of the training task; task status (TaskStatus): task status, such as in progress, abnormal, completed, etc.; training data Hash (DataHash): the Hash value of training data; training result Hash (ResultHash): the Hash value of training result; evidence time (CommitTime): the time to submit evidence; task information retrieval method (GetTaskInfo()): a method for retrieving training task related information.

[0067] In some embodiments, the embodiments of the present disclosure use distributed training to ensure that nodes complete training tasks honestly and ensure data security, such as Figure 6 As shown, the large model distributed training process based on smart contracts in the embodiment of the present disclosure specifically includes:

[0068] S602, smart contracts are deployed on the blockchain, combined with the decentralized characteristics of the blockchain to improve the overall credibility of the system.

[0069] S604: For model training tasks, a tensor parallel approach is used to distribute them to different processing devices for distributed training.

[0070] S606, distribution and feedback of training tasks are executed by triggering smart contracts.

[0071] S608, after the distributed task is completed, the task processing results are aggregated through the contract to form a complete model.

[0072] S610, all task execution records, including training data Hash, training result Hash (i.e., input and output of training tasks), etc., will be saved to the blockchain through contracts for evidence storage. If there is dishonest training, audit traceability can be performed to ensure that the nodes are not constrained to conduct model training honestly.

[0073] In some embodiments, the training data Hash in the disclosed embodiment, that is, the summary data obtained by hashing the tensor assigned to the node for this training task; the training result Hash, that is, the summary data obtained by hashing the training result obtained after the node uses the training data to complete the calculation; for given training data, check whether the uploaded result is consistent with the expected result. If so, the calculation task is completed honestly; otherwise, the calculation is wrong, that is, for a specific training data Hash, judge whether the feedback training result Hash is consistent with the expected result Hash. It can be seen that the relevant information in the model training in the disclosed embodiment, the training data Hash, the data result Hash and other task records are all stored on the chain, the data records on the chain cannot be tampered with, and can be retroactively audited.

[0074] In some embodiments, the embodiments of the present disclosure preprocess the training data to obtain a tensor corresponding to the training data, including: normalizing the training data to obtain standardized training data corresponding to the training data; converting the standardized training data into a tensor to obtain a tensor corresponding to the training data. Specifically, the embodiments of the present disclosure preprocess the training data and also include data cleaning, processing missing values ​​and other invalid data in the training data; and also include converting the categorical variables in the training data into numerical form, and then normalizing them to obtain standardized training data; converting the preprocessed data into a tensor to prepare for subsequent model training.

[0075] In some embodiments, the distributed tensor graph architecture in the embodiments of the present disclosure is oriented towards the current situation where data, calculations, and models may be dispersed in different locations and among participants, and can adapt to this decentralization while protecting data privacy; and, the embodiments of the present disclosure use tensor parallelism and zero-copy tensor operators, based on the shared memory and computing nodes pre-created by the embodiments of the present disclosure, to perform model training using tensor splitting and tensor parallelism, and use zero-copy technology for calculation during the training process to improve overall efficiency; further, the model training contract structure designed in the embodiments of the present disclosure, and the model training data assurance method based on this structure, utilize the automatic triggering execution of smart contracts and the tamper-proof nature of blockchain to ensure that each node honestly completes the model training task.

[0076] Based on the same inventive concept, the embodiments of the present disclosure also provide a distributed tensor computing device for model training, such as the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0077] Figure 7 A schematic diagram of a distributed tensor computing device for model training in an embodiment of the present disclosure is shown, and the device includes:

[0078] A data acquisition module 701 is used to acquire training data;

[0079] A data processing module 702 is used to pre-process the training data to obtain a tensor corresponding to the training data;

[0080] A tensor splitting module 703 is used to split the tensor and construct a distributed tensor graph based on multiple sub-tensors obtained after the splitting;

[0081] The model training module 704 is used to distribute multiple sub-tensors to multiple processing devices based on the distributed tensor graph to perform parallel training on the model to be trained to obtain the target training result.

[0082] A distributed tensor computing device for model training provided in an embodiment of the present disclosure acquires training data through a data acquisition module; pre-processes the training data through a data processing module to obtain a tensor corresponding to the training data; divides the tensor through a tensor splitting module, and constructs a distributed tensor graph based on multiple sub-tensors obtained after the splitting; and distributes multiple sub-tensors to multiple processing devices based on the distributed tensor graph to perform parallel training on the training model through a model training module to obtain a target training result. Compared with the related art that splits the training data and distributes it to different CPU / GPU devices for calculation and processing, and uses homomorphic encryption cryptography technology to ensure data security, there are problems such as high calculation overhead and low model training efficiency. The embodiment of the present disclosure greatly improves the model training efficiency by constructing a distributed tensor graph and using tensor parallel computing. Furthermore, during the distributed training of the model, the tensor is transmitted without directly revealing the training data, and the tensor can be disguised and transformed, which ensures the security of the data while ensuring efficiency.

[0083] In some embodiments, the tensor splitting module in the embodiments of the present disclosure is also used to split the tensor to obtain multiple sub-tensors corresponding to the tensor; and to construct a distributed tensor graph based on the multiple sub-tensors.

[0084] In some embodiments, the model training module in the embodiments of the present disclosure is also used to distribute multiple sub-tensors to multiple processing devices based on a distributed tensor graph to perform parallel training on the model to be trained, thereby obtaining multiple training results; and integrate and summarize the multiple training results to obtain the target training results.

[0085] In some embodiments, the data acquisition module in the embodiments of the present disclosure is also used to acquire the training data to be trained in a pre-created distributed shared memory.

[0086] In some embodiments, the model training module in the embodiments of the present disclosure is also used to set a zero-copy tensor operator based on a distributed tensor graph and a pre-created distributed shared memory; and to distribute multiple sub-tensors to multiple processing devices through the zero-copy tensor operator to perform parallel training on the model to be trained.

[0087] In some embodiments, the data processing module in the embodiments of the present disclosure is also used to normalize the training data to obtain standardized training data corresponding to the training data; convert the standardized training data into a tensor to obtain a tensor corresponding to the training data.

[0088] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0089] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any of the above-mentioned distributed tensor computing methods for model training by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above-mentioned method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0090] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0091] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 801, at least one storage unit 802, and a bus 803 connecting different system components (including the storage unit 802 and the processing unit 801).

[0092] The storage unit stores program codes, which can be executed by the processing unit 801, so that the processing unit 801 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0093] In some embodiments, when the electronic device is used to control, for example, the above-mentioned distributed tensor computing method for model training of the present disclosure, the processing unit 801 may perform the following steps of the above-mentioned method embodiment:

[0094] Acquire training data; preprocess the training data to obtain tensors corresponding to the training data; split the tensors, and construct a distributed tensor graph based on the multiple sub-tensors obtained after the splitting; based on the distributed tensor graph, distribute the multiple sub-tensors to multiple processing devices to perform parallel training on the training model to obtain the target training results.

[0095] The storage unit 802 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 8021 and / or a cache storage unit 8022 , and may further include a read-only storage unit (ROM) 8023 .

[0096] The storage unit 802 may also include a program / utility 8024 having a set (at least one) of program modules 8025, such program modules 8025 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0097] Bus 803 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0098] The electronic device 800 may also communicate with one or more external devices 804 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or may communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication may be performed via an input / output (I / O) interface 805. In addition, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 806. As shown, the network adapter 806 communicates with other modules of the electronic device 800 via a bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0099] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0100] Based on the same inventive concept, a computer-readable storage medium is also provided in the embodiment of the present disclosure, on which a computer program is stored, and when the computer program is executed by a processor, any of the above-mentioned distributed tensor calculation methods for model training is implemented. Since the principle of solving the problem in the computer-readable storage medium embodiment is similar to that in the above-mentioned method embodiment, the implementation of the computer-readable storage medium embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0101] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0102] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0103] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0104] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0105] Based on the same inventive concept, a computer program product is also provided in the embodiments of the present disclosure, including a computer program product, including: a computer program or an instruction, when the computer program or the instruction is executed by a processor, a distributed tensor calculation method for model training in any one of the above method embodiments is implemented. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0106] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0107] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0108] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0109] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A distributed tensor computing method for model training, characterized in that: include: Get training data; Preprocessing the training data to obtain a tensor corresponding to the training data; Splitting the tensor, and constructing a distributed tensor graph based on a plurality of sub-tensors obtained after the splitting; Based on the distributed tensor graph, multiple sub-tensors are distributed to multiple processing devices to perform parallel training on the model to be trained to obtain a target training result.

2. The distributed tensor computing method for model training according to claim 1, characterized in that: The tensor is segmented, and a distributed tensor graph is constructed based on a plurality of sub-tensors obtained after segmentation, including: Splitting the tensor to obtain multiple sub-tensors corresponding to the tensor; The distributed tensor graph is constructed based on multiple sub-tensors.

3. The distributed tensor computing method for model training according to claim 2, characterized in that: Based on the distributed tensor graph, multiple sub-tensors are distributed to multiple processing devices to perform parallel training on the model to be trained, and a target training result is obtained, including: Based on the distributed tensor graph, multiple sub-tensors are distributed to multiple processing devices to perform parallel training on the model to be trained, thereby obtaining multiple training results; Integrate and summarize multiple training results to obtain the target training result.

4. The distributed tensor computing method for model training according to claim 1, characterized in that: Get the training data to be trained, including: Get the training data to be trained in the pre-created distributed shared memory.

5. The distributed tensor computing method for model training according to claim 4, characterized in that: Based on the distributed tensor graph, multiple sub-tensors are distributed to multiple processing devices to perform parallel training on the model to be trained, and a target training result is obtained, including: Setting a zero-copy tensor operator based on the distributed tensor graph and the pre-created distributed shared memory; The zero-copy tensor operator distributes multiple sub-tensors to multiple processing devices to perform parallel training on the model to be trained.

6. The distributed tensor computing method for model training according to claim 1, characterized in that: Preprocessing the training data to obtain a tensor corresponding to the training data includes: Normalizing the training data to obtain standardized training data corresponding to the training data; The standardized training data is converted into a tensor to obtain a tensor corresponding to the training data.

7. A distributed tensor computing device for model training, characterized in that: include: A data acquisition module, used to acquire training data; A data processing module, used for preprocessing the training data to obtain a tensor corresponding to the training data; A tensor splitting module, used to split the tensor and construct a distributed tensor graph based on multiple sub-tensors obtained after the splitting; The model training module is used to distribute multiple sub-tensors to multiple processing devices based on the distributed tensor graph to perform parallel training on the model to be trained to obtain the target training result.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the distributed tensor computing method for model training described in any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distributed tensor computing method for model training described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the distributed tensor computing method for model training as described in any one of claims 1 to 6.

Images