Tensor Comparison across Distributed Machine Learning Environments
By determining tensor similarity through vector parameter comparison in distributed computing environments, the method reduces communication overhead and enhances data security by only transmitting dissimilar tensors, addressing inefficiencies in DML applications.
Patent Information
- Application Number
- JP2023512288
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-08-27
- Filing Date
- 2021-08-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-08-25
AI Technical Summary
Distributed machine learning (DML) applications face significant communication resource consumption due to the iterative transmission of large tensor objects between remotely distributed processes, which can be inefficient and wasteful, especially when tensors are similar.
A method that determines the similarity of tensors by exchanging and comparing vector parameters (length and angle) to decide whether to transmit tensors, reducing communication by only sending dissimilar tensors, using locality-sensitive hashing to encode vectors as (length, angle) pairs.
This approach significantly reduces communication overhead and improves data security by minimizing the transmission of similar tensors, optimizing resource utilization in distributed computing environments.
Smart Images

Figure 0007590093000058 
Figure 0007590093000059 
Figure 0007590093000060
Abstract
Description
[Technical field]
[0001] The present disclosure relates generally to tensor comparison across distributed computing environments, and more particularly to sharing tensors across a distributed network environment according to tensor similarity. [Background technology]
[0002] Training large models across large datasets, e.g., Turing-NLG, a 17 billion parameter language model, utilizes distributed machine learning (DML). Such applications require the repeated transmission of large tensor objects between remotely distributed processes. Techniques exist for compressing and decompressing tensors transmitted between processes. Sending large tensor objects between processors across a networked computing environment can consume significant communication resources. Summary of the Invention
[0003] The following presents a summary to provide a basic understanding of one or more embodiments of the present disclosure. This summary is not intended to identify key or critical elements, nor is it intended to delineate any scope of particular embodiments or any scope of the claims. The sole purpose of this summary is to present concepts in a simplified form as a prelude to the more detailed description presented later. In one or more embodiments described herein, a device, system, computer-implemented method, apparatus, or computer program product, or combination thereof, enables tensor comparison and communication associated with tensor similarity.
[0004] Aspects of the present invention disclose methods, systems and computer readable media associated with tensor comparison across a network by determining a first parameter of a first vector representation of a first tensor object associated with a first processor, receiving a second parameter associated with a second vector representation of a second tensor object associated with a second processor, determining a first difference between the first and second parameters, and transmitting the first vector representation from the first processor to the second processor according to the first difference. By exchanging and comparing the vector parameters (length and angle), the process can determine whether the original tensors are (generally) similar or not, and the sender can determine whether to send its tensor to the receiver or not. By transmitting tensors only when necessary (only when the tensors are generally dissimilar), communication resources are conserved.
[0005] The above and other objects, features and advantages of the present disclosure will become more apparent through more detailed description of several embodiments of the present disclosure in the accompanying drawings. The same reference numerals generally refer to the same components in the embodiments of the present disclosure. [Brief description of the drawings]
[0006] [Figure 1] 1 is a schematic diagram of a computing environment in accordance with an embodiment of the present invention. [Diagram 2] 4 is a flowchart illustrating an operational sequence according to an embodiment of the present invention. [Diagram 3] 2 is a schematic diagram of data flow according to an embodiment of the present invention. [Figure 4] 2 is a schematic diagram of data flow according to an embodiment of the present invention. [Diagram 5] FIG. 1 illustrates a cloud computing environment according to an embodiment of the present invention. [Figure 6] FIG. 2 illustrates abstract model layers according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0007] Some embodiments will now be described in more detail with reference to the accompanying drawings, in which embodiments of the present disclosure are illustrated, however the present disclosure may be embodied in various ways and therefore should not be construed as limited to the embodiments disclosed herein.
[0008] In one embodiment, one or more components of the system may use hardware or software or both to solve problems that are highly technical in nature (e.g., determining parameters of a vector representation of a first tensor, receiving parameter values associated with a vector representation of a second tensor, determining differences between the parameters, transmitting the first vector representation according to the differences, etc.). These solutions are not abstract and cannot be performed as a set of mental activities by a human being due to the processing power required to facilitate, for example, tensor comparisons across a network. Furthermore, some of the processes performed may be performed by a dedicated computer for performing defined tasks related to distributed communications. For example, a dedicated computer may be used to perform tasks related to managing network communication resource usage by comparing tensor values across a network, etc.
[0009] The use of compression and expansion techniques can reduce the communication burden of DML training and transmission of large tensor entities across the network in use. Compression and expansion techniques still require communication of large data entities. A compression / expansion cycle may not be necessary in every iteration of one or more processes because the processes can store tensors that are deemed similar enough to the requirements of the application (e.g., convergence criteria). In such cases, communication and computational resources are wasted in compressing, communicating, and expanding tensors. Additionally, in certain distributed applications, security constraints can allow data exchange to occur only if the data meets similarity requirements. For this reason, it is important to know whether tensors maintained between processes are similar enough before actually transmitting a tensor from a first server or processor to a second server or processor. By avoiding the compression, communication, and expansion steps, computational / communication resources are significantly saved. By enabling a process to decide whether to reveal (or not) its tensors to other processes based on similarity criteria, data security throughout the system is improved.
[0010] As an example, a DML application has local nodes that are used to train a Global Machine Learning (GML) model using local data. The local nodes are distributed across a network of servers and processors and are remote from each other. During training, the nodes need to exchange tensors containing the model parameters using network communication resources. For example, the tensor is an n1×n2 array and every node converts its tensor into a vector of dimension d=n1×n2. During a training iteration, the sender node sends its own vector of dimension d (number of model parameters)
[0011]
number
[0012] is the current vector at the receiving node
[0013]
number
[0014] Communication resources are saved if the sender transmits its own vector only if it is substantially different from
[0015]
number
[0016] Receive the length of the vector difference
[0017]
number
[0018] locally, and then, based on a relative tolerance threshold,
[0019]
number
[0020] However, this approach defeats the original intent, since now the entire vector is communicated between the servers in total d×m bits, where m is the "word" length, typically 32 or 64 (single or double precision) bits. Furthermore, the entire vector
[0021]
number
[0022] becomes shared and therefore exposed.
[0023] The disclosed system and method allows two separate processes executing on remote machines to determine whether two tensor objects are generally "close" (or similar) without sharing the tensor objects. In one embodiment, the invention works by vectorizing tensor objects, encoding each resulting vector as a (length, angle) pair, and sharing and comparing the encoded vectors instead of the original tensor objects, where the angles are calculated using locally sensitive hashing (LSH). The disclosed method substantially saves communication overhead and allows processes to control sharing of tensors with other processes based on tensor similarity criteria.
[0024] The disclosed systems and methods are applicable to networked computing environments in general, and particularly to DML system environments having nodes distributed across a networked environment executing multi-node algorithms at increased processing performance.
[0025] In one embodiment, the method uses k bits, where k is much lower than d×m, and an approximation of how close two vectors (and their associated original tensors) are is made without actually sharing any of the vectors between these two communicating agents. In this embodiment, the method determines whether two remote tensors are approximately "close" without transmitting any of them. The method works by first obtaining a vectorized version of the tensor. The resulting vector is then encoded as a (length, angle) pair, where the length is a real number representing the magnitude of the vector and the angle is a k-bit vector hash representing the angle of the vector.
[0026] In this embodiment, the angle of each vector is calculated using a locality sensitive hashing technique.
[0027]
number
[0028] and
[0029]
number
[0030] To estimate the angle θ between
[0031]
number
[0032] and
[0033]
number
[0034] is a vector
[0035]
number
[0036] and
[0037]
number
[0038] The length of the vector and its code vector representing the angle are the only information transmitted in this similarity comparison technique. This information is a small fraction of the original tensor (thus dramatically reducing communication overhead) and also reveals minimal information about the original tensor (thus making it applicable to applications with sharing-related or non-similarity-based security constraints).
[0039] By exchanging and comparing vector lengths and angles, the process can determine whether the original tensors are (roughly) similar, and the sender can decide whether to send its tensor to the receiver.
[0040] In one embodiment, the application of the disclosed method is applicable to distributed iterative algorithms that are particularly stable under perturbations. Examples include algorithms that retain most of their convergence properties when transferred to a fully (or partially) asynchronous computational model. The disclosed method is applicable to each pair of communicating agents, at each iteration, in particular towards the later stages of the algorithm when convergence can be assumed to increase the probability that the vectors are "close".
[0041] In one embodiment, the disclosed embodiment is applicable to learning algorithms based on stochastic approximation. As an example, stochastic gradient descent (SGD) variables, which are usually trained in mini-batches, are generally robust to input data "perturbations" such as resizing or shuffling. This implies that stochasticity can also be much more "tolerant" in terms of sensitivity when relaxing and approximating distance-like notions (for "proximity"), compared to other conventional methods with more deterministic data flows, where small input deviations can lead to irreversible instability. The method has further applicability in communication of large vectors, where it is assumed that the size d of the communicated real vectors is much larger than the size k of the encoded bit vectors. The disclosed method can be applied to trained models, which are typically represented as matrices and can be vectorized during application of the method. In one embodiment, the disclosed method increases application speed by reducing communication load, and further increases data security by reducing data sharing demands.
[0042] Aspects of the invention disclose methods, systems, and computer-readable media associated with tensor comparison across a network by determining a first parameter of a first vector representation of a first tensor object associated with a first processor, receiving a second parameter associated with a second vector representation of a second tensor object associated with a second processor, determining a third parameter of the first vector representation of the first tensor object, receiving a fourth parameter associated with the second vector representation of the second tensor object, determining a difference between the first and second vector representations according to the first, second, third, and fourth parameters, and transmitting the first vector representation from the first processor to the second processor according to the difference.
[0043] For the following description of the disclosed invention, the discussion will focus on the communication and interaction between a sender process (sender) and a receiver process (receiver). Each of the sender and receiver processes constitutes a process application that executes in a portion of a distributed computing environment. Each process executes in a different portion of the environment, and the processes communicate via a network communication system.
[0044] In one embodiment, the sender stores a tensor X a and the recipient is a tensor X b At the beginning of the process, the sender possesses X a Vector
[0045]
number
[0046] and the receiver is X b Vector
[0047]
number
[0048] Each vector provides a representation of the information contained in the respective tensor. In this embodiment, the sender calculates a length parameter a for its vector,
[0049]
number
[0050] The receiver is then prompted for the length parameter b associated with
[0051]
number
[0052] , and sends b to the sender in response to the request.
[0053] The sender determines the absolute difference between a and b and compares this difference to a predefined threshold. In one embodiment, the method attempts to avoid sending similar tensors, and the method proceeds to send tensors that have a determined difference greater than the predefined threshold.
[0054] Tensor X a and X b have similar lengths a and b, but
[0055]
number
[0056] In some cases, the difference tensors may be sufficiently different to justify the transmission of a difference tensor. To capture such instances and transmit the difference tensor, the method performs an additional similarity test on vectors with length differences less than a predefined threshold. For this similarity test, the sender first generates k random vectors r1, r2, r3, …r using a seed value s to initialize its pseudorandom number generator.k For each random vector, the sender generates a random vector and
[0057]
number
[0058] Calculate the sign (positive or negative) of the dot product with the sender k-bit sign vector
[0059]
number
[0060] The receiver also uses the seed value s to initialize its pseudorandom number generator, which is a set of k random vectors r1, r2, r3, … r k The receiver also generates a random vector
[0061]
number
[0062] Calculate the sign of the dot product with the receiver k-bit code vector
[0063]
number
[0064] As a result, we collect the calculated dot product codes, b and the vector
[0065]
number
[0066] To the sender, in one embodiment, the sender sends to the receiver the vector
[0067]
number
[0068] Alternatively, the sender and receiver each have predetermined values for s and k, respectively, for these calculations.
[0069] In one embodiment, k, s and the threshold are statically fixed and their values can be related to a particular application that utilizes tensor comparisons and computes resource, budget and other constraints. Increasing k results in a better approximation at the expense of computational load, i.e., more randomized projections. Increasing the threshold affects the number of iterations the method skips point-to-point communication and therefore the usefulness of the method to reduce communication resource utilization.
[0070] The sender sends two vectors
[0071]
number
[0072] and
[0073]
number
[0074] The probability p of having the same sign is
[0075]
number
[0076] and
[0077]
number
[0078] The method estimates the relative number of identical entries in p by counting them element by element. Given p, the method estimates the approximate angle θ between the two vectors using the formula p=1-θ / π. The approximate angle θ,
[0079]
number
[0080] length b and
[0081]
number
[0082] Now we obtain the length a of the vector difference,
[0083]
number
[0084] , the cosine theorem
[0085]
number
[0086] It can be roughly calculated by:
[0087] If the relative vector difference c / a is greater than the tolerance threshold r, the sender
[0088]
number
[0089] to the receiver, otherwise skip this communication. In one embodiment, the receiver is notified of the skip through the execution of a communication timeout between the nodes (processors) or by the sender sending an explicit skipped tensor communication message indicating that the tensor was not or will not be sent to the receiver.
[0090] In one embodiment, the sender and receiver each calculate the length and code vector for their respective vector representations using seeds s and k as described above. In this embodiment, the sender provides the receiver with b and the code vector
[0091]
number
[0092] The sender requests and receives from the receiver the difference between the two vectors.
[0093]
number
[0094] to the receiver or skip the communication. In one embodiment, the sender tries to avoid sending the difference tensor and only
[0095]
number
[0096] In one embodiment, the sender tries to avoid sending similar tensors and transmits
[0097]
number
[0098] Send.
[0099] Communication of a complete vector requires the transmission of d×m bits. In one embodiment, the method compares tensors using only 3×m+k bits. In an embodiment where the number of projections k and the seed value are pre-specified and not shared for each comparison, the comparison process requires only m+k bits of communication.
[0100] FIG. 1 provides a schematic diagram of exemplary network resources associated with the execution of the disclosed invention. The invention may be embodied in a processor of any of the disclosed elements that processes instruction streams. As shown in the figure, a networked client device 110 wirelessly connects to a server sub-system 102. A client device 104 wirelessly connects to the server sub-system 102 via a network 114. The client devices 104 and 110 include a tensor comparison program (not shown) along with sufficient computing resources (processors, memory, network communication hardware) to execute the program. In one embodiment, the client devices 104 and 110 and the server sub-system 102 form part of a distributed machine learning system, where training of a machine learning model includes the transfer of tensor objects between resources as the model training progresses.
[0101] As shown in Figure 1, server subsystem 102 includes server computer 150. Figure 1 illustrates a block diagram of components of server computer 150 in networked computer system 1000 in accordance with an embodiment of the present invention. It should be understood that Figure 1 provides an illustration of only one implementation and is not intended to imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.
[0102] The server computer 150 may include a processor 154, a memory 158, a persistent storage 170, a communication unit 152, an input / output (I / O) interface 156, and a communication fabric 140. The communication fabric 140 provides communication between a cache 162, the memory 158, the persistent storage 170, the communication unit 152, and the input / output (I / O) interface 156. The communication fabric 140 may be implemented using any architecture designed to pass data and / or control information between a processor (such as a microprocessor, communication and network processor, etc.), system memory, peripheral devices, and any other hardware components in a system. For example, the communication fabric 140 may be implemented using one or more buses.
[0103] Memory 158 and persistent storage 170 are computer-readable storage media. In this embodiment, memory 158 includes random access memory (RAM) 160. In general, memory 158 may include any suitable volatile or non-volatile computer-readable storage media. Cache 162 is a high-speed memory that enhances the performance of processor 154 by holding recently accessed data and data nearby recently accessed data from memory 158.
[0104] Program instructions and data used to implement embodiments of the invention, such as tensor comparison program 175, are stored in persistent storage 170 for execution and / or access by one or more of the respective processors 154 of server computer 150 via cache 162. In this embodiment, persistent storage 170 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage 170 may include a solid state hard drive, a semiconductor storage device, a read only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, or any other computer readable storage medium capable of storing program instructions or digital information.
[0105] The media used by persistent storage 170 may be removable. For example, a removable hard drive may be used for persistent storage 170. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of persistent storage 170.
[0106] The communications unit 152 provides for communication with other data processing systems or devices, including resources of the client computing devices 104 and 110, in these examples. In these examples, the communications unit 152 includes one or more network interface cards. The communications unit 152 can provide for communication through the use of either or both physical and wireless communication links. Software distribution programs, as well as other programs and data used in the implementation of the present invention, may be downloaded to the persistent storage 170 of the server computer 150 through the communications unit 152.
[0107] The I / O interface 156 allows for input and output of data with other devices that may be connected to the server computer 150. For example, the I / O interface 156 may provide a connection to an external device 190, such as a keyboard, a keypad, a touch screen, a microphone, a digital camera, or some other suitable input device or combination thereof. The external device 190 may also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to implement embodiments of the present invention, such as the tensor comparison program 175 on the server computer 150, may be stored on such portable computer-readable storage media and loaded onto the persistent storage 170 via the I / O interface 156. The I / O interface 156 also connects to a display 180.
[0108] Display 180 provides a mechanism for displaying data to a user and may be, for example, a computer monitor. Display 180 may also function as a touch screen, such as the display of a tablet computer.
[0109] 2 provides a flowchart 200 illustrating exemplary activities associated with practicing the present disclosure. After the program begins, at block 210, the method of the tensor comparison program 175 determines a first parameter value associated with a vector representation of a tensor maintained by a sending process. In one embodiment, the first parameter value constitutes the length of the vector representation. The method generates the first parameter value as part of the method execution by the sending process operating in a networked computing processor.
[0110] At block 220, the tensor comparison program 175 method receives a second parameter. A receiving process operating in a second networked computing processor determines a second parameter associated with a second vector representation of the second tensor accessed by the receiving process, where the first parameter and the second parameter each represent the same parameter of the respective associated vector representation. For example, each parameter may provide a determined length of the corresponding vector representation. The receiving process provides the second parameter to the sending process in response to a request for the second parameter from the sending process.
[0111] At block 230, the transmission process compares the first and second parameters, and in one embodiment, the comparison includes determining the absolute value of the difference between the two parameters.
[0112] At block 240, the sending process transmits the vector representation of the tensor over the network to the receiving process. In one embodiment, the sending process evaluates the determined parameter difference against a defined threshold and transmits the vector only when the correct relationship between the difference and the threshold exists, e.g., when the difference is above or below the threshold, as appropriate. In one embodiment, the transmission occurs only when the difference exceeds the threshold, and in another embodiment, the transmission occurs only when the difference is below the threshold. For example, if the difference does not meet the threshold requirement, the method does not transmit the vector. In one embodiment, the sender provides notification of the failure threshold, and in another embodiment, the sender remains silent in view of the failure threshold.
[0113] 3 illustrates a data flow 300 through a networked system in accordance with an embodiment of the present invention. As shown, in stage 0, a sender 310 and a receiver 350 each receive a vector representation X a , X b tensor to vector
[0114]
number
[0115] and
[0116]
number
[0117] Each of the sender 310 and the receiver 350 determines their own vector length parameters a and b. In stage 1, the sender 310 generates k random vectors from a seed value s. The sender 310 converts the code vector
[0118]
number
[0119] for each of the k random vectors
[0120]
number
[0121] The length and sign vectors are calculated according to the sign of the dot product with the vector
[0122]
number
[0123] is considered a comparison signature of
[0124] In stage 2, the sender 310 provides k and s to the receiver 350, and the receiver 350 uses k and s in the same way as the sender 310 to
[0125]
number
[0126] The receiver 350 responds to the request for a comparison signature from the sender 310 by determining the comparison signature b and
[0127]
number
[0128] to the sender 310.
[0129] In stage 3, the sender 310 compares the two comparison signatures to obtain the vector
[0130]
number
[0131] and
[0132]
number
[0133] The comparison involves determining the angle between the two vectors according to their respective code vectors.
[0134] In stage 4, the sender compares the determined difference between the vectors with a defined threshold to determine the vector
[0135]
number
[0136] should be transmitted to the receiver 350. In one embodiment, vectors with differences greater than a threshold r, indicating dissimilarity vectors, are transmitted at 315. In an alternative embodiment, vectors with differences less than r, i.e., similarity vectors, are transmitted at 325.
[0137] 4 illustrates a data flow 400 through a networked system in accordance with an embodiment of the present invention. As shown, in stage 0, a sender 410 and a receiver 450 each receive a vector representation X a , X b tensor to vector
[0138]
number
[0139] and
[0140]
number
[0141] Each of the sender 410 and the receiver 450 determines the length parameters a, b of their own vectors. In stage 1a, the sender 410 requests the length b from the receiver 450. The receiver 450 provides b, and the sender 410 compares a and b. In 415, the sender 410 determines that the difference between a and b exceeds a threshold, and then:
[0142]
number
[0143] to the receiver. After the sender determines in 425 that the difference between a and b is less than the threshold, it proceeds to step 1b.
[0144] In stage 1b, the sender 410 generates k random vectors from a seed value s. The sender 410 then generates the code vectors
[0145]
number
[0146] for each of the k random vectors
[0147]
number
[0148] The length and sign vectors are calculated according to the sign of the dot product with the vector
[0149]
number
[0150] is considered a comparison signature of
[0151] In stage 2, the sender 410 provides k and s to the receiver 450, and the receiver 450 uses k and s in the same way as the sender 410 to
[0152]
number
[0153] The receiver 450 responds to the request for a comparison signature from the sender 410 by determining the comparison signature b and
[0154]
number
[0155] to the sender 410.
[0156] In stage 3, the sender 410 compares the two comparison signatures to obtain the vector
[0157]
number
[0158] and
[0159]
number
[0160] The comparison involves determining the angle between the two vectors according to their respective code vectors.
[0161] In stage 4, the sender 410 compares the determined difference between the vectors with a defined threshold to determine whether the vector
[0162]
number
[0163] should be transmitted to the receiver 450. In one embodiment, vectors having a relative difference greater than a threshold r, indicating a dissimilarity vector, are transmitted at 435. In an alternative embodiment, vectors having a relative difference less than r, i.e., similarity vectors, are transmitted at 445.
[0164] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the invention may be implemented in conjunction with any other type of computing environment now known or later developed.
[0165] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.
[0166] The characteristics are as follows:
[0167] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically on an as-needed basis, without the need for human interaction with the service provider.
[0168] Broad Network Access: Capabilities are made available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs).
[0169] Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Location independence is significant in that consumers typically have no control or knowledge over the exact location of the resources provided, but may be able to specify location at a higher level of abstraction (e.g. country, state or data center).
[0170] Rapid Elasticity: Capacity can be rapidly and elastically provisioned, in some cases automatically, to instantly scale out, and rapidly released to instantly scale in. To the consumer, the capacity available for provisioning often appears unlimited and can be purchased in any amount at any time.
[0171] Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of utilized services.
[0172] The service model is as follows:
[0173] Software as a Service (SaaS): The capability offered to the consumer is the ability to use the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.
[0174] Platform as a Service (PaaS): The ability offered to a consumer is the ability to deploy applications that the consumer has created or acquired, generated using programming languages and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application hosting environment configuration.
[0175] Infrastructure as a Service (IaaS): The ability offered to the consumer is the ability to provision processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over select networking components (e.g., host firewalls).
[0176] The deployment model is as follows:
[0177] Private Cloud: The cloud infrastructure is operated exclusively for an organization. The cloud infrastructure may be managed by the organization or a third party and may be on-premise or off-premise.
[0178] Community Cloud: The cloud infrastructure is shared by multiple organizations to support a particular community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). The cloud infrastructure may be managed by the organizations or a third party and may reside on-premise or off-premise.
[0179] Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.
[0180] Hybrid Cloud: The cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain a unique entity but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).
[0181] Cloud computing environments are service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.
[0182] Referring now to FIG. 5, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or cell phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, may communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private, community, public, or hybrid cloud, or combinations thereof, as described above (not shown). This allows the cloud computing environment 50 to offer infrastructure, platform, or software, or combinations thereof, as a service without the cloud consumer having to maintain resources on the local computing device. The types of computing devices 54A-N shown in FIG. 5 are intended to be exemplary only, and it is understood that the computing node 10 and cloud computing environment 50 may communicate with any type of computerized device via any type of network or network-addressable connection or both (e.g., using a web browser).
[0183] Referring now to Figure 6, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 5) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 6 are intended to be merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:
[0184] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, reduced instruction set computing (RISC) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.
[0185] The virtualization layer 70 provides an abstraction layer over which the following examples of virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75.
[0186] In one example, the management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and provides billing or charging for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management such that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-positioning and procurement of cloud computing resources where future requirements are anticipated according to SLAs.
[0187] The workload layer 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and tensor comparison programs 175.
[0188] The present invention may be a system, method or computer program product, or combination thereof, at any possible level of technical detail of integration. The present invention may be advantageously implemented in any system, single or parallel, that processes instruction streams. The computer program product may include a computer readable storage medium having computer readable program instructions for causing a processor to perform aspects of the present invention.
[0189] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. Computer-readable storage medium or computer-readable storage device, as used herein, should not be construed to be a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or an electrical signal transmitted over a wire.
[0190] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium in the respective computing / processing device for storage.
[0191] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state configuration data, configuration data for an integrated circuit, or object oriented programming languages such as Smalltalk, C++, procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the invention.
[0192] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.
[0193] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, to produce a machine. These computer readable program instructions may be stored in a computer readable storage medium, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions implementing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, and direct the computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.
[0194] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.
[0195] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. It should be noted that each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or a combination of dedicated hardware and computer instructions.
[0196] References herein to "one embodiment," "one embodiment," "exemplary embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but all embodiments may not necessarily include that particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Moreover, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed to be within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.
[0197] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, it will be understood that the terms "comprises" and / or "comprises" when used in this patent specification specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.
[0198] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the spirit and scope of the present invention. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or improvements of the technology over the technology found in the market, and to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A computer-implemented method for tensor comparison across a network by computer information processing, comprising: determining, by one or more computer processors, a first parameter of a first vector representation of a first tensor object associated with the first processor; receiving, by the one or more computer processors, second parameters associated with a second vector representation of a second tensor object associated with a second processor; determining, by the one or more computer processors, a first difference value between the first parameter and the second parameter; transmitting, by the one or more computer processors, the first vector representation from the first processor to the second processor according to the first difference value relative to a first threshold; 4. A computer-implemented method comprising:
2. determining, by the one or more computer processors, a third parameter of the first vector representation of the first tensor object; and receiving, by the one or more computer processors, a fourth parameter associated with the second vector representation of the second tensor object; determining, by the one or more computer processors, a second difference value between the first vector representation and the second vector representation in accordance with the first parameter, the second parameter, the third parameter, and the fourth parameter; Further comprising:
2. The computer-implemented method of claim 1, wherein transmitting, by the one or more computer processors, the first vector representation from the first processor to the second processor is further performed according to the second difference value relative to a second threshold.
3. determining, by the one or more computer processors, the third parameter, further performed in accordance with a seed value; The computer-implemented method of claim 2 , wherein the fourth parameter is further determined by the one or more computer processors according to the seed value.
4. 3. The computer-implemented method of claim 2, wherein the third parameter comprises a code vector associated with the first vector representation, and the fourth parameter comprises a code vector associated with the second vector representation.
5. 3. The computer-implemented method of claim 2, further comprising: sending, by the one or more computer processors, an explicit "no tensor sends" message from the first processor to the second processor according to the second difference value relative to the second threshold.
6. 3. The computer-implemented method of claim 2, further comprising: implementing, by the one or more computer processors, a communications timeout between the first processor and the second processor according to the second difference value relative to the second threshold.
7. The computer-implemented method of claim 1 , wherein the first parameter comprises a length of the first vector representation and the second parameter comprises a length of the second vector representation.
8. A computer program for tensor comparison across networks, said computer program causing a computer to perform the computer implemented methods of claims 1 to 7.
9. 1. A computer system for tensor comparison across networks, the computer system comprising: one or more computer processors; one or more computer readable storage devices; and program instructions stored on the one or more computer readable storage devices for execution by the one or more computer processors, the computer system comprising: determining a first parameter of a first vector representation of a first tensor object associated with the first processor; receiving second parameters associated with a second vector representation of a second tensor object associated with a second processor; determining a first difference between the first parameter and the second parameter; transmitting the first vector representation from the first processor to the second processor according to the first difference value relative to a first threshold; A computer system that executes the above.
10. The computer system comprises: determining a third parameter of the first vector representation of the first tensor object; receiving a fourth parameter associated with the second vector representation of the second tensor object; determining a second difference value between the first vector representation and the second vector representation according to the first parameter, the second parameter, the third parameter, and the fourth parameter; Further execute 10. The computer system of claim 9, wherein transmitting the first vector representation from the first processor to the second processor is further performed according to the second difference value relative to a second threshold.
11. 11. The computer system of claim 10, wherein determining the third parameter is further performed according to a seed value, and the fourth parameter is further determined according to the seed value.
12. 11. The computer system of claim 10, wherein the third parameter comprises a code vector associated with the first vector representation, and the fourth parameter comprises a code vector associated with the second vector representation.
13. 11. The computer system of claim 10, further comprising: transmitting a "no tensor send" message from the first processor to the second processor according to the second difference value relative to the second threshold.
14. 11. The computer system of claim 10, further comprising: implementing a communications timeout between the first processor and the second processor according to the second difference value relative to the second threshold.
15. A computer-readable recording medium having the computer program according to claim 8 recorded thereon.
Citation Information
Patent Citations
Compiling optimization method and device for machine learning model
CN111580827A
Method and device for neural network generation
JP2020149663A
Unsupervised multisource temporal anomaly detection
US20170004166A1
System and Method with Federated Learning Model for Medical Research Applications
US20200293887A1
