Systems, methods, and computer program products for saving memory during training of knowledge graph neural networks

Abstract

Systems, methods, and computer program products are provided for saving memory during training of knowledge graph neural networks. The method comprises: receiving a training data set, the training data set comprising a first knowledge graph embedded set associated with a plurality of entities for a first layer of a knowledge graph; inputting the training data set into a knowledge graph neural network to generate at least one further knowledge graph embedded set associated with the plurality of entities for at least one further layer of the knowledge graph; quantizing the at least one further knowledge graph embedded set to provide at least one quantized knowledge graph embedded set; storing the at least one quantized knowledge graph embedded set in a memory; and de-quantizing the at least one quantized knowledge graph embedded set to provide at least one de-quantized knowledge graph embedded set.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 337,329, filed on May 2, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to training knowledge graph neural networks and, in some non-limiting embodiments or aspects, to systems, methods, and computer program products for conserving memory during training of knowledge graph neural networks. Background Art

[0004] A knowledge graph (KG) can use nodes to represent a network of real-world entities (e.g., objects, events, situations, concepts, etc.). A KG can use edges to connect nodes that have relationships with each other to illustrate the relationships between such real-world entities. The information represented in a KG can be visualized as a graph structure built using nodes and edges. A KG may include a formal representation of entities and relationships between entities in an underlying knowledge model. This formal representation (e.g., using metadata and a knowledge model) may define the context and semantics of the KG. A KG differs from other graphical representations by including the context and semantics surrounding the underlying data (e.g., entities and relationships between entities).

[0005] Some systems can use machine learning models to learn low-dimensional representations of entities and relations of a KG while preserving semantic meaning. Using the embedded representation of the KG, the KG can be used together with machine learning models in various applications such as link prediction, triple classification, entity recognition, clustering, and recommendation systems. For example, a deep learning model such as a deep neural network (DNN) can be trained to predict whether a customer will interact with an entity. These predictions can be used to provide product recommendations to customers in a recommendation system.

[0006] However, some deep learning models (e.g., knowledge graph neural networks (KGNNs)) may be relatively large and / or may require relatively large amounts of computing resources (e.g., processor resources, memory resources, and / or the like) to train and use. In addition, deep learning models using KGs may receive relatively large amounts of data for training, and / or may require a large amount of memory to store embeddings from the KG (e.g., KG embeddings) during training. For example, because the amount of data embedded in the KG is too large for the memory of a single computing device, training a deep learning model on the KG may require distributed resources (e.g., computing resources located on different networked computing devices communicating over a network) to store and access the KG embeddings. In order to take advantage of distributed resources, the KG may need to be partitioned so that individual partitions can be processed by each computing device. However, when the KG includes multi-hop relationships in the knowledge model (e.g., a first node has a relationship with a third node in the KG via a second (e.g., intermediate) node, and the first node and the third node have no direct relationship link to each other), because the complexity of learning multi-hop relationships in the KG may be high and / or because partitioning the KG may result in the loss of one or more hops in the multi-hop connection, a distributed framework for training deep learning models may not be sufficient. In addition, because training a KGNN may require low latency (e.g., delay between messages), training a KGNN using distributed resources may not be feasible for the KG. For example, low latency may be more difficult to achieve using distributed resources. Summary of the invention

[0007] It is therefore an object of the present disclosure to provide systems, methods, and computer program products for saving memory during training of KGNNs that overcome some or all of the above-identified deficiencies.

[0008] According to non-limiting embodiments or aspects, a computer-implemented method for saving memory during training of a KGNN is provided. In some non-limiting embodiments or aspects, the computer-implemented method may include: receiving a training data set, the training data set including a first knowledge graph embedding set associated with a plurality of entities for a first layer of a knowledge graph. In some non-limiting embodiments or aspects, the computer-implemented method may further include: inputting the training data set into a knowledge graph neural network to generate at least one additional knowledge graph embedding set associated with a plurality of entities for at least one additional layer of the knowledge graph. In some non-limiting embodiments or aspects, the computer-implemented method may further include: quantizing at least one additional knowledge graph embedding set to provide at least one quantized knowledge graph embedding set. In some non-limiting embodiments or aspects, the computer-implemented method may further include: storing at least one quantized knowledge graph embedding set in a memory. In some non-limiting embodiments or aspects, the computer-implemented method may further include: dequantizing at least one quantized knowledge graph embedding set to provide at least one dequantized knowledge graph embedding set, wherein the at least one dequantized knowledge graph embedding set is approximate to at least one additional knowledge graph embedding set, and wherein the gradient used for backpropagation is determined based on the at least one dequantized knowledge graph embedding set.

[0009] In some non-limiting embodiments or aspects, the computer-implemented method may further include: determining gradients for back-propagation based on at least one dequantized knowledge graph embedding set; and updating parameters of the knowledge graph neural network based on the gradients.

[0010] In some non-limiting embodiments or aspects, each respective additional knowledge graph embedding set in the at least one additional knowledge graph embedding set may include a respective activation map of a respective layer of the knowledge graph neural network for the at least one additional layer.

[0011] In some non-limiting embodiments or aspects, quantizing the at least one additional set of knowledge graph embeddings may include quantizing the at least one additional set of knowledge graph embeddings based on the following equation: in is the corresponding knowledge graph embedding in at least one other knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer, l is the corresponding layer of at least one additional layer, and is the unbiased random rounding operator.

[0012] In some non-limiting embodiments or aspects, dequantizing at least one knowledge graph embedding set may include dequantizing at least one knowledge graph embedding set based on the following equation: in is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer and l is the corresponding layer of at least one additional layer.

[0013] In some non-limiting embodiments or aspects, the knowledge graph neural network may have an expected value based on the following equation: in is the expected measure, is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, l is the corresponding layer of at least one additional layer, and is a corresponding knowledge graph embedding in at least one additional knowledge graph embedding set, and the knowledge graph neural network may have a variance value based on the following equation: where d is the dimension of the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the offset, It is for The range of B=2 b -1, where b is the number of digits for the integer.

[0014] In some non-limiting embodiments or aspects, each respective quantized knowledge graph embedding in at least one set of quantized knowledge graph embeddings may include an integer representation, and each respective additional knowledge graph embedding in at least one additional set of knowledge graph embeddings may include a floating point representation.

[0015] In some non-limiting embodiments or aspects, storing at least one quantized knowledge graph embedding set in a memory may include overwriting at least one additional knowledge graph embedding set with the at least one quantized knowledge graph embedding set in a memory of a graphics processing unit (GPU).

[0016] According to non-limiting embodiments or aspects, a system for saving memory during training of a KGNN is provided. In some non-limiting embodiments or aspects, the system may include: at least one processor; and at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium storing instructions, which, when executed by at least one processor, causes the at least one processor to receive a training data set, the training data set comprising a first knowledge graph embedding set associated with a plurality of entities for a first layer of a knowledge graph. In some non-limiting embodiments or aspects, the instructions may further cause at least one processor to input the training data set into a knowledge graph neural network to generate at least one additional knowledge graph embedding set associated with a plurality of entities for at least one additional layer of the knowledge graph. In some non-limiting embodiments or aspects, the instructions may further cause at least one processor to quantize at least one additional knowledge graph embedding set to provide at least one quantized knowledge graph embedding set. In some non-limiting embodiments or aspects, the instructions may further cause at least one processor to store at least one quantized knowledge graph embedding set in a memory. In some non-limiting embodiments or aspects, the instructions may further cause at least one processor to dequantize at least one quantized knowledge graph embedding set to provide at least one dequantized knowledge graph embedding set, wherein the at least one dequantized knowledge graph embedding set approximates at least one additional knowledge graph embedding set, wherein gradients for backpropagation are determined based on the at least one dequantized knowledge graph embedding set.

[0017] In some non-limiting embodiments or aspects, the plurality of entities may include a plurality of users and a plurality of projects, and the instructions may further cause at least one processor to determine, based on a knowledge graph neural network, a prediction indicating whether a user from the plurality of users would be interested in a project in which the user has not yet engaged.

[0018] In some non-limiting embodiments or aspects, at least one additional layer may include a second layer and a third layer, at least one additional knowledge graph embedding set may include a second knowledge graph embedding set associated with the second layer and a third knowledge graph embedding set associated with the third layer, and when a training dataset is input, the instructions may further cause at least one processor to: input the training dataset into the knowledge graph neural network to generate a second knowledge graph embedding set associated with multiple entities for the second layer of the knowledge graph; and input the second knowledge graph embedding set into the knowledge graph neural network to generate a third knowledge graph embedding set associated with multiple entities for the third layer of the knowledge graph.

[0019] In some non-limiting embodiments or aspects, the instructions may further cause at least one processor to: determine gradients for back-propagation based on at least one set of dequantized knowledge graph embeddings; and update parameters of the knowledge graph neural network based on the gradients.

[0020] In some non-limiting embodiments or aspects, each respective additional knowledge graph embedding set in the at least one additional knowledge graph embedding set may include a respective activation map of a respective layer of the knowledge graph neural network for the at least one additional layer.

[0021] In some non-limiting embodiments or aspects, when quantizing the at least one additional set of knowledge graph embeddings, the instructions may further cause the at least one processor to: quantize the at least one additional set of knowledge graph embeddings based on the following equation: in is the corresponding knowledge graph embedding in at least one other knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer, l is the corresponding layer of at least one additional layer, and is the unbiased random rounding operator.

[0022] In some non-limiting embodiments or aspects, when dequantizing the at least one knowledge graph embedding set, the instructions further cause the at least one processor to: dequantize the at least one knowledge graph embedding set based on the following equation: in is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer and l is the corresponding layer of at least one additional layer.

[0023] In some non-limiting embodiments or aspects, the knowledge graph neural network may have an expected value based on the following equation: in is the expected measure, is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, l is the corresponding layer of at least one additional layer, and is a corresponding knowledge graph embedding in at least one additional knowledge graph embedding set, and the knowledge graph neural network may have a variance value based on the following equation: where d is the dimension of the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the offset, It is for The range of B=2 b -1, where b is the number of digits for the integer.

[0024] In some non-limiting embodiments or aspects, each respective quantized knowledge graph embedding in at least one set of quantized knowledge graph embeddings may include an integer representation, and each respective additional knowledge graph embedding in at least one additional set of knowledge graph embeddings may include a floating point representation.

[0025] In some non-limiting embodiments or aspects, when at least one quantized knowledge graph embedding set is stored in a memory, the instructions may further cause at least one processor to: overwrite at least one additional knowledge graph embedding set with the at least one quantized knowledge graph embedding set in a memory of a graphics processing unit (GPU).

[0026] According to a non-limiting embodiment or aspect, a computer program product for saving memory during training of a KGNN is provided. In some non-limiting embodiments or aspects, the computer program product may include at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium including program instructions, which, when executed by at least one processor, causes the at least one processor to receive a training data set, the training data set including a first knowledge graph embedding set associated with a plurality of entities for a first layer of a knowledge graph. In some non-limiting embodiments or aspects, the program instructions may further cause at least one processor to input the training data set into a knowledge graph neural network to generate at least one additional knowledge graph embedding set associated with a plurality of entities for at least one additional layer of the knowledge graph. In some non-limiting embodiments or aspects, the program instructions may further cause at least one processor to quantize at least one additional knowledge graph embedding set to provide at least one quantized knowledge graph embedding set. In some non-limiting embodiments or aspects, the program instructions may further cause at least one processor to store at least one quantized knowledge graph embedding set in a memory. In some non-limiting embodiments or aspects, the program instructions may further cause at least one processor to dequantize at least one quantized knowledge graph embedding set to provide at least one dequantized knowledge graph embedding set, wherein the at least one dequantized knowledge graph embedding set approximates at least one additional knowledge graph embedding set, wherein gradients for back-propagation are determined based on the at least one dequantized knowledge graph embedding set.

[0027] In some non-limiting embodiments or aspects, the plurality of entities may include a plurality of users and a plurality of projects, and the program instructions may further cause at least one processor to determine, based on a knowledge graph neural network, a prediction indicating whether a user from the plurality of users would be interested in a project in which the user has not yet participated.

[0028] In some non-limiting embodiments or aspects, at least one additional layer may include a second layer and a third layer, at least one additional knowledge graph embedding set may include a second knowledge graph embedding set associated with the second layer and a third knowledge graph embedding set associated with the third layer, and when a training data set is input, the program instructions may further cause at least one processor to: input the training data set into the knowledge graph neural network to generate a second knowledge graph embedding set associated with multiple entities for the second layer of the knowledge graph; and input the second knowledge graph embedding set into the knowledge graph neural network to generate a third knowledge graph embedding set associated with multiple entities for the third layer of the knowledge graph.

[0029] In some non-limiting embodiments or aspects, the program instructions may further cause at least one processor to: determine gradients for back-propagation based on at least one dequantized knowledge graph embedding set; and update parameters of the knowledge graph neural network based on the gradients.

[0030] In some non-limiting embodiments or aspects, each respective additional knowledge graph embedding set in the at least one additional knowledge graph embedding set may include a respective activation map of a respective layer of the knowledge graph neural network for the at least one additional layer.

[0031] In some non-limiting embodiments or aspects, when quantizing the at least one additional set of knowledge graph embeddings, the program instructions may further cause the at least one processor to: quantize the at least one additional set of knowledge graph embeddings based on the following equation: in is the corresponding knowledge graph embedding in at least one other knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer, l is the corresponding layer of at least one additional layer, and is the unbiased random rounding operator.

[0032] In some non-limiting embodiments or aspects, when dequantizing at least one knowledge graph embedding set, the program instructions may further cause at least one processor to: dequantize at least one knowledge graph embedding set based on the following equation: in is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer and l is the corresponding layer of at least one additional layer.

[0033] In some non-limiting embodiments or aspects, the knowledge graph neural network may have an expected value based on the following equation: in is the expected measure, is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, l is the corresponding layer of at least one additional layer, and is a corresponding knowledge graph embedding in at least one additional knowledge graph embedding set, and the knowledge graph neural network may have a variance value based on the following equation: where d is the dimension of the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the offset, It is for The range of B=2 b -1, where b is the number of digits for the integer.

[0034] In some non-limiting embodiments or aspects, each respective quantized knowledge graph embedding in at least one set of quantized knowledge graph embeddings may include an integer representation, and each respective additional knowledge graph embedding in at least one additional set of knowledge graph embeddings may include a floating point representation.

[0035] In some non-limiting embodiments or aspects, when at least one quantized knowledge graph embedding set is stored in a memory, the program instructions may further cause at least one processor to: overwrite at least one additional knowledge graph embedding set with the at least one quantized knowledge graph embedding set in a memory of a graphics processing unit (GPU).

[0036] Other non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0037] Clause 1: A computer-implemented method comprising: receiving, with at least one processor, a training dataset comprising a first knowledge graph embedding set associated with a plurality of entities for a first layer of a knowledge graph; inputting, with at least one processor, the training dataset into a knowledge graph neural network to generate, with at least one processor, at least one additional knowledge graph embedding set associated with a plurality of entities for at least one additional layer of the knowledge graph; quantizing, with at least one processor, the at least one additional knowledge graph embedding set to provide at least one quantized knowledge graph embedding set; storing, with at least one processor, the at least one quantized knowledge graph embedding set in a memory; and dequantizing, with at least one processor, the at least one quantized knowledge graph embedding set to provide at least one dequantized knowledge graph embedding set, wherein the at least one dequantized knowledge graph embedding set approximates the at least one additional knowledge graph embedding set, wherein a gradient for back-propagation is determined based on the at least one dequantized knowledge graph embedding set.

[0038] Clause 2: A computer-implemented method as described in Clause 1, wherein the plurality of entities include a plurality of users and a plurality of projects, the method further comprising: utilizing at least one processor to determine, based on a knowledge graph neural network, a prediction indicating whether a user among the plurality of users will be interested in a project in which the user has not yet participated.

[0039] Clause 3: A computer-implemented method as described in any of the preceding clauses, wherein at least one additional layer includes a second layer and a third layer, wherein at least one additional knowledge graph embedding set includes a second knowledge graph embedding set associated with the second layer and a third knowledge graph embedding set associated with the third layer, and wherein inputting a training data set includes: inputting the training data set into a knowledge graph neural network using at least one processor to generate a second knowledge graph embedding set associated with multiple entities for the second layer of the knowledge graph; and inputting the second knowledge graph embedding set into the knowledge graph neural network using at least one processor to generate a third knowledge graph embedding set associated with multiple entities for the third layer of the knowledge graph.

[0040] Clause 4: A computer-implemented method as described in any of the preceding clauses, further comprising: determining, using at least one processor, gradients for backpropagation based on at least one dequantized knowledge graph embedding set; and updating, using at least one processor, parameters of the knowledge graph neural network based on the gradients.

[0041] Clause 5: A computer-implemented method as described in any preceding clause, wherein each respective additional knowledge graph embedding set of at least one additional knowledge graph embedding set comprises a respective activation map of a respective layer of the knowledge graph neural network for the at least one additional layer.

[0042] Clause 6: A computer-implemented method as described in any preceding clause, wherein quantizing at least one additional set of knowledge graph embeddings comprises quantizing at least one additional set of knowledge graph embeddings based on the following equation: in is the corresponding knowledge graph embedding in at least one other knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer, l is the corresponding layer of at least one additional layer, and is the unbiased random rounding operator.

[0043] Clause 7: A computer-implemented method as described in any preceding clause, wherein dequantizing at least one knowledge graph embedding set comprises dequantizing at least one knowledge graph embedding set based on the following equation: in is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer and l is the corresponding layer of at least one additional layer.

[0044] Clause 8: A computer-implemented method as described in any preceding clause, wherein the knowledge graph neural network has an expected value based on the following equation: in is the expected measure, is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, l is the corresponding layer of at least one additional layer, and is a corresponding knowledge graph embedding in at least one other set of knowledge graph embeddings, and wherein the knowledge graph neural network has a variance value based on the following equation: where d is the dimension of the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the offset, It is for The range of B=2 b -1, where b is the number of digits for the integer.

[0045] Clause 9: A computer-implemented method as described in any preceding clause, wherein each corresponding quantized knowledge graph embedding in at least one set of quantized knowledge graph embeddings comprises an integer representation, and each corresponding additional knowledge graph embedding in at least one additional set of knowledge graph embeddings comprises a floating point representation.

[0046] Clause 10: A computer-implemented method as described in any preceding clause, wherein storing at least one quantized knowledge graph embedding set in a memory comprises: overwriting at least one other knowledge graph embedding set with the at least one quantized knowledge graph embedding set in a memory of a graphics processing unit (GPU).

[0047] Item 11: A system comprising: at least one processor; and at least one non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to: receive a training dataset comprising a first knowledge graph embedding set associated with a plurality of entities for a first layer of a knowledge graph; input the training dataset into a knowledge graph neural network to generate at least one additional knowledge graph embedding set associated with a plurality of entities for at least one additional layer of the knowledge graph; quantize the at least one additional knowledge graph embedding set to provide at least one quantized knowledge graph embedding set; store the at least one quantized knowledge graph embedding set in a memory; and dequantize the at least one quantized knowledge graph embedding set to provide at least one dequantized knowledge graph embedding set, wherein the at least one dequantized knowledge graph embedding set approximates the at least one additional knowledge graph embedding set, wherein a gradient for back-propagation is determined based on the at least one dequantized knowledge graph embedding set.

[0048] Clause 12: The system of clause 11, wherein the plurality of entities comprises a plurality of users and a plurality of projects, and wherein the instructions further cause the at least one processor to: determine, based on a knowledge graph neural network, a prediction indicating whether a user from the plurality of users would be interested in a project in which the user has not yet engaged.

[0049] Clause 13: A system as described in clause 11 or clause 12, wherein at least one additional layer includes a second layer and a third layer, wherein at least one additional knowledge graph embedding set includes a second knowledge graph embedding set associated with the second layer and a third knowledge graph embedding set associated with the third layer, and wherein, when a training data set is input, the instructions further cause at least one processor to: input the training data set into the knowledge graph neural network to generate a second knowledge graph embedding set associated with multiple entities for the second layer of the knowledge graph; and input the second knowledge graph embedding set into the knowledge graph neural network to generate a third knowledge graph embedding set associated with multiple entities for the third layer of the knowledge graph.

[0050] Clause 14: A system as described in any of clauses 11-13, wherein the instructions further cause at least one processor to: determine gradients for backpropagation based on at least one dequantized knowledge graph embedding set; and update parameters of the knowledge graph neural network based on the gradients.

[0051] Clause 15: A system as described in any of clauses 11-14, wherein each respective additional knowledge graph embedding set of at least one additional knowledge graph embedding set comprises a respective activation map of a respective layer of the knowledge graph neural network for the at least one additional layer.

[0052] Clause 16: The system of any of clauses 11-15, wherein when quantizing the at least one additional set of knowledge graph embeddings, the instructions further cause the at least one processor to: quantize the at least one additional set of knowledge graph embeddings based on the following equation: in is the corresponding knowledge graph embedding in at least one other knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer, l is the corresponding layer of at least one additional layer, and is the unbiased random rounding operator.

[0053] Clause 17: The system of any of clauses 11-16, wherein when dequantizing the at least one knowledge graph embedding set, the instructions further cause the at least one processor to: dequantize the at least one knowledge graph embedding set based on the following equation: in is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer and l is the corresponding layer of at least one additional layer.

[0054] Clause 18: A system as described in any of clauses 11-17, wherein the knowledge graph neural network has an expected value based on the following equation: in is the expected measure, is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, l is the corresponding layer of at least one additional layer, and is a corresponding knowledge graph embedding in at least one other set of knowledge graph embeddings, and wherein the knowledge graph neural network has a variance value based on the following equation: where d is the dimension of the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the offset, It is for The range of B=2 b -1, where b is the number of digits for the integer.

[0055] Clause 19: A system as described in any of clauses 11-18, wherein each corresponding quantized knowledge graph embedding in at least one set of quantized knowledge graph embeddings comprises an integer representation, and each corresponding additional knowledge graph embedding in at least one additional set of knowledge graph embeddings comprises a floating point representation.

[0056] Clause 20: A system as described in any of clauses 11-19, wherein, when at least one quantized knowledge graph embedding set is stored in a memory, the instructions further cause at least one processor to: overwrite at least one additional knowledge graph embedding set with the at least one quantized knowledge graph embedding set in a memory of a graphics processing unit (GPU).

[0057] Clause 21: A computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: receive a training data set comprising a first knowledge graph embedding set associated with a plurality of entities for a first layer of a knowledge graph; input the training data set into a knowledge graph neural network to generate at least one additional knowledge graph embedding set associated with a plurality of entities for at least one additional layer of the knowledge graph; quantize the at least one additional knowledge graph embedding set to provide at least one quantized knowledge graph embedding set; store the at least one quantized knowledge graph embedding set in a memory; and dequantize the at least one quantized knowledge graph embedding set to provide at least one dequantized knowledge graph embedding set, wherein the at least one dequantized knowledge graph embedding set approximates the at least one additional knowledge graph embedding set, wherein a gradient for back-propagation is determined based on the at least one dequantized knowledge graph embedding set.

[0058] Clause 22: A computer program product as described in clause 21, wherein the plurality of entities include a plurality of users and a plurality of projects, and wherein the program instructions further cause at least one processor to: determine, based on a knowledge graph neural network, a prediction indicating whether a user from the plurality of users will be interested in a project in which the user has not yet engaged.

[0059] Clause 23: A computer program product as described in clause 21 or clause 22, wherein at least one additional layer includes a second layer and a third layer, wherein at least one additional knowledge graph embedding set includes a second knowledge graph embedding set associated with the second layer and a third knowledge graph embedding set associated with the third layer, and wherein, when a training data set is input, the program instructions further cause at least one processor to: input the training data set into the knowledge graph neural network to generate a second knowledge graph embedding set associated with multiple entities for the second layer of the knowledge graph; and input the second knowledge graph embedding set into the knowledge graph neural network to generate a third knowledge graph embedding set associated with multiple entities for the third layer of the knowledge graph.

[0060] Clause 24: A computer program product as described in any of clauses 21-23, wherein the program instructions further cause at least one processor to: determine gradients for backpropagation based on at least one dequantized knowledge graph embedding set; and update parameters of the knowledge graph neural network based on the gradients.

[0061] Clause 25: A computer program product as described in any of clauses 21-24, wherein each respective additional knowledge graph embedding set of at least one additional knowledge graph embedding set comprises a respective activation map of the knowledge graph neural network for a respective layer of the at least one additional layer.

[0062] Clause 26: A computer program product as described in any of clauses 21-25, wherein, when quantizing the at least one additional set of knowledge graph embeddings, the program instructions further cause the at least one processor to: quantize the at least one additional set of knowledge graph embeddings based on the following equation: in is the corresponding knowledge graph embedding in at least one other knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer, l is the corresponding layer of at least one additional layer, and is the unbiased random rounding operator.

[0063] Clause 27: A computer program product as described in any of clauses 21-26, wherein, when dequantizing at least one knowledge graph embedding set, the program instructions further cause at least one processor to: dequantize at least one knowledge graph embedding set based on the following equation: in is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer and l is the corresponding layer of at least one additional layer.

[0064] Clause 28: A computer program product as described in any of clauses 21-27, wherein the knowledge graph neural network has an expected value based on the following equation: in is the expected measure, is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, l is the corresponding layer of at least one additional layer, and is a corresponding knowledge graph embedding in at least one other set of knowledge graph embeddings, and wherein the knowledge graph neural network has a variance value based on the following equation: where d is the dimension of the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the offset, It is for The range of B=2 b -1, where b is the number of digits for the integer.

[0065] Clause 29: A computer program product as described in any of clauses 21-28, wherein each corresponding quantized knowledge graph embedding in at least one set of quantized knowledge graph embeddings comprises an integer representation, and each corresponding additional knowledge graph embedding in at least one additional set of knowledge graph embeddings comprises a floating point representation.

[0066] Clause 30: A computer program product as described in any of clauses 21-29, wherein, when at least one quantized knowledge graph embedding set is stored in a memory, the program instructions further cause at least one processor to: overwrite at least one additional knowledge graph embedding set with the at least one quantized knowledge graph embedding set in a memory of a graphics processing unit (GPU).

[0067] These and other features and characteristics of the present disclosure, as well as methods of operation and functions of the related structural elements and combinations of parts and economies of manufacture, will become more apparent when considering the following description and appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for purposes of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Additional advantages and details are explained in more detail below with reference to a non-limiting exemplary embodiment shown in the schematic drawings, in which:

[0069] Figure 1 is a schematic diagram of a system for saving memory during training of a KGNN according to some non-limiting embodiments or aspects;

[0070] Figure 2 is a flowchart of a method for saving memory during training of a KGNN according to some non-limiting embodiments or aspects;

[0071] Figure 3 is a diagram of an exemplary environment in which methods, systems, and / or computer program products described herein may be implemented according to some non-limiting embodiments or aspects;

[0072] Figure 4 According to some non-limiting embodiments or aspects Figure 1 and / or Figure 3 schematic diagrams of example components of one or more devices;

[0073] Figure 5 is an example diagram of a knowledge graph neural network according to some non-limiting embodiments or aspects;

[0074] Fig. 6A is an example diagram of a user-item rating matrix according to some non-limiting embodiments or aspects;

[0075] Figure 6B is an example graph of a user-item bipartite graph according to some non-limiting embodiments or aspects;

[0076] Figure 7 is an example diagram of a knowledge graph according to some non-limiting embodiments or aspects; and

[0077] Figure 8 is an example of a user-item entity graph according to some non-limiting embodiments or aspects. DETAILED DESCRIPTION

[0078] For the purposes of the following description, the terms "end," "upper," "lower," "right," "left," "vertical," "horizontal," "top," "bottom," "lateral," "longitudinal," and their derivatives shall relate to the orientation of the embodiments in the accompanying drawings. However, it shall be understood that the embodiments may employ various alternative variations and step sequences, except where expressly specified to the contrary. It shall also be understood that the specific devices and processes illustrated in the drawings and described in the following specification are merely exemplary embodiments or aspects of the disclosed subject matter. Accordingly, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein shall not be considered limiting.

[0079] Aspects, parts, elements, structures, actions, steps, functions, instructions and / or the like used herein should not be understood as critical or necessary unless explicitly described as such. Also, as used herein, the article "one" is intended to include one or more items, and can be used interchangeably with "one or more" and "at least one". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, a combination of related items and unrelated items, and / or the like), and can be used interchangeably with "one or more" or "at least one". In the case of intending to have only one item, the term "one" or similar language is used. Moreover, as used herein, the term "having" and / or the like is intended to be an open term. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "based at least in part on".

[0080] As used herein, the term "acquirer institution" may refer to an entity that is licensed and / or approved by a transaction service provider to initiate transactions (e.g., payment transactions) using a payment device associated with the transaction service provider. Transactions that may be initiated by an acquirer institution may include payment transactions (e.g., purchases, original letter of credit transactions (OCTs), account funding transactions (AFTs), and / or the like). In some non-limiting embodiments or aspects, an acquirer institution may be a financial institution, such as a bank. As used herein, the term "acquirer system" may refer to one or more computing devices operated by or on behalf of an acquirer institution, such as a server computer executing one or more software applications.

[0081] As used herein, the term "account identifier" may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term "token" may refer to an identifier used as a replacement or substitute identifier for an original account identifier such as a PAN. An account identifier may be alphanumeric or any combination of characters and / or symbols. A token may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases and / or the like) such that the token can be used to conduct transactions without directly using the original account identifier. In some examples, an original account identifier such as a PAN may be associated with multiple tokens for different individuals or purposes.

[0082] As used herein, the term "communication" may refer to the reception, acceptance, transmission, transfer, provision and / or the like of data (e.g., information, signals, messages, instructions, commands and / or the like). A unit (e.g., a device, a system, a component of a device or a system, a combination thereof and / or the like) communicating with another unit means that the unit is able to directly or indirectly receive information from the other unit and / or send information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection and / or the like) that is wired and / or wireless in nature. In addition, although the information sent may be modified, processed, relayed and / or routed between the first unit and the second unit, the two units may also communicate with each other. For example, even if the first unit passively receives information and does not actively send information to the second unit, the first unit may communicate with the second unit. As another example, if at least one intermediate unit processes the information received from the first unit and transmits the processed information to the second unit, the first unit may communicate with the second unit.

[0083] As used herein, the term "computing device" may refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components for receiving, processing, and outputting data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or a standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other similar devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0084] As used herein, the terms "electronic wallet" and "electronic wallet application" refer to one or more electronic devices and / or software applications configured to initiate and / or conduct payment transactions. For example, an electronic wallet may include a mobile device that executes an electronic wallet application, and may also include server-side software and / or databases for maintaining and providing transaction data to the mobile device. An "electronic wallet provider" may include an entity that provides and / or maintains electronic wallets for customers, such as Google Android Apple Samsung And / or other similar electronic payment systems. In some non-limiting examples, the issuing bank may be the electronic wallet provider.

[0085] As used herein, the term "issuer institution" may refer to one or more entities, such as a bank, that provide accounts to customers for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide customers with an account identifier, such as a PAN, that uniquely identifies one or more accounts associated with the customer. The account identifier may be embodied on a portable financial device, such as a physical financial instrument (e.g., a payment card), and / or may be electronic and used for electronic payments. The term "issuer system" refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing transactions.

[0086] As used herein, the term "merchant" may refer to a person or entity that provides goods and / or services, or access to goods and / or services, to a customer based on a transaction, such as a payment transaction. The term "merchant" or "merchant system" may also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer that executes one or more software applications.

[0087] As used herein, a "point of sale (POS) device" may refer to one or more devices that may be used by a merchant to conduct a transaction (e.g., a payment transaction) and / or process a transaction. For example, a POS device may include one or more client devices. Additionally or alternatively, a POS device may include a peripheral device, a card reader, a scanning device (e.g., a code scanner), Communication receiver, near field communication (NFC) receiver, radio frequency identification (RFID) receiver and / or other contactless transceiver or receiver, contact-based receiver, payment terminal and / or its analog. As used herein, "point of sale (POS) system" can refer to one or more client devices and / or peripheral devices used by merchants to conduct transactions. For example, a POS system may include one or more POS devices, and / or other similar devices that can be used to conduct payment transactions. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers that are programmed or configured to process online payment transactions through web pages, mobile applications and / or their analogs.

[0088] As used herein, the terms "client" and "client device" may refer to one or more client-side devices or systems (e.g., at a remote location of a transaction service provider) for initiating or facilitating a transaction (e.g., a payment transaction). As an example, a "client device" may refer to one or more POS devices used by a merchant, one or more acquirer host computers used by an acquirer, one or more mobile devices used by a user, and / or the like. In some non-limiting embodiments or aspects, a client device may be an electronic device configured to communicate with one or more networks and initiate or facilitate a transaction. For example, a client device may include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, and / or the like), PDAs, and / or the like. In addition, a "client" may also refer to an entity (e.g., a merchant, an acquirer, and / or the like) that owns, utilizes, and / or operates a client device for initiating a transaction (e.g., for initiating a transaction with a transaction service provider).

[0089] As used herein, the term "payment device" may refer to a payment card (e.g., a credit or debit card), a gift card, a smart card, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a key chain device or fob, an RFID transponder, a retailer discount or membership card, a cellular phone, an electronic wallet mobile application, a personal digital assistant (PDA), a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, and / or the like. In some non-limiting embodiments or aspects, a payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, an account holder's name, and / or the like).

[0090] As used herein, the term "payment gateway" may refer to an entity and / or a payment processing system operated by or on behalf of such an entity, wherein the entity (e.g., a merchant service provider, a payment service provider, a payment service provider, a payment service provider with a contract with an acquirer, a payment aggregator, and / or the like) provides payment services (e.g., a transaction service provider payment service, a payment processing service, and / or the like) to one or more merchants. The payment service may be associated with the use of a portable financial device managed by a transaction service provider. As used herein, the term "payment gateway system" may refer to one or more computer systems, computer devices, servers, server groups, and / or the like operated by or on behalf of a payment gateway.

[0091] As used herein, the term "server" may refer to or include one or more computing devices operated by or facilitating communications and processing by multiple parties in a network environment such as the Internet, but it should be understood that communications may be facilitated through one or more public or private network environments, and that various other arrangements are possible. In addition, multiple computing devices (e.g., servers, point of sale (POS) devices, mobile devices, etc.) that communicate directly or indirectly in a network environment may constitute a "system." As used herein, references to a "server" or "processor" may refer to a previously described server and / or processor stated to perform a previous step or function, a different server and / or processor, and / or a combination of servers and / or processors. For example, as used in the specification and claims, a first server and / or first processor stated to perform a first step or function may refer to the same or different server and / or processor stated to perform a second step or function.

[0092] As used herein, the term "transaction service provider" may refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment assurance through an agreement between the transaction service provider and an issuer institution. For example, a transaction service provider may include, for example The term "transaction processing system" may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server that executes one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0093] Non-limiting embodiments or aspects of the disclosed subject matter are directed to systems, methods, and computer program products for saving memory (including, but not limited to, saving memory during training of a KGNN). For example, non-limiting embodiments or aspects of the disclosed subject matter provide for receiving a training data set comprising a first KG embedding set (e.g., multiple KG embeddings) associated with multiple entities for a first layer of a KG. The training data set can be input to the KGNN to generate at least one additional KG embedding set associated with multiple entities for at least one additional layer of the KG. At least one additional KG embedding set can be quantized to provide at least one quantized KG embedding set. At least one quantized KG embedding set can be stored in a memory (e.g., without storing additional KG embedding sets). At least one quantized KG embedding set can be dequantized to provide at least one dequantized KG embedding set. At least one dequantized KG embedding set can be approximated to at least one additional KG embedding set. A gradient for backpropagation can be determined based on at least one dequantized KG embedding set. Such embodiments or aspects provide techniques and systems for training KGNNs while storing a relatively (e.g., significantly) small amount of data and / or requiring a relatively (e.g., significantly) small amount of computing resources (e.g., memory resources, network resources, processor resources, and / or the like) during the training of the KGNNs. In addition, non-limiting embodiments or aspects of the disclosed subject matter can reduce the amount of memory required to store embeddings (e.g., KG embeddings) from the KGNNs during training. Thus, because reduced memory may be required to store and access the KG embeddings, training the KGNNs can be performed on centralized resources (e.g., a single computing device, such as a single server using a single GPU and / or the like). In addition, when the KG includes multi-hop relationships, a centralized system for training the KGNNs can reduce (e.g., reduce, eliminate, etc.) inaccuracies and / or loss of information from partitioning the KG. In addition, a single centralized system for training the KGNNs can reduce latency issues (e.g., delays between messages transmitted across a network of distributed computing devices), and thus, the time required to train the KGNNs can be reduced (e.g., train the KGNNs faster). In addition, non-limiting embodiments or aspects of the disclosed subject matter can significantly reduce memory usage while maintaining or even improving performance during training of KGNNs. The reduction in memory resources enables the KGNNs to be trained using a single GPU and / or computing device. The reduction in memory improves the training of KGNNs because the size of the KGNNs may otherwise require multiple GPUs in a distributed system (e.g., using distributed resources). Otherwise, the use of multiple GPUs can degrade performance because the KG will need to be partitioned, potentially resulting in loss of information.Although quantization may result in a reduction in accuracy, the use of unbiased quantization ameliorates this reduction in accuracy, thereby enabling the aforementioned reduction in memory and / or computational resources while maintaining performance during training.

[0094] Figure 1 A system 100 for saving memory during training of a KGNN is depicted according to some non-limiting embodiments or aspects. The system 100 may include a knowledge graph quantization system 102, a graphics processing unit (GPU) 104, and a memory 106.

[0095] The knowledge graph quantization system 102 may include a computing device, such as a server (e.g., a single server), a server group, and / or other similar devices. In some non-limiting embodiments or aspects, as described herein, the knowledge graph quantization system 102 may include a processor (e.g., GPU 104) and / or a memory (e.g., memory 106).

[0096] The GPU 104 may include at least one graphics processing unit and / or at least one processor (e.g., a multi-core processor), such as a central processing unit (CPU), an accelerated processing unit (APU), a microprocessor, and / or the like. In some non-limiting embodiments or aspects, the GPU 104 may be programmed to perform the techniques described herein. In some non-limiting embodiments or aspects, the GPU 104 may communicate with the memory 106. In some non-limiting embodiments or aspects, the GPU 104 may be capable of receiving information from the memory 106, storing information in the memory 106, transmitting information to the memory 106, or searching for information stored in the memory 106.

[0097] Memory 106 may include cache memory, random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic storage, optical storage, etc.) that stores information and / or instructions for use by GPU 104.

[0098] In some non-limiting embodiments or aspects, the memory 106 may store KGs, KGNNs, KG embedding sets (e.g., multiple KG embeddings), quantized KG embedding sets (e.g., multiple quantized KG embeddings), dequantized KG embedding sets (e.g., multiple dequantized KG embeddings), and / or the like generated by the knowledge graph quantization system 102 as described herein. In some non-limiting embodiments or aspects, the memory 106 may be implemented by (part of) the GPU 104 (e.g., cache memory, RAM, and / or the like).

[0099] Figure 1 The number and arrangement of systems and devices shown in the FIG. are provided as examples. Figure 1 There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or systems and / or devices arranged in a different manner than the systems and / or devices shown in the drawings. Furthermore, a single system and / or device may be implemented Figure 1 Two or more systems or devices shown in Figure 1 The single system or device shown in the embodiment may be implemented as multiple distributed systems or devices. Additionally or alternatively, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another group of systems or another group of devices of system 100.

[0100] Reference now Figure 2 , illustrated is a process 200 for saving memory during training of a KGNN, according to some non-limiting embodiments or aspects. Figure 2 The steps shown are for exemplary purposes only. It should be understood that additional, fewer, different and / or different order steps may be used in non-limiting embodiments or aspects.

[0101] like Figure 2 As shown, at step 202, process 200 may include receiving a training dataset of KG embeddings. For example, the knowledge graph quantization system 102 (e.g., its GPU 104) may receive a training dataset including a first KG embedding set. In some non-limiting embodiments or aspects, the first KG embedding set may be associated with a plurality of entities for a first layer of the KG.

[0102] In some non-limiting embodiments or aspects, the plurality of entities may include a plurality of users and a plurality of items. In some non-limiting embodiments or aspects, the knowledge graph quantification system 102 may determine a prediction (e.g., an inference) indicating whether a user of the plurality of users will be interested in an item (e.g., a video, a product sold by a merchant, etc.) that the user has not yet engaged with (e.g., interacted with via a computing device such as the client device 306).

[0103] like Figure 2 As shown, at step 204, process 200 may include inputting the training data set into the KGNN. For example, the knowledge graph quantization system 102 (e.g., its GPU 104) may input the training data set into the KGNN to generate at least one additional KG embedding set. In some non-limiting embodiments or aspects, at least one additional KG embedding set may be associated with multiple entities for at least one additional layer of the KG.

[0104] In some non-limiting embodiments or aspects, at least one additional layer may include a second layer and a third layer. In some non-limiting embodiments or aspects, at least one additional KG embedding set may include a second KG embedding set associated with the second layer and a third KG embedding set associated with the third layer. In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 (e.g., its GPU 104) may generate at least one additional KG embedding set for each layer of the KGNN (e.g., additional layers other than the first layer, the second layer, and the third layer).

[0105] In some non-limiting embodiments or aspects, each respective additional KG embedding set in the at least one additional KG embedding set may include a respective activation map of the KGNN for a respective layer of the at least one additional layer.

[0106] In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 (e.g., its GPU 104) can input the training data set into the KGNN to generate a second KG embedding set. In some non-limiting embodiments or aspects, the second KG embedding set can be associated with multiple entities for the second layer of the KG.

[0107] In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 (e.g., its GPU 104) can input the second KG embedding set into the KGNN to generate a third KG embedding set. In some non-limiting embodiments or aspects, the third KG embedding set can be associated with multiple entities for the third layer of the KG.

[0108] In some non-limiting embodiments or aspects, the KGNN may have an expected value based on the following equation:

[0109]

[0110] in is the expected measure, is the corresponding KG embedding in the set of at least one dequantized KG embeddings, l is the corresponding layer of at least one additional layer, and is the corresponding KG embedding in at least one other embedding set.

[0111] In some non-limiting embodiments or aspects, ... can have a variance value based on the following equation:

[0112]

[0113] where d is the dimension of the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, It is for The range of B=2b -1, where b is the number of digits for the integer.

[0114] like Figure 2 As shown, process 200 may include quantizing the knowledge graph embeddings at step 206. For example, the knowledge graph quantization system 102 (e.g., its GPU 104) may quantize at least one additional knowledge graph embedding set to provide at least one quantized knowledge graph embedding set.

[0115] In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 may quantize at least one additional set of knowledge graph embeddings based on the following equation:

[0116]

[0117] in is the corresponding KG embedding in at least one other KG embedding set, is the corresponding quantized KG embedding scaled to [0, B] in at least one quantized KG embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer, l is the corresponding layer of at least one additional layer, and is the unbiased random rounding operator.

[0118] In some non-limiting embodiments or aspects, each corresponding quantized KG embedding in at least one quantized KG embedding set may include an integer representation (e.g., an 8-bit integer representation), and each corresponding additional KG embedding in at least one additional KG embedding set may include a floating point representation (e.g., a 32-bit floating point representation).

[0119] like Figure 2 As shown, at step 208, process 200 may include storing the quantized KG embeddings. For example, the knowledge graph quantization system 102 (e.g., its GPU 104) may store at least one quantized KG embedding set in the memory 106. In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 (e.g., its GPU 104) may store at least one quantized KG embedding set in the memory 106 without storing at least one additional KG embedding set in the memory 106, and / or may store at least one quantized KG embedding set in the memory 106 instead of at least one additional KG embedding set.

[0120] In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 (eg, the GPU 104 thereof) may overwrite at least one additional KG embedding set in the memory 106 with at least one quantized KG embedding set.

[0121] like Figure 2 As shown, at step 210, process 200 may include dequantizing the quantized KG embeddings. For example, the knowledge graph quantization system 102 (e.g., its GPU 104) may dequantize at least one quantized KG embedding set to provide at least one dequantized KG embedding set. In some non-limiting embodiments or aspects, at least one dequantized KG embedding set may be approximated to at least one additional KG embedding set. In some non-limiting embodiments or aspects, a gradient for back propagation may be determined based on at least one dequantized KG embedding set.

[0122] In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 (e.g., its GPU 104) may dequantize at least one quantized KG embedding set based on the following equation:

[0123]

[0124] in is the corresponding KG embedding in at least one dequantized KG embedding set, is the corresponding quantized KG embedding scaled to [0, B] in at least one quantized KG embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer and l is the corresponding layer of at least one additional layer.

[0125] In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 (e.g., its GPU 104) can determine the gradient for back propagation based on at least one dequantized KG embedding set. In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 (e.g., its GPU 104) can update the parameters of the KGNN based on the gradient.

[0126] Reference now Figure 3 , Figure 3 3 is a diagram of a non-limiting embodiment or aspect of an exemplary environment 300 in which systems, products, and / or methods as described herein may be implemented. Figure 3As shown, the environment 300 includes a transaction service provider system 302, an issuer system 304, a client device 306, a merchant system 308, an acquirer system 310, and a communication network 312. In some non-limiting embodiments or aspects, each of the knowledge graph quantization system 102, the GPU 104, and / or the memory 106 may be implemented by the transaction service provider system 302 (e.g., a portion thereof). In some non-limiting embodiments or aspects, at least one of each of the knowledge graph quantization system 102, the GPU 104, and / or the memory 106 may be implemented by another system, another device, another system group, or another device group (e.g., a portion thereof) such as the issuer system 304, the merchant system 308, the acquirer system 310, and / or the like that is separate from the transaction service provider system 302 or includes the transaction service provider system 302.

[0127] The transaction service provider system 302 may include one or more devices capable of receiving information from an issuer system 304, a client device 306, a merchant system 308, and / or an acquirer system 310 via a communication network 312 and / or transmitting information to the issuer system, the client device, the merchant system, and / or the acquirer system. For example, the transaction service provider system 302 may include a computing device, such as a server (e.g., a transaction processing server, etc.), a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the transaction service provider system 302 may be associated with a transaction service provider described herein. In some non-limiting embodiments or aspects, the transaction service provider system 302 may communicate with a data storage device, which may be local or remote to the transaction service provider system 302. In some non-limiting embodiments or aspects, the transaction service provider system 302 is capable of receiving information from the data storage device, storing information in the data storage device, transmitting information to the data storage device, or searching for information stored in the data storage device.

[0128] The issuer system 304 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 302, the client device 306, the merchant system 308, and / or the acquirer system 310 via a communication network 312. For example, the issuer system 304 may include a computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the issuer system 304 may be associated with an issuer organization described herein. For example, the issuer system 304 may be associated with an issuer organization that issues a credit account, a debit account, a credit card, a debit card, and / or the like to a user associated with the client device 306.

[0129] The client device 306 may include one or more devices capable of receiving information from and / or transmitting information to a transaction service provider system 302, an issuer system 304, a merchant system 308, and / or an acquirer system 310 via a communication network 312. Additionally or alternatively, each client device 306 may include a device capable of receiving information from and / or transmitting information to other client devices 306 via a communication network 312, another network (e.g., a temporary network, a local network, a private network, a virtual private network, and / or the like), and / or any other suitable communication technology. For example, a client device 306 may include a client device and / or the like. In some non-limiting embodiments or aspects, a client device 306 may or may not be capable of communicating via a short-range wireless communication connection (e.g., an NFC communication connection, an RFID communication connection, Communication connection, communication connection and / or the like) (e.g., from merchant system 308 or from another client device 306), and / or transmit information via a short-range wireless communication connection (e.g., to merchant system 308).

[0130] The merchant system 308 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 302, the issuer system 304, the customer device 306, and / or the acquirer system 310 via the communication network 312. The merchant system 308 may also include a communication connection (e.g., an NFC communication connection, an RFID communication connection, a NFC communication connection, a ...RFID communication connection, a NFC communication connection, a RFID communication connection, a RFID communication connection, a NFC communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, Communication connection, In some non-limiting embodiments or aspects, the merchant system 308 may include a computing device, such as a server, a server group, a client device, a client device group, and / or other similar devices. In some non-limiting embodiments or aspects, the merchant system 308 may be associated with the merchant described herein. In some non-limiting embodiments or aspects, the merchant system 308 may include one or more client devices. For example, the merchant system 308 may include a client device that allows the merchant to transmit information to the transaction service provider system 302. In some non-limiting embodiments or aspects, the merchant system 308 may include one or more devices that can be used by merchants to conduct transactions with users, such as computers, computer systems, and / or peripheral devices. For example, the merchant system 308 may include a POS device and / or a POS system.

[0131] The acquirer system 310 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 302, the issuer system 304, the client device 306, and / or the merchant system 308 via the communication network 312. For example, the acquirer system 310 may include a computing device, a server, a server group, and / or the like. In some non-limiting embodiments or aspects, the acquirer system 310 may be associated with an acquirer as described herein.

[0132] The communication network 312 may include one or more wired and / or wireless networks. For example, the communication network 312 may include a cellular network (e.g., Long Term Evolution). networks, third generation (3G) networks, fourth generation (4G) networks, fifth generation (5G) networks, code division multiple access (CDMA) networks, and / or the like), public land mobile networks (PLMNs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), telephone networks (e.g., public switched telephone networks (PSTNs)), private networks (e.g., private networks associated with transaction service providers), ad hoc networks, intranets, the Internet, fiber-optic-based networks, cloud computing networks, and / or the like, and / or combinations of these or other types of networks.

[0133] Provided as an example Figure 3 The number and arrangement of systems, devices, and / or networks shown are not intended to be construed as limiting the number and arrangement of systems, devices, and / or networks shown. Additional systems, devices, and / or networks may exist; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or networks in a manner similar to that shown. Figure 3The systems, devices and / or networks shown in the figure may be arranged in different ways. In addition, the system or device may be implemented in a single system or device. Figure 3 Two or more systems or devices shown, or Figure 3 The single system or device shown may be implemented as multiple distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) and / or a set of devices (e.g., one or more devices) of environment 300 may perform one or more functions described as being performed by another set of systems or another set of devices of environment 300.

[0134] Reference now Figure 4 , a diagram showing example components of a device 400 according to a non-limiting embodiment or aspect. As an example, the device 400 may correspond to Figure 1 At least one of the knowledge graph quantization system 102, GPU 104 and / or memory 106 in Figure 3 In some non-limiting embodiments or aspects, Figure 1 or Figure 3 Such a system or device in may include at least one device 400 and / or at least one component of a device 400. Figure 4 The number and arrangement of the components shown in FIG. are provided as examples. In some non-limiting embodiments or aspects, Figure 4 Compared to those components shown in , device 400 may include additional components, fewer components, different components, or components arranged in a different manner. Additionally or alternatively, a group of components (e.g., one or more components) of device 400 may perform one or more functions described as being performed by another group of components of device 400.

[0135] like Figure 4As shown, the device 400 may include a bus 402, a processor 404, a memory 406, a storage component 408, an input component 410, an output component 412, and a communication interface 414. The bus 402 may include components that permit communication between components of the device 400. In some non-limiting embodiments or aspects, the processor 404 may be implemented in hardware, firmware, or a combination of hardware and software. For example, the processor 404 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.). In some non-limiting embodiments or aspects, the processor 404 may be the same or similar to the GPU 104. The memory 406 may include a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by the processor 404. In some non-limiting embodiments or aspects, memory 406 can be the same as or similar to memory 106 .

[0136] Continue to refer Figure 4 , the storage component 408 may store information and / or software related to the operation and use of the device 400. For example, the storage component 408 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a solid-state disk, etc.) and / or another type of computer-readable medium. The input component 410 may include a component that allows the device 400 to receive information, for example, through user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). In addition or alternatively, the input component 410 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). The output component 412 may include a component that provides output information from the device 400 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). The communication interface 414 may include a transceiver-like component (e.g., a transceiver, a separate receiver and a transmitter, etc.) that enables the device 400 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 414 may allow the device 400 to receive information from another device and / or provide information to another device. For example, the communication interface 414 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, interface, cellular network interface, and / or the like.

[0137] Device 400 can perform one or more processes described herein. Device 400 can perform these processes based on processor 404 executing software instructions stored by computer-readable media such as memory 406 and / or storage component 408. Computer-readable media can include any non-temporary memory device. Memory device includes memory space located inside a single physical storage device or memory space spread over multiple physical storage devices. Software instructions can be read from another computer-readable medium or from another device to memory 406 and / or storage component 408 through communication interface 414. When executed, software instructions stored in memory 406 and / or storage component 408 can cause processor 404 to perform one or more processes described herein. In addition or alternatively, hard-wired circuits can be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any specific combination of hardware circuits and software. The term "programmed or configured" used herein refers to the arrangement of software, hardware circuit systems, or any combination thereof on one or more devices.

[0138] Reference now Figure 5 , which is an example diagram of a KGNN 500 according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, the KGNN 500 can be implemented by the knowledge graph quantization system 102.

[0139] In some non-limiting embodiments or aspects, KGNN 500 can capture high-order structural proximity between entities in the knowledge graph. In some non-limiting embodiments or aspects, KGNN 500 can have many layers L.

[0140] In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 may receive a training dataset comprising a first knowledge graph embedding set {E (0) In some non-limiting embodiments or aspects, the plurality of entities may include a plurality of users and / or a plurality of projects.

[0141] In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 may input the training data set into the KGNN 500. In response to receiving the training data set, the KGNN 500 may generate at least one additional knowledge graph embedding set {E t t} associated with a plurality of entities for at least one additional layer of the knowledge graph. (1) …E (L)}.

[0142] In some non-limiting embodiments or aspects, the at least one additional layer may include one or more layers. For example, the at least one additional layer may include a second layer and / or a third layer. In some non-limiting embodiments or aspects, the at least one additional knowledge graph embedding set {E (1) …E (L)} may include a second knowledge graph embedding set associated with the second layer and / or a third knowledge graph embedding set associated with the third layer.

[0143] In some non-limiting embodiments or aspects, the knowledge graph quantization system 102 may input the training data set into the KGNN 500 to generate a second knowledge graph embedding set associated with a plurality of entities for a second layer of the knowledge graph. Additionally or alternatively, the knowledge graph quantization system 102 may input the second knowledge graph embedding set into the knowledge graph neural network to generate a third knowledge graph embedding set associated with a plurality of entities for a third layer of the knowledge graph.

[0144] In some non-limiting embodiments or aspects, the KGNN 500 may embed at least one additional knowledge graph set {E (1) …E (L)} quantize to provide at least one quantized knowledge graph embedding set For example, at least one additional graph embedding set {E (1) …E (L)} can be quantized into low-order activation values ​​to propagate at least one additional graph embedding set {E (1) …E (L)} (e.g., an exact set of graph embeddings) to reduce memory footprint.

[0145] In some non-limiting embodiments or aspects, quantizing at least one additional set of knowledge graph embeddings may be based on the following equation:

[0146]

[0147] in is the corresponding knowledge graph embedding in at least one other knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, where b is the number of bits for the integer, l is the corresponding layer of at least one additional layer, and is the unbiased random rounding operator.

[0148] In some non-limiting embodiments or aspects, random rounding for any scalar x may be formulated as:

[0149]

[0150] in is the ceil operator and is the floor operator.

[0151] In some non-limiting embodiments or aspects, at least one quantized knowledge graph embedding set Each corresponding quantized knowledge graph embedding in may include an integer representation (e.g., a 2-bit integer, an 8-bit integer). In some non-limiting embodiments or aspects, at least one additional knowledge graph embedding set {E (1) …E (L) Each corresponding additional knowledge graph embedding in} may include a floating point representation (e.g., 32-bit floating point).

[0152] In some non-limiting embodiments or aspects, KGNN 500 may embed at least one quantized knowledge graph into a set In some non-limiting embodiments or aspects, when storing at least one quantized knowledge graph embedding set When , KGNN 500 can overwrite at least one other knowledge graph embedding set in memory with at least one quantized knowledge graph embedding set.

[0153] In some non-limiting embodiments or aspects, KGNN 500 may perform at least one quantized knowledge graph embedding set Dequantization is performed to provide at least one dequantized knowledge graph embedding set In some non-limiting embodiments or aspects, at least one dequantized knowledge graph embedding set can be approximated by at least one other knowledge graph embedding set {E (1) …E (L)}.

[0154] In some non-limiting embodiments or aspects, dequantizing at least one set of quantized knowledge graph embeddings may be based on the following equation:

[0155]

[0156] in is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the corresponding quantized knowledge graph embedding scaled to [0, B] in at least one quantized knowledge graph embedding set, It is for range, is the offset, B = 2 b -1, wherein b is the number of bits for the integer, and l is the corresponding layer of at least one additional layer. In some non-limiting embodiments or aspects, Can be a full precision tensor used to compute gradients for back propagation. For example, all operators (e.g., spmm(·)) can be performed with full precision. In some non-limiting embodiments or aspects, dequantization is necessary because most GPUs do not support low-bit operators.

[0157] In some non-limiting embodiments or aspects, the KGNN 500 may have an expected value based on the following equation:

[0158]

[0159] in is the expected measure, is the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, l is the corresponding layer of at least one additional layer, and is the corresponding knowledge graph embedding in at least one other knowledge graph embedding set.

[0160] In some non-limiting embodiments or aspects, the KGNN 500 may have a variance value based on the following equation:

[0161]

[0162] where d is the dimension of the corresponding knowledge graph embedding in at least one dequantized knowledge graph embedding set, is the offset, It is for The range of B=2 b -1, where b is the number of bits for the integer (e.g., 2 bits, 8 bits, 32 bits).

[0163] In some non-limiting embodiments or aspects, at least one dequantized knowledge graph embedding Can be used to calculate the gradient for back propagation (e.g., KG reverse). In some non-limiting embodiments or aspects, KGNN 500 can be based on at least one dequantized knowledge graph embedding set To determine (e.g., calculate) one or more gradients (e.g., ). Additionally or alternatively, KGNN 500 may be based on the calculated gradient values ​​(e.g., )Update the parameters of the knowledge graph neural network.

[0164] In some non-limiting embodiments or aspects, the KGNN 500 may retrieve at least one quantized knowledge graph embedding set from memory. to determine the gradient.

[0165] In some non-limiting embodiments or aspects, KGNN 500 may determine a prediction. For example, KGNN 500 may determine a prediction indicating whether a user among a plurality of users will be interested in an item that the user has not yet engaged in. For example, KGNN 500 may output a value (e.g., a prediction value) indicating a probability that a user will be interested in an item that the user has not yet engaged in.

[0166] In some non-limiting embodiments or aspects, KGNN 500 may use a message passing pattern, where the representation of an entity is v is updated iteratively in each layer by collecting messages from its neighbors in the knowledge graph. In some non-limiting embodiments or aspects, the lth layer can be simplified as:

[0167] E (l+1) =KG-Layer(A R ,E (l) ,θ (l) ),l=0,1,…,L-1

[0168] Where E (l) ∈R (Nxd) represents the d-dimensional embedding of the entity, N is the number of entities, and A R represents the relationship matrix containing multi-type relationships between entities, θ (l) represents the trainable parameters in the lth layer, and KG-Layer(·) is a propagation layer which can be a graph convolutional layer or a graph attention layer.

[0169] In some non-limiting embodiments or aspects, the polymerization process may be based on:

[0170]

[0171] in is A with self-connection R The normalized adjacency matrix of .

[0172] In some non-limiting embodiments or aspects, the computation graph may be decomposed as follows:

[0173] Forward:

[0174] Backward:

[0175] where spmm(·) is sparse-dense matrix multiplication, mm(·) is dense-dense matrix multiplication, σ(·) is a nonlinear function, represents the gradient of the activation / parameter always taken with respect to the loss L, and ctx(·) represents the context information that needs to be stored in the GPU memory for the backward pass, i.e. In some non-limiting embodiments or aspects, a reverse pass (eg, KG reverse) may require more messages than a forward pass (eg, KG forward).

[0176] In some non-limiting embodiments or aspects, in the inference phase, only a forward pass of the entire network may be performed, where the intermediate layers (e.g., H (l) ) can be subsequently discarded. However, during the training phase, an automatic differentiation engine (e.g., PyTorch) can store the following forward pass variables in memory for backpropagation: Normalized adjacency matrix Parameter θ (l) ∈R (dxd) and / or all intermediate results H (l) ∈R (Nxd) ,J (l) ∈R (Nxd) ,E (l) ∈R (Nxd) (For example, if layer polymerization is employed).

[0177] Reference now Fig. 6A and Figure 6B , respectively, are shown as example diagrams of a user-item rating matrix 600 and an example diagram of a user-item bipartite graph 610. In some non-limiting embodiments or aspects, the user-item rating matrix 600 and / or the user-item bipartite graph 610 may include one or more user nodes 602 (e.g., u1, u2, and / or u3) and / or one or more item nodes 604 (e.g., i1, i2, i3, i4, and / or i5). Each user node 602 may represent a different user, and each item node 604 may represent a different item.

[0178] In some non-limiting embodiments or aspects, a user-item rating matrix 600 may be defined based on implicit feedback from users, where a value on the matrix (e.g., 1) indicates that user u has participated in item i. Fig. 6A As seen in , users u1 and u2 have participated in project i1 as indicated by the value 1, and user u3 has not participated as indicated by the "?" on the matrix. Users can participate in a project in any way including, but not limited to, clicking, viewing, purchasing, and / or using the project.

[0179] like Figure 6BAs shown, as indicated by the user-item rating matrix 600, when the user represented by the user node 602 has participated in the project represented by the project node 604, there may be an edge 606 between the user node and the project node. In some non-limiting embodiments or aspects, the edge 606 may represent a relationship between two nodes connected by the edge. For example, the edge 606 may be formed between the user node u1 and the project node i1, where the user represented by the user node u1 has participated in the project represented by the project node i4, as indicated by the user-item rating matrix 600 by the value 1.

[0180] In some non-limiting embodiments or aspects, given a user-item interaction matrix Y and a knowledge graph G, the probability that user u will participate in item v can be predicted based on the following prediction function F Where θ represents the parameter of the function:

[0181]

[0182] Reference now Figure 7 , which shows an example diagram of the knowledge graph 700. Figure 7 As shown, the knowledge graph 700 may include multiple nodes 702 and / or multiple edges 704. In some non-limiting embodiments or aspects, the multiple nodes 702 may include user nodes (e.g., u1-u5), item nodes (e.g., i1-i4), and / or entity nodes (e.g., e1-i5). 1、 e2, e3). User nodes may each represent different users, project nodes may each represent different projects, and entity nodes may each represent different entities. In some non-limiting embodiments or aspects, multiple nodes 702 may be the same as user nodes 602 and / or project nodes 604, similar to them, and / or may be parts of them. In some non-limiting embodiments or aspects, multiple edges 704 may be the same as edge 606, similar to it, and / or may be parts of it.

[0183] In some non-limiting embodiments or aspects, an edge in the plurality of edges 702 may represent a relationship between two nodes in the plurality of nodes 702 that connect the edge. In some non-limiting embodiments or aspects, each edge in the plurality of edges 704 may represent a different relationship (e.g., r1, r2, r3, r4). In some non-limiting embodiments or aspects, an edge 606 (e.g., a connection) may connect a user node 602 and an item node 604 based on a relationship between a user represented by a user node u1 and an item represented by an item node i4.

[0184] In some non-limiting embodiments or aspects, the knowledge graph 700 may be used to predict whether a user is potentially interested in a project that the user has not previously engaged with.

[0185] Reference now Figure 8 , shown is an example of a user-item entity graph 800. The user-item entity graph 800 may include a plurality of nodes 802 and / or a plurality of edges 804. In some non-limiting embodiments or aspects, the plurality of nodes 802 may include a head node 806. In some non-limiting embodiments or aspects, the user-item entity graph 800 may be generated based on entity sets and / or relationship sets in the knowledge graph 700. In some non-limiting embodiments or aspects, the entity set may consist of items and / or non-items.

[0186] In some non-limiting embodiments or aspects, an edge can connect two nodes based on a relationship. For example, an edge between two nodes can indicate that an item corresponds to an entity.

[0187] In some non-limiting embodiments or aspects, user-item entity graph 800 may include a head, a relationship, and a tail of a knowledge triple. In some non-limiting embodiments or aspects, a knowledge triple may indicate a fact. For example, an edge connecting a node representing Forrest Gump and a node representing Tom Hanks may indicate that Tom Hanks is an actor in the movie Forrest Gump.

[0188] Although the embodiments have been described in detail for the purpose of illustration, it should be understood that such details are used only for that purpose and that the present disclosure is not limited to the disclosed embodiments or aspects, but on the contrary, is intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it should be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect.

Claims

1. A computer-implemented method, the method comprising: receiving, with at least one processor, a training dataset comprising a first set of knowledge graph embeddings associated with a plurality of entities for a first layer of a knowledge graph; inputting, with at least one processor, the training data set into a knowledge graph neural network to generate at least one additional set of knowledge graph embeddings associated with the plurality of entities for at least one additional layer of the knowledge graph; quantizing, with at least one processor, the at least one additional set of knowledge graph embeddings to provide at least one quantized set of knowledge graph embeddings; storing, with at least one processor, the at least one quantized knowledge graph embedding set in a memory; as well as Dequantizing the at least one quantized knowledge graph embedding set using at least one processor to provide at least one dequantized knowledge graph embedding set, wherein the at least one dequantized knowledge graph embedding set approximates the at least one additional knowledge graph embedding set, wherein gradients for backpropagation are determined based on the at least one dequantized knowledge graph embedding set.

2. The computer-implemented method of claim 1, wherein the plurality of entities comprises a plurality of users and a plurality of projects, the method further comprising: Determining, with at least one processor, a prediction based on the knowledge graph neural network indicating whether a user among the plurality of users would be interested in an item in which the user has not yet engaged.

3. The computer-implemented method of claim 1 , wherein the at least one additional layer comprises a second layer and a third layer, wherein the at least one additional knowledge graph embedding set comprises a second knowledge graph embedding set associated with the second layer and a third knowledge graph embedding set associated with the third layer, and The input of the training data set includes: inputting the training data set into the knowledge graph neural network using at least one processor to generate the second knowledge graph embedding set associated with the plurality of entities for the second layer of the knowledge graph; as well as The second knowledge graph embedding set is input into the knowledge graph neural network using at least one processor to generate the third knowledge graph embedding set associated with the plurality of entities for the third layer of the knowledge graph.

4. The computer-implemented method of claim 1 , further comprising: determining, with at least one processor, a gradient for back-propagation based on the at least one set of dequantized knowledge graph embeddings; as well as Using at least one processor, update parameters of the knowledge graph neural network based on the gradient.

5. The computer-implemented method of claim 1 , wherein each respective additional knowledge graph embedding set of the at least one additional knowledge graph embedding set comprises a respective activation map of the knowledge graph neural network for a respective layer of the at least one additional layer.

6. The computer-implemented method of claim 1, wherein each respective quantized knowledge graph embedding in the at least one set of quantized knowledge graph embeddings comprises an integer representation, and each respective additional knowledge graph embedding in the at least one additional set of knowledge graph embeddings comprises a floating point representation.

7. The computer-implemented method of claim 1 , wherein storing the at least one quantized knowledge graph embedding set in the memory comprises: The at least one additional knowledge graph embedding set is overwritten with the at least one quantized knowledge graph embedding set in a memory of a graphics processing unit (GPU).

8. A system, comprising: at least one processor; as well as at least one non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to: receiving a training dataset comprising a first set of knowledge graph embeddings associated with a plurality of entities for a first layer of a knowledge graph; inputting the training dataset into a knowledge graph neural network to generate at least one additional set of knowledge graph embeddings associated with the plurality of entities for at least one additional layer of the knowledge graph; quantizing the at least one additional set of knowledge graph embeddings to provide at least one quantized set of knowledge graph embeddings; storing the at least one quantized knowledge graph embedding set in a memory; as well as Dequantizing the at least one quantized knowledge graph embedding set to provide at least one dequantized knowledge graph embedding set, wherein the at least one dequantized knowledge graph embedding set approximates the at least one additional knowledge graph embedding set, wherein gradients for backpropagation are determined based on the at least one dequantized knowledge graph embedding set.

9. The system of claim 8, wherein the plurality of entities comprises a plurality of users and a plurality of projects, and wherein the instructions further cause the at least one processor to: A prediction is determined based on the knowledge graph neural network indicating whether a user among the plurality of users would be interested in an item with which the user has not yet engaged.

10. The system of claim 8, wherein the at least one additional layer comprises a second layer and a third layer, wherein the at least one additional knowledge graph embedding set comprises a second knowledge graph embedding set associated with the second layer and a third knowledge graph embedding set associated with the third layer, and in, When the training data set is input, the instructions further cause the at least one processor to: Inputting the training data set into the knowledge graph neural network to generate the second knowledge graph embedding set associated with the plurality of entities for the second layer of the knowledge graph; as well as The second knowledge graph embedding set is input into the knowledge graph neural network to generate the third knowledge graph embedding set associated with the plurality of entities for the third layer of the knowledge graph.

11. The system of claim 8, wherein the instructions further cause the at least one processor to: Determining a gradient for back-propagation based on the at least one set of dequantized knowledge graph embeddings; and Parameters of the knowledge graph neural network are updated based on the gradient.

12. The system of claim 8, wherein each respective additional knowledge graph embedding set of the at least one additional knowledge graph embedding set comprises a respective activation map of the knowledge graph neural network for a respective layer of the at least one additional layer.

13. The system of claim 8, wherein each respective quantized knowledge graph embedding in the at least one set of quantized knowledge graph embeddings comprises an integer representation, and each respective additional knowledge graph embedding in the at least one additional set of knowledge graph embeddings comprises a floating point representation.

14. The system of claim 8, wherein: When storing the at least one quantized knowledge graph embedding set in the memory, the instructions further cause the at least one processor to: The at least one additional knowledge graph embedding set is overwritten with the at least one quantized knowledge graph embedding set in a memory of a graphics processing unit (GPU).

15. A computer program product comprising at least one non-transitory computer readable medium, the at least one non-transitory computer readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: receiving a training dataset comprising a first set of knowledge graph embeddings associated with a plurality of entities for a first layer of a knowledge graph; inputting the training dataset into a knowledge graph neural network to generate at least one additional set of knowledge graph embeddings associated with the plurality of entities for at least one additional layer of the knowledge graph; quantizing the at least one additional set of knowledge graph embeddings to provide at least one quantized set of knowledge graph embeddings; storing the at least one quantized knowledge graph embedding set in a memory; as well as Dequantizing the at least one quantized knowledge graph embedding set to provide at least one dequantized knowledge graph embedding set, wherein the at least one dequantized knowledge graph embedding set approximates the at least one additional knowledge graph embedding set, wherein gradients for backpropagation are determined based on the at least one dequantized knowledge graph embedding set.

16. The computer program product of claim 15, wherein the plurality of entities comprises a plurality of users and a plurality of projects, and wherein the program instructions further cause the at least one processor to: A prediction is determined based on the knowledge graph neural network indicating whether a user among the plurality of users would be interested in an item with which the user has not yet engaged.

17. The computer program product of claim 15, wherein the at least one additional layer comprises a second layer and a third layer, wherein the at least one additional knowledge graph embedding set comprises a second knowledge graph embedding set associated with the second layer and a third knowledge graph embedding set associated with the third layer, and in, When the training data set is input, the program instructions further cause the at least one processor to: Inputting the training data set into the knowledge graph neural network to generate the second knowledge graph embedding set associated with the plurality of entities for the second layer of the knowledge graph; as well as The second knowledge graph embedding set is input into the knowledge graph neural network to generate the third knowledge graph embedding set associated with the plurality of entities for the third layer of the knowledge graph.

18. The computer program product of claim 15, wherein the program instructions further cause the at least one processor to: Determining a gradient for back-propagation based on the at least one set of dequantized knowledge graph embeddings; and Parameters of the knowledge graph neural network are updated based on the gradient.

19. The computer program product of claim 15, wherein each respective additional knowledge graph embedding set of the at least one additional knowledge graph embedding set comprises a respective activation map of the knowledge graph neural network for a respective layer of the at least one additional layer.

20. The computer program product of claim 15, wherein: When storing the at least one quantized knowledge graph embedding set in the memory, the program instructions further cause the at least one processor to: The at least one additional knowledge graph embedding set is overwritten with the at least one quantized knowledge graph embedding set in a memory of a graphics processing unit (GPU).

Images