Method, system and computer program product for embedding compression and regularization

Abstract

A method, system and computer program product for embedding compression and reconstruction are provided. The method includes receiving embedding vector data including a plurality of embedding vectors. A beta variational autoencoder is trained based on the embedding vector data and a loss equation. The method includes determining a corresponding entropy and a corresponding variance of a corresponding mean value for each corresponding dimension in a plurality of dimensions. A first subset of the plurality of dimensions is determined based on the corresponding entropy and the corresponding variance of the corresponding mean value for each corresponding dimension in the plurality of dimensions. A second subset of the plurality of dimensions is discarded based on the corresponding entropy and the corresponding variance of the corresponding mean value for each corresponding dimension in the plurality of dimensions. The method includes generating a compressed representation of the embedding vector data based on the first subset of dimensions.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 270,321, filed on October 21, 2021, the disclosure of which is incorporated herein by reference in its entirety. Background Art 1. Technical Field

[0004] The present disclosure relates generally to embedding compression and regularization and, in some non-limiting embodiments or aspects, to systems, methods, and computer program products for embedding compression and regularization using autoencoders.

[0005] 2. Technical considerations

[0006] Autoencoder architectures (e.g., encoders and / or decoders) are often used to transfer word embeddings into a latent space. A compressed representation of the embedding vector data may be desirable for certain tasks, such as converting between languages, comparing merchant embeddings from different time periods or different regions, etc.

[0007] However, with some techniques, the latent space is often of higher dimensionality than the input embeddings, and little is known about how individual dimensions of the latent space control the semantics encoded in the embeddings. In other techniques, the latent space has fewer dimensions than the original embeddings, but interpretability of the dimensions is difficult, as is determining which dimensions are useful. In either case, the resulting latent space may be more like a chaotic space. Therefore, techniques for embedding compression and regularization are needed to compress the latent space and focus the semantics on fewer latent dimensions. Summary of the invention

[0008] It is therefore an object of the present disclosure to provide a system, method, and computer program product for embedding compression and regularization that overcomes some or all of the deficiencies identified above.

[0009] Clause 1: A computer-implemented method for generating a compressed representation of embedded vector data, comprising: receiving, with at least one processor, the embedded vector data comprising a plurality of embedded vectors; training, with at least one processor, a beta variational autoencoder based on the embedded vector data and a loss equation, the beta variational autoencoder comprising an encoder network, a latent layer, and a decoder network, the loss equation comprising a first term associated with a reconstruction of an input by the beta variational autoencoder, a second term associated with a regularization of a latent space of the latent layer, and a hyperparameter weight associated with a ratio of the first term to the second term, wherein the latent space has a plurality of dimensions; determining, with at least one processor, a respective mean for each respective dimension of the plurality of dimensions; determining, with at least one processor, a respective variance for each respective dimension of the plurality of dimensions; determining, with at least one processor, a first subset of the plurality of dimensions based on the respective mean and the respective variance for each respective dimension of the plurality of dimensions; discarding, with at least one processor, a second subset of the plurality of dimensions based on the respective mean and the respective variance for each respective dimension of the plurality of dimensions; and generating, with at least one processor, the compressed representation of the embedded vector data based on the first subset of dimensions.

[0010] Clause 2: A computer-implemented method according to clause 1, wherein determining the corresponding mean value of each corresponding dimension in the multiple dimensions includes: determining the corresponding entropy of the corresponding mean value of each corresponding dimension in the multiple dimensions; and wherein training the beta variational autoencoder includes: iteratively adjusting the hyperparameter weights and repeating the training, determining the corresponding entropy of the corresponding mean value, determining the corresponding variance, determining the first subset, discarding the second subset, and generating the compressed representation.

[0011] Clause 3: A computer-implemented method according to Clause 1 or Clause 2, wherein determining the corresponding average value of each corresponding dimension in the multiple dimensions comprises: determining the corresponding entropy of the corresponding average value of each corresponding dimension in the multiple dimensions; and wherein the method further comprises: determining that each corresponding dimension in the multiple dimensions includes a useful dimension or a discarded dimension based on the corresponding average value and the corresponding variance of the corresponding dimension; wherein the corresponding entropy of the corresponding average value of each useful dimension is higher than a first threshold and the corresponding variance of each useful dimension is lower than a second threshold, and wherein the corresponding entropy of the corresponding average value of each discarded dimension is lower than the first threshold and the corresponding variance of each discarded dimension is higher than the second threshold.

[0012] Clause 4: The computer-implemented method of any of clauses 1 to 3, wherein the first subset of the plurality of dimensions includes each useful dimension, and wherein the second subset of the plurality of dimensions includes each deprecated dimension.

[0013] Clause 5: A computer-implemented method according to any one of clauses 1 to 4, wherein determining the corresponding average value of each corresponding dimension in the multiple dimensions comprises: determining the corresponding entropy of the corresponding average value of each corresponding dimension in the multiple dimensions using at least one processor; and wherein determining the first subset of the multiple dimensions comprises: determining the first subset of the multiple dimensions based on the corresponding entropy and the corresponding variance of the corresponding average value of each corresponding dimension in the multiple dimensions.

[0014] Clause 6: A computer-implemented method according to any one of clauses 1 to 5, wherein discarding the second subset of the multiple dimensions comprises: discarding the second subset of the multiple dimensions based on the corresponding entropy and the corresponding variance of the corresponding mean value of each corresponding dimension in the multiple dimensions.

[0015] Clause 7: The computer-implemented method of any one of clauses 1 to 6, wherein the loss equation is:

[0016]

[0017] Wherein the first term associated with the input reconstruction performed by the beta variational autoencoder is:

[0018]

[0019] wherein the second term associated with the regularization of the latent space of the latent layer is:

[0020] and

[0021] Wherein the hyperparameter weight associated with the ratio of the first term to the second term is β.

[0022] Item 8: A system for generating a compressed representation of embedded vector data, comprising at least one processor, the at least one processor being programmed or configured to: receive the embedded vector data comprising a plurality of embedded vectors; train a beta variational autoencoder based on the embedded vector data and a loss equation, the beta variational autoencoder comprising an encoder network, a latent layer, and a decoder network, the loss equation comprising a first term associated with input reconstruction by the beta variational autoencoder, a second term associated with regularization of a latent space of the latent layer, and a hyperparameter weight associated with a ratio of the first term to the second term, wherein the latent space has multiple dimensions; determine a corresponding mean for each corresponding dimension in the multiple dimensions; determine a corresponding variance for each corresponding dimension in the multiple dimensions; determine a first subset of the multiple dimensions based on the corresponding mean and the corresponding variance for each corresponding dimension in the multiple dimensions; discard a second subset of the multiple dimensions based on the corresponding mean and the corresponding variance for each corresponding dimension in the multiple dimensions; and generate the compressed representation of the embedded vector data based on the first subset of dimensions.

[0023] Clause 9: A system according to clause 8, wherein, when determining the corresponding mean value of each corresponding dimension in the multiple dimensions, the at least one processor is programmed or configured to: determine the corresponding entropy of the corresponding mean value of each corresponding dimension in the multiple dimensions; and wherein, when training the beta variational autoencoder, the at least one processor is programmed or configured to: iteratively adjust the hyperparameter weights and repeat the training, the determining the corresponding entropy of the corresponding mean value, the determining the corresponding variance, the determining the first subset, the discarding the second subset, and the generating the compressed representation.

[0024] Clause 10: A system according to clause 8 or clause 9, wherein, when determining the corresponding average value of each corresponding dimension in the multiple dimensions, the at least one processor is programmed or configured to: determine the corresponding entropy of the corresponding average value of each corresponding dimension in the multiple dimensions; and wherein the at least one processor is further programmed or configured to: determine that each corresponding dimension in the multiple dimensions includes a useful dimension or a discarded dimension based on the corresponding average value and the corresponding variance of the corresponding dimension; wherein the corresponding entropy of the corresponding average value of each useful dimension is higher than a first threshold and the corresponding variance of each useful dimension is lower than a second threshold, and wherein the corresponding entropy of the corresponding average value of each discarded dimension is lower than the first threshold and the corresponding variance of each discarded dimension is higher than the second threshold.

[0025] Clause 11: The system of any of clauses 8 to 10, wherein the first subset of the plurality of dimensions includes each useful dimension, and wherein the second subset of the plurality of dimensions includes each deprecated dimension.

[0026] Clause 12: A system according to any one of clauses 8 to 11, wherein, when determining the corresponding average value of each corresponding dimension in the multiple dimensions, the at least one processor is programmed or configured to: determine the corresponding entropy of the corresponding average value of each corresponding dimension in the multiple dimensions; and wherein, when determining the first subset of the multiple dimensions, the at least one processor is programmed or configured to: determine the first subset of the multiple dimensions based on the corresponding entropy and the corresponding variance of the corresponding average value of each corresponding dimension in the multiple dimensions.

[0027] Clause 13: A system according to any one of clauses 8 to 12, wherein, when discarding the second subset of the multiple dimensions, the at least one processor is programmed or configured to: discard the second subset of the multiple dimensions based on the corresponding entropy and the corresponding variance of the corresponding mean value of each corresponding dimension in the multiple dimensions.

[0028] Clause 14: The system of any one of clauses 8 to 13, wherein the loss equation is:

[0029]

[0030] Wherein the first term associated with the input reconstruction performed by the beta variational autoencoder is:

[0031]

[0032] wherein the second term associated with the regularization of the latent space of the latent layer is:

[0033] and

[0034] Wherein the hyperparameter weight associated with the ratio of the first term to the second term is β.

[0035] Clause 15: A computer program product for generating a compressed representation of embedded vector data, the computer program product comprising at least one non-transitory computer-readable medium, the non-transitory computer-readable medium comprising one or more instructions that, when executed by at least one processor, cause the at least one processor to: receive the embedded vector data comprising a plurality of embedded vectors; train a beta variational autoencoder based on the embedded vector data and a loss equation, the beta variational autoencoder comprising an encoder network, a latent layer, and a decoder network, the loss equation comprising a first term associated with a reconstruction of an input by the beta variational autoencoder, a second term associated with a regularization of a latent space of the latent layer, and a hyperparameter weight associated with a ratio of the first term to the second term, wherein the latent space has a plurality of dimensions; determine a respective mean for each respective dimension in the plurality of dimensions; determine a respective variance for each respective dimension in the plurality of dimensions; determine a first subset of the plurality of dimensions based on the respective mean and the respective variance for each respective dimension in the plurality of dimensions; discard a second subset of the plurality of dimensions based on the respective mean and the respective variance for each respective dimension in the plurality of dimensions; and generate the compressed representation of the embedded vector data based on the first subset of dimensions.

[0036] Clause 16: A computer program product according to clause 15, wherein, when determining the corresponding mean value of each corresponding dimension in the multiple dimensions, the one or more instructions cause the at least one processor to: determine the corresponding entropy of the corresponding mean value of each corresponding dimension in the multiple dimensions; and wherein, when training the beta variational autoencoder, the one or more instructions further cause the at least one processor to: iteratively adjust the hyperparameter weights and repeat the training, the determining the corresponding entropy of the corresponding mean value, the determining the corresponding variance, the determining the first subset, the discarding the second subset, and the generating the compressed representation.

[0037] Clause 17: A computer program product according to clause 15 or clause 16, wherein, when determining the corresponding average value of each corresponding dimension in the multiple dimensions, the one or more instructions cause the at least one processor to: determine the corresponding entropy of the corresponding average value of each corresponding dimension in the multiple dimensions; and wherein the one or more instructions further cause the at least one processor to: determine that each corresponding dimension in the multiple dimensions includes a useful dimension or a discarded dimension based on the corresponding average value and the corresponding variance of the corresponding dimension; wherein the corresponding entropy of the corresponding average value of each useful dimension is higher than a first threshold and the corresponding variance of each useful dimension is lower than a second threshold, and wherein the corresponding entropy of the corresponding average value of each discarded dimension is lower than the first threshold and the corresponding variance of each discarded dimension is higher than the second threshold.

[0038] Clause 18: The computer program product of any of clauses 15 to 17, wherein the first subset of the plurality of dimensions includes each useful dimension, and wherein the second subset of the plurality of dimensions includes each deprecated dimension.

[0039] Clause 19: A computer program product according to any one of clauses 15 to 18, wherein, when determining the corresponding average value of each corresponding dimension in the multiple dimensions, the one or more instructions cause the at least one processor to: determine the corresponding entropy of the corresponding average value of each corresponding dimension in the multiple dimensions; and wherein, when determining the first subset of the multiple dimensions, the one or more instructions cause the at least one processor to: determine the first subset of the multiple dimensions based on the corresponding entropy and the corresponding variance of the corresponding average value of each corresponding dimension in the multiple dimensions.

[0040] Clause 20: A computer program product according to any one of clauses 15 to 19, wherein when discarding the second subset of the plurality of dimensions, the one or more instructions cause the at least one processor to: discard the second subset of the plurality of dimensions based on the respective entropy and the respective variance of the respective mean of each respective dimension in the plurality of dimensions, wherein the loss equation is:

[0041]

[0042] Wherein the first term associated with the input reconstruction performed by the beta variational autoencoder is:

[0043]

[0044] wherein the second term associated with the regularization of the latent space of the latent layer is:

[0045] and

[0046] Wherein the hyperparameter weight associated with the ratio of the first term to the second term is β.

[0047] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related structural elements and combinations of parts and the economies of manufacturing will become more apparent when considering the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for purposes of illustration and description only and are not intended as a definition of the limits of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Additional advantages and details are explained in more detail below with reference to non-limiting, exemplary embodiments or aspects shown in the accompanying drawings, in which:

[0049] Figure 1 is an illustration of a system for embedding compression and regularization according to a non-limiting embodiment or aspect;

[0050] Figure 2 is a flow chart of a method for embedding compression and regularization according to a non-limiting embodiment or aspect;

[0051] Figure 3 is an illustration of an example environment in which methods, systems, and / or computer program products described herein may be implemented according to non-limiting embodiments or aspects;

[0052] Figure 4 illustrates example components of an apparatus for use in conjunction with a non-limiting embodiment or aspect;

[0053] Figure 5 is a diagram of a machine learning model architecture according to a non-limiting embodiment or aspect;

[0054] Fig. 6A and 6B is an illustration of quantitative embedding results of a machine learning model architecture according to a non-limiting embodiment or aspect. DETAILED DESCRIPTION

[0055] For the purposes of the following description, the terms "end," "upper," "lower," "right," "left," "vertical," "horizontal," "top," "bottom," "lateral," "longitudinal," and their derivatives shall relate to the orientation of the embodiment or aspect in the accompanying drawings. However, it shall be understood that the embodiments or aspects may employ various alternative variations and step sequences, except where expressly specified to the contrary. It shall also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely exemplary embodiments or aspects of the present invention. Accordingly, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein shall not be considered limiting.

[0056] The aspects, components, elements, structures, actions, steps, functions, instructions, etc. used herein should not be understood as critical or necessary unless explicitly described as such. Also, as used herein, the article "one" is intended to include one or more items, and can be used interchangeably with "one or more" and "at least one". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, a combination of related items and unrelated items, etc.), and can be used interchangeably with "one or more" or "at least one". In the case of wishing only one item, the term "one" or similar language is used. Also, as used herein, the term "having" etc. is intended to be an open term. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "based at least in part on".

[0057] As used herein, the term "acquirer institution" may refer to an entity that is licensed and / or approved by a transaction service provider to initiate a transaction (e.g., a payment transaction) using a payment device associated with the transaction service provider. Transactions that may be initiated by an acquirer institution may include payment transactions, such as purchases, original credit transactions (OCT), account funding transactions (AFT), etc. In some non-limiting embodiments or aspects, the acquirer institution may be a financial institution, such as a bank. As used herein, the term "acquirer system" may refer to one or more computing devices operated by or on behalf of an acquirer institution, such as a server computer executing one or more software applications.

[0058] As used herein, the term "account identifier" may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term "token" may refer to an identifier used as a replacement or substitute identifier for an original account identifier such as a PAN. An account identifier may be alphanumeric or any combination of characters and / or symbols. A token may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, etc.) so that the token can be used to conduct transactions without directly using the original account identifier. In some examples, an original account identifier such as a PAN may be associated with multiple tokens for different individuals or purposes.

[0059] As used herein, the term "communication" may refer to the reception, acceptance, transmission, transmission, provision, etc. of data (e.g., information, signals, messages, instructions, commands, etc.). A unit (e.g., a device, a system, a component of a device or system, a combination thereof, etc.) communicating with another unit means that the unit is able to directly or indirectly receive information from the other unit and / or send information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, etc.) that is wired and / or wireless in nature. In addition, although the information sent can be modified, processed, relayed and / or routed between the first unit and the second unit, the two units may also communicate with each other. For example, even if the first unit passively receives information and does not actively send information to the second unit, the first unit may communicate with the second unit. As another example, if at least one intermediate unit processes the information received from the first unit and transmits the processed information to the second unit, the first unit may communicate with the second unit.

[0060] As used herein, the term "computing device" may refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, etc. The computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or a standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, etc.), a personal digital assistant (PDA), and / or other similar devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0061] As used herein, the terms "electronic wallet" and "electronic wallet application" refer to one or more electronic devices and / or software applications configured to initiate and / or conduct payment transactions. For example, an electronic wallet may include a mobile device executing an electronic wallet application, and may also include server-side software and / or databases for maintaining and providing transaction data to the mobile device. An "electronic wallet provider" may include an entity that provides and / or maintains electronic wallets for customers, such as Google. Android Apple Samsung And / or other similar electronic payment systems. In some non-limiting examples, the issuing bank may be an electronic wallet provider.

[0062] As used herein, the term "issuer institution" may refer to one or more entities, such as a bank, that provide customers with accounts for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide a customer with an account identifier, such as a primary account number (PAN), that uniquely identifies one or more accounts associated with the customer. The account identifier may be implemented on a portable financial device, such as a physical financial instrument (e.g., a payment card), and / or may be electronic and used for electronic payments. The term "issuer system" refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer that executes one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing transactions.

[0063] As used herein, the term "merchant" may refer to a person or entity that provides goods and / or services, or access to goods and / or services, to a customer based on a transaction, such as a payment transaction. The term "merchant" or "merchant system" may also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer that executes one or more software applications. As used herein, a "point of sale (POS) system" may refer to one or more computers and / or peripheral devices used by a merchant to conduct payment transactions with customers, including one or more card readers, near field communication (NFC) receivers, radio frequency identification (RFID) receivers and / or other contactless transceivers or receivers, contact-based receivers, payment terminals, computers, servers, input devices, and / or other similar devices that can be used to initiate payment transactions.

[0064] As used herein, the term "payment device" may refer to a payment card (e.g., a credit or debit card), a gift card, a smart card, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a key chain device or fob, an RFID transponder, a retailer discount or membership card, a cellular telephone, an electronic wallet mobile application, a personal digital assistant (PDA), a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, etc. In some non-limiting embodiments or aspects, a payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, an account holder's name, etc.).

[0065] As used herein, the term "payment gateway" may refer to an entity and / or a payment processing system operated by or on behalf of such an entity, which provides payment services (e.g., transaction service provider payment services, payment processing services, etc.) to one or more merchants (e.g., transaction service provider payment services, payment processing services, etc.). The payment service may be associated with the use of a portable financial device managed by a transaction service provider. As used herein, the term "payment gateway system" may refer to one or more computer systems, computer devices, servers, server groups, etc. operated by or on behalf of a payment gateway.

[0066] As used herein, the term "server" may refer to or include one or more computing devices operated by or facilitating communications and processing by multiple parties in a network environment such as the Internet, but it should be understood that communications may be facilitated through one or more public or private network environments, and that there may be various other arrangements. In addition, multiple computing devices (e.g., servers, point of sale (POS) devices, mobile devices, etc.) that communicate directly or indirectly in a network environment may constitute a "system." As used herein, references to "servers" or "processors" may refer to previously described servers and / or processors, different servers and / or processors, and / or combinations of servers and / or processors stated to perform previous steps or functions. For example, as used in the specification and claims, a first server and / or first processor stated to implement a first step or function may refer to the same or different servers and / or processors stated to implement a second step or function.

[0067] As used herein, the term "transaction service provider" may refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment assurance through an agreement between the transaction service provider and the issuer organization. For example, a transaction service provider may include, for example The term "transaction processing system" may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server executing one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0068] The present disclosure provides a new technology for compressing and regularizing embedding vectors. For example, a non-limiting embodiment or aspect of the present disclosure describes receiving embedding vector data including multiple embedding vectors. A beta variational autoencoder is trained based on the embedding vector data and a loss equation. In some non-limiting embodiments or aspects, the beta variational autoencoder includes an encoder network, a potential layer, and a decoder network. In some non-limiting embodiments or aspects, the loss equation includes a first term associated with the input reconstruction performed by the beta variational autoencoder, a second term associated with the regularization of the potential space of the potential layer, and a hyperparameter weight associated with the ratio of the first term to the second term. In some non-limiting embodiments or aspects, the potential space includes multiple dimensions. The present disclosure describes determining the corresponding entropy and corresponding variance of the corresponding mean value of each corresponding dimension in the multiple dimensions. A first subset of the multiple dimensions can be determined based on the corresponding entropy and the corresponding variance of the corresponding mean value of each corresponding dimension in the multiple dimensions. A second subset of the multiple dimensions can be discarded based on the corresponding entropy and the corresponding variance of the corresponding mean value of each corresponding dimension in the multiple dimensions. The present disclosure enables the generation of a compressed representation of the embedding vector data based on the first subset of dimensions. Thus, the disclosed subject matter improves computer technology by compressing embeddings while achieving similar or better accuracy (e.g., as shown by similarity scores, semantic detection, and performance on downstream tasks) than techniques using more dimensions. This compression also saves computational resources in terms of memory required to store the compressed embeddings and the number of input dimensions required for downstream models.

[0069] Figure 1 Depicted is a system 100 for embedding compression and regularization according to a non-limiting embodiment or aspect. The system 100 includes an embedding vector database 102, a variational autoencoder system 104, and (one or more) machine learning model systems 106.

[0070] The embedding vector database 102 may include one or more devices capable of receiving information from the variational autoencoder system 104 and / or the machine learning model system 106 and / or transmitting information to the variational autoencoder system and / or the machine learning model system. For example, the embedding vector database 102 may include a computing device, such as a computer, a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the embedding vector database 102 may communicate with a data storage device, which may be local or remote relative to the embedding vector database 102. In some non-limiting embodiments or aspects, the embedding vector database 102 is capable of receiving information from the data storage device, storing information in the data storage device, transmitting information to the data storage device, or searching for information stored in the data storage device.

[0071] The variational autoencoder system 104 may include one or more devices capable of receiving information from the embedding vector database 102 and / or (one or more) machine learning model systems 106 and / or transmitting information to the embedding vector database and / or the machine learning model system. For example, the variational autoencoder system 104 may include a computing device, such as a computer, a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the variational autoencoder system 104 may be associated with a financial institution, a transaction service provider, and / or an issuer, as described herein. For example, the variational autoencoder system 104 may be operated by a financial institution, a transaction service provider, and / or an issuer.

[0072] The machine learning model system(s) 106 may include one or more devices capable of receiving information from the embedding vector database 102 and / or the variational autoencoder system 104 and / or transmitting information to the embedding vector database and / or the variational autoencoder system. For example, the machine learning model system(s) 106 may include a computing device, such as a computer, a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the machine learning model system(s) 106 may be associated with a financial institution, a transaction service provider, and / or an issuer, as described herein. For example, the machine learning model system 106 may be operated by a financial institution, a transaction service provider, and / or an issuer.

[0073] In some non-limiting embodiments or aspects, the variational self-encoder system 104 may receive embedded vector data (e.g., from the embedding vector database 102), as described herein. For example, the embedded vector data may include multiple embedded vectors, as described herein. In some non-limiting embodiments or aspects, the variational self-encoder system 104 may train a β variational self-encoder, as described herein. For example, training a β variational self-encoder may be based on embedded vector data and a loss equation, as described herein. The β variational self-encoder may include an encoder network, a potential layer, and / or a decoder network, as described herein. The loss equation may include a first term associated with the input reconstruction performed by the β variational self-encoder, a second term associated with the regularization of the potential space of the potential layer, and / or a hyperparameter weight associated with the ratio of the first term to the second term, as described herein. The potential space may have multiple dimensions, as described herein. In some non-limiting embodiments or aspects, the variational self-encoder system 104 may determine the corresponding entropy of the corresponding mean value of each corresponding dimension in the multiple dimensions, as described herein. Additionally or alternatively, the variational self-encoder system 104 may determine the corresponding variance of each corresponding dimension in a plurality of dimensions, as described herein. In some non-limiting embodiments or aspects, the variational self-encoder system 104 may determine a first subset of a plurality of dimensions, as described herein. For example, a first subset of a plurality of dimensions may be based on the corresponding entropy and the corresponding variance of the corresponding mean value of each corresponding dimension in a plurality of dimensions, as described herein. In some non-limiting embodiments or aspects, the variational self-encoder system 104 may discard a second subset of a plurality of dimensions, as described herein. For example, the variational self-encoder system 104 may discard a second subset of a plurality of dimensions based on the corresponding entropy and the corresponding variance of the corresponding mean value of each corresponding dimension in a plurality of dimensions, as described herein. In some non-limiting embodiments or aspects, the variational self-encoder system 104 may generate a compressed representation of the embedded vector data. For example, the variational self-encoder system 104 may generate a compressed representation of the embedded vector data based on the first subset of dimensions, as described herein.

[0074] Figure 1 The number and arrangement of devices and networks shown in are provided as examples. Figure 1 Additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks than those shown in . In addition, Figure 1 Two or more of the devices shown in may be implemented in a single device, or Figure 1 The single device shown in the figure may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (eg, one or more devices) of the system 100 may perform one or more functions described as being performed by another set of devices of the system 100.

[0075] Reference Figure 2 , showing a process 200 for embedding compression and regularization. Figure 2 The steps shown are for exemplary purposes only. It should be understood that in non-limiting embodiments or aspects, additional, fewer, different and / or different order steps may be used. In some non-limiting embodiments or aspects, one or more steps of process 200 may be performed by the variational self-encoder system 104 (e.g., one or more devices of the variational self-encoder system 104) (e.g., completely, partially, etc.). In some non-limiting embodiments or aspects, one or more steps of process 200 may be performed by another device or device group that is separated from the variational self-encoder system 104 or includes the variational self-encoder system, such as the embedding vector database 102 and / or (one or more) machine learning model system 106 (e.g., one or more devices of (one or more) machine learning model system 106) (e.g., completely, partially, etc.).

[0076] like Figure 2 As shown, at step 202, process 200 may include receiving embedding vector data including a plurality of embedding vectors. For example, variational autoencoder system 104 may receive the embedding vector data from embedding vector database 102. In some non-limiting embodiments or aspects, the embedding vector data may include a plurality of embedding vectors.

[0077] like Figure 2 As shown, at step 204, process 200 may include training a β variational autoencoder. For example, the variational autoencoder system 104 may train the β variational autoencoder based on the embedding vector data and / or the loss equation. In some non-limiting embodiments or aspects, the β variational autoencoder may include an encoder network, a potential layer, and / or a decoder network. In some non-limiting embodiments or aspects, the loss equation may include a first term associated with the input reconstruction performed by the β variational autoencoder, a second term associated with the regularization of the potential space of the potential layer, and / or a hyperparameter weight associated with the ratio of the first term to the second term. In some non-limiting embodiments or aspects, the potential space may include multiple dimensions.

[0078] like Figure 2 As shown, at step 206, process 200 may include determining a corresponding mean value for each respective dimension in the plurality of dimensions. For example, the variational autoencoder system 104 may determine a corresponding mean value for each respective dimension in the plurality of dimensions. In some non-limiting embodiments or aspects, the variational autoencoder system 104 may determine a corresponding entropy of the corresponding mean value for each respective dimension in the plurality of dimensions.

[0079] like Figure 2As shown, at step 208, process 200 may include determining a corresponding variance for each respective dimension in the plurality of dimensions. For example, variational autoencoder system 104 may determine a corresponding variance for each respective dimension in the plurality of dimensions.

[0080] like Figure 2 As shown, at step 210, process 200 may include determining a first subset of the plurality of dimensions. For example, the variational autoencoder system 104 may determine the first subset of the plurality of dimensions based on a corresponding mean value and a corresponding variance of each corresponding dimension in the plurality of dimensions. In some non-limiting embodiments or aspects, the variational autoencoder system 104 may determine the first subset of the plurality of dimensions based on a corresponding entropy and a corresponding variance of a corresponding mean value of each corresponding dimension in the plurality of dimensions.

[0081] like Figure 2 As shown, at step 212, process 200 may include discarding a second subset of the plurality of dimensions. For example, the variational autoencoder system 104 may discard the second subset of the plurality of dimensions based on the respective entropy and the respective variance of the respective mean of each respective dimension in the plurality of dimensions.

[0082] like Figure 2 As shown, at step 214, process 200 may include generating a compressed representation of the embedded vector data. For example, variational autoencoder system 104 may generate a compressed representation of the embedded vector data based on a first subset of dimensions. In some non-limiting embodiments or aspects, the compressed representation of the embedded vector data may include one or more embedded vectors whose number of dimensions is reduced compared to the number of dimensions of the plurality of embedded vectors associated with the embedded vector data.

[0083] Additionally or alternatively, process 200 may include iteratively adjusting hyperparameter weights and repeating training, determining corresponding entropies of corresponding means, determining corresponding variances, determining a first subset, discarding a second subset, and / or generating a compressed representation. For example, variational autoencoder system 104 may iteratively adjust hyperparameter weights and repeating training, determining corresponding entropies of corresponding means, determining corresponding variances, determining a first subset, discarding a second subset, and / or generating a compressed representation.

[0084] Additionally or alternatively, process 200 may include determining that each corresponding dimension in a plurality of dimensions includes a useful dimension or a discarded dimension. For example, the variational self-encoder system 104 may determine that each corresponding dimension in a plurality of dimensions is a useful dimension or a discarded dimension based on the corresponding entropy and the corresponding variance of the corresponding mean value of the corresponding dimension. In some non-limiting embodiments or aspects, the corresponding entropy of the corresponding mean value of each useful dimension may be higher than a first threshold, and the corresponding variance of each useful dimension may be lower than a second threshold. In some non-limiting embodiments or aspects, the corresponding entropy of the corresponding mean value of each discarded dimension may be lower than a first threshold, and the corresponding variance of each discarded dimension may be higher than a second threshold. In some non-limiting embodiments or aspects, a first subset of a plurality of dimensions may include each useful dimension. In some non-limiting embodiments or aspects, a second subset of a plurality of dimensions may include each discarded dimension.

[0085] Reference Figure 3 , a diagram showing a non-limiting embodiment or aspect of an environment in which the methods, systems and / or computer program products described herein may be implemented according to the principles of the presently disclosed subject matter. Figure 3 As shown, the environment 300 includes a transaction service provider system 302, an issuer system 304, a client device 306, a merchant system 308, an acquirer system 310, and a communication network 312. In some non-limiting embodiments or aspects, each of the embedding vector database 102, the variational autoencoder system 104, and the (one or more) machine learning model system 106 may be implemented by the transaction service provider system 302 (e.g., a portion of the transaction service provider system). In some non-limiting embodiments or aspects, at least one of the embedding vector database 102, the variational autoencoder system 104, and the (one or more) machine learning model system 106 may be implemented separately from the transaction service provider system 302 or include another system, another device, another system group, or another device group (e.g., the issuer system 304, the merchant system 308, the acquirer system 310, etc.) (e.g., a portion thereof) of the transaction service provider system.

[0086] The transaction service provider system 302 may include one or more devices capable of receiving information from and / or transmitting information to an issuer system 304, a client device 306, a merchant system 308, and / or an acquirer system 310 via a communication network 312. For example, the transaction service provider system 302 may include a computing device, such as a server (e.g., a transaction processing server), a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the transaction service provider system 302 may be associated with a transaction service provider, as described herein. In some non-limiting embodiments or aspects, the transaction service provider system 302 may communicate with a data storage device, which may be local or remote to the transaction service provider system 302. In some non-limiting embodiments or aspects, the transaction service provider system 302 may be capable of receiving information from, storing information in, transmitting information to, or searching for information stored in the data storage device.

[0087] The issuer system 304 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 302, the client device 306, the merchant system 308, and / or the acquirer system 310 via the communication network 312. For example, the issuer system 304 may include a computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the issuer system 304 may be associated with an issuer organization, as described herein. For example, the issuer system 304 may be associated with an issuer organization that issues a credit account, a debit account, a credit card, a debit card, etc. to a user associated with the client device 306.

[0088] The client devices 306 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 302, the issuer system 304, the merchant system 308, and / or the acquirer system 310 via the communication network 312. Additionally or alternatively, each client device 306 may include a device capable of receiving information from and / or transmitting information to other client devices 306 via the communication network 312, another network (e.g., an ad hoc network, a local network, a private network, a virtual private network, etc.), and / or any other suitable communication technology. For example, the client devices 306 may include client devices, etc. In some non-limiting embodiments or aspects, the client devices 306 may or may not be capable of communicating via a short-range wireless communication connection (e.g., an NFC communication connection, an RFID communication connection, Communication connection, communication connection, etc.) (e.g., from merchant system 308 or from another client device 306), and / or transmit information via a short-range wireless communication connection (e.g., to merchant system 308).

[0089] The merchant system 308 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 302, the issuer system 304, the client device 306, and / or the acquirer system 310 via the communication network 312. The merchant system 308 may also include a communication connection (e.g., an NFC communication connection, an RFID communication connection, a NFC communication connection, a ... Communication connection, In some non-limiting embodiments or aspects, the merchant system 308 may include a computing device, such as a server, a server group, a client device, a client device group, and / or other similar devices. In some non-limiting embodiments or aspects, the merchant system 308 may be associated with a merchant, as described herein. In some non-limiting embodiments or aspects, the merchant system 308 may include one or more client devices. For example, the merchant system 308 may include a client device that allows a merchant to transmit information to the transaction service provider system 302. In some non-limiting embodiments or aspects, the merchant system 308 may include one or more devices that can be used by merchants to conduct transactions with users, such as computers, computer systems, and / or peripheral devices. For example, the merchant system 308 may include a POS device and / or a POS system.

[0090] The acquirer system 310 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 302, the issuer system 304, the client device 306, and / or the merchant system 308 via the communication network 312. For example, the acquirer system 310 may include a computing device, a server, a server group, etc. In some non-limiting embodiments or aspects, the acquirer system 310 may be associated with an acquirer, as described herein.

[0091] The communication network 312 may include one or more wired and / or wireless networks. For example, the communication network 312 may include a cellular network (e.g., a long term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network (e.g., a private network associated with a transaction service provider), an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of these or other types of networks.

[0092] In some non-limiting embodiments or aspects, processing a transaction may include generating and / or transmitting at least one transaction message (e.g., an authorization request, an authorization response, any combination thereof, etc.). For example, a client device (e.g., a client device 306, a POS device of a merchant system 308, etc.) may initiate a transaction, for example, by generating an authorization request. Additionally or alternatively, a client device (e.g., a client device 306, at least one device of a merchant system 308, etc.) may transmit an authorization request. For example, the client device 306 may transmit an authorization request to a merchant system 308 and / or a payment gateway (e.g., a payment gateway of a transaction service provider system 302, a third-party payment gateway separate from the transaction service provider system 302, etc.). Additionally or alternatively, the merchant system 308 (e.g., its POS device) may transmit an authorization request to an acquirer system 310 and / or a payment gateway. In some non-limiting embodiments or aspects, the acquirer system 310 and / or the payment gateway may transmit an authorization request to a transaction service provider system 302 and / or an issuer system 304. Additionally or alternatively, the transaction service provider system 302 may transmit the authorization request to the issuer system 304. In some non-limiting embodiments or aspects, the issuer system 304 may determine an authorization decision (e.g., authorization, rejection, etc.) based on the authorization request. For example, the authorization request may cause the issuer system 304 to determine the authorization decision based on the authorization request. In some non-limiting embodiments or aspects, the issuer system 304 may generate an authorization response based on the authorization decision. Additionally or alternatively, the issuer system 304 may transmit the authorization response. For example, the issuer system 304 may transmit the authorization response to the transaction service provider system 302 and / or the payment gateway. Additionally or alternatively, the transaction service provider system 302 and / or the payment gateway may transmit the authorization response to the acquirer system 310, the merchant system 308, and / or the client device 306. Additionally or alternatively, the acquirer system 310 may transmit the authorization response to the merchant system 308 and / or the payment gateway. Additionally or alternatively, the payment gateway may transmit the authorization response to the merchant system 308 and / or the client device 306. Additionally or alternatively, merchant system 308 can transmit an authorization response to client device 306. In some non-limiting embodiments or aspects, merchant system 308 can receive the authorization response (e.g., from acquirer system 310 and / or payment gateway). Additionally or alternatively, merchant system 308 can complete the transaction (e.g., provide, ship, and / or deliver goods and / or services associated with the transaction; fulfill an order associated with the transaction; any combination thereof, etc.) based on the authorization response.

[0093] For purposes of illustration, processing a transaction may include generating a transaction message (e.g., an authorization request, etc.) based on an account identifier of a customer (e.g., associated with a customer device 306, etc.) and / or transaction data associated with the transaction. For example, a merchant system 308 (e.g., a client device of the merchant system 308, a POS device of the merchant system 308, etc.) may initiate a transaction, for example, by generating an authorization request (e.g., in response to receiving an account identifier from a portable financial device, etc. of the customer). Additionally or alternatively, the merchant system 308 may transmit the authorization request to an acquirer system 310. Additionally or alternatively, the acquirer system 310 may transmit the authorization request to a transaction service provider system 302. Additionally or alternatively, the transaction service provider system 302 may transmit the authorization request to an issuer system 304. The issuer system 304 may determine an authorization decision (e.g., authorization, rejection, etc.) based on the authorization request, and / or the issuer system 304 may generate an authorization response based on the authorization decision and / or the authorization request. Additionally or alternatively, the issuer system 304 may transmit the authorization response to the transaction service provider system 302. Additionally or alternatively, the transaction service provider system 302 may communicate the authorization response to the acquirer system 310 , which may communicate the authorization response to the merchant system 308 .

[0094] For purposes of illustration, clearing and / or settlement of a transaction may include generating a message (e.g., a clearing message, a settlement message, etc.) based on an account identifier of a customer (e.g., associated with a customer device 306, etc.) and / or transaction data associated with the transaction. For example, the merchant system 308 may generate at least one clearing message (e.g., a plurality of clearing messages, a batch of clearing messages, etc.). Additionally or alternatively, the merchant system 308 may transmit the clearing message to the acquirer system 310. Additionally or alternatively, the acquirer system 310 may transmit the clearing message to the transaction service provider system 302. Additionally or alternatively, the transaction service provider system 302 may transmit the clearing message to the issuer system 304. Additionally or alternatively, the issuer system 304 may generate at least one settlement message based on the clearing message. Additionally or alternatively, the issuer system 304 may transmit the settlement message and / or funds to the transaction service provider system 302 (and / or a settlement bank system associated with the transaction service provider system 302). Additionally or alternatively, transaction service provider system 302 (and / or settlement bank system) may transmit the settlement message and / or funds to acquirer system 310, which may transmit the settlement message and / or funds to merchant system 308 (and / or an account associated with merchant system 308).

[0095] Figure 3The number and arrangement of systems, devices, and / or networks shown are provided as examples. Additional systems, devices, and / or networks may exist; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or networks in a similar manner. Figure 3 The systems, devices and / or networks shown in the figure may be arranged in different ways. In addition, the system or device may be implemented in a single system or device. Figure 3 Two or more systems or devices shown, or Figure 3 The single system or device shown may be implemented as multiple distributed systems or devices. Additionally or alternatively, a system set (e.g., one or more systems) or a device set (e.g., one or more devices) of environment 300 may perform one or more functions described as being performed by another system set or another device set of environment 300.

[0096] Reference Figure 4 , a diagram showing example components of a device 900 according to a non-limiting embodiment or aspect. As an example, the device 900 may correspond to Figure 1 The embedding vector database 102, the variational autoencoder system 104 and / or the machine learning system 106 in Figure 3 Transaction service provider system 302, issuer system 304, client device 306, merchant system 308, and / or acquirer system 310 in the embodiment of the present invention. In some non-limiting embodiments or aspects, such systems or devices may include at least one device 900 and / or at least one component of device 900. The number and arrangement of components shown are provided as examples. In some non-limiting embodiments or aspects, Figure 1 Compared to those shown in , the device 900 may include additional components, fewer components, different components, or components arranged in a different manner. Additionally or alternatively, a component set (e.g., one or more components) of the device 900 may perform one or more functions described as being performed by another component set of the device 900.

[0097] like Figure 4As shown in , the device 900 may include a bus 902, a processor 904, a memory 906, a storage component 908, an input component 910, an output component 912, and a communication interface 914. The bus 902 may include components that permit communication between components of the device 900. In some non-limiting embodiments or aspects, the processor 904 may be implemented in hardware, firmware, or a combination of hardware and software. For example, the processor 904 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.). The memory 906 may include a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by the processor 904.

[0098] Continue to refer Figure 4 , the storage component 908 may store information and / or software related to the operation and use of the device 900. For example, the storage component 908 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a solid-state disk, etc.) and / or another type of computer-readable medium. The input component 910 may include a component that allows the device 900 to receive information, for example, through user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). In addition or alternatively, the input component 910 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). The output component 912 may include a component that provides output information from the device 900 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). The communication interface 914 may include a transceiver-type component (e.g., a transceiver, a separate receiver and a transmitter, etc.) that enables the device 900 to communicate with other devices, for example, through a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 914 may allow the device 900 to receive information from another device and / or provide information to another device. For example, the communication interface 914 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, Interface, cellular network interface, etc.

[0099] The device 900 may perform one or more processes described herein. The device 900 may perform these processes based on the processor 904 executing software instructions stored by a computer-readable medium such as a memory 906 and / or a storage component 908. The computer-readable medium may include any non-transient memory device. The memory device includes a memory space located inside a single physical storage device or a memory space spread over multiple physical storage devices. The software instructions may be read from another computer-readable medium or from another device to the memory 906 and / or the storage component 908 via the communication interface 914. When executed, the software instructions stored in the memory 906 and / or the storage component 908 may cause the processor 904 to perform one or more processes described herein. In addition or alternatively, the hard-wired circuit system may replace the software instructions or be used in combination with the software instructions to perform one or more processes described herein. Therefore, the embodiments or aspects described herein are not limited to any specific combination of hardware circuits and software. The term "programmed or configured" used herein refers to the arrangement of software, hardware circuit systems, or any combination thereof on one or more devices.

[0100] Reference Figure 5 , showing a diagram of a machine learning model architecture of an autoencoder 502 and a machine learning model architecture of a beta variational autoencoder 550. In some non-limiting embodiments or aspects, the autoencoder 502 may include a neural network (e.g., an unsupervised deep neural network) having two subnetworks, an encoder network 504 and a decoder network 506. The encoder network 504 may receive each training instance as input, and may transfer the input to a latent layer 508 (e.g., a latent space representation) to become a latent vector 508A (e.g., a high-dimensional latent vector). In some non-limiting embodiments or aspects, the latent layer 508 may include a set of scalars (e.g., one or more vectors). The decoder network 506 may take the latent vector 508A as input and reconstruct the training instance from the latent vector 508A. In some non-limiting embodiments or aspects, the encoder network 504 and the decoder network 506 may be jointly trained to minimize the difference between the input and the corresponding reconstruction.

[0101] In some non-limiting embodiments or aspects, the beta variational autoencoder 550 may include a neural network (e.g., an unsupervised deep neural network) having two sub-networks, an encoder network 554 and a decoder network 556. Figure 5As shown, the latent space 558 may include a set of Gaussian distributions, each having a mean value 558A and a variance 558B. In some non-limiting embodiments, the mean value 558A may be parameterized by a mean vector, and the variance 558B may be parameterized by a variance vector. The encoder network 554 may receive each training instance as input, and may transfer the input to the latent space 558 to generate a latent vector 558C (e.g., a high-dimensional latent vector based on a Gaussian distribution with a mean value 558A and a variance 558B). In some non-limiting embodiments, the decoder network 556 may receive samples (e.g., random samples) from each Gaussian distribution to generate a latent vector 558C, and the decoder network 556 may take the latent vector 558C as input and reconstruct the training instance (e.g., each training instance from a training data set) from the latent vector 558C.

[0102] In some non-limiting embodiments or aspects, the variational autoencoder system 104 may perform a process for embedding compression and regularization (e.g., process 200) using the variational autoencoder 550. In some non-limiting embodiments or aspects, the variational autoencoder system 104 may train the variational autoencoder 550 (e.g., a β variational autoencoder) based on the embedding vector data including multiple embedding vectors and the loss equation. In some non-limiting embodiments or aspects, the loss equation of the variational autoencoder 550 may include a term associated with the reconstruction of the input by the β variational autoencoder, referred to as the reconstruction loss (e.g., it may include the reconstruction loss inherited from the autoencoder 502), and a term associated with the regularization of the latent space 558 (e.g., the latent space 558 of the latent layer including a Gaussian distribution with a mean 558A and a variance 558B and a latent vector 558C), referred to as the regularization loss.

[0103] In some non-limiting embodiments or aspects, the regularization loss may minimize the Kullback-Leibler divergence (D KL ). The loss equation of the β variational autoencoder 550 may be:

[0104]

[0105] where x and are the input and reconstruction (e.g., the input reconstruction provided by the β variational autoencoder 550), respectively, with n dimensions. is a function of a normal distribution (e.g., a Gaussian distribution). The first loss term of equation (1) may include x and The difference between 2 The norm loss is minimized as:

[0106]

[0107] The first loss term may be referred to as the reconstruction loss. In some non-limiting embodiments or aspects, m is the dimensionality of the latent space (e.g., latent space 558), and the latent representation of x in each latent dimension is μ i and σ i 2 The second loss term, which can be called the regularization loss and can be used to constrain the latent space, is:

[0108]

[0109] β may be a weight term (e.g., a hyperparameter weight) associated with the ratio of the first term to the second term. β may control the reconstruction and regularization ratio, which may be used to balance the first loss term and the second loss term, and may be equal to 1. In some non-limiting embodiments or aspects, reducing the β value (e.g., relatively small) may reduce the dimensionality of the latent space, maintain the quality of the input reconstruction, and / or allow each latent vector to become more semantically salient. In some non-limiting embodiments or aspects, increasing the β value (e.g., relatively small) may increase the dimensionality of the latent space and / or prevent information loss about the input.

[0110] In some non-limiting embodiments or aspects, β may be a value other than 1. For example, β may be greater than 1 to further regularize the latent space. In some non-limiting embodiments or aspects, β values ​​may be selected to effectively disentangle the latent space so that each latent dimension encodes more orthogonal information than for other β values, which may further regularize individual latent dimensions, and some latent dimensions may converge to a unit Gaussian. Therefore, the remaining dimensions may have to encode more information and in a more efficient manner to keep the overall loss small.

[0111] In some non-limiting embodiments or aspects, the latent space 558 may have multiple dimensions, and the variational autoencoder system 104 may determine a respective mean value, μ , for each respective dimension in the multiple dimensions. i (e.g., the mean term of a Gaussian distribution), and / or the corresponding variance for each corresponding dimension in the multiple dimensions, σ i 2 (eg, a variance term of a Gaussian distribution.) For example, the variational autoencoder system 104 may determine a respective entropy of a respective mean for each respective dimension in the plurality of dimensions.

[0112] In some non-limiting embodiments or aspects, the variational self-encoder system 104 may determine that each of the multiple dimensions includes a useful dimension (e.g., to be retained) or a discarded dimension (e.g., to be discarded) based on the corresponding mean and the corresponding variance of the corresponding dimension. For example, the variational self-encoder system 104 may use an entropy threshold (e.g., an entropy cutoff value including a relatively steep negative slope) and a variance threshold (e.g., a variance cutoff value including a relatively steep positive slope) to screen multiple dimensions, so that if the entropy has a higher negative rate of change and the variance has a higher positive rate of change, the dimensions with respect to this condition are useful, otherwise these dimensions may be discarded.

[0113] In some non-limiting embodiments or aspects, the variational self-encoder system 104 may encode all n inputs into the latent space 558. For each dimension, there may be n Gaussians (e.g., n pairs of means and variances), one Gaussian for each input. The variational self-encoder system 104 may calculate the entropy of the n means. For useful dimensions, the corresponding entropy is larger because the n inputs are encoded using the full range of values ​​of the dimension. However, for discarded dimensions, the entropy is smaller, tends to 0, and may be limited by regularization loss (e.g., because the Kullback-Leibler (KL) loss of the beta variational self-encoder 550 pushes all Gaussians to unit Gaussians).

[0114] In some non-limiting embodiments or aspects, the variational self-encoder system 104 may compare the corresponding entropy of the corresponding mean value of each dimension with the entropy threshold. In addition or alternatively, the variational self-encoder system 104 may compare the corresponding variance of each dimension with the variance threshold. In some non-limiting embodiments or aspects, if the corresponding entropy of the corresponding mean value meets the entropy threshold, the variational self-encoder system 104 may determine that the dimension is useful (e.g., a useful dimension). In some non-limiting embodiments or aspects, if the corresponding entropy of the corresponding mean value does not meet the entropy threshold, the variational self-encoder system 104 may determine that the dimension is abandoned (e.g., an abandoned dimension).

[0115] In some non-limiting embodiments or aspects, the corresponding entropy of the corresponding mean value of each useful dimension may be higher than a first threshold and the corresponding variance of each useful dimension may be lower than a second threshold, and the corresponding entropy of the corresponding mean value of each discarded dimension may be lower than the first threshold and the corresponding variance of each discarded dimension may be higher than the second threshold.

[0116] In some non-limiting embodiments or aspects, the variational self-encoder system 104 may determine a first subset of the multiple dimensions based on a corresponding mean and a corresponding variance of each corresponding dimension in the multiple dimensions. For example, the variational self-encoder system 104 may determine a first subset of the multiple dimensions based on a corresponding entropy and a corresponding variance of a corresponding mean of each corresponding dimension in the multiple dimensions.

[0117] In some non-limiting embodiments or aspects, the variational self-encoder system 104 may discard a second subset of the multiple dimensions based on the corresponding mean and the corresponding variance of each corresponding dimension in the multiple dimensions. For example, the variational self-encoder system 104 may discard a second subset of the multiple dimensions based on the corresponding entropy and the corresponding variance of the corresponding mean of each corresponding dimension in the multiple dimensions. In some non-limiting embodiments or aspects, the number of potential dimensions in the second subset discarded by the variational self-encoder system 104 may be related to the β value.

[0118] In some non-limiting embodiments or aspects, the variational autoencoder system 104 may generate a compressed representation of the embedded vector data (e.g., latent vector 558C) based on a first subset of dimensions.

[0119] Reference Fig. 6A and 6B , showing a graphical representation of the quantitative embedding results. Fig. 6A is a table showing similarity scores of monolingual embeddings for 3 types of machine learning models. Fig. 6A The similarity scores shown (e.g., 0 to 1, where 1 indicates equality) provide an indication of the quality of the regularized embeddings for different languages. The table shows single-language results for five languages: English (En), German (De), Italian (It), Spanish (Es), and Persian (Fa). FastText provides the original embeddings (e.g., 300 dimensions), which are used as input to the autoencoder (AE) (e.g., autoencoder 502) and the beta variational autoencoder (βVAE) (e.g., beta variational autoencoder 550). Both the AE and the βVAE have been trained with 350 dimensions, and the βVAE is compressed to the number of dimensions shown in parentheses next to the similarity scores. Figure 6B is a plot of the similarity scores of AE and βVAE versus the dimensionality of the latent space. The β value is shown in parentheses next to the latent dimension.

[0120] Although embodiments or aspects have been described in detail for purposes of illustration, it should be understood that such details are used for that purpose only, and the present disclosure is not limited to the disclosed embodiments or aspects, but on the contrary, is intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it should be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment.

Claims

1. A computer-implemented method for generating a compressed representation of natural language semantic embedding vector data, comprising: receiving, with at least one processor, the natural language semantic embedding vector data comprising a plurality of natural language semantic embedding vectors; training, with at least one processor, a beta variational autoencoder based on the natural language semantic embedding vector data and a loss equation, the beta variational autoencoder comprising an encoder network, a latent layer, and a decoder network, the loss equation comprising a first term associated with input reconstruction of the beta variational autoencoder, a second term associated with regularization of a latent space of the latent layer, and a hyperparameter weight associated with a ratio of the first term to the second term, wherein the latent space has a plurality of dimensions; determining, with at least one processor, a respective mean value for each respective dimension of the plurality of dimensions, wherein determining the respective mean value for each respective dimension of the plurality of dimensions comprises determining a respective entropy of the respective mean value for each respective dimension of the plurality of dimensions; determining, with at least one processor, a respective variance for each respective dimension of the plurality of dimensions; Determining, with at least one processor, that each of the plurality of dimensions comprises a useful dimension or a discarded dimension based on the corresponding means and the corresponding variances of the corresponding dimensions, wherein a corresponding entropy of the corresponding mean of each useful dimension is above a first threshold and the corresponding variance of each useful dimension is below a second threshold; and wherein a corresponding entropy of the corresponding mean of each discarded dimension is below the first threshold and the corresponding variance of each discarded dimension is above the second threshold; determining, with at least one processor, a first subset of the plurality of dimensions based on a respective mean and a respective variance of each respective dimension in the plurality of dimensions, wherein the first subset of the plurality of dimensions includes each useful dimension; discarding, with at least one processor, a second subset of the plurality of dimensions based on a respective mean and a respective variance of each respective dimension in the plurality of dimensions, wherein the second subset of the plurality of dimensions is different from the first subset of the plurality of dimensions, wherein the second subset of the plurality of dimensions includes each discarded dimension, and wherein discarding the second subset of the plurality of dimensions reduces the dimensionality of the latent space such that the natural language semantic embedding vector data is more semantically salient; generating, with at least one processor, a compressed representation of the natural language semantic embedding vector data based on the first subset of dimensions; and Using at least one processor, the compressed representation of the natural language semantics is embedded into the vector data and input into at least one machine learning model.

2. The computer-implemented method of claim 1 , wherein training the beta variational autoencoder comprises: Iteratively adjusting the hyperparameter weights and repeating the training, the determining the respective entropies of the respective means, the determining the respective variances, the determining the first subset, the discarding the second subset, and the generating the compressed representation.

3. The computer-implemented method of claim 1 , wherein determining a first subset of the plurality of dimensions comprises: A first subset of the plurality of dimensions is determined, with at least one processor, based on a respective entropy and a respective variance of a respective mean of each respective dimension of the plurality of dimensions.

4. The computer-implemented method of claim 3, wherein discarding the second subset of the plurality of dimensions comprises: A second subset of the plurality of dimensions is discarded based on the respective entropy and the respective variance of the respective mean of each respective dimension in the plurality of dimensions.

5. The computer-implemented method of claim 1 , wherein the loss equation is: Wherein the first term associated with the reconstruction of the input of the beta variational autoencoder is: where the second term associated with the regularization of the latent space of the latent layer is: where x i is the input, is reconstructed from the input provided by the β variational autoencoder; D KL is a function of the Kullback-Leibler divergence; is a function of the normal distribution; and Wherein the hyperparameter weight associated with the ratio of the first term to the second term is β.

6. A system for generating a compressed representation of natural language semantic embedding vector data, comprising at least one processor, wherein the at least one processor is programmed or configured to: Receiving the natural language semantic embedding vector data including a plurality of natural language semantic embedding vectors; Based on the natural language semantic embedding vector data and a loss equation, training a beta variational autoencoder, the beta variational autoencoder comprising an encoder network, a latent layer, and a decoder network, the loss equation comprising a first term associated with input reconstruction of the beta variational autoencoder, a second term associated with regularization of a latent space of the latent layer, and a hyperparameter weight associated with a ratio of the first term to the second term, wherein the latent space has multiple dimensions; Determining a respective average value for each respective dimension in the plurality of dimensions, wherein When determining the respective average value for each respective dimension of the plurality of dimensions, the at least one processor is programmed or configured to: determining a respective entropy of a respective mean value for each respective dimension of the plurality of dimensions; determining a respective variance for each respective dimension in the plurality of dimensions; Based on the corresponding averages and the corresponding variances of the corresponding dimensions, determining that each corresponding dimension of the plurality of dimensions comprises a useful dimension or a discarded dimension; wherein the corresponding entropy of the corresponding average of each useful dimension is higher than a first threshold, and the corresponding variance of each useful dimension is lower than a second threshold; and wherein the corresponding entropy of the corresponding average of each discarded dimension is lower than the first threshold, and the corresponding variance of each discarded dimension is higher than the second threshold; determining a first subset of the plurality of dimensions based on a respective mean and a respective variance of each respective dimension in the plurality of dimensions, wherein the first subset of the plurality of dimensions includes each useful dimension; Based on the corresponding mean and the corresponding variance of each corresponding dimension in the multiple dimensions, discarding a second subset of the multiple dimensions, wherein the second subset of the multiple dimensions is different from the first subset of the multiple dimensions, wherein the second subset of the multiple dimensions includes each discarded dimension, and wherein discarding the second subset of the multiple dimensions can reduce the dimensionality of the latent space so that the natural language semantic embedding vector data is more semantically salient; based on the first subset of dimensions, generating a compressed representation of the natural language semantic embedding vector data; and The compressed representation of the natural language semantic embedding vector data is input into at least one machine learning model.

7. The system according to claim 6, wherein: When training the beta variational autoencoder, the at least one processor is programmed or configured to: Iteratively adjusting the hyperparameter weights and repeating the training, the determining the respective entropies of the respective means, the determining the respective variances, the determining the first subset, the discarding the second subset, and the generating the compressed representation.

8. The system according to claim 6, in, When determining the first subset of the plurality of dimensions, the at least one processor is programmed or configured to: A first subset of the plurality of dimensions is determined based on a respective entropy and a respective variance of a respective mean value of each respective dimension of the plurality of dimensions.

9. The system according to claim 8, wherein: When discarding the second subset of the plurality of dimensions, the at least one processor is programmed or configured to: A second subset of the plurality of dimensions is discarded based on the respective entropy and the respective variance of the respective mean of each respective dimension in the plurality of dimensions.

10. The system of claim 6, wherein the loss equation is: Wherein the first term associated with the reconstruction of the input of the beta variational autoencoder is: where the second term associated with the regularization of the latent space of the latent layer is: where x i is the input, is reconstructed from the input provided by the β variational autoencoder; D KL is a function of the Kullback-Leibler divergence; is a function of the normal distribution; and Wherein the hyperparameter weight associated with the ratio of the first term to the second term is β.

11. A computer program product for generating a compressed representation of natural language semantic embedding vector data, the computer program product comprising at least one non-transitory computer-readable medium, the non-transitory computer-readable medium comprising one or more instructions that, when executed by at least one processor, cause the at least one processor to: Receiving the natural language semantic embedding vector data including a plurality of natural language semantic embedding vectors; Based on the natural language semantic embedding vector data and a loss equation, training a beta variational autoencoder, the beta variational autoencoder comprising an encoder network, a latent layer, and a decoder network, the loss equation comprising a first term associated with input reconstruction of the beta variational autoencoder, a second term associated with regularization of a latent space of the latent layer, and a hyperparameter weight associated with a ratio of the first term to the second term, wherein the latent space has multiple dimensions; Determining a respective average value for each respective dimension in the plurality of dimensions, wherein When determining a respective average value for each respective dimension of the plurality of dimensions, the one or more instructions cause the at least one processor to perform: determining a respective entropy of a respective mean value for each respective dimension of the plurality of dimensions; determining a respective variance for each respective dimension in the plurality of dimensions; Determining, based on the corresponding means and the corresponding variances of the corresponding dimensions, that each corresponding dimension of the plurality of dimensions comprises a useful dimension or a discarded dimension, wherein the corresponding entropy of the corresponding mean of each useful dimension is higher than a first threshold, and the corresponding variance of each useful dimension is lower than a second threshold; and wherein the corresponding entropy of the corresponding mean of each discarded dimension is lower than the first threshold, and the corresponding variance of each discarded dimension is higher than the second threshold; determining a first subset of the plurality of dimensions based on a respective mean and a respective variance of each respective dimension in the plurality of dimensions, wherein the first subset of the plurality of dimensions includes each useful dimension; Based on the corresponding mean and the corresponding variance of each corresponding dimension in the multiple dimensions, discarding a second subset of the multiple dimensions, wherein the second subset of the multiple dimensions is different from the first subset of the multiple dimensions, wherein the second subset of the multiple dimensions includes each discarded dimension, and wherein discarding the second subset of the multiple dimensions can reduce the dimensionality of the latent space so that the natural language semantic embedding vector data is more semantically salient; based on the first subset of dimensions, generating a compressed representation of the natural language semantic embedding vector data; and The compressed representation of the natural language semantic embedding vector data is input into at least one machine learning model.

12. The computer program product according to claim 11, wherein: When training the beta variational autoencoder, the one or more instructions further cause the at least one processor to: Iteratively adjusting the hyperparameter weights and repeating the training, the determining the respective entropies of the respective means, the determining the respective variances, the determining the first subset, the discarding the second subset, and the generating the compressed representation.

13. The computer program product of claim 11, wherein: When determining the first subset of the plurality of dimensions, the one or more instructions cause the at least one processor to: A first subset of the plurality of dimensions is determined based on a respective entropy and a respective variance of a respective mean value of each respective dimension of the plurality of dimensions.

14. The computer program product of claim 13, wherein: When discarding the second subset of the plurality of dimensions, the one or more instructions cause the at least one processor to: discarding a second subset of the plurality of dimensions based on a respective entropy and a respective variance of a respective mean of each respective dimension in the plurality of dimensions, The loss equation is: Wherein the first term associated with the reconstruction of the input of the beta variational autoencoder is: where the second term associated with the regularization of the latent space of the latent layer is: where x i is the input, is reconstructed from the input provided by the β variational autoencoder; D KL is a function of the Kullback-Leibler divergence; is a function of the normal distribution; and Wherein the hyperparameter weight associated with the ratio of the first term to the second term is β.

Images