Method, system and storage medium for classifying input samples using a comparison set

Abstract

Method, system and apparatus for classifying new samples using a comparison set of comparison samples. A method includes maintaining a comparison set, the comparison set including comparison samples and a corresponding label vector for each comparison sample, each label vector including a corresponding score for each label in a predetermined label set; receiving a new sample; determining a corresponding attention weight for each comparison sample by applying a neural network attention mechanism to the new sample and the comparison sample; and generating a corresponding label score for each label in the predetermined label set for each comparison sample based on the corresponding attention weight of the comparison sample and the corresponding label vector of the comparison sample, wherein the corresponding label score for each label represents the likelihood that the label is a correct label for the new sample.

Description

[0001] This application is a divisional application of the invention patent application with application date of May 19, 2017 and application number 201780031165.7. Technical Field

[0002] This specification relates to classifying data using neural networks. Background Art

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. In addition to an output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters.

[0004] Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network is able to use some or all of the internal state of the network from previous time steps to compute the output at the current time step. Summary of the invention

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations. The system is a system for classifying an input sample using a comparison set of comparison samples.

[0006] An example aspect of the present disclosure relates to a method, performed by one or more computers, for classifying a new sample using a comparison set of comparison samples, the method comprising: maintaining a comparison set, the comparison set comprising a plurality of comparison samples and a corresponding label vector for each of the plurality of comparison samples, each label vector comprising a corresponding score for each label in a predetermined label set; receiving a new sample; determining a corresponding attention weight for each comparison sample by applying a neural network attention mechanism to the new sample and the comparison sample; and generating a corresponding label score for each label in the predetermined label set for each comparison sample based on the corresponding attention weight of the comparison sample and the corresponding label vector of the comparison sample, wherein the corresponding label score for each label represents the likelihood that the label is a correct label for the new sample.

[0007] In an embodiment of this aspect, the method may further include classifying the new sample based on the corresponding label scores of the labels in the predetermined label set. For example, the new sample may be classified using a classification derived from the label scores. As a further example, the new sample may be classified using / based on the label with the highest probability of being correct, or the new sample may be classified using / based on the n (n>1) labels with the highest probability of being correct, or the new sample may be classified using / based on all labels with a probability of being correct exceeding a threshold, and so on.

[0008] Another example aspect of the present disclosure relates to a method for controlling an agent in an environment performed by one or more computers. The method includes generating one or more control inputs for controlling the agent based on the output of the method of the first aspect (e.g., based on the generated label scores, or based on the classification derived from the generated label scores).

[0009] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. A data classification system including a neural network can effectively determine labels for unobserved classifications without making any changes to the neural network or retraining the neural network from scratch. In particular, the neural network can take as its input a new set of labeled samples and new input samples that need to be classified, and even if the new set of labeled samples is different from the set of labeled samples previously used to train the neural network, the neural network can use the new set of labeled samples to effectively classify the new input samples. Therefore, embodiments of the present subject matter can provide a neural network that can be trained to process input data (e.g., image data) to generate output data that indicates attributes associated with the image data in a more efficient manner.

[0010] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An example data classification system is shown.

[0012] Figure 2 is a flow chart of an example process for classifying new samples using a comparison set of comparison samples.

[0013] Figure 3 is a flowchart of an example process for determining the corresponding attention weight for each comparison sample.

[0014] Figure 4 is a flowchart of an example process for training a neural network attention mechanism.

[0015] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0016] This specification generally describes a data classification system that is configured to classify any of a variety of data (ie, any type of data object that can be classified as belonging to one or more categories).

[0017] For example, if the input data is an image, the data classification system may be a visual recognition system that determines whether the input image includes an image of an object belonging to an object class from a predetermined set of object classes. As another example, if the input data is a video or a portion of a video, the data classification system may be a video classification system that determines which topic or topics the input video or video portion is related to. As another example, if the input data is audio data, the data classification system may be a speech recognition system that determines, for a given spoken utterance, one or more terms represented by the utterance. As another example, if the input data is text data, the data classification system may be a text classification system that determines which topic or topics the input text segment is related to.

[0018] Figure 1 An example data classification system 100 is shown.

[0019] Data classification system 100 is an example of a system implemented on one or more computers in one or more locations as a computer program that implements the systems, components, and techniques described below.

[0020] The data classification system 100 is configured to receive an input sample 102 and generate an output 116 that classifies the input sample 102 , for example, determines the category to which the input sample 102 belongs or the subject matter to which the input sample 102 relates.

[0021] The data classification system 100 stores comparison sample data 104 including a comparison set. The comparison set includes a plurality of comparison samples and a corresponding label vector for each comparison sample. Each label vector includes a corresponding score for each label in a predetermined label set. Typically, the label score in a given label vector identifies one or more known labels of the corresponding comparison sample.

[0022] The data classification system 100 includes a classifier 112 that takes as input the comparison sample data 104 and the input sample 102. The classifier 112 generates an output 116 as a function of the comparison sample data and the input sample 102 using a neural network attention mechanism 114. Figure 2 and Figure 3 The process of generating output 116 using neural network attention mechanism 114 is described in more detail.

[0023] The output 116 from the data classification system 100 can be used in many applications. As an example application, the output 116 from the data classification system 100 can be used to select one or more actions for controlling an agent in an environment. For example, the output 116 of the data classification system 100 can be used as or used to generate one or more control inputs that provide control of an agent in the environment. The environment can be a real-world environment in which an object in the real-world environment is controlled. For example, the object / agent can be a vehicle, such as an autonomous ("self-driving") or partially autonomous vehicle, and the output 116 of the data classification system 100 can be used as or can be used to generate control inputs for controlling the vehicle (e.g., controlling the navigation of the vehicle). Therefore, some aspects can solve problems associated with efficiently and effectively selecting actions for controlling an agent based on input data.

[0024] Figure 2 2 is a flow chart of an example process 200 for classifying a new sample using a comparison set of comparison samples. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a data classification system (e.g., Figure 1 The data classification system 100 is capable of performing process 200.

[0025] The system maintains a comparison set (step 202). As described above, the comparison set can include k comparison samples and can be expressed as:

[0026]

[0027] where x i is the comparison sample, y i is the comparison sample x i The corresponding label vector of each label vector y i The corresponding scores for each label in the predetermined label set are included. The label scores in a given label vector generally identify the known one or more labels of the corresponding comparison sample.

[0028] In some cases, the comparison set can be different from the comparison set used to train the neural network. That is, the neural network can be used to classify new samples using a new comparison sample set that is different from the set used to train the neural network, without retraining the neural network. In some of these cases, the new comparison sample set can include some of the comparison samples used to train the neural network and some additional or "unseen" comparison samples. In other of these cases, the new comparison sample set can include only "unseen" comparison samples and does not include comparison samples used for training.

[0029] The system receives a new sample (step 204). The new sample can be represented as

[0030] After receiving the new sample, the system determines the corresponding attention weight of each comparison sample by applying the neural network attention mechanism to the new sample and the comparison sample (step 206). Figure 3 The process for determining the corresponding attention weights for each comparison sample is described in more detail.

[0031] Then, the system generates a corresponding label score for each label in the predetermined label set for the new sample, so that the corresponding label score of each label represents the likelihood that the label is the correct label for the new sample (step 208). In particular, the system generates a label score for each comparison sample based on the corresponding attention weight of the comparison sample and the score in the corresponding label vector of the comparison sample.

[0032] In some embodiments, the tag score can be calculated as follows:

[0033]

[0034] Among them, x i is the comparison sample, y i From the comparison set The label vector of the comparison samples in , Given a new sample Comparison sample x i The attention weight.

[0035] Equation 1 describes the output of a new sample as a linear collection of label vectors in the comparison set. For each comparison sample x i , the system will compare the sample x i The label vector y i Multiply by the comparison sample x i The attention weight To generate comparison samples x i The weighted label vector The system then sums all weighted label vectors to generate a label vector that includes the combination of the corresponding label scores for each label in the predetermined label set. Each label score for each label represents the likelihood that the label is the correct label for a new sample.

[0036] In some embodiments, the system can also classify new samples based on the corresponding label scores of the labels in the predetermined label set. For example, the new sample can be classified using the classification derived from the label score. As a further example, the new sample can be classified using / based on the label with the highest probability of being correct, or the new sample can be classified using / based on the n (n>1) labels with the highest probability of being correct, or the new sample can be classified using / based on all labels with a probability of being correct exceeding a threshold.

[0037] After classifying the new sample, the system can provide the corresponding label scores of the labels in the predetermined label set, the one or more labels with the highest probability of being correct, or both to another system for further processing. For example, the new sample can be sensor data (e.g., an image) captured by the agent's sensor, and the other system can be a control system that uses the received label scores, the one or more labels with the highest probability of being correct, or both to decide how to control the agent in the environment.

[0038] Figure 3 is a flow chart of an example process 300 for determining the corresponding attention weights for each comparison sample using a neural network attention mechanism. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a data classification system (e.g., Figure 1 The data classification system 100 is capable of performing process 300.

[0039] The system uses a comparison sample neural network g(x i ) to process the comparison sample x i , to determine the digital embedding of the comparison sample (step 302). Generally, the digital embedding is an ordered set of numerical values, such as a vector of floating point values ​​or a vector of quantized floating point values.

[0040] In some embodiments, when the sample is an image, the sample neural network g(x i ) is a convolutional neural network configured to receive an input comparison sample and generate an embedding of the comparison sample. In some other embodiments, when the sample is a word, phrase, or other text fragment, the comparison sample neural network is a text embedding neural network that embeds the input text into a vector.

[0041] In some other embodiments, comparing the sample neural network g(x i) is a bidirectional long short-term memory (LSTM) neural network configured to process the comparison samples in the context of a comparison sample sequence of comparison samples from the comparison set to generate a digital embedding of the comparison samples. That is, the system is capable of arranging the comparison samples as a sequence, and for each comparison sample, using a bidirectional LSTM neural network according to the sequence to process the comparison sample or a feature representation of the comparison sample, for example, the comparison sample or the feature representation of the comparison sample generated by a convolutional neural network or a text embedding neural network to generate a digital embedding of the comparison sample.

[0042] The system uses a new sample neural network To process new samples To determine the digital embedding of the new sample (step 304). In some embodiments, when the sample is an image, the new sample neural network is a convolutional neural network configured to receive an input new sample and generate an embedding of the new sample. In some other embodiments, when the sample is a word, a phrase, or other text fragment, the comparison sample neural network is a text embedding neural network that embeds the input text into a vector.

[0043] In some other embodiments, the new sample neural network It is an LSTM neural network with read attention on the digital embeddings of multiple comparison samples, which is configured to process a sequence including K instances of new samples or K instances of feature representations of new samples to generate digital embeddings of comparison samples. New Sample Neural Network Can be expressed as follows:

[0044]

[0045] Where f′(x) is a new sample or feature representation of the new sample, and g(S) is a set of digital embeddings for the comparison sample. That is, at each time step of K steps except the last time step, the system uses the LSTM neural network to process the feature representation of the new sample according to the current internal state of the LSTM neural network to generate an initial update to the internal state. The system then combines (e.g., sums) the initial update and the feature representation to generate an embedding key, and uses the embedding key to apply a content-based attention mechanism to the digital embedding of the comparison sample to generate an attention vector. The system then combines (e.g., concatenates) the initial update and the attention vector to generate an updated internal state, i.e., an internal state that will be used as the current internal state for the next time step. The system is then able to use the initial update or embedding key of the last time step (i.e., the Kth step) as the final digital embedding for the new sample.

[0046] In some cases, when the new sample neural network and the comparison sample neural network are both or include convolutional neural networks, the new sample neural network and the comparison sample neural network share at least some parameters.

[0047] The system determines the corresponding attention weight by determining a similarity measure between the digital embedding of the comparison sample and the digital embedding of the new sample (step 306). In some embodiments, the similarity measure is a cosine distance. In some embodiments, the system calculates the attention weight for a given new sample based on the following model Each comparison sample x i The corresponding attention weights

[0048]

[0049] in Is to embed new samples The new sample neural network, g(x i ) is the embedding comparison sample x i Comparison sample neural network, c() is the cosine distance function.

[0050] Figure 4 is a flow chart of an example process 400 for training a neural network attention mechanism.

[0051] For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a data classification system (e.g., Figure 1 The data classification system 100 is capable of performing process 400 to train a neural network attention mechanism.

[0052] The system samples label sets from a distribution of possible label sets (step 402). Each label set is a different combination of labels from the complete set of possible labels. In some cases, each label set includes the same number of labels, for example, two or three. In other cases, different label sets include different numbers of labels, for example, ranging from one or two labels to five labels. For example, the distribution can evenly weight all possible label sets, and the system can use the distribution to randomly sample label sets.

[0053] The system then uses the sampled label set to sample a training comparison set and a batch of input samples (step 404). In particular, for each label in the sampled label set, the system samples a comparison sample set with that label. The sampled comparison sample set and their corresponding labels for all labels in the sampled label set form the training comparison set. The batch of input samples includes a random sequence of comparison samples with a label in the sampled label set.

[0054] The system trains the neural network attention mechanism to minimize the error in predicting labels of samples in a batch of input samples conditioned on a training comparison set (step 406). In particular, the system trains the attention mechanism to adjust current values ​​of parameters of the attention mechanism to reduce the error using traditional neural network training techniques (e.g., training techniques based on gradient descent).

[0055] The system can repeatedly perform process 400, i.e., determine the trained values ​​of the parameters of the neural network attention mechanism by repeatedly sampling the label set and then training the neural network attention mechanism based on the sampled label set. By training the neural network attention mechanism in this manner, the system can quickly determine the trained values ​​of the parameters of the neural network attention mechanism that meet the training goal of reducing error, and the trained neural network can perform well even when the comparison set used to classify the input sample is different from any comparison set used to train the neural network.

[0056] For a system of one or more computers configured to perform a specific operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action in operation. For one or more computer programs configured to perform a specific operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0057] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them). The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-temporary program carrier for operation by a data processing device or control of the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on a propagation signal (e.g., a machine-generated electrical, optical or electromagnetic signal) that is artificially generated and generated to encode information for transmission to a suitable receiver device for operation by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. However, a computer storage medium is not a propagation signal.

[0058] The term "data processing apparatus" includes all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or a plurality of processors or computers. The apparatus can include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus can also include code that creates an operating environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0059] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that stores other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files that store portions of one or more modules, subroutines or codes. A computer program can be deployed to run on one computer, or on multiple computers located at one site or distributed at multiple sites and interconnected by a communication network.

[0060] As used in this specification, an "engine" or "software engine" refers to a software-implemented input / output system that provides outputs that are different from inputs. An engine can be an encoded functional block, such as a library, a platform, a software development kit (Software Development Kit, "SDK"), or an object. Each engine can be implemented on any suitable type of computing device, for example, a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smart phone, or other fixed or portable device including one or more processors and a computer-readable medium. Additionally, two or more engines can be implemented on the same computing device or on different computing devices.

[0061] The processes and logic flows described in this specification can be performed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuits, and the apparatus can also be implemented as special-purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits). For example, the processes and logic flows can be performed by a graphics processing unit (GPU), and the apparatus can also be implemented as a graphics processing unit.

[0062] Computers suitable for running computer programs include, for example, computers that can be based on general or special microprocessors or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to one or more large-capacity storage devices (such as disks, magneto-optical disks, or optical disks) for storing data, to receive data from one or more large-capacity storage devices or to transfer data to one or more large-capacity storage devices, or for receiving and transmitting both. However, the computer does not need such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (Personal Digital Assistant, PDA), a mobile audio or video player, a game console, a global positioning system (Global Positioning System, GPS) receiver or a portable storage device (for example, a universal serial bus (Universal Serial Bus, USB) flash drive), and the above are just a few examples.

[0063] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0064] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0065] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server), or in a computing system that includes a middleware component (e.g., an application server), or in a computing system that includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or in a computing system that includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (Local Area Network, "LAN") and wide area networks (Wide Area Network, "WAN"), such as the Internet.

[0066] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between client and server occurs by virtue of computer programs running on their respective computers and having a client-server relationship to each other.

[0067] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features described in the context of separate embodiments in this specification can also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately in multiple embodiments, or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even requested to be protected as such at the outset, one or more features from the combination requested for protection can be deleted from the combination in some cases, and the combination requested for protection can involve sub-combinations or changes in sub-combinations.

[0068] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order or requiring all of the operations shown to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0069] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method, performed by one or more computers, for classifying a new image using a comparison set of comparison images, the method comprising: maintaining a first comparison set, the first comparison set comprising a plurality of comparison images and a corresponding label vector for each of the plurality of comparison images, each label vector comprising a corresponding score for each label in a predetermined label set; receiving a new image; determining, for each of the compared images, a corresponding attention weight representing a level of similarity between the compared image and the new image by applying a neural network attention mechanism to the new image and the compared image; as well as Generate a corresponding label score for each label in a predetermined label set for each comparison image according to the corresponding attention weight of the comparison image and the corresponding label vector of the comparison image, wherein the corresponding label score of each label represents the possibility that the label is a correct label of the new image, Among them, the neural network attention mechanism is trained in the following way: Sampling a label set from a distribution over multiple possible label sets; Using the sampled label set to sample a training comparison set and a batch of input images; and The neural network attention mechanism is trained to minimize the error in predicting labels for a batch of input images conditioned on a training comparison set.

2. The method according to claim 1, wherein: Each tag set in the plurality of possible tag sets is a different tag combination from the complete set of possible tags.

3. The method according to claim 1, wherein: Each tag set of the plurality of possible tag sets includes the same number of tags.

4. The method according to claim 1, wherein: Different ones of the plurality of possible tag sets include different numbers of tags.

5. The method according to claim 1, wherein: The distribution uniformly weights all possible label sets, and wherein employing the label sets comprises using the distribution to randomly sample the label sets.

6. The method according to claim 1, wherein: Using the sampled label set to sample the training comparison set and a batch of input images includes: For each label in the sampled label set, a training comparison image set having the label is sampled, wherein the sampled training comparison image set and its corresponding labels for all labels in the sampled label set form a training comparison set, and The batch of input images includes a random comparison image sequence, each having a corresponding label in the sampled label set.

7. The method according to claim 1, wherein: The training comparison set is different from the first comparison set.

8. The method according to claim 1, wherein: Training a neural network attention mechanism includes adjusting current values ​​of parameters of the neural network attention mechanism to minimize the error using the neural network training technique.

9. The method according to claim 8, wherein: Neural network training technology is based on gradient descent training technology.

10. The method according to claim 1, wherein: By applying the neural network attention mechanism to the new image and the comparison image, determining the corresponding attention weights for each comparison image that characterize the level of similarity between the comparison image and the new image includes: processing the comparison images using a comparison image neural network to determine a digital embedding of the comparison images; Processing the new image using the new image neural network to determine a digital embedding for the new image; and The corresponding attention weights are determined by determining the similarity measure between the digit embedding of the comparison image and the digit embedding of the new image.

11. The method according to claim 10, wherein: The similarity measure is cosine distance.

12. The method according to claim 1, wherein: Generating the corresponding label scores for each label in the predetermined label set includes: For each comparison image, multiply the label vector of the comparison image by the attention weight of the comparison image to generate a weighted label vector of the comparison image, and The weighted label vectors are summed to generate a label vector comprising a combination of the respective label scores for each label in the predetermined label set.

13. A system comprising one or more computers and one or more non-transitory computer storage media encoded with instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of classifying a new image using a comparison set of comparison images, the operations comprising: maintaining a first comparison set, the first comparison set comprising a plurality of comparison images and a corresponding label vector for each of the plurality of comparison images, each label vector comprising a corresponding score for each label in a predetermined label set; receiving a new image; determining, for each of the compared images, a corresponding attention weight representing a level of similarity between the compared image and the new image by applying a neural network attention mechanism to the new image and the compared image; as well as Generate a corresponding label score for each label in a predetermined label set for each comparison image according to the corresponding attention weight of the comparison image and the corresponding label vector of the comparison image, wherein the corresponding label score of each label represents the possibility that the label is a correct label of the new image, Among them, the neural network attention mechanism is trained in the following way: Sampling a label set from a distribution over multiple possible label sets; Using the sampled label set to sample a training comparison set and a batch of input images; and The neural network attention mechanism is trained to minimize the error in predicting labels for a batch of input images conditioned on a training comparison set.

14. The system according to claim 13, wherein: Each tag set in the plurality of possible tag sets is a different tag combination from the complete set of possible tags.

15. The system of claim 13, wherein: The distribution uniformly weights all possible label sets, and wherein employing the label sets comprises using the distribution to randomly sample the label sets.

16. The system of claim 13, wherein: Using the sampled label set to sample the training comparison set and a batch of input images includes: For each label in the sampled label set, a training comparison image set having the label is sampled, wherein the sampled training comparison image set and its corresponding labels for all labels in the sampled label set form a training comparison set, and The batch of input images includes a random comparison image sequence having a label from a sampled label set.

17. The system of claim 13, wherein: The training comparison set is different from the first comparison set.

18. The system of claim 13, wherein: Training a neural network attention mechanism includes adjusting current values ​​of parameters of the neural network attention mechanism to minimize the error using the neural network training technique.

19. The system of claim 18, wherein: Neural network training technology is based on gradient descent training technology.

20. One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations of classifying a new image using a comparison set of comparison images, the operations comprising: maintaining a first comparison set, the first comparison set comprising a plurality of comparison images and a corresponding label vector for each of the plurality of comparison images, each label vector comprising a corresponding score for each label in a predetermined label set; receiving a new image; determining, for each of the compared images, a corresponding attention weight representing a level of similarity between the compared image and the new image by applying a neural network attention mechanism to the new image and the compared image; as well as Generate a corresponding label score for each label in a predetermined label set for each comparison image according to the corresponding attention weight of the comparison image and the corresponding label vector of the comparison image, wherein the corresponding label score of each label represents the possibility that the label is a correct label of the new image, Among them, the neural network attention mechanism is trained in the following way: Sampling a label set from a distribution over multiple possible label sets; Using the sampled label set to sample a training comparison set and a batch of input images; and The neural network attention mechanism is trained to minimize the error in predicting labels for a batch of input images conditioned on a training comparison set.

Images