Information processing device, classification device, information processing method, classification method, and program

Abstract

To reduce the effort required to improve the accuracy of image classification. [Solution] An information processing device comprising: a control unit that trains a domain classifier that infers the domain to which data having a numerical sequence as a feature belongs, based on an input numerical sequence, along with the likelihood of inference; in the training, the domain classifier infers the domain to which the training target numerical sequence belongs and the likelihood of inference based on the feature of the training target numerical sequence, which is a numerical sequence that includes a domain numerical sequence that is a numerical sequence that indicates a domain; the domain classifier and the domain numerical sequence generation model, which is a mathematical model that generates the domain numerical sequence, are updated to increase the likelihood based on the likelihood; the training target numerical sequence also includes a classification numerical sequence, which is a numerical sequence that indicates text related to an image and indicates the classification destination of the image.

Description

[Technical Field]

[0001] This invention relates to an information processing device, a classification device, an information processing method, a classification method, and a program. [Background technology]

[0002] The most common method for training image classifiers using machine learning is supervised learning. In supervised learning, a large number of pairs of image data and their corresponding ground truth labels are prepared, and the image classifier is trained to predict the correct label when given image data as input. These large numbers of data and their corresponding ground truth labels are called supervised training data.

[0003] Image classifiers trained using supervised learning are often known to suffer significant performance degradation due to a problem called domain shift. A domain is a term that describes the environment or region from which data is acquired, and domain shift refers to a situation where the training domain and the target domain are different. A concrete example of domain shift is when the training domain is image data generated using computer graphics, and the target domain is image data acquired from a real-world environment. In this case, an image classifier trained using computer graphics may not be able to achieve sufficient classification accuracy for real-world image data due to differences in how computer graphics and real-world images appear.

[0004] One way to address the domain shift problem is through a method called "domain generalization." In practice, since the target domain cannot be known during the training of an image classifier, domain generalization aims to "build an image classifier that can achieve sufficient classification accuracy for any domain." To achieve this goal, domain generalization involves collecting supervised training data from multiple different domains and using that data to train the image classifier.

[0005] As a specific domain generalization technique, Non-Patent Document 1 discloses a technique that realizes high classification accuracy for any domain by learning domain-invariant feature quantities between data in a plurality of different domains through adversarial learning.

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0007] The techniques disclosed in Non-Patent Document 1 and similar techniques can achieve the purpose of domain generalization, but there is a problem in that teacher-labeled training data for multiple domains must be collected. In order to prepare teacher-labeled training data for one domain, it is necessary to collect a large amount of data and assign correct labels to them, which requires a great deal of cost. Furthermore, preparing teacher-labeled training data for multiple domains is a factor that significantly increases the data collection cost and greatly impairs the usefulness of the technology.

[0008] Non-Patent Document 2 and Non-Patent Document 3 disclose learning methods for image classifiers that do not require teacher-labeled training data. Both techniques utilize a large-scale pre-trained image-language model called CLIP (Contrastive Language-Image Pre-training).

[0009] First, an explanation of CLIP will be given. CLIP is a model composed of an image feature extractor that extracts features of images and a text feature extractor that extracts features of text. In the training of CLIP, for paired data of images and corresponding texts collected in large quantities from the Internet, the image feature extractor and the text feature extractor are trained so that the similarity between the image feature amount and the text feature amount obtained from the paired data becomes high. By using the thus obtained image feature extractor and text feature extractor, any image classification problem can be solved. For example, when it is desired to classify whether an animal existing in a certain image is a "dog" or a "cat", by calculating which of the feature amounts of the text "This is a dog." and the text "This is a cat." has a higher similarity with the feature amount of the target image, it is possible to classify whether it is a "dog" or a "cat".

[0010] Since many pre-trained models of CLIP are already provided, it is a technology that does not require effort for the learning of CLIP itself.

[0011] Non-patent document 2 discloses a technique for achieving domain generalization by preparing texts from various domains and integrating and using text features from multiple domains. These texts from various domains include, for example, in the image classification of "dogs" and "cats," multiple texts created by adding domain-specific descriptions to the same image, such as "This is a computer-generated image of a dog" or "This is a photograph of a dog." By integrating the features of these multiple domain texts, it becomes possible to classify "dogs" with high accuracy, regardless of whether the input image was obtained from the computer-generated image domain or the photographic domain.

[0012] In this context, integration refers to abstraction. Abstraction is the process of obtaining commonalities between multiple things. Thus, the technology disclosed in Non-Patent Document 2 uses texts from various domains, but it is a technology that uses them in an abstracted form.

[0013] The technology disclosed in Non-Patent Document 3 is a technology for classifying text without using image data, and does not involve manually creating text. Non-Patent Document 3 discloses a generation method that constructs text representing domains such as "CG images" and "photographs" through self-supervised learning. Furthermore, Non-Patent Document 3 discloses a technology that trains an image classifier using supervised training data from multiple domains by treating the text features of the text generated from multiple domains in this way as features obtained from images from multiple domains.

[0014] The technologies disclosed in Non-Patent Documents 2 and 3 achieve the goal of domain generalization without using supervised training data for multiple domains, but both use the same configuration for the domain of the input image and do not achieve image classification according to the domain of the input image. For example, in the problem of classifying images of "dogs" and "cats" in "photographs," the classification result obtained by selecting only the text of "photographs" is a more accurate classification result that is adapted to the domain of the input image, compared to using a feature that integrates the text of "CG images" and "photographs," or using an image classifier trained on the text of "CG images" and "photographs."

[0015] In summary, the technologies disclosed in Non-Patent Documents 2 and 3 do not necessarily have high classification accuracy, and on the other hand, in order to improve classification accuracy, it was necessary to use techniques such as those described in Non-Patent Document 1, which involve manually collecting supervised training data. Therefore, with the technologies disclosed so far, in order to improve the accuracy of image classification, it is necessary to manually collect supervised training data, and the effort required to improve classification accuracy can be considerable.

[0016] In view of the above circumstances, the present invention aims to provide a technology that reduces the effort required to improve the accuracy of image classification. [Means for solving the problem]

[0017] One aspect of the present invention is an information processing device comprising: a domain classifier that infers the domain to which data having a numerical sequence as a feature belongs, based on an input numerical sequence, along with the likelihood of inference; a control unit that trains a domain classifier; and in the training, the domain classifier infers the domain to which the training target numerical sequence belongs and the likelihood of inference based on the feature of the training target numerical sequence, which is a numerical sequence that includes a domain numerical sequence that indicates a domain; and the domain classifier and the domain numerical sequence generation model, which is a mathematical model that generates the domain numerical sequence, are updated to increase the likelihood based on the likelihood, wherein the training target numerical sequence also includes a classification numerical sequence, which is a numerical sequence that indicates text related to an image and indicates the classification destination of the image.

[0018] One aspect of the present invention is a classification device comprising a control unit that trains a domain classifier which infers the domain to which data having a numerical sequence as a feature belongs, based on an input numerical sequence, along with the likelihood of inference, wherein in the training, the domain classifier infers the domain to which the training target numerical sequence belongs and the likelihood of inference based on the feature of the training target numerical sequence which is a numerical sequence that includes a domain numerical sequence which is a numerical sequence that indicates a domain, and the domain classifier and the domain numerical sequence generation model which is a mathematical model that generates the domain numerical sequence are updated to increase the likelihood based on the likelihood, wherein the training target numerical sequence is a numerical sequence that indicates text related to an image and also includes a classification numerical sequence which is a numerical sequence that indicates the classification destination of the image, and the classification unit which uses a trained training target numerical sequence obtained by an information processing device and a trained domain classifier to classify image data to be classified, wherein the trained training target numerical sequence is the training target numerical sequence at the time when a predetermined learning termination condition, which is a condition for the end of the training, is met, and the trained domain classifier is the domain classifier at the time when the learning termination condition is met.

[0019] One aspect of the present invention is an information processing method executed by an information processing device, comprising: a learning step in which the control unit performs the learning, the learning step comprising: a domain classifier that infers the domain to which data having a numerical sequence as a feature belongs, based on an input numerical sequence, along with the likelihood of inference, the learning step comprising: the domain classifier infers the domain to which the training target numerical sequence belongs and the likelihood of inference, based on the feature of the training target numerical sequence which is a numerical sequence that includes a domain numerical sequence which is a numerical sequence that indicates a domain, the domain classifier infers the domain to which the training target numerical sequence belongs and the likelihood of inference, the domain classifier and the domain numerical sequence generation model which is a mathematical model that generates the domain numerical sequence, are updated to increase the likelihood, the training target numerical sequence is a numerical sequence that indicates text relating to an image and also includes a classification numerical sequence which is a numerical sequence that indicates the classification destination of the image, the training target numerical sequence is a numerical sequence that indicates text relating to an image, the training step comprising: a learning step in which the control unit performs the learning, the learning step comprising: a domain classifier that infers the domain to which data having a numerical sequence as a feature belongs, based on an input numerical sequence, the training target numerical sequence which is a numerical sequence that includes a domain numerical sequence which is a numerical sequence that indicates the classification destination of the image, the training target numerical sequence is a numerical sequence that indicates text relating to an image, the training target numerical sequence is a numerical sequence that indicates the classification destination of the image, the training target numerical sequence is a learning step in which the control unit performs the learning, the learning step comprising:

[0020] One aspect of the present invention comprises a domain classifier that infers the domain to which data having a numerical sequence as a feature belongs, based on an input numerical sequence, along with the likelihood of inference, and a control unit that trains the domain classifier, inferring the domain to which the training target numerical sequence belongs and the likelihood of inference based on the feature of the training target numerical sequence, which is a numerical sequence containing a domain numerical sequence that represents a domain, and updating the domain classifier and a domain numerical sequence generation model, which is a mathematical model that generates the domain numerical sequence, to improve the likelihood, and the training target numerical sequence is an image A classification method performed by a classification device, comprising: a classification unit that performs classification of image data to be classified using a learned training target numerical sequence obtained by an information processing device, which is a numerical sequence representing text relating to the image and also includes a classification numerical sequence that is a numerical sequence indicating the classification destination of the image, wherein the learned training target numerical sequence is the training target numerical sequence at the time when a predetermined learning termination condition, which is a condition relating to the end of the learning, is met, and the learned domain classifier is the domain classifier at the time when the learning termination condition is met, the classification method having a classification step in which the classification unit performs the classification.

[0021] One aspect of the present invention is a program for causing a computer to function as the above-mentioned information processing device.

[0022] One aspect of the present invention is a program for causing a computer to function as the above-mentioned classification device. [Effects of the Invention]

[0023] This invention makes it possible to improve the accuracy of image classification. [Brief explanation of the drawing]

[0024] [Figure 1] An explanatory diagram illustrating the information processing system of an embodiment. [Figure 2] A diagram showing an example of the hardware configuration of an information processing device in an embodiment. [Figure 3]A flowchart showing an example of the processing flow executed by the information processing device in the embodiment. [Figure 4] A diagram showing an example of the hardware configuration of the separation device in the embodiment. [Figure 5] A flowchart showing an example of the processing flow performed by the classification device in the embodiment. [Figure 6] A diagram showing a first example of the configuration of the control unit included in the information processing device in a modified example. [Figure 7] A diagram showing a second example of the configuration of the control unit included in the information processing device in a modified example. [Figure 8] A flowchart showing an example of the processing flow executed by the control unit of the information processing device in a modified example. [Figure 9] A diagram showing a first example of the configuration of the control unit included in a classification device in a modified example. [Figure 10] A diagram showing a second example of the configuration of the control unit included in the classification device in a modified example. [Figure 11] A flowchart showing an example of the processing flow performed by the control unit of the classification device in a modified example. [Modes for carrying out the invention]

[0025] (Embodiment) Figure 1 is an explanatory diagram illustrating an information processing system 100 according to an embodiment. The information processing system 100 comprises an information processing device 1 and a classification device 2. The classification device 2 is a device that performs image classification and uses the results obtained by the information processing device 1 for image classification.

[0026] <Information Processing Device 1> The information processing device 1 includes a control unit 11 which is equipped with a processor 91 such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Network Processing Unit) connected by a bus, and memory 92, and executes a program.

[0027] The control unit 11 performs, for example, a learning process. The learning process is the process of training a domain classifier. The domain classifier is a mathematical model that, based on an input sequence of numerical values, infers the domain to which the data whose features are that sequence of numerical values ​​belong, along with the probability of the inference. The domain classifier is, for example, a neural network. The trained domain classifier is used by the classification device 2 to classify images.

[0028] A trained mathematical model refers to the mathematical model at the point in time when a predetermined condition for the termination of training in the training process (hereinafter referred to as the "training termination condition") is met. The training termination condition may be, for example, a condition that a predetermined number of training sessions have been performed, or a condition that the accuracy of the estimation by the domain classifier is greater than or equal to a predetermined accuracy.

[0029] In the above learning process, the domain classifier infers the domain to which the training numerical sequence belongs and the likelihood of the inference, based on the features of the training numerical sequence (hereinafter referred to as the "training numerical sequence") which contains the domain numerical sequence that represents the domain. Based on this likelihood, the domain classifier and the mathematical model that generates the domain numerical sequence (hereinafter referred to as the "domain numerical sequence generation model") are updated. The update is performed according to predetermined rules to improve the likelihood obtained by the domain classifier. The features of the training numerical sequence are also numerical sequences.

[0030] Updating a mathematical model means updating the values ​​of the parameters in that model.

[0031] In fact, the training numerical sequence includes not only a domain numerical sequence, which indicates the domain, but also a classification numerical sequence. The classification numerical sequence is a numerical sequence that represents text related to an image, and indicates the classification target of that image. In other words, the classification numerical sequence is text data of the text indicating the classification target of the image.

[0032] A domain numerical sequence generation model is a mathematical model that generates domain numerical sequences for each domain. For example, a domain numerical sequence generation model takes an identifier representing a domain as input and generates domain numerical sequences. Therefore, in this case, the domain numerical sequence generation model is a mathematical model that receives a domain identifier as input and outputs domain numerical sequences for each domain.

[0033] <Relationship between training numerical sequences and text> The numerical data used for training is equivalent to text data. The intention behind the expression "equivalent" is that it is closer to text data than image data, but not necessarily text data itself. Therefore, the numerical data used for training may be text data, but is not necessarily text data itself.

[0034] For the sake of simplicity, let's first explain the case where the numerical sequence to be trained is text data. In this case, the text represented by the numerical sequence indicates the classification target of the image and the domain to which the image belongs. In other words, the text represented by the numerical sequence in this case indicates the classification target of the image and how the image was obtained.

[0035] The text represented by such a numerical sequence of data to be trained might be, for example, "This is an image of a dog obtained through image synthesis." Here, the word "image synthesis" indicates the domain, and "dog" indicates the classification target of the image. Note that in the training process, "image" here does not necessarily refer to a specific image data. In the training process, if one of the numerical sequences of data to be trained represents the content "This is an image of a dog obtained through image synthesis," it simply needs to exist as a numerical sequence of data to represent the text "This is an image of a dog obtained through image synthesis."

[0036] A domain classifier infers the domain to which a numerical sequence of data belongs based on its features. When the text representing the numerical sequence is something like "This is an image of a dog obtained by image synthesis," the domain classifier is expected to infer with high confidence that the domain is "image synthesis." Therefore, during the learning process, if the domain classifier obtains an inference result that the probability of it being "image synthesis" is less than a predetermined probability, the parameters of this domain classifier are updated so that it can infer with higher confidence that it belongs to the "image synthesis" domain.

[0037] Incidentally, domain inference by a domain classifier also depends on the input data. For example, if the difference between the numerical sequence indicating that an image belongs to the "image synthesis" domain and the numerical sequence indicating that the image belongs to the "shooting result" domain is small, the domain classifier is unlikely to be able to infer the domain properly. Therefore, it is desirable that the numerical sequences indicating the domains themselves be updated in a way that improves the accuracy of inference by the domain classifier. Specifically, it is desirable that the numerical sequences indicating the domains themselves be updated to increase the difference between the numerical sequence indicating that an image belongs to the first domain and the numerical sequence indicating that the image belongs to a second domain that is different from the first domain. Therefore, in the training process, the domain numerical sequences themselves that are included in the numerical sequences being trained are also updated.

[0038] Updating a domain numeric sequence means updating the domain numeric sequence generation model. Updating the domain numeric sequence generation model changes its output. Therefore, even if the domain identifier input to the domain numeric sequence generation model is the same before and after the update, the output of the domain numeric sequence generation model will change.

[0039] In the learning process, the domain numerical sequence generation model is updated so as to increase the difference between the numerical sequences output by the model, namely, the numerical sequence indicating that the image belongs to the first domain and the numerical sequence indicating that the image belongs to the second domain. Therefore, at the time when the learning end condition is satisfied, the domain numerical sequence has a greater difference due to the domain difference than before the update. Note that the domain numerical sequence at the time when the learning end condition is satisfied is the domain numerical sequence output by the learned domain numerical sequence generation model. If the difference due to the domain difference between the domain numerical sequences is large, the difference between the training target numerical sequences is also large.

[0040] As described above, the training target numerical sequence is not necessarily the text data itself. More specifically, the domain numerical sequence included in the training target numerical sequence is not necessarily the text data itself. As a result of the domain numerical sequence generation model being updated by learning, the numerical sequence output by the domain numerical sequence generation model may be a numerical sequence that is not partially or entirely converted into text data. Text data is merely data that is easy for people to understand. In some cases, the inference accuracy of the domain classifier may be higher for data that is difficult for people to understand. Therefore, the training target numerical sequence does not necessarily have to be the text data itself.

[0041] Note that FIG. 1 shows a diagram for explaining the learning process. In the example of FIG. 1, the number of domains is N (N is an integer of 2 or more), and the domain numerical sequence generation model generates N domain numerical sequences from the domain numerical sequence indicating domain D 1 to the numerical sequence indicating domain D N Each of the domains from domain D 1 to domain D N is a different domain. In the example of FIG. 1, "D 1 prompt" means the domain numerical sequence indicating domain D 1 , and "D N prompt" means the domain numerical sequence indicating domain D N .

[0042] In the example of FIG. 1, "D 1Prompt + Classification destination 1", "D 1 Prompt + Classification Destination 2", "D N Prompt + Classification Destination 1" and "D N Prompt + Classification Target 2" are both examples of numerical sequences to be trained. 1 Prompt + Classification Destination 1" is Domain D 1 This refers to a training target numerical sequence that includes a domain numerical sequence indicating and a classification numerical sequence indicating classification target 1 as the classification target. 1 Prompt + Classification Destination 2" is Domain D 1 This refers to a training target numerical sequence that includes a domain numerical sequence indicating and a classification numerical sequence indicating classification target 2.

[0043] “D N Prompt + Classification Destination 1" is Domain D N This refers to a training target numerical sequence that includes a domain numerical sequence indicating and a classification numerical sequence indicating classification target 1 as the classification target. N Prompt + Classification Destination 2" is Domain D N This refers to a training target numerical sequence that includes a domain numerical sequence indicating a specific domain and a classification numerical sequence indicating classification target 2. In this way, during the learning process, a training target numerical sequence is generated for each domain numerical sequence, with a different classification numerical sequence indicating a different classification target.

[0044] In the example in Figure 1, the numerical sequence to be trained is input to an encoder, and the output of the encoder is output to a domain classifier. Then, based on the inference results from the domain classifier, the domain classifier and the domain numerical sequence generation model are updated.

[0045] <Classification device 2> The classification device 2 includes a control unit 21 equipped with a processor 93 such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Network Processing Unit) connected by a bus, and memory 94, and executes a program.

[0046] The control unit 21 performs classification processing. The classification processing uses the trained training target numerical sequence and the trained domain classifier to classify the image to be classified. The trained training target numerical sequence is the training target numerical sequence at the time when the above-mentioned training termination condition is met. Therefore, the trained training target numerical sequence is the training target numerical sequence that includes the domain numerical sequence at the time when the training termination condition is met as the domain numerical sequence. The trained domain classifier is the domain classifier at the time when the above-mentioned training termination condition is met.

[0047] In the classification process, the image data features of the image to be classified are input to a pre-trained domain classifier. The image data features of the image to be classified are also numerical sequences. In the training process, the domain classifier was input with the features of the numerical sequence to be trained, but in the classification process, it is input with the features of the image data. However, regardless of whether the source is the numerical sequence to be trained or the image data, the features input to the domain classifier are numerical sequences, so the domain classifier can perform inference based on the input numerical sequence.

[0048] In the classification process, the classification target of the image data to be classified and the probability of the inference are inferred based on the pre-trained domain numerical sequence and the inference results of a pre-trained domain classifier based on the features of the image data to be classified. In the classification process, any method of inference is acceptable as long as the classification target of the image data to be classified and the probability of the inference are inferred based on the pre-trained domain numerical sequence and the inference results of a pre-trained domain classifier based on the features of the image data to be classified; for example, CLIP technology may be used. A specific example will be introduced in the modified examples.

[0049] <An example of the hardware configuration of the information processing device 1> Figure 2 shows an example of the hardware configuration of the information processing device 1 in the embodiment. The information processing device 1 includes a control unit 11 that executes a program, and through the execution of the program, functions as a device comprising the control unit 11, an interface unit 12 equipped with a communication interface 121, and a storage unit 13.

[0050] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. By executing the program stored in the memory 92, the information processing device 1 functions as a device comprising a control unit 11, an interface unit 12, and a storage unit 13.

[0051] The control unit 11 controls the operation of each functional unit of the information processing device 1. As described above, the control unit 11 performs, for example, a learning process. The control unit 11 also retrieves, for example, information stored in the memory unit 13. Specifically, the process of retrieving information stored in the memory unit 13 is a read operation.

[0052] The interface unit 12 includes a communication interface for connecting the information processing device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless means.

[0053] The external device may be, for example, a classification device 2. In such a case, the classification device 2 can perform classification using the learning results obtained by the information processing device 1 through communication via the interface unit 12.

[0054] The interface unit 12 may include input devices such as a mouse, keyboard, or touch panel. The interface unit 12 may also be configured as an interface connecting these input devices to the information processing device 1. In this way, the input devices of the interface unit 12 receive various types of information to the information processing device 1 via wired or wireless connections. Note that the information does not necessarily have to be input to the communication interface of the interface unit 12; it may also be input to the input devices of the interface unit 12.

[0055] The interface unit 12 outputs various types of information, for example. The interface unit 12 includes, for example, a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, as well as a speaker. The interface unit 12 may be configured as an interface for connecting these display devices or speakers to the information processing device 1. Therefore, the interface unit 12 may output information input to its input device as an image or sound, for example.

[0056] The storage unit 13 is configured using a computer-readable recording medium such as a magnetic hard disk drive or a semiconductor memory device. The storage unit 13 stores various information related to the information processing device 1. Therefore, the storage unit 13 stores, for example, the parameter values ​​of the domain numerical sequence generation model and the parameter values ​​of the domain classifier. The storage unit 13 stores, for example, various information generated by the operation of the control unit 11. The storage unit 13 may reside, for example, on the cloud. The storage unit 13 stores, for example, information indicating the classification destination in advance. Therefore, for example, if the classification destination is dogs and cats, it stores information indicating that the classification destination is either dogs or cats.

[0057] Figure 3 is a flowchart showing an example of the processing flow performed by the information processing device 1 in the embodiment. The control unit 11 performs the learning process (step S101). The execution of the learning process yields a trained domain classifier and a trained numerical sequence to be trained.

[0058] <Example of hardware configuration for classification device 2> Figure 4 shows an example of the hardware configuration of the classification device 2 in the embodiment. The classification device 2 includes a control unit 21 which has a processor 93 such as a CPU, GPU, or NPU connected by a bus and memory 94, and executes a program. The classification device 2 functions as a device comprising the control unit 21, interface unit 22, and storage unit 23 by executing the program.

[0059] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. By executing the program stored in the memory 94, the processor 93 functions as a device comprising a control unit 21, an interface unit 22, and a storage unit 23.

[0060] The control unit 21 controls the operation of each functional unit of the classification device 2. For example, the control unit 21 performs a classification process. For example, the control unit 21 retrieves information stored in the storage unit 23. Specifically, the process of retrieving information stored in the storage unit 23 is a read operation.

[0061] The interface unit 22 includes a communication interface for connecting the classification device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless connection.

[0062] The external device is, for example, the device that transmits the object to be classified in the classification process. The object to be classified is image data. The interface unit 22 acquires the object to be classified in the classification process by communicating with such a device that transmits the object to be classified.

[0063] The external device may be, for example, an information processing device 1. In such a case, the classification device 2 can utilize the learning results obtained by the information processing device 1 through communication via the interface unit 22.

[0064] The interface unit 22 may include input devices such as a mouse, keyboard, or touch panel. The interface unit 22 may also be configured as an interface connecting these input devices to the classification device 2. In this way, the input devices of the interface unit 22 receive various types of information to the classification device 2 via wired or wireless connections. Note that the information does not necessarily have to be input to the communication interface of the interface unit 22; it may also be input to the input devices of the interface unit 22.

[0065] The interface unit 22 outputs various types of information, for example. The interface unit 22 includes, for example, a display device such as a CRT display, liquid crystal display, or organic EL display, and a speaker. The interface unit 22 may be configured as an interface for connecting these display devices or speakers to the classification device 2. Therefore, the interface unit 22 may output information input to its input device as an image or sound, for example.

[0066] The storage unit 23 is configured using a computer-readable recording medium such as a magnetic hard disk drive or a semiconductor memory device. The storage unit 23 stores various information related to the classification device 2. Therefore, the storage unit 23 may store, for example, the parameter values ​​of a trained domain numerical sequence generation model or the parameter values ​​of a trained domain classifier. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 may reside, for example, on the cloud.

[0067] Figure 5 is a flowchart showing an example of the processing flow performed by the classification device 2 in the embodiment. The control unit 21 acquires the object to be classified (step S201). The control unit 21 performs the classification process (step S202). In the classification process, the trained domain classifier obtained through the learning process and the trained numerical sequence of the target to be trained are used. By performing the classification process, it is inferred which classification destination the object to be classified into.

[0068] <Effects of the learning process> As described above, the classification process uses a pre-trained domain classifier obtained through the learning process and a pre-trained numerical sequence of the target to perform the classification of the target. In the learning process, the data input to the domain classifier is generated by the domain numerical sequence generation model. Therefore, the data input to the domain classifier in the learning process is not manually generated data. Furthermore, since the domain numerical sequence generation model itself is updated during the learning process, the data input to the domain classifier is updated by the update of the domain numerical sequence generation model. For this reason, there is no need to manually prepare a large amount of data.

[0069] Furthermore, updating the domain numerical sequence generation model increases the differences in the domains of the domain numerical sequences it outputs. For example, in the initial state of the domain numerical sequence generation model, even if the model outputs N domain numerical sequences (where N is an integer greater than or equal to 2) that belong to different domains, if the differences between them are very small, it effectively means that domain numerical sequences of the same domain are being generated. However, if the differences between the N domain numerical sequences increase through the domain learning process, it no longer means that domain numerical sequences of the same domain are being generated, but rather that N domain numerical sequences of different domains are being generated. Therefore, the learning process trains the domain classifier using multiple numerical sequences of different domains. As a result, domain-specific image classification is achieved. Consequently, the accuracy of image classification can be improved by performing the learning process.

[0070] In this way, the effort required to improve the accuracy of image classification is reduced by performing the learning process.

[0071] The information processing device 1 configured in this way performs a learning process. Therefore, as described in <Effects of the Learning Process>, the information processing device 1 can reduce the effort required to improve the accuracy of image classification.

[0072] Furthermore, the classification device 2, configured in this way, performs estimation using the learning results obtained by the information processing device 1. Therefore, the classification device 2 can reduce the effort required to improve the accuracy of image classification.

[0073] Furthermore, the information processing system 100 configured in this way includes an information processing device 1. Therefore, the information processing system 100 can reduce the effort required to improve the accuracy of image classification.

[0074] (modified version) Incidentally, the control unit 11 may have the configuration shown in Figure 6 below, for example. Figure 6 is a diagram showing a first example of the configuration of the control unit 11 in a modified example. In the example of Figure 6, the control unit 11 includes a numerical sequence generation unit 111, a first encoder 112, a domain classifier 113, a first loss acquisition unit 114, and a learning control unit 115.

[0075] The numerical sequence generation unit 111 executes a domain numerical sequence generation model and obtains a domain numerical sequence for each domain. Furthermore, the numerical sequence generation unit 111 obtains a classification numerical sequence for each candidate classification target stored in the storage unit 13. Furthermore, the numerical sequence generation unit 111 combines each classification numerical sequence for each obtained domain numerical sequence. The result of each combination is a numerical sequence to be trained. Therefore, for example, if the number of domains is P (P is an integer greater than or equal to 2) and the number of candidate classification targets is Q (Q is an integer greater than or equal to 2), the numerical sequence generation unit 111 obtains P × Q numerical sequences to be trained.

[0076] The first encoder 112 is an encoder that obtains feature quantities from an input numerical sequence. In the example in Figure 6, each training target numerical sequence obtained by the numerical sequence generation unit 111 is input to the first encoder 112. Therefore, in the example in Figure 6, the first encoder 112 is an encoder that obtains feature quantities from each training target numerical sequence obtained by the numerical sequence generation unit 111. The domain classifier 113 is a domain classifier. The encoding result obtained by the first encoder 112 is input to the domain classifier 113.

[0077] The first loss acquisition unit 114 obtains a first predetermined loss based on the inference result of the domain classifier. Here, the loss is, for example, the loss that shows the difference between the inference result of the domain classifier and the domain indicated by the domain numerical sequence included in the training numerical sequence whose features are the numerical sequence input to the domain classifier.

[0078] The learning control unit 115 updates the domain numerical sequence generation model and the domain classifier according to a predetermined rule based on the loss obtained by the first loss acquisition unit 114 (hereinafter referred to as the "first loss").

[0079] Figure 7 shows a second example of the configuration of the control unit 11 in a modified example. In addition to the functional unit shown in Figure 6, the control unit 11 further includes a reference text generation unit 116 and a second loss acquisition unit 117. The reference text generation unit 116 executes a reference text generation model. The reference text generation model is a mathematical model that generates data representing natural language that includes information indicating the classification target (hereinafter referred to as "reference text data"). Therefore, the reference text generation unit 116 generates reference text data. Since the reference text data represents natural language, it is a type of text data. Text data is a sequence of numbers.

[0080] More specifically, the reference text generation unit 116 generates reference text data for each training target numerical sequence generated by the numerical sequence generation unit 111, which includes information indicating the classification destination indicated by the classification numerical sequence contained in each training target numerical sequence.

[0081] In the example shown in Figure 7, the first encoder 112 receives not only the numerical sequences to be trained obtained by the numerical sequence generation unit 111, but also the reference text data obtained by the reference text generation unit 116. Therefore, in the example shown in Figure 7, the first encoder 112 also obtains the feature quantities of the reference text data.

[0082] The second loss acquisition unit 117 obtains a second predetermined loss based on the feature quantities of the training target numerical sequence obtained by the first encoder 112 and the feature quantities of the reference text data corresponding to the training target numerical sequence (hereinafter referred to as "corresponding reference text data"). The reference text data corresponding to the training target numerical sequence (i.e., corresponding reference text data) means reference text data that includes information indicating the classification target indicated by the classification numerical sequence contained in the training target numerical sequence.

[0083] The loss under the second predetermined definition is different from the loss under the first predetermined definition. The loss under the second predetermined definition is the loss that shows the difference between the features of the training numerical sequence and the features of the corresponding reference text data.

[0084] In the example shown in Figure 7, the learning control unit 115 updates the domain numerical sequence generation model and the domain classifier according to a predetermined rule, based not only on the loss obtained by the first loss acquisition unit 114, but also on the loss obtained by the second loss acquisition unit 117 (hereinafter referred to as the "second loss"). In this update, the update is performed in a way that reduces the second loss.

[0085] <<The effects of updating to reduce the second loss>> The second loss represents the difference between the features of the training numerical sequence and the features of the corresponding reference text data. Therefore, by updating the domain classifier and the domain numerical sequence generation model to reduce the second loss, the domain numerical sequence generation model becomes a mathematical model that outputs the training numerical sequence closer to the text data. The closer the training numerical sequence is to the text data, the higher the accuracy of the inference of mathematical models that infer the correspondence between text data and image data, such as CLIP. Thus, updating to reduce the second loss further improves the accuracy of classification in the classification process.

[0086] Figure 8 shows a flowchart illustrating an example of the processing flow performed by the control unit 11 in the example shown in Figure 7. Figure 8 is a flowchart illustrating an example of the processing flow performed by the control unit 11 in a modified example. Figure 8 can also be said to show an example of the learning process.

[0087] The numerical sequence generation unit 111 generates numerical sequences to be trained, and for each numerical sequence to be trained, the reference text generation unit 116 generates corresponding reference text data (step S301). Next, the first encoder 112 obtains the feature quantities of the numerical sequences to be trained obtained in step S301 and the feature quantities of the corresponding reference text data (step S302). Next, the feature quantities of the numerical sequences to be trained obtained in step S302 are input to the domain classifier 113, and the domain classifier of the domain classifier 113 performs inference based on the input feature quantities of the numerical sequences to be trained (step S303). Next, the second loss acquisition unit 117 obtains a second loss based on the result of step S302 (step S304).

[0088] Next, the first loss acquisition unit 114 obtains a first loss based on the result of step S303 (step S305). Next, the learning control unit 115 updates the domain numerical sequence generation model and the domain classifier based on the second loss obtained in step S304 and the first loss obtained in step S305 (step S306). Next, the learning control unit 115 determines whether or not the learning termination condition has been met (step S307). If the learning termination condition is met (step S307: YES), the process ends. On the other hand, if the learning termination condition is not met (step S307: NO), the process returns to step S301.

[0089] The processing in step S304 may be executed at any time as long as it is performed after the processing in step S303 and before the processing in step S306. Similarly, the processing in step S305 may be executed at any time as long as it is performed after the processing in step S303 and before the processing in step S306. Therefore, for example, the second loss may be acquired after the first loss, or the first and second losses may be acquired in parallel.

[0090] Incidentally, the control unit 21 may have the configuration shown in Figure 9 below, for example. Figure 9 is a diagram showing a first example of the configuration of the control unit 21 in a modified example. In the example of Figure 9, the control unit 21 includes a trained training target numerical sequence acquisition unit 211, a second encoder 212, a third encoder 213, a trained domain classifier 214, and an inference unit 215.

[0091] The trained numerical sequence acquisition unit 211 acquires the trained numerical sequence. The second encoder 212 is an encoder that obtains the features of the trained numerical sequence. The third encoder is an encoder that obtains the features of the image data to be classified.

[0092] The trained domain classifier 214 is the trained domain classifier 113. The trained domain classifier 214 is input with the features obtained by the third encoder 213 (i.e., the features of the image data to be classified), and the trained domain classifier 214 performs inference based on those features.

[0093] The inference unit 215 performs inference processing. The inference processing is the process of inferring the classification destination of the image data to be classified based on the features obtained by the second encoder 212, the features obtained by the third encoder 213, and the inference results obtained by the trained domain classifier 214. For example, the inference processing may be the process of inferring the classification destination of the image data to be classified using CLIP based on the features obtained by the second encoder 212, the features obtained by the third encoder 213, and the inference results obtained by the trained domain classifier 214.

[0094] Figure 10 shows a second example of the configuration of the control unit 21 in a modified example. In addition to the functional unit shown in Figure 9, the control unit 21 further includes an update control unit 216.

[0095] The update control unit 216 performs, for example, an update process. The update process updates the trained domain classifier based on the results of the inference process. In other words, the update process further updates the trained domain classifier. In the update process, the trained domain classifier is updated based on the results of the inference process to improve the accuracy of the inference.

[0096] Figure 11 shows a flowchart illustrating an example of the processing flow performed by the control unit 21 in the example shown in Figure 10. Figure 11 is a flowchart illustrating an example of the processing flow performed by the control unit 21 in a modified example. Figure 11 can also be said to show an example of classification processing.

[0097] The trained numerical sequence acquisition unit 211 acquires the trained numerical sequence (step S401). Next, the second encoder 212 acquires the feature quantities of the trained numerical sequence (step S402). The image data to be classified is input to the control unit 21 (step S403). Next, the third encoder 213 acquires the feature quantities of the image data input in step S403 (step S404).

[0098] Next, the features obtained in step S404 are input to the trained domain classifier 214, and the trained domain classifier 214 performs inference based on the input features (step S405). Next, the inference unit 215 performs inference processing (step S406). In this inference processing, the classification destination of the image data to be classified is inferred based on the features obtained in step S402, the features obtained in step S404, and the result of the inference in step S405.

[0099] Next, the update control unit 216 determines whether the branching condition is met based on the inference result of the inference process (step S407). The branching condition can be any predetermined condition relating to the determination of whether to execute the update process. For example, it may be a condition that the update process has been performed a predetermined number of times, a condition that the change in the trained domain classifier 214 due to the update process is less than a predetermined change, or a condition that the probability of the inference in the inference process is greater than or equal to a predetermined probability. Whether the branching condition is met may be determined, for example, using a loss with a predetermined definition.

[0100] If the branching condition is not met (step S407: NO), the update process is executed (step S408). After step S408, the process returns to step S405.

[0101] On the other hand, if the branching condition is met (step S407: YES), the result of the inference obtained in step S406 is output to a predetermined output destination (step S409). The predetermined output destination may be, for example, the storage unit 23, an output device such as a monitor provided by the interface unit 22, or a predetermined external device connected via the interface unit 22.

[0102] The process in step S402 may be executed at any time, including in parallel with other processes, as long as it is executed after the process in step S401 and before the process in step S406. The process in step S401 may be executed at any time, including in parallel with other processes, as long as it is executed before the process in step S402.

[0103] Furthermore, the processing in step S405 may be executed at any time, including in parallel with other processes, as long as it is executed after the processing in step S404 and before the processing in step S406. Furthermore, the processing in step S404 may be executed at any time, including in parallel with other processes, as long as it is executed after the processing in step S403 and before the processing in step S405. Furthermore, the processing in step S403 may be executed at any time, including in parallel with other processes, as long as it is executed before the processing in step S404.

[0104] <Explanation using mathematical formulas> Here, we will provide an explanation using mathematical formulas. Since a feature vector is an example of a feature, for simplicity, we will use a feature vector as an example in the following explanation.

[0105] The training numerical sequence consists of a domain numerical sequence, which indicates the domain, and a classification numerical sequence, which indicates the classification target of the image.

[0106] Domain numerical sequences are generated by a domain numerical sequence generation model. Here, the domain numerical sequence generation model is a mathematical model that outputs domain numerical sequences for selected domains out of M domains. Therefore, the domain numerical sequence D for domain m is generated. m This can be expressed by the following equation (1) using the domain numerical sequence generation model G(·;θ).

[0107]

number

[0108] θ is a parameter of the domain numerical sequence generation model, constructed to be differentiable with respect to the first and second losses, and m is an integer between 1 and M, inclusive.

[0109] The classification numerical sequence for the target y is c y In this case, the domain m and the training target numerical sequence t for the classification target y are as follows: m、y It is expressed by the following equation (2).

[0110]

number

[0111] Note that [·] represents the operation of combining a sequence of numbers into a single sequence of numbers. Also, y is an integer between 1 and C, and C is an integer representing the total number of classifications.

[0112] Hereafter, the reference text data for classification target y will be represented as $\hat{t_y}$. $\hat{t_y}$ represents the symbol shown in equation (3) below.

[0113]

number

[0114] The reference text data may consist of a set of continuous value vectors corresponding to each word. The reference text data is generated as text data of text that represents the same classification target as the numerical sequence being trained, but the method of generation can be any method as long as it is appropriate for text that represents the same classification target. For example, when generating reference text data for the classification target "dog", it may be used by converting the text data of the text "This is a dog." into a set of continuous value vectors, or it may be converted the text data of the detailed explanatory text about "dogs," "Dogs have two ears, four legs, and a tail," into a set of continuous value vectors. Furthermore, the text data of the generated text does not need to be uniquely determined for a given classification target, and may be text data of text randomly selected from a set of texts that represent a given classification target.

[0115] <<First Encoder 112>> The first encoder 112 takes text data t as input and outputs a function E that outputs the corresponding feature vector z. text This is achieved by using the following: that is, the training target numerical sequence t m、y Feature vector z m,y This can be expressed by the following equation (4).

[0116]

number

[0117] Therefore, the feature vector $\hat{z_y}$ of the reference text $\hat{t_y}$ is expressed by the following equation (5). Note that $\hat{z_y}$ represents the symbol on the left side of the following equation (5).

[0118]

number

[0119] <<Domain Classifier 113>> The domain classifier 113 uses the feature vector z of the numerical sequence to be trained.m,y Given as input, the estimated domain assignment probability vector w m、y This is achieved by using a function g with parameter θ that can output the feature vector z of the numerical sequence to be trained. m,y Estimated domain assignment probability vector w m、y This can be expressed by the following equation (6).

[0120]

number

[0121] The estimated domain assignment probability vector is a probability vector that represents the likelihood of a data set, whose features are input to the domain classifier, belonging to one of the M domains. It is an M-dimensional vector with the sum of its elements equaling 1. Therefore, the estimated domain assignment probability vector is an example of a value that indicates the accuracy of the domain classifier's inference.

[0122] <<Second loss acquisition section 117>> The second loss acquisition unit 117 acquires the feature vector z of the numerical sequence to be trained. m,y And the feature vector z of the numerical sequence being trained. m,y The loss is calculated such that the value decreases as the similarity with the feature vector $\hat{z_y}$ of the reference text data to the same classification target increases. The second loss for the set of pairs B of features of the training numerical sequence and features of the reference text data is expressed by the following equation (7).

[0123]

number

[0124] Here, the similarity function represented by equation (8) below can be any function that allows for the calculation of similarity between vectors, such as the norm of the difference between two vectors or the cosine similarity.

[0125]

number

[0126] <<First loss acquisition section 114>> The first loss acquisition unit 114 generates the estimated domain assignment probability vector w m、y However, the higher the assignment probability assigned to the corresponding domain m, the smaller the loss value calculated. Estimated domain assignment probability vector w m、y The first loss for the set B is given by equation (9) below.

[0127]

number

[0128] Here, the classification loss function represented by equation (10) below can be any function as long as it is appropriate as a classification loss function; for example, the cross-entropy function may be used. Note that classification refers to the classification of the object to be classified.

[0129]

number

[0130] <<Learning Control Unit 115>> A concrete example of the processing performed by the learning control unit 115 will be explained using mathematical formulas. The loss for updating the parameters of the domain numerical sequence generation model and the parameters of the domain classifier may be a loss expressed by, for example, the following equation (11) using a first loss and a second loss.

[0131]

number

[0132] Note λ domain This is a constant weighting coefficient.

[0133] The learning control unit 115 controls the loss L totalThe parameters of the domain numerical sequence generation model and the domain classifier are updated to minimize the coefficient. The updated parameter values ​​are recorded, for example, in the storage unit 13.

[0134] <<Second encoder 212>> The second encoder 212 processes each trained numerical sequence t m、y It takes as input and the feature vector z m、y It outputs the following. The second encoder 212 may be the same as the first encoder 112.

[0135] <<Third Encoder 213>> The third encoder 213 processes image data x (i) The third encoder 213 receives the image data x as input data. (i) Function E takes the input and outputs the corresponding feature vector $\bar{z^{(i)}}$. image This is achieved by using $\bar{z^{(i)}}$.

[0136]

number

[0137] In other words, image data x (i) The feature vector $\bar{z^{(i)}}$ is expressed by the following equation (12).

[0138]

number

[0139] <<Trained Domain Classifier 214>> The trained domain classifier 214 processes image data x (i)This can be achieved by using a function g with parameter θ that takes a feature vector $\bar{z^{(i)}}$ as input and outputs an estimated domain assignment probability vector $\bar{w^{(i)}}$. Note that $\bar{w^{(i)}}$ represents the symbol in equation (14) below.

[0140]

number

[0141] In other words, the estimated domain assignment probability vector $\bar{w^{(i)}}$ of the feature vector $\bar{z^{(i)}}$ is expressed by the following equation (15).

[0142]

number

[0143] Note that the initial value of the parameter θ may be set to the parameter value of a pre-trained domain classifier.

[0144] <<Inference section 215>> The inference unit 215 processes the feature vector z obtained from the second encoder 212. m、y The third encoder 213 obtains a feature vector $\bar{z^{(i)}}$ and the trained domain classifier 214 obtains an estimated domain assignment probability vector $\bar{w^{(i)}}$ which are then taken as input. m、y For this, the feature vector z of all domain and classification target combinations. m、y The following is input. Based on these inputs, the inference unit 215 estimates the class assignment probability p of the image data. (i) Obtain the estimated class assignment probability p of the image data. (i) For each class component y, it is expressed by the following equation (15). Note that the notation $\bar{w_m^{(i)}}$ in the following equation (16) represents the mth component of the estimated domain assignment probability vector $\bar{w^{(i)}}$.

[0145]

number

[0146]

number

[0147] The estimated class assignment probability is a probability vector representing the likelihood of which of the C classification targets the image being classified into belongs to. It is a C-dimensional vector with elements whose sum equals 1. The estimated class assignment probability is an example of a value that indicates the likelihood of the inference made by the inference unit 215. The class component y is the probability that the image data, as indicated by the estimated class assignment probability, belongs to image classification target y.

[0148] <<Update Control Unit 216>> The update control unit 216 generates the estimated class assignment probability vector p (i) The update control unit 216 receives the estimated class assignment probability vector p as input. (i) Based on this, for example, a loss represented by the following equation (17) may be obtained.

[0149]

number

[0150] In this case, there are multiple suitable loss functions represented by equation (19) below. Therefore, the loss function represented by equation (19) may be, for example, the function of equation (20) below, which calculates entropy, a value that represents the ambiguity of the inference result.

[0151]

number

[0152]

number

[0153] Furthermore, the loss function expressed by equation (19) is calculated by taking multiple input images {x (i)} i=N N The inference result {p (i)} i=1 N This may be a function represented by the following equation (21), which calculates the loss value based on mutual information.

[0154]

number

[0155] Other loss functions may be used, not limited to the above, as long as they do not utilize training information regarding the correct class of the input image.

[0156] This section describes an example where CLIP is used in the inference unit 215. When CLIP is used in the inference unit 215, for each domain numerical sequence included in the trained numerical sequence, the classification destination of the image data to be classified, conditioned on the domain indicated by that domain numerical sequence, is inferred. This is done by CLIP based on the features of the trained numerical sequence and the features of the image data to be classified. In the inference unit 215, the features of the image data to be classified are input to the trained domain classifier 214, and the domain to which the image data belongs, as inferred by the trained domain classifier 214, is inferred. As described above, CLIP inferred the classification destination for each domain. Therefore, the inference unit 215 outputs the classification destination inferred by CLIP, corresponding to the domain inferred by the trained domain classifier 214, as the classification destination of the image data.

[0157] The information processing device 1 may be implemented using multiple information processing devices connected to each other via a network. In this case, each process executed by the control unit 11 may be performed in a distributed manner by multiple information processing devices.

[0158] The classification device 2 may be implemented using multiple information processing devices connected to each other via a network. In this case, each process performed by the control unit 21 may be performed in a distributed manner by multiple information processing devices.

[0159] Furthermore, the learning process and the classification process do not necessarily have to be performed by different devices. Therefore, for example, information processing device 1 may perform not only the learning process but also the classification process, or classification device 2 may perform not only the classification process but also the learning process.

[0160] Furthermore, all or part of the functions of the information processing system 100, the information processing device 1, and the classification device 2 may be implemented using hardware such as ASICs (Application Specific Integrated Circuits), PLDs (Programmable Logic Devices), or FPGAs (Field Programmable Gate Arrays). The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The program may also be transmitted via a telecommunications line.

[0161] The control unit 21 is an example of a classification unit.

[0162] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Explanation of Symbols]

[0163] 100... Information processing system, 1... Information processing device, 2... Classification device, 11... Control unit, 12... Interface unit, 13... Memory unit, 21... Control unit, 22... Interface unit, 23... Memory unit, 111... Numerical sequence generation unit, 112... First encoder, 113... Domain classifier, 114... First loss acquisition unit, 115... Learning control unit, 116... Reference text generation unit, 117... Second loss acquisition unit, 211... Learned training target numerical sequence acquisition unit, 212... Second encoder, 213... Third encoder, 214... Learned domain classifier, 215... Inference unit, 216... Update control unit, 91... Processor, 92... Memory, 93... Processor, 94... Memory

Claims

1. A control unit that trains a domain classifier to infer the domain to which data whose features belong, based on an input numerical sequence, along with the probability of inference. Equipped with, In the learning process described above, the domain classifier infers the domain to which the training numerical sequence belongs and the likelihood of inference based on the features of the training numerical sequence, which is a numerical sequence containing a domain numerical sequence that represents a domain. Based on the likelihood, the domain classifier and the domain numerical sequence generation model, which is a mathematical model that generates the domain numerical sequence, are updated to increase the likelihood. The aforementioned training target numerical sequence also includes a classification numerical sequence, which is a numerical sequence representing text related to an image and a numerical sequence indicating the classification target of the image. Information processing device.

2. In the aforementioned learning process, the domain classifier and the domain numerical sequence generation model are updated to minimize the difference between the features of the reference text data representing natural language containing information indicating the classification target and the features of the numerical sequence to be trained. The information processing apparatus according to claim 1.

3. A domain classifier that learns a Equipped with, The aforementioned trained numerical sequence is the numerical sequence of the training target at the point when the training termination condition, which is a predetermined condition for the termination of the training, is met. The aforementioned trained domain classifier is the domain classifier at the point when the training termination condition is met. Classification device.

4. In the classification process, the classification destination of the image data to be classified is inferred, and the pre-trained domain classifier is updated to improve the accuracy of the inference. The classification device according to claim 3.

5. An information processing method executed by an information processing device, comprising: a control unit that trains a domain classifier that infers the domain to which data having a numerical sequence as a feature belongs, based on an input numerical sequence, along with the likelihood of inference; in the training, the domain classifier infers the domain to which the training target numerical sequence belongs and the likelihood of inference based on the feature of the training target numerical sequence, which is a numerical sequence containing a domain numerical sequence that indicates a domain; the domain classifier and the domain numerical sequence generation model, which is a mathematical model that generates the domain numerical sequence, are updated to improve the likelihood based on the likelihood; the training target numerical sequence is a numerical sequence that indicates text related to an image and also includes a classification numerical sequence that indicates the classification destination of the image; The control unit performs the learning step, An information processing method having

6. A classification method performed by a classification device, comprising: a control unit that trains a domain classifier that infers the domain to which data having a numerical sequence as a feature belongs, based on an input numerical sequence, along with the likelihood of inference; in the training, the domain classifier infers the domain to which the training target numerical sequence belongs and the likelihood of inference based on the feature of the training target numerical sequence, which is a numerical sequence containing a domain numerical sequence that indicates a domain; the domain classifier and the domain numerical sequence generation model, which is a mathematical model that generates the domain numerical sequence, are updated to improve the likelihood based on the likelihood; the training target numerical sequence is a numerical sequence that indicates text related to an image and also includes a classification numerical sequence that indicates the classification destination of the image; the classification unit that performs classification of image data to be classified using a trained training target numerical sequence obtained by an information processing device and a trained domain classifier; the trained training target numerical sequence is the training target numerical sequence at the time when a predetermined learning termination condition, which is a condition for the end of the training, is met; and the trained domain classifier is the domain classifier at the time when the learning termination condition is met. The classification unit performs the classification step, A classification method that includes [specific characteristics / features].

7. A program for causing a computer to function as an information processing device according to claim 1 or 2.

8. A program for causing a computer to function as a classification device according to claim 3 or 4.

Images