Content labeling device, content labeling method, content labeling program, model learning device, model learning method, and model learning program

Abstract

To appropriately train a model using an attention structure.SOLUTION: A content label imparting apparatus includes: a model training unit which trains a model using content data and correct labels imparted to the data, to generate a trained model; and an inference unit which generates a prediction label of a content, for data to be inferred, using the model trained by the model training unit. The model training unit includes a first partial model and a second partial model which greatly changes with time than the first partial model and has a unidirectional attention structure. In the unidirectional attention structure, the model for encoding a hidden state of the second partial model is trained using a hidden state of the first partial model and a hidden state of the second partial model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a content labeling device, a content labeling method, a content labeling program, a model learning device, a model learning method, and a model learning program. [Background technology]

[0002] Non-Patent Document 1 describes a machine learning method using a multimodal Transformer for images and language. This method enables learning of cross-modal relationships between images and language using a self-attention structure that uses both image and language tokens. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. ICML, pages 5583-5594, 2021. Summary of the Invention [Problem to be solved by the invention]

[0004] When training using attribute information such as user data and system data in addition to images and language, attribute information changes significantly over time compared to images and language, so frequent retraining is desirable. However, retraining a model using a self-attention structure that also includes images and language requires a large amount of computation.

[0005] In addition, multimodal training data that links language, images, and attribute information is required, but such data that links multiple modal data together is less common than single-modal data such as language-only data, so it may not be possible to prepare a sufficient amount of data for learning the self-attention structure.

[0006] The present application has been made in consideration of the above, and aims to appropriately train a model using an attention structure. [Means for solving the problem]

[0007] The content labeling device of the present application comprises a model learning unit that learns a model using content data and the correct labels assigned to that data to generate a learned model, and an inference unit that generates a predicted label for content using the model learned by the model learning unit for data to be inferred, wherein the model learning unit includes a first partial model and a second partial model that changes more over time than the first partial model and has a unidirectional attention structure, and in the unidirectional attention structure, the model that encodes the hidden state of the second partial model is learned using the hidden state of the first partial model and the hidden state of the second partial model. [Effects of the Invention]

[0008] According to one aspect of the embodiment, a model using an attention structure can be appropriately trained. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a content labeling device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration. [Figure 3A] FIG. 3A is a diagram showing an example of the flow of preprocessing on image data. [Figure 3B]FIG. 3B is a diagram illustrating an example of the flow of processing performed by the learning data preprocessing unit on text data. [Figure 3C] FIG. 3C is a diagram illustrating an example of the flow of processing performed by the learning data preprocessing unit on attribute data. [Figure 3D] FIG. 3D is a diagram illustrating an example of the processing flow of the learning data preprocessing unit for the correct label. [Figure 4] FIG. 4 is a diagram for explaining the forward propagation process. [Figure 5] FIG. 5 is a diagram showing an example of processing by an image encoder and a text encoder. [Figure 6] FIG. 6 is a diagram illustrating an example of the MLP process. [Figure 7] FIG. 7 is a diagram illustrating an example of the processing of the attribute encoder. [Figure 8] FIG. 8 is a diagram illustrating another example of the processing of the attribute encoder. [Figure 9] FIG. 9 is a diagram illustrating an example of patch division and image token generation. [Figure 10] FIG. 10 is a diagram illustrating an example of processing performed by the learning data preprocessing unit on text data. [Figure 11] FIG. 11 is a diagram illustrating an example of generating an attribute token. [Figure 12] FIG. 12 is a diagram illustrating an example of processing of a pre-learning model for an image and text. [Figure 13] FIG. 13 is a diagram illustrating an example of vector generation. [Figure 14] FIG. 14 is a diagram illustrating an example of vector generation. [Figure 15] FIG. 15 is a diagram illustrating an example of a model configuration. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a detailed description will be given of a content labeling device, a content labeling method, a content labeling program, a model learning device, a model learning method, and a model learning program (hereinafter referred to as "embodiments") according to the present application, with reference to the drawings. Note that the content labeling device, the content labeling method, the content labeling program, the model learning device, the model learning method, and the model learning program according to the present application are not limited to these embodiments. Furthermore, the same components in the following embodiments are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0011] [Embodiment] Hereinafter, an embodiment of the present invention will be described using an example of a device that assigns labels to content such as advertisements. The labels assigned to content can be used, for example, as information for controlling the distribution destination of advertisements. Note that while this embodiment will be described using an example of assigning labels to content, the present invention can also be applied to various types of machine learning, such as prediction and classification, by changing the learning data and changing the learning target.

[0012] FIG. 1 is a diagram illustrating an example of the configuration of a content labeling device according to an embodiment. Specifically, FIG. 1 is a diagram illustrating the configuration of a content labeling device 101. The content labeling device 101 is an information processing device (computer) that executes various information processes described below. For example, the content labeling device 101 is a model learning device that executes a process (learning process) to learn various models such as a trained model 111 described below. Furthermore, for example, the content labeling device 101 is an inference processing device that executes inference processing using various models such as the trained model 111.

[0013] Note that, although the following description will be given as an example in which the content labeling device 101 performs both the learning process and the inference process, the learning process and the inference process may be performed by different devices. That is, the model learning device that performs the learning process and the inference processing device that performs the inference process using the model may be separate entities. In this case, the model learning device may have the functions of the model generation unit 106, which will be described later, and the inference processing device may have the functions of the inference unit 112, which will be described later.

[0014] From here, various types of information and processing will be described using as an example a content labeling device 101 that executes both the learning process and the inference process shown in Fig. 1. The content labeling device 101 has image data 102, text data 103, attribute data 104, and correct labels 105. The image data 102, text data 103, attribute data 104, and correct labels 105 (hereinafter sometimes collectively referred to as "learning data") are data for training a model that assigns labels.

[0015] The content labeling device 101 has a storage unit realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. For example, learning data such as image data 102, text data 103, attribute data 104, and correct labels 105 are stored in the storage unit.

[0016] Image data 102 is data of an image included in content such as an advertisement (constituting the content). Like image data, text data 103 is text data included in the content, and is text data accompanying the image data (i.e., text data included in the same content as the image). For example, text data 103 is text extracted from text in an image using OCR (Optical Character Recognition / Reader), text displayed with the image, text on a web page linked to an advertisement image, and a description of the advertisement. Attribute data 104 includes attribute information of advertisers and users, attribute data and features related to the advertisement, etc. Correct label 105 is a correct label assigned in advance to each piece of content. It is assumed that such learning data is associated with data included in the same content.

[0017] The model generation unit 106 generates a trained model 111 for assigning labels based on the image data 102, the text data 103, the attribute data 104, and the correct label 105. In addition, the model generation unit 106 may use a pre-trained image model 109 and a pre-trained language model 110, which have been pre-trained, in order to improve the efficiency of training. For example, the content label assignment device 101 may store the pre-trained image model 109 and the pre-trained language model 110 in a storage unit.

[0018] The model generation unit 106 includes a training data preprocessing unit 107 and a model training unit 108. The training data preprocessing unit 107 converts training data into vectors. The model training unit 108 generates a trained model 111 using the output of the training data preprocessing unit 107, a pre-trained image model 109, and a pre-trained language model 110. For example, the content labeling device 101 may store the trained model 111 in a storage unit.

[0019] The inference unit 112 assigns a label to the input content data to be predicted using the trained model 111. The inference unit 112 includes an inference data pre-processing unit 113 and an inference processing unit 114. The inference data pre-processing unit 113 converts the content data to be predicted into vector format. The inference processing unit 114 uses the output of the inference data pre-processing unit 113 and the trained model 111 to output a predicted label for the content.

[0020] The input / output unit 115 receives data to be predicted via an input device or a communication device, and outputs a predicted label via an output device or a communication device. For example, the communication device is realized by a NIC (Network Interface Card) or the like, and is connected to a network via a wired or wireless connection to transmit and receive information to and from other information processing devices. In addition, various data input and output by the input / output unit 115 may be recorded in various storage devices such as the storage unit described above. For example, content data and predictive labels may be recorded in various storage devices such as the storage unit described above. The input / output unit may be, for example, a type that transmits and receives data via communication in the form of an API (Application Programming Interface), or may be linked to another system or may be user input and output via a keyboard, display, or the like.

[0021] The data input unit 116 receives data to be predicted via an input device or a communication device, and transmits it to the inference data preprocessing unit 113. The label output unit 117 receives predicted labels from the inference processing unit 114, and outputs them via an output device or a communication device.

[0022] For example, the content labeling device 101 has a control unit that is realized by a GPU (Graphics Processing Unit), a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc., executing various programs (corresponding to examples of information processing programs such as model learning programs and inference programs) stored in a storage device inside the content labeling device 101 using RAM as a working area. For example, the control unit has a model generation unit 106, an inference unit 112, and an input / output unit 115, and realizes or executes information processing functions and actions corresponding to the model generation unit 106, the inference unit 112, and the input / output unit 115, respectively.

[0023] Fig. 2 shows an example of a hardware configuration for realizing the processing of the content labeling device 101 in Fig. 1. Fig. 2 is a diagram showing an example of the hardware configuration. A computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output I / F (Interface) 1060, an input I / F 1070, and a network I / F 1080 are connected via a bus 1090.

[0024] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic device 1030 is realized by a processor such as a GPU, a CPU, or an MPU. Note that the arithmetic device 1030 may have any configuration as long as it can execute the desired arithmetic processing, and is not limited to a processor, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0025] The primary storage device 1040 is a memory device such as RAM that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a read-only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may also be a removable storage medium such as a universal serial bus (USB) memory or a secure digital (SD) memory card. The secondary storage device 1050 may also be cloud storage (online storage), a network attached storage (NAS), a file server, or the like.

[0026] The output I / F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, projector, printer, etc., and is realized by a connector conforming to a standard such as USB, DVI (Digital Visual Interface), or HDMI (registered trademark) (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, keypad, button, scanner, etc., and is realized by a USB, etc.

[0027] Furthermore, the output I / F 1060 and the input I / F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

[0028] The output device 1010 and the input device 1020 may be integrated into one device, such as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated into one device as an input / output I / F.

[0029] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0030] The network I / F 1080 receives data from other devices via the network N and sends it to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0031] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I / F 1060 and the input I / F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0032] For example, when the computer 1000 functions as the content labeling device 101, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040 to realize various functions described below.

[0033] Note that each component in FIG. 2 may have any configuration depending on the implementation method. Furthermore, each process in FIG. 1 may be performed by multiple computing devices instead of a single computing device. In particular, model learning can be performed efficiently by using parallel distributed processing using multiple computing devices, which allows learning to be performed in a short time. Furthermore, data, etc. may be stored in a database server or the like and transmitted and received via a communication device as necessary.

[0034] 1, a detailed description will be given of each process executed by the content labeling device 101. The content labeling device 101 executes the following processes.

[0035] The training data preprocessing unit 107 performs preprocessing on the image data 102, text data 103, attribute data 104, and correct answer label 105. An example of the flow of preprocessing performed by the training data preprocessing unit 107 will be described using FIGS. 3A to 3D. Note that FIGS. 3A to 3D show processing steps for images, text, attributes, and correct answer labels. In the following, it is assumed that one image, one sentence (a sentence or multiple sentences), and attribute data for that content are associated with one piece of content.

[0036] FIG. 3A is a diagram showing an example of the flow of preprocessing of image data. In image acquisition 301, image data 102 is acquired from a database or the like stored in a storage device or the like. Next, in image preprocessing 302, preprocessing is performed on the image acquired by image acquisition 301. In image preprocessing 302, tile processing, which duplicates and arranges images to create a single image, noise addition processing to improve the robustness of learning, cropping processing, which randomly cuts out part of the image, and the like may be performed. Furthermore, if not necessary, no processing may be performed in image preprocessing 302. In image preprocessing 302, the preprocessed image is output.

[0037] In image normalization 303, the preprocessed image output by image preprocessing 302 is transformed to a predetermined vertical and horizontal size by a method such as scaling. In the following, the number of channels of the image generated at this time is defined as C, the number of vertical pixels as H, and the number of horizontal pixels as W. For example, in the case of an RGB color image, C=3. As a result of this processing, the image used as training data can be regarded as a (C, H, W)-dimensional tensor.

[0038] In patch division 304, the normalized image output by image normalization 303 is divided into patches. An example of this is shown in FIG. 9. FIG. 9 is a diagram showing an example of patch division and image token generation. In the example shown in FIG. 9, nine patch images are generated by dividing normalized image 901 vertically and horizontally into three. For example, if the normalized image has a vertical size of 224 and a horizontal size of 224 and the number of vertical and horizontal divisions is 14, 196 patch images, each 16 vertically and 16 horizontally, are generated. Note that the number of divisions into patches may be different vertically and horizontally. Furthermore, if the vertical and horizontal sizes of the normalized image are not divisible by the number of divisions, the divisions may be performed in such a way that some of the patches overlap.

[0039] Next, in image token generation 305, a vector is generated from each patch image generated in patch division 304. For example, in image token generation 305, a vector is generated for each patch image. An example of vector generation will be described. For example, if a patch image has a vertical length of 16, a horizontal length of 16, and three channels, and each pixel value is used as an element of the vector, a 768-dimensional vector is generated for each patch. Furthermore, in image token generation 305, special tokens called class tokens are added. Class tokens are learnable parameters that are initialized with random values. Hereinafter, these vectors and class tokens will be referred to as image tokens. If there are 196 patch images, 197 image tokens with 768 dimensions, including class tokens, will be generated. Hereinafter, the number of image tokens will be referred to as NI, and the number of dimensions will be referred to as DI. Image token generation 305 outputs this image token sequence.

[0040] Returning to Figure 2, the explanation will continue. The training data preprocessing unit 107 performs the processes from image acquisition 301 to image token generation 305 shown in Figure 3A for each image. The token sequence generated here can be regarded as a (NI, DI)-dimensional tensor for each image.

[0041] Next, the processing of text data by the learning data pre-processing unit 107 will be described with reference to Fig. 3B. Fig. 3B is a diagram showing an example of the flow of processing of text data by the learning data pre-processing unit.

[0042] First, text acquisition 306 acquires text data 103 from a database stored in a storage device or the like. Next, morphological analysis 307 applies morphological analysis to the sentence to segment it into words. For example, FIG. 10 illustrates an example of processing performed by the training data preprocessing unit on text data. As illustrated in FIG. 10, word segmentation 1002 segments input sentence 1001 into words using morphological analysis. In the example illustrated in FIG. 10, a special token [CLS] called a class token is added to the beginning of each segment. Also, as illustrated in word segmentation 1002, a special token [SEP] indicating the end or separation of a sentence is added. Note that training data preprocessing unit 107 may segment a sentence not by word but by character. In this case, it is sufficient to simply segment the sentence into characters without performing morphological analysis. If segmentation is performed into characters, the following processing can be performed in the same way by replacing the word portion with a character. Also, segmentation does not have to be performed by grammatical words; a method of learning how to segment based on data may be adopted.

[0043] Returning to Figure 3B, the explanation will continue. In text token generation 308, each word in the segmented word string is converted into a vector. To this end, training data pre-processing unit 107 converts the word string generated by morphological analysis 307 into a predetermined fixed-length ID string. To this end, training data pre-processing unit 107 uses a term list prepared in advance.

[0044] For example, as shown in FIG. 10, the term list 1003 has entries that associate terms with IDs. For example, the term list 1003 includes entries for special tokens such as [PAD], [UNK], [CLS], and [SEP]. Using this term list 1003, the training data preprocessing unit 107 converts a word string into a fixed-length ID string. At this time, if the predetermined fixed length is longer than the word string, it is necessary to fill in the missing sentence. [PAD] is a special token that fills in the missing part. [UNK] is a special token that indicates an unknown word not included in the term list. [CLS] is a class token used in later processing, and [SEP] is a special token that indicates the end of a sentence or a delimiter. An ID is assigned to each of these special tokens and words.

[0045] The training data preprocessing unit 107 uses this term list to convert the word string into an ID string. For example, the training data preprocessing unit 107 uses the term list 1003 shown in FIG. 10 to convert each word in the word segmentation 1002 shown in FIG. 10 into an ID, thereby generating the ID string 1004 shown in FIG. 10. The ID string 1004 shown in FIG. 10 shows an example of the format of an ID string. Here, if the fixed length is 20, the length of the ID string corresponding to the word string generated from the word segmentation 1002 is 14. Therefore, the training data preprocessing unit 107 generates the fixed-length ID string 1004 by adding ID=0 corresponding to the six special tokens ([PAD]). In other words, if the fixed length is 20, the word string in the example of the word segmentation 1002 has a length of 14, so it is complemented with ID=0 corresponding to the six special tokens ([PAD]).

[0046] Returning to FIG. 3B, the explanation will be continued. In text token generation 308, the training data preprocessing unit 107 converts each ID into a vector using the ID string. In this process, one vector is generated for each ID, so if the fixed length is 20, the training data preprocessing unit 107 generates 20 vectors. Here, an example of vector conversion will be explained. For example, if the total number of terms in the term list is DT, the training data preprocessing unit 107 converts the term list into a DT-dimensional one-hot vector in which the element corresponding to the ID is 1 and the others are 0. For example, a token with an ID of 2 is the third token counting from 0. Therefore, the training data preprocessing unit 107 converts the token with an ID of 2 into a DT-dimensional vector (0, 0, 1, 0, ..., 0) in which the third element is 1 and the others are 0.

[0047] The training data preprocessing unit 107 outputs a text token sequence generated by the text token generation unit 308. For example, a text token sequence including the class token 1005 shown in FIG. 10 is output. In actual applications, any fixed length, such as 512, can be adopted. When the fixed length is 512, the training data preprocessing unit 107 generates 512 DT-dimensional tokens. The token sequence generated here can be regarded as a (NI, DI)-dimensional tensor, where NI is the fixed length.

[0048] Next, the processing of attribute data by the learning data pre-processing unit 107 will be described with reference to Fig. 3C. Fig. 3C is a diagram showing an example of the flow of processing of attribute data by the learning data pre-processing unit.

[0049] The training data pre-processing unit 107 converts the attribute data 104 into an attribute token string in attribute token generation 309. The training data pre-processing unit 107 adds class tokens 1005 (see, for example, FIG. 10 ) to the token string in the same way as images and text. The training data pre-processing unit 107 also generates the remaining tokens from the attribute data 104. As with ordinary machine learning methods, various conversion methods can be used to generate a token string from attribute data, but several examples will be described here.

[0050] For example, FIG. 11 is a diagram illustrating the attribute token generation process 309. FIG. 11 illustrates an example of attribute token generation. The attribute features 1101 shown in FIG. 11 are various pieces of information associated with content, such as the ID of a content advertiser, data related to the usage of the advertiser's service, information about the content itself, and features generated from these. The "content ID" included in the attribute features 1101 is, for example, a sequential ID assigned to each piece of content. The "ID" included in the attribute features 1101 is, for example, the advertiser's ID, which is an example of a user ID. A one-hot vector can be used to generate tokens from IDs, as with words. In this case, if the total number of IDs is large, the number of dimensions of the vector increases. To reduce this, advertiser IDs can be clustered into several groups based on the advertiser's usage, etc., to generate "cluster IDs," which can then be used instead. Alternatively, only the most frequently used IDs can be used, and the other IDs can be collectively treated as an unknown token [UNK]. In this way, the content labeling device 101 can adopt any token and divide a user ID to use it as multiple tokens. Also, there may be a huge number of IDs, and although they may be clustered as described above, the content labeling device 101 may divide a one-hot vector generated from the ID and handle it as separate tokens.

[0051] Furthermore, "feature A" and "feature B" included in attribute feature 1101 are feature amounts generated from information linked to content, and are different types of feature amounts. The learning data preprocessing unit 107 may use feature amounts generated from information linked to content, such as feature amount A and feature amount B of attribute feature 1101, and generate them in an order such as (feature amount A, feature amount B, ...). In this way, feature amounts created by another machine learning method or data aggregation may be vectorized and used as tokens. Also, a simple predictor may be created using another machine learning method, and the prediction results may be regarded as feature amounts and used as tokens.

[0052] The training data preprocessing unit 107 generates vectors and class tokens by converting the information of the attribute features 1101 into a vector sequence. Hereinafter, the vectors and class tokens generated in this way may be referred to as attribute tokens. If the number of tokens is NA and the number of dimensions of the vector is DA, the token sequence generated here can be regarded as an (NA, DA)-dimensional tensor.

[0053] Next, the processing of the training data preprocessing unit 107 for the correct label will be described with reference to FIG. 3D. FIG. 3D is a diagram showing an example of the processing flow of the training data preprocessing unit for the correct label. The training data preprocessing unit 107 converts the label into a vector using the label vector generation unit 310. For example, if there are T types of labels and one type of label is assigned to one piece of content, the correct label can be expressed as a T-dimensional one-hot vector. That is, if the correct label is the I-th label among the T types, the training data preprocessing unit 107 generates a vector such that the I-th element of the T-dimensional vector is 1 and the other elements are 0. Note that in the case of a multi-label task in which multiple labels are assigned to one piece of content, it is sufficient to set corresponding elements of the vector to 1. Through the above processing, the label vector generation unit 310 converts the correct label into a T-dimensional vector.

[0054] As described above, the processing of the training data preprocessing unit 107 generates a (NI, DI)-dimensional tensor from the image, a (NT, DT)-dimensional tensor from the text, a (NA, DA)-dimensional tensor from the attribute, and a T-dimensional correct vector corresponding to the number of label types T from the correct label. Of these, the class token is a learnable parameter, and the class token parameters are also learned in the next model training unit 108.

[0055] Next, the model learning unit 108 learns model parameters from these tensors and the correct vector. Model learning includes forward propagation and backpropagation processes. In backpropagation, the model parameters are updated by a backpropagation method based on a loss function. The loss function can be, for example, cross entropy error. For parameter updating by backpropagation, methods such as AdamW and SGD (Stochastic Gradient Descent) can be used as optimizers. Since the backpropagation method based on forward propagation is used to update the model, the following describes the calculation process of forward propagation to obtain output from input. Note that the model learning unit 108 processes multiple samples together in batches. The batch size is B.

[0056] An example of the forward propagation process will now be described with reference to Fig. 4 and other figures. Fig. 4 is a diagram for explaining the forward propagation process. For example, each configuration corresponding to the image token embedding 401 to the prediction head 417 shown in Fig. 4 and other figures may be included in a trained model (for example, the trained model 111). For example, Fig. 4 shows an example of the configuration of a model trained by the content labeling device 101.

[0057] First, in image token embedding 401, each DI-dimensional token is converted into a D-dimensional token by a linear transformation from DI to D using learnable parameters. Note that if the values ​​of DI and D are equal, this linear transformation does not have to be performed.

[0058] Next, in image position embedding 402, a learnable D-dimensional vector is generated for each token. That is, in image position embedding 402, a position vector having N1 D-dimensional learnable parameters is generated. Then, in image position embedding 402, the generated position vector is added to each of the N1 tokens in the image token sequence and output.

[0059] In addition, in text token embedding 403 and text position embedding 404, similar to image token embedding 401 and image position embedding 402 for image token sequences, a linear transformation with learnable parameters is applied to the text token sequence, and position vectors with learnable parameters are added. At this time, position encoding based on a sine function may be used. In this case, the addition is performed after multiplying by a coefficient that aligns the magnitude of the norm to the same level.

[0060] In the attribute token embedding 405, a linear transformation with learnable parameters is applied to the attribute token sequence, similar to the processing for the image token sequence and the text token sequence, except that the addition of the position vector may not be necessary.

[0061] Next, in attribute type embedding 407, information for distinguishing the type of attribute data is embedded. Attribute information patterns may differ depending on differences in the process and timing of data generation, external factors such as the freshness of the original data, differences in the user's terminal, differences in the form of provision to users, differences in user groups, differences in systems, and external factors such as other events. When each token in an attribute token sequence is generated from heterogeneous mixed data with different patterns, in order to distinguish them, for example, in attribute type embedding 407, a learnable D-dimensional attribute type embedding vector is generated for each type of such data and added to the corresponding attribute token. Note that attribute type embedding is not required.

[0062] For example, if attribute token A is a token generated from an ID of a certain period HA and attribute token B is a token generated from an ID of a certain period HB, in order to distinguish the freshness of the data, for example, in attribute type embedding 407, different attribute type embedding vectors TA and TB, each of which has learnable parameters, are added.

[0063] Also, for example, if attribute token A is a token generated from data such as the number of times the user terminal DA is used, and attribute token B is a token generated from data such as the number of times the user terminal DB is used, then, for example, in attribute type embedding 407, different attribute type embedding vectors TA and TB, each having learnable parameters, are added as information for distinguishing between different user terminals. Similarly, in attribute type embedding 407, for example, it is possible to embed information such as differences in user terminals, provision formats, user sets, systems, and other external factors.

[0064] Furthermore, when each token in an attribute token string is generated from heterogeneous mixed data having a different pattern, the attribute type embedding 407 may use embedding of the attribute type in order to distinguish them.

[0065] For example, if attribute token A is a token generated from an ID and attribute token B is a token generated from data such as the number of times of use, the types of original data are different and they can be considered to be data with different patterns.Therefore, for example, in attribute type embedding 407, different attribute type embedding vectors TA and TB, each of which has learnable parameters, are added together.

[0066] However, if the data patterns are significantly different, it is possible to treat these data as different tokens and use a method other than the method of distinguishing them by embedding attribute types. For example, if the data patterns are significantly different, it is possible to treat these data as one token, select a large number of dimensions so that both sets of data can be stored, and assign these data to different dimensions of one token.

[0067] For example, if attribute token A is generated from data such as the number of times HA was used during a certain period, and attribute token B is generated from data such as the number of times HB was used during a certain period, they can be considered as data with the same pattern, even though they are generated for different periods. Therefore, in this case, for example, in attribute type embedding 407, the same attribute type embedding vector T with learnable parameters may be added to both.

[0068] Attribute information may contain data that has a variety of heterogeneous patterns that differ depending on the process and timing of data generation, external factors, etc. By embedding this information in the tokens using attribute type embedding 407, information regarding the pattern differences can be added, making it possible to distinguish and learn tokens generated from data with different patterns. Note that attribute type embedding 407 is not necessary.

[0069] Through the above process, including the batch size B, the image token sequence is converted into a (B, NI, D)-dimensional tensor, the text token sequence is converted into a (B, NT, D)-dimensional tensor, and the attribute token sequence is converted into a (B, NA, D)-dimensional tensor.

[0070] Next, the process from the encoding process start 408 to the process performed a predetermined number of times 415 is the encoding layer, and is executed a predetermined number of times. For example, in the case of 12 layers, it is executed 12 times.

[0071] The image hidden state input / output 409 takes the output of the image position embedding 402 as input in the first execution, and the output of the image encoder 410 in the second and subsequent executions, and outputs it directly to the image encoder 410 and attribute encoder 414.

[0072] Next, the image encoder 410 encodes the (B, NI, D)-dimensional tensor into a (B, NI, D)-dimensional tensor of the same dimensions. This encoding process will be described later.

[0073] As described above, the image encoder 410 does not change the dimensions of the tensor, so the output of the image position embedding 402 handled by the image hidden state input / output 409 and the output of the image encoder 410 are tensors of the same (B, NI, D) dimensions. Hereinafter, this (B, NI, D) dimensional tensor handled by the image hidden state input / output 409 is also referred to as the image hidden state.

[0074] As described above, in the case of 12 layers, the processing in the encoding layer involves converting (encoding) the (B, NI, D)-dimensional tensor generated in image position embedding 402 into tensors of the same dimensions 12 times.

[0075] The processing of the text hidden state input / output 411 and the text encoder 412 is similar to the processing of the image hidden state input / output 409 and the image encoder 410, and involves encoding a (B, NT, D)-dimensional tensor a specified number of times. Hereinafter, the (B, NT, D)-dimensional tensor handled by this text hidden state input / output 411 is also referred to as the text hidden state.

[0076] The attribute hidden state input / output 413 and the attribute encoder 414 also perform a process of encoding a (B, NA, D)-dimensional tensor a specified number of times. Hereinafter, the (B, NA, D)-dimensional tensor handled by the attribute hidden state input / output 413 is also referred to as the attribute hidden state. However, the attribute encoder 414 uses not only the attribute hidden state but also the image hidden state and the text hidden state for the encoding process. In this way, the attribute encoder 414 uses the image hidden state and the text hidden state in addition to the attribute hidden state for the encoding process.

[0077] If it is determined in the execution of the specified number of times 415 that the processing of the encoding layer has been executed the specified number of times (determination result in the execution of the specified number of times 415: YES), the processing proceeds to the vector concatenation 416. Note that if it is determined in the execution of the specified number of times 415 that the processing of the encoding layer has not been executed the specified number of times (determination result in the execution of the specified number of times 415: NO), the processing of the encoding layer (image hidden state input / output 409 to attribute encoder 414) is repeated.

[0078] The vector concatenation 416 concatenates tensors corresponding to the class tokens of the image, text, and attributes. That is, from the N-I (B, D)-dimensional tensors output by the image encoder 410, tensors (B, D) corresponding to the class tokens are extracted. Similarly, (B, D)-dimensional tensors corresponding to the class tokens are extracted from the output of the text encoder 412 and the attribute encoder 414. Next, the vector concatenation 416 concatenates the three (B, D)-dimensional tensors along the second dimension axis (the axis corresponding to D) to generate a (B, 3D)-dimensional tensor.

[0079] Finally, the prediction head 417 uses the (B, 3D)-dimensional tensor generated by the vector concatenation 416 as input to generate a T-dimensional predicted vector corresponding to the number of labels T. Each element of the predicted vector becomes the probability value of the corresponding label.

[0080] The prediction head 417 can use simple prediction functions, such as normalizing tensors using methods such as layer normalization with learnable parameters, linearly transforming 3D dimensions to T dimensions, and applying softmax or sigmoid functions.

[0081] Furthermore, when learning the model, the model learning unit 108 calculates a loss value from the T-dimensional predicted vector generated by the prediction head 417 and the T-dimensional correct vector using a loss function such as Cross Entropy Loss or Binary Cross Entropy Loss, and updates the model parameters using the backpropagation method based on the loss value.

[0082] For example, the model learning unit 108 performs the above-mentioned forward propagation and backward propagation processes a predetermined number of times while changing the batch extracted from the learning data, updates the learnable parameters, and saves the parameters in the learned model 111.

[0083] The initial values ​​of trainable parameters may be initialized by setting random values, but by creating pre-trained models for images and text in advance and using them to initialize parameters related to images and text, it is possible to efficiently generate highly accurate models.

[0084] Here, an example of processing of the pre-learning model for images and text will be described using Fig. 12. Fig. 12 is a diagram showing an example of processing of the pre-learning model for images and text. Each process in Fig. 12 handles image and text tokens independently, but is similar to the corresponding process for the image and text parts in Fig. 4, so detailed description will be omitted.

[0085] 12, the content labeling device 101 generates a pre-trained model for images and a pre-trained model for text, and stores the trainable parameters of the image token embedding 1201, image position embedding 1202, and image encoder 1210 generated therein in the pre-trained image model 109. When training the model of FIG. 4, the content labeling device 101 loads the parameters of the pre-trained image model 109 into the trainable parameters of the image token embedding 401, image position embedding 402, and image encoder 410 as initial values.

[0086] Similarly, for text, the content labeling device 101 stores the trainable parameters of the text token embedding 1203, the text position embedding 1204, and the text encoder 1212 in the pre-trained language model 110. When training the model of Figure 4, the content labeling device 101 loads the parameters of the pre-trained language model 110 as initial values ​​into the trainable parameters of the text token embedding 403, the text position embedding 404, and the text encoder 412. Note that the loaded parameters may be updated in the model parameter update process of the model training unit 108, or these parameters may be fixed and frozen.

[0087] Note that the pre-learning is not limited to the example in Figure 12, and training may be performed on a task different from the labeling task, or different data may be used. Also, an unsupervised learning method without a correct answer label may be used. Various pre-learning methods are known, including pre-learning methods used in Transformer training, such as ViT (Vision Transformer) and BERT (Bidirectional Encoder Representations from Transformers), and contrastive learning, and these methods can be utilized.

[0088] Next, the processing of the image encoder 410, text encoder 412, and attribute encoder 414 will be described with reference to Fig. 5 and Fig. 7. Fig. 5 is a diagram showing an example of the processing of the image encoder and text encoder. Fig. 7 is a diagram showing an example of the processing of the attribute encoder.

[0089] First, the image encoder 410 will be described using Fig. 5. Normalization 501 to residual addition 509 shown in Fig. 5 are an example of processing by the image encoder 410. For example, the image encoder 410 executes processing corresponding to normalization 501 to residual addition 509. Note that the method described here uses a method based on ViT as an example, but other configurations may also be used.

[0090] In normalization 501, the image hidden state, which is a (B, NI, D)-dimensional tensor, is normalized using a method such as layer normalization with learnable parameters. The next steps, from attention embedding 502 to attention weighting 505, are attention calculations.

[0091] First, in attention embedding 502, vectors called Q (Query), K (Key), and V (Value) are generated. First, for each of Q, K, and V, a linear transformation matrix with learnable parameters from D dimensions to D dimensions is maintained. This is a D x D matrix. Then, this linear transformation is applied to each D-dimensional token in the hidden image state to obtain the same D-dimensional token. For each token, three D-dimensional tokens are generated corresponding to the linear transformations of Q, K, and V. Below, the three D-dimensional token sequences generated at this time will be referred to as Q, K, and V, respectively.

[0092] An example of vector generation for the above Q, K, and V is shown in Figure 13. Figure 13 is a diagram showing an example of vector generation. By performing linear transformations corresponding to Q, K, and V on the N-I vectors in the hidden image state, N-I D-dimensional vectors corresponding to Q, K, and V are generated.

[0093] Next, in head division 503, each of the Q, K, and V tokens is divided into H tokens based on a predetermined number of heads H. The number of dimensions of the divided tokens is D / H. It is desirable to set D and H so that D is divisible by H. As a result, H N I H / D-dimensional vectors are generated for each of Q, K, and V. The N I H / D dimensions corresponding to the h-th division are denoted as Qh. This is shown in vector information 1301, vector information 1302, vector information 1303, and vector information 1304. FIG. 13 shows the case where H=3. In actual applications, H may be set to, for example, 12.

[0094] Next, in attention dot product 504, the dot product of vector Qh and vector Kh corresponding to each head is calculated. In FIG. 13, as shown in matrix information 1305, the dot product of vector Qh and vector Kh corresponding to each head is calculated. The ij-th component of this dot product value matrix is ​​the dot product value of the i-th vector of Qh and the j-th vector of Kh. Note that to prevent this dot product value from becoming too large, the dot product value divided by the square root of the number of dimensions D may be used as aij.

[0095] Next, in the attention weighting 505, Vh is weighted based on the dot product value, and the result is generated as a weighted vector. In FIG. 13, as shown in the weighted vector data 1306, a weighted vector is generated by weighting Vh based on the dot product value. For example, for each row i of the dot product value matrix, a softmax function is applied to the vector (ai1, ai2, ..., aiNI). Next, a weighted sum of the NI vectors of Vh is taken using the normalized vector as a weight, and this is used as the i-th weighted vector. Let this weighted vector be Sh. The weighted vector V is generated by combining S1, S2, ..., SH in the reverse order of when they were divided. This is composed of NI D-dimensional tokens.

[0096] For example, the content labeling device 101 applies the above process to each sample in the batch, which results in a weight vector V being a (B, N, D)-dimensional tensor, where the dimension of the weight vector V is the same as the dimension of the image hidden state.

[0097] Next, in residual addition 506, this weight vector and the image hidden state are added and output. After that, in normalization 507, normalization is performed using a method such as layer normalization with learnable parameters. Furthermore, in MLP 508, conversion is performed into a tensor of the same dimension using a feedforward network or the like. Finally, in residual addition 509, the output tensor of residual addition 506 is added to the tensor generated by MLP 508, and output to the next process. The output result at this time is also a tensor of the same dimension as the image hidden state, with (B, NI, D) dimensions.

[0098] An example of the processing of the MLP 508 will now be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the processing of the MLP. The processing of the MLP 508 includes the processing of linear transformation 601 to linear transformation 603 shown in Fig. 6. For example, the MLP 508 is a multilayer perceptron.

[0099] First, in linear transformation 601, each token is transformed from D-dimensional to E-dimensional using a linear transformation with learnable D×E-dimensional parameters. Next, in activation function application 602, an activation function such as ReLU (Rectified Linear Unit) is applied. In linear transformation 603, each token is transformed from E-dimensional to D-dimensional using a linear transformation with learnable E×D-dimensional parameters.

[0100] Through the above encoding process, the image hidden state is encoded into a tensor of the same dimensions (B, NI, D).

[0101] Next, the text encoder 412 will be described with reference to Fig. 5. Attention embedding 511 to normalization 522 shown in Fig. 5 are an example of processing by the text encoder 412. For example, the text encoder 412 executes processing corresponding to attention embedding 511 to normalization 522. Note that the method described here uses a method based on BERT as an example, but other configurations may also be used.

[0102] The processes from attention embedding 511 to attention weighting 514 are similar to the processes from attention embedding 502 to attention weighting 505 in the image encoder. Next, in residual addition 515, the text hidden state and the output of attention weighting 514 are added together and output to the next process. In linear transformation 516, the D-dimensional vector of each token is transformed using a D×D-dimensional linear transformation with learnable parameters.

[0103] Next, normalization 517 normalizes the vectors using a method such as layer normalization with learnable parameters. Linear transformation 518 transforms the D-dimensional vectors of each token using a D×D dimensional linear transformation with learnable parameters.

[0104] In activation function application 519, an activation function such as GELU (Gaussian Error Linear Units) is applied. In residual addition 520, the output of normalization 517 and the output of activation function application 519 are added together and output to the next process. In linear transformation 521, the D-dimensional vector of each token is transformed using a D x D-dimensional linear transformation with learnable parameters. Finally, in normalization 522, normalization is performed using a method such as layer normalization with learnable parameters.

[0105] Through the above encoding process, the text hidden state is encoded into a tensor of the same dimensions (B, NT, D).

[0106] Next, the processing of the attribute encoder 414 will be described with reference to Fig. 7. Vector combination 701 to residual addition 710 shown in Fig. 7 are an example of processing by the attribute encoder 414. For example, the attribute encoder 414 executes processing corresponding to vector combination 701 to residual addition 710. Note that the processing configuration described here is just an example, and another configuration using a unidirectional attention structure (also simply referred to as "unidirectional attention") may also be used.

[0107] First, in the vector combination 701, three tensors, namely, the (B, NI, D)-dimensional image hidden state tensor, the (B, NT, D)-dimensional text hidden state tensor, and the (B, NA, D)-dimensional attribute hidden state tensor output from the image hidden state input / output 409, text hidden state input / output 411, and attribute hidden state input / output 413, are combined along the second dimension (the dimension corresponding to NI, NT, and NA). As a result, N = NI + NA + NT tokens are generated for each sample, and a (B, N, D)-dimensional tensor is generated. This is called a combined vector. This combined vector combines image tokens, text tokens, and attribute tokens.

[0108] Next, in normalization 702, the (B, N, D)-dimensional tensor generated in vector combination 701 is normalized using a method such as layer normalization with learnable parameters. Next, in unidirectional attention embedding 703, vectors called Q, K, and V are generated, similar to attention embedding 502. However, Q is generated only for NA tokens corresponding to attribute tokens. K and V are generated for all N tokens. The subsequent process of head division 704 is similar to head division 503. An example of this process is shown in vector information 1401 and vector information 1402 in Figure 14. Figure 14 is a diagram showing an example of vector generation. As shown in vector information 1402, Q is NA corresponding to attribute tokens, while K and V are N. Hereinafter, Qh, Kh, and Vh are the vectors of Q, K, and V, respectively, resulting from the h-th division.

[0109] Next, in the unidirectional attention dot product 705, the dot product of Qh and Kh is calculated. However, the dot product is not calculated between all tokens, but between only attribute tokens and image tokens, text tokens, and attribute tokens. In other words, the dot product is calculated between tokens corresponding to attributes in Qh and all tokens in Kh. An example of the matrix format of this dot product value is shown in matrix information 1403.

[0110] Next, in unidirectional attention weighting 706, a softmax function is applied to the vector (ai1, ai2, ..., aiN) in the i-th row of the dot product matrix. Next, in unidirectional attention weighting 706, this vector is used as a weight to take a weighted sum of the N tokens in V, generating a D-dimensional weight vector. This generates NA weight vectors. This is applied to each sample, generating a (B, NA, D)-dimensional tensor for the batch size.

[0111] Next, in residual addition 707, the output of unidirectional attention weighting 706 is added to the output of attribute hidden state input / output 413, and the result is output to the next process. The processes from normalization 708 to residual addition 710 are the same as the processes from normalization 507 to residual addition 509 in the image encoder.

[0112] Through the above encoding process, the attribute hidden state is encoded into a tensor of the same dimensions (B, NI, D).

[0113] In a typical self-attention architecture, all tokens are encoded by calculating the dot products between all tokens and weighting them. In the configuration of the present application (e.g., the content labeling device 101), image and text tokens are weighted to attribute tokens based on their dot product values, but image and text tokens are not weighted. This unidirectional configuration does not change the image occlusion state or the text occlusion state. That is, the content labeling device 101 can embed three types of multimodal relationships (image, text, and attribute) in the attribute encoder without changing the image model or text model. In the above example, in a model (e.g., the trained model 111) trained by the content labeling device 101, the model portion corresponding to the image or text corresponds to the first partial model, and the model portion corresponding to attributes that change more over time than the image or text corresponds to the second partial model. For example, the first partial model has a permanent model structure that changes less over time. For example, the second partial model has a non-permanent model structure that changes more over time. For example, in the model example shown in Fig. 4, the portion corresponding to the image token sequence corresponds to the first partial model. Also, for example, in the model example shown in Fig. 4, the portion corresponding to the text token sequence corresponds to the first partial model. Also, for example, in the model example shown in Fig. 4, the portion corresponding to the attribute token sequence corresponds to the second partial model.

[0114] The above-described configuration allows the content labeling device 101 to achieve several advantages. For example, in media processing, particularly for images and text, a method is widely used in which a highly versatile pre-trained model is created in advance, and then fine-tuning is used to train the model to suit the task. Generally, pre-training requires a large amount of training data and a large amount of calculation, which makes it expensive, but fine-tuning allows training at a relatively low cost. With this configuration, a base model is trained in advance, and task-specific training can be performed at a relatively low cost and with high accuracy.

[0115] In the configuration of the present application (e.g., the content labeling device 101, etc.), unidirectional attention does not affect the parameters or hidden states of image or text models, so it is possible to use a pre-trained model created solely for images or text. If attention is used to weight tokens of other modalities to image or text tokens, it would significantly change the hidden state values ​​of the image model or text model, making it difficult to use a pre-trained model. As a result, the configuration of the present application (e.g., the content labeling device 101, etc.) can utilize pre-trained models created in advance for images or text, while learning the cross-modal relationships between images, text, and attributes with a highly accurate attention structure that can learn wide-area features.

[0116] Furthermore, an image pre-training model can be trained using image-only data, and a text pre-training model can be trained using text-only data. Compared to data in which multiple modalities such as images, text, and attributes are linked, image-only data or text-only data is often available in large quantities. The configuration of the present application (e.g., the content labeling device 101, etc.) can use a pre-training model that can be trained in a single modality, so high accuracy can be expected by not only training from multimodal data but also by pre-training with a large amount of single-modal data.

[0117] Furthermore, user-derived data such as attribute data is prone to change over time, making frequent model retraining desirable. On the other hand, image and text models are more versatile and are thought to degrade less over time than these types of data. When modals with less degradation and modals with more degradation coexist, pre-training modals with less degradation, such as images and text, allows for the use of pre-trained models for modals with less degradation, enabling model retraining at low cost.

[0118] In addition, if the attribute data requires high-frequency model retraining, but the image and language parts can be trained at low frequency, unidirectional attention does not change the parameters corresponding to the image or text, so it is possible to fix the image and text parameters or lower the learning rate and retrain only the attribute data.

[0119] In this way, the unidirectional attention structure allows for highly modular learning of the attribute part. Note that the process of the model learning unit 108 may provide dropout processing as appropriate to improve the robustness of learning.

[0120] Next, the inference unit 112 performs preprocessing on the content to be inferred that is input from the input / output unit 115. The preprocessing method is the same as that of the training data preprocessing unit 107. However, if random noise or the like is added in the training data preprocessing unit 107 to improve the robustness of learning, the inference data preprocessing unit 113 does not need to apply such noise processing.

[0121] Next, the inference processing unit 114 calculates a predicted label from the input using the forward propagation calculation method, similar to the model learning unit 108. However, dropout processing to improve the robustness of learning is not applied.

[0122] The data input unit 116 receives content data from the system or a user via an input device or a communication device. The label output unit 117 returns the predicted label of the inference processing unit 114 to the system or the user via an output device or a communication device. For example, the label output unit 117 transmits the predicted label of the inference processing unit 114 to a device included in the system or a user terminal used by the user via the output device or a communication device.

[0123] In the above description, the dimensions of the image hidden state vector, the dimensions of the text hidden state vector, and the dimensions of the attribute hidden state vector are all the same number of dimensions D. However, different numbers of dimensions may be set appropriate for each modality. An example of this point will be described with reference to FIG. 8. FIG. 8 is a diagram illustrating another example of the processing of the attribute encoder. Note that in FIG. 8, the same points as in FIG. 7 will not be described as appropriate. In the example shown in FIG. 8, in order to match the number of dimensions in the unidirectional attention calculation, dimension transformation 801 and dimension transformation 802 are performed to match the number of dimensions to the dimension of the attribute hidden state vector. Dimension transformations such as dimension transformation 801 and dimension transformation 802 may use linear transformation matrices with learnable parameters. This allows the content labeling device 101 to select a number of dimensions appropriate for each modality.

[0124] In the above explanation, the number of encoding layers, i.e., the number of encodings, is assumed to be the same for images, text, and attributes. However, if you want to adjust the number of layers to suit each modality, you can simply specify a layer that calculates unidirectional attention and a layer that calculates it independently, and then separate the processing methods. Independently calculated layers do not pass hidden states from the image hidden state input / output 409 and text hidden state input / output 411 to the attribute encoder 414, but instead calculate attention within the attribute token.

[0125] An example is shown in Figure 15. Figure 15 is a diagram showing an example of the model configuration. In the example shown in Figure 15, there are 8 layers for images, 6 layers for text, and 5 layers for attributes. Only the last 4 layers are used to calculate unidirectional attention, while the other layers are processed independently for each modality. Note that in the example in Figure 15, the last 4 layers are associated, but other association methods may also be used. This allows the total number appropriate for each modality to be selected.

[0126] In the example of Figure 4, we have explained a model that uses an attention structure for images and language, but the image and language models may not have an attention structure. In that case, attribute tokens are encoded using the hidden states of the image and language models.

[0127] In addition, the attribute information model may use only class tokens without using data. In that case, attribute data is not used and the data is bimodal, consisting of only image and language. However, since the class tokens of attributes are encoded using the hidden states of image and language, the cross-modal relationship between image and language can be learned using these tokens.

[0128] Although the above example uses images, text, and attribute data, it can also be used for images and attribute data, text and attribute data, or other modalities other than images, text, and attributes. In such cases, it is sufficient to use the hidden state of a versatile modal that changes little over time to calculate unidirectional attention that encodes the hidden state of a modal that changes greatly over time.

[0129] The content labeling device 101 calculates the relationship between modal tokens that change little over time, such as images and language, and modal tokens that change much over time, such as attribute information, while using a unidirectional attention structure that encodes only tokens that correspond to attribute information.This makes it possible to incorporate cross-modal relationships in a model of modals that change much over time and learn them without disrupting the parameters and states of modals that change little over time.

[0130] As a result, the content labeling device 101 can use a pre-trained model trained in a single modality for modalities that change little over time. That is, the content labeling device 101 can utilize single-modality training data. Furthermore, the content labeling device 101 can retrain only the parts that change greatly over time, without disrupting the parameters or states of modalities that change little over time, by reducing or fixing the learning rate for modalities that change little over time. As a result, the content labeling device 101 can perform efficient cross-modal learning in modalities that change little over time, by utilizing single-modality training data and reusing models, while combining them with modalities that require frequent re-training.

[0131] 〔others〕 Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of so-called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0132] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0133] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0134] For example, the above-described server device 10 may be realized by a plurality of server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform or the like using an API or network computing.

[0135] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0136] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit," etc. For example, an inference section can be read as an inference means or an inference circuit. [Explanation of symbols]

[0137] 101 Content labeling device 106 Model Generation Unit 107 Learning data preprocessing section 108 Model Learning Department 112 Reasoning Department 113 Inference data preprocessing unit 114 Inference processing unit 115 Input / output section 116 Data Entry Section 117 Label output unit

Claims

1. a model learning unit that learns a model using content data and correct labels assigned to the data, and generates a learned model; an inference unit that generates a predicted label of content using the model trained by the model training unit for data to be inferred; Equipped with The model learning unit The method includes: a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, wherein the hidden state of the first partial model and the hidden state of the second partial model are used to train the model that encodes the hidden state of the second partial model. A content labeling device comprising:

2. The model includes a model in which the number of dimensions of the hidden state of the first partial model is different from the number of dimensions of the second partial model, and before calculating the unidirectional attention structure, a dimensional transformation is performed on the hidden state of the first partial model by applying a linear change having a learnable parameter from the number of dimensions of the hidden state of the first partial model to the number of dimensions of the second partial model.

2. The content labeling device according to claim 1.

3. The model has an attention structure in the first partial model.

2. The content labeling device according to claim 1.

4. The model learning unit The second partial model has a plurality of layers for calculating an attention structure, and some of the layers are selected and associated with any hidden state of the first partial model. For the selected layers, unidirectional attention is calculated, and for layers other than the selected layers, attention is calculated using only the hidden state of the second partial model.

2. The content labeling device according to claim 1.

5. The model learning unit In order to handle various types of data in the second partial model, attribute types that distinguish between different data patterns are embedded in the tokens.

2. The content labeling device according to claim 1.

6. The model has only class tokens in the second partial model.

2. The content labeling device according to claim 1.

7. The model learning unit Use the user ID as the token in the second partial model.

2. The content labeling device according to claim 1.

8. The model learning unit In the second partial model, the user ID is divided and used as multiple tokens.

2. The content labeling device according to claim 1.

9. A computer-implemented content labeling method, comprising: a model training step of training a model using content data and correct labels assigned to the data to generate a trained model; an inference step of generating a predicted label of content using the model trained in the model training step for data to be inferred; Including, The model learning step includes: The method includes: a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, wherein the hidden state of the first partial model and the hidden state of the second partial model are used to train the model that encodes the hidden state of the second partial model. A content labeling method comprising:

10. a model training procedure for training a model using content data and correct labels assigned to the data to generate a trained model; an inference procedure for generating a predicted label of content using the model trained by the model training procedure for data to be inferred; on the computer, The model learning procedure includes: The method includes: a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, wherein the hidden state of the first partial model and the hidden state of the second partial model are used to train the model that encodes the hidden state of the second partial model. A content labeling program comprising:

11. a model learning unit that learns a model using content data and correct labels assigned to the data, and generates a learned model; an inference unit that generates a predicted label of content using the model trained by the model training unit for data to be inferred; Equipped with The model learning unit The method includes: a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, wherein the hidden state of the first partial model and the hidden state of the second partial model are used to train the model that encodes the hidden state of the second partial model. A model learning device characterized by:

12. The model includes a model in which the number of dimensions of the hidden state of the first partial model is different from the number of dimensions of the second partial model, and before calculating the unidirectional attention structure, a dimensional transformation is performed on the hidden state of the first partial model by applying a linear change having a learnable parameter from the number of dimensions of the hidden state of the first partial model to the number of dimensions of the second partial model.

12. The model learning device according to claim 11.

13. The model has an attention structure in the first partial model.

12. The model learning device according to claim 11.

14. The model learning unit The second partial model has a plurality of layers for calculating an attention structure, and some of the layers are selected and associated with any hidden state of the first partial model. For the selected layers, unidirectional attention is calculated, and for layers other than the selected layers, attention is calculated using only the hidden state of the second partial model.

12. The model learning device according to claim 11.

15. The model learning unit In order to handle various types of data in the second partial model, attribute types that distinguish between different data patterns are embedded in the tokens.

12. The model learning device according to claim 11.

16. The model has only class tokens in the second partial model.

12. The model learning device according to claim 11.

17. The model learning unit Use the user ID as the token in the second partial model.

12. The model learning device according to claim 11.

18. The model learning unit In the second partial model, the user ID is divided and used as multiple tokens.

12. The model learning device according to claim 11.

19. A computer-implemented model learning method, comprising: a model training step of training a model using content data and correct labels assigned to the data to generate a trained model; an inference step of generating a predicted label of content using the model trained in the model training step for data to be inferred; Including, The model learning step includes: The method includes: a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, wherein the hidden state of the first partial model and the hidden state of the second partial model are used to train the model that encodes the hidden state of the second partial model. A model learning method characterized by:

20. a model training procedure for training a model using content data and correct labels assigned to the data to generate a trained model; an inference procedure for generating a predicted label of content using the model trained by the model training procedure for data to be inferred; on the computer, The model learning procedure includes: The method includes: a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, wherein the hidden state of the first partial model and the hidden state of the second partial model are used to train the model that encodes the hidden state of the second partial model. A model learning program characterized by:

Images