Content label imparting apparatus, content label imparting method, content label imparting program, model training apparatus, model training method, and model training program

Abstract

To appropriately train a model using an attention structure.SOLUTION: A content label imparting apparatus includes: a model training unit which trains a model using content data and correct labels imparted to the data, to generate a trained model; and an inference unit which generates a prediction label of a content, for data to be inferred, using the model trained by the model training unit. The model training unit includes a first partial model and a second partial model which greatly changes with time than the first partial model and has a unidirectional attention structure. In the unidirectional attention structure, the model for encoding a hidden state of the second partial model is trained using a hidden state of the first partial model and a hidden state of the second partial model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a content labeling device, a content labeling method, a content labeling program, a model learning device, a model learning method, and a model learning program. [Background technology]

[0002] Non-Patent Document 1 describes a machine learning method using a multimodal Transformer for images and language. This method makes it possible to learn cross-modal relationships between images and language by using a self-attention structure that uses tokens of both images and language. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. ICML, pages 5583-5594, 2021. Summary of the Invention [Problem to be solved by the invention]

[0004] When learning using attribute information such as user data and system data in addition to images and language, attribute information changes more significantly over time than images and language, so it is desirable to re-learn it frequently. However, re-learning a model using a self-attention structure that also includes images and language requires a large amount of computation.

[0005] In addition, multi-modal training data that links language, images, and attribute information is required, but such data in which multiple modal data are linked together is scarce compared to single-modal data such as language-only data, so it may not be possible to prepare a sufficient amount of data for learning the self-attention structure.

[0006] The present application has been made in consideration of the above, and aims to appropriately train a model using attention structure. [Means for solving the problem]

[0007] The content labeling device of the present application comprises a model learning unit that learns a model using content data and a correct label assigned to the data to generate a learned model, and an inference unit that generates a predicted label of content for data to be inferred using the model learned by the model learning unit, wherein the model learning unit includes a first partial model and a second partial model that changes more over time than the first partial model and has a unidirectional attention structure, and in the unidirectional attention structure, the model that encodes the hidden state of the second partial model is learned using the hidden state of the first partial model and the hidden state of the second partial model. Effect of the Invention

[0008] According to one aspect of the embodiment, a model using an attention structure can be appropriately trained. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a content labeling device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration. [Figure 3A] FIG. 3A is a diagram showing an example of the flow of pre-processing on image data. [Figure 3B]FIG. 3B is a diagram illustrating an example of the flow of processing performed by the learning data preprocessing unit on text data. [Figure 3C] FIG. 3C is a diagram illustrating an example of the flow of processing performed by the learning data preprocessing unit on attribute data. [Figure 3D] FIG. 3D is a diagram illustrating an example of the flow of processing performed by the learning data preprocessing unit for a correct label. [Figure 4] FIG. 4 is a diagram for explaining the forward propagation process. [Diagram 5] FIG. 5 is a diagram showing an example of the processing of the image encoder and the text encoder. [Figure 6] FIG. 6 is a diagram illustrating an example of the MLP process. [Figure 7] FIG. 7 illustrates an example of the process of the attribute encoder. [Figure 8] FIG. 8 is a diagram illustrating another example of the process of the attribute encoder. [Figure 9] FIG. 9 is a diagram illustrating an example of patch division and image token generation. [Figure 10] FIG. 10 is a diagram illustrating an example of processing performed by the learning data preprocessing unit on text data. [Figure 11] FIG. 11 is a diagram illustrating an example of generating an attribute token. [Figure 12] FIG. 12 is a diagram showing an example of processing of a pre-learning model of an image and text. [Figure 13] FIG. 13 is a diagram illustrating an example of vector generation. [Figure 14] FIG. 14 is a diagram illustrating an example of vector generation. [Figure 15] FIG. 15 is a diagram illustrating an example of a model configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a detailed description will be given of a content labeling device, a content labeling method, a content labeling program, a model learning device, a model learning method, and a model learning program (hereinafter, referred to as "embodiments") for carrying out the present application with reference to the drawings. Note that the content labeling device, the content labeling method, the content labeling program, the model learning device, the model learning method, and the model learning program according to the present application are not limited to the embodiments. Furthermore, the same components in the following embodiments are given the same reference numerals, and duplicated descriptions will be omitted.

[0011] [Embodiment] In the following, an embodiment of the present invention will be described by taking an apparatus for assigning a label to content such as an advertisement as an example. The label assigned to the content can be used, for example, as information for controlling the delivery destination of the advertisement. Note that, although this embodiment will be described by taking an example of assigning a label to the content, the present invention can also be applied to various machine learning methods such as prediction and classification by changing the learning data and changing the learning target.

[0012] FIG. 1 is a diagram showing an example of the configuration of a content labeling device according to an embodiment. Specifically, FIG. 1 is a diagram showing the configuration of a content labeling device 101. The content labeling device 101 is an information processing device (computer) that executes various information processes described below. For example, the content labeling device 101 is a model learning device that executes a process (learning process) of learning various models such as a trained model 111 described later. Also, for example, the content labeling device 101 is an inference processing device that executes an inference process using various models such as the trained model 111.

[0013] In the following, a case where both the learning process and the inference process are performed by the content labeling device 101 will be described as an example, but the learning process and the inference process may be performed by different devices. That is, the model learning device that performs the learning process and the inference processing device that performs the inference process using the model may be separate entities. In this case, the model learning device may have the function of the model generation unit 106 described later, and the inference processing device may have the function of the inference unit 112 described later.

[0014] From here, various information and processes will be described using as an example a content labeling device 101 that executes both the learning process and the inference process shown in Fig. 1. The content labeling device 101 has image data 102, text data 103, attribute data 104, and correct labels 105. The image data 102, text data 103, attribute data 104, and correct labels 105 (hereinafter sometimes collectively referred to as "learning data") are data for learning a model that assigns labels.

[0015] The content labeling device 101 has a storage unit realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. For example, learning data such as image data 102, text data 103, attribute data 104, and correct answer labels 105 are stored in the storage unit.

[0016] The image data 102 is data of an image included in a content such as an advertisement (constituting the content). The text data 103 is data of text included in the content, similar to the image data, and is text data accompanying the image data (i.e., text data included in the same content as the image). For example, the text data 103 is text extracted from the text in the image by OCR (Optical Character Recognition / Reader), text displayed together with the image, text of a web page linked to the advertisement image, and a description of the advertisement. The attribute data 104 is attribute information of the advertiser or user, attribute data related to the advertisement, feature amount, and the like. The correct answer label 105 is a correct answer label previously assigned to each piece of content. It is assumed that such learning data is associated with data included in the same content.

[0017] The model generation unit 106 generates a trained model 111 for assigning a label based on the image data 102, the text data 103, the attribute data 104, and the correct label 105. In addition, the model generation unit 106 may use a pre-trained image model 109 and a pre-trained language model 110 that have been trained in advance in order to improve the efficiency of training. For example, the content label assignment device 101 may store the pre-trained image model 109 and the pre-trained language model 110 in a storage unit.

[0018] The model generation unit 106 includes a training data pre-processing unit 107 and a model training unit 108. The training data pre-processing unit 107 converts training data into a vector. The model training unit 108 generates a trained model 111 using the output of the training data pre-processing unit 107, a pre-trained image model 109, and a pre-trained language model 110. For example, the content labeling device 101 may store the trained model 111 in a storage unit.

[0019] The inference unit 112 assigns a label to the input content data to be predicted using the trained model 111. The inference unit 112 includes an inference data pre-processing unit 113 and an inference processing unit 114. The inference data pre-processing unit 113 converts the content data to be predicted into a vector format. The inference processing unit 114 uses the output of the inference data pre-processing unit 113 and the trained model 111 to output a predicted label of the content.

[0020] The input / output unit 115 receives data to be predicted via an input device or a communication device, and outputs a predicted label via an output device or a communication device. For example, the communication device is realized by a NIC (Network Interface Card) or the like, is connected to a network by wire or wirelessly, and transmits and receives information with other information processing devices. In addition, various data input and output by the input / output unit 115 may be recorded in various storage devices such as the above-mentioned storage unit. For example, content data and predicted labels may be recorded in various storage devices such as the above-mentioned storage unit. The input / output unit may be, for example, a type that transmits and receives data via communication in the form of an API (Application Programming Interface), or may be a link with another system or a user input / output via a keyboard, a display, or the like.

[0021] The data input unit 116 receives data to be predicted via an input device or a communication device, and transmits it to the inference data preprocessing unit 113. In addition, the label output unit 117 receives a predicted label from the inference processing unit 114, and outputs it via an output device or a communication device.

[0022] For example, the content labeling device 101 has a control unit that is realized by executing various programs (corresponding to examples of information processing programs such as a model learning program and an inference program) stored in a storage device inside the content labeling device 101 using a RAM as a working area by a GPU (Graphics Processing Unit), a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. For example, the control unit has a model generation unit 106, an inference unit 112, and an input / output unit 115, and realizes or executes functions and actions of information processing corresponding to each of the model generation unit 106, the inference unit 112, and the input / output unit 115.

[0023] Fig. 2 shows an example of a hardware configuration for realizing the processing of the content labeling device 101 in Fig. 1. Fig. 2 is a diagram showing an example of the hardware configuration. A computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I / F (Interface) 1060, an input I / F 1070, and a network I / F 1080 are connected via a bus 1090.

[0024] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 or the secondary storage device 1050, programs read from the input device 1020, etc., and executes various processes. The arithmetic device 1030 is realized by a processor such as a GPU, a CPU, or an MPU. Note that the arithmetic device 1030 may have any configuration as long as it can execute the desired arithmetic processing, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) without being limited to a processor.

[0025] The primary storage device 1040 is a memory device such as a RAM that primarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. The secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

[0026] The output I / F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, a projector, a printer, etc., and is realized by a connector conforming to a standard such as USB, DVI (Digital Visual Interface), or HDMI (registered trademark) (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, a keyboard, a keypad, a button, a scanner, etc., and is realized by a USB, etc.

[0027] Furthermore, the output I / F 1060 and the input I / F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

[0028] The output device 1010 and the input device 1020 may be integrated into one device, such as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated into one device, such as an input / output I / F.

[0029] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0030] The network I / F 1080 receives data from other devices via the network N and sends it to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0031] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I / F 1060 and the input I / F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0032] For example, when the computer 1000 functions as the content labeling device 101, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040 to realize various functions described below.

[0033] Each component in FIG. 2 may have any configuration depending on the implementation method. Each process in FIG. 1 may be implemented by multiple computing devices, not a single computing device. In particular, model learning can be efficiently performed in a short time by using multiple computing devices for parallel distributed processing. Data, etc. may be stored in a database server, etc., and transmitted and received via a communication device as necessary.

[0034] 1, there will be described in detail each process executed by the content labeling device 101. The content labeling device 101 executes each process shown below.

[0035] The learning data preprocessing unit 107 performs preprocessing on the image data 102, text data 103, attribute data 104, and correct label 105. An example of the flow of preprocessing executed by the learning data preprocessing unit 107 will be described with reference to Figs. 3A to 3D. Figs. 3A to 3D show processing steps for an image, text, attribute, and correct label. In the following, it is assumed that one image, one sentence (a sentence or multiple sentences), and attribute data for that content are associated with one piece of content.

[0036] FIG. 3A is a diagram showing an example of a flow of preprocessing of image data. In image acquisition 301, image data 102 is acquired from a database or the like stored in a storage device or the like. Next, in image preprocessing 302, preprocessing is performed on the image acquired by image acquisition 301. In image preprocessing 302, tile processing for duplicating and arranging an image to make it into one image, noise addition processing for improving the robustness of learning, crop processing for randomly cutting out a part of an image, and the like may be performed. In addition, in image preprocessing 302, nothing may be performed if not necessary. In image preprocessing 302, the image after preprocessing is output.

[0037] In image normalization 303, the preprocessed image output by image preprocessing 302 is transformed to a predetermined vertical and horizontal size by a method such as enlargement or reduction. In the following, the number of channels of the image generated at this time is C, the number of pixels in the vertical direction is H, and the number of pixels in the horizontal direction is W. For example, in the case of an RGB color image, C=3. As a result of such processing, the image used as learning data can be regarded as a (C, H, W)-dimensional tensor.

[0038] In the patch division 304, the normalized image output by the image normalization 303 is divided into patches. An example of this is shown in FIG. 9. FIG. 9 is a diagram showing an example of patch division and image token generation. In the example shown in FIG. 9, nine patch images are generated by dividing the normalized image 901 vertically and horizontally into three parts in the patch image 902. For example, if the normalized image has a vertical size of 224 and a horizontal size of 224, and the number of vertical and horizontal divisions is 14, 196 patch images with a vertical length of 16 and a horizontal length of 16 are generated. The number of divisions of the patch may be different vertically and horizontally. In addition, if the vertical and horizontal sizes of the normalized image are not divisible by the number of divisions, the normalized image may be divided in such a way that some of the images overlap.

[0039] Next, in image token generation 305, a vector is generated from each patch image generated in patch division 304. For example, in image token generation 305, a vector is generated for each patch image. An example of vector generation will be described. For example, in the case where the vertical length of a patch image is 16, the horizontal length is 16, and the number of channels is 3, if each pixel value is an element of a vector, a 768-dimensional vector is generated for each patch. Furthermore, in image token generation 305, a special token called a class token is added. The class token is a learnable parameter and is initialized with a random value. Hereinafter, these vectors and class tokens are referred to as image tokens. In the case where there are 196 patch images, 197 image tokens of 768 dimensions including the class token are generated. Hereinafter, the number of image tokens is NI, and the number of dimensions is DI. In image token generation 305, this image token string is output.

[0040] Returning to Fig. 2, the explanation will be continued. The learning data pre-processing unit 107 performs the processes from image acquisition 301 to image token generation 305 shown in Fig. 3A for each image. The token sequence generated here can be regarded as a (NI, DI)-dimensional tensor for each image.

[0041] Next, the processing of text data by the learning data pre-processing unit 107 will be described with reference to Fig. 3B. Fig. 3B is a diagram showing an example of the flow of processing of text data by the learning data pre-processing unit.

[0042] First, in the text acquisition 306, text data 103 is acquired from a database or the like stored in a storage device or the like. Next, in the morphological analysis 307, morphological analysis is applied to the sentence to divide it into words. For example, FIG. 10 is a diagram showing an example of processing of the learning data pre-processing unit for text data. As shown in FIG. 10, the word division 1002 divides the input sentence 1001 into words by morphological analysis. In the example shown in FIG. 10, a special token [CLS] called a class token is added to the beginning of the single division. Also, as shown in the word division 1002, a special token [SEP] indicating the end or separation of the sentence is added. Note that the learning data pre-processing unit 107 may divide the sentence into characters instead of words. In that case, it is sufficient to simply divide the sentence into characters without performing morphological analysis. When divided into characters, the following processing can be performed in the same way by replacing the word part with a character. Also, the division does not have to be into grammatical words, and a method of learning how to divide based on data may be adopted.

[0043] Returning to Fig. 3B, the explanation will be continued. In text token generation 308, each word in the segmented word string is converted into a vector. To this end, training data pre-processing unit 107 converts the word string generated in morphological analysis 307 into a predetermined fixed-length ID string. To this end, training data pre-processing unit 107 uses a term list prepared in advance.

[0044] For example, as shown in FIG. 10, the term list 1003 has entries that associate terms with IDs. For example, the term list 1003 includes entries for special tokens such as [PAD], [UNK], [CLS], and [SEP]. Using such term list 1003, the learning data preprocessing unit 107 converts a word string into a fixed-length ID string. At this time, if the predetermined fixed length is longer than the word string, it is necessary to fill in the missing sentences. [PAD] is a special token that fills in the missing parts. [UNK] is a special token that indicates an unknown word that is not included in the term list. [CLS] is a class token used in later processing, and [SEP] is a special token that indicates the end of a sentence or a delimiter. An ID is assigned to each of these special tokens and words.

[0045] The training data preprocessing unit 107 uses this term list to convert the word string into an ID string. For example, the training data preprocessing unit 107 uses the term list 1003 shown in FIG. 10 to convert each word in the word segmentation 1002 shown in FIG. 10 into an ID, thereby generating an ID string 1004 shown in FIG. 10. The ID string 1004 shown in FIG. 10 shows an example of the format of an ID string. Here, if the fixed length is 20, the length of the ID string corresponding to the word string generated from the word segmentation 1002 is 14, so the training data preprocessing unit 107 generates the ID string 1004 of a fixed length by adding ID=0 corresponding to six special tokens ([PAD]). That is, if the fixed length is 20, the word string in the example of the word segmentation 1002 has a length of 14, so it is complemented with ID=0 corresponding to the six special tokens ([PAD]).

[0046] Returning to FIG. 3B, the description will be continued. In text token generation 308, the training data preprocessing unit 107 converts each ID into a vector using the ID string. In this process, one vector is generated for each ID, so if the fixed length is 20, the training data preprocessing unit 107 generates 20 vectors. Here, an example of vector conversion will be described. For example, if the total number of terms in the term list is DT, the training data preprocessing unit 107 converts the term list into a DT-dimensional one-hot vector in which the element corresponding to the ID is 1 and the others are 0. For example, a token with an ID of 2 is the third, counting from 0. Therefore, the training data preprocessing unit 107 converts the token with an ID of 2 into a DT-dimensional vector (0, 0, 1, 0, ..., 0) in which the third element is 1 and the others are 0.

[0047] The training data pre-processing unit 107 outputs a text token string generated by the text token generation unit 308. For example, a text token string including a class token 1005 shown in FIG. 10 is output. In actual applications, any fixed length, such as 512, can be adopted. When the fixed length is 512, the training data pre-processing unit 107 generates 512 DT-dimensional tokens. The token string generated here can be regarded as a (NI, DI)-dimensional tensor, where NI is the fixed length.

[0048] Next, the processing of the attribute data by the learning data pre-processing unit 107 will be described with reference to Fig. 3C. Fig. 3C is a diagram showing an example of the flow of processing of the learning data pre-processing unit on the attribute data.

[0049] In attribute token generation 309, the training data pre-processing unit 107 converts the attribute data 104 into an attribute token string. The training data pre-processing unit 107 adds class tokens 1005 (see FIG. 10, for example) to the token string in the same manner as images and text. The training data pre-processing unit 107 also generates the remaining tokens from the attribute data 104. As with ordinary machine learning methods, various conversion methods can be used to generate a token string from attribute data, but several examples will be described here.

[0050] For example, FIG. 11 is a diagram for explaining the process of attribute token generation 309. FIG. 11 is a diagram showing an example of attribute token generation. Attribute features 1101 shown in FIG. 11 are various information linked to content, such as, for example, the ID of the advertiser of the content, data on the usage status of the service of the advertiser ID, or information on the content itself, and features generated from them. The "content ID" included in the attribute feature 1101 is, for example, a consecutive ID assigned to each content. The "ID" included in the attribute feature 1101 is, for example, the advertiser ID, which is an example of a user ID, and a one-hot vector can be used as a method for generating a token from the ID, just like a word. In this case, if the total number of IDs is large, the number of dimensions of the vector becomes large, so in order to reduce this, the advertiser IDs may be divided into several groups by clustering based on the usage status of the advertiser, and a "cluster ID" may be generated and used instead of the advertiser ID. Also, only the most frequently used IDs may be used, and the other IDs may be collectively treated as an unknown token [UNK]. In this way, the content labeling device 101 can adopt any token, and divide the user ID to use it as multiple tokens. Also, there may be a huge number of IDs, and although they may be clustered as described above, the content labeling device 101 may divide the one-hot vector generated from the ID and handle it as separate tokens.

[0051] Furthermore, "feature amount A" and "feature amount B" included in the attribute feature 1101 are feature amounts generated from information linked to the content, and are different types of feature amounts. The learning data preprocessing unit 107 may use feature amounts generated from information linked to the content, such as feature amount A and feature amount B of the attribute feature 1101, and generate them in an order such as (feature amount A, feature amount B, ...). In this manner, features created by another machine learning method or data aggregation may be vectorized and used as tokens. Also, a simple predictor may be created by another machine learning method, and the prediction result may be regarded as a feature amount and used as a token.

[0052] The learning data preprocessing unit 107 generates vectors and class tokens by converting the information of the attribute features 1101 into a vector sequence. Hereinafter, the vectors and class tokens generated in this way may be referred to as attribute tokens. If the number of tokens is NA and the number of dimensions of a vector is DA, the token sequence generated here can be regarded as a (NA, DA)-dimensional tensor.

[0053] Next, the processing of the learning data preprocessing unit 107 for the correct label will be described with reference to FIG. 3D. FIG. 3D is a diagram showing an example of the processing flow of the learning data preprocessing unit for the correct label. The learning data preprocessing unit 107 converts the label into a vector in the label vector generation unit 310. For example, if there are T types of labels and one type of label is assigned to one content, the correct label can be expressed as a T-dimensional one-hot vector. That is, if the correct label is the I-th label among the T types, the learning data preprocessing unit 107 generates a vector such that the I-th element of the T-dimensional vector is 1 and the other elements are 0. In the case of a multi-label task in which multiple labels are assigned to one content, it is sufficient to make the corresponding multiple elements of the vector 1. Through the above processing, the correct label is converted into a T-dimensional vector in the label vector generation unit 310.

[0054] As described above, by the processing of the training data preprocessing unit 107, a (NI, DI)-dimensional tensor is generated from the image, a (NT, DT)-dimensional tensor is generated from the text, a (NA, DA)-dimensional tensor is generated from the attributes, and a T-dimensional correct vector corresponding to the number of label types T is generated from the correct label. Of these, the class token is a learnable parameter, and in the next step, the model training unit 108, the parameters of the class token are also learned.

[0055] Next, the model learning unit 108 learns model parameters from these tensors and the ground truth vector. Model learning includes forward propagation and backpropagation processes. In backpropagation, the model parameters are updated by a backpropagation method based on a loss function. Cross entropy error or the like can be used as the loss function. For parameter updating by backpropagation, methods such as AdamW and SGD (Stochastic Gradient Descent) can be used as optimizers. Since the backpropagation method based on forward propagation is used for updating the model, the following describes the calculation process of forward propagation for obtaining output from input. Note that the model learning unit 108 processes multiple samples together in batches. The size of the batch is B.

[0056] From now, an example of the forward propagation process will be described with reference to Fig. 4 and the like. Fig. 4 is a diagram for explaining the forward propagation process. For example, each configuration corresponding to the image token embedding 401 to the prediction head 417 shown in Fig. 4 and the like may be included in a trained model (for example, trained model 111). For example, Fig. 4 shows an example of the configuration of a model trained by the content labeling device 101.

[0057] First, in image token embedding 401, each DI-dimensional token is converted into a D-dimensional token by a linear transformation from the DI dimension to the D dimension using a learnable parameter. Note that if the values ​​of DI and D are equal, this linear transformation is not necessary.

[0058] Next, in image position embedding 402, a learnable D-dimensional vector is generated for each token. That is, in image position embedding 402, NI position vectors having D-dimensional learnable parameters are generated. Then, in image position embedding 402, the generated position vector is added to each of the NI tokens in the image token sequence, and output.

[0059] In addition, in text token embedding 403 and text position embedding 404, a linear transformation with learnable parameters is applied to the text token string, and a position vector with learnable parameters is added, similar to image token embedding 401 and image position embedding 402 for the image token string. At this time, position coding based on a sine function may be used. In that case, the addition is performed after multiplying by a coefficient that makes the norm magnitudes approximately the same.

[0060] In the attribute token embedding 405, a linear transformation having a learnable parameter is applied to the attribute token string, similar to the processing for the image token string and the text token string, except that the addition of the position vector is not necessary.

[0061] Next, in attribute type embedding 407, information for distinguishing types of attribute data is embedded. The pattern of attribute information may differ depending on the process or time of data generation, external factors such as the freshness of the original data, differences in the terminal used by the user, differences in the form of provision to the user, differences in user groups, differences in systems, and external factors such as other events. When each token of the attribute token string is generated from heterogeneous mixed data having different patterns, in order to distinguish them, for example, in attribute type embedding 407, a learnable D-dimensional attribute type embedding vector is generated for each type of such data and added to the corresponding attribute token. Note that attribute type embedding may not be performed.

[0062] For example, if attribute token A is a token generated from an ID of a certain period HA, and attribute token B is a token generated from an ID of a certain period HB, in order to distinguish the freshness of the data, for example, in attribute type embedding 407, different attribute type embedding vectors TA and TB, each of which has a learnable parameter, are added.

[0063] Also, for example, when attribute token A is a token generated from data such as the number of times the user terminal DA is used, and attribute token B is a token generated from data such as the number of times the user terminal DB is used, for example, different attribute type embedding vectors TA and TB, each having learnable parameters, are added as information for distinguishing between different user terminals in attribute type embedding 407. Similarly, for example, in attribute type embedding 407, information such as differences in user terminals, differences in provision forms, differences in user sets, differences in systems, differences in other external factors, and the like can be embedded.

[0064] Furthermore, when each token in an attribute token string is generated from heterogeneous mixed data having a different pattern, in order to distinguish them, for example, in attribute type embedding 407, embedding of attribute types may be used.

[0065] For example, if attribute token A is a token generated from an ID and attribute token B is a token generated from data such as the number of times of use, the original data types are different and can be considered as data with different patterns. Therefore, for example, in attribute type embedding 407, different attribute type embedding vectors TA and TB, each of which has learnable parameters, are added.

[0066] However, when the data patterns are significantly different, it is possible to use a method other than treating these data as different tokens and distinguishing them by embedding attribute types. For example, when the data patterns are significantly different, it is possible to treat these data as one token, select a large number of dimensions so that both data can be stored, and assign these data to different dimensions of one token.

[0067] For example, if attribute token A is a token generated from data such as the number of times HA is used during a certain period, and attribute token B is a token generated from data such as the number of times HB is used during a certain period, they can be considered as data with the same pattern even though they are from different periods. Therefore, in this case, for example, in attribute type embedding 407, the same attribute type embedding vector T with learnable parameters may be added to both.

[0068] Attribute information may contain data that has a variety of different types and patterns of heterogeneous mixtures due to differences in the process and time of data generation, external factors, etc., and by embedding this information in the tokens using attribute type embedding 407, it is possible to add information on the difference in patterns and distinguish and learn tokens generated from data with different patterns. Note that attribute type embedding 407 is not necessary.

[0069] Through the above process, including the batch size B, the image token sequence is converted into a (B, NI, D)-dimensional tensor, the text token sequence is converted into a (B, NT, D)-dimensional tensor, and the attribute token sequence is converted into a (B, NA, D)-dimensional tensor.

[0070] Next, the process from start of encoding process 408 to execution of a prescribed number of times 415 is an encoding layer, and is executed a prescribed number of times. For example, in the case of 12 layers, it is executed 12 times.

[0071] The image hidden state input / output 409 takes as input the output of the image position embedding 402 in the first execution, and the output of the image encoder 410 in the second and subsequent executions, and outputs it as is to the image encoder 410 and attribute encoder 414 .

[0072] Next, the image encoder 410 encodes the (B, NI, D)-dimensional tensor into a (B, NI, D)-dimensional tensor of the same dimension. This encoding process will be described later.

[0073] As described above, since the image encoder 410 does not change the dimension of the tensor, the output of the image position embedding 402 handled by the image hidden state input / output 409 and the output of the image encoder 410 are tensors of the same (B, NI, D) dimensions. Hereinafter, this (B, NI, D) dimensional tensor handled by the image hidden state input / output 409 is also referred to as the image hidden state.

[0074] As described above, in the case of 12 layers, the processing in the encoding layer involves converting (encoding) the (B, NI, D)-dimensional tensor generated in image position embedding 402 into tensors of the same dimensions 12 times.

[0075] The processing of the text hidden state input / output 411 and the text encoder 412 is similar to that of the image hidden state input / output 409 and the image encoder 410, and performs processing to encode a (B, NT, D)-dimensional tensor a specified number of times. Hereinafter, the (B, NT, D)-dimensional tensor handled by this text hidden state input / output 411 is also referred to as the text hidden state.

[0076] The attribute hidden state input / output 413 and the attribute encoder 414 also perform a process of encoding a (B, NA, D)-dimensional tensor a prescribed number of times. Hereinafter, the (B, NA, D)-dimensional tensor handled by the attribute hidden state input / output 413 is also referred to as the attribute hidden state. However, in the attribute encoder 414, not only the attribute hidden state but also the image hidden state and the text hidden state are used in the encoding process. In this way, in addition to the attribute hidden state, the image hidden state and the text hidden state are also used in the encoding process in the attribute encoder 414.

[0077] If it is determined in the execution of the specified number of times 415 that the processing of the encoding layer has been executed the specified number of times (determination result in the execution of the specified number of times 415: YES), the process proceeds to vector concatenation 416. Note that, if it is determined in the execution of the specified number of times 415 that the processing of the encoding layer has not been executed the specified number of times (determination result in the execution of the specified number of times 415: NO), the processing of the encoding layer (image hidden state input / output 409 to attribute encoder 414) is repeated.

[0078] In the vector concatenation 416, tensors corresponding to the class tokens of the image, text, and attributes are concatenated. That is, from the NI (B, D)-dimensional tensors of the (B, NI, D)-dimensional tensors output by the image encoder 410, tensors (B, D) corresponding to the class tokens are extracted. Similarly, from the output of the text encoder 412 and the attribute encoder 414, tensors of (B, D) dimensions corresponding to the class tokens are extracted. Next, in the vector concatenation 416, the three (B, D)-dimensional tensors are concatenated on the second dimension axis (the axis corresponding to D) to generate a (B, 3D)-dimensional tensor.

[0079] Finally, in the prediction head 417, the (B, 3D)-dimensional tensor generated by the vector concatenation 416 is used as input to generate a T-dimensional predicted vector corresponding to the number of labels T. Each element of the predicted vector becomes the probability value of the corresponding label.

[0080] The prediction head 417 can use a simple prediction function, for example by normalizing the tensors using methods such as layer normalization with learnable parameters, linearly transforming the 3D dimensions to T dimensions, and applying a softmax function, a sigmoid function, etc.

[0081] In addition, when learning the model, the model learning unit 108 calculates a loss value from the T-dimensional predicted vector generated by the prediction head 417 and the T-dimensional correct vector using a loss function such as Cross Entropy Loss or Binary Cross Entropy Loss, and updates the model parameters based on the loss value using the backpropagation method.

[0082] For example, the model learning unit 108 performs the above-mentioned forward propagation and backward propagation processes a predetermined number of times while changing the batches extracted from the learning data, updates the learnable parameters, and stores the parameters in the learned model 111.

[0083] The initial values ​​of trainable parameters may be initialized by setting random values, but it is also possible to efficiently generate highly accurate models by creating pre-trained models for images and text in advance and initializing parameters related to images and text using these models.

[0084] Here, an example of processing of the pre-learning model of the image and text will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of processing of the pre-learning model of the image and text. Each process in Fig. 12 handles image and text tokens independently, but is similar to the corresponding process of the image and text parts in Fig. 4, so detailed description will be omitted.

[0085] 12, the content labeling device 101 generates a pre-trained model for an image and a pre-trained model for a text, and stores the trainable parameters of the image token embedding 1201, the image position embedding 1202, and the image encoder 1210 generated therein in the pre-trained image model 109. When training the model of FIG. 4, the content labeling device 101 loads the parameters of the pre-trained image model 109 into the trainable parameters of the image token embedding 401, the image position embedding 402, and the image encoder 410 as initial values.

[0086] Similarly, for text, the content labeling device 101 stores the learnable parameters of the text token embedding 1203, the text position embedding 1204, and the text encoder 1212 in the pre-trained language model 110. When training the model of Fig. 4, the content labeling device 101 loads the parameters of the pre-trained language model 110 as initial values ​​into the learnable parameters of the text token embedding 403, the text position embedding 404, and the text encoder 412. Note that the loaded parameters may be updated in the model parameter update process of the model training unit 108, or these parameters may be fixed and frozen.

[0087] Note that the pre-learning is not limited to the example in FIG. 12, and may be performed using a task different from the labeling task, or may use different data. Also, an unsupervised learning method without a correct answer label may be used. Various pre-learning methods are known, such as pre-learning methods used in Transformer learning, including ViT (Vision Transformer) and BERT (Bidirectional Encoder Representations from Transformers), and Contrastive Learning, and these methods may be utilized.

[0088] Next, the processing of the image encoder 410, the text encoder 412, and the attribute encoder 414 will be described with reference to Fig. 5 and Fig. 7. Fig. 5 is a diagram showing an example of the processing of the image encoder and the text encoder. Fig. 7 is a diagram showing an example of the processing of the attribute encoder.

[0089] First, the image encoder 410 will be described with reference to Fig. 5. Normalization 501 to residual addition 509 shown in Fig. 5 show an example of processing by the image encoder 410. For example, the image encoder 410 executes processing corresponding to normalization 501 to residual addition 509. Note that the method described here uses a method based on ViT as an example, but other configurations may be used.

[0090] In normalization 501, the image hidden state, which is a (B, NI, D)-dimensional tensor, is normalized using a method such as layer normalization with learnable parameters. The next steps from attention embedding 502 to attention weighting 505 are attention calculations.

[0091] First, in the attention embedding 502, vectors called Q (Query), K (Key), and V (Value) are generated. First, for each of Q, K, and V, a linear transformation matrix with learnable parameters from D dimensions to D dimensions is held. This is a D×D matrix. For each D-dimensional token in the image hidden state, this linear transformation is applied to obtain the same D-dimensional token. For one token, three D-dimensional tokens are generated corresponding to the linear transformations for Q, K, and V. In the following, the three D-dimensional token sequences generated at this time will be called Q, K, and V, respectively.

[0092] An example of vector generation for the above Q, K, and V is shown in Fig. 13. Fig. 13 is a diagram showing an example of vector generation. By performing linear transformations corresponding to Q, K, and V on the NI vectors in the image hidden state, NI D-dimensional vectors corresponding to Q, K, and V are generated.

[0093] Next, in head division 503, each of tokens Q, K, and V is divided into H tokens based on a predetermined number of heads H. The number of dimensions of the divided tokens is D / H dimensions. It is desirable to set D and H so that D is divisible by H. As a result, H vectors of NI H / D dimensions are generated for each of Q, K, and V. The NI H / D dimensions corresponding to the h-th division are set as Qh. Vector information 1301, vector information 1302, vector information 1303, and vector information 1304 show how this is done. FIG. 13 shows the case of H=3. In actual applications, for example, H may be set to 12 or the like.

[0094] Next, in the attention inner product 504, the inner product of the vector Qh and the vector Kh corresponding to each head is calculated. In FIG. 13, as shown in the matrix information 1305, the inner product of the vector Qh and the vector Kh corresponding to each head is calculated. The ij-th component of this inner product value matrix is ​​the inner product value of the i-th vector of Qh and the j-th vector of Kh. Note that in order to prevent this inner product value from becoming too large, the value obtained by dividing the inner product value by the square root of the number of dimensions D may be set as aij.

[0095] Next, in the attention weighting 505, Vh is weighted based on the inner product value, and the result is generated as a weighted vector. In FIG. 13, as shown in the weighted vector data 1306, a weighted vector is generated by weighting Vh based on the inner product value. For example, for each row i of the inner product value matrix, a softmax function is applied to the vector (ai1, ai2, ..., aiNI). Next, the normalized vector is used as a weight, and a weighted sum of NI vectors of Vh is taken to be the i-th weighted vector. This weighted vector is defined as Sh. The weighted vector V is generated by combining S1, S2, ..., SH in the opposite manner to when they were divided. This is composed of NI tokens of D dimension.

[0096] For example, the content labeling device 101 applies the above process to each sample in the batch, so that the weight vector V is a (B, NI, D)-dimensional tensor, and the dimension of the weight vector V is the same as the dimension of the image hidden state.

[0097] Next, in residual addition 506, this weight vector and the image hidden state are added and output. After that, in normalization 507, normalization is performed using a method such as layer normalization with learnable parameters. Furthermore, in MLP 508, conversion is performed to a tensor of the same dimension using a feedforward network or the like. Finally, in residual addition 509, the output tensor of residual addition 506 is added to the tensor generated by MLP 508, and output to the next process. The result output at this time is also a (B, NI, D)-dimensional tensor with the same dimensions as the image hidden state.

[0098] Here, an example of the processing of the MLP 508 will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the processing of the MLP. The processing of the MLP 508 includes the processing of linear transformation 601 to linear transformation 603 shown in Fig. 6. For example, the MLP 508 is a multilayer perceptron.

[0099] First, in linear transformation 601, each token is transformed from D dimension to a larger E dimension by a linear transformation having learnable D×E dimension parameters. Next, in activation function application 602, an activation function such as ReLU (Rectified Linear Unit) is applied. In linear transformation 603, each token is transformed from E dimension to D dimension by a linear transformation having learnable E×D dimension parameters.

[0100] Through the above encoding process, the image hidden state is encoded into a tensor of the same dimensions (B, NI, D).

[0101] Next, the text encoder 412 will be described with reference to Fig. 5. Attention embedding 511 to normalization 522 shown in Fig. 5 are an example of processing by the text encoder 412. For example, the text encoder 412 executes processing corresponding to attention embedding 511 to normalization 522. Note that the method described here uses a method based on BERT as an example, but other configurations may be used.

[0102] The processes from attention embedding 511 to attention weighting 514 are similar to those from attention embedding 502 to attention weighting 505 in the image encoder. Next, in residual addition 515, the text hidden state and the output of attention weighting 514 are added together and output to the next process. In linear transformation 516, the D-dimensional vector of each token is transformed by a D×D-dimensional linear transformation with learnable parameters.

[0103] Next, normalization 517 normalizes using a method such as layer normalization with learnable parameters. Linear transformation 518 transforms the D-dimensional vector of each token using a D×D dimensional linear transformation with learnable parameters.

[0104] In the activation function application 519, an activation function such as GELU (Gaussian Error Linear Units) is applied. In the residual addition 520, the output of the normalization 517 and the output of the activation function application 519 are added together and output to the next process. In the linear transformation 521, the D-dimensional vector of each token is transformed by a D×D-dimensional linear transformation having a learnable parameter. Finally, in the normalization 522, normalization is performed using a method such as layer normalization having a learnable parameter.

[0105] Through the above encoding process, the text hidden state is encoded into a tensor of the same dimensions (B, NT, D).

[0106] Next, the processing of the attribute encoder 414 will be described with reference to Fig. 7. Vector combination 701 to residual addition 710 shown in Fig. 7 show an example of processing by the attribute encoder 414. For example, the attribute encoder 414 executes processing corresponding to vector combination 701 to residual addition 710. Note that the processing configuration described here is just an example, and another configuration using a unidirectional attention structure (also simply referred to as "unidirectional attention") may also be used.

[0107] First, in the vector combination 701, three tensors, namely, a (B, NI, D)-dimensional image hidden state tensor, a (B, NT, D)-dimensional text hidden state tensor, and a (B, NA, D)-dimensional attribute hidden state tensor output from the image hidden state input / output 409, text hidden state input / output 411, and attribute hidden state input / output 413, are combined on the axis of the second dimension (the dimension corresponding to NI, NT, and NA). As a result, N=NI+NA+NT tokens are generated for each sample, and a (B, N, D)-dimensional tensor is generated. This is called a combined vector. This combined vector combines the image token, text token, and attribute token.

[0108] Next, in normalization 702, the (B, N, D)-dimensional tensor generated in vector combination 701 is normalized by a method such as layer normalization with learnable parameters. Next, in unidirectional attention embedding 703, vectors called Q, K, and V are generated in the same way as in attention embedding 502. However, Q is generated only for NA tokens corresponding to the attribute token. K and V are generated for all N tokens. The process of head division 704 thereafter is the same as that of head division 503. An example of this process is shown in vector information 1401 and vector information 1402 in FIG. 14. FIG. 14 is a diagram showing an example of vector generation. As shown in vector information 1402, Q is NA corresponding to the attribute token, but K and V are N. Hereinafter, Qh, Kh, and Vh are the vectors of Q, K, and V divided into h-th, respectively.

[0109] Next, in the unidirectional attention dot product 705, the dot product of Qh and Kh is calculated. However, the dot product is not calculated between all tokens, but between only attribute tokens and image tokens, text tokens, and attribute tokens. In other words, the dot product is calculated between tokens corresponding to attributes of Qh and all tokens of Kh. An example of the matrix format of this dot product value is shown in matrix information 1403.

[0110] Next, in unidirectional attention weighting 706, a softmax function is applied to the vector (ai1, ai2, ..., aiN) in the i-th row of the inner product matrix. Next, in unidirectional attention weighting 706, this vector is used as a weight to take a weighted sum of N tokens in V to generate a D-dimensional weight vector. This generates NA weight vectors. This is applied to each sample, and a (B, NA, D)-dimensional tensor for the batch size is generated.

[0111] Next, in the residual addition 707, the output of the unidirectional attention weighting 706 is added to the output of the attribute hidden state input / output 413, and the result is output to the next process. The processes from normalization 708 to residual addition 710 are the same as the processes from normalization 507 to residual addition 509 in the image encoder.

[0112] Through the above encoding process, the attribute hidden state is encoded into a tensor of the same dimensions (B, NI, D).

[0113] In a normal self-attention structure, all tokens are encoded by calculating the inner product between all tokens and taking weights. In the configuration of the present application (e.g., the content labeling device 101, etc.), image and text tokens are weighted to attribute tokens based on the inner product value, but image tokens and text tokens are not weighted. This unidirectional configuration does not cause changes in the image occlusion state or the text occlusion state. That is, the content labeling device 101 can embed three types of multimodal relationships of image, text, and attribute in the attribute encoder without causing changes in the image model or text model. In the above example, in the model (trained model 111, etc.) learned by the content labeling device 101, the model part corresponding to the image or text corresponds to the first partial model, and the model part corresponding to the attribute that changes more over time than the image or text corresponds to the second partial model. For example, the first partial model has a permanent model structure that changes less over time. For example, the second partial model has a non-permanent model structure that changes more over time. For example, in the model example shown in Fig. 4, a portion corresponding to an image token string corresponds to the first partial model. Also, for example, in the model example shown in Fig. 4, a portion corresponding to a text token string corresponds to the first partial model. Also, for example, in the model example shown in Fig. 4, a portion corresponding to an attribute token string corresponds to the second partial model.

[0114] With the above-mentioned configuration, the content labeling device 101 can obtain several effects. For example, in media processing, particularly for images and texts, a method is widely used in which a highly versatile pre-learning model is created in advance, and then learning is performed according to the task by fine tuning. Generally, pre-learning requires a large amount of learning data and a large amount of calculation, and is therefore expensive, but fine tuning can be performed at a relatively low cost. With such a configuration, a base model is learned by pre-learning, and learning according to the task can be performed with high accuracy at a relatively low cost.

[0115] In the configuration of the present application (e.g., the content labeling device 101, etc.), since unidirectional attention does not affect the parameters or hidden states of the image or text models, a pre-trained model created solely for images or a pre-trained model created solely for text can be used. When a token of a different modality is weighted to the tokens of an image or text due to attention, the hidden state values ​​of the image model or text model are significantly changed, making it difficult to use a pre-trained model. As a result, in the configuration of the present application (e.g., the content labeling device 101, etc.), while utilizing a pre-trained model created in advance for images or text, the cross-modal relationship between images, text, and attributes can be learned with a highly accurate attention structure that can learn wide-area features.

[0116] In addition, the image pre-learning model can be trained using image data only, and the text pre-learning model can be trained using text data only. Compared with data in which multiple modalities such as images, text, and attributes are linked, image data only or text data only is often available in large quantities. In the configuration of the present application (e.g., the content labeling device 101, etc.), a pre-learning model that can be trained in a single modality can be used, so high accuracy can be expected not only by training only from multimodal data, but also by pre-training with a large amount of single-modal data.

[0117] Furthermore, user-derived data such as attribute data is prone to change over time, and frequent model retraining is desirable. On the other hand, image and text models are more versatile and less prone to model degradation over time than these data. When modals with less degradation and modals with greater degradation are mixed, pre-training modals with less degradation, such as images and text, allows for the use of pre-trained models of modals with less degradation, making it possible to retrain models at low cost.

[0118] In addition, if the attribute data requires high-frequency re-learning of the model, but the image and language portions require low frequency, then since unidirectional attention does not make any changes to the parameters corresponding to the image or text, it is possible to fix the image and text parameters or lower the learning rate and re-learn only the attribute data.

[0119] In this way, the structure of unidirectional attention allows for high modularity learning of the attribute part. In addition, in the process of the model learning unit 108, dropout processing may be provided as appropriate to increase the robustness of learning.

[0120] Next, the inference unit 112 performs preprocessing on the content to be inferred input from the input / output unit 115. The preprocessing method is the same as that of the learning data preprocessing unit 107. However, if random noise or the like is added by the learning data preprocessing unit 107 in order to improve the robustness of learning, the inference data preprocessing unit 113 does not need to apply such noise processing.

[0121] Next, the inference processing unit 114 calculates a predicted label from the input using the forward propagation calculation method, similar to the model learning unit 108. However, the dropout process for improving the robustness of learning is not applied.

[0122] The data input unit 116 receives content data from the system or a user via an input device or a communication device. The label output unit 117 returns the predicted label of the inference processing unit 114 to the system or the user via an output device or a communication device. For example, the label output unit 117 transmits the predicted label of the inference processing unit 114 to a device included in the system or a user terminal used by the user via the output device or a communication device.

[0123] In the above description, the dimension of the vector in the image hidden state, the dimension of the vector in the text hidden state, and the dimension of the vector in the attribute hidden state are the same number of dimensions D. However, different numbers of dimensions suitable for each modality may be set. An example of this point will be described with reference to FIG. 8. FIG. 8 is a diagram showing another example of the processing of the attribute encoder. Note that in FIG. 8, the same points as in FIG. 7 will not be described as appropriate. In the example shown in FIG. 8, in order to match the number of dimensions in the calculation of unidirectional attention, a process of matching the number of dimensions to the dimension of the vector in the attribute hidden state is performed in dimension transformation 801 and dimension transformation 802. A linear transformation matrix having a learnable parameter may be used for the dimension transformation such as the dimension transformation 801 and the dimension transformation 802. This allows the content labeling device 101 to select a number of dimensions suitable for each modality.

[0124] In the above explanation, the number of encoding layers, i.e., the number of encoding times, is the same for images, text, and attributes. However, when adjusting the number of layers for each modality, it is sufficient to specify a layer that calculates unidirectional attention and a layer that calculates independently, and to separate the processing methods. Independently calculated layers do not pass hidden states from the image hidden state input / output 409 and text hidden state input / output 411 to the attribute encoder 414, and calculate attention within the attribute token.

[0125] An example is shown in Figure 15. Figure 15 is a diagram showing an example of the model configuration. In the example shown in Figure 15, there are 8 layers for images, 6 layers for text, and 5 layers for attributes. Only the last 4 layers are used to calculate unidirectional attention, and the other layers are processed independently for each modality. Note that in the example in Figure 15, the last 4 layers are associated, but other association methods may be used. This makes it possible to select an appropriate total number for each modality.

[0126] In the example of Figure 4, the image and language models are explained using the attention structure, but the image and language models may not have the attention structure. In that case, the attribute tokens are encoded using the hidden state of the image and language models.

[0127] In addition, the attribute information model may use only class tokens without using data. In that case, attribute data etc. are not used, and the data is bimodal, consisting of only images and language, but in the class token, the class token of the attribute is encoded using the hidden state of the image and language, so that the cross-modal relationship between the image and language can be learned in this token.

[0128] In addition, although the above description has been given using an example of image, text, and attribute data, the present invention can be used in the same way for image and attribute data, text and attribute data, and other modalities other than image, text, and attribute. In such cases, the hidden state of a highly versatile modal with little time-varying change can be used to calculate unidirectional attention that encodes the hidden state of a modal with large time-varying change.

[0129] The content labeling device 101 calculates the relationship between modal tokens that change little over time, such as images and language, and modal tokens that change much over time, such as attribute information, while using a unidirectional attention structure that encodes only tokens that correspond to attribute information.This makes it possible to incorporate cross-modal relationships in models of modals that change much over time and learn them without destroying the parameters and states of modals that change little over time.

[0130] As a result, the content labeling device 101 can use a pre-trained model trained in a single modality in a modality with little change over time. That is, the content labeling device 101 can utilize training data of a single modality. Furthermore, the content labeling device 101 can reduce or fix the learning rate in a modality with little change over time, and re-train only the part with large change over time without destroying the parameters or state of the modality with little change over time. As a result, the content labeling device 101 can perform efficient cross-modal learning in a modality with little change over time by utilizing training data of a single modality and reusing a model, while combining it with a modality that requires frequent re-training.

[0131] 〔others〕 Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. The above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0132] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0133] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0134] For example, the above-mentioned server device 10 may be realized by a plurality of server computers, and depending on the function, an external platform or the like may be called by an API or network computing, etc., so that the configuration can be flexibly changed.

[0135] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.

[0136] Moreover, the above-mentioned "section, module, unit" can be read as "means" or "circuit", etc. For example, an inference section can be read as an inference means or an inference circuit. [Explanation of symbols]

[0137] 101 Content labeling device 106 Model Generation Unit 107 Training data preprocessing section 108 Model Learning Department 112 Reasoning Department 113 Inference data preprocessing unit 114 Inference processing unit 115 Input / output section 116 Data Entry Section 117 Label output unit

Claims

1. a model learning unit that learns a model using content data and a correct label assigned to the content data, and generates a learned model; an inference unit that generates a predicted label of content using the model trained by the model training unit for data to be inferred; Equipped with The model learning unit includes: The method includes: learning a model including a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, the model encoding the hidden state of the second partial model using a hidden state of the first partial model and a hidden state of the second partial model in the unidirectional attention structure. A content labeling device comprising:

2. The model includes a model in which the number of dimensions of the hidden state of the first partial model is different from the number of dimensions of the second partial model, and before calculating the unidirectional attention structure, a dimensional transformation is performed on the hidden state of the first partial model by applying a linear change having a learnable parameter from the number of dimensions of the hidden state of the first partial model to the number of dimensions of the second partial model.

2. The content labeling device according to claim 1.

3. The model has an attention structure in the first partial model.

2. The content labeling device according to claim 1.

4. The model learning unit includes: The second partial model has a plurality of layers for calculating an attention structure, a part of the layers is selected, and a correspondence is made with any of the hidden states of the first partial model, and unidirectional attention is calculated for the selected layers, and attention using only the hidden states of the second partial model is calculated for layers other than the selected layers.

2. The content labeling device according to claim 1.

5. The model learning unit includes: In order to handle various types of data in the second partial model, attribute types that distinguish differences in data patterns are embedded in tokens.

2. The content labeling device according to claim 1.

6. The model has only class tokens in the second partial model.

2. The content labeling device according to claim 1.

7. The model learning unit includes: Use the user ID as the token in the second partial model.

2. The content labeling device according to claim 1.

8. The model learning unit includes: In the second partial model, the user ID is divided and used as multiple tokens.

2. The content labeling device according to claim 1.

9. A model training process for training a model using content data and a correct label assigned to the content data to generate a trained model; an inference step of generating a predicted label of content using the model trained in the model training step for data to be inferred; Including, The model learning step includes: The method includes: learning a model including a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, the model encoding the hidden state of the second partial model using a hidden state of the first partial model and a hidden state of the second partial model in the unidirectional attention structure. A content labeling method comprising:

10. A model learning procedure for learning a model using content data and a correct label assigned to the content data to generate a learned model; an inference step of generating a predictive label of content using the model trained by the model training step for data to be inferred; on the computer, The model learning procedure includes: The method includes: learning a model including a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, the model encoding the hidden state of the second partial model using a hidden state of the first partial model and a hidden state of the second partial model in the unidirectional attention structure. A content labeling program comprising:

11. a model learning unit that learns a model using content data and a correct label assigned to the content data, and generates a learned model; an inference unit that generates a predicted label of content using the model trained by the model training unit for data to be inferred; Equipped with The model learning unit includes: The method includes: learning a model including a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, the model encoding the hidden state of the second partial model using a hidden state of the first partial model and a hidden state of the second partial model in the unidirectional attention structure. A model learning device comprising:

12. The model includes a model in which the number of dimensions of the hidden state of the first partial model is different from the number of dimensions of the second partial model, and before calculating the unidirectional attention structure, a dimensional transformation is performed on the hidden state of the first partial model by applying a linear change having a learnable parameter from the number of dimensions of the hidden state of the first partial model to the number of dimensions of the second partial model. The model learning device according to claim 11 .

13. The model has an attention structure in the first partial model. The model learning device according to claim 11 .

14. The model learning unit includes: The second partial model has a plurality of layers for calculating an attention structure, a part of the layers is selected, and a correspondence is made with any of the hidden states of the first partial model, and unidirectional attention is calculated for the selected layers, and attention using only the hidden states of the second partial model is calculated for layers other than the selected layers. The model learning device according to claim 11 .

15. The model learning unit includes: In order to handle various types of data in the second partial model, attribute types that distinguish differences in data patterns are embedded in tokens. The model learning device according to claim 11 .

16. The model has only class tokens in the second partial model. The model learning device according to claim 11 .

17. The model learning unit includes: Use the user ID as the token in the second partial model. The model learning device according to claim 11 .

18. The model learning unit includes: In the second partial model, the user ID is divided and used as multiple tokens. The model learning device according to claim 11 .

19. A model training process for training a model using content data and a correct label assigned to the content data to generate a trained model; an inference step of generating a predicted label of content using the model trained in the model training step for data to be inferred; Including, The model learning step includes: The method includes: learning a model including a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, the model encoding the hidden state of the second partial model using a hidden state of the first partial model and a hidden state of the second partial model in the unidirectional attention structure. A model learning method comprising:

20. A model learning procedure for learning a model using content data and a correct label assigned to the content data to generate a learned model; an inference step of generating a predictive label of content using the model trained by the model training step for data to be inferred; Run the following on your computer: The model learning procedure includes: The method includes: learning a model including a first partial model; and a second partial model that changes more significantly over time than the first partial model and has a unidirectional attention structure, the model encoding the hidden state of the second partial model using a hidden state of the first partial model and a hidden state of the second partial model in the unidirectional attention structure. A model learning program characterized by:

Images