Training program, training method and information processing device

Abstract

In the present invention, for first data that includes a first object feature amount and position information for each of a plurality of subjects in first image data, at least one item of second data is generated by replacing at least one first object feature amount of the plurality of subjects with a second object feature amount that is obtained for at least one other object that has been classified as the same class as the subjects and that is in at least one item of second image data other than the first image data, and at least one item of the second data is input to an encoder, thereby training the encoder.

Description

[Technical Field]

[0001] The present invention relates to a training program, a training method, and an information processing device. [Background technology]

[0002] There is a known technology for generating a knowledge graph called a scene graph from image data. A scene graph contains information about the relationships between multiple objects in the image data. When generating a scene graph, a machine learning model is trained to calculate feature quantities about the relationships between the objects using supervised data (labeled data) that contains correct labels about the relationships between the objects. From the perspective of increasing the amount of information contained in a scene graph, it is desirable to calculate concrete relationships between multiple objects rather than abstract relationships. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-508737 Summary of the Invention [Problem to be solved by the invention]

[0004] However, depending on the content of the prepared supervised data (labeled data), it may not be easy to improve the classification accuracy of the relationships by reflecting the specific relationships between objects.

[0005] In one aspect, an object of the present invention is to improve the accuracy of relationship classification by reflecting specific relationships between multiple objects in image data. [Means for solving the problem]

[0006] In one aspect, the training program causes a computer to execute a process of replacing, for first data including first object features and position information of each of a plurality of objects in first image data, the first object feature of at least one of the plurality of objects with a second object feature obtained for at least one other object classified into the same class as the object in at least one second image data different from the first image data, to generate at least one second data, and inputting at least one of the second data into an encoder to train the encoder. The training program causes the computer to execute a process of performing machine learning in a process of training the encoder to increase the degree of agreement between a first relationship feature about the relationship between the plurality of objects, which is obtained by inputting the first data into the encoder, and a second relationship feature about the relationship between the plurality of objects, which is obtained by inputting the second data into the encoder. [Effects of the Invention]

[0007] In one aspect, it is possible to improve the accuracy of classifying relationships that reflect specific relationships between multiple objects in image data. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a training process for an encoder in the related art. [Figure 2] FIG. 1 is a block diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. [Figure 3] FIG. 1 is a diagram illustrating a definition of a scene graph. [Figure 4] FIG. 1 is a diagram schematically illustrating an example of a scene graph. [Figure 5] FIG. 1 illustrates an example of self-supervised learning. [Figure 6] FIG. 10 is a diagram illustrating an example of contrastive learning. [Figure 7] 3 is a block diagram schematically illustrating a first example of a software configuration in a training phase of an encoder performed by the information processing device illustrated in FIG. 2. FIG. [Figure 8] 3 is a diagram illustrating an example of a training process for an object detector performed by the information processing device illustrated in FIG. 2. FIG. [Figure 9] 3 is a diagram illustrating an example of a training process of an encoder performed by the information processing device illustrated in FIG. 2. FIG. [Figure 10]FIG. 10 is a diagram illustrating an example of a process using contrastive learning in the training process of the encoder in FIG. 9. [Figure 11] 3 is a block diagram schematically illustrating a second example of a software configuration in a training phase of an encoder performed by the information processing device illustrated in FIG. 2. FIG. [Figure 12] 3 is a block diagram schematically illustrating an example of a software configuration in a training phase of a classifier by the information processing device illustrated in FIG. 2. FIG. [Figure 13] 3 is a diagram illustrating an example of a classifier training process performed by the information processing device illustrated in FIG. 2. FIG. [Figure 14] 3 is a block diagram schematically illustrating an example of a software configuration in an inference phase by the information processing device illustrated in FIG. 2. FIG. [Figure 15] 3 is a flowchart illustrating an example of a training process for an encoder in a training phase of the information processing device shown in FIG. 2. [Figure 16] 3 is a flowchart illustrating an example of a classifier training process in a training phase of the information processing device shown in FIG. 2. [Figure 17] 3 is a flowchart showing an example of a scene graph creation process in an inference phase of the information processing device shown in FIG. 2. [Figure 18] 3 is a diagram showing an example of a result of the reproducibility of a relationship label obtained by the information processing device shown in FIG. 2. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] [A] Related Examples FIG. 1 is a diagram illustrating a training process for an encoder 23 in the related art.

[0010] In a related technique, the encoder 23 is trained by supervised learning using a deep neural network (DNN).

[0011] The input image data is labeled data 30. The labeled data 30 is also called supervised data.

[0012] The labeled data 30 includes image data 31. In the example shown in FIG. 1, the image data 31 contains multiple objects 32a (cow) and 32b (woman). The labeled data 30 may include object labels 33a and 33b and a relationship label 34. The object labels 33a and 33b are information indicating the types (classes) of the multiple objects 32a and 32b in the image data 31. In FIG. 1, the object label 33a is a "cow" and the object label 33b is a "woman." The relationship label 34 is information indicating the relationship between the multiple objects 32a and 32b, and is "feed" in FIG. 1.

[0013] The trained object detector 21 acquires object labels 33a and 33b and location information 36a and 36b for each of the objects 32a and 32b from the labeled data 30. The objects 32a and 32b may include a subject and an object. In this example, the object label 33a and location information 36a are acquired for the object "cow." Similarly, the object label 33b and location information 36b are acquired for the subject "woman."

[0014] The feature extractor 22 calculates a first object feature 37a for the object 32a (object) and a first object feature 37b for the object 32b (subject) based on the position information 36a and 36b of the objects 32a and 32b, respectively.

[0015] The encoder 23 outputs a relationship label classification result 38 based on the first object feature amount 37a of the object and the first object feature amount 37b of the subject. In this example, the encoder 23 outputs the relationship label classification result 38 of "feed". The encoder optimization unit 24 compares the relationship label classification result 38 output from the encoder 23 with the correct relationship label 34 of the labeled data 30, and trains the encoder 23 to reduce the error between the classification result 38 and the relationship label 34.

[0016] In the related technology, it is necessary to prepare labeled data 30. If the prepared labeled data 30 has biased correct relationship labels 34, it is not easy to improve the training accuracy of the encoder 23. For example, there is a case where the frequency of appearance of abstract relationships such as "on" or "have" is higher than the frequency of appearance of concrete relationships such as "sitting on" or "walking on." In this case, it is not easy to assign relationship labels taking the concrete relationships into consideration.

[0017] One possible approach is to consider specific relationships by increasing the variety of labeled data 30. However, increasing the number of pieces of labeled data 30 having relationship labels 34 increases the burden on workers.

[0018] Therefore, it is conceivable to employ self-supervised learning, particularly contrastive learning, in which the encoder 23 is trained using not only labeled data 30 but also unlabeled data.

[0019] [B] Embodiment An embodiment will be described below with reference to the drawings. However, the embodiment described below is merely an example, and is not intended to exclude various modifications or application of techniques not explicitly stated in the embodiment. In other words, this embodiment can be implemented with various modifications within the scope of its purpose. Furthermore, each figure does not intend to include only the components shown in the figure, but may include other functions, etc.

[0020] In the following drawings, the same reference numerals denote similar parts, and therefore the description thereof will be omitted.

[0021] [B-1] Configuration example FIG. 2 is a block diagram schematically illustrating an example of the hardware configuration of the information processing device 1 according to the embodiment.

[0022] 2, the information processing device 1 is a computer and includes a processor 11, a memory unit 12, a display control unit 13, a storage device 14, an input interface (IF) 15, an external recording medium processing unit 16, and a communication IF 17.

[0023] The memory unit 12 is an example of a storage unit, and is illustratively a read-only memory (ROM) or a random access memory (RAM). A program such as a basic input / output system (BIOS) may be written to the ROM of the memory unit 12. The software program in the memory unit 12 may be read and executed by the processor 11 as appropriate. The RAM of the memory unit 12 may be used as a temporary storage memory or a working memory.

[0024] The display control unit 13 is connected to the display device 130 and controls the display device 130. The display device 130 is a liquid crystal display, an organic light-emitting diode (OLED) display, a cathode ray tube (CRT), an electronic paper display, or the like, and displays various information to an operator, etc. The display device 130 may be combined with an input device, and may be, for example, a touch panel.

[0025] The storage device 14 is a storage device with high IO performance, and may be, for example, a dynamic random access memory (DRAM), a solid state drive (SSD), a storage class memory (SCM), or a hard disk drive (HDD).

[0026] The input IF 15 may be connected to input devices such as a mouse 151 and a keyboard 152, and may control the input devices such as the mouse 151 and the keyboard 152. The mouse 151 and the keyboard 152 are examples of input devices, and an operator performs various input operations via these input devices.

[0027] The external recording medium processing unit 16 is configured to allow a recording medium 160 to be attached. The external recording medium processing unit 16 is configured to be able to read information recorded on the recording medium 160 when the recording medium 160 is attached. In this example, the recording medium 160 is portable. For example, the recording medium 160 is a flexible disk, an optical disk, a magnetic disk, a magneto-optical disk, a semiconductor memory, or the like.

[0028] The communication IF 17 is an interface that enables communication with an external device.

[0029] The processor 11 is a processing device that performs various controls and calculations. The processor 11 may include a CPU (Central Processing Unit). The processor 11 may also include a dGPU (Discrete Graphics Processing Unit). The dGPU refers to a GPU that exists on a graphics chip or graphics board that is independent of the CPU. The processor 11 realizes various functions by executing an operating system (OS) and programs loaded into the memory unit 12. The processor 11 may also realize the function of a control unit 100, which will be described later.

[0030] The device for controlling the overall operation of the information processing device 1 is not limited to a CPU or a dGPU, and may be, for example, any one of an MPU, a DSP, an ASIC, a PLD, and an FPGA. The device for controlling the overall operation of the information processing device 1 may also be a combination of two or more of a CPU, an MPU, a DSP, an ASIC, a PLD, and an FPGA. MPU is an abbreviation for Micro Processing Unit, DSP is an abbreviation for Digital Signal Processor, and ASIC is an abbreviation for Application Specific Integrated Circuit. PLD is an abbreviation for Programmable Logic Device, and FPGA is an abbreviation for Field Programmable Gate Array.

[0031] FIG. 3 is a diagram for explaining the definition of a scene graph 300. The scene graph 300 is a directed graph in which objects (subjects) in an image or video are used as nodes and relationships between the objects are expressed as edges. Images and videos are examples of image data. The object (subject) may be a living thing, including a person, an article, or a part of a living thing or article.

[0032] FIG. 4 is a diagram schematically illustrating an example of a scene graph 300. The dataset illustrated in FIG. 4 includes image data 200 and a scene graph 300. In FIG. 4, objects in the image data 200 include a person, a computer, a table, a chair, and a window. Position information 211-1 to 211-5 is given for each of the objects. The position information 211 may include coordinates (x, y), height, and width information. The position information 211 may be expressed as a bounding box. The bounding box is a rectangular box that surrounds the object.

[0033] The scene graph 300 includes an object label 321-1 (person), an object label 321-2 (computer), an object label 321-3 (table), an object label 321-4 (chair), and an object label 321-5 (window). The relationships between the objects are indicated by directed edges 323-1 to 323-4. The directed edges 323 may be lines with arrows.

[0034] As information indicating the relationship between a person and a computer, relationship label 322-1 indicates "using," relationship label 322-2 between a computer and a table indicates "on," and relationship label 322-3 between a person and a chair indicates "sitting on."

[0035] The relationship label 322 can increase the information content of the scene graph 300 by indicating a specific relationship (for example, sitting on or standing on) rather than an abstract relationship (for example, on).

[0036] The information processing device 1 in this embodiment improves the accuracy of classifying relationships by reflecting specific relationships between multiple objects in image data. To achieve this, the information processing device 1 uses a self-supervised learning technique, particularly a contrastive learning technique.

[0037] Fig. 5 is a diagram showing an example of self-supervised learning. In Fig. 5, as a simple example, a case will be described in which image data showing a dog is input as input data 51 and an object label "dog" is output as output.

[0038] When input data 51 is input to the encoder 43, the encoder 43 outputs a latent vector (object feature vector in FIG. 5) Z. The latent vector Z is input to the classifier 44, which outputs a label.

[0039] Self-supervised learning has a two-stage training (learning) process. In the first stage, the encoder 43 is trained using unlabeled data. Unlabeled data requires less effort to generate than labeled data. Therefore, the amount of data can be increased when training using unlabeled data compared to when training using labeled data. This allows the encoder 43 to learn many variations, thereby improving training accuracy (learning accuracy).

[0040] Fig. 6 is a diagram showing an example of contrastive learning. Contrastive learning is a type of self-supervised learning. As a simple example, Fig. 6 explains a case where image data showing a dog is input as input data 51 and an object label "dog" is output as output.

[0041] Two types of data extension (T1, T2) are applied to the input x (input data 51) to generate two extension data x i ~T1(x) and x j∼T2(x) is obtained. Data augmentation can be achieved by, for example, applying translation, rotation, scaling, up / down flipping, left / right flipping, brightness adjustment, and a combination of these to the original image.

[0042] The data obtained by the two types of data extension are respectively encoded as encoder 43(f Φ ) are input to the two latent vectors z i =f Φ (x i )(first latent vector),z j =f Φ (x j ) (second latent vector) is obtained.

[0043] 2 x extended data i ~T1(x) and x j ~T2(x) is the data that has been transformed differently without changing the essence of the object. Two latent vectors z i =f Φ (x i ),zj=f Φ (x j ) are identical or similar due to the fact that the nature of the object has not changed. Therefore, in contrastive learning, two latent vectors z i =f Φ (x i ),z j =f Φ (x j The encoder 43 is machine-learned to increase the degree of match (similarity) between the two. In one example, the loss function L φ =‐sim(z i ,z j ) is calculated, and the parameter φ may be updated so that this value is minimized.

[0044] 5 and 6, the case of training the encoder 43 that outputs object labels of objects has been described, but in the information processing device 1 of this embodiment, contrastive learning is applied to training the encoder 143 that calculates the relationships between multiple objects. The encoder 143 is an example of an encoder that calculates the relationships between multiple objects.

[0045] [B-1-1] Training Phase FIG. 7 is a block diagram schematically illustrating a first example of a software configuration in the training phase of the encoder performed by the information processing device 1 shown in FIG.

[0046] The control unit 100 may include a labeled data acquisition unit 101, an object label acquisition unit 102, a location information acquisition unit 103, a detection confidence acquisition unit 104, a first object feature acquisition unit 105, a pair creation unit 106, an unlabeled data acquisition unit 108, a second object feature acquisition unit 109, and an encoder optimization unit 110.

[0047] The control unit 100 realizes a learning process (training process) in machine learning using the training data. That is, the information processing device 1 functions as a training device that trains a machine learning model using the control unit 100.

[0048] In this example, the object detector 121, the object feature extractor 122, and the encoder 143 are an example of a machine learning model. The machine learning model may be, for example, a deep learning model (deep neural network). The neural network may be a hardware circuit, or may be a virtual network using software that connects layers virtually constructed on a computer program by the processor 11 or the like.

[0049] The labeled data acquisition unit 101 acquires labeled data 30. The labeled data 30 may be a data set including image data, object labels, and relationship labels. The acquired labeled data 30 may be input to the object detector 121.

[0050] The object detector 121 may be an existing object detector based on a DNN. For example, the object detector 121 may be a Faster Region-based Convolutional Neural Network (Faster RCNN) or a DETR (DEtection Transformer). A detailed description of the object detector 121 itself will be omitted.

[0051] The object label acquisition unit 102 acquires object labels 33a and 33b of the objects 32a and 32b in the image data, respectively, from the labeled data 30. Note that the labeled data 30, image data 31, objects 32a and 32b, object labels 33a and 33b, relationship label 34, position information 36a and 36b, and object feature amounts 37a and 37b may be similar to those described in the related art shown in FIG.

[0052] The position information acquisition unit 103 acquires position information 36a and 36b of the objects 32a and 32b in the image data 31. The position information acquisition unit 103 uses an object detector 121 to acquire the position information 36a and 36b.

[0053] The detection confidence level acquisition unit 104 acquires a confidence level (detection confidence level) for the identification result of the object labels 33a and 33b by the object detector 121. When multiple bounding boxes are identified in the image data 31, the detection confidence level may be the probability that the label predicted for each bounding box (e.g., cow) is the label of the actual object. The detection confidence level acquisition unit 104 acquires the detection confidence level using the object detector 121.

[0054] The first object feature amount acquisition unit 105 acquires first object feature amounts 37a and 37b, which are object feature amounts for the targets 32a and 32b, based on the position information 36a and 36b. The first object feature amounts 37a and 37b are examples of first object feature amounts.

[0055] In one example, the position information 36a and 36b identified by the object detector 121 are input to the object feature extractor 122. The object feature extractor 122 identifies object features 37a and 37b based on image data within the bounding boxes that are the position information 36a and 36b. The object detector 121 and the object feature extractor 122 may be formed as a single machine learning model. The first object feature acquisition unit 105 may acquire the object features 37a and 37b using the object feature extractor 122.

[0056] The pair creation unit 106 creates subject-object pairs for the multiple objects 32a, 32b based on the detection confidence and a predetermined upper limit number of pairs. The pair creation unit 106 may extract only pairs that occur between the correct rectangle and the rectangle that covers it among the predicted bounding boxes (rectangles). In one example, for the image data of FIG. 4, the pair creation unit 106 creates subject-object pairs of "person and computer," "computer and table," "person and chair," and "person and window."

[0057] The unlabeled data acquisition unit 108 acquires unlabeled data 39, which is image data including at least one other object classified into the same class as at least one object 32a among the objects 32a, 32b (multiple objects of a subject and an object pair).

[0058] The unlabeled data acquisition unit 108 may use either the subject or object label as a key to extract images containing the same label from an external dataset.

[0059] The unlabeled data 39 is data that does not have a relationship label between multiple objects. For example, the unlabeled data acquisition unit 108 can acquire image data that is classified into the same class as the object 32a, "cow," i.e., image data of a cow. The unlabeled data acquisition unit 108 can widely acquire a large amount of unlabeled data 39 from outside the information processing device 1 via the Internet.

[0060] The second object feature acquisition unit 109 acquires second object feature 40, which is an object feature for an object in the unlabeled data 39. The second object feature 40 is an example of a second object feature.

[0061] In one example, the second object feature acquisition unit 109 acquires the second object feature 40 based on image data within a bounding box corresponding to an object in the unlabeled data 39. The second object feature acquisition unit 109 may acquire the second object feature 40 using an object detector 121 and an object feature extractor 122.

[0062] The control unit 100 acquires first data including a first object feature 37a of an object, a first object feature 37b of a subject, position information 36a of the object, and position information 36b of the subject, using the position information acquisition unit 103 and the first object feature acquisition unit 105. The control unit 100 generates second data by replacing at least one of the first object feature 37a of the object and the first object feature 37b of the subject in the first data with a different second object feature 40, using the position information acquisition unit 103 and the second object feature acquisition unit 109. The control unit 100 trains the encoder 143 by inputting the second data as augmented data for encoder training using contrastive learning.

[0063] In one example, the encoder 143 may be configured as a multi-layer perceptron (MLP). In this case, the encoder 143 is configured with at least three layers of nodes. The encoder 143 may be trained (learned) using a learning method called backpropagation.

[0064] The encoder optimization unit 110 may obtain a first latent vector 62a(Z) regarding the relationship between the multiple objects 32a and 32b, which is obtained by inputting the first data to the encoder 143. The encoder optimization unit 110 may obtain a second latent vector 62b(Z') regarding the relationship between the multiple objects 32a and 32b, which is obtained by inputting the second data to the encoder 143.

[0065] The encoder optimization unit 110 may perform machine learning on the encoder 143 to increase the degree of match (similarity) between the first latent vector 62a(Z) and the second latent vector 62b(Z'). The first latent vector 62a(Z) is an example of a first relationship feature regarding the relationship between the multiple objects 32a and 32b, and the second latent vector 62b(Z) is an example of a second relationship feature regarding the relationship between the multiple objects 32a and 32b. A function that serves as a criterion for evaluating the degree of match (similarity) may be a function (InforNCE) or the like used in contrastive learning (SimCLR, etc.).

[0066] Fig. 8 is a diagram showing an example of a training process for the object detector 121 by the information processing device 1 shown in Fig. 2. In training the object detector 121, image data 31 and labeled data 30 including correct labels and correct position information for object labels 33a and 33b of objects 32a and 32b in the image data 31 are input to the object detector 121. The labeled data 30 may be, for example, a dataset such as a visual genome dataset.

[0067] The object detector 121 outputs object labels 33a and 33b and position information 36a and 36b of the objects 32a and 32b, respectively, based on the image data 31. The object detector 121 may also output a detection confidence level, as described in FIG.

[0068] The position information 36a, 36b may include information on plane coordinates (x, y), height (h), and width (w). The plane coordinates may be the coordinates of one of the vertices of a bounding box (rectangle). The height (h) may be the length of a side of the bounding box in the x direction, and the width (w) may be the length of a side of the bounding box in the y direction perpendicular to the x direction.

[0069] Fig. 9 is a diagram showing an example of a training process for encoder 143 by information processing device 1 shown in Fig. 2. When training of object detector 121 is completed, the parameters of object detector 121 are fixed. Fixing the parameters of object detector 121 is sometimes referred to as "freezing."

[0070] Of the first data including the first object feature 37a of the object (not shown in FIG. 9), the first object feature 37b of the subject, the position information 36a of the object, and the position information 36b of the subject, at least one of the first object feature 37a of the object and the first object feature 37b of the subject is replaced with the second object feature 40 to generate second data.

[0071] 9 shows a case where the second data 42 is generated by replacing the first object feature 37a with the second object feature 40, but the first object feature 37b may be replaced with the second object feature 40. Also, both the first object feature 37a and the first object feature 37b may be replaced with the second object feature 40.

[0072] In generating the second data 42, unlabeled data 39 that does not include a relational label can be used. The unlabeled data 39 may be image data (image data of a cow, image data of a woman) that is classified into at least one class (e.g., cow, woman) of the object 32a and the object 32b. Therefore, a large amount of external data sets that can be used as the unlabeled data 39 exist in the Internet space. Therefore, the encoder 143 can be trained (learned) using a large amount of relationally unlabeled data.

[0073] Fig. 10 is a diagram showing an example of processing using contrastive learning in the training processing of the encoder 143 in Fig. 9. In Fig. 10, first data 41 including a first object feature 37a of an object, a first object feature 37b of a subject, position information 36a of the object, and position information 36b of the subject, and second data 42 in which the first object feature 37a is replaced with a second feature 40 are input to the encoder 143.

[0074] The encoder 143 outputs a first latent vector 62a (relationship feature vector Z) based on the input of the first data 41. The encoder 143 also outputs a second latent vector 62b (relationship feature vector Z') based on the input of the second data 42. In other words, the second latent vector 62b (Z') is obtained by data expansion of the first data 41 that is input to the encoder 143.

[0075] In the second data 42 obtained by data augmentation, the position information 36a and 36b are identical to those in the original input first data 41. The class of another object in the second data 42 (e.g., cow) is the same as the class of the object (object 32a) in the first data 41. The object features of the subject object 32b in the second data 42 are identical to those of the object 32b in the first data 41. Therefore, essential parts (position information, object labels) of the relationships between the objects in the first data 41 and the second data 42 are maintained. Therefore, the first latent vector 62a (relationship feature vector Z) and the second latent vector 62b (relationship feature vector Z') should be similar. The encoder 143 is subjected to contrastive learning to increase the degree of agreement (similarity) between the first latent vector 62a (Z) and the second latent vector 62b (Z').

[0076] FIG. 11 is a block diagram schematically illustrating a second example of the software configuration in the training phase of the encoder 143 by the information processing device 1 illustrated in FIG.

[0077] In the explanation of FIG. 6, just as two types of extended data are obtained from the input x by two types of data extension (T1, T2), second data and third data may be generated from the first data 41 by two types of data extension.

[0078] 11, a second and third object feature amount acquisition unit 111 is provided instead of the second object feature amount acquisition unit 109 in FIG.

[0079] In FIG. 11, the control unit 100 generates second data 42 by replacing at least one of the first object feature 37a of the object and the first object feature 37b of the subject in the first data 41 with another second object feature 40 corresponding to the subject or object.

[0080] Furthermore, the control unit 100 generates third data 53 as data obtained by replacing at least one of the first object feature 37a of the object and the first object feature 37b of the subject in the first data 41 with a different third object feature 54 corresponding to the subject or the object. In the second data 42 and the third data 53, the position information 36a and 36b of the first data may also be maintained as the position information 36a and 36b.

[0081] The third object feature 54 and the third data 53 are generated in the same manner as the second object feature 40 and the second data 42 are generated.

[0082] 11, unlabeled data 39 without any relational labels can be used without being limited by the number of data in labeled data 30. This allows the number of data used to train encoder 143 to be increased.

[0083] Fig. 12 is a block diagram schematically illustrating an example of a software configuration in the training phase of the classifier 144 by the information processing device 1 shown in Fig. 2. Fig. 12 illustrates a state in which the training of the object detector 121 and the encoder 143 is completed and the parameters are fixed.

[0084] The control unit 100 may include a labeled data acquisition unit 101 , an object label acquisition unit 102 , a position information acquisition unit 103 , a first object feature acquisition unit 105 , a pair creation unit 106 , and a classifier optimization unit 112 .

[0085] Fig. 13 is a diagram showing an example of a training process of the classifier 144 by the information processing device 1 shown in Fig. 2. A first object feature amount 37a of an object, a first object feature amount 37b of a subject, position information 36a of the object, and position information 36b of the subject are acquired. Each acquired data is input to the encoder 143.

[0086] In this example, a classifier 144 is used as an example of a machine learning model. The classifier 144 may be, for example, a deep learning model (deep neural network). The neural network may be a hardware circuit, or may be a virtual network created by software that connects layers virtually constructed on a computer program by the processor 11 or the like. In one example, the classifier 144 may be a logistic regression, which is an identification model consisting of only one fully connected layer, or may be configured as a multilayer perceptron (MLP) having multiple layers.

[0087] The control unit 100 acquires a first object feature 37a of the object, a first object feature 37b of the subject, position information 36a of the object, and position information 36b of the subject via the position information acquisition unit 103 and the first object feature acquisition unit 105. Each of the acquired data is input to the encoder 143.

[0088] The control unit 100 calculates a latent vector 62(Z) using the encoder 143. The latent vector Z indicates a position in the latent space.

[0089] The latent vector 62 (Z) is input to the classifier 144. The classifier 144 outputs logits, which may be a final, unnormalized score for the classification of the relationship label. The logits may be converted to predicted values ​​of the relationship label using a softmax function or the like.

[0090] The classifier optimization unit 112 optimizes the parameters of the classifier 144 based on the correct labels of the relationship labels in the labeled data 30 and the logits.

[0091] The encoder 143 is trained using augmented unlabeled data 39, while the classifier 144 may not be trained using unlabeled data 39.

[0092] [B-1-2] Inference phase FIG. 14 is a block diagram schematically illustrating an example of a software configuration in the inference phase by the information processing device 1 shown in FIG.

[0093] The control unit 100 may include an object label acquisition unit 102, a position information acquisition unit 103, a first object feature acquisition unit 105, a pair creation unit 106, an input image acquisition unit 113, a relationship label acquisition unit 114, and a scene graph creation unit 115. The object label acquisition unit 102, the position information acquisition unit 103, the first object feature acquisition unit 105, and the pair creation unit 106 may have the same functions as those described using FIG. 7 except that the data to be processed is input image data instead of the labeled training data 30.

[0094] The input image acquisition unit 113 acquires input image data to be processed. The input image data to be processed is input to a trained object detector 121. The object detector 121 is used to acquire object labels and position information of objects (subjects, subjects) in the input image data. The object feature extractor 122 is used to acquire object features of the objects (subjects, subjects).

[0095] Position information and object feature quantities for objects (subjects, subjects) in the input image data are input to the encoder 143. The encoder 143 infers a latent vector 62 (Z) based on the position information and object feature quantities for the objects (subjects, subjects). The latent vector 62 (relationship feature vector) indicates the relationship between multiple objects (between objects and subjects).

[0096] The latent vector 62 is input to the classifier 144. In one example, the classifier 144 outputs logits. The relationship label acquisition unit 114 uses the logits to acquire relationship labels that indicate relationships between multiple objects (subjects and subjects).

[0097] The scene graph creation unit 115 creates a scene graph 300. The scene graph creation unit 115 compiles, as the scene graph 300, the object labels of the subject and the object acquired from the input image data by the object label acquisition unit 102 and the relationship labels of the subject and the object acquired from the input image data by the relationship label acquisition unit 114.

[0098] In this embodiment, in training the encoder 143 that obtains the relationship between multiple objects, data augmentation is performed by replacing the object feature of at least one of the subject and the object in the first data 41 with the object feature of another object of the same class. The encoder 143 can learn many variations, thereby improving the training accuracy.

[0099] [B-2] Example of operation [B-2-1] Learning phase An example of the training process of the encoder 143 in the training phase of the information processing device 1 shown in FIG. 2 will be described with reference to the flowchart (steps S11 to S19) shown in FIG.

[0100] The control unit 100 acquires position information 36a, 36b(B) of the objects 32a, 32b in the image data, object labels 33a, 32b(L), a detection confidence level C, and first object features 37a, 37b(H) from the input labeled data 30(x) (step S11). The control unit 100 acquires each value by inputting the labeled data 30 to an object detector 121(od). The first object features 37a, 37b(H) are acquired using an object feature extractor 122.

[0101] Let the input x∈labeled data (image dataset) Di. Location information (including coordinate information) B(b1, b2, , b i ,···,b N )=od(x) Object label L(l1,l2,...,l i ,···,l N )=od(x) Detection confidence C(c1,c2,...,c i ,···,c N )=od(x) The first object feature H(h1, h2, , h i ,···,h N )=od(x)

[0102] The detection confidence C and the upper limit number of pairs Nmax are input to the pair generation unit 106 (PairGenerator). The pair generation unit 106 generates pairs P of subjects and objects from among the target objects 32 (step S12).

[0103] P=PairGenerator(C,Nmax) P=(P1,P2,...,P i ,···,P Nmax )where P i =(s i ,o i ) s i is the index of the principal in the i-th pair, o i is the index of the object in the i-th pair

[0104] The control unit 100 generates a pair P of a subject and an object from among the plurality of objects 32 (step S12). si and h oi , location information b si and b oi is input to the encoder 143. φ ) to the feature vector Z i The first latent vector 62a is calculated as follows (step S13). si and h oi , location information b si and b oiis an example of the first data.

[0105] In FIG. 15, the unlabeled data acquisition unit 108 acquires the object label l of the subject. si , is used as a key to extract the data from the external dataset De. si, The unlabeled data acquisition unit 108 extracts an image Xe (unlabeled data 39) containing another object corresponding to the object label l of the object (step S14). oi , is used as a key to extract the data from the external dataset De. oi, The unlabeled data acquisition unit 108 may extract an image Xe (unlabeled data 39) containing another object corresponding to the object label l of the subject and the object. si, and l oi , and l from the external dataset De using both as keys. si, Image Xe contains another object corresponding to l oi, An image Xe containing another object corresponding to

[0106] The second object feature acquisition unit 109 inputs the image Xe (unlabeled data 39) to the object detector 121(od), and determines the object label l in the image Xe from the output of the object detector 121(od). si The second object feature 40(h´ si ) are extracted (step S15). si If there is a second object feature 40(h'), the second object feature acquisition unit 109 randomly acquires the second object feature 40(h' si ) may be selected.

[0107] The control unit 100 calculates the second object feature 40(h') as the object feature of the subject and the object. si ) and the first object feature (h oi ) and location information b si ,b oi and the encoder 143 (f φ ) The control unit 100 inputs the signal to the encoder 143 (f φ ) based on the output of the feature vector Z i The second latent vector 62b is calculated as follows (step S16).si ), first object feature (h oi ), and location information b si ,b oi is an example of the second data 42. The second data 42 is a first object feature amount h si and h oi However, the second data 42 is data in which at least one of the position information b si and b oi Maintain.

[0108] First latent vector Zi=f φ (h si ,h oi ,b si ,b oi ) First latent vector Z´i=f φ (h´ si ,h oi ,b si ,b oi )

[0109] The control unit 100 determines whether the calculation of feature vectors for all pairs has been completed (step S17). If the calculation of feature vectors for all pairs has not been completed (see the No route of step S17), the process may return to step S13. If the calculation of feature vectors for all pairs has been completed (see the Yes route of step S17), the process proceeds to step S18.

[0110] The encoder optimization unit 110 calculates the loss function E x~p(x) [-sim(Z, Z')] is calculated, and the parameter φ is updated so that this value becomes minimum (step S18).

[0111] φ=argmin(E x~p(x) [‐sim(Z,Z´)〕)where Z=(z1,z2,···,z i ,···,z Nmax ),Z´=(z1´,z2´,···,z i ´,···,z Nmax ´) where argmin is a function that obtains the parameter φ that gives the minimum value.

[0112] The control unit 100 repeats the processes of steps S11 to S19 until the processes converge (see the No route of step S19). The control unit 100 waits for the processes to converge (see the Yes route of step S19) and ends the training process of the encoder 143 in the training phase.

[0113] An example of the training process of the classifier 144 in the training phase of the information processing device 1 shown in Fig. 2 will be described with reference to the flowchart (steps S21 to S27) shown in Fig. 16. Step S21 is the same as step S11 in Fig. 15.

[0114] The position information B is input to the pair generation unit 106 (Pair Generator). The pair generation unit 106 generates a pair P of a subject and an object from among the target objects 32 (step S22).

[0115] P=PairGenerator(B) P=(P1,P2,...,P i ,···,P Nmax )where P i =(s i ,o i ) s i means the index of the principal of the i-th pair, and o i means the index of the object in the i-th pair.

[0116] The control unit 100 generates a pair P of a subject and an object from among the plurality of objects 32 (step S12). si and h oi , location information b si and b oi is input to the encoder 143. φ ) to the feature vector Z i A latent vector 62 (feature vector) is calculated (step S23).

[0117] Latent vector Zi=f φ(h si ,h oi ,b si ,b oi )

[0118] If there is a pair P for which the latent vector (feature vector) has not been calculated (No route in step S24), the process returns to step S23. After the latent vector (feature vector) has been calculated for all pairs P (Yes route in step S24), the feature vector Z is input to the classifier 144 (g θ ) is input to the classifier 144(g θ ) and then the logit Y is calculated (step S25).

[0119] Logit Y=g θ (Z)where Z=(Z1,Z2,...,Z i ,···,Z Nmax )

[0120] The classifier optimization unit 112 calculates the loss function E based on the following equation: x~p(x) Calculate φ=argmin(E x~p(x) [CrossEntropy(Softmax(Y), T)], where T is the correct relationship label. The logit Y is converted into a predicted value of the class classification of the relationship label using a softmax function or the like. Then, the distance between the predicted value and the correct relationship label is calculated using a cross entropy function. Then, the parameter θ is updated using an argmin function so as to minimize the distance between the predicted value and the correct relationship label (step S26).

[0121] The control unit 100 repeats the processes of steps S21 to S27 until the processes converge (see the No route of step S27). The control unit 100 waits for the processes to converge (see the Yes route of step S27) and then ends the training process of the classifier 144 in the training phase.

[0122] An example of the process of creating a scene graph in the inference phase of the information processing device 1 shown in FIG. 2 will be described with reference to the flowchart (steps S31 to S38) shown in FIG.

[0123] The input image acquisition unit 113 acquires input image data (input image x) to be processed. Except for the difference in input data, the processes in steps S31 to S35 are the same as the processes in steps S21 to S25 in FIG.

[0124] The relationship label acquisition unit 114 calculates the softmax value softmax(Y) of the logit Y, and extracts the relationship label corresponding to the index (subscript) that gives the largest value of softmax(Y) (step S36).

[0125] The scene graph creation unit 115 collects the object labels and relationship labels that make up each pair to create a scene graph 300 for the input image (step S37). If the process does not collect the object labels and relationship labels, the process returns to step S31 (see the No route in step S38). After the process collects the object labels and relationship labels (see the Yes route in step S38), the scene graph creation process in the inference phase is completed.

[0126] [C] Effect FIG. 18 is a diagram showing an example of the results of the reproducibility of relationship labels obtained by the information processing device 1 shown in FIG. 2. When the encoder 143 is trained using the related technology (supervised learning) of FIG. 1, the reproducibility, which is the degree to which the encoder 143 can reproduce correct relationship labels, is high for abstract relationships and low for concrete relationships. In contrast, in the case of this embodiment using contrastive learning, the reproducibility can be improved even for concrete relationships, and the uneven distribution of class distributions of relationship labels is alleviated. In particular, the reproducibility of concrete relationship labels can be improved compared to the case of the related technology. Therefore, when creating a scene graph 300, a scene graph 300 with improved expressiveness can be created.

[0127] According to the example of the embodiment described above, for example, the following advantageous effects can be achieved.

[0128] The control unit 100 acquires first data 41 including first object features 37a, 37b and position information 36a, 36b of each of multiple objects 32a, 32b in the first image data. The control unit 100 replaces the first object features 37a, 37b of at least one of the multiple objects 32a, 32b in the first data 41 with a second object feature 40 to generate at least one second data 42. The second object feature 40 is an object feature acquired for at least one other object classified in the same class as the object in at least one second image data different from the first image data. The control unit 100 inputs at least one piece of second data 42 to an encoder 143 to train the encoder 143.

[0129] This allows the specific relationship between the plurality of objects 32a, 32b in the image data to be reflected, thereby improving the accuracy of classifying the relationship.

[0130] In the process of training the encoder 143, the control unit 100 inputs the first data 41 and the second data 42 to the encoder 143 to train the encoder 143.

[0131] This allows existing labeled data 30 to be used as training data for the encoder 143, and also allows unlabeled data 39 to be acquired and used. This allows the encoder 143 to be trained using a wide variety of training data, thereby improving training accuracy.

[0132] In the process of training the encoder 143, the control unit 100 performs machine learning to increase the degree of agreement between the first relationship feature (first latent vector Z) and the second relationship feature (second latent vector Z'). The first relationship feature is a feature related to the relationship between the multiple objects 32a and 32b, and is obtained by inputting first data 41 to the encoder 143. The second relationship feature is a feature related to the relationship between the multiple objects 32a and 32b, and is obtained by inputting second data 42 to the encoder 143.

[0133] This allows the accuracy of classifying relationships to be improved by reflecting the specific relationships between multiple objects in the image data.

[0134] Control unit 100 acquires first object features 37a, 37b and position information 36a, 36b by inputting first image data to trained object detector 121. Control unit 100 acquires second object features 40 by inputting second image data to trained object detector 121.

[0135] This allows the specific relationship between the plurality of objects 32a, 32b in the image data to be reflected, thereby improving the accuracy of classifying the relationship.

[0136] [D] Other The disclosed technology is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present embodiment. The configurations and processes of the present embodiment can be selected or combined as needed. [Explanation of symbols]

[0137] 1: Information processing equipment 11: Processor 12: Memory section 13: Display control section 130:Display device 14:Storage device 15: Input IF 151: Mouse 152: Keyboard 16: External recording medium processing unit 160: Recording media 17: Communication IF 100: Control unit 101: Labeled data acquisition unit 102: Object label acquisition unit 103: Location information acquisition unit 104: Detection confidence acquisition unit 105: First object feature acquisition unit 106: Pair Creation Department 108: Unlabeled data acquisition unit 109: Second object feature acquisition unit 110: Encoder optimization section 111: Second and third object feature acquisition units 112: Classifier optimization unit 113: Input image acquisition unit 114: Relationship label acquisition unit 115: Scene graph creation unit 121: Object detector 122: Object feature extractor 143: Encoder 144:Classifier

Claims

1. generating at least one second data by replacing, with respect to first data including first object features and position information of each of a plurality of objects in first image data, the first object feature of at least one of the plurality of objects with a second object feature acquired for at least one other object classified into the same class as the object in at least one second image data different from the first image data; inputting at least one of the second data into an encoder to train the encoder; Have the computer execute the process, In the process of training the encoder, performing machine learning so as to increase the degree of agreement between a first relationship feature regarding the relationship between the plurality of objects, which is obtained by inputting the first data into the encoder, and a second relationship feature regarding the relationship between the plurality of objects, which is obtained by inputting the second data into the encoder; A training program that causes the computer to execute a process.

2. acquiring the first object feature amount and the position information by inputting the first image data into a trained object detector; acquiring the second object feature by inputting the second image data into the trained object detector; The training program according to claim 1 , wherein the program causes the computer to execute processing.

3. generating at least one second data by replacing, with respect to first data including first object features and position information of each of a plurality of objects in first image data, the first object feature of at least one of the plurality of objects with a second object feature acquired for at least one other object classified into the same label as the object in at least one second image data different from the first image data; inputting at least one of the second data into an encoder to train the encoder; The computer executes the processing, In the process of training the encoder, performing machine learning so as to increase the degree of agreement between a first relationship feature regarding the relationship between the plurality of objects, which is obtained by inputting the first data into the encoder, and a second relationship feature regarding the relationship between the plurality of objects, which is obtained by inputting the second data into the encoder; A training method in which the processing is performed by the computer.

4. generating at least one second data by replacing, with respect to first data including first object features and position information of each of a plurality of objects in first image data, the first object feature of at least one of the plurality of objects with a second object feature acquired for at least one other object classified into the same label as the object in at least one second image data different from the first image data; inputting at least one of the second data into an encoder to train the encoder; a processor; The processor, in training the encoder, performing machine learning so as to increase the degree of agreement between a first relationship feature regarding the relationship between the plurality of objects, which is obtained by inputting the first data into the encoder, and a second relationship feature regarding the relationship between the plurality of objects, which is obtained by inputting the second data into the encoder; Information processing device.

Images