Training program, training method, and information processing device

Abstract

Generating at least one piece of second data by replacing at least one piece of first object feature quantity of the plurality of objects with a second object feature quantity with respect to first data including the first object feature quantity and position information of each of the plurality of objects in the first image data; and training the encoding device by inputting at least one piece of second data to the encoding device, the second object feature amount being a second object feature amount acquired for at least one other object classified as the same category as the target object in at least one piece of second image data different from the first image data.

Description

Technical Field

[0001] The invention relates to a training program, a training method and an information processing device. Background Art

[0002] There is a known technique for generating a knowledge graph called a scene graph from image data. The scene graph contains information about the relationship between multiple objects in the image data. When generating a scene graph, a machine learning model for calculating feature quantities of the relationship between objects is trained using teacher data (labeled data) containing correct labels of the relationship between objects. From the perspective of increasing the amount of information contained in the scene graph, it is desirable to calculate a specific relationship rather than an abstract relationship between multiple objects.

[0003] Patent Document 1: Japanese Patent Application No. 2022-508737

[0004] However, depending on the content of the prepared teacher data (labeled data), there is a concern that it may not be easy to reflect the specific relationship between objects and improve the classification accuracy of the relationship. Summary of the invention

[0005] In one aspect, the purpose is to reflect the specific relationship between a plurality of objects in image data and to improve the classification accuracy of the relationship.

[0006] In one aspect, a training program causes a computer to perform the following processing: for first data including first object feature quantities and position information of each of a plurality of objects in first image data, generating at least one second data by replacing at least one of the first object feature quantities of the plurality of objects by a second object feature quantity, wherein the second object feature quantity is a second object feature quantity obtained from at least one other object classified into the same category as the object in at least one second image data different from the first image data; and inputting at least one of the second data into an encoding device to train the encoding device.

[0007] In one aspect, it is possible to improve the classification accuracy of the relationship that reflects the specific relationship between a plurality of objects in the image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 The diagram is a diagram for explaining a training process of an encoder in the related art.

[0009] Figure 2 It is a block diagram schematically showing an example of the hardware configuration of the information processing device according to the embodiment.

[0010] Figure 3 This is a diagram that explains the definition of the scene graph.

[0011] Figure 4 This is a diagram schematically showing an example of a scene graph.

[0012] Figure 5 is a diagram showing an example of self-supervised learning.

[0013] Figure 6 is a diagram showing an example of contrast learning.

[0014] Figure 7 It is a schematic representation based on Figure 2 FIG. 1 is a block diagram showing a first example of a software configuration in a training phase of an encoder of an information processing device shown in FIG.

[0015] Figure 8 Is based on Figure 2 FIG. 1 is a diagram showing an example of a training process of an object detector of an information processing device shown.

[0016] Fig. 9 Is based on Figure 2 A diagram showing an example of a training process of an encoder of an information processing device shown.

[0017] Fig.10 It means in Fig. 9 A diagram showing an example of a process in which contrast learning is used in the training process of an encoder in FIG.

[0018] Fig.11 It is a schematic representation based on Figure 2 FIG. 2 is a block diagram showing a second example of the software configuration in the training phase of the encoder of the information processing device shown.

[0019] Fig.12 It is a schematic representation based on Figure 2 FIG. 1 is a block diagram showing an example of a software configuration in a training phase of a classifier of an information processing device.

[0020] Fig.13 Is based on Figure 2 FIG. 1 is a diagram showing an example of a training process of a classifier of an information processing device shown.

[0021] Fig.14 It indicates schematically Figure 2 A block diagram showing an example of a software configuration in the estimation phase of the information processing device shown.

[0022] Fig.15 Yes Description Figure 2 FIG. 1 is a flowchart of an example of a training process of an encoder in a training phase of an information processing device shown in FIG.

[0023] Fig.16 Yes Description Figure 2FIG. 1 is a flowchart showing an example of a training process of a classifier in a training phase of an information processing device shown.

[0024] Fig.17 Yes means Figure 2 A flowchart of an example of a process of creating a scene graph in the inference phase of the information processing device shown.

[0025] Fig.18 Yes means Figure 2 FIG. 1 is a diagram showing an example of the result of the reproducibility of relational labels of the information processing device shown. DETAILED DESCRIPTION

[0026] 〔A〕Related examples

[0027] Figure 1 It is a diagram for explaining the training process of the encoder 23 in the related art.

[0028] In the related art, the encoder 23 is trained by supervised learning using a deep neural network (DNN).

[0029] As input image data, labeled data 30 is used. The labeled data 30 is also called trained data.

[0030] The labeled data 30 includes image data 31. Figure 1 In the example shown, there are multiple objects 32a (cows) and 32b (women) in the image data 31. The labeled data 30 may include object labels 33a, 33b and a relationship label 34. The object labels 33a, 33b are information indicating the type (category) of each of the multiple objects 32a, 32b in the image data 31. Figure 1 In the example, object label 33a is "cow", and object label 33b is "woman". Relationship label 34 is information indicating the relationship between multiple objects 32a and 32b. Figure 1 In Chinese, it means "feed".

[0031] The object detector 21 that has completed learning obtains the object labels 33a, 33b and the position information 36a, 36b of each object 32a, 32b based on the labeled data 30. The objects 32a, 32b may include a subject and an object. In this example, the object label 33a and the position information 36a are obtained for the "cow" as the object. Similarly, the object label 33b and the position information 36b are obtained for the "woman" as the subject.

[0032] The feature quantity extractor 22 calculates a first object feature quantity 37a of the object 32a (subject) and a first object feature quantity 37b of the object 32b (subject) based on the position information 36a and 36b respectively occupied by the objects 32a and 32b.

[0033] The encoder 23 outputs the classification result 38 of the relational label based on the first object feature 37a of the object and the first object feature 37b of the subject. In this example, the encoder 23 outputs the classification result 38 of the relational label such as "feeding". The encoder optimization unit 24 compares the classification result 38 of the relational label output from the encoder 23 with the relational label 34 of the correct answer of the labeled data 30, and trains the encoder 23 so that the error between the classification result 38 and the relational label 34 is reduced.

[0034] In the related art, it is necessary to prepare the labeled data 30 in advance. When the relationship labels 34 of the correct answers of the prepared labeled data 30 are biased, it is not easy to improve the training accuracy of the encoder 23. In one example, the frequency of occurrence of abstract relationships such as "on" and "have" is higher than the frequency of occurrence of specific relationships such as "sitting on" and "walking on". In this case, it is not easy to attach relationship labels in consideration of the specific relationship.

[0035] A specific method of determining the relationship can be considered by increasing the number of labeled data 30. However, increasing the number of labeled data 30 having the relationship labels 34 increases the burden on the operator.

[0036] Therefore, it is considered to adopt self-supervised learning, especially contrast learning, which uses not only labeled data 30 but also unlabeled data to train the encoder 23.

[0037] [B] Implementation Methods

[0038] Hereinafter, an embodiment will be described with reference to the accompanying drawings. However, the embodiments shown below are only for illustration and do not exclude the application of various modifications and technologies not explicitly shown in the embodiments. That is, the present embodiment can be implemented by various modifications within the scope of the main purpose. In addition, each figure does not only have the main purpose of the components shown in the figure, but can also include other functions, etc.

[0039] Hereinafter, in the drawings, the same reference numerals denote the same parts, and thus description thereof will be omitted.

[0040] 〔B-1〕Configuration example

[0041] Figure 2It is a block diagram schematically showing a hardware configuration example of the information processing device 1 according to the embodiment.

[0042] The information processing device 1 is a computer. Figure 2 As shown, the information processing device 1 includes a processor 11 , a memory unit 12 , a display control unit 13 , a storage device 14 , an input interface (IF) 15 , an external recording medium processing unit 16 , and a communication IF 17 .

[0043] The memory unit 12 is an example of a storage unit, and for example, it is a Read Only Memory (ROM) and a Random Access Memory (RAM). Programs such as a Basic Input / Output System (BIOS) can be written into the ROM of the memory unit 12. The software program of the memory unit 12 can be appropriately read into the processor 11 and executed. In addition, the RAM of the memory unit 12 can be used as a temporary recording memory or a working memory.

[0044] The display control unit 13 is connected to the display device 130 and controls the display device 130. The display device 130 is a liquid crystal display, an organic light-emitting diode (OLED) display, a cathode ray tube (CRT), an electronic paper display, etc., and displays various information for operators, etc. The display device 130 may be combined with an input device, for example, a touch panel.

[0045] The storage device 14 is a storage device with high IO performance, for example, Dynamic Random Access Memory (DRAM), Solid State Drive (SSD), Storage Class Memory (SCM), Hard Disk Drive (HDD) can be used.

[0046] The input IF 15 can be connected to input devices such as a mouse 151 and a keyboard 152, and can control the input devices such as the mouse 151 and the keyboard 152. The mouse 151 and the keyboard 152 are examples of input devices, and an operator performs various input operations via these input devices.

[0047] The external recording medium processing unit 16 is configured to be able to mount a recording medium 160. The external recording medium processing unit 16 is configured to be able to read information recorded in the recording medium 160 when the recording medium 160 is mounted. In this example, the recording medium 160 is portable. For example, the recording medium 160 is a floppy disk, an optical disk, a magnetic disk, an optical magneto-disk, or a semiconductor memory.

[0048] The communication IF 17 is an interface for enabling communication with an external device.

[0049] The processor 11 is a processing device that performs various controls and calculations. The processor 11 may include a CPU (Central Processing Unit). In addition, the processor 11 may include a dGPU (Discrete Graphics Processing Unit), which refers to a GPU that exists on a graphics chip or graphics board that is independent of the CPU. The processor 11 implements various functions by executing an Operating System (OS) or a program read into the memory unit 12. The processor 11 can implement the function of the control unit 100 described later.

[0050] The device for controlling the overall action of the information processing device 1 is not limited to the CPU and the dGPU, and may be, for example, any one of an MPU, a DSP, an ASIC, a PLD, and an FPGA. In addition, the device for controlling the overall action of the information processing device 1 may also be a combination of two or more of a CPU, an MPU, a DSP, an ASIC, a PLD, and an FPGA. In addition, MPU is the abbreviation of MicroProcessing Unit, DSP is the abbreviation of Digital Signal Processor, and ASIC is the abbreviation of Application Specific Integrated Circuit. In addition, PLD is the abbreviation of Programmable Logic Device, and FPGA is the abbreviation of Field Programmable Gate Array.

[0051] Figure 3 This is a diagram illustrating the definition of scene graph 300. Scene graph 300 refers to a directed graph that uses objects (objects) in an image or video as nodes and represents the relationship between objects through edges. Images and videos are examples of image data. Objects (objects) can be living beings including humans, articles, or parts of living beings or articles.

[0052] Figure 4 3 is a diagram schematically showing an example of a scene graph 300 . Figure 4 The dataset shown includes image data 200 and a scene graph 300. Figure 4 In the image data 200, there are a person, a computer, a table, a chair, and a window as objects. Position information 211-1 to 211-5 are given to each object. The position information 211 may have information of coordinates (x, y), height, and width. The position information 211 may be expressed by a bounding box. The bounding box is a rectangular frame surrounding the object.

[0053] The scene graph 300 includes object labels 321-1 (man), 321-2 (computer), 321-3 (table), 321-4 (chair), and 321-5 (window). In addition, directed edges 323-1 to 328-4 represent the relationship between the objects. The directed edge 323 may be a line with an arrow.

[0054] As information indicating the relationship between a person and a computer, a relationship label 322-1 is shown as "using", a relationship label 322-2 is shown as "on" between a computer and a table, and a relationship label 322-3 is shown as "sitting on" between a person and a chair.

[0055] The relationship labels 322 can increase the information content of the scene graph 300 by showing specific relationships (eg, sitting on or standing on) rather than abstract relationships (eg, on).

[0056] The information processing device 1 of this embodiment reflects the specific relationship between a plurality of objects in the image data and improves the classification accuracy of the relationship. To this end, the information processing device 1 uses self-supervised learning, in particular, a comparison learning method.

[0057] Figure 5 is a diagram showing an example of self-supervised learning. Figure 5 In the example, as a simple example, an example is given in which image data showing a dog is input as input data 51, and an object label "dog" is output as output.

[0058] If input data 51 is input to encoder 43, encoder 43 outputs a potential vector (in Figure 5 , is the object feature vector) Z. The potential vector Z is input to the classifier 44, and the classifier 44 outputs a label.

[0059] Self-supervised learning has a two-stage training (learning) process. In the first stage, the encoder 43 is learned using unlabeled data. Unlabeled data has a smaller generation burden than labeled data. Therefore, compared with the case of learning using labeled data, the case of learning using unlabeled data can increase the number of data. Therefore, the encoder 43 can learn more changes and improve the training accuracy (learning accuracy).

[0060] Figure 6 is a diagram showing an example of contrastive learning. Contrastive learning is a type of self-supervised learning. Figure 6 In the example, as a simple example, the case where image data showing a dog is input as input data 51 and the object label "dog" is output as output is described.

[0061] According to the input x (input data 51), two data extensions (T1, T2) are obtained to obtain two extended data x i ~T1(x) and x j ~T2(x). For example, the data expansion is obtained by performing translation, rotation, enlargement and reduction, upside-down inversion, left-right inversion, brightness adjustment, and a combination of these deformations on the original image.

[0062] By inputting the data obtained by the two data expansions into the encoder 43 (f Φ ), we get two potential vectors z i =f Φ (x i )(first potential vector), z j =f Φ (x j )(second latent vector).

[0063] Two extension data x i ~T1(x) and x j ~T2(x) is the data that has been transformed without changing the nature of the object. i =f Φ (x i ), z j =f Φ (x j ) Since the nature of the object has not changed, it is consistent or similar. Therefore, in contrast learning, two latent vectors z i =f Φ (x i ), z j =f Φ (x j) is improved in consistency (similarity) by performing machine learning on the encoder 43. In one example, the loss function can be calculated And update the parameters to minimize this value.

[0064] exist Figure 5 as well as Figure 6 In the above description, the case of training the encoder 43 that outputs the object label of the object is described. However, in the information processing device 1 of this embodiment, contrast learning is applied to the training of the encoder 143 that calculates the relationship between multiple objects. The encoder 143 is an example of an encoding device that calculates the relationship between multiple objects.

[0065] 〔B-1-1〕Training phase

[0066] Figure 7 It is a schematic representation based on Figure 2 1 is a block diagram showing a first example of a software configuration in a training phase of an encoder of the information processing device 1 shown.

[0067] The control unit 100 can include a labeled data acquisition unit 101, an object label acquisition unit 102, a position information acquisition unit 103, a detection confidence acquisition unit 104, a first object feature acquisition unit 105, a pairing unit 106, an unlabeled data acquisition unit 108, a second object feature acquisition unit 109, and an encoder optimization unit 110.

[0068] The control unit 100 implements a learning process (training process) in machine learning using the training data. That is, the information processing device 1 functions as a training device that trains a machine learning model through the control unit 100.

[0069] In this example, the object detector 121, the object feature extractor 122, and the encoder 143 are examples of a machine learning model. The machine learning model may also be, for example, a deep learning model (deep neural network). The neural network may be a hardware circuit or a software-based virtual network that connects layers virtually constructed on a computer program by the processor 11 or the like.

[0070] The labeled data acquisition unit 101 acquires the labeled data 30 . The labeled data 30 may be a data set including image data, object labels, and relational labels. The acquired labeled data 30 may be input to the object detector 121 .

[0071] The object detector 121 may be an existing object detector based on DNN. For example, the object detector 121 may be FasterRCNN (Faster Region-based Convolutional Neural Network) or DETR (DEtection Transformer). The detailed description of the object detector 121 itself is omitted.

[0072] The object label acquisition unit 102 acquires the object labels 33a and 33b of the objects 32a and 32b in the image data based on the labeled data 30. In addition, the labeled data 30, the image data 31, the objects 32a and 32b, the object labels 33a and 33b, the relationship labels 34, the position information 36a and 36b, and the object feature quantities 37a and 37b can be combined with the labeled data 30, the image data 31, the objects 32a and 32b, the object labels 33a and 33b, the relationship labels 34, the position information 36a and 36b, and the object feature quantities 37a and 37b. Figure 1 The contents described in the related art are the same.

[0073] The position information acquisition unit 103 acquires position information 36a and 36b of the objects 32a and 32b in the image data 31. The position information acquisition unit 103 acquires the position information 36a and 36b using the object detector 121.

[0074] The detection confidence acquisition unit 104 acquires the confidence (detection confidence) of the determination result of the object labels 33a and 33b based on the object detector 121. The detection confidence may be the probability that the label (e.g., cow) predicted for each bounding box is the label of the actual object when there are multiple determined bounding boxes in the image data 31. The detection confidence acquisition unit 104 acquires the detection confidence using the object detector 121.

[0075] The first object feature quantity acquisition unit 105 acquires first object feature quantities 37a and 37b, which are object feature quantities of the objects 32a and 32b, based on the position information 36a and 36b. The first object feature quantities 37a and 37b are examples of first object feature quantities.

[0076] In one example, the position information 36a and 36b determined by the object detector 121 are respectively input to the object feature quantity extractor 122. The object feature quantity extractor 122 determines the object feature quantities 37a and 37b based on the image data within the bounding box as the position information 36a and 36b. The object detector 121 and the object feature quantity extractor 122 can also be formed as a machine learning model. The first object feature quantity acquisition unit 105 can acquire the object feature quantities 37a and 37b using the object feature quantity extractor 122.

[0077] The pairing unit 106 creates pairs of subject and object in the plurality of objects 32a and 32b based on the detection confidence and the upper limit of the number of pairs. The pairing unit 106 may extract only the pairs generated between the correct answer rectangle and the overlapping rectangle in the predicted bounding box (rectangle). In one example, if Figure 4 , the pairing unit 106 creates “person and computer”, “computer and table”, “person and chair”, and “person and window” as pairs of subject and object.

[0078] The unlabeled data acquisition unit 108 acquires unlabeled data 39 including image data of at least one other object classified into the same category as at least one object 32a among the objects 32a and 32b (a plurality of objects forming a pair of subject and object).

[0079] The unlabeled data acquisition unit 108 may use any object label of the subject or the object as a keyword and retrieve images containing the same label from an external data set.

[0080] The unlabeled data 39 is data without relational labels between multiple objects. For example, the unlabeled data acquisition unit 108 can acquire image data classified into the same category as the object 32a, that is, image data of a cow. The unlabeled data acquisition unit 108 can widely acquire a large amount of unlabeled data 39 from outside the information processing device 1 via the Internet.

[0081] The second object feature quantity acquisition unit 109 acquires a second object feature quantity 40 which is an object feature quantity of an object in the unlabeled data 39. The second object feature quantity 40 is an example of a second object feature quantity.

[0082] In one example, the second object feature acquisition unit 109 acquires the second object feature 40 based on the image data in the bounding box corresponding to the object in the unlabeled data 39. The second object feature acquisition unit 109 may acquire the second object feature 40 using the object detector 121 and the object feature extractor 122.

[0083] The control unit 100 acquires first data including the first object feature 37a of the object, the first object feature 37b of the subject, the position information 36a of the object, and the position information 36b of the subject through the position information acquisition unit 103 and the first object feature acquisition unit 105. The control unit 100 generates second data by replacing at least one of the first object feature 37a of the object and the first object feature 37b of the subject in the first data with another second object feature 40 through the position information acquisition unit 103 and the second object feature acquisition unit 109. The control unit 100 trains the encoder 143 by inputting the second data as extended data when training the encoder using contrast learning.

[0084] In one example, the encoder 143 may be formed by a multilayer perceptron (MLP). In this case, the encoder 143 is formed by a layer of at least three nodes. The encoder 143 may be trained (learned) using a learning method called back propagation (error back propagation method).

[0085] The encoder optimization unit 110 can obtain a first latent vector 62a(Z) of the relationship between the plurality of objects 32a and 32b obtained by inputting the first data to the encoder 143. The encoder optimization unit 110 can obtain a second latent vector 62b(Z′) of the relationship between the plurality of objects 32a and 32b obtained by inputting the second data to the encoder 143.

[0086] The encoder optimization unit 110 can perform machine learning on the encoder 143 to improve the consistency (similarity) between the first latent vector 62a (Z) and the second latent vector 62b (Z'). The first latent vector 62a (Z) is an example of a first relational feature quantity of the relationship between multiple objects 32a, 32b, and the second latent vector 62b (Z) is an example of a second relational feature quantity of the relationship between multiple objects 32a, 32b. The function that serves as a benchmark for evaluating consistency (similarity) can be a function (InforNCE) used in comparative learning (SimCLR, etc.), etc.

[0087] Figure 8 Is based on Figure 2 FIG. 1 is a diagram showing an example of a training process of an object detector 121 of the information processing device 1 shown in FIG. In the training of the object detector 121, labeled data 30 including image data 31, correct labels of object labels 33a and 33b of objects 32a and 32b in the image data 31, and correct position information is input to the object detector 121. In one example, the labeled data 30 may be a dataset such as Visual Genome.

[0088] The object detector 121 outputs object labels 33a, 33b and position information 36a, 36b of the objects 32a, 32b, respectively, based on the image data 31. The object detector 121 may be configured as follows: Figure 7 The detection confidence is also output as described in .

[0089] The position information 36a and 36b may include information of plane coordinates (x, y), height (h), and width (w). The plane coordinates may be the coordinates of one of the vertices of the bounding box (rectangle). The height (h) may be the length of the side of the bounding box in the x direction, and the width (w) may be the length of the side of the bounding box in the y direction that is perpendicular to the x direction.

[0090] Fig. 9 Is based on Figure 2 FIG. 1 is a diagram showing an example of a training process of the encoder 143 of the information processing device 1 shown in FIG. When the training of the object detector 121 is completed, the parameters of the object detector 121 are fixed. The process of fixing the parameters of the object detector 121 is sometimes referred to as "freezing".

[0091] Generate a first object feature 37a (in Fig. 9 The second data is a second data in which at least one of the first object feature quantity 37a of the object and the first object feature quantity 37b of the subject in the first data of the position information 36a of the subject is replaced by the second object feature quantity 40 (not shown in the figure).

[0092] exist Fig. 9 , the case where the first object feature 37a is replaced by the second object feature 40 to generate the second data 42 is shown, but the first object feature 37b may be replaced by the second object feature 40. In addition, both the first object feature 37a and the first object feature 37b may be replaced by the second object feature 40.

[0093] In the generation of the second data 42, unlabeled data 39 that does not include a relational label can be used. The unlabeled data 39 only needs to be image data (image data of cows, image data of women) classified into at least one of the categories (cows, women, etc.) of the object 32a and the object 32b. Therefore, there are a large number of external data sets that can be used as unlabeled data 39 in the Internet space. Therefore, the encoder 143 can be trained (learned) using a large amount of relational unlabeled data.

[0094] Fig.10 It means in Fig. 9 FIG. 1 is a diagram showing an example of a process in which contrast learning is used in the training process of the encoder 143 in FIG. Fig.10In the process, first data 41 including the first object feature 37a of the object, the first object feature 37b of the subject, the position information 36a of the object, and the position information 36b of the subject, and second data 42 after the first object feature 37a is replaced by the second feature 40 are input into the encoder 143.

[0095] The encoder 143 outputs a first latent vector 62a (relational feature vector Z) based on the input of the first data 41. In addition, the encoder 143 outputs a second latent vector 62b (relational feature vector Z′) based on the input of the second data 42. In other words, the second latent vector 62b (Z′) is obtained by data expansion of the first data 41, which is the input to the encoder 143.

[0096] In the second data 42 obtained by data expansion, the position information 36a, 36b is the same as the original input, that is, the first data 41. Moreover, the category of other objects in the second data 42 (for example, cow) is the same as the category of the object (object 32a) of the first data 41. The object feature quantity of the subject, that is, object 32b in the second data 42 is the same as the object feature quantity of object 32b in the first data 41. Therefore, the essential part (position information, object label) of the relationship between objects in the first data 41 and the relationship between objects in the second data 42 is maintained. Therefore, the first latent vector 62a (relational feature vector Z) and the second latent vector 62b (relational feature vector Z′) should be similar. The encoder 143 is subjected to comparative learning to improve the consistency (similarity) of the first latent vector 62a (Z) and the second latent vector 62b (Z′).

[0097] Fig.11 It is a schematic representation based on Figure 2 1 is a block diagram of a second example of the software configuration in the training phase of the encoder 143 of the information processing device 1 shown.

[0098] with Figure 6 In the description in, the two extended data obtained by performing two data extensions (T1, T2) according to the input x are the same, and the second data and the third data can be generated according to the first data 41 by performing two data extensions.

[0099] exist Fig.11 In place of Figure 7 The second object feature quantity acquisition unit 109 is provided, and the second and third object feature quantity acquisition units 111 are provided.

[0100] exist Fig.11In the embodiment, the control unit 100 generates the second data 42 as data obtained by replacing at least one of the first object feature 37a of the object and the first object feature 37b of the subject in the first data 41 with another second object feature 40 corresponding to the subject or the object.

[0101] Furthermore, the control unit 100 generates the third data 53 as data obtained by replacing at least one of the first object feature 37a of the object and the first object feature 37b of the subject in the first data 41 with another third object feature 54 corresponding to the subject or the object. In the second data 42 and the third data 53, the position information 36a, 36b of the first data may be maintained as the position information 36a, 36b.

[0102] The generation of the third object feature 54 and the third data 53 is the same as the generation of the second object feature 40 and the second data 42 .

[0103] according to Fig.11 The processing shown can use the unlabeled data 39 without relational labels without being limited by the number of labeled data 30. The number of data for training the encoder 143 can be increased.

[0104] Fig.12 It is a schematic representation based on Figure 2 FIG. 1 is a block diagram showing an example of a software configuration in the training phase of the classifier 144 of the information processing device 1. Fig.12 , the training of the object detector 121 and the encoder 143 is completed, and the parameters are fixed.

[0105] The control unit 100 may include a labeled data acquisition unit 101 , an object label acquisition unit 102 , a position information acquisition unit 103 , a first object feature quantity acquisition unit 105 , a pairing unit 106 , and a classifier optimization unit 112 .

[0106] Fig.13 Is based on Figure 2 1 is a diagram showing an example of training processing of the classifier 144 of the information processing device 1. The first object feature 37a of the object, the first object feature 37b of the subject, the position information 36a of the object, and the position information 36b of the subject are acquired. The acquired data are input to the encoder 143.

[0107] In this example, the classifier 144 is used as an example of a machine learning model. The classifier 144 may also be, for example, a deep learning model (deep neural network). A neural network may be a hardware circuit or a software-based virtual network that connects layers virtually constructed on a computer program by a processor 11 or the like. In one example, the classifier 144 may be a logistic regression as a recognition model consisting of only one fully coupled layer, or may be composed of a multilayer perceptron (MLP) having multiple layers.

[0108] The control unit 100 acquires the first object feature 37 a of the object, the first object feature 37 b of the subject, the position information 36 a of the object, and the position information 36 b of the subject through the position information acquisition unit 103 and the first object feature acquisition unit 105 . The acquired data are input to the encoder 143 .

[0109] The control unit 100 calculates a latent vector 62 (Z) using the encoder 143. The latent vector Z indicates a position in the latent space.

[0110] The latent vector 62 (Z) is input to the classifier 144. The classifier 144 outputs a logit (logit). The logit may be the final score of the classification of the relational label that is not normalized. The logit may also be converted into a predicted value of the relational label using a softmax function or the like.

[0111] The classifier optimization unit 112 optimizes the parameters of the classifier 144 based on the correct answer labels and Logit of the relational labels in the labeled data 30 .

[0112] The training of the encoder 143 uses the unlabeled data 39 to increase the amount of data, while the training of the classifier 144 does not need to use the unlabeled data 39.

[0113] 〔B-1-2〕Inference stage

[0114] Fig.14 It indicates schematically Figure 2 1 is a block diagram showing an example of a software configuration of the information processing device 1 in the estimation phase.

[0115] The control unit 100 may include an object label acquisition unit 102, a position information acquisition unit 103, a first object feature quantity acquisition unit 105, a pairing unit 106, an input image acquisition unit 113, a relational label acquisition unit 114, and a scene graph creation unit 115. In addition, the object label acquisition unit 102, the position information acquisition unit 103, the first object feature quantity acquisition unit 105, and the pairing unit 106 may include the same as the one used for the training except that the data to be processed is not the labeled data 30 for training but the input image data. Figure 7 The same functions as those described above are performed.

[0116] The input image acquisition unit 113 acquires input image data of the processing object. The input image data of the processing object is input to the trained object detector 121. The object label and position information of the object (object, subject) in the input image data are acquired using the object detector 121. The object feature quantity extractor 122 is used to acquire the object feature quantity of the object (object, subject).

[0117] The encoder 143 is input with the position information and object features of the object (object, subject) in the input image data. The encoder 143 estimates the latent vector 62 (Z) based on the position information and object features of the object (object, subject). The latent vector 62 (relational feature vector) represents the relationship between multiple objects (objects and subjects).

[0118] The latent vector 62 is input to the classifier 144. In one example, the classifier 144 outputs Logit. The relational label acquisition unit 114 acquires relational labels indicating the relationship between a plurality of objects (objects and subjects) using Logit.

[0119] The scene graph generator 115 generates the scene graph 300 by integrating the subject and object object labels acquired by the object label acquirer 102 from the input image data and the subject and object relational labels acquired by the relational label acquirer 114 from the input image data into the scene graph 300 .

[0120] In this embodiment, in the training of the encoder 143 for obtaining the relationship between multiple objects, data expansion is performed by replacing the object feature of at least one of the subject and the object with the object feature of another object of the same category in the first data 41. The encoder 143 can learn many changes and improve the training accuracy.

[0121] 〔B-2〕Action Example

[0122] 〔B-2-1〕Learning stage

[0123] according to Fig.15 The flowchart (steps S11 to S19) shown in Figure 2 An example of the training process of the encoder 143 in the training phase of the information processing device 1 shown in FIG. 1 will be described.

[0124] The control unit 100 obtains the position information 36a, 36b (B), the object labels 33a, 32b (L), the detection confidence C, and the first object feature quantities 37a, 37b (H) of the objects 32a, 32b in the image data based on the input labeled data 30 (x) (step S11). The control unit 100 obtains each value by inputting the labeled data 30 to the object detector 121 (od). The first object feature quantities 37a, 37b (H) are obtained using the object feature quantity extractor 122.

[0125] Input x∈labeled data (image dataset) Di.

[0126] Position information (including coordinate information) B (b1, b2, ..., b i 、···、b N ) = od(x)

[0127] Object label L(l1, l2, ···, l i 、···、l N ) = od(x)

[0128] Detection confidence C(c1, c2, ···, c i 、···、c N ) = od(x)

[0129] The first object feature quantity H (h1, h2, ..., h i 、···、h N ) = od(x)

[0130] The detection certainty C and the upper limit number Nmax of pairs are input to the pairing unit 106 (PairGenerator). The pairing unit 106 generates a pair P of a subject and an object in the object 32 (step S12).

[0131] P = PairGenerator(C, Nmax)

[0132] P=(P1、P2、···、P i 、···、P Nmax )where P i =(s i , o i )

[0133] s i is the index of the subject of the ith pair, o i is the index of the object of the ith pair

[0134] The control unit 100 generates a pair P of a subject and an object from the plurality of objects 32 (step S12). si and hoi , location informationb si and b oi is input to the encoder 143. Φ ) Calculate the characteristic vector Z i That is, the first latent vector 62a (step S13). The first object feature h of the subject and the object si and h oi , location informationb si and b oi is an example of the first data.

[0135] exist Fig.15 In the process, the unlabeled data acquisition unit 108 sets the object label l of the subject si, As a keyword, extract the data from the external dataset De containing the equivalent of l si, The image Xe (unlabeled data 39) of another object (step S14). However, the unlabeled data acquisition unit 108 may also use the object label l oi, As a keyword, extract the data from the external dataset De containing the equivalent of l oi, The unlabeled data acquisition unit 108 may also store the object labels of the subject and the object. si, and oi, Both parties are used as keywords, and the data set De is extracted from the external dataset containing the equivalent of l si, The image Xe of other objects, and the image Xe of other objects, including the image Xe of other objects, including the image Xe of other objects, and ... oi, Images of other objects Xe.

[0136] The second object feature quantity acquisition unit 109 inputs the image Xe (unlabeled data 39) to the object detector 121 (od), and extracts the feature quantity corresponding to the object label l in the image Xe from the output of the object detector 121 (od). si The second object feature 40(h′ si )(Step S15). When there are a plurality of second object feature quantities 40(h′) si ), the second object feature quantity acquisition unit 109 may randomly select the second object feature quantity 40(h′ si ).

[0137] The control unit 100 converts the second object feature value 40 (h′ si ), the first object feature (h oi ), location information b si 、b oi The object feature quantities of the subject and the object are input to the encoder 143 (f Φ ). The control unit 100 is based on the encoder 143 (f Φ) output to calculate the feature vector Z i That is, the second latent vector 62b (step S16). The second object feature 40 (h') si ), the first object feature (h oi ), and location information b si 、b oi is an example of the second data 42. The second data 42 is obtained by replacing the first object feature value h in the first data 41. si and h oi However, the second data 42 retains the position information b in the first data 41. si and b oi .

[0138] The first potential vector Z i =f Φ (h si 、h oi 、b si 、b oi )

[0139] The first latent vector Z′ i =f Φ (h′ si 、h oi 、b si 、b oi )

[0140] The control unit 100 determines whether the feature vector calculation is completed for all pairs (step S17). If the feature vector calculation is not completed for all pairs (refer to the negative path of step S17), the process can return to step S13. If the feature vector calculation is completed for all pairs (refer to the positive path of step S17), the process proceeds to step S18.

[0141] The encoder optimization unit 110 calculates the loss function E x～p(x) 〔-sim(Z, Z′)〕, and update the parameters To minimize the value (step S18).

[0142] where Z=(z1、z2、···、

[0143] z i 、···、z Nmax ), Z′=(z1′, z2′,···,z i ′、···、z Nmax ').in,

[0144] argmin is to get the parameter that gives the minimum value function.

[0145] The control unit 100 repeats the processing of steps S11 to S19 until the processing converges (see the negative path of step S19). The control unit 100 waits for the processing to converge (see the positive path of step S19) and ends the training processing of the encoder 143 in the training phase.

[0146] according to Fig.16 The flowchart (steps S21 to S27) shown in Figure 2 An example of the training process of the classifier 144 in the training phase of the information processing device 1 shown in FIG. Fig.15 The same as step S11.

[0147] The position information B is input to the pairing unit 106 (PairGenerator). The pairing unit 106 generates a pair P of a subject and an object in the object 32 (step S22).

[0148] P=PairGenerator(B)

[0149] P=(P1、P2、···、P i 、···、P Nmax )where P i =(s i , o i )

[0150] s i is the index of the subject of the ith pair, o i is the index of the object of the i-th pair.

[0151] The control unit 100 generates a pair P of a subject and an object from the plurality of objects 32 (step S12). The first object feature quantity h of the subject and the object si and h oi , location informationb si and b oi is input to the encoder 143. Φ ) Calculate the characteristic vector Z i That is, the latent vector 62 (feature vector) (step S23).

[0152] Potential vector Z i =f Φ (h si 、h oi 、b si 、b oi )

[0153] If there is a pair P for which the latent vector (feature vector) has not been calculated (the negative path of step S24), the process returns to step S23. Wait until the latent vector (feature vector) is calculated for all pairs P (the positive path of step S24), and the feature vector Z is input to the classifier 144 (g θ ). According to classifier 144 (g θ ) output to calculate LogitY (step S25).

[0154] LogitY=g θ (Z)where Z=(Z1、Z2、···、Z i 、···、Z Nmax )

[0155] The classifier optimization unit 112 calculates the loss function E based on the following formula: x～p(x) . Where T is the relational label of the correct answer. LogitY is converted into the predicted value of the category classification of the relational label using the Softmax function, etc. Then, the distance between the predicted value and the relational label of the correct answer is calculated by the cross entropy function. Then, the parameter θ is updated by the argmin function to minimize the distance between the predicted value and the relational label of the correct answer (step S26).

[0156] The control unit 100 repeats the processing of steps S21 to S27 until the processing converges (see the negative path of step S27). The control unit 100 waits for the processing to converge (see the positive path of step S27) and ends the training processing of the classifier 144 in the training phase.

[0157] according to Fig.17 The flowchart (steps S31 to S38) shown in FIG. Figure 2 An example of scene graph creation processing in the inference phase of the information processing device 1 shown in FIG. 1 is described.

[0158] The input image acquisition unit 113 acquires the input image data (input image x) to be processed. Except for the difference in input data, the processing of steps S31 to S35 is the same as that of Fig.16 The processing of steps S21 to S25 is the same.

[0159] The relational label acquisition unit 114 calculates the maximum softmax(Y) of LogitY, and extracts the relational label corresponding to the index (subscript) with the largest value of softmax(Y) (step S36 ).

[0160] The scene graph generation unit 115 aggregates the object labels and relational labels that constitute each pair and generates a scene graph 300 for the input image (step S37). If the process has not converged, the process returns to step S31 (refer to the negative path of step S38). The process waits for the process to converge (refer to the positive path of step S38), and the scene graph generation process in the inference phase ends.

[0161] 〔C〕Effect

[0162] Fig.18 Is based on Figure 2 The figure is an example of the result of the reproducibility of the relational label of the information processing device 1 shown. Figure 1 When the encoder 143 is trained using the related art (supervised learning), the degree to which the encoder 143 can reproduce the correct relational labels, that is, the degree of reproduction, is higher in the case of abstract relations and lower in the case of specific relations. In contrast, in the case of the present embodiment using comparative learning, the degree of reproduction can be improved to a certain extent in the case of specific relations, alleviating the uneven distribution of the categories of relational labels. In particular, compared with the case of the related art, the degree of reproduction of specific relational labels can be improved. Therefore, when the scene graph 300 is produced, a scene graph 300 with improved expressiveness can be produced.

[0163] According to the example of the above-mentioned embodiment, for example, the following effects can be achieved.

[0164] The control unit 100 obtains first data 41 including first object feature quantities 37a, 37b and position information 36a, 36b of each of the plurality of objects 32a, 32b in the first image data. The control unit 100 generates at least one second data 42 by replacing at least one first object feature quantity 37a, 37b of the plurality of objects 32a, 32b with a second object feature quantity 40 in the first data 41. The second object feature quantity 40 is an object feature quantity obtained for at least one other object classified into the same category as the object in at least one second image data different from the first image data. The control unit 100 inputs at least one of the second data 42 to the encoder 143 to train the encoder 143.

[0165] This makes it possible to reflect the specific relationship between the plurality of objects 32a and 32b in the image data and improve the classification accuracy of the relationship.

[0166] In the process of training the encoder 143 , the control unit 100 inputs the first data 41 and the second data 42 to the encoder 143 to train the encoder 143 .

[0167] Thus, the existing labeled data 30 can also be used as training data for the encoder 143, and the unlabeled data 39 can be acquired and used. Therefore, the encoder 143 can be trained with training data rich in variety. Therefore, the training accuracy is improved.

[0168] The control unit 100 performs machine learning to improve the consistency between the first relational feature quantity (first latent vector Z) and the second relational feature quantity (second latent vector Z′) in the process of training the encoder 143. The first relational feature quantity is a feature quantity related to the relationship between the plurality of objects 32a and 32b, and is obtained by inputting the first data 41 to the encoder 143. The second relational feature quantity is a feature quantity related to the relationship between the plurality of objects 32a and 32b, and is obtained by inputting the second data 42 to the encoder 143.

[0169] This makes it possible to reflect the specific relationship between a plurality of objects in the image data and improve the classification accuracy of the relationship.

[0170] The control unit 100 acquires the first object feature 37a, 37b and the position information 36a, 36b by inputting the first image data to the trained object detector 121. The control unit 100 acquires the second object feature 40 by inputting the second image data to the trained object detector 121.

[0171] This makes it possible to reflect the specific relationship between the plurality of objects 32a and 32b in the image data and improve the classification accuracy of the relationship.

[0172] 〔D〕Other

[0173] The disclosed technology is not limited to the above-mentioned embodiment, and can be implemented with various modifications within the scope of the gist of the present embodiment. The various configurations and processes of the present embodiment can be selected or omitted as needed, or can be appropriately combined.

[0174] Description of Reference Numerals

[0175] 1: information processing device, 11: processor, 12: memory unit, 13: display control unit, 130: display device, 14: storage device, 15: input IF, 151: mouse, 152: keyboard, 16: external recording medium processing unit, 160: recording medium, 17: communication IF, 100: control unit, 101: label data acquisition unit, 102: object label acquisition unit, 103: position information acquisition unit, 104: detection confidence acquisition unit, 105 : First object feature acquisition unit, 106: pairing unit, 108: unlabeled data acquisition unit, 109: second object feature acquisition unit, 110: encoder optimization unit, 111: second and third object feature acquisition units, 112: classifier optimization unit, 113: input image acquisition unit, 114: relational label acquisition unit, 115: scene graph creation unit, 121: object detector, 122: object feature extractor, 143: encoder, 144: classifier.

Claims

1. A training procedure, wherein: Make the computer execute the following processing: For first data including first object feature quantities and position information of each of a plurality of objects in the first image data, at least one second data is generated by replacing at least one of the first object feature quantities of the plurality of objects with a second object feature quantity, the second object feature quantity being a second object feature quantity obtained for at least one other object classified into the same category as the object in at least one second image data different from the first image data; as well as At least one of the second data is input into the encoding device to train the encoding device.

2. The training program according to claim 1, wherein: In the process of training the above encoding device, The computer is caused to execute a process of inputting the first data and the second data into an encoding device to train the encoding device.

3. The training program according to claim 2, wherein: In the process of training the above encoding device, The computer is made to execute the following processing: Machine learning is performed to improve the consistency between a first relational feature quantity of the relationship between the plurality of objects obtained by inputting the first data into the encoding device and a second relational feature quantity of the relationship between the plurality of objects obtained by inputting the second data into the encoding device.

4. The training program according to any one of claims 1 to 3, wherein: The computer is made to execute the following processing: acquiring the first object feature quantity and the position information by inputting the first image data into a trained object detector; and The second object feature amount is acquired by inputting the second image data into the trained object detector.

5. A training method, wherein: The above computer performs the following processing: For first data including first object feature quantities and position information of each of a plurality of objects in the first image data, at least one second data is generated by replacing at least one of the first object feature quantities of the plurality of objects by a second object feature quantity, the second object feature quantity being a second object feature quantity obtained for at least one other object classified as having the same label as the object in at least one second image data different from the first image data; as well as At least one of the second data is input into the encoding device to train the encoding device.

6. The training method according to claim 5, wherein: In the process of training the above encoding device, The computer executes a process of inputting the first data and the second data into an encoding device to train the encoding device.

7. The training method according to claim 6, wherein: In the process of training the above encoding device, The above computer performs the following processing: Machine learning is performed to improve the consistency between a first relational feature quantity of the relationship between the plurality of objects obtained by inputting the first data into the encoding device and a second relational feature quantity of the relationship between the plurality of objects obtained by inputting the second data into the encoding device.

8. The training method according to any one of claims 5 to 7, wherein: The above computer performs the following processing: acquiring the first object feature quantity and the position information by inputting the first image data into a trained object detector; and The second object feature amount is acquired by inputting the second image data into the trained object detector.

9. An information processing device, wherein: A processor is provided, the processor executing: For first data including first object feature quantities and position information of each of a plurality of objects in the first image data, at least one second data is generated by replacing at least one of the first object feature quantities of the plurality of objects by a second object feature quantity, the second object feature quantity being a second object feature quantity obtained for at least one other object classified as having the same label as the object in at least one second image data different from the first image data; as well as At least one of the second data is input into the encoding device to train the encoding device.

10. The information processing device according to claim 9, wherein: In the process of training the above encoding device, The processor inputs the first data and the second data into an encoding device to train the encoding device.

11. The information processing device according to claim 10, wherein: In the process of training the above encoding device, The processor performs machine learning to improve the consistency between a first relational feature quantity of the relationship between the plurality of objects obtained by inputting the first data into the encoding device and a second relational feature quantity of the relationship between the plurality of objects obtained by inputting the second data into the encoding device.

12. The information processing device according to any one of claims 9 to 11, wherein: The above processor executes: acquiring the first object feature and the position information by inputting the first image data into a trained object detector; as well as The second object feature amount is acquired by inputting the second image data into the trained object detector.

Images