Information processing program, information processing method, and information processing device

Abstract

To solve the problem that in conventional scene graph generation, since individual objects are cut out from an image, contextually important information existing in regions other than a BBOX region of the cut-out objects may be missed, i.e., a relation between the objects is not necessarily correctly recognized only with information of the BBOX region of the cut-out objects.SOLUTION: An information processing program causes a computer to execute processing of: extracting a first feature amount from an image; detecting, from the extracted first feature amount, a plurality of objects included in the image; generating a second feature amount obtained by combining the first feature amount of each object in a set of the plurality of detected objects; generating, based on the first feature amount and the second feature amount, a first map that indicates a relation between the objects; extracting a fourth feature amount based on the first map and a third feature amount obtained by converting the first feature amount; and estimating a relation from the fourth feature amount.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] Technologies are being developed to realize a safe and secure society by sensing the real world from video and feeding back the knowledge gained. In order to deeply understand, or sense, the complex real world, this technology goes beyond simply detecting people and objects as in the past; it also needs to recognize the relationships between people and objects in the video (also known as "context").

[0003] Conventional object detection technology for detecting people and objects may miss a lot of information, such as human actions and the relationship between people and objects, even if it can detect people and objects correctly. Therefore, semantically different scenes may be detected as the same result. For example, in the analysis of surveillance and monitoring systems, there is a possibility of incorrect recognition, such as recognizing something as dangerous when it is not, or vice versa.

[0004] Therefore, in order to deeply sense and understand the real world, scene graphs are being generated that graphically represent the relationships between objects, including people, in an image. Scene graph generation involves detecting the location (BBOX: bounding box) and type (class) of objects in an image (scene), and generating a scene graph corresponding to them. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Li, Rongjie, et al. "Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021. [Non-patent document 2] Xu, Danfei, et al. "Scene graph generation by iterative message passing." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. Summary of the Invention [Problem to be solved by the invention]

[0006] However, conventional scene graph generation extracts individual objects from an image, which can miss important contextual information outside the BBOX region of the extracted object. For example, it may be possible to better understand the relationships between extracted objects by looking at the objects surrounding the BBOX region of the extracted object or the entire image. In other words, the relationships between objects may not be correctly recognized based solely on the information in the BBOX region of the extracted object.

[0007] In one aspect, an object is to provide an information processing program, an information processing method, and an information processing device that can more accurately recognize the relationship between objects in an image. [Means for solving the problem]

[0008] In one aspect, the information processing program causes a computer to execute the following processes: extracting a first feature from an image; detecting multiple objects contained in the image from the extracted first feature; generating a second feature that combines the first feature possessed by each object in the set of detected multiple objects; generating a first map that shows the relationship between the objects based on the first feature and the second feature; extracting a fourth feature based on a third feature converted from the first feature and the first map; and estimating the relationship from the fourth feature. [Effects of the Invention]

[0009] On the one hand, it can more accurately recognize relationships between objects in an image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram for explaining object detection and scene graph generation. [Figure 2] FIG. 2 is a diagram illustrating an example of a scene graph generation process. [Figure 3] FIG. 3 is a diagram for explaining the problem of scene graph generation. [Figure 4] FIG. 4 is a diagram illustrating an example of attention map generation according to the first embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of the information processing device 10 according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of an NN configuration according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of image feature extraction according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of object detection according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of generating paired features according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of relationship feature extraction according to the first embodiment. [Figure 11]FIG. 11 is a diagram illustrating an example of relationship estimation according to the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of the configuration of the machine learning device 100 according to the first embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of training of the machine learning model according to the first embodiment. [Figure 14] FIG. 14 is a flowchart illustrating an example of the flow of the relationship estimation process according to the first embodiment. [Figure 15] FIG. 15 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0011] Examples of the information processing program, information processing method, and information processing device according to the present embodiment will be described in detail below with reference to the accompanying drawings. Note that the present embodiment is not limited to these examples. Furthermore, each example can be appropriately combined within a consistent range. [Example]

[0012] First, a conventional technique for object detection and scene graph generation from captured images executed by an information processing device will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining object detection and scene graph generation. In Fig. 1, the upper part shows object detection, and the lower part shows scene graph generation.

[0013] As shown in the upper part of Figure 1, objects including people are detected from captured images using existing detection algorithms such as YOLO (YOU Only Look Once), SSD (Single Shot Multibox Detector), and RCNN (Region Based Convolutional Neural Networks). In the example of Figure 1, at least a man and a horse are detected from captured images 50 and 51, and the detected objects are indicated by bounding boxes (BBOXes) that surround their respective areas in the images. However, even though captured images 50 and 51 are images of different scenes, object detection alone may produce similar detection results, which may be insufficient as detection results.

[0014] Therefore, as shown in the lower part of Fig. 1, a scene graph that also expresses the relationships between people and objects is generated from the captured image. The lower part of Fig. 1 shows an example in which a scene graph 60 is generated from a captured image 51. In the example of Fig. 1, the scene graph expresses not only the objects represented by a man and a horse, but also the relationships between objects represented by wearing and feeding.

[0015] The scene graph is generated using existing techniques described in Non-Patent Document 1, Non-Patent Document 2, etc. FIG. 2 is a diagram showing an example of scene graph generation processing. As shown in FIG. 2, the information processing device uses an existing detection algorithm to detect all objects, including people, from an input captured image, and generates a BBOX on the image for each detected object, surrounding the object. Next, the information processing device cuts out a BBOX area from the image and extracts features of each area. Then, the information processing device estimates the relationship between pairs of objects (subject, object) from the features, and generates a scene graph that represents each object and the relationship between them, i.e., the context.

[0016] Figure 3 is a diagram illustrating the problems with scene graph generation. However, as shown in Figure 3, scene graph generation extracts individual objects from a captured image, and therefore may miss important contextual information outside the BBOX regions of the Subject and Object. More specifically, in the example of Figure 3, the BBOX regions of the cat and curtain are extracted as shown on the right side, but information from other regions is not used to estimate the relationship between the cat and the curtain, resulting in the overlooking of important information. For example, by looking at the objects and room structure surrounding the cat and curtain, it can be inferred that the curtain is behind the cat. However, conventional scene graph generation does not use surrounding information to infer the relationship between the cat and the curtain. By using surrounding information when inferring the relationship between objects, it becomes possible to infer, for example, from the image on the left side of Figure 3, that there is an apple next to the cat, that there is a high possibility that there is a table under the cat and apple, and that the curtain is behind the cat because it is hanging on the door.

[0017] Therefore, in this embodiment, contextually important areas are adaptively extracted from the entire image for each subject and object whose relationship is to be estimated, and the relationship between the objects is recognized. The areas that are important for relationship recognition are realized as a map (hereinafter referred to as an "Attention map") that takes on a value between 0 and 1 depending on the importance, for example.

[0018] FIG. 4 is a diagram illustrating an example of attention map generation according to the first embodiment. As shown in FIG. 4, instead of extracting only the BBOX region as in conventional scene graph generation, an attention map 65 is generated that represents important regions in regions other than the subject and object for which a relationship is to be estimated, and is used to estimate the relationship between objects. As shown in FIG. 4, the attention map 65 expresses the importance of important regions in estimating the relationship between the subject and the object using color, shading, or the like. Note that the attention map 65 is generated using a machine learning model that is trained to input, for example, feature amounts of objects detected from an image and output important regions in the image as correct labels, so as to increase the recognition rate of the relationship between objects. Furthermore, since the location of contextually important regions in an image cannot be known in advance, generating the attention map 65 for the entire image enables more accurate extraction of important regions, and ultimately enables more accurate recognition of the relationship between objects.

[0019] [Functional configuration of information processing device 10] Next, a functional configuration of the information processing device 10 that executes the present embodiment will be described. Fig. 5 is a diagram illustrating an example of the configuration of the information processing device 10 according to Example 1. As shown in Fig. 5, the information processing device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

[0020] The communication unit 20 is a processing unit that controls communication with other information processing devices, and is, for example, a communication interface such as a network interface card. The communication unit 20 may receive, for example, captured images that are targets for recognizing relationships between objects, from a camera device or another information processing device.

[0021] The storage unit 30 is an example of a storage device, such as a memory or a hard disk, that stores various data and programs executed by the control unit 40. The storage unit 30 stores input data 31, a model DB 32, and the like.

[0022] The input data 31 stores data to be input to the information processing device 10 for recognizing relationships between objects. The data is, for example, an image or video of a scene. The data may be uploaded to the information processing device 10 from another information processing device via the communication unit 20, or may be read by the information processing device 10 via an arbitrary computer-readable recording medium.

[0023] The model DB 32 stores, for example, information about a machine learning model for acquiring object classes and inter-object relations for generating a scene graph from a captured image, and model parameters for constructing the machine learning model. The machine learning model, i.e., the scene graph generation model, is generated by machine learning using the captured image as a feature and the locations (BBOXes) of objects included in the captured image, the object classes, and the inter-object relations as correct labels.

[0024] The model DB 32 also stores information about a machine learning model generated by inputting feature amounts of an object detected from an image and training the model to output important regions on the image as correct labels, as well as the model parameters. The machine learning model is a model for generating an attention map, and is referred to as an attention map generation model. The scene graph generation model and the attention map generation model may be generated by the information processing device 10, or may be trained and generated by another information processing device.

[0025] The above information stored in the storage unit 30 is merely an example, and the storage unit 30 can store various information other than the above information.

[0026] The control unit 40 is a processing unit, such as a processor, that controls the entire information processing device 10. The control unit 40 includes an image feature extraction unit 41, an object detection unit 42, a pair feature generation unit 43, a relationship feature extraction unit 44, and a relationship estimation unit 45. Each processing unit is an example of an electronic circuit included in the processor or an example of a process executed by the processor.

[0027] The image feature extraction unit 41 extracts a first feature amount from an image.

[0028] The object detection unit 42 detects a plurality of objects included in the image from the first feature amount extracted by the image feature extraction unit 41.

[0029] Paired feature generation unit 43 generates, as paired features, second features that combine first features owned by each object in a set of multiple objects detected by object detection unit 42. The process of generating second features includes a process of generating second features that combine first features owned by each object in a set of a subject and an object, with each object in all sets of the detected multiple objects being treated as a subject and an object.

[0030] The relationship feature extraction unit 44 generates a first map, i.e., an attention map, indicating the relationship between each object based on the first feature amount and the second feature amount. The process of generating the first map includes a process of finding a correlation between the first feature amount and the second feature amount for each set of multiple detected objects, and generating the first map based on the correlation.

[0031] The relationship estimation unit 45 extracts a fourth feature based on a third feature obtained by converting the first feature and the first map, and estimates the relationship between each object from the fourth feature. The fourth feature is, for example, a feature of an area represented in the attention map as an area important for recognizing the relationship between each object. The process of extracting the fourth feature includes a process of multiplying the first map by the third feature to obtain a weighted sum, and extracting the fourth feature based on the weighted sum.

[0032] Furthermore, the relationship estimation unit 45 generates a fifth feature based on the fourth feature and the second feature, and estimates the relationship between each object from the fifth feature. The process of estimating the relationship includes processes of combining the fourth feature and the second feature to generate a fifth feature, calculating a probability value of the type of relationship between each object from the fifth feature, and estimating the relationship between each object based on the probability value. For example, the relationship estimation unit 45 estimates the type of relationship (on, behind, hold, etc.) with the highest probability value from among multiple probability values ​​calculated for each object as the relationship between the objects.

[0033] [Function details] The relationship estimation process that can more accurately recognize the relationship between objects in an image will be described in more detail with reference to Figures 6 to 11. Each processing unit that executes the relationship estimation process in this embodiment may be configured with a neural network (NN).

[0034] FIG. 6 is a diagram illustrating an example of a NN configuration according to the first embodiment. As illustrated in FIG. 6, the image feature extraction unit 41, the object detection unit 42, the pair feature generation unit 43, the relationship feature extraction unit 44, and the relationship estimation unit 45, which perform the relationship estimation process in this embodiment, may be configured by an NN 70. When an image capturing a certain scene is input, the NN 70 starts with the processing by the image feature extraction unit 41 and outputs a BBOX of an object detected by the object detection unit 42 and its type (class). The NN 70 also outputs the relationship (relation) of each object detected by the object detection unit 42, estimated by the relationship estimation unit 45. Next, the processing by each processing unit included in the NN 70 will be described for each processing.

[0035] First, feature extraction from a captured image performed by the image feature extraction unit 41 will be described. FIG. 7 is a diagram illustrating an example of image feature extraction according to the first embodiment. As illustrated in FIG. 7, the image feature extraction unit 41 extracts, for example, an image feature of the C channel from a captured image 52 having three RGB channels. The image feature extraction unit 41 may be configured using, for example, existing technologies such as a convolutional neural network (CNN) or a transformer. Furthermore, the CNN or transformer may be one that has been pre-trained on a classification task or the like using a large-scale image dataset such as ImageNet. Furthermore, the width W and height H of the image feature are generally smaller than the width and height of the input image, i.e., the captured image 52 in the example of FIG. 7, due to CNN or pooling processing.

[0036] Next, object detection from image features executed by the object detection unit 42 will be described. FIG. 8 is a diagram illustrating an example of object detection according to the first embodiment. As illustrated in FIG. 8, the object detection unit 42 receives, for example, image features extracted by the image feature extraction unit 41, and outputs a BBOX, which is a rectangle surrounding each object included in the captured image and indicates the location of each object, and a class, which indicates the type of the object. As an output example, the object detection unit 42 can overlay and draw a BBOX of each detected object on the captured image, as illustrated on the right side of FIG. 8, and output the image. At this time, a class, which indicates the type of each object, may be output near each BBOX.

[0037] The rectangle of the BBOX can be expressed by four real values, such as the upper left coordinates (x1, y2) and the lower right coordinates (x2, y2) of the rectangle. Furthermore, the "class" is a probability value that the detected object is a predetermined object to be detected. More specifically, for example, if the predetermined detection objects are {cat, table, car}, and as shown on the right side of FIG. 8, the class of the BBOX corresponding to a cat has a probability value such as (0.9, 0.1, 0.2). In this case, the "class" indicates that the probability that the detected object is a cat, a table, and a car is 0.9, 0.1, and 0.2, respectively. The object detection unit 42 may be configured, for example, by a Faster Region Based Convolutional Neural Network (R-CNN), Mask R-CNN, or DEtection Transformer (DETR), which is an object detection NN based on existing technology.

[0038] Next, the paired feature of each detected object, which is executed by the paired feature generation unit 43, will be described. FIG. 9 is a diagram illustrating an example of paired feature generation according to the first embodiment. As illustrated in FIG. 9, the paired feature generation unit 43 extracts, for each object, a feature of each detected object based on, for example, the object detected by the object detection unit 42, i.e., the output class and BBOX. In this case, the paired feature generation unit 43 can use RoIAlign in Mask R-CNN, for example. Furthermore, when the object detection unit 42 is configured using DETR, the feature for each object output by the Transformer decoder can be used as is. Furthermore, the extracted feature for each object can be, for example, a real vector of 256 dimensions, for example.

[0039] Then, the paired feature generation unit 43 pairs all combinations of detected objects, with one object as the subject and the other as the object. The paired features shown on the right side of Fig. 9 are obtained by arranging the features of the subject and the object for each pair. Furthermore, in order to adjust the number of dimensions of the paired features, the paired feature generation unit 43 may convert the features of the paired subject and object using MLP (Multilayer Perceptron), which is an existing technology.

[0040] Next, extraction of features indicating the relationship between detected and paired objects, which is executed by the relationship feature extraction unit 44, will be described. FIG. 10 is a diagram illustrating an example of relationship feature extraction according to the first embodiment. First, as shown in FIG. 10, the relationship feature extraction unit 44 converts the image features extracted by the image feature extraction unit 41 using conversion units 1 and 2, respectively, for use in subsequent processing. The conversion units 1 and 2 may be realized by, for example, an MLP (Multilayer Perceptron). In the example of FIG. 10, the conversion units 1 and 2 are distinguished as 1 and 2 because they are realized by MLPs with different weights.

[0041] Next, the relationship feature extraction unit 44 generates an attention map by using the attention map generation unit to correlate, with the image features converted by the conversion unit 1, each line of the paired features generated by the paired feature generation unit 43, i.e., for each pair of subject and object. Note that the relationship feature extraction unit 44 may convert the attention map using MLP or layer normalization after correlating.

[0042] Here, we will explain in more detail the correlation process between a certain pair feature and the image feature converted by the conversion unit 1. It is assumed that the pair feature has been adjusted to a C-dimensional vector in the previous process. It is also assumed that the image feature converted by the conversion unit 1 is an H×W tensor with a C-dimensional channel direction. A pixel (x, y) containing the image feature converted by the conversion unit 1 is focused on and designated as the pixel of interest. Since the pixel of interest is 1×1×C, it can be considered a C-dimensional vector. The attention map generation unit then calculates a correlation value (scalar) between the C-dimensional vector of the pixel of interest and the pair feature adjusted to the C-dimensional vector. This determines the correlation value at the pixel of interest (x, y). The attention map generation unit performs this process for all pixels to generate an H×W×1 attention map.

[0043] Then, the relationship feature extraction unit 44 extracts features of important regions in the entire image corresponding to the subject-object pair by taking a weighted sum of the generated attention map and the image features converted by the conversion unit 2. Note that since the weighted sum is taken over the entire image, the feature obtained by taking the weighted sum becomes a C-dimensional feature for one subject-object pair.

[0044] Here, the weighted sum of the attention map and the image feature converted by the conversion unit 2 will be described in more detail. It is assumed that the image feature converted by the conversion unit 2 is an H×W×C tensor. First, the relationship feature extraction unit 44 multiplies the image feature converted by the conversion unit 2 by the attention map. At this time, since the attention map is H×W×1, the channel is copied to the C dimension. The relationship feature extraction unit 44 also adds all the C-dimension vectors of each pixel for the multiplication. This generates one C-dimension vector. That is, one C-dimension vector is generated for each attention map. Furthermore, since attention maps are actually generated for the number of paired features, the number of C-dimension vectors generated is also the same as the number of paired features. Through the above process, the relationship feature extraction unit 44 performs a weighted sum of the image feature converted by the conversion unit 2, using the attention map as a weight.

[0045] Then, the relationship feature extraction unit 44 uses a synthesis unit to synthesize the extracted feature of the important region and the paired feature generated by the paired feature generation unit 43, and outputs the synthesized feature as a relationship feature. More specifically, the relationship feature extraction unit 44 can use a combination of the feature of the important region and the paired feature in the dimensional direction. After synthesizing the feature of the important region and the paired feature, the relationship feature extraction unit 44 may convert the combined feature using MLP or the like to adjust the number of dimensions.

[0046] Next, estimation of the relationship between each pair of Subject and Object, which is executed by the relationship estimation unit 45, will be described. FIG. 11 is a diagram illustrating an example of relationship estimation according to the first embodiment. The relationship estimation unit 45 receives the relationship feature output by the relationship feature extraction unit 44, and outputs the relationship corresponding to each pair of Subject and Object. The relationship estimation unit 45 may be configured using, for example, existing techniques such as MLP or batch normalization. The output relationship is a probability value of a relationship to be estimated in advance. For example, when the estimation target is {on, behind, hold}, in the example of FIG. 11, a probability value for each of {on, behind, hold}, such as (0.1, 0.9, 0.2), is output as the relationship between a curtain and a cat.

[0047] Above, a relationship estimation process that can more accurately recognize relationships between objects in an image has been described in more detail using FIGS. 6 to 11. Next, a training process for a neural network (NN) that constitutes each processing unit that executes the estimation process will be described. The NN training process can use the error backpropagation method (also called backpropagation), which is an existing technology. The training process may be executed by the information processing device 10 that executes the estimation process, or may be executed by another information processing device. In this embodiment, the training process is described as being executed by a machine learning device 100 that is another information processing device.

[0048] [Functional configuration of machine learning device 100] The functional configuration of the machine learning device 100 that executes the training process of the NN will be described. FIG. 12 is a diagram illustrating an example of the configuration of the machine learning device 100 according to the first embodiment. As illustrated in FIG. 12, the machine learning device 100 includes a communication unit 120, a storage unit 130, and a control unit 140.

[0049] The communication unit 120 is a processing unit that controls communication with other information processing devices, and is, for example, a communication interface such as a network interface card. The communication unit 20 can transmit and receive information to and from the information processing device 10, for example.

[0050] The storage unit 130 is an example of a storage device, such as a memory or a hard disk, that stores various data and programs executed by the control unit 140. The storage unit 130 stores training data 131, a model DB 132, and the like.

[0051] The training data 131 stores training data in which a captured image is used as a feature, and the location (BBOX) of an object included in the captured image, the type (class) of the object, and the relationship (relation) between the objects are used as correct labels.

[0052] The model DB 132 stores, for example, information about a machine learning model for obtaining object classes and relationships between objects from a captured image to generate a scene graph, as well as model parameters for constructing the machine learning model.

[0053] The above information stored in the storage unit 130 is merely an example, and the storage unit 130 can store various information other than the above information.

[0054] The control unit 140 is a processing unit that controls the entire machine learning device 100, and is, for example, a processor. The control unit 140 includes a scene graph generation unit 141, an error calculation unit 142, and a weight update value calculation unit 143. The scene graph generation unit 141 may be configured by a neural network (NN) as shown in FIG. 6. Each processing unit is an example of an electronic circuit included in the processor or an example of a process executed by the processor.

[0055] The processing of each processing unit will be described with reference to FIG. 13. FIG. 13 is a diagram illustrating an example of training of a machine learning model according to the first embodiment. As illustrated in FIG. 13, first, the machine learning device 100 inputs an input image acquired from training data 131 to a scene graph generation unit 141. The input image may be an image of various landscapes captured using a camera device or a camera function. Then, the scene graph generation unit 141 detects and outputs the locations of objects (BBOX), the types of objects (classes), and the relationships between objects (relation) from the input image using existing technology.

[0056] Next, the machine learning device 100 calculates an error by comparing the BBOX, class, and relation output from the scene graph generation unit 141 with each piece of training data corresponding to the input image, i.e., the correct label, using the error calculation unit 142. Note that the error can be calculated using existing techniques such as cross entropy and class balanced loss.

[0057] Next, the machine learning device 100 uses the weight update value calculation unit 143 to calculate the sum of the errors calculated by the error calculation unit 142, and uses this sum to calculate an update value for the weight of the NN according to the error backpropagation algorithm. Then, the machine learning device 100 updates the weights of the NN within the scene graph generation unit 141 according to the update value for the weight calculated by the weight update value calculation unit 143. This trains the NNs that constitute the processing units that perform relationship estimation processing in this embodiment, enabling more accurate recognition of relationships between objects in an image. Note that the information processing device 10 is the entity that performs this estimation processing, and therefore, when performing this estimation processing, for example, a trained NN is copied in advance to the machine learning device 100 and used.

[0058] [Processing flow] Next, a flow of a relationship estimation process by the information processing device 10 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the flow of the relationship estimation process according to the first embodiment. The relationship estimation process shown in Fig. 14 is a process for estimating a relationship between objects in an image or video captured of a certain scene from the captured image or video. The relationship estimation process shown in Fig. 14 may be started, for example, when a captured image or video to be processed is uploaded to the information processing device 10, or may be started at any timing.

[0059] First, the information processing device 10 acquires an input image to be processed from the input data 31 (step S101). Note that the input image includes an image of one frame of a video, and if the input image is stored as a video in the input data 31, one frame is acquired from the video as the input image.

[0060] Next, the information processing device 10 extracts image feature amounts as image features from the input image acquired in step S101 (step S102).

[0061] Next, the information processing device 10 detects a plurality of objects included in the image extracted in step S102 (step S103). For example, by using existing technology, the objects are detected by detecting a BBOX indicating the location of each object included in the image and a class indicating the type of each object.

[0062] Next, the information processing device 10 generates, as a pair feature, a second feature that combines the first feature of each object in the set of multiple objects detected in step S103 (step S104).

[0063] Next, the information processing device 10 generates an attention map from the pair features extracted in step S104, and combines the features of the areas important for relationship estimation extracted by the attention map with the pair features to extract them as relationship features (step S105).

[0064] Then, the information processing device 10 estimates the relationship between each object detected from the image based on the relationship feature amount extracted in step S105 (step S106). Note that the estimation of the relationship may be, for example, by calculating a probability value for each type of relationship. After step S106 is executed, the relationship estimation process shown in FIG. 14 ends.

[0065] [effect] As described above, the information processing device 10 extracts a first feature from an image, detects multiple objects contained in the image from the extracted first feature, generates a second feature that combines the first feature possessed by each object in the set of detected multiple objects, generates a first map that indicates the relationship between the objects based on the first feature and the second feature, extracts a fourth feature based on the first map and a third feature converted from the first feature, and estimates the relationship from the fourth feature.

[0066] As a result, the information processing device 10 estimates the relationship between objects while taking into consideration image feature amounts other than those of the objects whose relationship is to be estimated, and therefore can more accurately recognize the relationship between objects in the image.

[0067] In addition, the process of estimating the relationship executed by the information processing device 10 includes a process of generating a fifth feature based on the fourth feature and the second feature, and estimating the relationship from the fifth feature.

[0068] This allows the information processing device 10 to more accurately estimate the relationship between objects in the image.

[0069] In addition, the process of generating a second feature executed by the information processing device 10 includes a process of generating a second feature that combines the first feature possessed by each object in all pairs of detected objects as a subject and an object.

[0070] This allows the information processing device 10 to more accurately estimate the relationship between objects in the image.

[0071] In addition, the process of generating the first map executed by the information processing device 10 includes a process of calculating the correlation between the first feature amount and the second feature amount for each set of detected objects, and generating the first map based on the correlation.

[0072] This allows the information processing device 10 to take into consideration important areas for estimating the relationship between objects in an image, and to more accurately estimate the relationship between the objects.

[0073] Furthermore, the process of extracting the fourth feature amount, which is executed by the information processing device 10, includes a process of multiplying the first map by the third feature amount to obtain a weighted sum, and extracting the fourth feature amount based on the weighted sum.

[0074] This allows the information processing device 10 to more accurately estimate the relationship between the objects.

[0075] In addition, the process of estimating a relationship executed by the information processing device 10 includes a process of combining the fourth feature and the second feature to generate a fifth feature, calculating a probability value of the type of relationship from the fifth feature, and estimating the relationship based on the probability value.

[0076] This allows the information processing device 10 to more accurately recognize the relationship between objects in the image.

[0077] The machine learning device 100 also extracts a first feature from an image, detects multiple objects contained in the image from the extracted first feature, generates a second feature that combines the first feature possessed by each object in the set of detected multiple objects, generates a first map that shows the relationship between the objects based on the first feature and the second feature, extracts a fourth feature based on a third feature converted from the first feature and the first map, inputs the fourth feature, and generates a machine learning model by training using the type and relationship of each object as a correct label.

[0078] This enables the machine learning device 100 to generate a machine learning model for more accurately recognizing the relationships between objects in an image.

[0079] [system] The information, including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified. Furthermore, the specific examples, distributions, and numerical values ​​described in the embodiments are merely examples and can be changed as desired.

[0080] Furthermore, the components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution or integration of each device is not limited to that shown. In other words, all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads and usage conditions. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) and a program analyzed and executed by the CPU or GPU, or can be realized as hardware using wired logic.

[0081] [Hardware] Fig. 15 is a diagram illustrating an example of a hardware configuration. Note that, although the example of Fig. 15 illustrates the hardware configuration of the information processing device 10, the machine learning device 100 may also have a similar configuration. In the following description, the hardware configuration will be described using the information processing device 10 as an example.

[0082] 15, the information processing device 10 includes a communication interface 10a, a hard disk drive (HDD) 10b, a memory 10c, and a processor 10d. The components shown in FIG. 15 are connected to each other via a bus or the like.

[0083] The communication interface 10a is a network interface card or the like, and communicates with other servers. The HDD 10b stores programs and DBs that operate the functions shown in FIG.

[0084] The processor 10d is a hardware circuit that operates a process that executes each function described in FIG. 5 and other figures by reading a program that executes the same processing as each processing unit shown in FIG. 5 from the HDD 10b or the like and expanding the program into the memory 10c. That is, this process executes the same functions as each processing unit of the information processing device 10. Specifically, the processor 10d reads a program having the same functions as the image feature extraction unit 41, the object detection unit 42, and the like from the HDD 10b or the like. Then, the processor 10d executes a process that executes the same processing as the image feature extraction unit 41, the object detection unit 42, and the like.

[0085] In this way, the information processing device 10 operates as an information processing device that executes operation control processing by reading and executing a program that executes processing similar to that of each processing unit shown in Fig. 5. The information processing device 10 can also realize functions similar to those of the above-described embodiment by reading a program from a recording medium using a medium reading device and executing the read program. Note that the program in these other embodiments is not limited to being executed by the information processing device 10. For example, this embodiment can also be applied in the same way to cases where another computer or server executes a program, or where these execute a program in cooperation with each other.

[0086] A program that executes the same processes as those of the processing units shown in Fig. 5 can be distributed via a network such as the Internet. This program can be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disc (DVD), and can be executed by being read from the recording medium by a computer.

[0087] The following additional notes are provided regarding the embodiments including the above examples.

[0088] (Appendix 1) Extract a first feature from the image, detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The relationship is estimated from the fourth feature amount. An information processing program that causes a computer to execute a process.

[0089] (Supplementary Note 2) The process of estimating the relationship is generating a fifth feature amount based on the fourth feature amount and the second feature amount; The relationship is estimated from the fifth feature amount. 2. The information processing program according to claim 1, further comprising:

[0090] (Supplementary Note 3) The process of generating the second feature amount includes: Each of the objects in all pairs of the detected objects is treated as a subject and an object, and the second feature amount is generated by combining the first feature amount of each of the objects in the pair of the subject and the object. 3. The information processing program according to claim 1, further comprising:

[0091] (Supplementary Note 4) The process of generating the first map includes: calculating a correlation between the first feature amount and the second feature amount for each set of the plurality of detected objects; generating the first map based on the correlation; 4. The information processing program according to claim 1, further comprising:

[0092] (Supplementary Note 5) The process of extracting the fourth feature amount is multiplying the first map by the third feature amount to obtain a weighted sum; extracting the fourth feature amount based on the weighted sum; 5. The information processing program according to claim 1, further comprising:

[0093] (Supplementary Note 6) The process of estimating the relationship is generating a fifth feature by combining the fourth feature and the second feature; calculating a probability value of the type of the relationship from the fifth feature amount; Estimating the relationship based on the probability value 2. The information processing program according to claim 1, further comprising:

[0094] (Appendix 7) Extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The fourth feature is input, and the type of each object and the relationship are used as correct labels to train a machine learning model. An information processing program that causes a computer to execute a process.

[0095] (Appendix 8) Extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The relationship is estimated from the fourth feature amount. An information processing method characterized in that the processing is executed by a computer.

[0096] (Supplementary Note 9) The process of estimating the relationship is generating a fifth feature amount based on the fourth feature amount and the second feature amount; The relationship is estimated from the fifth feature amount. 9. The information processing method according to claim 8, further comprising:

[0097] (Supplementary Note 10) The process of generating the second feature amount includes: Each of the objects in all pairs of the detected objects is treated as a subject and an object, and the second feature amount is generated by combining the first feature amount of each of the objects in the pair of the subject and the object. 10. The information processing method according to claim 8 or 9, comprising the steps of:

[0098] (Supplementary Note 11) The process of generating the first map includes: calculating a correlation between the first feature amount and the second feature amount for each set of the plurality of detected objects; generating the first map based on the correlation; 11. The information processing method according to any one of appendices 8 to 10, comprising:

[0099] (Supplementary Note 12) The process of extracting the fourth feature amount is multiplying the first map by the third feature amount to obtain a weighted sum; extracting the fourth feature amount based on the weighted sum; 12. The information processing method according to any one of claims 8 to 11, comprising:

[0100] (Supplementary Note 13) The process of estimating the relationship includes: generating a fifth feature by combining the fourth feature and the second feature; calculating a probability value of the type of the relationship from the fifth feature amount; Estimating the relationship based on the probability value 9. The information processing method according to claim 8, further comprising:

[0101] (Appendix 14) Extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The relationship is estimated from the fourth feature amount. An information processing device comprising a control unit that executes processing.

[0102] (Supplementary Note 15) The process of estimating the relationship is generating a fifth feature amount based on the fourth feature amount and the second feature amount; The relationship is estimated from the fifth feature amount. 15. The information processing device according to claim 14, further comprising:

[0103] (Supplementary Note 16) The process of generating the second feature amount includes: Each of the objects in all pairs of the detected objects is treated as a subject and an object, and the second feature amount is generated by combining the first feature amount of each of the objects in the pair of the subject and the object. 16. The information processing device according to claim 14 or 15, further comprising:

[0104] (Supplementary Note 17) The process of generating the first map includes: calculating a correlation between the first feature amount and the second feature amount for each set of the plurality of detected objects; generating the first map based on the correlation; 17. The information processing device according to any one of appendices 14 to 16, comprising:

[0105] (Supplementary Note 18) The process of extracting the fourth feature amount is multiplying the first map by the third feature amount to obtain a weighted sum; extracting the fourth feature amount based on the weighted sum; 18. The information processing device according to any one of appendices 14 to 17, comprising:

[0106] (Supplementary Note 19) The process of estimating the relationship includes: generating a fifth feature by combining the fourth feature and the second feature; calculating a probability value of the type of the relationship from the fifth feature amount; Estimating the relationship based on the probability value 15. The information processing device according to claim 14, further comprising:

[0107] (Appendix 20) Extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The fourth feature is input, and the type of each object and the relationship are used as correct labels to train a machine learning model. An information processing method characterized in that the processing is executed by a computer.

[0108] (Appendix 21) Extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The fourth feature is input, and the type of each object and the relationship are used as correct labels to train a machine learning model. An information processing device comprising a control unit that executes processing.

[0109] (Appendix 22) A processor; a memory operatively connected to the processor; An information processing device comprising: extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The relationship is estimated from the fourth feature amount. An information processing device that executes processing.

[0110] (Appendix 23) A processor; a memory operatively connected to the processor; An information processing device comprising: extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The fourth feature is input, and the type of each object and the relationship are used as correct labels to train a machine learning model. An information processing device that executes processing. [Example]

[0111] Although the embodiments of the present invention have been described above, the present invention may be embodied in various different forms other than the above-described embodiments. [Explanation of symbols]

[0112] 10. Information processing equipment 10a communication interface 10b HDD 10c memory 10d processor 20 Communications Department 30 Storage section 31 Input Data 32 Model DB 40 Control Unit 41 Image feature extraction unit 42 Object detection unit 43 Pair feature generation unit 44 Relationship feature extraction unit 45 Relationship Estimation Unit 50, 51, 52 Captured images 60 Scene Graph 65 Attention Maps 70NN 100 Machine Learning Devices 120 Communications Department 130 Storage section 131 training data 132 Model DB 140 Control Unit 141 Scene Graph Generation Unit 142 Error calculation section 143 Weight update value calculation unit

Claims

1. extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount that is a feature amount for each set of objects by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount, which is a feature amount of a region in the first map that is expressed as a region important for recognizing a relationship between the objects, based on a third feature amount obtained by converting the first feature amount and the first map; A relationship between the objects whose relationship is to be estimated is estimated from the fourth feature amount. An information processing program that causes a computer to execute a process.

2. The process of estimating the relationship includes: generating a fifth feature amount based on the fourth feature amount and the second feature amount; The relationship is estimated from the fifth feature amount.

2. The information processing program according to claim 1, further comprising:

3. The process of generating the second feature amount includes: Each of the objects in all pairs of the detected objects is treated as a subject and an object, and the second feature amount is generated by combining the first feature amount of each of the objects in the pair of the subject and the object.

3. The information processing program according to claim 1, further comprising:

4. The process of generating the first map includes: calculating a correlation between the first feature amount and the second feature amount for each set of the plurality of detected objects; generating the first map based on the correlation; 4. The information processing program according to claim 1, further comprising:

5. The process of extracting the fourth feature amount includes: multiplying the first map by the third feature amount to obtain a weighted sum; extracting the fourth feature amount based on the weighted sum; 5. The information processing program according to claim 1, further comprising:

6. The process of estimating the relationship includes: generating a fifth feature by combining the fourth feature and the second feature; calculating a probability value of the type of the relationship from the fifth feature amount; Estimating the relationship based on the probability value 2. The information processing program according to claim 1, further comprising:

7. extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount that is a feature amount for each set of objects by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount, which is a feature amount of a region in the first map that is expressed as a region important for recognizing a relationship between the objects, based on a third feature amount obtained by converting the first feature amount and the first map; The fourth feature is input, and the type of each object and the relationship between the objects to be estimated are used as correct labels for training to generate a machine learning model. An information processing program that causes a computer to execute a process.

8. extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount that is a feature amount for each set of objects by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount, which is a feature amount of a region in the first map that is expressed as a region important for recognizing a relationship between the objects, based on a third feature amount obtained by converting the first feature amount and the first map; A relationship between the objects whose relationship is to be estimated is estimated from the fourth feature amount. An information processing method characterized in that the processing is executed by a computer.

9. extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount that is a feature amount for each set of objects by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount, which is a feature amount of a region in the first map that is expressed as a region important for recognizing a relationship between the objects, based on a third feature amount obtained by converting the first feature amount and the first map; A relationship between the objects whose relationship is to be estimated is estimated from the fourth feature amount. An information processing device comprising a control unit that executes processing.

10. extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount that is a feature amount for each set of objects by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount, which is a feature amount of a region in the first map that is expressed as a region important for recognizing a relationship between the objects, based on a third feature amount obtained by converting the first feature amount and the first map; The fourth feature is input, and the type of each object and the relationship between the objects to be estimated are used as correct labels for training to generate a machine learning model. An information processing method characterized in that the processing is executed by a computer.

11. extracting a first feature from the image; detecting a plurality of objects included in the image from the extracted first feature amount; generating a second feature amount that is a feature amount for each set of objects by combining the first feature amounts of each object in the set of the detected plurality of objects; generating a first map indicating a relationship between the objects based on the first feature amount and the second feature amount; extracting a fourth feature amount, which is a feature amount of a region in the first map that is expressed as a region important for recognizing a relationship between the objects, based on a third feature amount obtained by converting the first feature amount and the first map; The fourth feature is input, and the type of each object and the relationship between the objects to be estimated are used as correct labels for training to generate a machine learning model. An information processing device comprising a control unit that executes processing.

Images