Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device includes a database scene graph generation unit, a query scene graph generation unit, a scene graph matching unit, and an image search unit. The database scene graph generation unit generates a scene graph for each database image to be subjected to image search. The query scene graph generation unit generates a scene graph for a query image. The scene graph matching unit generates an attention map on the basis of the result of matching between the scene graph for each database image and the scene graph for the query image. The attention map shows a weight distribution in image analogy determination. The image search unit searches for a database image similar to the query image on the basis of the attention map.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present invention relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] A Visual Positioning System (VPS) is a position estimation system that, when a query image is given, matches it with a group of images in an image database or a 3D map, and returns the shooting position of the query image. In recent years, many applications based on VPS have been developed.

[0003] International Publication No. 2021 / 192686

[0004] It has been pointed out that VPS can cause false matching of feature points of moving objects in a database image when the moving object moves to a different position or disappears in a query image. While masking using semantic segmentation has been attempted as a countermeasure, masking can occur even when the moving object does not move, which can reduce the matching rate. It would be desirable to estimate the position using only peripheral information only when the moving object has moved, but this has not been achieved with existing technologies.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that are capable of suppressing false matches in image searches.

[0006] According to the present disclosure, there is provided an information processing device having a database scene graph generation unit that generates a scene graph of each database image to be the target of image search, a query scene graph generation unit that generates a scene graph of a query image, a scene graph matching unit that generates an attention map indicating a distribution of weights in image similarity judgment based on a matching result between the scene graph of each database image and the scene graph of the query image, and an image search unit that searches for the database images similar to the query image based on the attention map.The present disclosure also provides an information processing method in which information processing of the information processing device is executed by a computer, and a computer-readable non-transitory storage medium that stores a program for causing a computer to realize the information processing of the information processing device.

[0007] 1 is an explanatory diagram of a VPS. FIG. ... a diagram explaining a problem in image search. FIG. 1 is a diagram explaining a problem in image search. FIG. 1 is a diagram showing an example of generation of an attention map AT. FIG. 2 is a diagram showing an example of a self-position estimation process by a server. FIG. 2 is an explanatory diagram of a scene graph. FIG. 2 is an explanatory diagram of a scene graph. FIG. 3 is a diagram showing an overview of a 3D scene graph generation process. FIG. 3 is a diagram showing an example of a method for generating a 3D panoptic map. FIG. 4 is a diagram showing an example of an edge estimation technique. FIG. 4 is a diagram showing an example of a processing flow of a mapping unit. FIG. 5 is a diagram showing an example of a processing flow of a scene graph matching unit. FIG. 5 is a diagram showing a specific example of a score calculation method. FIG. 6 is a diagram showing a specific example of a score calculation method. FIG. 6 is a diagram showing an example of setting weights for an attention map. FIG. 7 is a diagram showing an example of a system configuration of a VPS of a second embodiment. FIG. 7 is a diagram showing a specific example of a score calculation method. FIG. 8 is a diagram showing an example of a hardware configuration of an information processing device.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The description will be given in the following order: [1. Background] [1-1. Overview of VPS] [1-2. Issues in image retrieval] [1-3. Overview of image retrieval according to the present disclosure] [2. First embodiment] [2-1. Example of system configuration] [2-2. Overview of scene graph] [2-3. Method for generating 3D scene graph] [2-4. 2D mapping of 3D scene graph] [2-5. Scene graph matching] [2-6. Generation of attention map] [2-7. Effects] [3. Second embodiment] [4. Example of hardware configuration]

[0010] [1. Background] [1-1. Overview of VPS] FIGS. 1 to 4 are explanatory diagrams of the VPS.

[0011] VPS is a cloud-based location identification system that uses images as input. VPS processing is performed offline using a map database (DB). Ｍ The process consists of the construction of the self-position PS and the online process of estimating the self-position PS.

[0012] The server SV uses SLAM (Simultaneous Localization and Mapping) to estimate a 3D model of the surrounding environment and the terminal's own position PS from the sensing data DT. The sensing data DT includes a group of images captured by the terminal EP while it is moving, IMU (Inertial Measurement Unit) information of the terminal EP, and the like. Color information (RGB sequence) and distance information (depth sequence) are acquired from the group of images. The sensing data DT may also include GPS (Global Positioning System) information of the terminal EP.

[0013] The server SV creates a map database DB based on the information estimated by the SLAM. Ｍ Build a map database Ｍ The data includes texture, shape, and 3D map data. The 3D map includes a group of images of the surrounding environment taken by the terminal EP while it is moving, coordinate information (feature point information) of feature points (key points) extracted from each image, and information about the shooting position and shooting posture of each image (camera pose information).

[0014] Map database DB Ｍ The image database DB stores the image group, feature point information, and camera pose information. Ｋ The server SV stores each image in a database image IM for image search. Ｋ The database image IM Ｋ is linked to feature point information and camera pose information and stored in the image database DB Ｋ is stored in

[0015] The server SV receives the query image IM from the client. Ｑ When the query image IM is received, the query image IM is generated by using a known method. Ｑ The server SV extracts feature point information from the image database DB. Ｋ The feature point information of the database images is stored in the feature database DB. Ｃ The server SV acquires the query image IM Ｑ The feature point information is stored in the feature database DB. Ｃ (feature point matching), and the matching database image IM Ｋ Match image IM Ｃ Search as.

[0016] The server SV stores the image database DB. Ｋ From, match image IM Ｃ The server SV acquires the camera pose information associated with the query image IM based on the acquired camera pose information. Ｑ The server SV notifies the client of the estimated photographing position and photographing attitude as the client's own position PS.

[0017] [1-2. Issues in Image Search] FIGS. 5 and 6 are diagrams for explaining issues in image search.

[0018] Database image IM Ｋ is a previously acquired image. Therefore, the current query image IM Ｑ If the difference is large, a mismatch may occur, which may reduce the estimation accuracy of the self-position PS. In the example of FIG. 5, the database image IM ＫWhen the moving object (the part surrounded by a frame) included in the query image moves to another position or disappears, the feature points of the moving object part induce false matching.

[0019] False matching can also occur when a repeated pattern exists in an image. In the example of Figure 6, a repeated pattern (the area surrounded by a frame) occurs when multiple vehicles with the same structure are connected. In this case, similar patterns are captured even when photographed from different positions, making false matching more likely.

[0020] As mentioned above, VPS has issues with the robustness of estimation. Regarding the first issue shown in Figure 5, in particular, some attempts have been made to address this issue by using masking techniques based on semantic segmentation. However, this masking process is performed even when the moving object does not move, which can reduce the matching rate. It would be desirable to estimate the position of a moving object using only peripheral information only when the moving object moves, but this has not been achieved with existing technologies.

[0021] One example of a robust VPS method that uses feature point filtering using semantic segmentation is the technology disclosed in Japanese Patent Application Laid-Open No. 2021-099275. This method proposes a self-localization system using an in-vehicle camera, extracts landmarks using semantic segmentation, and performs feature point matching focused on the landmarks.

[0022] The above method can perform feature point matching without including moving objects, and therefore can deal with false matching caused by moving objects such as cars and pedestrians. However, this method cannot solve the problem of the repeated pattern of a single object shown in Figure 6. In addition, in non-vehicle VPS systems, many images are included in which landmarks do not occupy the majority, so a significant drop in the matching rate is expected.

[0023] [1-3. Overview of Image Search in the Present Disclosure] The present disclosure has been made in consideration of the above-mentioned problems. Ｋ and the query image IM ＱWe generate scene graphs for both the first and second images, and generate an attention map based on matching between the scene graphs.

[0024] 7 is a diagram showing an example of generating an attention map AT. In the example of FIG. 7, Ｋ A 3D scene graph SG is generated from the database images stored in Ｓ The 3D scene graph SG is generated. Ｓ Database image IM Ｋ By mapping to the database image IM Ｋ The scene graph (database scene graph SG Ｋ ) is generated. In the example of FIG. Ｓ via the database scene graph SG Ｋ is generated, but the 3D scene graph SG Ｓ Without going through the database image IM Ｋ Directly from the database scene graph SG Ｋ may be generated.

[0025] A scene graph is a graph that abstracts the overall situation of a scene based on the types of elements in the scene and the relationships between them (positional and semantic relationships). The attention map AT shows the distribution of weights in image analogy judgments. Based on the results of matching between scene graphs, high weights are assigned to image regions that are recognized as similar. When matching between scene graphs, similarity is judged not only based on the proportion of similar elements but also on the relationships between the elements. This allows for accurate extraction of areas that should be focused on when judging image analogy.

[0026] In image retrieval, similarity judgment of images is performed while adjusting the weight of each feature point in the image based on the attention map AT. For example, in the example of Fig. 5, the weight of the feature point of a moving object changes depending on whether the moving object has moved. In the example of Fig. 6, the weight of each feature point is adjusted taking into consideration not only the proportion of the image occupied by the repeating pattern, but also the relationship between the repeating pattern and surrounding objects. This improves the accuracy of feature point matching and reduces erroneous matching.

[0027] 2. First Embodiment 2-1. System Configuration Example FIG. 8 is a diagram showing an example of the system configuration of a VPS according to the first embodiment.

[0028] The VPS service is provided by a server SV, which is an information processing device. The server SV functions as a database generation / VPS execution unit. Fig. 8 shows only the functional blocks related to the self-location estimation process. The server SV has a database scene graph generation unit GK, a query scene graph generation unit GQ, a scene graph matching unit MC, and an image search unit RT.

[0029] In FIG. 8, the query image capturing unit CM Ｑ The VPS result receiving unit RE is a client (a terminal such as a smartphone) that uses the VPS service. Ｑ is the query image IM Ｑ The VPS result receiving unit RE captures the query image IM and transmits it to the server SV. Ｑ The self-position PS estimated from the above is received from the server SV.

[0030] FIG. 9 is a diagram illustrating an example of the self-position estimation process performed by the server SV.

[0031] Image database DB Ｋ is the individual database image IM to be searched for. Ｋ The database scene graph generator GK stores each database image IM Ｋ Hereinafter, the database image IM Ｋ The scene graph of the database scene graph SG Ｋ Each database scene graph SG Ｋ is a 2D scene graph database Ｓ is stored in

[0032] For example, the database scene graph generator GK includes a 3D scene graph generator GS and a mapping unit MA. The 3D scene graph generator GS generates a plurality of database images IM to be used as the target of image search. Ｋ From 3D scene graph SG ＳThe mapping unit MA generates a 3D scene graph SG Ｓ Database image IM Ｋ The database scene graph SG is a 2D scene graph. Ｋ Generate.

[0033] The query scene graph generator GQ generates a query image IM Ｑ Hereinafter, the query image IM Ｑ Query the scene graph of SG Ｑ The query scene graph SG Ｑ is the database scene graph SG Ｋ is a 2D scene graph that is to be matched with.

[0034] The scene graph matching unit MC matches each database scene graph SG Ｋ and the query scene graph SG Ｑ Based on the matching results, the scene graph matching unit MC generates an attention map AT that indicates the distribution of weights in image similarity judgments during image retrieval.

[0035] The scene graph matching unit MC determines the contribution of each location to the analogy judgment based on the correspondence between the elements of the scene graphs and the relationships between the elements. The scene graph matching unit MC assigns a high weight to feature points of important locations that should be focused on in image retrieval.

[0036] The image search unit RT searches the image database DB based on the attention map AT. Ｋ A plurality of database images IM stored in Ｋ From the query image IM Ｑ Database images IM similar to Ｋ Match image IM Ｃ Search as.

[0037] [2-2. Overview of Scene Graph] FIGS. 10 and 11 are explanatory diagrams of the scene graph SG.

[0038] The scene graph SG defines components in an image as nodes ND and defines relationships between the nodes ND as edges EG. The scene graph SG abstracts the overall situation of a scene based on the types of nodes ND and the relationships between the nodes ND (edges EG).

[0039] The nodes ND are identified based on category information indicating object classes. The relationships indicated by the edges EG include positional relationships between the nodes ND (such as "on," "behind," "above," and "over") and semantic relationships between the nodes ND (such as "have," "wear," and "use").

[0040] In the example of FIG. 10, a 2D scene graph SG Ｐ The person in the center, the shirt, pants, helmet, and band worn by the person, and the skateboard used by the person are shown as nodes ND. In the example of FIG. 11, a 3D scene graph SG representing a three-dimensional scene is shown. Ｓ The sofa, table, shelf, and potted plants arranged in the living room are shown as nodes ND. Ｐ As a method for generating the metric, known methods using a Relation Proposal Network, Attentional GCVs, or the like can be used.

[0041] [2-3. 3D Scene Graph Generation Method] FIG. 12 shows the 3D scene graph SG Ｓ FIG. 10 is a diagram showing an outline of the generation process.

[0042] 3D Scene Graph SG Ｓ is generated based on a group of images obtained by capturing images of the sensing space. For example, each image is acquired as an RGB-D image containing color information and distance information within one pixel. In the example of FIG. 12, the color information is shown as an RGB image IM, and the distance information is shown as a depth image DP.

[0043] The 3D scene graph generator GS generates a 3D panoptic map PM of the sensing space based on the image group. Ｓ Generate a 3D panoptic map PMＳ The 3D scene graph generator GS uses machine learning and other techniques to estimate the relationships between the components and generate a 3D scene graph SG. Ｓ Generate.

[0044] FIG. 13 shows the 3D panoptic map PM. Ｓ FIG. 10 is a diagram illustrating an example of a method for generating the

[0045] The 3D scene graph generator GS applies panoptic segmentation to the RGB image IM to generate a 2D panoptic image PM. Ｐ Generates a 2D panoptic image PM Ｐ means an RGB image in which category information is assigned to each pixel.

[0046] The 3D scene graph generation unit GS acquires point cloud data of the sensing space and camera pose information of each RGB image IM using Visual SLAM. The 3D scene graph generation unit GS generates a corresponding 2D panoptic image PM based on the camera pose information of the RGB image IM. Ｐ This creates a 3D panoptic map PM in which each component is classified by category information. Ｓ is generated.

[0047] FIG. 14 is a diagram showing an example of a method for estimating the edge EG.

[0048] The 3D scene graph generator GS generates a 3D panoptic map PM Ｓ A plurality of components OB are extracted from the 3D scene graph SG, which are distinguished based on category information. Ｓ The node ND is set as the node ND.

[0049] The 3D scene graph generation unit GS sets, for each component OB, a bounding box BX (3D oriented bounding box) that fits the shape of the component OB. The 3D scene graph generation unit GS heuristically estimates the front direction of the bounding box BX based on affordances and geometric information.

[0050] The 3D scene graph generator GS heuristically estimates the positional relationships between the bounding boxes BX, and estimates the relationships between the constituent elements OB from the estimated positional relationships. The estimated relationships are stored in the 3D scene graph SG Ｓ The edge EG is expressed by a label indicating a relationship (relationship label), such as "on", "behind", "have", or "use".

[0051] [2-4. 2D Mapping of 3D Scene Graph] FIG. 15 is a diagram showing an example of the processing flow of the mapping unit MA.

[0052] In this example, ray casting is used to generate a 3D scene graph SG Ｓ and database image IM Ｋ In other words, the mapping unit MA links the objects included in the database image IM Ｋ The mapping unit MA uses ray casting to identify one or more objects contained in the 3D scene graph SG Ｓ The mapping unit MA extracts one or more nodes ND corresponding to one or more objects and edges EG between the nodes ND from the database image IM. Ｋ Mapping to the database scene graph SG Ｋ This will be explained in detail below.

[0053] First, the mapping unit MA Ｋ From the database image IM as the input image ＫThe mapping unit MA applies instance segmentation such as Mask R-CNN to the input image to identify objects in the input image that will become nodes ND (step S1). Thereafter, the mapping unit MA performs the following process for each object.

[0054] The mapping unit MA casts a ray from the camera position toward the center position of the object (step S2). The mapping unit MA divides the ray at regular intervals (step S3), and for each divided ray, it finds a 3D scene graph SG that is within a threshold distance from the ray and has not yet been mapped. Ｓ The mapping unit MA determines whether or not a node ND of the same category as the result of instance segmentation exists (step S5), and if a node ND of the same category exists (step S5: Yes), extracts the node ND with the closest distance (step S6).

[0055] The mapping unit MA determines whether or not the node ND has been extracted (step S7). If the node ND has been extracted (step S7: Yes), the mapping unit MA extracts the 3D scene graph SG Ｓ The nodes ND are mapped to the centers of the objects in the input image detected by the instance segmentation (step S8). The mapping unit MA maps the relationships (edges EG) between the mapped nodes ND to the 3D scene graph SG. Ｓ and map it onto the input image (step S9). Ｓ a database scene graph SG with nodes ND and edges EG corresponding to Ｋ is generated.

[0056] The technique in FIG. 15 uses ray casting to generate a 3D scene graph SG Ｓ and database image IM Ｋ However, the 2D mapping method is not limited to this. It is also possible to generate an image equivalent to the database image by rendering using point cloud data in the sensing space.

[0057] For example, the mapping unit MA may map a 3D scene graph SG Ｓ The mapping unit MA maps the 3D mesh to the database image IM. Ｋ The mapping unit MA generates a viewpoint image by rendering from the shooting viewpoint of the 3D scene graph SG Ｓ The mapping unit MA extracts one or more nodes ND and edges EG between the nodes ND that appear in the viewpoint image from the database image IM. Ｋ Mapping to database scene graph SGK を Generate.

[0058] 2-5. Scene Graph Matching FIG. 16 is a diagram showing an example of the processing flow of the scene graph matching unit MC.

[0059] In the method of this example, the correspondence between nodes ND and the correspondence between edges EG are scored, and the similarity between scene graphs SG is determined based on the magnitude of the score. Ｋ About the database scene graph SG Ｋ and the query scene graph SG Ｑ The scene graph matching unit MC calculates a score for the location of each node ND based on the presence or absence of a corresponding node ND, the presence or absence of a corresponding edge EG, and the identity of the relationship labels between the corresponding edges EG.

[0060] The scene graph matching unit MC uses the database scene graph SG Ｋ and the query scene graph SG ＱAll nodes ND included in the map image MI (see FIG. 19) are aggregated onto a single map image MI. For each pixel PX (see FIG. 19) in the map image MI, the scene graph matching unit MC extracts one or more nodes ND that are within a threshold distance from the pixel PX, and calculates a weighted average of the scores of each extracted node ND according to the distance from the node ND as the weight of the pixel PX. The scene graph matching unit MC obtains the map image MI in which weights are set for each pixel PX as an attention map AT. This will be explained in detail below.

[0061] First, the scene graph matching unit MC matches a query scene graph SG Ｑ For each node ND in the database scene graph SG Ｋ All nodes ND in the same category are listed (step S11).

[0062] The scene graph matching unit MC matches the query scene graph SG Ｑ For each node ND in Ｋ The scene graph matching unit MC creates all patterns by selecting the nodes ND in the database scene graph SG Ｋ The patterns having overlapping nodes ND are excluded (step S13).

[0063] This results in a query scene graph SG Ｑ and database scene graph SG Ｋ All combination patterns of nodes ND are obtained for both the scene graph SG and the scene graph SG. However, the number of nodes ND in the other scene graph SG corresponding to one node ND is either 1 or 0. Furthermore, the corresponding nodes ND are limited to nodes ND in the same category.

[0064] The scene graph matching unit MC performs the following process for each combination pattern: First, the scene graph matching unit MC matches the query scene graph SG Ｑ Each node ND and the corresponding database scene graph SG Ｋ(Step S14). The scene graph matching unit MC extracts the nodes ND of the extracted database scene graph SG Ｋ The scene graph matching unit MC extracts all edges EG between the nodes ND in the scene graph (step S15). The scene graph matching unit MC records the total number of extracted nodes ND and edges EG (step S16).

[0065] After the above-mentioned process is completed for each combination pattern, the scene graph matching unit MC selects the pattern with the largest total number of nodes ND and edges EG (step S17).

[0066] [2-6. Generation of Attention Map] The scene graph matching unit MC generates an attention map AT based on the combination selected in step S17.

[0067] The scene graph matching unit MC determines the score of the location where each node ND exists in the above combination pattern according to the similarity of the correspondence between the node ND and the edge EG as follows: For example, if only one node ND exists, the score is set to 0. If both nodes ND exist and the edges EG match perfectly, the score is set to 1. If both nodes ND exist but the edges EG do not match perfectly, the score is set to 0 to 1 according to the number of matching edges EG.

[0068] The scene graph matching unit MC determines a score for each pixel by taking a weighted average of the values ​​of nodes ND within a distance of a threshold value according to the distance from the pixel. The scene graph matching unit MC generates an attention map AT based on the score of each pixel. However, if no node ND exists within a distance of the threshold value, the scene graph matching unit MC refers to the value of the node ND closest to the pixel.

[0069] 17 and 18 are diagrams showing specific examples of score calculation methods.

[0070] Query Scene Graph SG Ｑ and database scene graph SG ＫThe score is determined for all nodes ND included in the list ND according to the following rule: First, if a corresponding node ND exists, the number of matching edges EG and the number of non-matching edges EG are counted. The score is determined by dividing the number of matching edges EG by the total number of edges. If a corresponding node ND does not exist, the score is calculated as 0.

[0071] In the example of FIG. 17, the query scene graph SG Ｑ The node ND is the node ND Ｑ１ and the corresponding database scene graph SG Ｋ The node ND is the node ND Ｂ１ Node ND Ｑ１ There are two nodes ND (ND Ｑ２ , N.D. Ｑ３ ) exists and the node ND Ｂ１ There are three nodes ND (ND Ｂ２ , N.D. Ｂ３ , N.D. Ｂ４ ) exist. There are two corresponding nodes ND Ｑ１ , N.D. Ｂ１ The number of matching edges EG is two and the number of mismatching edges EG is three between the nodes ND Ｑ１ and node ND Ｂ１ The score is 2 / (2+3)=0.4.

[0072] FIG. 18 shows the node ND for which the score is calculated. Ｂ４ As shown in FIG. Ｂ４ The query scene graph SG corresponding to Ｑ Therefore, there is no node ND Ｂ４ will receive a score of 0.

[0073] FIG. 19 is a diagram showing an example of setting the weights of the attention map AT.

[0074] The scene graph matching unit MC uses the database scene graph SG Ｋ and the query scene graph SG ＱAll the nodes ND included in the map image MI are aggregated onto one map image MI. The scene graph matching unit MC sets weights for all the pixels PX on the map image MI according to the following rules.

[0075] First, if one or more nodes ND are present within a distance from the pixel PX that is within a threshold value, the scene graph matching unit MC obtains the scores of all nodes ND that are within the distance from the pixel PX that is within the threshold value, and the distance from the pixel PX to each node ND. The scene graph matching unit MC calculates a weighted average of the scores of each node ND according to the distance as the weight of the pixel PX. The threshold value can be set arbitrarily by the system developer.

[0076] If there is no node ND within the threshold distance from the pixel PX, the scene graph matching unit MC calculates the score of the node ND closest to the pixel PX as the weight of the pixel PX.

[0077] In the example at the top of FIG. 19, the pixel PX is ｔｈ There are two nodes ND in the range. One node ND has a score of 0.4 and a distance to pixel PX of 18 pixels. The other node ND has a score of 0.0 and a distance to pixel PX of 12 pixels. Therefore, the weight of pixel PX is (0.4 x 18 + 0.0 x 12) / (18 + 12) = 0.24.

[0078] In the example at the bottom of FIG. 19, the pixel PX is ｔｈ There is no node ND in the range of . The score of the node ND closest to the pixel PX is 0.8. Therefore, the weight of the pixel PX is 0.8.

[0079] In the above example, the attention map AT is generated according to an algorithm. However, the above process can also be performed using, for example, a graph neural network that takes two scene graphs SG as inputs and outputs the attention map AT.

[0080] [2-7. Effects] The server SV has a database scene graph generation unit GK, a query scene graph generation unit GQ, a scene graph matching unit MC, and an image search unit RT. The database scene graph generation unit GK generates individual database images IM to be the target of image search. Ｋ Scene graph SG (database scene graph SG Ｋ The query scene graph generator GQ generates a query image IM Ｑ The scene graph SG (query scene graph SG Ｑ The scene graph matching unit MC generates a scene graph SG Ｋ and the query scene graph SG Ｑ The attention map AT indicates the distribution of weights in the image similarity judgment. The image retrieval unit RT generates an attention map AT based on the matching result with the query image IM. Ｑ Database images IM similar to Ｋ In the information processing method of the present disclosure, the processing of the server SV is executed by a computer. In the program of the present disclosure, the processing of the server SV is executed by a computer.

[0081] According to this configuration, the attention map AT can be used to specify areas to focus on in analogy judgment. The attention map AT is acquired by matching the scene graphs SG of the images to be compared. This enables analogy judgment of images that also takes into account the relationship with surrounding objects. For example, the movement of a moving object can be estimated based on the matching results between the scene graphs SG. Since the weights of feature points in image retrieval can be changed depending on whether the moving object has moved, erroneous matching in image retrieval can be reduced. Even when repetitive patterns occupy a large proportion of an image, the weights of individual feature points are adjusted taking into account the relationship with surrounding objects, thereby improving the accuracy of feature point matching and reducing erroneous matching.

[0082] The database scene graph generator GK includes a 3D scene graph generator GS and a mapping unit MA. The 3D scene graph generator GS generates a plurality of database images IM to be used as the target of image search. Ｋ From 3D scene graph SG Ｓ The mapping unit MA generates a 3D scene graph SG Ｓ Database image IM Ｋ The database scene graph SG is a 2D scene graph. Ｋ Generate.

[0083] According to this configuration, the database scene graph SG Ｋ 3D scene graph SG Ｓ The 3D scene graph SG is generated via Ｓ The 3D scene graph SG is generated from a database of images of the subject taken from multiple angles, resulting in highly accurate spatial data with the effects of occlusion suppressed. Ｓ The database scene graph SG generated via Ｋ Query the scene graph SG Ｑ By matching with this, the accuracy of the matching process is improved.

[0084] The mapping unit MA maps the database image IM Ｋ The mapping unit MA uses ray casting to identify one or more objects contained in the 3D scene graph SG Ｓ The mapping unit MA extracts one or more nodes ND corresponding to one or more objects and edges EG between the nodes ND from the database image IM. Ｋ Mapping to the database scene graph SG Ｋ Generate.

[0085] According to this configuration, a highly accurate database scene graph SG Ｋ is generated.

[0086] The mapping unit MA is a 3D scene graph SG Ｓ The mapping unit MA obtains a 3D mesh of the space represented by the database image IM.Ｋ The mapping unit MA generates a viewpoint image by rendering from the shooting viewpoint of the 3D scene graph SG Ｓ The mapping unit MA extracts one or more nodes ND and edges EG between the nodes ND that appear in the viewpoint image from the database image IM. Ｋ Mapping to the database scene graph SG Ｋ Generate.

[0087] According to this configuration, a highly accurate database scene graph SG Ｋ is generated.

[0088] The scene graph matching unit MC matches each database scene graph SG Ｋ About the database scene graph SG Ｋ and the query scene graph SG Ｑ The scene graph matching unit MC calculates a score for the location of each node ND based on the presence or absence of a corresponding node ND, the presence or absence of a corresponding edge EG, and the identity of the relationship labels between the corresponding edges EG. The scene graph matching unit MC calculates a score for the location of each node ND based on the presence or absence of a corresponding node ND, the presence or absence of a corresponding edge EG, and the identity of the relationship labels between the corresponding edges EG. Ｋ and the query scene graph SG Ｑ The scene graph matching unit MC aggregates all the nodes ND included in the map image MI onto one map image MI. For each pixel PX of the map image MI, the scene graph matching unit MC calculates a threshold value r ｔｈ The scene graph matching unit MC extracts one or more nodes ND within a distance of 1 / 2, and calculates a weighted average of the scores of each extracted node ND according to the distance from the node ND as the weight of the pixel PX. The scene graph matching unit MC obtains a map image MI in which a weight is set for each pixel PX as an attention map AT.

[0089] According to this configuration, an appropriate attention map AT is generated based on the correspondence between the nodes ND, the correspondence between the edges EG, and the identity of the relation labels between the corresponding edges EG.

[0090] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0091] 3. Second Embodiment FIG. 20 is a diagram showing an example of the system configuration of a VPS according to a second embodiment.

[0092] The present embodiment differs from the first embodiment in that emphasis is placed on edges EG that indicate specific relationships (relationship labels) when calculating the score of a node ND. The differences from the first embodiment will be described below.

[0093] The server SV has a relationship prior management unit RP. The relationship prior management unit RP stores the weights of relationship labels. The system developer can arbitrarily set what weights are assigned to which relationship labels. For example, to suppress false matching due to moving objects, a high weight can be assigned to a relationship label that indicates a strong connection between nodes ND.

[0094] The scene graph matching unit MC determines whether the relational labels of the edges EG are in the query image IM. Ｑ The database image IM is identical to the scene graph SG of Ｋ The scene graph matching unit MC calculates the ratio of the scene graph SG of the node ND to the weight of the relation label. The scene graph matching unit MC multiplies the calculated ratio by the weight of the relation label to calculate a weighted matching rate as the score of the node ND.

[0095] FIG. 21 is a diagram showing a specific example of a score calculation method.

[0096] 21, the weights of the relationship labels indicating "near," "above," "left," and "right" are set to 1.0, 0.3, 0.5, and 0.5, respectively. The relationship prior management unit RP provides this weight information to the scene graph matching unit MC as prior knowledge.

[0097] As in the example of FIG. 17, the query scene graph SG Ｑ The node ND is the node ND Ｑ１ and the corresponding database scene graph SG ＫThe node ND is the node ND Ｂ１ Node ND Ｑ１ There are two nodes ND (ND Ｑ２ , N.D. Ｑ３ ) exists and the node ND Ｂ１ There are three nodes ND (ND Ｂ２ , N.D. Ｂ３ , N.D. Ｂ４ ) exist. There are two corresponding nodes ND Ｑ１ , N.D. Ｂ１ Between nodes ND and ND, the number of matching edges EG is two and the number of mismatching edges EG is three. Ｑ１ and node ND Ｂ１ The score is 1.0×2 / (1.0×2+1.0×2+0.3×1)=0.47.

[0098] According to this embodiment, an appropriate attention map AT is generated taking into consideration the identity of the relation labels between corresponding edges EG and the weights of the relation labels, thereby improving the robustness of image retrieval.

[0099] 4. Example of Hardware Configuration FIG. 22 is a diagram showing an example of the hardware configuration of the server SV.

[0100] The information processing of the server SV is realized by, for example, a computer 1000. The computer 1000 has a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0101] The CPU 1100 operates and controls each component based on a program (program data 1450) stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0102] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000 .

[0103] The HDD 1400 is a non-transitory computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to an embodiment as an example of program data 1450.

[0104] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0105] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0106] For example, when the computer 1000 functions as the server SV according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of the aforementioned components. The information processing program, various models, and various data according to the present disclosure are stored in the HDD 1400. The CPU 1100 reads and executes the program data 1450 from the HDD 1400. Alternatively, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0107] [Additional Notes] The present technology may also be configured as follows: (1) An information processing device comprising: a database scene graph generation unit that generates a scene graph of each database image that is the target of image retrieval; a query scene graph generation unit that generates a scene graph of a query image; a scene graph matching unit that generates an attention map indicating a distribution of weights in image similarity judgment based on a matching result between the scene graph of each database image and the scene graph of the query image; and an image retrieval unit that searches for the database images that are similar to the query image based on the attention map. (2) The information processing device described in (1), wherein the database scene graph generation unit comprises: a 3D scene graph generation unit that generates a 3D scene graph from multiple database images that are the target of image retrieval; and a mapping unit that maps the 3D scene graph onto the database image to generate a scene graph for the database image that is a 2D scene graph. (3) The information processing device according to (2), wherein the mapping unit identifies one or more objects included in the database image, extracts from the 3D scene graph one or more nodes and edges between the nodes corresponding to the one or more objects using ray casting, and generates a scene graph for the database image by mapping the extracted one or more nodes and edges between the nodes onto the database image. (4) The information processing device according to (2), wherein the mapping unit obtains a 3D mesh of a space represented by the 3D scene graph, generates a viewpoint image by rendering the 3D mesh from a shooting viewpoint of the database image, extracts from the 3D scene graph one or more nodes and edges between the nodes that appear in the viewpoint image, and maps the extracted one or more nodes and edges between the nodes onto the database image.(5) The information processing device according to any one of (2) to (4), wherein the scene graph matching unit: detects, for each of the scene graphs of the database image, correspondences between nodes and edges included in the scene graph of the database image and nodes and edges included in the scene graph of the query image; calculates a score for the location of each node based on the presence or absence of corresponding nodes, the presence or absence of corresponding edges, and the identity of the relationship labels between the corresponding edges; aggregates all nodes included in the scene graph of the database image and the scene graph of the query image onto a single map image; extracts, for each pixel of the map image, one or more nodes that are within a threshold distance from the pixel; calculates a weight for the pixel by weighting the scores of each extracted node according to the distance from the node; and obtains the map image with the weight set for each pixel as the attention map. (6) The information processing device according to (5), further comprising: a relation prior management unit that stores weights of the relation labels, and the scene graph matching unit calculates a proportion of scene graphs of the database image in which the relation labels of the edges are the same as those of the scene graph of the query image, and calculates a weighted match rate as the score by multiplying the proportion by the weight of the relation labels. (7) An information processing method executed by a computer, comprising: generating scene graphs of individual database images to be subjected to image retrieval; generating a scene graph of a query image; generating an attention map indicating a distribution of weights in image similarity judgment based on a matching result between the scene graph of each database image and the scene graph of the query image; and searching for the database images similar to the query image based on the attention map.(8) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: generate a scene graph for each database image that is the subject of image search; generate a scene graph for a query image; generate an attention map indicating the distribution of weights in image similarity judgments based on the matching results between the scene graphs of each database image and the scene graph of the query image; and search for the database images that are similar to the query image based on the attention map.

[0108] AT Attention map EG Edge GK Database scene graph generator GQ Query scene graph generator GS 3D scene graph generator IM Ｋ Database image IM Ｑ Query image MA Mapping unit MC Scene graph matching unit MI Map image ND Node PX Pixel RP Relation prior management unit RT Image search unit r ｔｈ Threshold SG Scene graph SG Ｋ Database Scene Graph SG Ｑ Query Scene Graph SG Ｓ 3D scene graph SV server (information processing device)

Claims

1. An information processing device having: a database scene graph generation unit that generates a scene graph of each database image to be the subject of image search; a query scene graph generation unit that generates a scene graph of a query image; a scene graph matching unit that generates an attention map indicating the distribution of weights in image similarity judgment based on the matching results between the scene graph of each database image and the scene graph of the query image; and an image search unit that searches for database images similar to the query image based on the attention map.

2. The information processing device according to claim 1, wherein the database scene graph generation unit comprises: a 3D scene graph generation unit that generates a 3D scene graph from a plurality of database images that are the subject of image search; and a mapping unit that maps the 3D scene graph onto the database images to generate a scene graph for the database images, which is a 2D scene graph.

3. The information processing device of claim 2, wherein the mapping unit identifies one or more objects included in the database image, uses ray casting to extract one or more nodes and edges between the nodes corresponding to the one or more objects from the 3D scene graph, and maps the extracted one or more nodes and edges between the nodes onto the database image to generate a scene graph for the database image.

4. The information processing device of claim 2, wherein the mapping unit: acquires a 3D mesh of the space represented by the 3D scene graph; renders the 3D mesh from the shooting viewpoint of the database image to generate a viewpoint image; extracts from the 3D scene graph one or more nodes and edges between the nodes that appear in the viewpoint image; and maps the extracted one or more nodes and edges between the nodes onto the database image to generate a scene graph for the database image.

5. The information processing device of claim 2, wherein the scene graph matching unit: detects, for each of the scene graphs of the database image, a correspondence between the nodes and edges included in the scene graph of the database image and the nodes and edges included in the scene graph of the query image; calculates a score for the location of each node based on the presence or absence of corresponding nodes, the presence or absence of corresponding edges, and the identity of the relationship labels between the corresponding edges; aggregates all nodes included in the scene graphs of the database image and the scene graphs of the query image onto a single map image; extracts, for each pixel of the map image, one or more nodes that are within a threshold distance from the pixel; calculates a weighted average of the scores of each extracted node according to the distance from the node as the weight of the pixel; and obtains the map image in which the weight is set for each pixel as the attention map.

6. An information processing device as described in claim 5, further comprising a relation prior management unit that stores weights of the relation labels, and wherein the scene graph matching unit calculates a proportion of scene graphs of the database image in which the relation labels of the edges are identical to the scene graph of the query image, and calculates as the score a weighted match rate obtained by multiplying the proportion by the weight of the relation label.

7. An information processing method executed by a computer, comprising: generating a scene graph for each database image to be subjected to image search; generating a scene graph for a query image; generating an attention map indicating a distribution of weights in image similarity judgment based on a matching result between the scene graph of each database image and the scene graph of the query image; and searching for the database images similar to the query image based on the attention map.

8. A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following steps: generate a scene graph for each database image that is to be the subject of image search; generate a scene graph for a query image; generate an attention map indicating a distribution of weights in image similarity judgment based on the matching results between the scene graphs of each database image and the scene graph of the query image; and search for the database images that are similar to the query image based on the attention map.

Images