Cross-modal retrieval method and device, electronic device, and storage medium

Abstract

The present disclosure provides a cross-modal retrieval method and device, electronic device, and storage medium, which belongs to the field of image processing technology. The method includes: extracting a first scene graph based on a target image; extracting a second scene graph based on point cloud data; projecting the second scene graph based on the pose of the target image, and screening multiple third scene graphs; the third scene graph is a local two-dimensional scene graph in the second scene graph; respectively calculating the first similarity between the first scene graph and the third scene graph, screening a fourth scene graph from the third scene graph based on the first similarity calculation result, and determining the fourth scene graph as the retrieval result corresponding to the target image. The cross-modal retrieval method and device, electronic device, and storage medium provided by the present disclosure can improve the accuracy of cross-modal retrieval.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and more specifically, relates to a cross-modal retrieval method and device, an electronic device, and a storage medium. Background Art

[0002] Cross-modal retrieval technology refers to establishing a correspondence between two or more modalities (such as text, images, audio, video, 3D models, etc.), so that users can retrieve information from one modality through information from another modality. For example, image-point cloud cross-modal retrieval is a technology that associates and matches 2D images and 3D point cloud data, and has broad application prospects in the fields of autonomous driving, robot navigation, 3D reconstruction, etc.

[0003] Since images and point cloud data have significant heterogeneity in their underlying representations, images are composed of pixels and contain rich color, texture, and shape information, while point cloud data is composed of points in space and their attributes (such as color, intensity, depth, etc.), and focuses more on expressing the geometric shape and spatial distribution of objects. This difference makes it very difficult to directly associate and compare the data of the two modalities. Existing technologies usually learn a common high-dimensional space and represent the features of images and point clouds in this space, and use similarity calculation to capture the potential association between the modalities to achieve cross-modal retrieval. However, due to the heterogeneity of the two modal data, it is not easy to find a common feature representation space that can simultaneously express their key information. At the same time, due to the large difference between the two modal data, traditional similarity measurement methods may not accurately reflect the similarity relationship between them. In addition, with the increase in the number and scale of point cloud models, the computational overhead of similarity calculation and retrieval is also a problem that needs to be solved, which also limits the scalability of related technologies. Summary of the invention

[0004] The purpose of the present disclosure is to provide a cross-modal retrieval method and device, an electronic device, and a storage medium to improve the accuracy of cross-modal retrieval.

[0005] A first aspect of the embodiments of the present disclosure provides a cross-modal retrieval method, including:

[0006] Extracting a first scene graph based on the target image; the first scene graph is used to represent multiple objects in the target image and the relationship between the objects;

[0007] Extracting a second scene graph based on the point cloud data; the second scene graph is used to represent multiple objects in the point cloud data and the relationship between each object;

[0008] Projecting the second scene graph based on the pose of the target image, and screening a plurality of third scene graphs; the third scene graph is a local two-dimensional scene graph in the second scene graph;

[0009] A first similarity between the first scene graph and the third scene graph is calculated respectively, a fourth scene graph is selected from the third scene graph based on the first similarity calculation result, and the fourth scene graph is determined as a retrieval result corresponding to the target image.

[0010] A second aspect of the embodiments of the present disclosure provides a cross-modal retrieval device, including:

[0011] A first extraction module, used to extract a first scene graph from a target image; the first scene graph is used to represent multiple objects in the target image and the relationship between the objects;

[0012] A second extraction module is used to extract a second scene graph from the point cloud data; the second scene graph is used to represent multiple objects in the point cloud data and the relationship between each object;

[0013] A first screening module, configured to perform projection of the second scene graph based on the position and posture of the target image, and screen a plurality of third scene graphs; the third scene graph is a local two-dimensional scene graph in the second scene graph;

[0014] The second screening module is used to calculate the first similarity between the first scene graph and the third scene graph respectively, screen the fourth scene graph from the third scene graph based on the first similarity calculation result, and determine the fourth scene graph as the retrieval result corresponding to the target image.

[0015] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned cross-modal retrieval method when executing the computer program.

[0016] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned cross-modal retrieval method are implemented.

[0017] The cross-modal retrieval method and device, electronic device, and storage medium provided by the embodiments of the present disclosure have the following beneficial effects:

[0018] In the disclosed embodiment, by extracting a two-dimensional first scene graph from the target image, image information can be abstracted from a semantic level; by extracting a three-dimensional second scene graph from the point cloud data, point cloud information can be abstracted from a semantic level; the second scene graph is projected based on the pose of the target image, and multiple third scene graphs are screened, so that the second scene graph can be cropped to reduce the amount of calculation in the subsequent matching process; the first scene graph and the third scene graph are matched by calculating the first similarity, thereby obtaining a fourth scene graph with the highest matching degree with the first scene graph.

[0019] The disclosed embodiments extract two-dimensional and three-dimensional scene graphs from image and point cloud data respectively, and uniformly abstract the two modal data information of image and point cloud to a higher-dimensional semantic level, thereby avoiding latent space learning and similarity calculation from the feature level, which is beneficial to improving the accuracy of retrieval. Furthermore, it increases scalability while reducing the computational overhead of cross-modal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram of a cross-modal search method according to an embodiment of the present disclosure;

[0022] Figure 2 A structural diagram of an example segmentation network provided by an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of an instance segmentation network training process provided by an embodiment of the present disclosure;

[0024] Figure 4 A diagram of a transformation network structure provided by an embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of a single-layer graph convolution operation provided by an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of a scene graph generation network training process based on an attention mechanism provided in an embodiment of the present disclosure;

[0027] Figure 7 A structural block diagram of a cross-modal retrieval device provided by an embodiment of the present disclosure;

[0028] Figure 8 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0030] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0031] Please refer to Figure 1 , Figure 1 A schematic diagram of a cross-modal search method provided by an embodiment of the present disclosure, the method comprising:

[0032] S101: extracting a first scene graph based on a target image; the first scene graph is used to represent multiple objects in the target image and the relationship between the objects.

[0033] In this embodiment, the target image is the image input by the user during retrieval. A pre-trained target detection network (such as R-CNN, YOLO, etc.) can be used to detect different objects from the target image, extract the bounding box, category label and corresponding feature representation of the object. Among them, the bounding box is usually represented by a rectangular box, which is used to determine the position and size of the object in the image; the category label is used to indicate the category to which the object belongs, such as "person", "car", "tree", etc.; the feature representation is an abstract representation of the object's features, usually a high-dimensional vector, which contains the object's color, shape, texture and other feature information, and these features can be used for subsequent analysis and processing.

[0034] After obtaining the above information about the object, a relationship prediction network based on a graph neural network, such as GraphSAGE (Graph Sample and Aggregate) and GAT (Graph Attention Network), can be used to predict the semantic relationship between the objects based on their features and spatial information, and obtain the relationship edges between the objects. Based on the generated object nodes and relationship edges, a graph neural network is used to obtain the first scene graph, which represents the objects in the image and the relationships between them in a structured way.

[0035] S102: Extracting a second scene graph based on the point cloud data; the second scene graph is used to represent multiple objects in the point cloud data and the relationship between the objects. The process of extracting the second scene graph based on the point cloud data includes:

[0036] S1021: Perform instance segmentation on the point cloud data.

[0037] In this embodiment, a clustering algorithm (such as density-based clustering algorithm DBSCAN, K-Means, etc.) can be first used to segment the point cloud data into different objects. The above algorithm divides the point cloud into different clusters based on features such as the distance and density between points, and each cluster represents a possible object.

[0038] Then calculate the spatial relationship between different objects, such as distance, relative position (up and down, left and right, front and back, etc.), containment relationship, etc. The spatial relationship can be determined by calculating the distance between the centroids of the objects, direction vectors, etc.

[0039] S1022: Generate the second scene graph based on the instance segmentation result.

[0040] Based on instance segmentation of point cloud data, each segmented object is represented as a node in the second scene graph. The attributes of the node include the features of the object (such as geometric features, local features, etc.) and the category label. According to the relationship between the extracted objects, edges are added to the second scene graph. The attributes of the edges may include the type of relationship (such as spatial relationship, semantic relationship, etc.) and the strength of the relationship.

[0041] S103: Projecting the second scene graph based on the position and posture of the target image, and screening a plurality of third scene graphs; the third scene graph is a local two-dimensional scene graph in the second scene graph.

[0042] In this embodiment, the pose of the target image is the pose of the camera that captures the target image. The camera pose is used to project the 3D point cloud to determine the visible instance objects in the corresponding 2D image perspective. On this basis, a local 2D scene graph is obtained by screening from the 3D scene graph. During the projection process, the bounding box of each instance object is obtained by instance annotation and annotated in the image.

[0043] Specifically, the image camera pose is used as input, and the camera spatial pose during image acquisition can be obtained through the projection process. The pose is used to calculate the visible local information in each point cloud data. , the object's center of mass can be obtained by calculating its bounding box , and the length, width and height attributes of the three-dimensional bounding box , , By projecting the object onto the XY plane, the range occupied by the object in space is obtained, and by determining whether the point cloud instance object is within the camera projection perspective, the three-dimensional scene graph is projected to obtain the local two-dimensional scene graph (also known as the third scene graph) corresponding to the image.

[0044] S104: Calculate first similarities between the first scene graph and the third scene graph respectively, select a fourth scene graph from the third scene graph based on the first similarity calculation result, and determine the fourth scene graph as a retrieval result corresponding to the target image.

[0045] In this embodiment, the graph similarity comparison is transformed into a set similarity comparison problem, and the Simpson metric is used to calculate the similarity score and sort it to obtain the optimal retrieval matching result. When comparing the similarity of two sets, the elements added to the set include the semantic label of the instance object, the semantic label of the relationship between objects, and the triple composed of the semantic relationship. The similarity measurement of the graph topology is replaced by the comparison of the similarity between sets, and the calculation of the similarity score is specifically as follows:

[0046]

[0047] in, The Simpson function is selected as the standard for similarity measurement. Defined as an enhanced graph structure ,in , , R represent the semantic label of the instance object, the semantic label of the relationship between objects, and the semantic relationship, respectively. Represents a collection.

[0048] The Simpson metric expression is as follows:

[0049]

[0050] A and B are two independent sets. When the number of elements in set A and set B is very different, the similarity between the two sets can be compared more discriminatively. Since a two-dimensional image is only a partial projection of a scene, there is often a certain gap in the number of instance objects compared to a three-dimensional scene graph. Therefore, in the task of retrieving two-dimensional images to three-dimensional scenes, the Simpson metric is a more appropriate choice.

[0051] From the above, it can be concluded that this embodiment can abstract image information from a semantic level by extracting a two-dimensional first scene graph from the target image; can abstract point cloud information from a semantic level by extracting a three-dimensional second scene graph from the point cloud data; can project the second scene graph based on the pose of the target image, screen multiple third scene graphs, and crop the second scene graph to reduce the amount of calculation in the subsequent matching process; and can match the first scene graph and the third scene graph by calculating the first similarity to obtain a fourth scene graph that has the highest match with the first scene graph.

[0052] The disclosed embodiments extract two-dimensional and three-dimensional scene graphs from image and point cloud data respectively, and uniformly abstract the two modal data information of image and point cloud to a higher-dimensional semantic level, thereby avoiding latent space learning and similarity calculation from the feature level, which is beneficial to improving the accuracy of retrieval. Furthermore, it increases scalability while reducing the computational overhead of cross-modal retrieval.

[0053] In one embodiment of the present disclosure, instance segmentation is performed on point cloud data, including:

[0054] Extract local feature vectors of point cloud data;

[0055] Calculate the second similarity between each local feature vector to obtain a similarity matrix;

[0056] The point cloud data is grouped based on the similarity matrix to obtain multiple prediction candidate groups;

[0057] Perform convolution operation on each local feature vector to obtain the confidence level corresponding to each prediction candidate group;

[0058] Based on the confidence level corresponding to each prediction candidate group, multiple prediction candidate groups are screened to obtain instance segmentation results.

[0059] In this embodiment, the 3D instance segmentation task takes the original point cloud data as input and generates the instance label of the instance object for each point. The point cloud data of 3D points is used as input and passed into the feedforward feature extraction dynamic graph convolution network. The features are reduced in dimension and learnable parameters are added through a 1×1 convolutional neural network. The similarity matrix is ​​constructed by calculating the Euclidean distance of the feature space output by the network, thereby realizing instance segmentation of the point cloud. The network structure is shown in the figure below. Figure 3 shown.

[0060] Figure 3In this paper, point cloud feature extraction, feature similarity measurement, and candidate group prediction and elimination are the core. Point cloud feature extraction based on dynamic graph structure and candidate group prediction based on similarity matrix are proposed in turn to improve the effectiveness of feature extraction and the accuracy of instance segmentation tasks. The core of the algorithm is to normalize the point cloud data to the unit space, use symmetric equations to solve the disorder of point cloud data, and construct a dynamic graph adjacency structure between points to extract the local geometric feature vector of the point cloud; use 1×1 convolution to reduce the dimension of the point cloud feature extraction vector, use the point cloud feature vector generated by the network to calculate the Euclidean distance between vectors to construct a similarity matrix, and group the points in the point cloud data according to the similarity matrix. Points with high similarity are likely to belong to the same instance object and can be grouped together, thereby obtaining multiple prediction candidate groups; use 1×1 convolution to construct a confidence map branch, and use the predicted confidence map to generate a confidence for each prediction candidate group. According to the confidence, possible instance objects can be screened from multiple prediction candidate groups.

[0061] During the instance segmentation network training phase, the network parameters are learned using the intersection-over-union ratio between the candidate groups and the true values ​​as supervision, and the candidate groups are screened using the predicted confidence.

[0062] The dynamic graph network structure can be used to extract local and global features of the point cloud and generate a matrix containing feature vectors. , the instance segmentation network structure of this paper consists of two branches, the matrix Pass in a single-layer dynamic graph structure to obtain two sizes The characteristic matrix , using these two feature matrices, we can obtain the similarity matrix and confidence map. The feature matrix A row is a representation The embedding space dimensional feature vector, and the overall loss function of the network is also composed of the loss functions of the two branches, and the specific expression is:

[0063] (1)

[0064] Define the possible degree of association between points. The following association categories may exist:

[0065] 1) Category 1: and Belong to the same instance object and have the same semantic category label;

[0066] 2) Category 2: and Have the same semantic category label, but do not belong to the same instance object;

[0067] 3) Category Three: and They do not belong to the same instance object and have different semantic category labels. As the similarity of the correlation between points increases, that is, as the sequence number defined above increases from small to large, the distance between point pairs in the feature space needs to be gradually increased. For binary classification tasks, the hinge loss is often used as the training basis. For the three correlation categories defined above, a similar idea can be used to define the double hinge loss. The specific loss function is:

[0068]

[0069]

[0070] in, The mark is the point pair The similarity category to which it belongs, and are some predefined hyperparameters that satisfy Although there is no distinction between category 2 and category 3 in the instance segmentation task, i.e. and Point Belong to different instance objects, but use the double hinge loss in Distinguishing different point pair relationships in can provide additional feature information between the network branches of the similarity matrix and the semantic segmentation task, thereby improving the accuracy and accelerating the network convergence. In the semantic segmentation task, the network attempts to bring the distance between the point relationships of category 2 closer in the embedding space. Add an additional penalty factor when it is 2 To perform the product, this parameter needs to satisfy , by definition The parameter increases the weight ratio of the semantic segmentation task branch in the network fitting process, making it more sensitive to the loss function during the gradient descent process.

[0071] The loss function of the confidence map prediction branch is defined as the predicted intersection-over-union ratio and truth value between loss:

[0072]

[0073] The similarity matrix acquisition branch is The feature vector is taken as input and a convolutional network is used to generate a The similarity matrix of The candidate group of the instance where the confidence graph is located is incorrect and redundant. The confidence graph branch of the network is needed to prune the candidate group. The matrix is ​​used as input and a convolutional neural network structure is used to generate a The confidence map uses the intersection over union (IoU) between the similarity matrix and the true value as the basis for regression supervision. During the training process, it needs to be performed after the similarity matrix is ​​calculated. Each element in the confidence map indicates the degree of trustworthiness of the candidate group obtained from the similarity matrix. This network branch is used to fit a discriminator to determine whether the candidate group is trustworthy. The confidence map generated by the network can be used to screen the candidate group extracted from the similarity matrix. The output results of the two branches can be combined to assign a valid instance label to each point in the point cloud. The complete training idea of ​​the instance segmentation network model is summarized as follows: Figure 3 shown.

[0074] In one embodiment of the present disclosure, before performing instance segmentation on point cloud data, the method further includes:

[0075] Normalize the point cloud data.

[0076] In this embodiment, in the process of adding semantic category labels to the point cloud, it is necessary to overcome the transformation independence of the point cloud, that is, in the process of performing specific geometric transformations on the point cloud, the features learned by the network should not affect the subsequent tasks due to the introduction of geometric transformations. Therefore, in the process of network design, it is also necessary to prevent the features learned from the point cloud from being affected by geometric transformations. The point cloud data is normalized to make the data scale-independent. Since the point cloud data itself can be multiplied by the transformation matrix, it can be learned through a smaller network. By introducing an affine transformation matrix, the learned transformation matrix is ​​directly applied to the input point cloud. The transformation network structure diagram is shown in the figure. Figure 4 shown.

[0077] In one embodiment of the present disclosure, generating a second scene graph based on the instance segmentation result includes:

[0078] Input the point set corresponding to each instance into the node prediction network to obtain the node features;

[0079] The union of the point sets corresponding to two adjacent instances is input into the edge prediction network to obtain the relationship feature;

[0080] The node features and relationship features are combined into triplets and input into the graph convolutional network to obtain the second scene graph.

[0081] In this embodiment, given three-dimensional point cloud data And instance segmentation results , the goal of the scene graph generation task is to obtain , which describes the objects contained in the scene Relationships between objects The features of nodes and edges in the scene graph are predicted and generated respectively, and two dynamic graph convolutional networks are used to extract the node features of the scene point cloud. and edge features .

[0082] The orientation information of the point cloud will affect the prediction of spatial position relationships such as up, down, left, and right. In order to prevent the dynamic graph convolutional network from learning redundant scale-related information, it is necessary to normalize the input instance object point cloud and the union point cloud between objects, and avoid adding rotation operations in the data enhancement part. Using the center coordinates of the bounding box contained in the data set, the target point set is translated as a whole, its center coordinates are translated to the origin, and then the point cloud is normalized to a sphere with a radius of 1; for the point set union composed of instance objects, the midpoint of the center of the bounding box of the two objects is used as the new center, and the center is translated to the origin, and then all points in the point cloud are scaled normalized.

[0083] For the scene , we need to use the instance segmentation results To extract For each instance object in the object point cloud ,in represents the Kronecker function, when hour The value is 1, otherwise it is 0. and They respectively represent and Instance object in It represents a collection The module of The number of elements contained in each individual point set As the input of the node prediction network. For each pair of instances and , using their respective bounding boxes The union of the two sets of points constitutes a new point set , It can be defined as ,and This will be used as the input of the edge prediction network. In addition to inputting the point cloud within the bounding box union of the instance object as the input of the network, it is also necessary to add labels to the points in the point set to indicate that the points belong to the instance , Example , or background information, the mask is defined as , when its value is 1, it means that the point belongs to the instance , a value of 2 indicates that the point belongs to an instance , when the value is 0, it indicates that the point belongs to background information.

[0084] The object features and relationship features are organized into a triple form of (subject, predicate, object) as the input of the scene graph optimization graph convolutional network, that is, using the occupies the position of subject and object, while the Occupies the position of the predicate.

[0085] Specifically, for all instance objects , , given the input features , using three multilayer perceptrons (MLPs), abstracted into three functions To calculate the output features , all three multilayer perceptrons take feature vectors in the form of triples as input. To update the predicate relation features, use Function operation, its expression is In contrast, updating the features of instance objects is a more complex process, because each object may participate in multiple relationships. Need to depend on all objects Input features that are related in the graph topology , and associated instance objects and objects The relationship characteristics of For each edge of To calculate the candidate feature vector, the feature collects all Features in a collection; similarly, using To collect all the Characteristics and collections of nodes and edges The features in are:

[0086] (2)

[0087] For objects The output vector pass Calculate, where It is a symmetric function that performs pooling operations on a series of input features. All of them are implemented through a single-layer graph convolution operation. The single-layer graph convolution operation is as follows Figure 5 shown.

[0088] Figure 5 In this network, The concatenated result vector is used as input, and the network design is implemented using MLP. The output features are then pooled. , and then input the pooled results into another MLP network to further increase the depth of the network and improve its generalization ability.

[0089] In one embodiment of the present disclosure, generating a second scene graph based on the instance segmentation result further includes:

[0090] Using a scene graph generation network based on an attention mechanism to update node features and relationship features in the second scene graph to obtain an updated second scene graph;

[0091] For each message passing layer in the second scene graph, the update process of node features and relationship features includes:

[0092] The triple consisting of the node feature and the relationship feature is passed as input to the first multi-layer perceptron to obtain an intermediate feature vector of the node and an updated relationship feature vector;

[0093] Aggregate the intermediate feature vectors of the nodes to obtain the hidden layer features of each node;

[0094] The hidden layer features of each node are input into the second multi-layer perceptron, and residual connections are added to obtain the updated feature vector of each node.

[0095] In this embodiment, the attention mechanism is introduced to optimize the instance label prediction of nodes and edges. For a single-layer graph network structure, the input is a series of node features. ,in is the number of nodes in the topology graph, is the dimension of each node feature. After this layer of network, a series of new node features can be obtained. As output, the feature dimension of the output is not necessarily the same as that of the input. In order for the model to have sufficient generalization ability to learn high-level information features, the network needs at least one learnable linear transformation. As part of the network initialization, a shared linear transformation is applied to each node, using a weight matrix Parameterize, , and then apply the self-attention mechanism on each node, which is defined as a arrive A shared attention module on , specifically:

[0096] (3)

[0097] in, Represents the attention score, i.e., the node Pair Node The importance coefficient of .

[0098] The structural information in the graph is introduced through the mask attention mechanism, and only all nodes are calculated. The connected edges ,here Represents a node The node set consists of the first-order neighbors of . In order to make the attention scores comparable between different nodes, the softmax function is used to normalize all attention scores. The calculation expression of the attention score is as follows:

[0099] (4)

[0100] The attention mechanism is a single-layer feedforward neural network in the network implementation. The attention vector is parameterized and a nonlinear function is applied to activate the output vector. The normalized attention scores are then used to calculate the linear combination of the feature vectors as the output features of each node.

[0101] Introducing the attention mechanism into the scene graph generation network, the feature update of the nodes in the second scene graph (i.e., the graph convolutional network) requires the calculation and introduction of attention scores. For each message passing layer in the graph convolutional network , we need to transform the triple As input, it is passed into the first multi-layer perceptron. The calculation process is as follows:

[0102] (5)

[0103] in is the processed intermediate eigenvector, Indicates the subject, Indicates the predicate, Indicates the object, the upper right corner Indicates The feature vector of the layer, It means the Here, only the relationship features are updated to obtain the updated relationship features. .

[0104] After that, the intermediate feature vectors of the nodes need to be aggregated. At this stage, each node will receive the features of all nodes that are associated with it. In this stage, the attention score is introduced in the summation process to obtain the hidden layer features of each node. , the specific calculation expression is:

[0105] (6)

[0106] in, and The current nodes are As a collection of connecting points between subject and object, It is a node Pair Node The output result is passed to another multi-layer perceptron. At the same time, in order to avoid the possible Laplace smoothing in the graph structure, a residual connection is added to the final output vector to obtain the node Output characteristics .

[0107] The iterative update method of each layer feature in the network is consistent with the above ideas. After the graph convolutional network of the layer, the receptive field of the node will change from the first-order neighbor to its own n-order neighbor. The network uses two multi-layer perceptrons as classifiers for downstream tasks to complete the category prediction of instance objects and the relationship between objects.

[0108] In one embodiment of the present disclosure, the loss function of the scene graph generation network based on the attention mechanism is:

[0109]

[0110]

[0111]

[0112] in, represents the total loss function, represents the object classification loss, represents the predicate classification loss, is the weight coefficient; represents the probability of a positive sample, is the predicted label level, is the label level of the true value; For the The true value of each category in the binary classification task, For the The prediction confidence of each category, c is the total number of predicate label categories; , , , , These are debugging parameters.

[0113] In this embodiment, the training phase uses an attention-based scene graph generation network to perform end-to-end learning, taking 3D scene point cloud data with instance labels as input to regress the 3D semantic scene graph. and predicate classification loss After optimization, the overall loss function of the network can be expressed by summing up, and its specific expression is:

[0114] (7)

[0115] The parameters It is used to balance the weight ratio of object classification and predicate classification. During the experiment, the parameter is adjusted by saving the loss calculation results of the two classification tasks. Object prediction and predicate relationship label prediction can be regarded as two independent classification tasks. According to the observation of actual scene data, it can be found that the classification task can be regarded as a single classification task or a multi-classification task. It is a more direct idea to regard the classification task as a single classification task. Each point cloud object and the relationship between objects has only one semantic label corresponding to it. In order to balance the imbalance between classes in the data set, that is, the situation where a few categories dominate the entire data set, Focal Loss is introduced to alleviate this problem. In this case, the loss function of object classification and predicate relationship classification is specifically:

[0116] (8)

[0117] By introducing the coefficient To balance the importance of positive and negative samples and overcome the problem of uneven proportion of positive and negative samples. It is used to adjust the speed at which the weight of simple samples is reduced. When increases, the adjustment effect on the difficult and easy samples will also increase. For simple samples, it accounts for a smaller proportion in the final loss function, while for difficult samples, it accounts for a larger proportion in the loss function.

[0118] (9)

[0119] in, , and It is the hyperparameter in Focal Loss. Defined as the predicted label level, is the label level of the true value, and The definition of is used to measure the degree to which the proportion of rough prediction results in the loss function is reduced. The lower This means that even if a rough prediction result is not considered an error, its impact on the loss function is small. When the network obtains a more refined prediction result, since the correctness of the prediction result cannot be judged based on the true value given in the dataset, the design of its loss function is consistent with the loss of predicting the correct result.

[0120] Since object labels have a hierarchical relationship, more detailed label classification should not give wrong labels during the prediction process, and less attention should be paid to coarser object label annotation. The optimization direction of the network should be towards finer division rather than coarser label prediction. Therefore, the loss function under the multi-classification task can be designed as follows:

[0121] (10)

[0122] in For the The true value of each category in the binary classification task, For the The prediction confidence of each category, c is the total number of predicate label categories. The complete training idea of ​​the scene graph generation network model based on the attention mechanism is summarized as follows Figure 6 shown.

[0123] Corresponding to the cross-modal retrieval method in the above embodiment, Figure 7 This is a structural block diagram of a cross-modal retrieval device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 7 The cross-modal retrieval device 20 includes: a first extraction module 21, a second extraction module 22, a first screening module 23 and a second screening module 24.

[0124] The first extraction module 21 is used to extract a first scene graph based on the target image; the first scene graph is used to represent multiple objects in the target image and the relationship between the objects;

[0125] A second extraction module 22 is used to extract a second scene graph from the point cloud data; the second scene graph is used to represent multiple objects in the point cloud data and the relationship between the objects;

[0126] A first screening module 23 is used to perform a second scene graph projection based on the position and posture of the target image, and screen a plurality of third scene graphs; the third scene graph is a local two-dimensional scene graph in the second scene graph;

[0127] The second screening module 24 is used to calculate the first similarity between the first scene graph and the third scene graph respectively, screen the fourth scene graph from the third scene graph based on the first similarity calculation result, and determine the fourth scene graph as the retrieval result corresponding to the target image.

[0128] In one embodiment of the present disclosure, the second extraction module 22 is specifically used for:

[0129] Perform instance segmentation on point cloud data;

[0130] A second scene graph is generated based on the instance segmentation result.

[0131] In one embodiment of the present disclosure, the second extraction module 22 is further configured to:

[0132] Extract local feature vectors of point cloud data;

[0133] Calculate the second similarity between each local feature vector to obtain a similarity matrix;

[0134] The point cloud data is grouped based on the similarity matrix to obtain multiple prediction candidate groups;

[0135] Perform convolution operation on each local feature vector to obtain the confidence level corresponding to each prediction candidate group;

[0136] Based on the confidence level corresponding to each prediction candidate group, multiple prediction candidate groups are screened to obtain instance segmentation results.

[0137] In one embodiment of the present disclosure, the second extraction module 22 is further configured to:

[0138] Input the point set corresponding to each instance into the node prediction network to obtain the node features;

[0139] The union of the point sets corresponding to two adjacent instances is input into the edge prediction network to obtain the relationship feature;

[0140] The node features and relationship features are combined into triplets and input into the graph convolutional network to obtain the second scene graph.

[0141] In one embodiment of the present disclosure, the second extraction module 22 is further configured to:

[0142] Using a scene graph generation network based on an attention mechanism to update node features and relationship features in the second scene graph to obtain an updated second scene graph;

[0143] For each message passing layer in the second scene graph, the update process of node features and relationship features includes:

[0144] The triple consisting of the node feature and the relationship feature is passed as input to the first multi-layer perceptron to obtain an intermediate feature vector of the node and an updated relationship feature vector;

[0145] Aggregate the intermediate feature vectors of the nodes to obtain the hidden layer features of each node;

[0146] The hidden layer features of each node are input into the second multi-layer perceptron, and residual connections are added to obtain the updated feature vector of each node.

[0147] In one embodiment of the present disclosure, the second extraction module 22 is further used to calculate the loss function of the scene graph generation network based on the attention mechanism, including:

[0148] The loss function of the scene graph generation network based on the attention mechanism is:

[0149]

[0150]

[0151]

[0152] in, represents the total loss function, represents the object classification loss, represents the predicate classification loss, is the weight coefficient; represents the probability of a positive sample, is the predicted label level, is the label level of the true value; For the The true value of each category in the binary classification task, For the The prediction confidence of each category, c is the total number of predicate label categories; , , , , These are debugging parameters.

[0153] In one embodiment of the present disclosure, the second extraction module 22 is specifically used for:

[0154] Normalize the point cloud data.

[0155] See Figure 8 , Figure 8 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 8 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of each module / unit in the above-mentioned device embodiments, such as Figure 7 The functions of modules 21 to 24 are shown.

[0156] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0157] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0158] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0159] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure may execute the implementation methods described in the first and second embodiments of the cross-modal retrieval method provided in the embodiments of the present disclosure, and may also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0160] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0161] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0162] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0163] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0164] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0165] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0166] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0167] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A cross-modal retrieval method, characterized in that: include: extracting a first scene graph based on the target image; The first scene graph is used to represent multiple objects in the target image and the relationship between the objects; Extracting a second scene graph based on the point cloud data; the second scene graph is used to represent multiple objects in the point cloud data and the relationship between each object; Projecting the second scene graph based on the pose of the target image, and screening a plurality of third scene graphs; the third scene graph is a local two-dimensional scene graph in the second scene graph; A first similarity between the first scene graph and the third scene graph is calculated respectively, a fourth scene graph is selected from the third scene graph based on the first similarity calculation result, and the fourth scene graph is determined as a retrieval result corresponding to the target image.

2. The cross-modal retrieval method according to claim 1, characterized in that: The step of extracting a second scene graph based on the point cloud data comprises: Performing instance segmentation on the point cloud data; The second scene graph is generated based on the instance segmentation result.

3. The cross-modal retrieval method according to claim 2, characterized in that: The performing instance segmentation on the point cloud data comprises: Extracting local feature vectors of the point cloud data; Calculate the second similarity between each local feature vector to obtain a similarity matrix; Grouping the point cloud data based on the similarity matrix to obtain a plurality of prediction candidate groups; Perform convolution operation on each local feature vector to obtain the confidence level corresponding to each prediction candidate group; The multiple prediction candidate groups are screened based on the confidence level corresponding to each prediction candidate group to obtain an instance segmentation result.

4. The cross-modal retrieval method according to claim 2, wherein: The generating the second scene graph based on the instance segmentation result comprises: Input the point set corresponding to each instance into the node prediction network to obtain the node features; The union of the point sets corresponding to two adjacent instances is input into the edge prediction network to obtain the relationship feature; The node features and the relationship features are combined into a triplet form and input into a graph convolutional network to obtain the second scene graph; wherein any triplet includes two node features and corresponding relationship features.

5. The cross-modal retrieval method according to claim 4, characterized in that: The generating the second scene graph based on the instance segmentation result further includes: Using a scene graph generation network based on an attention mechanism to update node features and relationship features in the second scene graph to obtain an updated second scene graph; For each message passing layer in the second scene graph, the update process of node features and relationship features includes: The triple consisting of the node feature and the relationship feature is passed as input to the first multi-layer perceptron to obtain an intermediate feature vector of the node and an updated relationship feature vector; Aggregate the intermediate feature vectors of the nodes to obtain the hidden layer features of each node; The hidden layer features of each node are input into the second multi-layer perceptron, and residual connections are added to obtain the updated feature vector of each node.

6. The cross-modal retrieval method according to claim 5, characterized in that: The loss function of the scene graph generation network based on the attention mechanism is: in, represents the total loss function, represents the object classification loss, represents the predicate classification loss, is the weight coefficient; represents the probability of a positive sample, is the predicted label level, is the label level of the true value; For the The true value of each category in the binary classification task, For the The prediction confidence of each category, c is the total number of predicate label categories; , , , , These are debugging parameters.

7. The cross-modal retrieval method according to claim 2, characterized in that: Before performing instance segmentation on the point cloud data, the method further includes: A normalization operation is performed on the point cloud data.

8. A cross-modal retrieval device, characterized in that: include: A first extraction module, used for extracting a first scene graph from a target image; The first scene graph is used to represent multiple objects in the target image and the relationship between the objects; A second extraction module, used to extract a second scene graph from the point cloud data; The second scene graph is used to represent multiple objects in the point cloud data and the relationship between the objects; A first screening module, configured to perform projection of the second scene graph based on the position and posture of the target image, and screen a plurality of third scene graphs; the third scene graph is a local two-dimensional scene graph in the second scene graph; The second screening module is used to calculate the first similarity between the first scene graph and the third scene graph respectively, screen the fourth scene graph from the third scene graph based on the first similarity calculation result, and determine the fourth scene graph as the retrieval result corresponding to the target image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images