Cross-modal Stereo Vision Object Retrieval Method and Device

Abstract

The present application discloses a cross-modal stereo vision object retrieval method and apparatus. The method includes: extracting depth features of each modality to obtain instances; constructing a dynamic graph structure within the modality domain, and using dynamic graph convolution to encode instance features and intra-instance domain relationships to obtain enhanced intra-instance domain features; constructing a cross-modal dynamic bipartite graph structure, and using dynamic bipartite graph convolution to encode instance features and cross-domain relationships between instances to obtain enhanced cross-domain instance features; performing transformation encoding on the instance features to obtain instance self-transformation features; fusing the above features to generate an instance fusion representation, further generating a class prediction score, and using the gradient descent algorithm to optimize the weights, and then calculating the similarity score using the cosine distance between the instance fusion representations to obtain relevant cross-modal retrieval data of the instance object. Thereby, the problems in the related art such as the inability to directly retrieve between modalities and the limitation of cross-modal retrieval accuracy and speed are solved.

Description

Technical Field

[0001] This application relates to the technical field of stereo vision object retrieval, and particularly to a cross-modal stereo vision object retrieval method and device. Background Art

[0002] Stereo vision is an important field in computer vision. There are a wide variety of data modalities in the field of stereo vision, including point clouds, voxels, views, meshes, etc. The diversity of data modalities is determined by the diversity of usage scenarios and sensors. Many directions in the field of stereo vision involve stereo data of multiple data modalities. For example, 3D printing, stereo modeling, robots, etc. often require cross-modal retrieval of stereo object data.

[0003] However, in related technologies, the conversion cost between different modalities is relatively expensive, usually accompanied by irrecoverable information loss, and in practical applications, some modal data of stereo objects may be missing, and it cannot be converted into a single-modal stereo object retrieval, which urgently needs to be improved.

[0004] Content of the Application

[0005] This application provides a cross-modal stereo vision object retrieval method and device to solve problems such as the inability to directly retrieve between modalities, limited cross-modal retrieval accuracy and speed in related technologies.

[0006] The first aspect embodiment of this application provides a cross-modal stereo vision object retrieval method, including the following steps: extracting depth features of each modality to obtain at least one instance; constructing a dynamic graph structure within the modal domain based on the at least one instance, and using dynamic graph convolution to encode instance features and in-instance domain relationships to obtain enhanced in-instance domain features; constructing a cross-modal dynamic bipartite graph structure based on the at least one instance, and using dynamic bipartite graph convolution to encode the instance features and instance cross-domain relationships to obtain enhanced instance cross-domain features; performing transformation encoding on the instance features of the at least one instance to obtain instance self-transformation features; fusing the enhanced in-instance domain features, enhanced instance cross-domain features, and instance self-transformation features to generate an instance fusion representation; generating category prediction scores according to the instance fusion representation, and using the gradient descent algorithm to optimize the weights; and calculating similarity scores based on the cosine distance between the instance fusion representations of the at least one instance to obtain relevant cross-modal retrieval data of the instance object.

[0007] Optionally, in an embodiment of this application, the extracting depth features of each modality to obtain at least one instance includes: using a preset depth representation model to extract depth features of each modality sample, where the depth representation model is trained based on at least one classification task by one or more of point cloud stereo data, mesh stereo data, and view stereo data.

[0008] Optionally, in an embodiment of the present application, constructing a dynamic graph structure within the modal domain based on the at least one instance, and using dynamic graph convolution to encode instance features and relationships within the instance domain to obtain enhanced features within the instance domain, includes: for each instance, calculating the first cosine distance between features, and using the nearest neighbor algorithm to determine the neighbors of each instance one by one, establishing the dynamic graph structure within the modal domain; based on the depth features of the instance and the intra-domain connections of the dynamic graph structure within the modal domain, using the dynamic graph convolution to encode and generate the enhanced features within the instance domain.

[0009] Optionally, in an embodiment of the present application, constructing a cross-modal dynamic bipartite graph structure based on the at least one instance, and using dynamic bipartite graph convolution to encode the instance features and instance cross-domain relationships to obtain enhanced instance cross-domain features, includes: for each instance, calculating the second cosine distance between the features, using the nearest neighbor algorithm to obtain the intra-domain neighbors of each instance, establishing cross-domain connections between the instance and the instances of other modalities corresponding to its intra-domain neighbors, constructing the dynamic bipartite graph structure; based on the depth features of the instance and the cross-domain connections of the dynamic bipartite graph structure, using the dynamic bipartite graph convolution to encode and generate the enhanced instance cross-domain features.

[0010] Optionally, in an embodiment of the present application, generating a class prediction score according to the instance fusion representation and optimizing the weights using the gradient descent algorithm, includes: processing the instance fusion representation using at least one fully connected layer to generate the class score of the instance; according to the class score and the labeled class, using the gradient descent method to optimize the learnable weights.

[0011] An embodiment of the second aspect of the present application provides a cross-modal stereo vision object retrieval device, including: a feature extraction module, configured to extract depth features of each modality to obtain at least one instance; an intra-domain feature enhancement module, configured to construct a dynamic graph structure within the modal domain based on the at least one instance, and use dynamic graph convolution to encode instance features and relationships within the instance domain to obtain enhanced features within the instance domain; a cross-domain feature enhancement module, configured to construct a cross-modal dynamic bipartite graph structure based on the at least one instance, and use dynamic bipartite graph convolution to encode the instance features and instance cross-domain relationships to obtain enhanced instance cross-domain features; a feature transformation module, configured to perform transformation encoding on the instance features of the at least one instance to obtain instance self-transformation features; a feature fusion module, configured to fuse the enhanced features within the instance domain, the enhanced instance cross-domain features, and the instance self-transformation features to generate an instance fusion representation; a weight optimization module, configured to generate a class prediction score according to the instance fusion representation and optimize the weights using the gradient descent algorithm; and a retrieval module, configured to calculate a similarity score based on the cosine distance between the instance fusion representations of the at least one instance based on the optimized weights to obtain relevant cross-modal retrieval data of the instance object.

[0012] Optionally, in an embodiment of the present application, the feature extraction module is further configured to: extract depth features of each modality example by using a preset depth representation model, where the depth representation model is trained based on at least one classification task by one or more of point cloud stereo data, mesh stereo data, and view stereo data.

[0013] Optionally, in an embodiment of the present application, the intra-domain feature enhancement module includes: a first calculation unit configured to calculate a first cosine distance between features for each instance, and use the nearest neighbor algorithm to determine neighbors of each instance one by one, and establish an intra-modal domain dynamic graph structure; a first generation unit configured to generate an intra-instance enhanced feature by using dynamic graph convolution encoding based on the depth feature of the instance and the intra-domain connection of the intra-modal domain dynamic graph structure.

[0014] Optionally, in an embodiment of the present application, the cross-domain feature enhancement module includes: a second calculation unit configured to calculate a second cosine distance between the features for each instance, use the nearest neighbor algorithm to obtain intra-domain neighbors of each instance, establish cross-domain connections between the instance and instances of other modalities corresponding to its intra-domain neighbors, and construct the dynamic bipartite graph structure; a second generation unit configured to generate an instance cross-domain enhanced feature by using dynamic bipartite graph convolution encoding based on the depth feature of the instance and the cross-domain connection of the dynamic bipartite graph structure.

[0015] Optionally, in an embodiment of the present application, the weight optimization module includes: a fusion unit configured to process the instance fusion representation by using at least one fully connected layer to generate a class score of the instance; an optimization unit configured to optimize the learnable weight by using the gradient descent method according to the class score and the labeled class.

[0016] An embodiment of the third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the cross-modal stereo vision object retrieval method as described in the above embodiments.

[0017] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores computer instructions for causing the computer to execute the cross-modal stereo vision object retrieval method as described in the above embodiments.

[0018] Based on the construction of a dynamic graph and a dynamic bipartite graph and their feature fusion, the embodiments of the present application achieve the purpose of retrieving data from a retrieval data of one modality to a data domain of a specified modality, improve the accuracy of the retrieval representation using the complex relationships between data, and effectively ensure the accuracy and reliability of the retrieval. Thus, the problems in the related art such as the inability to directly retrieve between modalities and the limited accuracy and speed of cross-modal retrieval are solved.

[0019] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, in which:

[0021] Figure 1 is a flowchart of a cross-modal stereo vision object retrieval method according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of a cross-modal stereo vision object retrieval method according to a specific embodiment of the present application;

[0023] Figure 3 is a schematic structural diagram of a cross-modal stereo vision object retrieval device according to an embodiment of the present application;

[0024] Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the drawings, in which the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application and should not be construed as limiting the present application.

[0026] The cross-modal stereo vision object retrieval method and device according to the embodiments of the present application will be described below with reference to the drawings. In view of the problems in the related art mentioned in the above background art that it is impossible to directly retrieve between modalities and the accuracy and speed of cross-modal retrieval are limited, the present application provides a cross-modal stereo vision object retrieval method. In this method, based on the construction of a dynamic graph and a dynamic bipartite graph and their feature fusion, the purpose of retrieving data from a retrieval data of one modality to a data domain of a specified modality is achieved, and the accuracy of the retrieval representation is improved using the complex relationships between data, effectively ensuring the accuracy and reliability of the retrieval. Thus, the problems in the related art such as the inability to directly retrieve between modalities and the limited accuracy and speed of cross-modal retrieval are solved.

[0027] Specifically, Figure 1 FIG. is a schematic flowchart of a cross-modal stereo vision object retrieval method provided by an embodiment of the present application.

[0028] As Figure 1 shown, the cross-modal stereo vision object retrieval method includes the following steps:

[0029] In step S101, depth features of each modality are extracted to obtain at least one instance.

[0030] In the actual execution process, the embodiment of the present application can extract the depth features of each modality, thereby obtaining at least one instance, and providing a basis for subsequent in-domain feature enhancement, cross-domain feature enhancement, etc., which is beneficial to achieving high-precision cross-modal stereo vision object retrieval.

[0031] Optionally, in an embodiment of the present application, extracting the depth features of each modality to obtain at least one instance includes: using a preset depth representation model to extract the depth features of each modality example, where the depth representation model is trained based on at least one classification task by one or more of point cloud stereo data, mesh stereo data, and view stereo data.

[0032] Specifically, the embodiment of the present application can use a preset depth representation model to extract the depth features of each modality example. Among them, the depth features include but are not limited to stereo data, mesh stereo data, and view stereo data. The embodiment of the present application can train a depth representation model for the specific structure of single-modal stereo data based on a classification task, and use the high-level features of this layer as the depth feature representation of an example for an instance based on the fully connected layer at the end of the model.

[0033] For example, the embodiment of the present application can use the DGCNN model algorithm to learn point cloud stereo data, use the MeshNet model algorithm to learn mesh stereo data, use the MVCNN model algorithm to learn multi-view stereo data, and then use the high-level features of this layer as the depth feature representation of an example for an instance based on the fully connected layer at the end of the model.

[0034] In step S102, an in-modal domain dynamic graph structure is constructed based on at least one instance, and instance features and in-modal domain relationships are encoded using dynamic graph convolution to obtain in-modal domain enhanced features.

[0035] As a possible implementation manner, the embodiment of the present application can construct an in-modal domain dynamic graph structure based on at least one instance, and obtain in-modal domain enhanced features by encoding instance features and in-modal domain relationships using dynamic graph convolution. By obtaining the in-modal domain enhanced features, the embodiment of the present application is beneficial to improving the accuracy of cross-modal stereo vision object retrieval.

[0036] Optionally, in an embodiment of the present application, a dynamic graph structure within the modal domain is constructed based on at least one instance, and the instance features and the in-domain relationships of the instances are encoded using dynamic graph convolution to obtain in-domain enhanced features, including: for each instance, calculating the first cosine distance between features, and using the nearest neighbor algorithm to determine the neighbors of each instance one by one to establish a dynamic graph structure within the modal domain; based on the depth features of the instances and the in-domain connections of the dynamic graph structure within the modal domain, using dynamic graph convolution encoding to generate in-domain enhanced features of the instances.

[0037] In some specific embodiments, for the instances of each modality, the cosine distance between the instance depth features can be calculated, and the nearest neighbor algorithm is used to determine the neighbors of the instances one by one, thereby establishing a dynamic graph structure, and based on the depth features of the instances and the in-domain connections, the proposed dynamic graph convolution encoding is used to generate in-domain enhanced features of the instances.

[0038] Specifically, the calculation formula of dynamic graph convolution can be as follows:

[0039] ,

[0040] where, 、 represent the instance feature with node modality and serial number j , and the initial instance feature is the instance obtained in step S101, represents a non-linear activation function, k is a parameter of the nearest neighbor algorithm, 、 represent the set of neighbor serial numbers of the node with serial number i constructed by the nearest neighbor algorithm, 、 are learnable matrix parameters, 、 respectively represent the obtained in-domain enhanced features of the instances.

[0041] In step S103, a cross-modal dynamic bipartite graph structure is constructed based on at least one instance, and the instance features and the instance cross-domain relationships are encoded using dynamic bipartite graph convolution to obtain instance cross-domain enhanced features.

[0042] In the actual execution process, the embodiment of the present application can construct a cross-modal dynamic bipartite graph structure and use dynamic bipartite graph convolution to encode the instance features and the instance cross-domain relationships, and then obtain instance cross-domain enhanced features. By obtaining the instance cross-domain enhanced features, the embodiment of the present application is beneficial to improving the accuracy of cross-modal 3D object retrieval.

[0043] Optionally, in an embodiment of the present application, a cross-modal dynamic bipartite graph structure is constructed based on at least one instance, and the instance features and instance cross-domain relationships are encoded using dynamic bipartite graph convolution to obtain instance cross-domain enhanced features, including: for each instance, calculating the second cosine distance between features, using the nearest neighbor algorithm to obtain the in-domain neighbors of each instance, establishing cross-domain connections between the instance and the instances of other modalities corresponding to its in-domain neighbors, and constructing a dynamic bipartite graph structure; based on the depth features of the instances and the cross-domain connections of the dynamic bipartite graph structure, using dynamic bipartite graph convolution encoding to generate instance cross-domain enhanced features.

[0044] It can be understood that the dynamic bipartite graph describes the instance connections between two domains. For the instances of each modality, the embodiments of the present application can calculate the cosine distance between the instance depth features, use the nearest neighbor algorithm to obtain the in-domain neighbors of the instances, and establish cross-domain connections between the instance and the instances of other modalities corresponding to its in-domain neighbors to construct a dynamic bipartite graph. At the same time, based on the depth features of the instances and the cross-domain connections, the proposed dynamic bipartite graph convolution encoding is used to generate the cross-domain enhanced features of the instances.

[0045] Among them, the dynamic bipartite graph convolution formula is as follows:

[0046] ,

[0047] Among them, , represents the instance feature with the node modality of and the serial number of j , which is obtained in step S101. represents a non-linear activation function. b is the parameter of the nearest neighbor algorithm. , represents the set of neighbor serial numbers of the node with the serial number of i constructed by the nearest neighbor algorithm. , are learnable matrix parameters. , respectively represent the obtained instance cross-domain enhanced features.

[0048] In step S104, the instance features of at least one instance are transformed and encoded to obtain instance self-transformed features.

[0049] For example, the embodiments of the present application can multiply the instance features with a learnable matrix and perform non-linear activation function operations to obtain the self-transformed features of the instances.

[0050] Among them, the calculation formula of the instance self-transformed features is as follows:

[0051] ,

[0052] Among them, and represent that the node mode is The serial number of i is the instance feature obtained from step S101. represents a non-linear activation function. and are learnable matrix parameters. and respectively represent the obtained instance self-transformation features.

[0053] In step S105, the instance domain-enhanced feature, the instance cross-domain enhanced feature, and the instance self-transformation feature are fused to generate an instance fusion representation.

[0054] Specifically, the embodiments of the present application can fuse the in-domain enhanced feature, the cross-domain enhanced feature, and the self-transformation feature of the instance obtained in the above steps, and use hyperparameters to perform weighted summation to obtain the fusion representation of the instance.

[0055] Among them, the calculation formula for generating the instance fusion representation can be as follows:

[0056] ,

[0057] Among them, and and are hyperparameters set artificially. and is the fusion representation of the instance.

[0058] In step S106, a class prediction score is generated based on the instance fusion representation, and the weights are optimized using the gradient descent algorithm.

[0059] As a possible implementation, the embodiments of the present application can generate a class prediction score based on the fusion representation obtained in the above steps, and optimize the weights using the gradient descent algorithm. The embodiments of the present application achieve the purpose of retrieving data from a specified modality from the retrieval data of one modality based on the construction of a dynamic graph and a dynamic bipartite graph and their feature fusion, and improve the accuracy of the retrieval representation using the complex relationships between data, effectively ensuring the accuracy and reliability of the retrieval.

[0060] Optionally, in an embodiment of the present application, generating a class prediction score based on the instance fusion representation and optimizing the weights using the gradient descent algorithm includes: processing the instance fusion representation using at least one fully connected layer to generate a class score of the instance; optimizing the learnable weights using the gradient descent method according to the class score and the labeled class.

[0061] Specifically, the embodiments of the present application can use at least one layer of fully connected layers to process the fused representation of instances, thereby generating the class scores of the instances, and optimize the learnable weights in the process using the gradient descent method based on the predicted class scores of the data and the labeled classes. The embodiments of the present application achieve the purpose of retrieving data from one modality to the data domain of a specified modality based on constructing a dynamic graph, a dynamic bipartite graph, and their feature fusion, and improve the accuracy of the retrieval representation using the complex relationships between data, effectively ensuring the accuracy and reliability of the retrieval.

[0062] In step S107, based on the optimized weights, calculate the similarity scores using the cosine distance between the fused representations of at least one instance to obtain the relevant cross-modal retrieval data of the instance object.

[0063] It can be understood that the fused representation of an instance refers to the instance features obtained in step S105. The embodiments of the present application can calculate all the fused representations of the instances, and obtain the relevant cross-modal retrieval data of the strength object by calculating the distances between the instance representations to be retrieved and those within the retrieval range. The embodiments of the present application achieve the purpose of retrieving data from one modality to the data domain of a specified modality based on constructing a dynamic graph, a dynamic bipartite graph, and their feature fusion, and improve the accuracy of the retrieval representation using the complex relationships between data, effectively ensuring the accuracy and reliability of the retrieval.

[0064] Next, in conjunction with Figure 2 as shown, the working principle of the stereo vision object retrieval method of the embodiments of the present application will be elaborated in detail with a specific embodiment.

[0065] As Figure 2 shown, the embodiments of the present application may include the following steps:

[0066] Step S201: Use a feature extractor to extract the depth features of each modality example. Specifically, the embodiments of the present application can use a preset depth representation model to extract the depth features of each modality example. Among them, the depth features include but are not limited to stereo data, mesh stereo data, and view stereo data. The embodiments of the present application can train a depth representation model with a specific structure for single-modal stereo data based on a classification task, and use the high-level features of this layer as the depth feature representation of an example based on the fully connected layer at the end of the model.

[0067] For example, the embodiments of the present application can use the DGCNN model algorithm to learn point cloud stereo data, the MeshNet model algorithm to learn mesh stereo data, and the MVCNN model algorithm to learn multi-view stereo data, and then use the high-level features of this layer as the depth feature representation of an example based on the fully connected layer at the end of the model.

[0068] Step S202: Construct a dynamic graph structure within the modality domain, use dynamic graph convolution to encode instance features and relationships within the instance domain, and obtain enhanced features within the instance domain. In some specific embodiments, for instances of each modality, the cosine distance between instance depth features can be calculated, and the nearest neighbor algorithm can be used to determine the neighbors of each instance one by one, thereby establishing a dynamic graph structure. Based on the instance depth features and intra-domain connections, the proposed dynamic graph convolution is used to encode and generate enhanced features within the instance domain for the instance.

[0069] Specifically, the calculation formula of dynamic graph convolution is as follows:

[0070] ,

[0071] where, 、 represent the instance features of the node with modality and serial number j . The initial instance features are the instances obtained in step S201. represents a non-linear activation function. k is the parameter of the nearest neighbor algorithm. 、 represent the set of neighbor serial numbers of the node with serial number i constructed by the nearest neighbor algorithm. 、 are learnable matrix parameters. 、 respectively represent the obtained enhanced features within the instance domain.

[0072] Step S203: Construct a cross-modal dynamic bipartite graph structure, use dynamic bipartite graph convolution to encode instance features and instance cross-domain relationships, and obtain enhanced cross-domain features for the instance. It can be understood that the dynamic bipartite graph describes the instance connections between two domains. For instances of each modality, in the embodiments of the present application, the cosine distance between instance depth features can be calculated, the nearest neighbor algorithm can be used to obtain the intra-domain neighbors of the instance, and cross-domain connections between the instance and the instances of other modalities corresponding to its intra-domain neighbors can be established to construct a dynamic bipartite graph. At the same time, based on the instance depth features and cross-domain connections, the proposed dynamic bipartite graph convolution is used to encode and generate enhanced cross-domain features for the instance.

[0073] Among them, the formula of dynamic bipartite graph convolution is as follows:

[0074] ,

[0075] where, 、 represent the instance features of the node with modality and serial number j . This instance feature is obtained in step S201. Represents a non - linear activation function, b Is a parameter of the nearest - neighbor algorithm, 、 Represents the set of neighbor sequence numbers of the node with sequence number i constructed by the nearest - neighbor algorithm, 、 Are learnable matrix parameters, 、 Respectively represent the obtained instance cross - domain enhanced features.

[0076] Step S204: Transform and encode the features of the instance itself to obtain the instance self - transformation features. For example, in the embodiments of the present application, a learnable matrix can be multiplied by the instance features and then operated through a non - linear activation function to obtain the self - transformation features of the instance.

[0077] Among them, the calculation formula of the instance self - transformation features is as follows:

[0078] ,

[0079] Among them, 、 Represent the instance features of the node with modality and sequence number i , which are obtained in step S201, Represents a non - linear activation function, 、 Are learnable matrix parameters, 、 Respectively represent the obtained instance self - transformation features.

[0080] Step S205: Fuse the in - domain enhanced features, cross - domain enhanced features, and self - transformation features of the instance to generate a fused representation of the instance. Specifically, in the embodiments of the present application, the in - domain enhanced features, cross - domain enhanced features, and self - transformation features of the instance obtained in the above steps can be fused, and weighted addition is used with hyperparameters to obtain the fused representation of the instance.

[0081] Among them, the calculation formula for generating the fused representation of the instance is as follows:

[0082] ,

[0083] Among them, 、 、 Are hyperparameters set as needed, 、 Is the fused representation of the instance.

[0084] Step S206: Generate class prediction scores based on the fused representation and optimize the weights using the gradient descent algorithm. Specifically, in the embodiments of the present application, at least one fully connected layer can be used to process the fused representation of the instance, so as to generate the class scores of the instance, and according to the class scores predicted from the data and the labeled classes, the gradient descent method is used to optimize the learnable weights in the process.

[0085] Step S207: Calculate the similarity scores using the cosine distance between the instance fused representations to obtain the relevant cross-modal retrieval data of the instance object. It can be understood that the instance fused representation refers to the instance features obtained in step S205. In the embodiments of the present application, all instance fused representations can be calculated, and by calculating the distances between the representations of the instances to be retrieved and within the retrieval range, the relevant cross-modal retrieval data of the strength object can be obtained. The embodiments of the present application achieve the purpose of retrieving data from the retrieval data of one modality to the data domain of the specified modality based on constructing a dynamic graph and a dynamic bipartite graph and their feature fusion, and improve the accuracy of the retrieval representation using the complex relationships between the data, effectively ensuring the accuracy and reliability of the retrieval.

[0086] According to the cross-modal stereo vision object retrieval method proposed in the embodiments of the present application, the purpose of retrieving data from the retrieval data of one modality to the data domain of the specified modality is achieved based on constructing a dynamic graph and a dynamic bipartite graph and their feature fusion, and the accuracy of the retrieval representation is improved using the complex relationships between the data, effectively ensuring the accuracy and reliability of the retrieval. Thereby, the problems in the related art such as the inability to directly retrieve between modalities and the limited cross-modal retrieval accuracy and speed are solved.

[0087] Next, a cross-modal stereo vision object retrieval device according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0088] Figure 3 It is a block diagram of a cross-modal stereo vision object retrieval device according to an embodiment of the present application.

[0089] As Figure 3 shown, the cross-modal stereo vision object retrieval device 10 includes: a feature extraction module 100, an intra-domain feature enhancement module 200, a cross-domain feature enhancement module 300, a feature transformation module 400, a feature fusion module 500, a weight optimization module 600, and a retrieval module 700.

[0090] Specifically, the feature extraction module 100 is configured to extract the depth features of each modality to obtain at least one instance.

[0091] The intra-domain feature enhancement module 200 is configured to construct an intra-modal domain dynamic graph structure based on at least one instance, and use dynamic graph convolution to encode the instance features and the intra-instance domain relationships to obtain the intra-instance domain enhanced features.

[0092] The cross-domain feature enhancement module 300 is used to construct a cross-modal dynamic bipartite graph structure based on at least one instance, and use the dynamic bipartite graph convolution to encode instance features and instance cross-domain relationships to obtain instance cross-domain enhanced features.

[0093] The feature transformation module 400 is configured to perform transformation encoding on the instance features of at least one instance to obtain instance self-transformation features.

[0094] The feature fusion module 500 is used to fuse instance intra-domain enhancement features, instance cross-domain enhancement features, and instance self-transformation features to generate instance fusion representation.

[0095] The weight optimization module 600 is used to generate a category prediction score based on the instance fusion representation and optimize the weight using a gradient descent algorithm.

[0096] The retrieval module 700 is configured to calculate a similarity score based on the optimized weights using the cosine distance between instance fusion representations of at least one instance to obtain relevant cross-modal retrieval data of the instance object.

[0097] Optionally, in one embodiment of the present application, the feature extraction module 100 is further used to extract depth features of each modal sample using a preset depth representation model, wherein the depth representation model is obtained by training one or more of point cloud stereo data, mesh stereo data, and view stereo data based on at least one classification task.

[0098] Optionally, in one embodiment of the present application, the intra-domain feature enhancement module 200 includes: a first calculation unit and a first generation unit.

[0099] The first calculation unit is used to calculate the first cosine distance between features for each instance, and use the nearest neighbor algorithm to determine the neighbors of each instance one by one to establish a dynamic graph structure in the modal domain.

[0100] The first generation unit is used to generate instance domain enhanced features using dynamic graph convolutional coding based on the deep features of the instance and the intra-domain connection of the dynamic graph structure in the modal domain.

[0101] Optionally, in one embodiment of the present application, the cross-domain feature enhancement module 300 includes: a second computing unit and a second generating unit.

[0102] Among them, the second computing unit is used to calculate the second cosine distance between features for each instance, use the nearest neighbor algorithm to obtain the intra-domain neighbors of each instance, establish cross-domain connections between the instance and instances of other modalities corresponding to its intra-domain neighbors, and construct a dynamic bipartite graph structure.

[0103] A second generation unit, configured to generate instance cross-domain enhanced features by using dynamic bipartite graph convolution encoding based on instance depth features and cross-domain connections of a dynamic bipartite graph structure.

[0104] Optionally, in an embodiment of the present application, the weight optimization module 600 includes: a fusion unit and an optimization unit.

[0105] The fusion unit is configured to process the instance fusion representation by using at least one layer of fully connected layers to generate class scores of the instances.

[0106] The optimization unit is configured to optimize learnable weights by using a gradient descent method according to the class scores and the labeled classes.

[0107] It should be noted that the foregoing explanation of the embodiments of the cross-modal stereo vision object retrieval method also applies to the cross-modal stereo vision object retrieval device of this embodiment, and will not be elaborated here.

[0108] The cross-modal stereo vision object retrieval device provided by the embodiments of the present application realizes the purpose of retrieving data from the retrieval data of one modality to the data domain of a specified modality based on constructing a dynamic graph and a dynamic bipartite graph and their feature fusion, improves the accuracy of the retrieval representation by using the complex relationships between data, and effectively ensures the accuracy and reliability of the retrieval. Thus, the problems in the related art that cannot directly retrieve between modalities and the cross-modal retrieval accuracy and speed are limited are solved.

[0109] Figure 4 The structural schematic diagram of the electronic device provided by the embodiments of the present application. The electronic device may include:

[0110] A memory 401, a processor 402, and a computer program stored on the memory 401 and executable on the processor 402.

[0111] When the processor 402 executes the program, it implements the cross-modal stereo vision object retrieval method provided in the foregoing embodiments.

[0112] Further, the electronic device further includes:

[0113] A communication interface 403, configured for communication between the memory 401 and the processor 402.

[0114] The memory 401 is configured to store a computer program executable on the processor 402.

[0115] The memory 401 may include a high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

[0116] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 only a thick line is used in Figure 4 , but it does not mean that there is only one bus or one type of bus.

[0117] Optionally, in a specific implementation, if the memory 401, the processor 402, and the communication interface 403 are integrated on a single chip, the memory 401, the processor 402, and the communication interface 403 can communicate with each other via an internal interface.

[0118] The processor 402 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0119] This embodiment also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above cross-modal stereo vision object retrieval method is implemented.

[0120] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0121] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0122] Any process or method description represented in a flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code that includes one or more N executable instructions for implementing a customized logical function or process. The scope of the preferred embodiments of the present application includes additional implementations, where the functions may be performed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0123] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing a logical function, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with such instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or N wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing as necessary, and then stored in a computer memory.

[0124] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one of the following techniques well known in the art or a combination thereof can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0125] Those of ordinary skill in the art can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0126] In addition, each functional unit in various embodiments of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0127] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A cross-modal stereo vision object retrieval method, characterized in that Including the following steps: Extract the depth features of each modality to obtain at least one instance; Construct a dynamic graph structure within the modality domain based on the at least one instance, and use dynamic graph convolution to encode the instance features and the in-domain relationships of the instances to obtain enhanced in-domain features of the instances; Construct a cross-modal dynamic bipartite graph structure based on the at least one instance, and use dynamic bipartite graph convolution to encode the instance features and the cross-domain relationships of the instances to obtain enhanced cross-domain features of the instances; Perform transformation encoding on the instance features of the at least one instance to obtain self-transformed features of the instances; Fuse the enhanced in-domain features of the instances, the enhanced cross-domain features of the instances, and the self-transformed features of the instances to generate a fused representation of the instances; Generate class prediction scores according to the fused representation of the instances, and optimize the weights using the gradient descent algorithm; And Based on the optimized weights, calculate the similarity scores using the cosine distance between the fused representations of the at least one instance to obtain relevant cross-modal retrieval data of the instance objects; Among them, the constructing a dynamic graph structure within the modality domain based on the at least one instance, and using dynamic graph convolution to encode the instance features and the in-domain relationships of the instances to obtain enhanced in-domain features of the instances includes: for each instance, calculate the first cosine distance between the features, and use the nearest neighbor algorithm to determine the neighbors of each instance one by one, and establish the dynamic graph structure within the modality domain; based on the depth features of the instances and the in-domain connections of the dynamic graph structure within the modality domain, use the dynamic graph convolution to encode and generate the enhanced in-domain features of the instances; Among them, the constructing a cross-modal dynamic bipartite graph structure based on the at least one instance, and using dynamic bipartite graph convolution to encode the instance features and the cross-domain relationships of the instances to obtain enhanced cross-domain features of the instances includes: for each instance, calculate the second cosine distance between the features, use the nearest neighbor algorithm to obtain the in-domain neighbors of each instance, establish cross-domain connections between the instance and the instances of other modalities corresponding to its in-domain neighbors, and construct the dynamic bipartite graph structure; based on the depth features of the instances and the cross-domain connections of the dynamic bipartite graph structure, use the dynamic bipartite graph convolution to encode and generate the enhanced cross-domain features of the instances.

2. The method according to claim 1, wherein The extracting the depth features of each modality to obtain at least one instance includes: Use a preset depth representation model to extract the depth features of each modality example, where the depth representation model is trained based on at least one classification task by one or more of point cloud stereo data, mesh stereo data, and view stereo data.

3. The method according to claim 1 or 2, characterized in that, The generating class prediction scores according to the fused representation of the instances, and optimizing the weights using the gradient descent algorithm includes: Use at least one fully connected layer to process the fused representation of the instances to generate class scores of the instances; According to the class scores and the labeled classes, optimize the weights using the gradient descent method.

4. A cross-modal stereoscopic vision object retrieval device, characterized in that, Including: A feature extraction module for extracting the depth features of each modality to obtain at least one instance; An in-domain feature enhancement module for constructing a dynamic graph structure within the modality domain based on the at least one instance, and using dynamic graph convolution to encode the instance features and the in-domain relationships of the instances to obtain enhanced in-domain features of the instances; A cross-domain feature enhancement module, which is used to construct a cross-modal dynamic bipartite graph structure based on the at least one instance, and use dynamic bipartite graph convolution to encode the instance features and instance cross-domain relationships to obtain instance cross-domain enhanced features; A feature transformation module, which is used to perform transformation encoding on the instance features of the at least one instance to obtain instance self-transformation features; A feature fusion module, which is used to fuse the instance in-domain enhanced features, instance cross-domain enhanced features, and instance self-transformation features to generate an instance fusion representation; A weight optimization module, which is used to generate class prediction scores according to the instance fusion representation and optimize the weights using the gradient descent algorithm; and A retrieval module, which is used to calculate similarity scores based on the optimized weights and use the cosine distance between the instance fusion representations of the at least one instance to obtain relevant cross-modal retrieval data of the instance object; wherein, the in-domain feature enhancement module includes: a first calculation unit, which is used to calculate the first cosine distance between features for each instance, and use the nearest neighbor algorithm to determine the neighbors of each instance one by one to establish the in-modal-domain dynamic graph structure; a first generation unit, which is used to generate the instance in-domain enhanced features by encoding using the dynamic graph convolution based on the deep features of the instance and the in-domain connections of the in-modal-domain dynamic graph structure; wherein, the cross-domain feature enhancement module includes: a second calculation unit, which is used to calculate the second cosine distance between the features for each instance, use the nearest neighbor algorithm to obtain the in-domain neighbors of each instance, establish cross-domain connections between the instance and the instances of other modalities corresponding to its in-domain neighbors, and construct the dynamic bipartite graph structure; a second generation unit, which is used to generate the instance cross-domain enhanced features by encoding using the dynamic bipartite graph convolution based on the deep features of the instance and the cross-domain connections of the dynamic bipartite graph structure.

5. An electronic device, characterized in that, including: A memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the cross-modal stereo vision object retrieval method according to any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to be used to implement the cross-modal stereo vision object retrieval method according to any one of claims 1-3.

Images