A method and system for identifying and retrieving 3D objects

Abstract

The present application relates to the technical field of object recognition, and proposes a 3D object recognition and retrieval method and system, comprising the following steps: based on the intrinsic features of known 3D objects, the mutual information between the adjacent view intrinsic features extracted from unknown 3D objects is maximized, the mutual information between the intrinsic features after mutual information maximization and the view features is minimized, the mutual information between each view feature after mutual information processing is minimized, and the intrinsic features after mutual information processing are fused by PoE; based on the fused intrinsic features of known 3D objects, the mutual information between the unknown fused intrinsic features and the intrinsic features after mutual information processing is maximized, the fused intrinsic features after mutual information processing and the view features after mutual information minimization are fused by PoE to obtain global features, the fused intrinsic features after mutual information processing are iteratively updated, and the updated fused intrinsic features are used for classification and recognition or retrieval of unknown 3D objects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of object recognition, and more particularly to a 3D object recognition retrieval method and system. BACKGROUND

[0002] A three-dimensional (3D) object multi-view is an image obtained by mapping different perspectives, each perspective corresponding to a view containing an intrinsic feature and a perspective feature, and the intrinsic feature of each perspective corresponding view is different, and the intrinsic features of adjacent perspectives have more common features, while the intrinsic features of perspectives far apart have fewer common features. The current main research direction of three-dimensional object recognition technology is to increase the connection between the perspectives of the 3D object through various methods, so that the training model learns more useful information.

[0003] The prior art proposes a three-dimensional object recognition method, including the following steps: obtaining multiple images of an unknown object, and performing feature extraction processing on the multiple images through an artificial intelligence model to obtain multiple two-dimensional view features; combining a plurality of preset registration view features according to the artificial intelligence model to obtain a combined view feature and a classification result corresponding to the combined view feature; performing classification decision processing on the multiple two-dimensional view features and the combined view feature according to the artificial intelligence model to obtain a positive and negative example score result; when the positive and negative example score result is a positive example score, determining the category of the unknown object based on the classification result. This method can perform three-dimensional object recognition on multiple images through multiple artificial intelligence models, thereby improving the accuracy and universality of object recognition, but this method only considers the intrinsic features of the three-dimensional object multi-view and does not consider the perspective features of the three-dimensional object multi-view. Although it is generally believed that the perspective feature does not need to be considered, the perspective feature will also affect the recognition result of the three-dimensional object, and this method does not exclude the influence of the perspective feature, which greatly affects the accuracy of three-dimensional object recognition. SUMMARY

[0004] To overcome the above-mentioned defects of the prior art that do not exclude the influence of the perspective feature and greatly affect the accuracy of three-dimensional object recognition, the present application provides a 3D object recognition retrieval method and system that comprehensively considers the intrinsic features and perspective features of three-dimensional objects and can accurately recognize.

[0005] To solve the above technical problems, the technical solution of the present application is as follows:

[0006] A 3D object recognition retrieval method, characterized by comprising the following steps:

[0007] S1: obtaining all views of an unknown 3D object, and extracting the intrinsic feature and perspective feature corresponding to each view;

[0008] S2: obtain intrinsic features of a known 3D object, and maximize mutual information between adjacent view intrinsic features extracted from the unknown 3D object based on the intrinsic features of the known 3D object, to obtain intrinsic features after mutual information maximization;

[0009] S3: minimize mutual information between the intrinsic features after mutual information maximization and view features, to obtain intrinsic features after mutual information processing and view features after mutual information processing;

[0010] S4: minimize mutual information between each view feature after mutual information processing, to obtain view features after mutual information minimization;

[0011] S5: PoE fusion is performed on the intrinsic features after mutual information processing, to obtain unknown fused intrinsic features;

[0012] S6: obtain fused intrinsic features of a known 3D object, and maximize mutual information between the unknown fused intrinsic features and the intrinsic features after mutual information processing based on the fused intrinsic features of the known 3D object, to obtain intrinsic features after mutual information processing of fusion;

[0013] S7: PoE fusion is performed on the intrinsic features after mutual information processing of fusion and the view features after mutual information minimization, to obtain global features, and the intrinsic features after mutual information processing of fusion are iteratively updated based on the global features, to obtain updated fused intrinsic features;

[0014] S8: the unknown 3D object is classified and recognized or retrieved by using the updated fused intrinsic features.

[0015] The application further provides a 3D object recognition and retrieval system for implementing the 3D object recognition and retrieval method, comprising:

[0016] a feature extraction module, which obtains all views of an unknown 3D object, and extracts intrinsic features and view features corresponding to each view;

[0017] a mutual information maximization module, which is used to obtain intrinsic features of a known 3D object, and maximize mutual information between adjacent view intrinsic features extracted from the unknown 3D object based on the intrinsic features of the known 3D object, to obtain intrinsic features after mutual information maximization;

[0018] a mutual information processing module, which is used to minimize mutual information between the intrinsic features after mutual information maximization and view features, to obtain intrinsic features after mutual information processing and view features after mutual information processing;

[0019] a mutual information minimization module, which is used to minimize mutual information between each view feature after mutual information processing, to obtain view features after mutual information minimization;

[0020] a feature fusion module, configured to perform PoE fusion on the mutual information processed intrinsic features to obtain unknown fused intrinsic features;

[0021] a fused feature mutual information processing module, configured to obtain fused intrinsic features of a known 3D object, and maximize mutual information between the unknown fused intrinsic features and the mutual information processed intrinsic features based on the fused intrinsic features of the known 3D object, to obtain mutual information processed fused intrinsic features;

[0022] a fused feature updating module, configured to perform PoE fusion on the mutual information processed fused intrinsic features and the mutual information minimized view angle features to obtain global features, and iteratively update the mutual information processed fused intrinsic features based on the global features to obtain updated fused intrinsic features;

[0023] a recognition module, configured to perform classification recognition or retrieval on the unknown 3D object by using the updated fused intrinsic features.

[0024] Compared with the prior art, the technical scheme of the present application has the beneficial effects that:

[0025] By maximizing mutual information between intrinsic features extracted from adjacent view angles of the unknown 3D object, minimizing mutual information between the intrinsic features after mutual information maximization and view angle features, and minimizing mutual information between each mutual information processed view angle feature, the present application enhances the consistency of the intrinsic features while reducing the influence of the view angle features on the recognition result of the three-dimensional object; by performing PoE fusion on the mutual information processed intrinsic features to obtain unknown fused intrinsic features, the present application obtains a more comprehensive representation of the unknown 3D object, and iteratively updates the mutual information processed fused intrinsic features based on the global features, further excluding the influence of the view angle features on the recognition result of the three-dimensional object, so as to achieve the purpose of accurately recognizing the unknown 3D object. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 a flowchart of the 3D object recognition and retrieval method of embodiment 1;

[0027] Figure 2 a schematic diagram of classification recognition or retrieval by using the updated fused intrinsic features of embodiment 1;

[0028] Figure 3 a whole framework diagram of the 3D object recognition and retrieval system of embodiment 2. DETAILED DESCRIPTION

[0029] The accompanying drawings are only used for illustrative purposes, and cannot be understood as a limitation on the present patent;

[0030] For better illustrating the embodiments, some components in the drawings can be omitted, enlarged or reduced, which do not represent the actual product size.

[0031] It is understandable for those skilled in the art that some known structures and their descriptions in the drawings can be omitted.

[0032] The technical solutions of the present application will be further described below in combination with the drawings and embodiments.

[0033] Embodiment 1

[0034] The present embodiment proposes a 3D object recognition retrieval method, Figure 1 The flowchart of the 3D object recognition retrieval method of the present embodiment is shown in the figure.

[0035] The 3D object recognition retrieval method proposed in the present embodiment includes the following steps:

[0036] S1: Obtain all views of an unknown 3D object, and extract the intrinsic features and view angle features corresponding to each view;

[0037] S2: Obtain the intrinsic features of a known 3D object, and maximize the mutual information between the adjacent view angle intrinsic features extracted from the unknown 3D object based on the intrinsic features of the known 3D object, to obtain the intrinsic features after mutual information maximization;

[0038] S3: Minimize the mutual information between the intrinsic features after mutual information maximization and the view angle features, to obtain the intrinsic features after mutual information processing and the view angle features after mutual information processing;

[0039] S4: Minimize the mutual information between each view angle feature after mutual information processing, to obtain the view angle features after mutual information minimization;

[0040] S5: PoE fusion of the intrinsic features after mutual information processing, to obtain the unknown fusion intrinsic features;

[0041] S6: Obtain the fusion intrinsic features of the known 3D object, and maximize the mutual information between the unknown fusion intrinsic features and the intrinsic features after mutual information processing based on the fusion intrinsic features of the known 3D object, to obtain the fusion intrinsic features after mutual information processing;

[0042] S7: PoE fusion of the fusion intrinsic features after mutual information processing and the view angle features after mutual information minimization, to obtain the global features, and iterative update the fusion intrinsic features after mutual information processing based on the global features, to obtain the updated fusion intrinsic features;

[0043] S8: Use the updated fusion intrinsic features to classify and identify or retrieve the unknown 3D object.

[0044] As an example, in the embodiment, a variational autoencoder encoder (Enc) is provided for each view of the unknown 3D object, and an intrinsic feature and a view feature are extracted in each encoder; the process of obtaining the intrinsic feature of the known 3D object is to obtain the view of the known 3D object first, and then to extract the intrinsic feature and the view feature corresponding to each view by using the encoder of the variational autoencoder;

[0045] In the specific implementation, by maximizing the mutual information between the intrinsic features extracted from adjacent views of the unknown 3D object, minimizing the mutual information between the intrinsic features and the view features after the mutual information is maximized, and minimizing the mutual information between the view features after each mutual information processing, the consistency of the intrinsic features is enhanced, and the influence of the view features on the recognition result of the three-dimensional object is reduced; by fusing the intrinsic features after the mutual information processing by PoE (Product of Experts), the fused intrinsic features of the unknown are obtained, so that a more comprehensive representation of the unknown 3D object is obtained, and the fused intrinsic features after the mutual information processing are iteratively updated based on the global feature, so as to further exclude the influence of the view features on the recognition result of the three-dimensional object, thereby achieving the purpose of accurately recognizing the unknown 3D object.

[0046] In an optional embodiment, the process of obtaining the intrinsic feature after the mutual information is maximized includes:

[0047] Based on the intrinsic feature of the known 3D object, an intrinsic feature mutual information maximization model for maximizing the mutual information between the extracted intrinsic features of adjacent views is constructed;

[0048] wherein the expression of the intrinsic feature mutual information maximization model is:

[0049]

[0050] C=[c1,c2,...,c n ]

[0051] C1=[c 11 ,c 12 ,...,c 1n ]

[0052] In the formula, denotes a joint distribution function, sp(·) denotes a softplus function, denotes an edge distribution function, c i denotes the feature code corresponding to the i th intrinsic feature c i , c j denotes the feature code corresponding to the j th intrinsic feature c j , and the intrinsic feature ci The intrinsic feature c j is adjacent to the view angle;

[0053] c′ j represents the feature code corresponding to the intrinsic feature of the known 3D object in the same view angle as c j ; c n represents the nth intrinsic feature of the intrinsic feature set C;

[0054] c 1n represents the intrinsic feature c n corresponding to the mutual information maximization, and also represents the nth intrinsic feature of the intrinsic feature set C1;

[0055] The intrinsic feature mutual information maximization model is solved iteratively, and when the objective function A q reaches maximization, the iteration is ended, and the intrinsic feature after mutual information maximization is obtained.

[0056] As an exemplary illustration, the intrinsic feature mutual information maximization model is constructed based on the MINE algorithm (Mutual Information Neural Estimator, a neural network method based on KL divergence dual representation), and the variational autoencoder is used to solve the intrinsic feature mutual information maximization model, so as to obtain the intrinsic feature after mutual information maximization;

[0057] In this optional embodiment, the mutual information between the intrinsic features extracted from adjacent view angles is maximized, which can reduce the influence of view angle change on the intrinsic feature, enhance the consistency of the intrinsic feature, and further improve the accuracy of identifying unknown 3D objects.

[0058] In an optional embodiment, the step of obtaining the intrinsic feature after mutual information processing and the view angle feature after mutual information processing comprises:

[0059] An intrinsic information processing model for minimizing the mutual information between the intrinsic feature after mutual information maximization and the view angle feature is constructed;

[0060] The expression of the mutual information processing model is:

[0061]

[0062] E=[e1,e2,...,e n ]

[0063] E1=[e 11 ,e 12 ,...,e 1n ]

[0064] C2=[c 21 ,c 22 ,...,c2n ]

[0065] where ||·|| denotes vector norm, denotes the encoder of c 1i , denotes the encoder of the i-th view feature e i , denotes the i-th view feature of c 1i , i denotes the i-th view feature of e i , n denotes the n-th view feature of e

[0066] denotes the i-th intrinsic feature of c 2n , 1n denotes the n-th intrinsic feature of the intrinsic feature set C2

[0067] denotes the i-th view feature of e 1n , n denotes the n-th view feature of the view feature set E1

[0068] iteratively solving the mutual information processing model, and ending the iteration when the objective function B q is minimized, to obtain the mutual information processed intrinsic feature and the mutual information processed view feature.

[0069] As an example, the mutual information processing model is solved by using a variational autoencoder, and then the mutual information processed intrinsic feature and the mutual information processed view feature are obtained.

[0070] In this optional embodiment, the independence of the intrinsic feature and the view feature is improved by minimizing the linear correlation, so as to minimize the mutual information of the intrinsic feature and the view feature; the mutual information between the intrinsic feature and the view feature after maximizing the mutual information is minimized, so that the obtained mutual information processed intrinsic feature has a more essential representation of the unknown object, which can reduce the negative influence of the view change on the 3D object recognition, and further improve the accuracy of recognizing the unknown 3D object.

[0071] In an optional embodiment, the step of obtaining the view feature after minimizing the mutual information comprises:

[0072] constructing a view feature mutual information minimization model for minimizing the mutual information between each mutual information processed view feature;

[0073] wherein the expression of the view feature mutual information minimization model is:

[0074]

[0075] E2 = [e 21 , e 22 ,..., e 2n ]

[0076] wherein, denotes an encoder processing e 1i , denotes an encoder processing the i-th view feature e 1j , denotes the corresponding feature encoding of e 1i , denotes the corresponding view feature encoding of e 1j ;

[0077] e 2n denotes the view feature after mutual information minimization corresponding to e 1n ;

[0078] The view feature mutual information minimization model is solved iteratively, and when the objective function C q reaches minimization, the iteration is ended, and the view feature after mutual information minimization is obtained.

[0079] As an exemplary illustration, the view feature mutual information minimization model is solved by using a variational autoencoder, and then the view feature after mutual information minimization is obtained.

[0080] In this optional embodiment, the mutual information between each view feature after mutual information processing is minimized, which can further eliminate the negative impact of the view feature on 3D object recognition, and further improve the accuracy of recognizing unknown 3D objects.

[0081] In an optional embodiment, the intrinsic feature after mutual information processing is fused by PoE to obtain the expression of the unknown fused intrinsic feature F1 as follows:

[0082] F1 = z c

[0083]

[0084]

[0085] X = [x1, x2,..., x n ]

[0086] x n = [c 2n , e 2n ]

[0087] wherein, X denotes the view corresponding to the intrinsic feature after mutual information processing and the view feature after mutual information processing, x n denotes the intrinsic feature c2n and view feature e 2n , which also represents the nth view in X; z c represents a latent variable, q(z c |X) represents the conditional probability density function of observing z c given X, c p(z c ) represents the probability density function of the prior probability distribution of z represents the corresponding latent variable of c 2i , represents the conditional probability density function of observing z i given x , i represents the corresponding latent variable of x i , q(z i |x i ) represents the conditional probability density function of observing z i given x i , represents the corresponding latent variable of e 2n , represents the conditional probability density function of observing z i and and given x 2i .

[0088] In the optional embodiment, the intrinsic features processed by mutual information are fused by PoE, so that the obtained unknown fused intrinsic features can represent the intrinsic features of all views, reducing the dimension of the features and reducing the calculation cost.

[0089] In an optional embodiment, the step of obtaining the fused intrinsic features processed by mutual information comprises:

[0090] Based on the known fused intrinsic features of the 3D object, a fused intrinsic feature mutual information maximization model is constructed to maximize the mutual information between the unknown fused intrinsic features and the intrinsic features processed by mutual information;

[0091] wherein the expression of the fused intrinsic feature mutual information maximization model is:

[0092]

[0093] In the formula, and respectively represent the feature codes corresponding to c 2i and c 2j , F1' represents the known fused intrinsic features of the 3D object;

[0094] The fused intrinsic feature mutual information maximization model is iteratively solved, and the target function D qWhen the maximization is reached, the iteration is ended, and the mutual information processed fused intrinsic feature F is obtained 11 , wherein the mutual information processed fused intrinsic feature F 11 equals F1 at the end of the iteration.

[0095] As an example, the mutual information processed fused intrinsic feature is obtained by solving the fused intrinsic feature mutual information maximization model using a variational autoencoder;

[0096] In this optional embodiment, the mutual information between the unknown fused intrinsic feature and the mutual information processed intrinsic feature is maximized, so that the obtained mutual information processed fused intrinsic feature contains as much intrinsic feature of all views as possible, which can more comprehensively identify the 3D object, and further improve the accuracy of identifying the 3D object.

[0097] In an optional embodiment, the step of obtaining the updated fused intrinsic feature comprises:

[0098] S7.1: PoE fusion is performed on the mutual information processed fused intrinsic feature and the view angle feature minimized in mutual information to obtain a global feature r;

[0099] wherein the calculation expression of the global feature r is:

[0100] q(r|X) = q(F 11 |X)q(E2|X)

[0101]

[0102]

[0103] In the formula, q(r|X) represents the conditional probability density function of observing the global feature r given X, q(F 11 |X) represents the conditional probability density function of observing F 11 given X, and q(E2|X) represents the conditional probability density function of observing E2 given X; μ r and respectively represent the mean and variance of q(r|X), μ c and respectively represent the mean and variance of q(F 11 |X), μ e and respectively represent the mean and variance of q(E2|X).

[0104] S7.2: Based on the global feature r, a fused intrinsic feature updating model is constructed.

[0105] wherein the expression of the fused intrinsic feature updating model is:

[0106]

[0107] wherein, represents a processing C 2i encoder, represents a processing i-th view feature e 2j encoder, represents an ELBO function, represents a reconstruction loss function of global feature r, represents a KL divergence function of F 11 , d(x|r) represents a decoder processing global feature r; represents a KL divergence function of E2; d(x|r) represents a decoder processing global feature r;

[0108] The fusion intrinsic feature update model is solved iteratively, and the target function is minimized, the iteration is ended, and the updated fusion intrinsic feature F 12 is obtained, wherein the updated fusion intrinsic feature F 12 is equal to F 11 at the end of iteration.

[0109] As an exemplary illustration, the encoder is a variational autoencoder encoder; the decoder is a variational autoencoder decoder (Dec); the ELBO (Evidence Lower Bound) function is a variational autoencoder ELBO function; and the decoder can reconstruct the unknown 3D object view using the global feature.

[0110] In this optional embodiment, the mutual information processed fusion intrinsic feature is iteratively updated based on the global feature, to obtain an updated fusion intrinsic feature, so that the updated fusion intrinsic feature is closer to the essence of the unknown 3D object, thereby more accurately representing the unknown 3D object, and further achieving the purpose of accurately identifying the 3D object.

[0111] In an optional embodiment, Figure 2 is a schematic diagram for classification identification or retrieval using the updated fusion intrinsic feature, before the unknown 3D object is classified and identified or retrieved using the updated fusion intrinsic feature, steps S1 to S6 are performed to obtain the updated fusion intrinsic feature M 12 of another 3D object M of the same category as the unknown 3D object.

[0112] The updated fusion intrinsic feature F 12 and the updated fusion intrinsic feature M 12 are spliced and fused together to obtain the updated fusion intrinsic feature FM;

[0113] The updated fusion intrinsic feature FM is used for classification identification or retrieval of the unknown 3D object.

[0114] In the optional embodiment, the updated fusion intrinsic feature F 12 and the updated fusion intrinsic feature M 12 are spliced together, which can further learn the intrinsic feature information of the object related to the unknown 3D object category, so as to more comprehensively and accurately identify the 3D object.

[0115] In an optional embodiment, when the updated fusion intrinsic feature is used for classification identification of the unknown 3D object, a full connection layer and a normalization layer are set after the updated fusion intrinsic feature, and the classification identification result of the unknown 3D object is calculated by using the full connection layer and the normalization layer.

[0116] When the updated fusion intrinsic feature is used for retrieval of the unknown 3D object, the Euclidean distance between the updated fusion intrinsic feature and the updated fusion intrinsic feature of all known 3D objects is calculated, and the known 3D object with the smallest Euclidean distance calculation result is taken as the retrieval result of the unknown 3D object.

[0117] Embodiment 2

[0118] The 3D object identification and retrieval method proposed in Embodiment 1 is applied in this embodiment, and the following performance comparison examples are proposed.

[0119] In this embodiment, three data sets of ModelNet40, ModelNet10 and ShapeNet55 are used.

[0120] Among them, ModelNet40 is composed of 12311 3D objects of 40 categories, including 9843 training objects and 2468 test objects for object shape classification, and different categories have different numbers of objects. ModelNet10 is a subset of ModelNet40. ShapeNet55 data set contains 51162 3D objects, which are divided into 55 categories and 204 subcategories. Among the 51162 3D objects, the training set, the verification set and the test set are 70% (35764), 10% (5133) and 20% (10265) respectively.

[0121] (1) On the ShapeNet55 dataset, the method of the embodiment (Ours) is compared with view-based methods, wherein the compared algorithms include: RotationNet algorithm, View-GCN algorithm and CAR algorithm; the comparison results include precision (P@N), recall (R@N), F1 value (F1@N), average precision (mAP) and normalized discounted cumulative gain (NDCG@N) on micro average (Micro) and macro average (Macro); the macro average is used to give the unweighted average of the entire dataset, and the score is averaged with the same weight; the micro average treats queries and retrieval results equally between classes, so the result is averaged without adjusting the weight according to the class size; that is, the macro average is to first average each class and then average; the micro average is to directly average the total.

[0122] The comparison results are shown in the following table:

[0123] Table 1

[0124]

[0125]

[0126] As can be seen from Table 1, on the ShapeNet55 dataset, the difference between the method of the embodiment and the view-based method is less than 0.05, which is sufficient to show that the three-dimensional recognition accuracy of the method under ideal conditions is still at an advanced level, and the method can still complete the three-dimensional recognition task with priority to accuracy.

[0127] (2) On the ModelNet40 and ModelNet10 datasets, the method of the embodiment (Ours) is compared with view-based methods, wherein the compared algorithms include: MHBN algorithm, CAR-Net algorithm and DAN algorithm; the comparison results include average instance accuracy and retrieval average accuracy (Retrieval mAP), and average classification accuracy (average class accuracy).

[0128] The comparison results are shown in Tables 2 and 3:

[0129] Table 2 Comparison results on the modelnet40, 12 view dataset

[0130]

[0131] Table 3 Comparison results on the modelnet10, 12 view dataset

[0132]

[0133] As can be seen from Table 2 and Table 3, the classification accuracy and the retrieval average accuracy of the method of the embodiment are both high on the ModelNet40 and ModelNet10 data sets, which is sufficient to show that the three-dimensional recognition precision of the model is still at an advanced level under ideal conditions, and the model can still complete the three-dimensional recognition task with priority to precision.

[0134] Embodiment 3

[0135] The embodiment proposes a 3D object recognition and retrieval system for implementing the 3D object recognition and retrieval method proposed in Embodiment 1.

[0136] As shown in Figure 3 , it is a whole framework diagram of the 3D object recognition and retrieval system of the embodiment.

[0137] The 3D object recognition and retrieval system comprises:

[0138] The feature extraction module acquires all views of the unknown 3D object, and extracts the intrinsic feature and the view angle feature corresponding to each view;

[0139] The mutual information maximization module is used to acquire the intrinsic feature of the known 3D object, and based on the intrinsic feature of the known 3D object, the mutual information between the adjacent view angle intrinsic features extracted from the unknown 3D object is maximized, and the intrinsic feature after mutual information maximization is obtained;

[0140] The mutual information processing module is used to minimize the mutual information between the intrinsic feature after mutual information maximization and the view angle feature, and obtain the intrinsic feature after mutual information processing and the view angle feature after mutual information processing;

[0141] The mutual information minimization module is used to minimize the mutual information between each view angle feature after mutual information processing, and obtain the view angle feature after mutual information minimization;

[0142] The feature fusion module is used to perform PoE fusion on the intrinsic feature after mutual information processing, and obtain the unknown fusion intrinsic feature;

[0143] The fusion feature mutual information processing module is used to acquire the fusion intrinsic feature of the known 3D object, and based on the fusion intrinsic feature of the known 3D object, the mutual information between the unknown fusion intrinsic feature and the intrinsic feature after mutual information processing is maximized, and the fusion intrinsic feature after mutual information processing is obtained;

[0144] The fusion feature updating module is configured to perform PoE fusion on the mutual information processed fusion intrinsic feature and the mutual information minimized view feature to obtain a global feature, and iteratively update the mutual information processed fusion intrinsic feature based on the global feature to obtain an updated fusion intrinsic feature;

[0145] The recognition module is configured to perform classification recognition or retrieval on the unknown 3D object by using the updated fusion intrinsic feature.

[0146] It can be understood that the system of the embodiment is applied to the method of the above embodiment 1, and the optional items in the above embodiment 1 are also applicable to the embodiment, and thus are not repeatedly described herein.

[0147] The same or similar reference numerals correspond to the same or similar components;

[0148] The terms describing the positional relationship in the drawings are only used for exemplary illustration, and should not be understood as a limitation on the patent;

[0149] Obviously, the above embodiments of the present application are merely exemplary for clearly illustrating the present application, and are not intended to limit the implementation modes of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, it is not necessary and impossible to exhaust all the implementation modes. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the claims of the present application.

Claims

1. A method of identifying a 3D object, characterized by, The method comprises the following steps: S1: acquiring all views of an unknown 3D object, and extracting intrinsic features and view angle features corresponding to each view; S2: acquiring intrinsic features of a known 3D object, and maximizing mutual information between adjacent view angle intrinsic features extracted from the unknown 3D object based on the intrinsic features of the known 3D object, to obtain intrinsic features after mutual information maximization; S3: minimizing mutual information between the intrinsic features after mutual information maximization and the view angle features, to obtain intrinsic features after mutual information processing and view angle features after mutual information processing; S4: minimizing mutual information between each view angle feature after mutual information processing, to obtain view angle features after mutual information minimization; S5: performing PoE fusion on the intrinsic features after mutual information processing, to obtain fused intrinsic features of the unknown; S6: acquiring fused intrinsic features of the known 3D object, and maximizing mutual information between the fused intrinsic features of the unknown and the intrinsic features after mutual information processing based on the fused intrinsic features of the known 3D object, to obtain fused intrinsic features after mutual information processing; S7: performing PoE fusion on the fused intrinsic features after mutual information processing and the view angle features after mutual information minimization, to obtain global features, and iteratively updating the fused intrinsic features after mutual information processing based on the global features, to obtain updated fused intrinsic features; S8: using the updated fused intrinsic features to classify and identify or search the unknown 3D object.

2. The identification search method of a 3D object according to claim 1, characterized in that, The process of obtaining the intrinsic features after mutual information maximization in step S2 comprises: constructing an intrinsic feature mutual information maximization model for maximizing mutual information between adjacent view angle intrinsic features extracted based on the intrinsic features of the known 3D object; wherein the expression of the intrinsic feature mutual information maximization model is: wherein, denotes a joint distribution function, denotes a softplus function, denotes an edge distribution function, denotes the th eigenfeature corresponding feature encoding, denotes the th eigenfeature corresponding feature encoding, and the eigenfeature is adjacent in perspective to the eigenfeature . a feature encoding corresponding to an intrinsic feature of a same perspective of a known 3D object; a feature encoding corresponding to an intrinsic feature of a same perspective of a known 3D object; a feature encoding corresponding to an intrinsic feature of a same perspective of a known 3D object; a feature encoding corresponding to an intrinsic feature of a same perspective of a known 3D object; a feature encoding corresponding to an intrinsic feature of a same perspective of a known 3D object; representing corresponding to the mutual information maximization, also represents the intrinsic feature set the first intrinsic feature The intrinsic feature mutual information maximization model is solved iteratively, and the objective function is maximized, the iteration is ended, and the intrinsic feature after the mutual information maximization is obtained.

3. The identification search method of a 3D object according to claim 2, characterized in that, The steps of obtaining the intrinsic features after mutual information processing and the view angle features after mutual information processing comprise: constructing a mutual information processing model for minimizing mutual information between the intrinsic features after mutual information maximization and the view angle features; wherein the expression of the mutual information processing model is: In the formula, denotes vector modulo, denotes processing of the encoder, denotes processing the first view feature of the encoder, denotes the corresponding intrinsic feature encoding, denotes the corresponding view feature encoding; denotes the first view feature; representing corresponding mutual information processed intrinsic features, also denoted as intrinsic feature set the first intrinsic feature of the intrinsic feature set representing corresponding mutual information processed view features, also denoted as a view feature set a first view feature of the view feature set The mutual information processing model is solved iteratively, and the iteration is ended when the objective function reaches a minimum, obtaining the intrinsic features after mutual information processing and the perspective features after mutual information processing.

4. The identification search method of a 3D object according to claim 3, characterized in that, The step of obtaining the view angle features after mutual information minimization comprises: constructing a view angle feature mutual information minimization model for minimizing mutual information between each view angle feature after mutual information processing; wherein the expression of the view angle feature mutual information minimization model is: In the formula, Indicate processing The encoder, Indicates processing the first Features from a single perspective The encoder, express The corresponding feature encoding, express Corresponding viewpoint feature encoding; representations of corresponding mutual information minimized view features; The iteration is solved for the perspective feature mutual information minimization model, and the target function is minimized, the iteration is ended, and the perspective feature after mutual information minimization is obtained.

5. The identification search method of a 3D object according to claim 4, characterized in that, The mutual information processed intrinsic features are fused by PoE to obtain unknown fused intrinsic features The expression is: wherein denotes the view corresponding to the intrinsic feature and the mutual information processed view angle feature, denotes the view containing the intrinsic feature and the view angle feature , also denotes the th view in ; denotes the latent variable, denotes the conditional probability density function of observing given ; denotes the probability density function of the prior distribution of ; denotes the latent variable corresponding to ; denotes the conditional probability density function of observing given ; denotes the latent variable corresponding to ; denotes the conditional probability density function of observing given ; denotes the latent variable corresponding to ; denotes the conditional probability density function of observing given and .

6. The identification search method of a 3D object according to claim 5, wherein, The step of obtaining the fused intrinsic features after mutual information processing comprises: constructing a fused intrinsic feature mutual information maximization model for maximizing mutual information between the fused intrinsic features of the unknown and the intrinsic features after mutual information processing based on the fused intrinsic features of the known 3D object; wherein the expression of the fused intrinsic feature mutual information maximization model is: wherein, and respectively represent and corresponding feature encodings, represent fused intrinsic features of known 3D objects; iteratively solving the fusion intrinsic feature mutual information maximization model, and ending iteration when the objective function reaches maximization to obtain the fusion intrinsic feature processed by mutual information , wherein the fusion intrinsic feature processed by mutual information equals the fusion intrinsic feature processed by mutual information at the time of ending iteration . .

7. The identification search method of a 3D object according to claim 6, characterized in that, The step of obtaining the updated fused intrinsic features comprises: S7.1: PoE fusion is performed on the fusion intrinsic features processed by mutual information and the view angle features minimized by mutual information to obtain global features ; wherein the global feature is calculated by the expression: where denotes the conditional probability density function of the observation when the global feature is known, denotes the conditional probability density function of the observation when the global feature is known, denotes the conditional probability density function of the observation when the global feature is known; and denote the mean and variance of respectively, and denote the mean and variance of respectively, and denote the mean and variance of respectively. S7.2: based on global features , construct a fusion intrinsic feature update model; wherein the expression of the fused intrinsic feature updating model is: In the formula, Indicate processing The encoder, Indicates processing the first Features from a single perspective The encoder, express function, Representing global features The reconstruction loss function express The KL divergence function, express The KL divergence function; Indicates processing global features The decoder; iteratively solving the fusion eigensignature updating model, and ending iteration when the objective function reaches minimization, to obtain updated fusion eigensignature wherein the updated fusion eigensignature equals the objective function at the end of iteration.

8. The method of claim 7, wherein, Before using the updated fused intrinsic feature to classify and recognize or retrieve the unknown 3D object, steps S1 to S6 are performed to obtain another 3D object of the same category as the unknown 3D object the updated fused intrinsic feature of the other 3D object ; the updated fused intrinsic feature and the updated fused intrinsic feature stitching together to obtain the updated fused intrinsic feature ; In the classification recognition or retrieval of unknown 3D objects, the updated fusion intrinsic features are utilized for classification recognition or retrieval.

9. The identification and search method of the 3D object according to any one of claims 1-8, characterized in that: when the unknown 3D object is classified and identified by using the updated fused intrinsic features, a full connection layer and a normalization layer are arranged after the updated fused intrinsic features, and the classification and identification result of the unknown 3D object is calculated by using the full connection layer and the normalization layer. When retrieving the unknown 3D object by using the updated fused intrinsic feature, the Euclidean distance between the updated fused intrinsic feature and the updated fused intrinsic feature of all known 3D objects is calculated, and the known 3D object with the smallest Euclidean distance is taken as the retrieval result of the unknown 3D object.

10. A system for identifying and retrieving a 3D object, for implementing the method for identifying and retrieving a 3D object according to any one of claims 1 to 8, characterized in that, Comprise: The feature extraction module acquires all views of the unknown 3D object, and extracts the intrinsic feature and the view angle feature corresponding to each view; The mutual information maximization module is used to acquire the intrinsic feature of the known 3D object, and based on the intrinsic feature of the known 3D object, the mutual information between the adjacent view angle intrinsic features extracted from the unknown 3D object is maximized to obtain the intrinsic feature after mutual information maximization; The mutual information processing module is used to minimize the mutual information between the intrinsic feature after mutual information maximization and the view angle feature, to obtain the intrinsic feature after mutual information processing and the view angle feature after mutual information processing; The mutual information minimization module is used to minimize the mutual information between each view angle feature after mutual information processing, to obtain the view angle feature after mutual information minimization; The feature fusion module is used to PoE fuse the intrinsic feature after mutual information processing to obtain the unknown fused intrinsic feature; The fused feature mutual information processing module is used to acquire the fused intrinsic feature of the known 3D object, and based on the fused intrinsic feature of the known 3D object, the mutual information between the unknown fused intrinsic feature and the intrinsic feature after mutual information processing is maximized to obtain the intrinsic feature after mutual information processing of the fused feature; The fused feature update module is used to PoE fuse the intrinsic feature after mutual information processing of the fused feature and the view angle feature after mutual information minimization to obtain the global feature, and based on the global feature, iteratively update the intrinsic feature after mutual information processing of the fused feature to obtain the updated fused intrinsic feature; The recognition module is used to classify and identify or retrieve the unknown 3D object by using the updated fused intrinsic feature.

Images