A three-dimensional model retrieval method and system based on high-order adjacency relation constraint

Abstract

The application provides a three-dimensional model retrieval method and system based on high-order adjacency relation constraint, and belongs to the technical field of three-dimensional model retrieval, and comprises the following steps: obtaining a three-dimensional model to be queried, rendering an illumination view and a depth view of the three-dimensional model to be queried, and splicing the illumination view and the depth view into multi-channel information; based on the multi-channel information, an initial feature is extracted by using a feature extraction model to obtain initial feature descriptors of each channel; the initial feature descriptors between channels are used for initial grouping of multi-views based on cosine similarity; a high-order adjacency relation matrix is used for representing the structural correlation degree of each view, the initial grouping is optimized based on the high-order adjacency relation matrix to obtain a final grouping result, and final feature descriptors are obtained based on the final grouping result; and similarity between the final feature descriptors and model features in a feature database is measured to obtain a three-dimensional model retrieval result. The application improves the stability and accuracy of three-dimensional model retrieval.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional model retrieval, and particularly relates to a three-dimensional model retrieval method and system based on high-order adjacency relationship constraint. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Three-dimensional model retrieval technology has broad application prospects in virtual reality, industrial manufacturing, autonomous driving and other fields. The core task of three-dimensional model retrieval is to quickly and accurately retrieve similar models in a three-dimensional model library for a user input model to be retrieved.

[0004] According to different representation forms of three-dimensional models, existing three-dimensional model retrieval methods are mainly divided into four categories: point cloud-based, voxel-based, mesh-based and view-based methods. The point cloud-based method usually uses deep learning to analyze point cloud data to extract its geometric structure features. This method has high flexibility and can adapt to irregular point cloud distribution. However, due to the sparsity and disorder of point cloud data, feature extraction is a great challenge; the voxel-based method uses a regular voxel grid to represent a three-dimensional model and uses a three-dimensional convolutional neural network to extract features. Since this method can preserve the spatial structure information of the model more completely, it has high retrieval accuracy. However, as the voxel resolution increases, the spatial and temporal complexity of this method will also increase significantly, limiting the scale and efficiency of its practical application; the mesh-based method extracts model features by analyzing the surface mesh data of the model. This method can better capture the surface continuity and topological structure of the model, but it is highly dependent on mesh quality; the view-based method projects a three-dimensional model from multiple perspectives to generate a series of two-dimensional images, extracts multi-view image features using existing image processing networks, and then implements model retrieval. With the development of deep learning and image processing technology, this method performs well in retrieval performance and has good engineering feasibility. However, the existing methods often fail to fully express the structure and semantic information of the three-dimensional model in the view generation process, and the potential relationship between views is not fully explored, which restricts the model retrieval accuracy.

[0005] The existing three-dimensional model rendering view generation method is prone to cause the loss of model geometric structure and spatial feature information, and it is difficult to fully reflect the shape semantic feature of the three-dimensional model. In addition, the existing multi-view grouping feature fusion method usually adopts a direct grouping method, which fails to effectively explore the potential high-order structure relationship between views, and the traditional pooling operation will cause the loss of key features, resulting in limited model feature expression ability and affecting the retrieval performance. SUMMARY

[0006] In order to overcome the above-mentioned deficiencies of the prior art, the present application proposes a three-dimensional model retrieval method and system based on high-order adjacency relation constraint, which uses illumination view and depth view to represent three-dimensional model, and mines potential relation between views based on view double grouping strategy, so as to enhance the robustness and discriminability of three-dimensional model feature representation, and further improve the stability and accuracy of three-dimensional model retrieval.

[0007] In order to achieve the above-mentioned purpose, one or more embodiments of the present application provide the following technical solutions:

[0008] In the first aspect, the present application discloses a three-dimensional model retrieval method based on high-order adjacency relation constraint, comprising:

[0009] Obtaining a three-dimensional model to be queried, rendering illumination view and depth view of the three-dimensional model to be queried, and channel splicing the illumination view and the depth view into multi-channel information;

[0010] Based on the multi-channel information, using a feature extraction model to extract initial features to obtain initial feature descriptors of each channel;

[0011] Based on the cosine similarity of the initial feature descriptors between each channel, the multi-view is initially grouped;

[0012] Using a high-order adjacency relation matrix to represent the structural correlation degree of each view, optimizing the initial grouping based on the high-order adjacency relation matrix to obtain a final grouping result, and obtaining a final feature descriptor based on the final grouping result;

[0013] Measuring the similarity between the final feature descriptor and the model features in the feature database to obtain a three-dimensional model retrieval result.

[0014] In the second aspect, the present application discloses a three-dimensional model retrieval system based on high-order adjacency relation constraint, comprising:

[0015] A data acquisition module is configured to obtain a three-dimensional model to be queried, render illumination view and depth view of the three-dimensional model to be queried, and channel splice the illumination view and the depth view into multi-channel information;

[0016] A feature extraction module is configured to extract initial features based on the multi-channel information, using a feature extraction model to obtain initial feature descriptors of each channel;

[0017] An initial grouping module is configured to initially group the multi-view based on the cosine similarity of the initial feature descriptors between each channel;

[0018] An optimization grouping module is configured to represent the structural correlation degree of each view by using a high-order adjacency relation matrix, to optimize the initial grouping based on the high-order adjacency relation matrix to obtain a final grouping result, and to obtain a final feature descriptor based on the final grouping result.

[0019] A model retrieval module is configured to perform similarity measurement on the final feature descriptor and model features in a feature database to obtain a three-dimensional model retrieval result.

[0020] In a third aspect, the present application discloses an electronic device, which comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, and when the computer instructions are executed by the processor, the steps of the three-dimensional model retrieval method based on high-order adjacency relation constraint are completed.

[0021] In a fourth aspect, the present application discloses a computer readable storage medium, which is configured to store computer instructions, and when the computer instructions are executed by a processor, the steps of the three-dimensional model retrieval method based on high-order adjacency relation constraint are completed.

[0022] In a fifth aspect, the present application discloses a computer program product, which comprises executable instructions stored in a computer readable storage medium, and when a processor of an electronic device reads the executable instructions from the computer readable storage medium and executes the executable instructions, the three-dimensional model retrieval method based on high-order adjacency relation constraint is implemented.

[0023] Compared with the prior art, the present application has the following beneficial effects:

[0024] The present application introduces a high-order adjacency relation modeling mechanism to overcome the problem that the traditional multi-view feature fusion method fails to fully exploit the potential relationship between views. By constructing a multi-order adjacency matrix, the deep structural correlation between multi-views is explored, and the feature interaction and fusion of cross-group views are realized on the basis of initial grouping, thereby improving the overall consistency and local sensitivity of model feature representation, and enhancing the expressiveness and robustness of model descriptors.

[0025] To further improve the aggregation effect of feature expression, a view double-grouping strategy is designed. The strategy first groups the multi-channel views according to the cosine similarity between view features. Then, the grouping result is optimized and adjusted based on the high-order adjacency relation to guide the view aggregation on a more reasonable structure. The in-group features are fused to form group-level descriptors, and all group-level features are uniformly aggregated to generate a final model descriptor. The strategy effectively improves the global expression ability of the model descriptor, and helps to improve the accuracy and stability of three-dimensional model retrieval.

[0026] The application takes full advantage of the depth image in expressing the three-dimensional model geometry and spatial structure, combines the light views and depth views from multiple perspectives, and fuses in the channel dimension, significantly enhancing the expression integrity and discrimination ability of view information, thereby effectively overcoming the problem that the existing rendered views lose structural information.

[0027] Advantages of the additional aspects of the application will be partially given in the following description, partially will become apparent from the following description, or will be understood by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0028] The accompanying drawings, which form a part of the specification, are included to provide further understanding of the application and are incorporated in and constitute a part of this specification. The illustrative embodiments of the application and their description are used to explain the application.

[0029] Figure 1 A flow chart of the three-dimensional model retrieval method based on high-order adjacency relationship constraint described in embodiment one of the application.

[0030] Figure 2 A framework chart of the three-dimensional model retrieval method based on high-order adjacency relationship constraint described in embodiment one of the application. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0032] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the application.

[0033] In the case of no conflict, the embodiments in the application and the features in the embodiments can be combined with each other.

[0034] Embodiment one

[0035] In one or more embodiments, in order to solve the two core problems of three-dimensional model structure information loss and potential relationship between views not being effectively mined and modeled caused by rendered views in three-dimensional model retrieval, a three-dimensional model retrieval method based on high-order adjacency relationship constraint is disclosed, as shown in Figures 1-2 The method comprises the following steps:

[0036] Step S1, obtaining a three-dimensional model to be queried, rendering a light view and a depth view of the three-dimensional model to be queried, and channel splicing the light view and the depth view into multi-channel information;

[0037] Step S1-1, obtaining a three-dimensional model to be queried;

[0038] Step S1-2, setting a virtual camera and generating a view;

[0039] Step S1-2-1, setting one or more virtual cameras in a three-dimensional space, which are placed around the three-dimensional model. Usually, the cameras are uniformly distributed around the center point of the model, ensuring that the entire three-dimensional model can be covered from multiple directions.

[0040] Step S1-2-2, the virtual camera takes a three-dimensional model from its respective position to generate a two-dimensional image. Specifically, each camera view captures a specific side or local detail of the model.

[0041] In this embodiment, the multi-view acquisition method is to place the three-dimensional model at the center, and the multiple virtual cameras are located at a regular circle with a 30° elevation angle, and the azimuth angle of the virtual camera is spaced apart by 30°, thereby generating 12 views, including view to view .

[0042] Step S1-3, based on the process of generating views in step S1-2, generating the light multi-view and the depth multi-view of the three-dimensional model . Input the light multi-view , depth multi-view rendered by the three-dimensional model; wherein, represents the light view of the th view of the th model, represents the depth view of the th view of the th model.

[0043] Specifically, in the view generation stage, the three-dimensional model is rendered using a fixed light source to obtain a light view reflecting the model's contour and surface texture information. At the same time, by calculating the distance from the virtual camera to the model surface, a depth view is generated that can express the geometric structure and spatial depth information.

[0044] Step S1-4, multi-channel information fusion;

[0045] Splice and fuse the light multi-view and the depth multi-view along the channel dimension to generate multi-channel information ; wherein, represents the multi-channel information of the th view of the th model.

[0046] This approach generates two types of views from multiple fixed perspectives and fuses them along the channel dimension to construct a multi-channel view representation. This allows for the consideration of both the shape and contour features and depth structure information of the model within the same input space. This strategy effectively alleviates the problems of limited view information representation and insufficient shape description in traditional methods.

[0047] Step S2: Based on the multi-channel information, use a feature extraction model to extract initial features to obtain the initial feature descriptors for each channel;

[0048] Extracting and fusing multi-view features: multi-channel information after channel stitching and fusion. The data is fed into a convolutional neural network for initial feature extraction, resulting in initial feature descriptors. ;in, Indicates the first The model of the first Initial feature descriptors for each perspective.

[0049] The convolutional neural network can adopt network structures such as ResNet50, and there is no limitation here.

[0050] Step S3: Perform initial grouping of the multi-view based on the cosine similarity of the initial feature descriptors between each channel;

[0051] Initial grouping based on cosine similarity. For the ... Multichannel descriptor Calculate its relationship with The cosine similarity between channels is used to obtain a cosine similarity score between that channel and other channels. Based on this score, the multiple views are grouped. .

[0052] Specifically, the cosine similarity of multiple views is mapped to an interval of 0 to 1, where this interval is divided into 3 groups on average. Each view is then mapped to a specific interval according to its cosine similarity, thus obtaining the initial grouping. ;in Indicates the first The preliminary grouping results of the model, the first The formula for calculating the cosine similarity between multi-channel information and other channel information is as follows:

[0053] (1)

[0054] In the formula, Let be the cosine similarity, and n be the number of other channels.

[0055] Step S4: Use a high-order adjacency relation matrix to characterize the structural correlation of each view, optimize the initial grouping based on the high-order adjacency relation matrix to obtain the final grouping result, and obtain the final feature descriptor based on the final grouping result;

[0056] Step S4-1: Optimize grouping results based on higher-order adjacency relations and mine potential structural associations among multiple views. This embodiment introduces a multi-order adjacency relation matrix to characterize the potential structural associations between different views, and in the initial grouping... The feature fusion process of multiple views is optimized based on this. Specifically, firstly, a multi-order adjacency relation matrix is ​​defined. , used to indicate the first The view and the first The view in the first The degree of structural correlation of order is calculated using the following formula:

[0057] (2)

[0058] In the formula, Here, s is the distance scaling parameter, and s-nearest neighbor relationship refers to the result obtained by the k-nearest neighbor algorithm. Indicates the first The and the first The view in the first The distance relationship under the order is defined as follows:

[0059] (3)

[0060] In the formula, Let i be the vector in the i-th row of the k-1 order relation matrix. Let be the vector in the j-th row of the k-1 order relation matrix.

[0061] In this embodiment, a second-order adjacency relationship is mainly adopted (i.e., Structural modeling is performed to fully explore the indirect relationships between views. Next, based on the second-order adjacency matrix... Optimize view grouping. The specific steps are as follows: Iterate through the initial groups. The degree of structural correlation between each pair of views in ( If the structural correlation between the two is greater than a set threshold, i.e. Then the first Zhang View and the First There must be a significant correlation between the views; otherwise, the first view is considered to be... Zhang View and the First There is no significant correlation between the tensor views; if the structural correlation is greater than a set threshold and the initial groupings do not belong to the same group, i.e. and and If they did not originally belong to the same group, then... and Each group is then placed into its corresponding subgroup. Through these steps, the optimized final grouping result can be obtained. The fusion result comprises initial grouping structure information and high-order adjacency relationship.

[0062] The embodiment optimizes the grouping result through the high-order adjacency relationship, and enhances the modeling capability of local and global structure information in multi-view feature expression.

[0063] Step S4-2, performing in-group average pooling on the final grouping result to obtain level feature descriptors, and then performing average pooling on the group level feature descriptors to obtain a final feature descriptor ; wherein, indicates the final feature descriptor of the mth model.

[0064] Step S5, performing similarity measurement on the final feature descriptor and model features in a feature database to obtain a three-dimensional model retrieval result.

[0065] Specifically, the final feature descriptor is input into the feature database as a model feature to be retrieved to perform similarity measurement, and a retrieval result is output according to the highest similarity feature. All feature descriptors are pre-stored in the feature database.

[0066] The embodiment introduces a view two-stage grouping strategy, the first stage performs preliminary grouping based on the cosine similarity between views, and the second stage optimizes and reconstructs the initial grouping result in combination with the high-order adjacency relationship, so that the mining and reconstruction of the potential structure relationship between views are realized. Through the mechanism, not only the structure consistency in the multi-view feature fusion process is improved, but also the sensitivity of the model descriptor to the three-dimensional shape difference is significantly enhanced.

[0067] The three-dimensional model retrieval method proposed in the embodiment can significantly improve the accuracy and stability of the retrieval result, and has strong practical value and broad application prospect.

[0068] Embodiment two

[0069] In one or more embodiments, a three-dimensional model retrieval system based on high-order adjacency relationship constraint is disclosed, specifically comprising:

[0070] A data acquisition module is configured to acquire a three-dimensional model to be queried, render an illumination view and a depth view of the three-dimensional model to be queried, and splice the illumination view and the depth view into multi-channel information.

[0071] A feature extraction module is configured to extract initial features based on the multi-channel information, and extract initial feature descriptors of each channel by using a feature extraction model.

[0072] An initial grouping module is configured to perform initial grouping on multi-views based on the cosine similarity of the initial feature descriptors between each channel. ​

[0073] The optimization grouping module is configured to represent the structural correlation degree of each view by using a high-order adjacency relation matrix, optimize the initial grouping based on the high-order adjacency relation matrix to obtain a final grouping result, and obtain a final feature descriptor based on the final grouping result.

[0074] The model retrieval module is configured to perform similarity measurement on the final feature descriptor and model features in a feature database to obtain a three-dimensional model retrieval result.

[0075] Embodiment three

[0076] The embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the three-dimensional model retrieval method based on high-order adjacency relation constraint are completed.

[0077] Embodiment four

[0078] The embodiment provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the steps of the three-dimensional model retrieval method based on high-order adjacency relation constraint are completed.

[0079] Embodiment five

[0080] The embodiment provides a computer program product, which includes executable instructions, the executable instructions are a kind of computer instructions;The executable instructions are stored in computer readable storage medium.When the processor of electronic device reads the executable instructions from the computer readable storage medium, the processor executes the executable instructions, so that the electronic device executes the three-dimensional model retrieval method based on high-order adjacency relation constraint provided in the above embodiment of the present embodiment.

[0081] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of general-purpose computer, special-purpose computer, embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of computer or other programmable data processing device realize the functions specified in the flow Figure 1 The device for realizing the functions specified in one flow or multiple flows and / or blocks Figure 1 The device for realizing the functions specified in one flow or multiple flows and / or blocks

[0082] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in accordance with the embodiments herein can be implemented as computer program instructions. Figure 1

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in accordance with the embodiments herein can be implemented as computer program instructions. Figure 1

[0084] The above description of the various embodiments can have emphasized certain aspects of the various embodiments, which description relating to a certain embodiment can be applicable to other embodiments. For the sake of completeness, the description related to one embodiment can be considered to apply to other embodiments described herein, and vice versa.

[0085] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. The present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the present application.​​

Claims

1. A three-dimensional model retrieval method based on high-order adjacency constraints, characterized in that, include: Obtain the 3D model to be queried, render the lighting view and depth view of the 3D model to be queried, and stitch the lighting view and depth view into multi-channel information. Based on the multi-channel information, an initial feature descriptor for each channel is obtained by using a feature extraction model to extract initial features. Initial grouping of multiple views is performed based on the cosine similarity of the initial feature descriptors between each channel; A high-order adjacency matrix is ​​used to characterize the structural correlation of each view. The initial grouping is optimized based on the high-order adjacency matrix to obtain the final grouping result. The final feature descriptor is obtained based on the final grouping result. The final feature descriptor is compared with the model features in the feature database to obtain the 3D model retrieval results; The use of a high-order adjacency matrix to characterize the structural correlation of each view is specifically as follows: The adjacency relation matrix is ​​used to represent the first... The view and the first The view in the first The degree of structural correlation of order is calculated using the following formula: In the formula, Multi-order adjacency relation matrix For distance scaling parameters, Indicates the first The and the first The view in the first The distance relationships at each level are as follows: In the formula, For the first One initial feature descriptor, For the j-th initial feature descriptor, Let i be the vector in the i-th row of the k-1 order relation matrix. Let be the vector in the j-th row of the k-1 order relation matrix.

2. The 3D model retrieval method based on high-order adjacency constraints as described in claim 1, characterized in that, The initial grouping of multiple views based on the cosine similarity of the initial feature descriptors between each channel is specifically as follows: The cosine similarity of multiple views is mapped to an interval of 0 to 1. This interval is divided into several groups on average. Each view is then mapped to a specific interval according to its cosine similarity, resulting in preliminary grouping.

3. The 3D model retrieval method based on high-order adjacency constraints as described in claim 1, characterized in that, The optimization of the initial grouping based on the higher-order adjacency matrix to obtain the final grouping result is specifically as follows: traverse the feature structure correlation of each pair of views in the initial group. If the correlation between the two views is greater than a set threshold, it is determined that there is a significant correlation between the two views; otherwise, it is determined that there is no significant correlation between the two views. If the correlation between the two views is greater than the set threshold and the initial grouping does not belong to the same group, the two views are respectively included in their corresponding groups.

4. The 3D model retrieval method based on high-order adjacency constraints as described in claim 1, characterized in that, The process of obtaining the final feature descriptor based on the final grouping result specifically involves: performing intra-group average pooling on the final grouping result to obtain group-level feature descriptors, and then performing average pooling on the group-level feature descriptors to obtain the final feature descriptor.

5. The 3D model retrieval method based on high-order adjacency constraints as described in claim 1, characterized in that, Render the lighting and depth views of the 3D model to be queried, specifically as follows: A fixed light source is used to render the 3D model, obtaining a lighting view that reflects the model's outline and surface texture information; at the same time, by calculating the distance from the virtual camera to the model's surface, a depth view that can express geometric structure and spatial depth information is generated.

6. A 3D model retrieval system based on high-order adjacency constraints, characterized in that, include: The data acquisition module is used to acquire the 3D model to be queried, render the lighting view and depth view of the 3D model to be queried, and stitch the lighting view and depth view into multi-channel information. The feature extraction module is used to extract initial features based on the multi-channel information using a feature extraction model to obtain initial feature descriptors for each channel; The initial grouping module is used to initially group multiple views based on the cosine similarity of the initial feature descriptors between each channel; The optimization grouping module is used to characterize the structural correlation of each view using a high-order adjacency relation matrix, optimize the initial grouping based on the high-order adjacency relation matrix to obtain the final grouping result, and obtain the final feature descriptor based on the final grouping result; The model retrieval module is used to measure the similarity between the final feature descriptor and the model features in the feature database to obtain the three-dimensional model retrieval results; The use of a high-order adjacency matrix to characterize the structural correlation of each view is specifically as follows: The adjacency relation matrix is ​​used to represent the first... The view and the first The view in the first The degree of structural correlation of order is calculated using the following formula: In the formula, Multi-order adjacency relation matrix For distance scaling parameters, Indicates the first The and the first The view in the first The distance relationships at each level are as follows: In the formula, For the first One initial feature descriptor, For the j-th initial feature descriptor, Let i be the vector in the i-th row of the k-1 order relation matrix. Let be the vector in the j-th row of the k-1 order relation matrix.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the three-dimensional model retrieval method based on higher-order adjacency constraints as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the three-dimensional model retrieval method based on higher-order adjacency constraints as described in any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the three-dimensional model retrieval method based on higher-order adjacency constraints as described in any one of claims 1-5.

Images