Single-view 3D model retrieval method and system based on feature fusion and view grouping

Abstract

The application discloses a single-image three-dimensional model retrieval method and system based on feature fusion and view grouping, and the method comprises the following steps: taking the known query image, the background-free image and a plurality of views corresponding to each three-dimensional model as input values of a three-dimensional model retrieval network, taking a three-dimensional model most similar to the query image as an output value of the three-dimensional model retrieval network, training the three-dimensional model retrieval network by using first, second and third loss functions in sequence to obtain a trained three-dimensional model retrieval network; inputting a query image, a mask image of the query image and a three-dimensional model set into the fine-tuned three-dimensional model retrieval network to obtain image features corresponding to the query image and final features corresponding to each three-dimensional model; calculating the similarity between the image features corresponding to the query image and the final features corresponding to each three-dimensional model, and selecting a three-dimensional model corresponding to the maximum similarity value as a final retrieval result output.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional model retrieval, in particular to a single-image three-dimensional model retrieval method and system based on feature fusion and view grouping. BACKGROUND

[0002] With the rapid development of three-dimensional model database and the easy access to real-world pictures, single-image based three-dimensional model retrieval has gradually become a hot topic. The goal is to retrieve a series of three-dimensional models similar to the query image through the given query image.

[0003] Single-image based three-dimensional model retrieval can be divided into instance level and category level according to the retrieval level. In category level retrieval, only three-dimensional models of the same category as the query image need to be found, so the accuracy requirement is lower. However, in instance level retrieval, the most similar three-dimensional model to the query image needs to be found, so the accuracy is higher. Since most databases have provided instance labels of two-dimensional images and three-dimensional models, and instance level retrieval has higher application value, the present application selects instance level single-image based three-dimensional model retrieval as the research direction.

[0004] In the past few years, people have been committed to improving the accuracy of single-image based three-dimensional model retrieval, and have proposed many related works. In practical applications, single-image based three-dimensional model retrieval often has the following problems. First, since the query image is a two-dimensional image taken in the real world, its complex background is inevitable. Second, since three-dimensional models are difficult to obtain, their training quantity is limited. Finally, since two-dimensional images and three-dimensional models are two different modal data, there is a huge domain gap. These problems will affect the final retrieval effect. SUMMARY

[0005] In order to solve the problems of the prior art, the present application provides a single-image three-dimensional model retrieval method and system based on feature fusion and view grouping, which can effectively improve the accuracy of image retrieval three-dimensional model and solve the problem of affecting the retrieval accuracy.

[0006] On the one hand, a single-image three-dimensional model retrieval method based on feature fusion and view grouping is provided, comprising:

[0007] Obtaining a known query image, a mask image corresponding to the known query image, and a three-dimensional model set; each three-dimensional model in the three-dimensional model set is provided with a label; the label includes: a three-dimensional model corresponding to the known query image, or a three-dimensional model not corresponding to the known query image;

[0008] obtain a background-free image based on the known query image and the mask image corresponding to the known query image; preprocess each three-dimensional model in the three-dimensional model set in a set manner to obtain a plurality of views corresponding to each three-dimensional model;

[0009] input the known query image, the background-free image, and the plurality of views corresponding to each three-dimensional model into the three-dimensional model retrieval network as input values, input the three-dimensional model most similar to the query image into the three-dimensional model retrieval network as an output value, train the three-dimensional model retrieval network using the first, second, and third loss functions in sequence to obtain a trained three-dimensional model retrieval network, and fine-tune the trained three-dimensional model retrieval network to obtain a fine-tuned three-dimensional model retrieval network;

[0010] obtain a query image, a mask image of the query image, and a three-dimensional model set; input the query image, the mask image of the query image, and the three-dimensional model set into the fine-tuned three-dimensional model retrieval network to obtain an image feature corresponding to the query image and a final feature corresponding to each three-dimensional model; calculate the similarity between the image feature corresponding to the query image and the final feature corresponding to each three-dimensional model, and select the three-dimensional model corresponding to the maximum similarity value as the final retrieval result.

[0011] In another aspect, a single-image three-dimensional model retrieval system based on feature fusion and view grouping is provided, comprising:

[0012] The obtaining module is configured to: obtain a known query image, a mask image corresponding to the known query image, and a three-dimensional model set; each three-dimensional model in the three-dimensional model set is provided with a label; the label includes: a three-dimensional model corresponding to the known query image, or a three-dimensional model not corresponding to the known query image;

[0013] The processing module is configured to: obtain a background-free image based on the known query image and the mask image corresponding to the known query image; preprocess each three-dimensional model in the three-dimensional model set in a set manner to obtain a plurality of views corresponding to each three-dimensional model;

[0014] The training module is configured to: input the known query image, the background-free image, and the plurality of views corresponding to each three-dimensional model into the three-dimensional model retrieval network as input values, input the three-dimensional model most similar to the query image into the three-dimensional model retrieval network as an output value, train the three-dimensional model retrieval network using the first, second, and third loss functions in sequence to obtain a trained three-dimensional model retrieval network, and fine-tune the trained three-dimensional model retrieval network to obtain a fine-tuned three-dimensional model retrieval network;

[0015] The application module is configured to: acquire a query image, a mask image of the query image, and a three-dimensional model set; input the query image, the mask image of the query image, and the three-dimensional model set into the fine-tuned three-dimensional model retrieval network to obtain image features corresponding to the query image and final features corresponding to each three-dimensional model; calculate the similarity between the image features corresponding to the query image and the final features corresponding to each three-dimensional model, and select a three-dimensional model corresponding to a maximum similarity value as a final retrieval result output.

[0016] In another aspect, an electronic device is also provided, comprising:

[0017] a memory for non-transitorily storing computer readable instructions; and

[0018] a processor for running the computer readable instructions,

[0019] When the computer readable instructions are run by the processor, the method of the first aspect is executed.

[0020] In another aspect, a storage medium is also provided, which non-transitorily stores computer readable instructions, wherein when the non-transitory computer readable instructions are executed by a computer, the instructions of the method of the first aspect are executed.

[0021] In another aspect, a computer program product is also provided, which comprises a computer program for implementing the method of the first aspect when running on one or more processors.

[0022] The above technical solution has the following advantages or beneficial effects:

[0023] The method of the present application is used to improve the three-dimensional model retrieval accuracy based on a single image. Firstly, the feature fusion solves the problem of complex query image background, and highlights the important information of the query image. Secondly, the view grouping solves the problem of limited number of models in the training process. Finally, the distance between different features of the same three-dimensional model and the distance between the image and the three-dimensional model are considered to reduce the domain gap between them. In addition, the design principle of this method is reliable, the structure is simple, and it has very wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the description of the exemplary embodiments of the present application and the explanation thereof serve to explain the present application, and do not constitute an improper limitation of the present application.

[0025] Figure 1 The training phase method flowchart of Example 1;

[0026] Figure 2A test phase method flowchart for Example One. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0028] Example One

[0029] The present embodiment provides a single-image three-dimensional model retrieval method based on feature fusion and view grouping;

[0030] As shown in Figure 1 and Figure 2 , the single-image three-dimensional model retrieval method based on feature fusion and view grouping comprises:

[0031] S101: Obtain a known query image, a mask image corresponding to the known query image, and a three-dimensional model set; each three-dimensional model in the three-dimensional model set is provided with a label; the label includes: a three-dimensional model corresponding to the known query image, or a three-dimensional model not corresponding to the known query image;

[0032] S102: Obtain a background-free image based on the known query image and the mask image corresponding to the known query image; each three-dimensional model in the three-dimensional model set is preprocessed in a set manner to obtain a plurality of views corresponding to each three-dimensional model;

[0033] S103: Take the known query image, the background-free image, and the plurality of views corresponding to each three-dimensional model as input values of a three-dimensional model retrieval network, take a three-dimensional model most similar to the query image as an output value of the three-dimensional model retrieval network, train the three-dimensional model retrieval network using first, second, and third loss functions in sequence, obtain a trained three-dimensional model retrieval network, and fine-tune the trained three-dimensional model retrieval network to obtain a fine-tuned three-dimensional model retrieval network;

[0034] S104: Obtain a to-be-queried image, a mask image of the to-be-queried image, and a three-dimensional model set; input the to-be-queried image, the mask image of the to-be-queried image, and the three-dimensional model set into the fine-tuned three-dimensional model retrieval network to obtain an image feature corresponding to the to-be-queried image and a final feature corresponding to each three-dimensional model; calculate the similarity between the image feature corresponding to the to-be-queried image and the final feature corresponding to each three-dimensional model, and select a three-dimensional model corresponding to a maximum similarity value as a final retrieval result output.

[0035] Further, the S101: obtaining a known query image, a mask image corresponding to the known query image, and a three-dimensional model set;

[0036] The known query image refers to an image used in a training stage. The image can be an image containing a target object, and the target object includes a human body, an animal, or an object.

[0037] The mask image corresponding to the known query image, wherein the mask image is a mask image obtained by performing mask processing on the known query image. The mask processing specifically includes: first obtaining the main part of the image through an instance segmentation algorithm, and then processing the main part of the image into a binary image (black and white image) for display.

[0038] The three-dimensional model set includes a plurality of three-dimensional models.

[0039] Exemplarily, the S101: obtaining a known query image, a mask image corresponding to the known query image, and a three-dimensional model set, includes:

[0040] Inputting a query image q, a mask image m, and a three-dimensional model set S = {s 1 ,s 2 ,...,s n}, wherein s i represents the i-th three-dimensional model. For better explanation, the i-th three-dimensional model s i is taken as an example for description.

[0041] Further, each three-dimensional model in the three-dimensional model set is provided with a label. The label includes: a three-dimensional model corresponding to the known query image, or a three-dimensional model not corresponding to the query image. The label includes: an animal of a certain kind or an object of a certain kind, etc.

[0042] Further, the S102: obtaining a background-free image based on the known query image and the mask image corresponding to the known query image, includes:

[0043] First, the main part of the image is obtained through an instance segmentation algorithm, the main part of the query image is retained, and the background of the query image is deleted to obtain a background-free image.

[0044] Exemplarily, the background-free image g is obtained by using the query image q and the mask image m.

[0045] Further, the S102: each three-dimensional model in the three-dimensional model set is preprocessed in a set manner to obtain a plurality of views corresponding to each three-dimensional model, includes:

[0046] The three-dimensional model is rendered into a gray-scale image from different angles by a plane coloring rendering method, and each image obtained from an angle is taken as a view of the three-dimensional model, and multiple angles obtain multiple views of the three-dimensional model.

[0047] Exemplarily, the three-dimensional model is preprocessed to obtain 12 views Among them, The zth view of the ith three-dimensional model is represented.

[0048] Further, the S103: taking the known query image, the background-free image and the several views corresponding to each three-dimensional model as input values of a three-dimensional model retrieval network, taking a three-dimensional model most similar to the query image as an output value of the three-dimensional model retrieval network, training the three-dimensional model retrieval network in turn by using first, second and third loss functions, and obtaining a trained three-dimensional model retrieval network, wherein the three-dimensional model retrieval network comprises:

[0049] The first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the attention mechanism layer and the pooling layer;

[0050] The known query image is input into the first convolutional neural network to obtain a query image feature;

[0051] The background-free image is input into the first convolutional neural network to obtain a background-free image feature;

[0052] The query image feature and the background-free image feature are fused by an addition operation to obtain a fused feature;

[0053] The fused feature is input into the second convolutional neural network to obtain a final query image feature;

[0054] The several views corresponding to each three-dimensional model are input into the third convolutional neural network to obtain several view features corresponding to each three-dimensional model;

[0055] The final query image feature and the several view features corresponding to each three-dimensional model are input into the attention mechanism layer, the attention mechanism layer performs grouping operation on the features, and the grouping operation result is input into the pooling layer, and the pooling layer outputs several features of each three-dimensional model under the current final query image feature;

[0056] According to the several features of the three-dimensional model, the distance between multiple features of the same three-dimensional model is narrowed and the distance between multiple features of different three-dimensional models is widened by the first loss function.

[0057] According to the query image features and the plurality of features of the three-dimensional model, the distance between each query image feature and the corresponding plurality of features of the three-dimensional model is narrowed through the second loss function and the third loss function.

[0058] Further, the attention mechanism layer groups the features, including:

[0059] Through the attention mechanism, each view obtains a weight, and according to the weight size, the plurality of view features of each three-dimensional model is grouped to obtain a plurality of groups of features: according to the set threshold interval range to which the weight value belongs, a plurality of groups of features are obtained, and the features in each interval range are a group;

[0060] Or, the views with a weight value greater than the set threshold value are grouped into a group, and the views with a weight value less than the set threshold value are grouped into another group.

[0061] Further, the known query image is input into the first convolutional neural network to obtain the query image feature; the background-free image is input into the first convolutional neural network to obtain the background-free image feature, and the query image feature and the background-free image feature are fused to obtain the fused feature; including:

[0062] The query image q and the background-free image g obtain the query image feature and the background-free image feature through the convolutional neural network F1, and the two features are fused to obtain the final query image feature f through the convolutional neural network F2.

[0063] Further, the plurality of views corresponding to each three-dimensional model is input into the third convolutional neural network to obtain a plurality of view features corresponding to each three-dimensional model; the final query image feature and the plurality of view features corresponding to each three-dimensional model are input into the attention mechanism layer, the attention mechanism layer groups the features, and the grouping operation result is input into the pooling layer, and the pooling layer outputs a plurality of features of each three-dimensional model under the current final query image feature, including:

[0064] The features of the multi-view three-dimensional model are extracted through the convolutional neural network F3 Wherein, The z-th view feature of the i-th three-dimensional model is represented. The query image feature f and the plurality of view features of the i-th three-dimensional model After grouping through the attention mechanism, a plurality of features of the i-th three-dimensional model with respect to the query image feature f are obtained

[0065] Further, the query image feature f and the plurality of view features of the i-th three-dimensional model After grouping through the attention mechanism, a plurality of features of the i-th three-dimensional model with respect to the query image feature f are obtained comprising:

[0066] Given query image feature f and multiple view features of the i-th 3D model Firstly, given query image feature f and multiple view features of the i-th 3D model Through the attention mechanism, the weight of each view with respect to the query image feature f is obtained;

[0067] Secondly, according to the weight size of each view, the views are grouped, and the views are grouped according to the weight belonging to the set threshold interval range, and the view features after grouping obtain multiple features of the i-th 3D model with respect to the query image feature f through the pooling operation

[0068] Further, according to the plurality of features of the 3D model, the distance between the plurality of features of the same 3D model is narrowed and the distance between the plurality of features of different 3D models is pushed away through the first loss function, comprising:

[0069] Given the plurality of features of all 3D models, according to the plurality of features of all 3D models, the distance between the plurality of features of the same 3D model is continuously narrowed and the distance between the plurality of features of different 3D models is continuously pushed away by continuously optimizing the first loss function L1:

[0070]

[0071] Wherein, the distance between two features is calculated by inner product, one feature of the first 3D model is selected as an anchor point, represent the distance between the anchor point feature of the first 3D model and the remaining features of the first 3D model, m represents the number of multiple features of the same 3D model, represent the distance between the feature and the plurality of features of other 3D models, k represents the number of multiple features of all 3D models, represent the distance between the selected feature of the n-th model and the remaining features of the n-th model, represent the distance between the selected feature of the n-th model and the plurality of features of other 3D models.

[0072] Further, according to the query image feature and the plurality of features of the 3D model, the distance between each query image feature and the corresponding plurality of features of the 3D model is narrowed through the second loss function and the third loss function, comprising:

[0073] Given the query image features and the plurality of features of all three-dimensional models, at the instance level, according to the given query image features and the features of all three-dimensional models, find the three-dimensional model corresponding to the query image, by continuously optimizing the second loss function L2, continuously shorten the distance between the query image and the corresponding three-dimensional model, and lengthen the distance between the query image and other three-dimensional models:

[0074]

[0075] wherein d 1 1,...,d ′ m denotes the distance between the plurality of features of the query image and the corresponding three-dimensional model, d ′ 1,...,d ′ k denotes the distance between the plurality of features of the query image and other three-dimensional models; the distance between the features is calculated by the inner product of two features. Each image has a corresponding three-dimensional model with a label, and other three-dimensional models refer to all models other than the corresponding three-dimensional model with the label.

[0076] At the category level, according to the given query image features and the features of all three-dimensional models, find the query image and three-dimensional model of the same category, by continuously optimizing the third loss function, continuously shorten the distance between them, and lengthen the distance between the query image and three-dimensional model of different categories:

[0077]

[0078] wherein d ′ 1,d ′ 2...,d ′ p denotes the distance between the plurality of features of the query image and the three-dimensional model of the same category to which the query image belongs, p denotes the number of the plurality of features of the three-dimensional model of the same category to which the query image belongs, d ′ 1,d ′ 2...,d ′ k denotes the distance between the plurality of features of the query image and the three-dimensional model of the different category to which the query image belongs. The distance between the features is calculated by the inner product of two features.

[0079] Further, the S103: fine-tuning the trained three-dimensional model retrieval network to obtain a fine-tuned three-dimensional model retrieval network, comprising:

[0080] Remove the attention mechanism layer, and directly connect the output end of the third convolutional neural network with the input end of the pooling layer.

[0081] Further, the S104: obtaining the to-be-queried image, the mask image of the to-be-queried image, and the three-dimensional model set; inputting the to-be-queried image, the mask image of the to-be-queried image, and the three-dimensional model set into the fine-tuned three-dimensional model retrieval network to obtain an image feature corresponding to the to-be-queried image and a final feature corresponding to each three-dimensional model, including:

[0082] inputting the to-be-queried image into the first convolutional neural network to obtain a to-be-queried image feature;

[0083] inputting the background-free image into the first convolutional neural network to obtain a background-free image feature;

[0084] performing feature fusion on the to-be-queried image feature and the background-free image feature to obtain a fused feature;

[0085] inputting the fused feature into the second convolutional neural network to obtain a final query image feature;

[0086] inputting the plurality of views corresponding to each three-dimensional model into the third convolutional neural network to obtain a plurality of view features corresponding to each three-dimensional model;

[0087] inputting the final query image feature and the plurality of view features corresponding to each three-dimensional model into the pooling layer, and the pooling layer outputs a plurality of features of each three-dimensional model under the current final query image feature.

[0088] Further, the similarity between the image feature corresponding to the to-be-queried image and the final feature corresponding to each three-dimensional model is calculated, and the three-dimensional model corresponding to the maximum similarity value is selected as the final retrieval result output, and the similarity is calculated by using the cosine similarity.

[0089] The method first fuses the features of the query image and the background-free image to highlight the key information in the image and reduce the interference of the complex background; secondly, the plurality of feature representations of the same model are generated by view grouping to increase the number of model training; finally, the distance between different feature representations of the same model is narrowed, and the distance between the image and the three-dimensional model is narrowed at the instance level and the class level, thereby reducing the domain gap between the two. The method can help the image retrieval to more correct three-dimensional models, and solves the problem of affecting the retrieval accuracy.

[0090] The method focuses on feature fusion and view grouping, reduces the distance between images and three-dimensional models, and improves the accuracy of retrieving three-dimensional models from a single image. First, by introducing a background-free image and a query image for feature fusion, the key information in the query image can be effectively highlighted, and the background interference can be reduced. Second, by view grouping, multiple feature representations can be obtained to increase the number of trained models to some extent. Finally, by reducing the distance between multiple feature representations of the same three-dimensional model and narrowing the distance between images and three-dimensional models at the class and instance levels, the accuracy and reliability of the results can be significantly improved, thereby further improving the retrieval precision.

[0091] Embodiment Two

[0092] The embodiment provides a single-image three-dimensional model retrieval system based on feature fusion and view grouping, comprising:

[0093] An acquisition module configured to acquire a known query image, a mask image corresponding to the known query image, and a three-dimensional model set; each three-dimensional model in the three-dimensional model set is provided with a label; the label includes: a three-dimensional model corresponding to the known query image, or a three-dimensional model not corresponding to the known query image;

[0094] A processing module configured to obtain a background-free image based on the known query image and the mask image corresponding to the known query image; and pre-process each three-dimensional model in the three-dimensional model set in a set manner to obtain a plurality of views corresponding to each three-dimensional model.

[0095] A training module configured to take the known query image, the background-free image, and the plurality of views corresponding to each three-dimensional model as input values of a three-dimensional model retrieval network, take a three-dimensional model most similar to the query image as an output value of the three-dimensional model retrieval network, train the three-dimensional model retrieval network using first, second, and third loss functions in sequence, and obtain a trained three-dimensional model retrieval network; and fine-tune the trained three-dimensional model retrieval network to obtain a fine-tuned three-dimensional model retrieval network.

[0096] An application module configured to acquire a to-be-queried image, a mask image of the to-be-queried image, and the three-dimensional model set; input the to-be-queried image, the mask image of the to-be-queried image, and the three-dimensional model set into the fine-tuned three-dimensional model retrieval network to obtain an image feature corresponding to the to-be-queried image and a final feature corresponding to each three-dimensional model; calculate the similarity between the image feature corresponding to the to-be-queried image and the final feature corresponding to each three-dimensional model, and select a three-dimensional model corresponding to a maximum similarity value as a final retrieval result output.

[0097] It should be noted that the above acquisition module, processing module, training module and application module correspond to steps S101-S104 in Embodiment One, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment One. It should be noted that the above modules can be executed in a computer system such as a set of computer executable instructions as part of a system.

[0098] The description of each embodiment in the above embodiments has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0099] The proposed system can be implemented in other ways. For example, the system embodiments described above are only illustrative, for example, the division of the above modules is only a logical function division, and in actual implementation, there can be another division method, for example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0100] Embodiment Three

[0101] The embodiment also provides an electronic device, including one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method described in Embodiment One.

[0102] It should be understood that in the embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0103] The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0104] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software.

[0105] The method in the embodiment one can be directly embodied as being completed by a hardware processor, or being completed by a combination of hardware and software modules in the processor. The software modules can be located in a storage medium in the art, such as a random access memory, a flash memory, a read only memory, a programmable read only memory, an electrically erasable programmable memory, a register, or the like. The storage medium is located in the storage, and the processor reads information in the storage, and combines the hardware to complete the steps of the above method. To avoid repetition, no further detailed description is given here.

[0106] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0107] Embodiment four: the embodiment also provides a computer readable storage medium for storing computer instructions, the computer instructions are executed by a processor to complete the method described in embodiment one.

[0108] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A single-image 3D model retrieval method based on feature fusion and view grouping, characterized by: include: Obtain the known query image, the mask image corresponding to the known query image, and the set of 3D models; Each 3D model in the set of 3D models is labeled; the label includes: whether it is a 3D model corresponding to a known query image, or whether it is not a 3D model corresponding to a known query image. Based on the known query image and the mask image corresponding to the known query image, a background-free image is obtained; each 3D model in the 3D model set is preprocessed according to a set method to obtain several views corresponding to each 3D model; The known query image, the backgroundless image, and several views corresponding to each 3D model are used as input values ​​for the 3D model retrieval network. The 3D model most similar to the query image is used as the output value of the 3D model retrieval network. The 3D model retrieval network is trained by sequentially applying the first, second, and third loss functions to obtain the trained 3D model retrieval network. The trained 3D model retrieval network is then fine-tuned to obtain the fine-tuned 3D model retrieval network. The 3D model retrieval network includes: First convolutional neural network, second convolutional neural network, third convolutional neural network, attention mechanism layer and pooling layer; In this process, the known query image is input into the first convolutional neural network to obtain the query image features; The background-free image is input into the first convolutional neural network to obtain the background-free image features; The query image features and background-free image features are fused by addition to obtain the fused features. The fused features are then input into the second convolutional neural network to obtain the final query image features. The several views corresponding to each 3D model are input into the third convolutional neural network to obtain the features of the several views corresponding to each 3D model. The final query image features and several view features corresponding to each 3D model are input into the attention mechanism layer. The attention mechanism layer performs grouping operations on the features, and the grouping results are input into the pooling layer. The pooling layer outputs several features of each 3D model under the current final query image features. Based on several features of the 3D model, the first loss function is used to narrow the distance between multiple features of the same 3D model and widen the distance between multiple features of different 3D models. Based on the features of the query image and several features of the 3D model, the second loss function and the third loss function are used to narrow the distance between the features of each query image and the corresponding multiple features of the 3D model; Obtain the query image, its mask image, and a set of 3D models; input the query image, its mask image, and the set of 3D models into a fine-tuned 3D model retrieval network to obtain the image features corresponding to the query image and the final features corresponding to each 3D model; calculate the similarity between the image features corresponding to the query image and the final features corresponding to each 3D model, and select the 3D model with the highest similarity value as the final retrieval result output.

2. The single-image 3D model retrieval method based on feature fusion and view grouping as described in claim 1, characterized in that, The attention mechanism layer performs grouping operations on features, including: Through the attention mechanism, each view is assigned a weight. Based on the weight, the features of several views of each 3D model are grouped into multiple groups of features: based on the set threshold range to which the weight values ​​belong, several groups of features are obtained, and the features within each range constitute a group. Alternatively, views with weight values ​​greater than a set threshold can be grouped into one group, and views with weight values ​​less than a set threshold can be grouped into another group.

3. The single-image 3D model retrieval method based on feature fusion and view grouping as described in claim 1, characterized in that, The step of narrowing the distance between multiple features of the same 3D model and widening the distance between multiple features of different 3D models by using a first loss function based on several features of the 3D model includes: Given multiple features of all 3D models, the first loss function is continuously optimized based on these features. This continuously narrows the distance between multiple features of the same 3D model and widens the distance between multiple features of different 3D models. ；（1） In this process, the distance between two features is calculated using the inner product, and one of the features of the first 3D model is selected as the anchor point. This represents the distance between the anchor point feature of the first 3D model and the remaining features of the first 3D model. This indicates the number of multiple features in the same 3D model. This represents the distance between this feature and multiple features of the other 3D model. This represents the number of features in all 3D models. Representing the The selected features of the model and the first The distance between the remaining features of each model Representing the The distance between selected features of one model and multiple features of other 3D models.

4. The single-image 3D model retrieval method based on feature fusion and view grouping as described in claim 1, characterized in that, The step of narrowing the distance between the features of each query image and the corresponding multiple features of the 3D model based on the features of the query image and several features of the 3D model, using a second loss function and a third loss function, includes: Given query image features and multiple features of all 3D models, at the instance level, based on the given query image features and all 3D model features, find the 3D model corresponding to the query image, by continuously optimizing the second loss function. It continuously narrows the distance between the query image and the corresponding 3D model, and widens the distance between the query image and other 3D models: in, , . . . , This represents the distance between multiple features of the query image and its corresponding 3D model. , .. . , This represents the distance between multiple features of the query image and other 3D models; the distance between features is calculated by the inner product of two features.

5. The single-image 3D model retrieval method based on feature fusion and view grouping as described in claim 4, characterized in that, At the category level, based on the given query image features and all 3D model features, query images and 3D models of the same category are found. By continuously optimizing the third loss function, the distance between them is continuously reduced, while the distance between query images and 3D models of different categories is increased. in, , . . , This represents the distance between the query image and multiple features of a 3D model belonging to the same category as the query image. This indicates and queries the number of multiple features of a 3D model that belong to the same category in an image. , . . , This represents the distance between the query image and multiple features of a 3D model that do not belong to the same category as the query image.

6. The single-image 3D model retrieval method based on feature fusion and view grouping as described in claim 1, characterized in that, Obtain the query image, its mask image, and a set of 3D models; input the query image, its mask image, and the set of 3D models into a fine-tuned 3D model retrieval network to obtain the image features corresponding to the query image and the final features corresponding to each 3D model, including: The image to be queried is input into the first convolutional neural network to obtain the features of the image to be queried; The background-free image is input into the first convolutional neural network to obtain the background-free image features; The features of the image to be queried and the features of the image without background are fused to obtain the fused features; The fused features are then input into the second convolutional neural network to obtain the final query image features. The several views corresponding to each 3D model are input into the third convolutional neural network to obtain the features of the several views corresponding to each 3D model. The final query image features and several view features corresponding to each 3D model are input into the pooling layer, and the pooling layer outputs several features of each 3D model under the current final query image features.

7. A single-image 3D model retrieval system based on feature fusion and view grouping, characterized in that: include: The acquisition module is configured to acquire a known query image, a mask image corresponding to the known query image, and a set of 3D models. Each 3D model in the set of 3D models is labeled; the label includes: whether it is a 3D model corresponding to a known query image, or whether it is not a 3D model corresponding to a known query image. The processing module is configured to: obtain a background-free image based on the known query image and the mask image corresponding to the known query image; preprocess each 3D model in the 3D model set according to a set method to obtain several views corresponding to each 3D model; The training module is configured to: take the known query image, the backgroundless image, and several views corresponding to each 3D model as input values ​​to the 3D model retrieval network; take the 3D model most similar to the query image as the output value of the 3D model retrieval network; train the 3D model retrieval network sequentially using the first, second, and third loss functions to obtain the trained 3D model retrieval network; and fine-tune the trained 3D model retrieval network to obtain the fine-tuned 3D model retrieval network. The 3D model retrieval network includes: First convolutional neural network, second convolutional neural network, third convolutional neural network, attention mechanism layer and pooling layer; In this process, the known query image is input into the first convolutional neural network to obtain the query image features; The background-free image is input into the first convolutional neural network to obtain the background-free image features; The query image features and background-free image features are fused by addition to obtain the fused features. The fused features are then input into the second convolutional neural network to obtain the final query image features. The several views corresponding to each 3D model are input into the third convolutional neural network to obtain the features of the several views corresponding to each 3D model. The final query image features and several view features corresponding to each 3D model are input into the attention mechanism layer. The attention mechanism layer performs grouping operations on the features, and the grouping results are input into the pooling layer. The pooling layer outputs several features of each 3D model under the current final query image features. Based on several features of the 3D model, the first loss function is used to narrow the distance between multiple features of the same 3D model and widen the distance between multiple features of different 3D models. Based on the features of the query image and several features of the 3D model, the second loss function and the third loss function are used to narrow the distance between the features of each query image and the corresponding multiple features of the 3D model; The application module is configured to: acquire the query image, the mask image of the query image, and a set of 3D models; input the query image, the mask image of the query image, and the set of 3D models into a fine-tuned 3D model retrieval network to obtain the image features corresponding to the query image and the final features corresponding to each 3D model; calculate the similarity between the image features corresponding to the query image and the final features corresponding to each 3D model, and select the 3D model with the highest similarity value as the final retrieval result output.

8. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-6.

9. A storage medium characterized by being non-transitory. The system stores computer-readable instructions, wherein, when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-6 are executed.

Images