Single-image three-dimensional model retrieval method and system based on feature fusion and view grouping

Abstract

The invention discloses a single-image three-dimensional model retrieval method and system based on feature fusion and view grouping, and the method comprises the steps: taking a known query image, a background-free image and a plurality of views corresponding to each three-dimensional model as input values of a three-dimensional model retrieval network, taking the three-dimensional model most similar to the query image as an output value of a three-dimensional model retrieval network, and training the three-dimensional model retrieval network by adopting a first loss function, a second loss function and a third loss function in sequence to obtain a trained three-dimensional model retrieval network; inputting the to-be-queried image, the mask image of the to-be-queried image and the three-dimensional model set into the fine-tuned three-dimensional model retrieval network to obtain image features corresponding to the to-be-queried image and final features corresponding to each three-dimensional model; and calculating the similarity between the image feature corresponding to the image to be queried and the final feature corresponding to each three-dimensional model, and selecting the three-dimensional model corresponding to the maximum similarity value as a final retrieval result to be output.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional model retrieval, and in particular to a single-image three-dimensional model retrieval method and system based on feature fusion and view grouping. Background Art

[0002] Due to the rapid development of 3D model databases and the easy access to real-world images, 3D model retrieval based on a single image has gradually become a hot topic. Its goal is to retrieve a series of similar 3D models given a query image.

[0003] Three-dimensional model retrieval based on a single image can be divided into instance level and category level according to the retrieval level. In category level retrieval, only the three-dimensional model of the same category as the query image needs to be found, so the accuracy requirement is relatively low. On the other hand, instance level retrieval requires finding the three-dimensional model that is most similar to the query image, so the accuracy is higher. Since most databases have provided instance labels of two-dimensional images and three-dimensional models, and instance level retrieval has higher application value, the present invention selects instance level three-dimensional model retrieval based on a single image as the research direction.

[0004] In the past few years, people have been committed to improving the accuracy of 3D model retrieval based on a single image, and many related works have been proposed. In practical applications, 3D model retrieval based on a single image often has the following problems: first, since the query image is a 2D image taken in the real world, its background is complex and inevitable; second, since 3D models are difficult to obtain, the number of training is limited; finally, since 2D images and 3D models are two different modal data, there is a huge domain gap. These problems will affect the final retrieval effect. Summary of the invention

[0005] In order to address the deficiencies of the prior art, the present invention provides a single-image three-dimensional model retrieval method and system based on feature fusion and view grouping. The method can effectively improve the accuracy of image retrieval of three-dimensional models and solve the problem that affects the retrieval accuracy.

[0006] On the one hand, a single-image 3D model retrieval method based on feature fusion and view grouping is provided, including:

[0007] Acquire a known query image, a mask image corresponding to the known query image, and a set of three-dimensional models; each three-dimensional model in the set of three-dimensional models is provided with a label; the label includes: a three-dimensional model corresponding to the known query image, or a three-dimensional model not corresponding to the known query image;

[0008] Based on the known query image and the mask image corresponding to the known query image, a background-free image is obtained; each three-dimensional model in the three-dimensional model set is preprocessed according to a set method to obtain a plurality of views corresponding to each three-dimensional model;

[0009] The known query image, the background-free image and the several views corresponding to each 3D model are used as input values ​​of the 3D model retrieval network, the 3D model most similar to the query image is used as the output value of the 3D model retrieval network, the 3D model retrieval network is trained using the first, second and third loss functions in sequence to obtain a trained 3D model retrieval network; the trained 3D model retrieval network is fine-tuned to obtain a fine-tuned 3D model retrieval network;

[0010] Obtain a query image, a mask image of the query image, and a set of three-dimensional models; input the query image, the mask image of the query image, and the set of three-dimensional models into a fine-tuned three-dimensional model retrieval network to obtain image features corresponding to the query image and final features corresponding to each three-dimensional model; calculate the similarity between the image features corresponding to the query image and the final features corresponding to each three-dimensional model, and select the three-dimensional model corresponding to the maximum similarity value as the final retrieval result output.

[0011] On the other hand, a single-image 3D model retrieval system based on feature fusion and view grouping is provided, including:

[0012] An acquisition module is configured to: acquire a known query image, a mask image corresponding to the known query image, and a set of three-dimensional models; each three-dimensional model in the set of three-dimensional models is provided with a label; the label includes: a three-dimensional model corresponding to the known query image, or a three-dimensional model not corresponding to the known query image;

[0013] A processing module is configured to: obtain a background-free image based on the known query image and a mask image corresponding to the known query image; pre-process each three-dimensional model in the three-dimensional model set according to a set method to obtain a plurality of views corresponding to each three-dimensional model;

[0014] A training module is configured to: use the known query image, the background-free image, and the several views corresponding to each 3D model as input values ​​of a 3D model retrieval network, use the 3D model most similar to the query image as an output value of the 3D model retrieval network, train the 3D model retrieval network using the first, second, and third loss functions in sequence to obtain a trained 3D model retrieval network; and fine-tune the trained 3D model retrieval network to obtain a fine-tuned 3D model retrieval network;

[0015] The application module is configured to: obtain an image to be queried, a mask image of the image to be queried, and a set of three-dimensional models; input the image to be queried, the mask image of the image to be queried, and the set of three-dimensional models into a fine-tuned three-dimensional model retrieval network to obtain image features corresponding to the image to be queried and final features corresponding to each three-dimensional model; calculate the similarity between the image features corresponding to the image to be queried and the final features corresponding to each three-dimensional model, and select the three-dimensional model corresponding to the maximum similarity value as the final retrieval result output.

[0016] On the other hand, there is also provided an electronic device, comprising:

[0017] a memory for non-transitory storage of computer-readable instructions; and

[0018] a processor for executing the computer readable instructions,

[0019] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0020] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0021] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0022] The above technical solution has the following advantages or beneficial effects:

[0023] The method proposed in the present invention is used to improve the accuracy of 3D model retrieval based on a single image. First, the problem of complex background of the query image is solved by feature fusion, and the important information of the query image is highlighted; secondly, the problem of limited number of models in the training process is solved by view grouping; finally, the distance between different features of the same 3D model and the distance between the image and the 3D model are considered to narrow the domain gap between the two. In addition, the method has a reliable design principle, a simple structure, and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 This is a flow chart of the training phase method of embodiment 1;

[0026] Figure 2This is a flow chart of the testing phase method of Example 1. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0028] Embodiment 1

[0029] This embodiment provides a single-image 3D model retrieval method based on feature fusion and view grouping;

[0030] like Figure 1 and Figure 2 As shown, the single-image 3D model retrieval method based on feature fusion and view grouping includes:

[0031] S101: Acquire a known query image, a mask image corresponding to the known query image, and a set of three-dimensional models; each three-dimensional model in the set of three-dimensional models is provided with a label; the label includes: a three-dimensional model corresponding to the known query image, or a three-dimensional model not corresponding to the known query image;

[0032] S102: obtaining a background-free image based on the known query image and the mask image corresponding to the known query image; preprocessing each 3D model in the 3D model set according to a set method to obtain a plurality of views corresponding to each 3D model;

[0033] S103: using the known query image, the background-free image, and the several views corresponding to each 3D model as input values ​​of the 3D model retrieval network, using the 3D model most similar to the query image as the output value of the 3D model retrieval network, and training the 3D model retrieval network using the first, second, and third loss functions in sequence to obtain a trained 3D model retrieval network; and fine-tuning the trained 3D model retrieval network to obtain a fine-tuned 3D model retrieval network;

[0034] S104: Obtain an image to be queried, a mask image of the image to be queried, and a set of three-dimensional models; input the image to be queried, the mask image of the image to be queried, and the set of three-dimensional models into the fine-tuned three-dimensional model retrieval network to obtain image features corresponding to the image to be queried and final features corresponding to each three-dimensional model; calculate the similarity between the image features corresponding to the image to be queried and the final features corresponding to each three-dimensional model, and select the three-dimensional model corresponding to the maximum similarity value as the final retrieval result output.

[0035] Further, the S101: obtaining a known query image, a mask image corresponding to the known query image, and a set of three-dimensional models;

[0036] The known query image refers to an image used in the training phase, and the image may be an image containing a target object, and the target object includes: a human body, an animal, or an object.

[0037] The mask image corresponding to the known query image is obtained by performing mask processing on the known query image. The mask processing specifically includes: first obtaining the main part of the image through an instance segmentation algorithm, and then processing the main part of the image into a binary image (black and white image) for display.

[0038] The three-dimensional model set includes: a plurality of three-dimensional models.

[0039] Exemplarily, the step S101 of acquiring a known query image, a mask image corresponding to the known query image, and a set of three-dimensional models includes:

[0040] Input query image q, mask image m, 3D model set S = {s 1 ,s 2 ,...,s n}, where s i represents the i-th three-dimensional model. For better explanation, next, we take the i-th three-dimensional model s i Take this as an example for description.

[0041] Furthermore, each three-dimensional model in the three-dimensional model set is provided with a label; the label includes: a three-dimensional model corresponding to a known query image, or a three-dimensional model that does not correspond to the query image; the label includes: a certain animal or a certain object, etc.

[0042] Furthermore, the step S102: obtaining a background-free image based on the known query image and a mask image corresponding to the known query image, includes:

[0043] First, the main part of the image is obtained through the instance segmentation algorithm, the main part of the query image is retained, and the background of the query image is deleted to obtain a background-free image.

[0044] Exemplarily, a background-free image g is obtained using the query image q and the mask image m.

[0045] Furthermore, the S102: preprocessing each 3D model in the 3D model set according to a set method to obtain a plurality of views corresponding to each 3D model, including:

[0046] The three-dimensional model is rendered into a grayscale image from different angles through a flat shading rendering method. The image obtained from each angle is used as a view of the three-dimensional model, and multiple views of the three-dimensional model are obtained from multiple angles.

[0047] Exemplarily, the 3D model is preprocessed to obtain 12 views in, Represents the zth view of the i-th 3D model.

[0048] Further, the S103: using the known query image, the background-free image and the several views corresponding to each 3D model as input values ​​of the 3D model retrieval network, using the 3D model most similar to the query image as the output value of the 3D model retrieval network, and training the 3D model retrieval network using the first, second and third loss functions in sequence to obtain a trained 3D model retrieval network, wherein the 3D model retrieval network includes:

[0049] The first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the attention mechanism layer and the pooling layer;

[0050] Wherein, a known query image is input into a first convolutional neural network to obtain query image features;

[0051] Input the background-free image into the first convolutional neural network to obtain the background-free image features;

[0052] The query image features and the background-free image features are fused by addition operation to obtain fused features;

[0053] Input the fused features into the second convolutional neural network to obtain the final query image features;

[0054] Inputting several views corresponding to each 3D model into a third convolutional neural network to obtain several view features corresponding to each 3D model;

[0055] The final query image features and several view features corresponding to each 3D model are input into the attention mechanism layer. The attention mechanism layer performs a grouping operation on the features. The grouping operation results are input into the pooling layer. The pooling layer outputs several features of each 3D model under the current final query image features.

[0056] According to the features of the three-dimensional model, the distance between multiple features of the same three-dimensional model is shortened by a first loss function, and the distance between multiple features of different three-dimensional models is extended;

[0057] According to the query image features and several features of the three-dimensional model, the distance between each query image feature and the corresponding multiple features of the three-dimensional model is shortened through the second loss function and the third loss function.

[0058] Furthermore, the attention mechanism layer performs grouping operations on features, including:

[0059] Through the attention mechanism, each view is given a weight, and several view features of each 3D model are grouped according to the weight to obtain multiple groups of features: according to the set threshold interval range to which the weight value belongs, several groups of features are obtained, and the features within each interval are a group;

[0060] Alternatively, views with weight values ​​greater than a set threshold are grouped into one group, and views with weight values ​​less than the set threshold are grouped into another group.

[0061] Further, the method of inputting a known query image into a first convolutional neural network to obtain query image features; inputting a background-free image into the first convolutional neural network to obtain background-free image features, and fusing the query image features and the background-free image features to obtain fused features comprises:

[0062] The query image q and the background-free image g are obtained through the convolutional neural network F1 to obtain the query image features and the background-free image features, and the two features are fused through the convolutional neural network F2 to obtain the final query image features f.

[0063] Furthermore, the several views corresponding to each three-dimensional model are input into the third convolutional neural network to obtain several view features corresponding to each three-dimensional model; the final query image features and the several view features corresponding to each three-dimensional model are input into the attention mechanism layer, the attention mechanism layer performs a grouping operation on the features, and the grouping operation results are input into the pooling layer, and the pooling layer outputs several features of each three-dimensional model under the current final query image features, including:

[0064] Extracting features from multiple views of 3D models using convolutional neural network F3 in, Represents the zth view feature of the i-th 3D model. The query image feature f and the i-th 3D model multi-view feature After grouping through the attention mechanism, multiple features of the i-th 3D model about the query image feature f are obtained

[0065] Furthermore, the query image feature f and the i-th 3D model multi-view feature After grouping through the attention mechanism, multiple features of the i-th 3D model about the query image feature f are obtained include:

[0066] Given the query image feature f and the features of multiple views of the i-th 3D model First, the given query image feature f and the multiple view features of the i-th 3D model are combined Through the attention mechanism, the weight of each view with respect to the query image feature f is obtained;

[0067] Secondly, the views are grouped according to the weight of each view, and the views are grouped according to the set threshold interval range to which the weight belongs. The grouped view features are pooled to obtain multiple features of the i-th 3D model about the query image feature f

[0068] Furthermore, the method of shortening the distance between multiple features of the same three-dimensional model and pushing the distance between multiple features of different three-dimensional models by using a first loss function according to the multiple features of the three-dimensional model includes:

[0069] Given multiple features of all three-dimensional models, by continuously optimizing the first loss function L1, the distance between multiple features of the same three-dimensional model is continuously shortened, and the distance between multiple features of different three-dimensional models is pushed away:

[0070]

[0071] Among them, the distance between two features is calculated by the inner product, and one of the features of the first 3D model is selected as the anchor point. represents the distance between the anchor feature of the first 3D model and the remaining features of the first 3D model, m represents the number of multiple features of the same 3D model, represents the distance between the feature and multiple features of other 3D models, k represents the number of multiple features of all 3D models, Represents the distance between the features selected by the nth model and the remaining features of the nth model, Represents the distance between the selected feature of the nth model and multiple features of other 3D models.

[0072] Furthermore, the method of shortening the distance between each query image feature and the corresponding multiple features of the three-dimensional model by using a second loss function and a third loss function according to the query image feature and the multiple features of the three-dimensional model includes:

[0073] Given the query image features and multiple features of all 3D models, at the instance level, find the 3D model corresponding to the query image based on the given query image features and all 3D model features, and continuously optimize the second loss function L2 to continuously narrow the distance between the query image and the corresponding 3D model and push the distance between the query image and other 3D models away:

[0074]

[0075] Among them, d 1 1,...,d ′ m Represents the distance between the query image and the corresponding 3D model features, d ′ 1,...,d ′ k Represents the distance between multiple features of the query image and other 3D models; the distance between features is calculated by the inner product of two features. Each image has a 3D model corresponding to a label, and other 3D models refer to all models except the 3D model corresponding to the corresponding label.

[0076] At the category level, based on the given query image features and all 3D model features, we find query images and 3D models of the same category, and continuously optimize the third loss function to continuously narrow the distance between them and push the distance between query images and 3D models of different categories further apart:

[0077]

[0078] Among them, d ′ 1,d ′ 2...,d ′ p represents the distance between the query image and multiple features of the 3D model of the same category as the query image, p represents the number of multiple features of the 3D model of the same category as the query image, d ′ 1,d ′ 2...,d ′ k Represents the distance between a query image and multiple features of the 3D model that do not belong to the same category as the query image. The distance between features is calculated by the inner product of two features.

[0079] Furthermore, the step S103: fine-tuning the trained three-dimensional model retrieval network to obtain a fine-tuned three-dimensional model retrieval network includes:

[0080] The attention mechanism layer is removed, and the output of the third convolutional neural network is directly connected to the input of the pooling layer.

[0081] Furthermore, the step S104: obtaining a query image, a mask image of the query image, and a set of three-dimensional models; inputting the query image, the mask image of the query image, and the set of three-dimensional models into the fine-tuned three-dimensional model retrieval network to obtain image features corresponding to the query image and final features corresponding to each three-dimensional model, including:

[0082] Inputting the query image into the first convolutional neural network to obtain the query image features;

[0083] Input the background-free image into the first convolutional neural network to obtain the background-free image features;

[0084] Fusing the features of the image to be queried and the features of the image without background to obtain fused features;

[0085] Input the fused features into the second convolutional neural network to obtain the final query image features;

[0086] Inputting several views corresponding to each 3D model into a third convolutional neural network to obtain several view features corresponding to each 3D model;

[0087] The final query image features and several view features corresponding to each 3D model are input into the pooling layer, and the pooling layer outputs several features of each 3D model under the current final query image features.

[0088] Furthermore, the similarity between the image feature corresponding to the query image and the final feature corresponding to each three-dimensional model is calculated, and the three-dimensional model corresponding to the maximum similarity value is selected as the final retrieval result output, and the similarity is calculated using cosine similarity.

[0089] This method first fuses the features of the query image and the background-free image to highlight the key information in the image and reduce the interference of the complex background; secondly, it generates multiple feature representations of the same model through view grouping, thereby increasing the number of model training; finally, it reduces the domain gap between the two by narrowing the distance between different feature representations of the same model and narrowing the distance between the image and the 3D model at the instance level and class level. This method can help image retrieval more correct 3D models and solve the problem of affecting retrieval accuracy.

[0090] This method focuses on feature fusion and view grouping, narrowing the distance between images and 3D models, and improving the accuracy of retrieval of 3D models from a single image. First, by introducing background-free images and query images for feature fusion, the key information in the query image can be effectively highlighted and background interference can be reduced; second, by obtaining multiple feature representations through view grouping, the number of training models can be increased to a certain extent; finally, by narrowing the distance between multiple feature representations of the same 3D model and shortening the distance between images and 3D models at the category and instance levels, the accuracy and reliability of the results can be significantly improved, thereby further improving the retrieval accuracy.

[0091] Embodiment 2

[0092] This embodiment provides a single-image 3D model retrieval system based on feature fusion and view grouping, including:

[0093] An acquisition module is configured to: acquire a known query image, a mask image corresponding to the known query image, and a set of three-dimensional models; each three-dimensional model in the set of three-dimensional models is provided with a label; the label includes: a three-dimensional model corresponding to the known query image, or a three-dimensional model not corresponding to the known query image;

[0094] A processing module is configured to: obtain a background-free image based on the known query image and a mask image corresponding to the known query image; pre-process each three-dimensional model in the three-dimensional model set according to a set method to obtain a plurality of views corresponding to each three-dimensional model;

[0095] A training module is configured to: use the known query image, the background-free image, and the several views corresponding to each 3D model as input values ​​of a 3D model retrieval network, use the 3D model most similar to the query image as an output value of the 3D model retrieval network, train the 3D model retrieval network using the first, second, and third loss functions in sequence to obtain a trained 3D model retrieval network; and fine-tune the trained 3D model retrieval network to obtain a fine-tuned 3D model retrieval network;

[0096] The application module is configured to: obtain an image to be queried, a mask image of the image to be queried, and a set of three-dimensional models; input the image to be queried, the mask image of the image to be queried, and the set of three-dimensional models into a fine-tuned three-dimensional model retrieval network to obtain image features corresponding to the image to be queried and final features corresponding to each three-dimensional model; calculate the similarity between the image features corresponding to the image to be queried and the final features corresponding to each three-dimensional model, and select the three-dimensional model corresponding to the maximum similarity value as the final retrieval result output.

[0097] It should be noted that the acquisition module, processing module, training module and application module described above correspond to steps S101 to S104 in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules described above as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0098] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0100] Embodiment 3

[0101] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

[0102] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0103] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0104] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0105] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0106] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0107] Embodiment 4: This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in Embodiment 1 is completed.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A single-image 3D model retrieval method based on feature fusion and view grouping, characterized in that: include: Obtaining a known query image, a mask image corresponding to the known query image, and a set of three-dimensional models; Each three-dimensional model in the three-dimensional model set is provided with a label; the label includes: a three-dimensional model corresponding to a known query image, or a three-dimensional model not corresponding to a known query image; Based on the known query image and the mask image corresponding to the known query image, a background-free image is obtained; each three-dimensional model in the three-dimensional model set is preprocessed according to a set method to obtain a plurality of views corresponding to each three-dimensional model; The known query image, the background-free image and the several views corresponding to each 3D model are used as input values ​​of the 3D model retrieval network, the 3D model most similar to the query image is used as the output value of the 3D model retrieval network, the 3D model retrieval network is trained using the first, second and third loss functions in sequence to obtain a trained 3D model retrieval network; the trained 3D model retrieval network is fine-tuned to obtain a fine-tuned 3D model retrieval network; Obtain a query image, a mask image of the query image, and a set of three-dimensional models; input the query image, the mask image of the query image, and the set of three-dimensional models into a fine-tuned three-dimensional model retrieval network to obtain image features corresponding to the query image and final features corresponding to each three-dimensional model; calculate the similarity between the image features corresponding to the query image and the final features corresponding to each three-dimensional model, and select the three-dimensional model corresponding to the maximum similarity value as the final retrieval result output.

2. The single-image 3D model retrieval method based on feature fusion and view grouping as claimed in claim 1, characterized in that: The known query image, the background-free image and the several views corresponding to each 3D model are used as input values ​​of the 3D model retrieval network, the 3D model most similar to the query image is used as the output value of the 3D model retrieval network, and the 3D model retrieval network is trained using the first, second and third loss functions in sequence to obtain a trained 3D model retrieval network, wherein the 3D model retrieval network includes: The first convolutional neural network, the second convolutional neural network, the third convolutional neural network, the attention mechanism layer and the pooling layer; Wherein, a known query image is input into a first convolutional neural network to obtain query image features; Input the background-free image into the first convolutional neural network to obtain the background-free image features; The query image features and the background-free image features are fused by addition operation to obtain fused features; Input the fused features into the second convolutional neural network to obtain the final query image features; Inputting several views corresponding to each 3D model into a third convolutional neural network to obtain several view features corresponding to each 3D model; The final query image features and several view features corresponding to each 3D model are input into the attention mechanism layer. The attention mechanism layer performs a grouping operation on the features. The grouping operation results are input into the pooling layer. The pooling layer outputs several features of each 3D model under the current final query image features. According to the features of the three-dimensional model, the distance between multiple features of the same three-dimensional model is shortened by a first loss function, and the distance between multiple features of different three-dimensional models is extended; According to the query image features and several features of the three-dimensional model, the distance between each query image feature and the corresponding multiple features of the three-dimensional model is shortened through the second loss function and the third loss function.

3. The single-image 3D model retrieval method based on feature fusion and view grouping as claimed in claim 2, characterized in that: The attention mechanism layer performs grouping operations on features, including: Through the attention mechanism, each view is given a weight, and several view features of each 3D model are grouped according to the weight to obtain multiple groups of features: according to the set threshold interval range to which the weight value belongs, several groups of features are obtained, and the features within each interval are a group; Alternatively, views with weight values ​​greater than a set threshold are grouped into one group, and views with weight values ​​less than the set threshold are grouped into another group.

4. The single-image 3D model retrieval method based on feature fusion and view grouping as claimed in claim 2, characterized in that: The method of shortening the distance between multiple features of the same three-dimensional model and pushing the distance between multiple features of different three-dimensional models by using a first loss function according to multiple features of the three-dimensional model includes: Given multiple features of all three-dimensional models, by continuously optimizing the first loss function L1, the distance between multiple features of the same three-dimensional model is continuously shortened, and the distance between multiple features of different three-dimensional models is further increased: Among them, the distance between two features is calculated by the inner product, and one of the features of the first 3D model is selected as the anchor point. represents the distance between the anchor feature of the first 3D model and the remaining features of the first 3D model, m represents the number of multiple features of the same 3D model, represents the distance between the feature and multiple features of other 3D models, k represents the number of multiple features of all 3D models, Represents the distance between the features selected by the nth model and the remaining features of the nth model, Represents the distance between the selected feature of the nth model and multiple features of other 3D models.

5. The single-image 3D model retrieval method based on feature fusion and view grouping as claimed in claim 2, characterized in that: The method of shortening the distance between each query image feature and the corresponding multiple features of the three-dimensional model by using a second loss function and a third loss function according to the query image feature and the multiple features of the three-dimensional model includes: Given the query image features and multiple features of all 3D models, at the instance level, find the 3D model corresponding to the query image based on the given query image features and all 3D model features, and continuously optimize the second loss function L2 to continuously narrow the distance between the query image and the corresponding 3D model and push the distance between the query image and other 3D models away: Among them, d 1 1,...,d ′ m Represents the distance between the query image and the corresponding 3D model features, d ′ 1,...,d ′ k Represents the distance between multiple features of the query image and other 3D models; the distance between features is calculated by the inner product of two features.

6. The single-image 3D model retrieval method based on feature fusion and view grouping as claimed in claim 5, characterized in that: At the category level, based on the given query image features and all 3D model features, we find query images and 3D models of the same category, and continuously optimize the third loss function to continuously narrow the distance between them and push the distance between query images and 3D models of different categories further apart: Among them, d ′ 1,d ′ 2...,d ′ p represents the distance between the query image and multiple features of the 3D model of the same category as the query image, p represents the number of multiple features of the 3D model of the same category as the query image, d ′ 1,d ′ 2...,d ′ k Represents the distance between a query image and multiple features of a 3D model that do not belong to the same category as the query image.

7. The single-image 3D model retrieval method based on feature fusion and view grouping as claimed in claim 1, characterized in that: Obtain a query image, a mask image of the query image, and a set of 3D models; input the query image, the mask image of the query image, and the set of 3D models into the fine-tuned 3D model retrieval network to obtain image features corresponding to the query image and final features corresponding to each 3D model, including: Inputting the query image into the first convolutional neural network to obtain the query image features; Input the background-free image into the first convolutional neural network to obtain the background-free image features; Fusing the features of the image to be queried and the features of the image without background to obtain fused features; Input the fused features into the second convolutional neural network to obtain the final query image features; Inputting several views corresponding to each 3D model into a third convolutional neural network to obtain several view features corresponding to each 3D model; The final query image features and several view features corresponding to each 3D model are input into the pooling layer, and the pooling layer outputs several features of each 3D model under the current final query image features.

8. A single-image 3D model retrieval system based on feature fusion and view grouping, characterized by: include: An acquisition module is configured to: acquire a known query image, a mask image corresponding to the known query image, and a set of three-dimensional models; Each three-dimensional model in the three-dimensional model set is provided with a label; the label includes: a three-dimensional model corresponding to a known query image, or a three-dimensional model not corresponding to a known query image; A processing module is configured to: obtain a background-free image based on the known query image and a mask image corresponding to the known query image; pre-process each three-dimensional model in the three-dimensional model set according to a set method to obtain a plurality of views corresponding to each three-dimensional model; A training module is configured to: use the known query image, the background-free image, and the several views corresponding to each 3D model as input values ​​of a 3D model retrieval network, use the 3D model most similar to the query image as an output value of the 3D model retrieval network, train the 3D model retrieval network using the first, second, and third loss functions in sequence to obtain a trained 3D model retrieval network; and fine-tune the trained 3D model retrieval network to obtain a fine-tuned 3D model retrieval network; The application module is configured to: obtain an image to be queried, a mask image of the image to be queried, and a set of three-dimensional models; input the image to be queried, the mask image of the image to be queried, and the set of three-dimensional models into a fine-tuned three-dimensional model retrieval network to obtain image features corresponding to the image to be queried and final features corresponding to each three-dimensional model; calculate the similarity between the image features corresponding to the image to be queried and the final features corresponding to each three-dimensional model, and select the three-dimensional model corresponding to the maximum similarity value as the final retrieval result output.

9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 7 is executed.

10. A storage medium, characterized in that: The computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1 to 7 are executed.

Images