An image retrieval method and device, and a storage medium

Abstract

The embodiment of the application discloses an image retrieval method and device, and a storage medium, comprising: in the case of obtaining a two-dimensional image and a three-dimensional image of a sample object, extracting a two-dimensional image feature from the two-dimensional image and extracting a three-dimensional image feature from the three-dimensional image; performing metric learning on the two-dimensional image feature and the three-dimensional image feature to obtain a corresponding relationship between the two-dimensional image feature and the three-dimensional image feature; in the case of receiving a target two-dimensional image of a target object, obtaining a target two-dimensional image feature of the target two-dimensional image; and determining a target three-dimensional image corresponding to the target object according to the target two-dimensional image feature and the corresponding relationship between the two-dimensional image feature and the three-dimensional image feature.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image retrieval method, apparatus, and storage medium. Background Technology

[0002] In recent years, major e-commerce platforms have introduced features such as image-based and text-based image search. They are also actively exploring how to effectively use text or two-dimensional images to search for three-dimensional objects and present the three-dimensional data to users in a visually appealing way.

[0003] In existing technologies, the main approach is to match two-dimensional images with three-dimensional objects to construct multiple triples, then input these triples into a constructed three-branch neural network, and finally train it using Triplet Loss. Retrieving three-dimensional objects from two-dimensional images requires constructing triples based on both the two-dimensional image and the three-dimensional object. This input of triples significantly increases the computational load and complexity of the algorithm, prolonging the retrieval time for three-dimensional images from two-dimensional images, thus reducing the speed of image retrieval. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application aim to provide an image retrieval method, apparatus, and storage medium that can improve the speed of image retrieval.

[0005] The technical solution of this application is implemented as follows:

[0006] This application provides an image retrieval method, which includes:

[0007] Given two-dimensional and three-dimensional images of the sample object, extract two-dimensional image features from the two-dimensional image and extract three-dimensional image features from the three-dimensional image;

[0008] The two-dimensional image features and the three-dimensional image features are subjected to metric learning to obtain the correspondence between the two-dimensional image features and the three-dimensional image features;

[0009] Upon receiving a target two-dimensional image of the target object, the target two-dimensional image features of the target two-dimensional image are obtained;

[0010] Based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features, the target three-dimensional image corresponding to the target object is determined.

[0011] This application provides an image retrieval device, the device comprising:

[0012] The extraction unit is used to extract two-dimensional image features from the two-dimensional image and three-dimensional image features from the three-dimensional image when two-dimensional and three-dimensional images of the sample object are obtained.

[0013] The learning unit is used to perform metric learning on the two-dimensional image features and the three-dimensional image features to obtain the correspondence between the two-dimensional image features and the three-dimensional image features;

[0014] The acquisition unit is used to acquire the target two-dimensional image features of the target two-dimensional image when a target two-dimensional image of the target object is received;

[0015] The determining unit is used to determine the target three-dimensional image corresponding to the target object based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features.

[0016] This application provides an image retrieval device, the device comprising:

[0017] The system includes a memory, a processor, and a communication bus. The memory communicates with the processor via the communication bus. The memory stores an image retrieval program that can be executed by the processor. When the image retrieval program is executed, the image retrieval method described above is performed by the processor.

[0018] This application provides a storage medium storing a computer program for use in an image retrieval device, characterized in that the computer program, when executed by a processor, implements the image retrieval method described above.

[0019] This application provides an image retrieval method, apparatus, and storage medium. The image retrieval method includes: when a two-dimensional image and a three-dimensional image of a sample object are obtained, extracting two-dimensional image features from the two-dimensional image and extracting three-dimensional image features from the three-dimensional image; performing metric learning on the two-dimensional and three-dimensional image features to obtain the correspondence between the two-dimensional and three-dimensional image features; when a target two-dimensional image of a target object is received, obtaining the target two-dimensional image features of the target two-dimensional image; and determining the target three-dimensional image corresponding to the target object based on the target two-dimensional image features and the correspondence between the two-dimensional and three-dimensional image features. By employing the above method, the image retrieval apparatus, through metric learning on the two-dimensional and three-dimensional image features, eliminates the need to construct matching triples between the two-dimensional and three-dimensional images, reducing the computational load during metric learning, decreasing the retrieval time when retrieving the target object's three-dimensional image using the target object's two-dimensional image, and improving the image retrieval speed. Attached Figure Description

[0020] Figure 1A flowchart of an image retrieval method provided in an embodiment of this application;

[0021] Figure 2 This is an exemplary schematic diagram of determining a three-dimensional image from a two-dimensional image, provided as an embodiment of this application.

[0022] Figure 3 An exemplary image retrieval diagram provided for an embodiment of this application;

[0023] Figure 4 A schematic diagram of the composition structure of an image retrieval device provided in this application embodiment. Figure 1 ;

[0024] Figure 5 A schematic diagram of the composition structure of an image retrieval device provided in this application embodiment. Figure 2 . Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0026] This application provides an image retrieval method, which is applied to an image retrieval device. Figure 1 A flowchart of an image retrieval method provided in this application embodiment is shown below. Figure 1 As shown, image retrieval methods may include:

[0027] S101. Given two-dimensional and three-dimensional images of the sample object, extract two-dimensional image features from the two-dimensional image and extract three-dimensional image features from the three-dimensional image.

[0028] The image retrieval method provided in this application embodiment is applicable to scenarios where a two-dimensional image of a target object is used to retrieve a three-dimensional image corresponding to the target object.

[0029] In the embodiments of this application, the image retrieval device can be implemented in various forms. For example, the image retrieval device described in this application may include devices such as mobile phones, cameras, tablet computers, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as devices such as digital TVs, desktop computers, servers, etc.

[0030] In the embodiments of this application, the three-dimensional image can be an image obtained from multiple two-dimensional images from different angles.

[0031] For example, such as Figure 2 As shown, the sample object can be a chair. The sample object is placed at position (0,0,0) in the three-dimensional coordinate system. A virtual camera is set at every 30° angle with the horizontal plane, for a total of 12 virtual cameras to capture multiple views of the sample object, thereby obtaining the corresponding three-dimensional image of the sample object.

[0032] In this embodiment of the application, the sample object can be a chair, an airplane, a table, or other objects; the specific sample object can be determined according to the actual situation, and this embodiment of the application does not limit it.

[0033] In this embodiment, the process of the image retrieval device extracting two-dimensional image features from a two-dimensional image and three-dimensional image features from a three-dimensional image can be as follows: the image retrieval device inputs the two-dimensional image into a Convolutional Neural Network (CNN) to obtain two-dimensional image features, and the image retrieval device inputs the three-dimensional image into the CNN to obtain three-dimensional image features. Alternatively, the image retrieval device can extract two-dimensional image features from a two-dimensional image and three-dimensional image features from a three-dimensional image in other ways. The specific method by which the image retrieval device extracts two-dimensional image features from a two-dimensional image and three-dimensional image features from a three-dimensional image can be determined according to the actual situation, and this embodiment does not limit this.

[0034] In this embodiment, the number of sample objects can be multiple, and the number of two-dimensional images corresponding to multiple sample objects is also multiple, as is the number of three-dimensional images corresponding to multiple sample objects. When multiple two-dimensional images and multiple three-dimensional images are obtained, the image retrieval device will perform positive and negative sample matching on the multiple two-dimensional images and multiple three-dimensional images to determine the two-dimensional image and three-dimensional image corresponding to the same sample object. Specifically, the image retrieval device numbers the two-dimensional images according to the rule of sample object category name - number of samples, and numbers the three-dimensional data according to the rule of sample object category name - number of samples - number of views. Specifically, the two-dimensional image dataset is as shown in formula (1):

[0035] S1={x 1-1 ,x 1-2 ,x 1-3 ,...,x 1-m ,x 2-1 ,x 2-2 ,...,x 2-m ,...,x n-1 ,x n-2 ,...,x n-m} (1)

[0036] There are n major categories, and each major category contains m minor categories. For example, the major categories include objects such as airplanes, tanks, cannons, tables, and chairs; the minor categories of airplanes include different models of airplanes such as fighter jets and civilian airplanes.

[0037] In this embodiment of the application, the three-dimensional dataset is as shown in formula (2):

[0038] S2={y i-k-1 ,y i-k-2 ,...,y i-k-12 ,y i-h-1 ,y i-h-2 ,...,y i-h-12 ,...,y i-f-1 ,y i-f-2 ,...,y i-f-12} (2)

[0039] In this embodiment of the application, each two-dimensional image is traversed, and for each two-dimensional image x i-j (x i-j (Refers to the j-th sample of the i-th class). Five sample objects of the same class are randomly selected and labeled using random numbers. The result is shown in formula (3):

[0040] {y i-k-1 ,y i-k-2 ,...,y i-k-12 ,y i-h-1 ,y i-h-2 ,...,y i-h-12 ,...,y i-f-1 ,y i-f-2 ,...,y i-f-12} (3)

[0041] It should be noted that i represents the major category and k represents the minor category; there are a total of 5 categories: k, h, ..., f.

[0042] In this embodiment of the application, the labeling results of 10 different classes of sample objects are shown in formula (4):

[0043] {y u-g-1 ,y u-g-2 ,...,y u-g-12 ,y p-d-1 ,y p-d-2 ,...,y p-d-12 ,...,y b-t-1 ,y b-t-2 ,...,y b-t-12}(4)

[0044] In this embodiment of the application, the selected matching result is shown in formula (5):

[0045] {x i-j ,y i-k-1 ,y i-k-2 ,...,y i-k-12}, {x i-j ,y i-h-1 ,y i-h-2 ,...,y i-h-12},...,{x i-j ,y b-t-1 ,y b-t-2 ,...,y b-t-12} (5)

[0046] In the embodiments of this application, each sample is assigned a number from 1 to N during random selection, and then a random number from 1 to N is randomly generated for random selection.

[0047] S102. Perform metric learning on the two-dimensional image features and the three-dimensional image features to obtain the correspondence between the two-dimensional image features and the three-dimensional image features.

[0048] In this embodiment of the application, after the image retrieval device extracts two-dimensional image features from a two-dimensional image and three-dimensional image features from a three-dimensional image, the image retrieval device can perform metric learning on the two-dimensional image features and the three-dimensional image features to obtain the correspondence between the two-dimensional image features and the three-dimensional image features.

[0049] In this embodiment of the application, the process by which the image retrieval device performs metric learning on two-dimensional image features and three-dimensional image features to obtain the correspondence between the two-dimensional image features and the three-dimensional image features includes: the image retrieval device performing interference removal processing on the two-dimensional image features to obtain a first feature; the image retrieval device performing retrieval capability evaluation processing on the three-dimensional image features to obtain a second feature; and the image retrieval device performing metric learning on the first feature and the second feature to obtain the correspondence between the two-dimensional image features and the three-dimensional image features.

[0050] In this embodiment, the process by which the image retrieval device performs interference removal processing on two-dimensional image features to obtain the first feature can be as follows: the image retrieval device assigns weights to different positions in the two-dimensional image, assigning low weights to the background and high weights to the sample object portion of the image, thereby reducing the interference of the background portion in the two-dimensional image and obtaining the first feature; or the image retrieval device can perform interference removal processing on the two-dimensional image features in other ways to obtain the first feature. The specific method by which the image retrieval device performs interference removal processing on the two-dimensional image features to obtain the first feature can be determined according to the actual situation, and this embodiment does not limit this.

[0051] In the embodiments of this application, the process by which the image retrieval device evaluates the retrieval capability of three-dimensional image features to obtain a second feature can involve the image retrieval device determining factors such as the shooting angle and illumination intensity of the three-dimensional image, and determining the retrieval capability of the three-dimensional image based on these factors. The second feature is then determined based on the retrieval capability and the three-dimensional image features. Alternatively, the image retrieval device can perform retrieval capability evaluation processing on the three-dimensional image features in other ways to obtain the second feature. The specific method by which the image retrieval device performs retrieval capability evaluation processing on the three-dimensional image features to obtain the second feature can be determined according to the actual situation, and this application embodiment does not limit this.

[0052] In this embodiment of the application, the process by which the image retrieval device performs retrieval capability evaluation processing on three-dimensional image features to obtain a second feature includes: the image retrieval device inputting the three-dimensional image features into the retrieval capability evaluation module to obtain a retrieval capability evaluation value for the three-dimensional image; and the image retrieval device fusing the three-dimensional image features and the retrieval capability evaluation value to obtain a second feature.

[0053] In this embodiment, the process by which the image retrieval device fuses three-dimensional image features and retrieval capability evaluation values ​​to obtain a second feature can be as follows: the image retrieval device multiplies the three-dimensional image features and the retrieval capability evaluation values ​​and then sums them to obtain a fused value, i.e., the second feature; or the image retrieval device can fuse the three-dimensional image features and the retrieval capability evaluation values ​​in other ways to obtain the second feature. The specific method by which the image retrieval device fuses the three-dimensional image features and the retrieval capability evaluation values ​​to obtain the second feature can be determined according to the actual situation, and this embodiment does not limit this.

[0054] In this embodiment of the application, the process by which the image retrieval device fuses three-dimensional image features and retrieval capability evaluation values ​​to obtain a second feature can be as follows: the image retrieval device fuses three-dimensional image features and retrieval capability evaluation values ​​to obtain a first fused feature, and the image retrieval device inputs the first fused feature into a fully connected layer to obtain the second feature.

[0055] In this embodiment of the application, the process by which the image retrieval device inputs three-dimensional image features into the retrieval capability evaluation module to obtain a retrieval capability evaluation value for the three-dimensional image includes: the image retrieval device performing dimensionality reduction processing on the three-dimensional image features to obtain dimensionality-reduced three-dimensional image features; and the image retrieval device inputting the dimensionality-reduced three-dimensional image features into the retrieval capability evaluation module to obtain a retrieval capability evaluation value.

[0056] In this embodiment, the retrieval capability evaluation module can be a Sigmoid activation function, used to obtain a set of influence factors between 0 and 1 based on the three-dimensional image features. The retrieval capability evaluation module can also be other modules used to obtain a set of influence factors between 0 and 1 based on the three-dimensional image features. The specific retrieval capability evaluation module can be determined according to the actual situation, and this embodiment does not limit it.

[0057] In the embodiments of this application, the process of the image retrieval device performing dimensionality reduction processing on the three-dimensional image features to obtain the dimensionality-reduced three-dimensional image features can be that the image retrieval device reduces the feature data dimension of the three-dimensional image features, or it can reduce the channel dimension of the three-dimensional image features; it can also reduce both the channel dimension and the feature data dimension of the three-dimensional image features. The specific method can be determined according to the actual situation, and the embodiments of this application do not limit this.

[0058] For example, the image retrieval device merges the multi-view features (3D image features) extracted after CNN processing (dimension Batch×12×56×56×3) and inputs them into a 1X1 convolutional layer to reduce the feature data dimensionality, reducing the channel dimension from 3 to 1. The dimension becomes Batch×12×56×56×1. Then, the image retrieval device inputs the features output from the 1X1 convolutional layer into a fully connected layer (the number of fully connected layers can be 2 or 3; the number of fully connected layers can also be other values; the specific number can be determined according to the actual situation, and this application embodiment does not limit this). The feature dimension changes from Batch×12×3136 to Batch×12×1, thus obtaining the dimensionality-reduced 3D image features. Then, the image retrieval device takes the absolute value of the dimensionality-reduced 3D image features and inputs it into a Sigmoid activation function (retrieval capability evaluation module) to obtain a set of influence factors (retrieval capability evaluation values) with values ​​between 0 and 1, used to characterize the role of each view in retrieval. Finally, the image retrieval device multiplies and sums the influencing factors with the features corresponding to the view (3D image features) to obtain the second feature, and inputs the second feature into the loss function.

[0059] In this embodiment of the application, the process by which the image retrieval device performs interference removal processing on two-dimensional image features to obtain a first feature includes: the image retrieval device performing dimensionality reduction processing on the two-dimensional image features to obtain dimensionality-reduced two-dimensional image features; the image retrieval device performing feature normalization processing on the dimensionality-reduced two-dimensional image features to obtain normalized features; and the image retrieval device determining the first feature based on the two-dimensional image features and the normalized features.

[0060] In the embodiments of this application, the process of the image retrieval device performing dimensionality reduction processing on two-dimensional image features to obtain dimensionality-reduced two-dimensional image features can be that the image retrieval device reduces the feature data dimension of the two-dimensional image features, or it can reduce the channel dimension of the two-dimensional image features; it can also reduce both the channel dimension and the feature data dimension of the two-dimensional image features. The specific method can be determined according to the actual situation, and the embodiments of this application do not limit this.

[0061] In this embodiment of the application, the process by which the image retrieval device obtains a first feature based on two-dimensional image features and normalized features includes the image retrieval device multiplying and adding the normalized features with the two-dimensional image features, and then inputting the sum into the subsequent fully connected layer to obtain the first feature.

[0062] In this embodiment, after the two-dimensional image is processed by a CNN network, it outputs two-dimensional image features, and the feature dimension changes from 3×224×224 (channels × width × height) to 3×64×64. The two-dimensional image features are then processed by a 1×1 convolution to reduce the number of channels to 1, and the feature dimension becomes 1×64×64 (the feature dimension corresponding to the two-dimensional image features after dimensionality reduction). A normalization function is used to normalize the feature value range of the dimensionality-reduced two-dimensional image features to between 0 and 1. The specific calculation formula is shown in formula (6):

[0063] θ=sigmoid(log(abs(α))) (6)

[0064] Where α represents the dimensionality-reduced two-dimensional image features, and θ represents the normalized features.

[0065] In this embodiment, the normalized features are multiplied and added to the two-dimensional image features to obtain the second fused feature, which is then input into the fully connected layer to obtain the first feature. The specific calculation formula is shown in formula (7):

[0066] out=F+F*θ (7)

[0067] Where F represents the two-dimensional image features output after CNN processing, and out represents the first feature.

[0068] It is understandable that by performing dimensionality reduction on the features of a two-dimensional image, and then performing feature normalization on the dimensionality-reduced two-dimensional image features, the influence of complex backgrounds on the image retrieval process is weakened, thereby enhancing the retrieval capability of the target object.

[0069] In this embodiment of the application, the process of metric learning for two-dimensional image features and three-dimensional image features is as shown in formula (8):

[0070]

[0071] Where y represents whether the sample objects are similar; y = 1 if the sample objects are similar, and y = 0 if the sample objects are dissimilar. N is the number of sample objects, and margin is the threshold of the loss function, which is a constant. The calculation method of d in formula (8) is shown in formula (9):

[0072] d=||q1-q2||2 (9)

[0073] It should be noted that q1 and q2 are the features of two samples (i.e., two-dimensional image features and three-dimensional image features). The Loss(L) in formula (8) can be used to widen the distance between samples of different classes and narrow the distance between samples of the same class.

[0074] S103. Upon receiving a target two-dimensional image of the target object, acquire the target two-dimensional image features of the target two-dimensional image.

[0075] In this embodiment of the application, the image retrieval device performs metric learning on two-dimensional image features and three-dimensional image features to obtain the correspondence between the two-dimensional image features and the three-dimensional image features. When a target two-dimensional image of a target object is received, the image retrieval device can obtain the target two-dimensional image features of the target two-dimensional image.

[0076] In this embodiment of the application, the target object may be the same as the sample object or may be different from the sample object. The specific target object may be determined according to the actual situation, and this embodiment of the application does not limit it.

[0077] In this embodiment, the image retrieval device can obtain the target two-dimensional image features of the target two-dimensional image in the following ways: the image retrieval device inputs the target two-dimensional image into a CNN network to obtain the target two-dimensional image features; or the image retrieval device can extract the target two-dimensional image features from the target two-dimensional image in other ways. The specific way of extracting the target two-dimensional image features from the target two-dimensional image can be determined according to the actual situation, and this embodiment does not limit it.

[0078] S104. Based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features, determine the target three-dimensional image corresponding to the target object.

[0079] In this embodiment of the application, after the image retrieval device obtains the target two-dimensional image features of the target two-dimensional image, the image retrieval device can determine the target three-dimensional image corresponding to the target object based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features.

[0080] In this embodiment of the application, the process by which the image retrieval device determines the target three-dimensional image corresponding to the target object based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features includes: the image retrieval device obtaining the target three-dimensional image features corresponding to the target object based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features; and the image retrieval device acquiring the target three-dimensional image based on the target three-dimensional image features.

[0081] In this embodiment, the process by which the image retrieval device obtains the target three-dimensional image features corresponding to the target object based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features can be as follows: the image retrieval device performs dimensionality reduction processing on the target two-dimensional image features to obtain dimensionality-reduced target two-dimensional image features; the image retrieval device performs feature normalization processing on the dimensionality-reduced target two-dimensional image features to obtain normalized target features; a first target feature is obtained based on the target two-dimensional image features and the normalized target features; and the image retrieval device determines the target three-dimensional image features corresponding to the first target features based on the correspondence between the two-dimensional image features and the three-dimensional image features.

[0082] For example, such as Figure 3As shown: The image processing device can place the sample object at position (0,0,0) in a three-dimensional coordinate system. A virtual camera is set at preset angles at certain points relative to the horizontal plane. Multiple virtual cameras are used to capture multiple views of the sample object (multi-view projection rendering), thus obtaining the corresponding three-dimensional image of the sample object. After acquiring the two-dimensional and three-dimensional images of the sample object, the image retrieval device performs positive and negative sample matching (positive and negative sample matching algorithm) to determine the two-dimensional and three-dimensional images corresponding to the same sample object. Then, the image retrieval device inputs the two-dimensional image into a CNN to obtain two-dimensional image features, and the image retrieval device inputs the three-dimensional image into a CNN to obtain three-dimensional image features. The two-dimensional image features are then subjected to dimensionality reduction processing (1×1 convolution) to obtain dimensionality-reduced two-dimensional image features. The dimensionality-reduced two-dimensional image features are then normalized (using the sigmoid function) to obtain normalized features. Finally, the two-dimensional image features and the normalized features are... The image retrieval device performs fusion of features to obtain a second fused feature, which is then input into a fully connected layer (FC) to obtain a first feature. The 3D image features are then dimensionality-reduced to obtain dimensionality-reduced 3D image features. These dimensionality-reduced features are input into a retrieval capability evaluation module to obtain a retrieval capability evaluation value (view influence calculation, yielding an influence factor). The 3D image features and the retrieval capability evaluation value are fused to obtain a first fused feature, which is then input into the fully connected layer (FC) to obtain a second feature. A metric learning (Contrastive Loss) is performed on the first and second features to obtain the correspondence between the 2D and 3D image features. When a target 2D image of the target object is received, the target 2D image features of the target 2D image are obtained. Based on the target 2D image features and the correspondence between the 2D and 3D image features, the target 3D image features corresponding to the target object are obtained. Finally, the target 3D image is obtained based on the target 3D image features.

[0083] Understandably, image retrieval devices perform metric learning on two-dimensional and three-dimensional image features, eliminating the need to construct matching triples between two-dimensional and three-dimensional images. This reduces the computational load during metric learning, decreases the retrieval time when retrieving a three-dimensional image of a target object from its two-dimensional image, and improves image retrieval speed.

[0084] Based on the same inventive concept as the above-mentioned image retrieval method, this application provides an image retrieval device 1, corresponding to an image retrieval method; Figure 4 A schematic diagram of the composition structure of an image retrieval device provided in this application embodiment. Figure 1 The image retrieval device 1 may include:

[0085] Extraction unit 11 is used to extract two-dimensional image features from the two-dimensional image and three-dimensional image features from the three-dimensional image when two-dimensional and three-dimensional images of the sample object are obtained.

[0086] Learning unit 12 is used to perform metric learning on the two-dimensional image features and the three-dimensional image features to obtain the correspondence between the two-dimensional image features and the three-dimensional image features;

[0087] The acquisition unit 13 is used to acquire the target two-dimensional image features of the target two-dimensional image when a target two-dimensional image of the target object is received;

[0088] The determining unit 14 is used to determine the target three-dimensional image corresponding to the target object based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features.

[0089] In some embodiments of this application, the apparatus further includes a processing unit;

[0090] The processing unit is used to perform interference removal processing on the two-dimensional image features to obtain a first feature; and to perform retrieval capability evaluation processing on the three-dimensional image features to obtain a second feature.

[0091] The learning unit 12 is used to perform metric learning on the first feature and the second feature to obtain the correspondence between the two-dimensional image features and the three-dimensional image features.

[0092] In some embodiments of this application, the apparatus further includes an input unit and a fusion unit.

[0093] The input unit is used to input the three-dimensional image features into the retrieval capability evaluation module to obtain the retrieval capability evaluation value of the three-dimensional image;

[0094] The fusion unit is used to fuse the three-dimensional image features and the retrieval capability evaluation value to obtain the second feature.

[0095] In some embodiments of this application, the processing unit is used to perform dimensionality reduction processing on the three-dimensional image features to obtain dimensionality-reduced three-dimensional image features;

[0096] The input unit is used to input the dimensionality-reduced 3D image features into the retrieval capability evaluation module to obtain the retrieval capability evaluation value.

[0097] In some embodiments of this application, the processing unit is configured to perform dimensionality reduction processing on the two-dimensional image features to obtain dimensionality-reduced two-dimensional image features; and to perform feature normalization processing on the dimensionality-reduced two-dimensional image features to obtain normalized features.

[0098] The determining unit 14 is used to determine the first feature based on the two-dimensional image features and the normalized features.

[0099] In some embodiments of this application, the determining unit 14 is used to obtain the target three-dimensional image features corresponding to the target object based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features;

[0100] The acquisition unit 13 is used to acquire the target three-dimensional image based on the target three-dimensional image features.

[0101] It should be noted that, in practical applications, the extraction unit 11, learning unit 12, acquisition unit 13, and determination unit 14 described above can be implemented by the processor 15 on the image retrieval device 1, specifically by a CPU (Central Processing Unit), MPU (Microprocessor Unit), DSP (Digital Signal Processor), or FPGA (Field Programmable Gate Array), etc.; the data storage described above can be implemented by the memory 16 on the image retrieval device 1.

[0102] This application also provides an image retrieval device 1, such as... Figure 5 As shown, the image retrieval device 1 includes a processor 15, a memory 16, and a communication bus 17. The memory 16 communicates with the processor 15 through the communication bus 17. The memory 16 stores programs executable by the processor 15. When the program is executed, the image retrieval method described above is executed by the processor 15.

[0103] In practical applications, the aforementioned memory 16 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 15.

[0104] This application provides a computer-readable storage medium having a computer program thereon, which, when executed by a processor 15, implements the image retrieval method as described above.

[0105] Understandably, image retrieval devices perform metric learning on two-dimensional and three-dimensional image features, eliminating the need to construct matching triples between two-dimensional and three-dimensional images. This reduces the computational load during metric learning, decreases the retrieval time when retrieving a three-dimensional image of a target object from its two-dimensional image, and improves image retrieval speed.

[0106] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0107] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. An image retrieval method, characterized in that, The method includes: Given two-dimensional and three-dimensional images of the sample object, extract two-dimensional image features from the two-dimensional image and extract three-dimensional image features from the three-dimensional image; The two-dimensional image features and the three-dimensional image features are subjected to metric learning to obtain the correspondence between the two-dimensional image features and the three-dimensional image features; Upon receiving a target two-dimensional image of the target object, the target two-dimensional image features of the target two-dimensional image are obtained; Based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features, the target three-dimensional image corresponding to the target object is determined; The step of performing metric learning on the two-dimensional image features and the three-dimensional image features to obtain the correspondence between the two-dimensional image features and the three-dimensional image features includes: The 3D image features are input into the retrieval capability evaluation module to obtain the retrieval capability evaluation value of the 3D image; The second feature is obtained by fusing the three-dimensional image features and the retrieval capability evaluation value; Metric learning is performed on the two-dimensional image features and the second feature to obtain the correspondence between the two-dimensional image features and the three-dimensional image features.

2. The method according to claim 1, characterized in that, The step of performing metric learning on the two-dimensional image features and the three-dimensional image features to obtain the correspondence between the two-dimensional image features and the three-dimensional image features includes: The two-dimensional image features are subjected to interference removal processing to obtain the first feature; Metric learning is performed on the first feature and the three-dimensional image feature to obtain the correspondence between the two-dimensional image feature and the three-dimensional image feature.

3. The method according to claim 1, characterized in that, The step of inputting the 3D image features into the retrieval capability evaluation module to obtain the retrieval capability evaluation value of the 3D image includes: The dimensionality reduction process is performed on the three-dimensional image features to obtain the dimensionality-reduced three-dimensional image features; The dimensionality-reduced 3D image features are input into the retrieval capability evaluation module to obtain the retrieval capability evaluation value.

4. The method according to claim 2, characterized in that, The step of removing interference from the two-dimensional image features to obtain the first feature includes: The two-dimensional image features are subjected to dimensionality reduction processing to obtain the dimensionality-reduced two-dimensional image features; The dimensionality-reduced two-dimensional image features are then normalized to obtain the normalized features. The first feature is determined based on the two-dimensional image features and the normalized features.

5. The method according to claim 1, characterized in that, The step of determining the target three-dimensional image corresponding to the target object based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features includes: Based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features, the target three-dimensional image features corresponding to the target object are obtained; The target 3D image is obtained based on the target 3D image features.

6. An image retrieval device, characterized in that, The device includes: The extraction unit is used to extract two-dimensional image features from the two-dimensional image and three-dimensional image features from the three-dimensional image when two-dimensional and three-dimensional images of the sample object are obtained. The learning unit is used to perform metric learning on the two-dimensional image features and the three-dimensional image features to obtain the correspondence between the two-dimensional image features and the three-dimensional image features; The acquisition unit is used to acquire the target two-dimensional image features of the target two-dimensional image when a target two-dimensional image of the target object is received; The determining unit is used to determine the target three-dimensional image corresponding to the target object based on the target two-dimensional image features and the correspondence between the two-dimensional image features and the three-dimensional image features; The device also includes an input unit and a fusion unit. The input unit is used to input the three-dimensional image features into the retrieval capability evaluation module to obtain the retrieval capability evaluation value of the three-dimensional image; The fusion unit is used to fuse the three-dimensional image features and the retrieval capability evaluation value to obtain a second feature; The learning unit is used to perform metric learning on the two-dimensional image features and the second feature to obtain the correspondence between the two-dimensional image features and the three-dimensional image features.

7. The apparatus according to claim 6, characterized in that, The device also includes a processing unit; The processing unit is used to perform interference removal processing on the two-dimensional image features to obtain the first feature; The learning unit is used to perform metric learning on the first feature and the three-dimensional image feature to obtain the correspondence between the two-dimensional image feature and the three-dimensional image feature.

8. An image retrieval device, characterized in that, The device includes: The system includes a memory, a processor, and a communication bus, wherein the memory communicates with the processor via the communication bus, and the memory stores an image retrieval program executable by the processor. When the image retrieval program is executed, the processor performs the method as described in any one of claims 1 to 5.

9. A storage medium storing a computer program thereon for use in an image retrieval device, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 5.

Images