Image retrieval method and device, computer equipment, storage medium and program product

Abstract

The invention relates to an image retrieval method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring a target text feature of a target image and a target image feature of at least one image view angle of the target image; determining a first retrieval result corresponding to the target text feature according to the target text feature, and determining a second retrieval result corresponding to each target image feature according to each target image feature; and performing fusion processing on the first retrieval result and each second retrieval result to obtain a target retrieval result corresponding to the target image. According to the method, the final target retrieval result is fused and determined based on the retrieval results corresponding to the target text feature and the multiple target image features, and the retrieval result corresponding to the text feature and the retrieval results corresponding to the multiple image features verify each other, so that the target retrieval result is more accurate.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to an image retrieval method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] Image retrieval refers to the technology of retrieving images that meet the conditions from an image database. According to the different ways of describing the image content, image retrieval technology can be divided into two categories: one is text-based image retrieval technology (TBIR), and the other is content-based image retrieval technology (CBIR).

[0003] With the development of artificial intelligence, deep learning-based image retrieval technology and multimodal image retrieval technology have gradually emerged in related technologies. At present, image retrieval is usually achieved by selecting one of the deep learning-based image retrieval technology and the multimodal image retrieval technology.

[0004] However, the above method has the problem of low retrieval accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide an image retrieval method, apparatus, computer equipment, storage medium and program product that can improve the accuracy of image retrieval in response to the above technical problems.

[0006] In a first aspect, the present application provides an image retrieval method, comprising:

[0007] Acquire a target text feature of a target image and a target image feature of at least one image perspective of the target image;

[0008] Determine a first search result corresponding to the target text feature according to the target text feature, and determine a second search result corresponding to each target image feature according to each target image feature;

[0009] The first search result and each second search result are fused to obtain a target search result corresponding to the target image.

[0010] The above method can more accurately obtain the features of the target image by combining the target text features of the target image and the target image features of at least one image perspective, and provide accurate data support for the subsequent determination of the target retrieval results based on the target text features and the target image features of at least one image perspective; at the same time, compared with the existing methods that determine the retrieval results based only on text features or image features, the above method is based on the retrieval results corresponding to the target text features and multiple target image features, and the retrieval results corresponding to the text features and the retrieval results corresponding to the multiple image features are mutually verified, making the target retrieval results more accurate.

[0011] In one embodiment, the determining of the first search result corresponding to the target text feature according to the target text feature includes:

[0012] Performing similarity matching between the target text feature and the candidate text feature of each candidate image in the image database, respectively, to obtain a first similarity between the target text feature and each candidate text feature;

[0013] According to the preset sorting order of each first similarity, a first similarity sequence is determined, and the first similarity sequence is used as the first search result; each first data item in the first similarity sequence includes at least the first similarity and the image identifier of the candidate image corresponding to the first similarity.

[0014] The method for obtaining the first search object provided in the embodiment of the present application sorts multiple candidate images in the image database through the first similarities between the target text features and each candidate text feature, and obtains a first search result, thereby achieving a preliminary search result for the target image in the image database based on the text features, and providing a data basis for subsequently determining the target search result based on this preliminary search result.

[0015] In one embodiment, the determining of the second search result corresponding to each target image feature according to each target image feature includes:

[0016] For each target image feature, the target image feature is matched with the candidate image features of each candidate image respectively to obtain a second similarity between the target image feature and each candidate image feature;

[0017] According to the preset sorting order of each second similarity, a second similarity sequence is determined, and the second similarity sequence is used as the second search result; each second data item in the second similarity sequence includes at least the second similarity and the image identifier of the candidate image corresponding to the second similarity.

[0018] The method for obtaining the second retrieval object provided in the embodiment of the present application sorts multiple candidate images in the image database through the second similarity between the target image feature and the candidate image features, and obtains the second retrieval result, thereby achieving a preliminary retrieval result for the target image in the image database based on the image features, thereby providing a data basis for subsequently determining the target retrieval result based on this preliminary retrieval result.

[0019] In one embodiment, the above-mentioned fusion processing of the first search result and each second search result to obtain the target search result corresponding to the target image includes:

[0020] For each image identifier, determining multiple similarities corresponding to the image identifier based on the first similarity sequence and each second similarity sequence;

[0021] Determine, according to the image identifier and the multiple similarities, a third data item corresponding to the image identifier;

[0022] A third similarity sequence is determined according to the third data items corresponding to the image identifiers, and the third similarity sequence is used as the target retrieval result.

[0023] The method for obtaining target retrieval results provided in the embodiment of the present application fuses multiple similarities corresponding to each image identifier to obtain a third data item corresponding to each image identifier, and determines the target retrieval result based on the third data item corresponding to each image identifier. Compared with the existing retrieval results determined only based on text features or image features, the method combines text features and image features to jointly determine the target retrieval results. The text features and image features check and balance each other, so that the determined target retrieval results are more accurate.

[0024] In one embodiment, determining the third data item corresponding to the image identifier according to the image identifier and the multiple similarities includes:

[0025] Perform statistical processing on multiple similarities to obtain the target similarity;

[0026] The image identification and target similarity are taken as the third data item.

[0027] The method for obtaining the third data item corresponding to the image identifier provided in the embodiment of the present application obtains the target similarity corresponding to the image identifier by statistically processing multiple similarities corresponding to the image identifier. Compared with the existing method of determining the retrieval result based only on any similarity of the image, the above method combines various similarities to jointly determine the target similarity, which can more accurately reflect the similarity between the target image and each candidate image, and thus accurately determine the target retrieval result.

[0028] In one embodiment, the determining of multiple similarities corresponding to the image identifier based on the first similarity sequence and each second similarity sequence includes:

[0029] Determine a reference similarity sequence from the first similarity sequence and each second similarity sequence;

[0030] According to the reference similarity sequence, reordering a plurality of other similarity sequences in the first similarity sequence and each second similarity sequence except the reference similarity sequence to obtain a plurality of updated similarity sequences;

[0031] According to the image identifier, the similarities respectively corresponding to the image identifiers are determined from the reference similarity sequence and each updated similarity sequence to obtain a plurality of similarities.

[0032] The method for determining multiple similarities corresponding to image identifiers provided in the embodiments of the present application reorders other similarity sequences based on a reference similarity sequence to obtain multiple updated similarity sequences, so that when obtaining multiple similarities corresponding to each image identifier, there is no need to first determine the data items corresponding to the image identifier in each similarity sequence one by one, and then determine the corresponding similarities in the data items. The multiple similarities corresponding to the image identifier can be efficiently determined according to the updated similarity sequence, greatly improving the efficiency of determining the multiple similarities corresponding to each image identifier.

[0033] In one embodiment, the above-mentioned reordering process is performed on the first similarity sequence and multiple other similarity sequences except the reference similarity sequence in each second similarity sequence according to the reference similarity sequence to obtain multiple updated similarity sequences, including:

[0034] According to the sorting order of each image identifier in the reference similarity sequence, the sorting order of each image identifier in each other similarity sequence is adjusted to obtain each updated similarity sequence.

[0035] The reordering method provided in the embodiment of the present application reorders the sorting order of each image identifier in each other similarity sequence based on the sorting order of each image identifier in the benchmark similarity sequence, so as to subsequently determine multiple similarities corresponding to the image identifier based on the other similarity sequences after the reordering, thereby greatly improving the efficiency of determining multiple similarities corresponding to each image identifier.

[0036] In one embodiment, the first data item further includes a first serial number, the second data item further includes a second serial number, and the first search result and each second search result are fused to obtain a target search result corresponding to the target image, including:

[0037] According to each first sequence number and the number of images of each candidate image, each first similarity in the first similarity sequence is corrected to obtain a corrected first similarity sequence;

[0038] According to each second sequence number and the number of images of each candidate image, each second similarity in the second similarity sequence is corrected to obtain a corrected second similarity sequence;

[0039] The corrected first similarity sequence is fused with each corrected second similarity sequence to obtain a target retrieval result corresponding to the target image.

[0040] The method for obtaining target retrieval results provided in the present application corrects each first similarity in the first similarity sequence and corrects each second similarity in the second similarity sequence, so that the target retrieval results determined according to the corrected first similarity sequence and the corrected second similarity sequence are more accurate.

[0041] In one embodiment, the method further comprises:

[0042] Acquire multiple candidate images;

[0043] For each candidate image, obtaining, according to the candidate image, a candidate text feature of the candidate image and a candidate image feature of at least one image perspective of the candidate image;

[0044] An image database is constructed based on the candidate text features of each candidate image and the candidate image features of at least one image perspective.

[0045] The method for constructing an image database provided in the embodiment of the present application makes a one-to-one correspondence between the candidate text features corresponding to the candidate images and the candidate image features of at least one perspective, thereby providing a data basis for subsequent image retrieval.

[0046] In one embodiment, the step of obtaining a candidate text feature of a candidate image and a candidate image feature of at least one image perspective of the candidate image includes:

[0047] For an image viewing angle, obtaining image features of at least two different magnifications of the image viewing angle;

[0048] The image features of at least two different magnifications are fused to obtain candidate image features of the image viewing angle of the candidate image.

[0049] A method for acquiring candidate image features of an image perspective provided in an embodiment of the present application obtains and fuses image features of at least two image perspectives with different magnifications to obtain a candidate image feature of an image perspective, thereby avoiding the problem of inaccurate retrieval results due to different image magnifications when performing image retrieval.

[0050] In a second aspect, the present application also provides an image retrieval device, comprising:

[0051] An acquisition module, used to acquire a target text feature of a target image and a target image feature of at least one image perspective of the target image;

[0052] A determination module, used to determine a first search result corresponding to a target text feature according to a target text feature, and to determine a second search result corresponding to each target image feature according to each target image feature;

[0053] The processing module is used to fuse the first search result and each second search result to obtain a target search result corresponding to the target image.

[0054] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0055] Acquire a target text feature of a target image and a target image feature of at least one image perspective of the target image;

[0056] Determine a first search result corresponding to the target text feature according to the target text feature, and determine a second search result corresponding to each target image feature according to each target image feature;

[0057] The first search result and each second search result are fused to obtain a target search result corresponding to the target image.

[0058] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0059] Acquire a target text feature of a target image and a target image feature of at least one image perspective of the target image;

[0060] Determine a first search result corresponding to the target text feature according to the target text feature, and determine a second search result corresponding to each target image feature according to each target image feature;

[0061] The first search result and each second search result are fused to obtain a target search result corresponding to the target image.

[0062] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0063] Acquire a target text feature of a target image and a target image feature of at least one image perspective of the target image;

[0064] Determine a first search result corresponding to the target text feature according to the target text feature, and determine a second search result corresponding to each target image feature according to each target image feature;

[0065] The first search result and each second search result are fused to obtain a target search result corresponding to the target image.

[0066] The above-mentioned image retrieval method, apparatus, computer equipment, storage medium and program product obtain the target text features of the target image and the target image features of at least one image perspective of the target image, and determine the first retrieval result corresponding to the target text features according to the target text features, and determine the second retrieval results corresponding to each target image feature according to each target image feature, and then fuse the first retrieval result and each second retrieval result to obtain the target retrieval result corresponding to the target image. In this way, by combining the target text features of the target image and the target image features of at least one image perspective, the features of the target image can be more accurately obtained, providing accurate data support for the subsequent determination of the target retrieval results based on the target text features and the target image features of at least one image perspective. At the same time, compared with the existing method of determining the retrieval results based only on text features or image features, the above-mentioned method is based on the retrieval results corresponding to the target text features and multiple target image features, respectively, and fuses to determine the final target retrieval result. The retrieval results corresponding to the text features and the retrieval results corresponding to the multiple image features verify each other, making the target retrieval results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0068] FIG1 is a diagram showing an application environment of an image retrieval method in one embodiment;

[0069] FIG2 is a schematic diagram of a flow chart of an image retrieval method in one embodiment;

[0070] Figure 3 is a schematic flow chart of an image retrieval method in another embodiment;

[0071] Figure 4 is a schematic flow chart of an image retrieval method in another embodiment;

[0072] FIG5 is a schematic diagram of a flow chart of an image retrieval method in another embodiment;

[0073] Figure 6 is a schematic flow chart of an image retrieval method in another embodiment;

[0074] FIG7 is a schematic diagram of a flow chart of an image retrieval method in another embodiment;

[0075] FIG8 is a schematic diagram of a flow chart of an image retrieval method in another embodiment;

[0076] Fig. 9 is a schematic flow chart of an image retrieval method in another embodiment;

[0077] Fig.10 is a schematic flow chart of an image retrieval method in another embodiment;

[0078] Fig.11 is a schematic flow chart of an image retrieval method in another embodiment;

[0079] FIG12 is a block diagram of an image retrieval device in one embodiment;

[0080] FIG. 13 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0082] Image retrieval refers to the technology of retrieving images that meet the conditions from an image database. Image retrieval technology can be divided into two categories according to the different ways of describing the image content: one is text-based image retrieval technology (TBIR) and the other is content-based image retrieval technology (CBIR). Image retrieval technology is widely used. For example, in the field of e-commerce, users can search for similar products by uploading product images. In the field of intelligent transportation, similar vehicles can be found through vehicle images captured by cameras. In the field of security monitoring, it can be used for face recognition, object recognition, etc.

[0083] With the development of artificial intelligence, deep learning-based image retrieval technology and multimodal image retrieval technology have gradually emerged in related technologies. At present, image retrieval is usually achieved by selecting one of the deep learning-based image retrieval technology and the multimodal image retrieval technology. For general simple images, because their features are simple and easy to distinguish, and the network learning cost is low, the use of the above-mentioned deep learning-based image retrieval technology or the multimodal-based image retrieval technology can achieve good results. However, for complex images, for example, images including complex textures, the differences between different categories are small, and the use of the above-mentioned deep learning-based image retrieval technology or the multimodal-based image retrieval technology has the problem of low retrieval accuracy.

[0084] In view of this, the embodiments of the present application provide an image retrieval method, apparatus, computer equipment, storage medium and program product, which obtains the target text features of the target image and the target image features of at least one image perspective of the target image, and determines the first retrieval result corresponding to the target text features according to the target text features, and determines the second retrieval results corresponding to each target image feature according to each target image feature, and then fuses the first retrieval result and each second retrieval result to obtain the target retrieval result corresponding to the target image. In this way, by combining the target text features of the target image and the target image features of at least one image perspective, the features of the target image can be more accurately obtained, and accurate data support is provided for the subsequent determination of the target retrieval results based on the target text features and the target image features of at least one image perspective. At the same time, compared with the existing method of determining the retrieval results based only on text features or image features, the above method is based on the retrieval results corresponding to the target text features and multiple target image features, respectively, and fuses to determine the final target retrieval result. The retrieval results corresponding to the text features and the retrieval results corresponding to the multiple image features verify each other, so that the target retrieval results are more accurate.

[0085] The image retrieval method provided in the embodiment of the present application can be applied in the implementation environment as shown in FIG1 , which includes a server 104. The server 104 can be implemented as an independent server 104 or a server cluster composed of multiple servers 104. The data storage system 102 can store the data that the server 104 needs to process. The data storage system 102 can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the server 104 can obtain the target text features of the target image and the target image features of at least one image perspective, and then determine the first retrieval result corresponding to the target text features, and the second retrieval results corresponding to each target image feature, and then, the first retrieval result and each second retrieval result are fused to obtain the target retrieval result.

[0086] In other possible implementations, the image retrieval method provided in the embodiment of the present application can also be applied to terminals, which can be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.

[0087] In one embodiment, as shown in FIG. 2 , an image retrieval method is provided, which is described by taking the method applied to the computer device in FIG. 1 as an example, and includes the following steps 201 to 203:

[0088] Step 201 : acquiring a target text feature of a target image and a target image feature of at least one image perspective of the target image.

[0089] The target image refers to an image to be retrieved, that is, it is necessary to retrieve from multiple candidate images stored in the image database to obtain an image similar to it.

[0090] The image perspective refers to different perspectives of the target object in the target image, including the main image perspective, the top view of the image, and the side view of the image.

[0091] In the embodiment of the present application, when it is necessary to retrieve a target candidate image similar to a target image in an image database, a target text feature of the target image and a target image feature of at least one image perspective of the target image may be first obtained.

[0092] Optionally, text features of multiple images and image features of at least one image perspective of each image can be acquired in advance, and the acquired text features of multiple images and image features of at least one image perspective of each image can be stored in a feature database. When it is necessary to acquire features corresponding to a target image, target text features of the target image and target image features of at least one image perspective of the target image can be directly acquired from the feature database.

[0093] Optionally, a feature extraction model, such as a CLIP model, can be pre-trained, and after acquiring the target image, the text information of the target image and the image information of at least one image perspective are first determined, and then the text information of the target image and the image information of at least one image perspective are input into the trained feature extraction model for feature extraction to obtain the target text features of the target image and the target image features of at least one image perspective of the target image.

[0094] The following is a training method for a CLIP model, including:

[0095] (1) Collect sample data of sample images; the sample data includes text information and image information. The image information includes three-view information of the target instance, such as first view information, second view information and third view information. The text information should be specific and accurate, and target instances of different categories should have certain differences. The image information should be as clear and accurate as possible.

[0096] (2) Combining sample data to form data pairs; each data pair includes any view information and text information, forming three types of data pairs including text information and first view information, text information and second view information, and text information and third view information. In each round of model training, the training order of the data pairs can be disrupted to complete the model training;

[0097] (3) Data feature extraction: The Clip model includes image-encoder and text-encoder. The image-encoder uses CNN network, vit network, etc. to extract image features, and the text-encoder uses transformer network to extract text features. For a single data {img, text}, the following formulas (1)-(2) are used to extract image features and text features respectively:

[0098]

[0099]

[0100] in, is the image feature, refers to the image encoding function, Refers to image information; is the image feature, refers to the text encoding function, Refers to text information;

[0101] (4) Feature mapping: Map the above text features and image features into a feature space to obtain text mapping features and image mapping features, respectively, satisfying the following formulas (3)-(4):

[0102]

[0103]

[0104] in, represents standardization, dot represents matrix dot multiplication;

[0105] (5) Calculate the cosine similarity; perform similarity calculation based on the text mapping features and the image mapping features to obtain the cosine similarity between the text mapping features and the image mapping features, which satisfies the following formula (5):

[0106]

[0107] Among them, T represents the transpose operation, exp represents the exponential operation, and t is the scaling factor;

[0108] (6) Calculate the loss and satisfy the following formula (6):

[0109]

[0110] (7) The network model gradient is calculated and back-propagated according to the loss value obtained by formula (6), and the model parameters are updated to complete one step of training. After the training is completed, a round of training is completed by looping the data loader. A total of 100 rounds are completed, or when the classification accuracy of the validation set begins to decline, the final CLIP model after training is obtained.

[0111] Step 202: determining a first search result corresponding to the target text feature according to the target text feature, and determining a second search result corresponding to each target image feature according to each target image feature.

[0112] Among them, the first retrieval result refers to the candidate image result that is matched according to the target text features in the candidate text features of each candidate image in the image database. For example, the first retrieval result can be the sorting result of each candidate image determined according to a preset sorting order by comparing the candidate text features of each candidate image in the image database with the target text features respectively.

[0113] Among them, the second retrieval result refers to the candidate image result that is matched according to the target image features in the candidate image features of each candidate image in the image database. For example, the second retrieval result can be the sorting result of each candidate image determined according to a preset sorting order by comparing the candidate image features of each candidate image in the image database with the target image features respectively.

[0114] It should be noted that the first search result and the second search result may be the same or different.

[0115] In an embodiment of the present application, after the target text feature of the target image is acquired as mentioned above, the target text feature and the candidate image features of multiple candidate images in the image database can be respectively similarly matched to obtain a first retrieval result corresponding to the target text feature, and, after at least one target image feature of the target image is acquired as mentioned above, each target image feature and the candidate image features of multiple candidate images in the image database can be respectively similarly matched to obtain a second retrieval result corresponding to each target image feature.

[0116] Step 203: The first search result and each second search result are fused to obtain a target search result corresponding to the target image.

[0117] The target retrieval result refers to a candidate image result obtained by comprehensively matching the candidate image features and the candidate text features of each candidate image in the image database according to the target image features and the target text features. For example, the target retrieval result may be a ranking result of each candidate image determined according to a preset ranking order after comprehensively considering the first retrieval result and each second retrieval result.

[0118] In an embodiment of the present application, after the first search result corresponding to the target text feature and the second search result corresponding to each target image feature are obtained, the first search result and each second search result are fused to obtain a target search result corresponding to the target image.

[0119] Optionally, the target retrieval result corresponding to the target image can be determined based on the ranking of each candidate image in the first retrieval result and each second retrieval result. For example, the image database includes three candidate images, namely candidate image A, candidate image B and candidate image C. The ranking order of the three candidate images in the first retrieval result is: candidate image A, candidate image B and candidate image C. The ranking order of the three candidate images in the second retrieval result is: candidate image A, candidate image C and candidate image B. The ranking order of the three candidate images in the third retrieval result is: candidate image A, candidate image C, and candidate image B. Then, through comprehensive evaluation, according to the ranking of the candidate images in each retrieval result, the target retrieval result corresponding to the target image is determined to be candidate image A, candidate image C and candidate image B.

[0120] The image retrieval method provided in the embodiment of the present application obtains the target text features of the target image and the target image features of at least one image perspective of the target image, determines the first retrieval result corresponding to the target text features according to the target text features, and determines the second retrieval results corresponding to each target image feature according to each target image feature, and fuses the first retrieval result and each second retrieval result to obtain the target retrieval result corresponding to the target image. The above method can more accurately obtain the features of the target image by combining the target text features of the target image and the target image features of at least one image perspective, and provides accurate data support for the subsequent determination of the target retrieval results based on the target text features and the target image features of at least one image perspective; at the same time, compared with the existing method of determining the retrieval results based only on text features or image features, the above method is based on the retrieval results corresponding to the target text features and multiple target image features, respectively, and fuses to determine the final target retrieval results, and the retrieval results corresponding to the text features and the retrieval results corresponding to the multiple image features verify each other, making the target retrieval results more accurate.

[0121] In one embodiment, based on the embodiment shown in FIG. 2, see Figure 3 This embodiment relates to a process of how a computer device determines a first search result corresponding to a target text feature according to the target text feature. Figure 3 As shown, in this embodiment, step 202 includes Figure 3 Step 301 and step 302 shown:

[0122] Step 301 : performing similarity matching between the target text feature and the candidate text feature of each candidate image in the image database, and obtaining a first similarity between the target text feature and each candidate text feature.

[0123] In an embodiment of the present application, after the target text features of the target image are acquired as described above, the target text features can be subjected to cosine similarity operations with the candidate text features of each candidate image in the image database to obtain the first similarities between the target text features and the candidate text features of each candidate image.

[0124] Step 302: determine a first similarity sequence according to a preset sorting order of each first similarity, and use the first similarity sequence as a first search result.

[0125] The preset sorting order may be an order from large to small according to the first similarity, or an order from small to large according to the first similarity.

[0126] The first similarity sequence is composed of multiple first data items, each of which corresponds to relevant information of a candidate image, and the relevant information of the candidate image includes at least the first similarity between the candidate text feature of the candidate image and the target text feature, the image identifier of the candidate image, and the serial number of the candidate image in the first similarity sequence.

[0127] In an embodiment of the present application, after the first similarities between the target text features and each candidate text feature are obtained, the candidate images in the image database can be sorted in descending order of the first similarities to obtain a first similarity sequence, and the first similarity sequence is used as the first retrieval result.

[0128] The method for obtaining the first search object provided in the embodiment of the present application sorts multiple candidate images in the image database through the first similarities between the target text features and each candidate text feature, and obtains a first search result, thereby achieving a preliminary search result for the target image in the image database based on the text features, and providing a data basis for subsequently determining the target search result based on this preliminary search result.

[0129] In one embodiment, based on Figure 3 The embodiment shown, see Figure 4 This embodiment relates to a process of how a computer device determines the second search result corresponding to each target image feature according to each target image feature. Figure 4 As shown, in this embodiment, step 202 includes Figure 4 Step 401 and step 402 are shown:

[0130] Step 401 : for each target image feature, the target image feature is matched with the candidate image features of each candidate image respectively in similarity to obtain a second similarity between the target image feature and each candidate image feature.

[0131] In an embodiment of the present application, after the multiple target image features of the target image are acquired as described above, for each target image feature, a cosine similarity operation can be performed on the target image feature and the candidate image features of each candidate image in the image database to obtain a second similarity between the target image feature and the candidate image features of each candidate image.

[0132] Step 402: determine a second similarity sequence according to a preset sorting order of each second similarity, and use the second similarity sequence as a second search result.

[0133] The preset sorting order may be a descending order of the second similarity, or a descending order of the second similarity.

[0134] The second similarity sequence is composed of multiple second data items, each second data item corresponds to relevant information of a candidate image, and the relevant information of the candidate image includes at least the second similarity between the candidate image features of the candidate image and the target image features, the image identifier of the candidate image, and the serial number of the candidate image in the second similarity sequence.

[0135] In an embodiment of the present application, after the second similarities between the target image features and the candidate image features are obtained, the candidate images in the image database can be sorted in descending order of the second similarities to obtain a second similarity sequence, and the second similarity sequence is used as the second retrieval result.

[0136] The method for obtaining the second retrieval object provided in the embodiment of the present application sorts multiple candidate images in the image database through the second similarity between the target image feature and the candidate image features, and obtains the second retrieval result, thereby achieving a preliminary retrieval result for the target image in the image database based on the image features, thereby providing a data basis for subsequently determining the target retrieval result based on this preliminary retrieval result.

[0137] In one embodiment, based on Figure 4 The embodiment shown in FIG5 is related to how the computer device fuses the first search result and the second search result to determine the target search result. As shown in FIG5, in this embodiment, step 203 includes steps 501, 502 and 503 shown in FIG5:

[0138] Step 501 : for each image identifier, based on the first similarity sequence and each second similarity sequence, determine a plurality of similarities corresponding to the image identifier.

[0139] The image identifier represents a unique identifier of a candidate image in the image database, and one candidate image corresponds to one image identifier.

[0140] The first similarity sequence and the second similarity sequence both include a plurality of candidate images, and the candidate images in the first similarity sequence and the second similarity sequence may be the same or different.

[0141] In an embodiment of the present application, for an image identifier corresponding to each candidate image in a first similarity sequence and a second similarity sequence, according to each first data item in the first similarity sequence and each second data item in the second similarity sequence, for any image identifier, a first data item corresponding to any image identifier in the first similarity sequence is determined, and then a first similarity is determined according to the corresponding first data item in the first similarity sequence, and a second data item corresponding to any image identifier in each second similarity sequence is determined, and then each second similarity is determined according to the corresponding second data item in the second similarity sequence.

[0142] Step 502: Determine a third data item corresponding to the image identifier according to the image identifier and multiple similarities.

[0143] The third data items respectively correspond to relevant information of a candidate image, and the relevant information of the candidate image at least includes a third similarity between features of the candidate image and features of the target image, and an image identifier of the candidate image.

[0144] In an embodiment of the present application, after obtaining the first similarity of each image identifier in the first similarity sequence and the second similarities in the second similarity sequences, for any image identifier, the sum of the first similarity and the second similarities corresponding to the image identifier is determined as the third similarity corresponding to the image identifier, and then, based on the third similarity corresponding to the image identifier, the third data item corresponding to the image identifier is determined.

[0145] Step 503: determine a third similarity sequence according to the third data item corresponding to each image identifier, and use the third similarity sequence as a target retrieval result.

[0146] In an embodiment of the present application, after the third data item of an image identifier is obtained as described above, according to the same method, the sum of the first similarity corresponding to other image identifiers and each second similarity is determined as the third similarity corresponding to the other image identifiers, and the third data items corresponding to the other image identifiers are determined according to the third similarities corresponding to the other image identifiers, and then the third data items corresponding to each image identifier are obtained, and then the third data items are sorted according to the preset arrangement order of the third similarity to obtain a third similarity sequence, and the third similarity sequence is determined as the target retrieval result.

[0147] The method for obtaining target retrieval results provided in the embodiment of the present application fuses multiple similarities corresponding to each image identifier to obtain a third data item corresponding to each image identifier, and determines the target retrieval result based on the third data item corresponding to each image identifier. Compared with the existing retrieval results determined only based on text features or image features, the method combines text features and image features to jointly determine the target retrieval results. The text features and image features check and balance each other, so that the determined target retrieval results are more accurate.

[0148] In one embodiment, based on the embodiment shown in FIG. 5 , see Figure 6 This embodiment relates to a process of how a computer device determines a third data item based on an image identifier and multiple similarities. Figure 6 As shown, in this embodiment, step 502 includes Figure 6 Step 601 and step 602 are shown:

[0149] Step 601, statistically process multiple similarities to obtain a target similarity.

[0150] In an embodiment of the present application, for any image identifier, when the multiple similarities corresponding to the image identifier are obtained as mentioned above, the multiple similarities corresponding to the image identifier can be corrected respectively to obtain multiple corrected similarities corresponding to the image identifier, and the sum of the multiple corrected similarities corresponding to the image identifier is used as the target similarity corresponding to the image identifier.

[0151] Optionally, each similarity correction value corresponding to the image identifier can be determined based on the serial number corresponding to the image identifier in each similarity sequence and the total number of candidate images in multiple similarity sequences, and then each similarity corresponding to the image identifier is corrected based on each similarity correction value to obtain multiple corrected similarities corresponding to the image identifier.

[0152] The total number of candidate images in multiple similarity sequences refers to the number M of candidate images in the image set obtained by first forming a candidate image set from the candidate images in the multiple similarity sequences and then removing repeated candidate images in the candidate image set.

[0153] Optionally, after obtaining multiple corrected similarities corresponding to the image identification, weight calculation can be performed on the multiple corrected similarities according to pre-set weight values ​​of each similarity to obtain multiple corrected similarities after weight calculation, and the sum of the multiple corrected similarities after weight calculation can be used as the target similarity corresponding to the image identification.

[0154] For example, after the corrected similarity Score1 corresponding to a certain image identifier in any similarity sequence is determined as above, the corrected similarities Score1, Score2, Score3 and Score4 corresponding to the image identifier in each similarity sequence are determined in the same way, and then each corrected similarity is weighted to obtain the corrected similarity after each weight calculation, and then the sum of the corrected similarities after each weight calculation is determined as the target similarity corresponding to the image identifier. The process of determining the target similarity corresponding to the image identifier is shown in the following formula (7):

[0155]

[0156] in, It refers to the target similarity corresponding to the image label. , , and 4 refers to the weights corresponding to different similarity sequences.

[0157] Step 602: Use the image identifier and the target similarity as the third data item.

[0158] In the embodiment of the present application, after the target similarity corresponding to the image identifier is obtained as described above, the image identifier and the target similarity corresponding to the image identifier may be used as the third data item.

[0159] Optionally, after each image identifier and the target similarity respectively corresponding to each image identifier are determined, each image identifier and the target similarity respectively corresponding to each image identifier may be determined as the third data item.

[0160] The method for obtaining the third data item corresponding to the image identifier provided in the embodiment of the present application obtains the target similarity corresponding to the image identifier by statistically processing multiple similarities corresponding to the image identifier. Compared with the existing method of determining the retrieval result based only on any similarity of the image, the above method combines various similarities to jointly determine the target similarity, which can more accurately reflect the similarity between the target image and each candidate image, and thus accurately determine the target retrieval result.

[0161] In one embodiment, based on the embodiment shown in FIG5 , referring to FIG7 , this embodiment relates to a process of how a computer device determines multiple similarities corresponding to an image identifier based on multiple similarity sequences. As shown in FIG7 , in this embodiment, step 501 includes steps 701, 702, and 703 shown in FIG7 :

[0162] Step 701: determine a reference similarity sequence from the first similarity sequence and each second similarity sequence.

[0163] In the embodiment of the present application, after the first similarity sequence and the second similarity sequences are obtained, any one of the first similarity sequence and the second similarity sequences may be used as a reference similarity sequence.

[0164] Step 702: reorder the first similarity sequence and a plurality of other similarity sequences in each second similarity sequence except the reference similarity sequence according to the reference similarity sequence to obtain a plurality of updated similarity sequences.

[0165] In the embodiment of the present application, after the benchmark similarity sequence is obtained as described above, multiple similarity sequences other than the benchmark similarity sequence in the first similarity sequence and each second similarity sequence are used as the residual similarity sequence, and then the sorting order of each image identifier in the residual similarity sequence is adjusted according to the sorting order of each image identifier in the benchmark similarity sequence to obtain the sorting order of each image identifier in the residual similarity sequence, and the sorting order of each image identifier in the residual similarity sequence is used as the updated similarity sequence.

[0166] Step 703 : According to the image identifier, the similarities corresponding to the image identifiers are determined from the reference similarity sequence and each updated similarity sequence to obtain a plurality of similarities.

[0167] In the embodiment of the present application, after the baseline similarity sequence and each updated similarity sequence are determined as described above, the similarities corresponding to any image identifier in the baseline similarity sequence and each updated similarity sequence can be determined according to the image identifiers in the baseline similarity sequence and each updated similarity sequence, thereby obtaining multiple similarities of any image identifier.

[0168] The method for determining multiple similarities corresponding to image identifiers provided in the embodiments of the present application reorders other similarity sequences based on a reference similarity sequence to obtain multiple updated similarity sequences, so that when obtaining multiple similarities corresponding to each image identifier, there is no need to first determine the data items corresponding to the image identifier in each similarity sequence one by one, and then determine the corresponding similarities in the data items. The multiple similarities corresponding to the image identifier can be efficiently determined according to the updated similarity sequence, greatly improving the efficiency of determining the multiple similarities corresponding to each image identifier.

[0169] In one embodiment, based on the embodiment shown in FIG. 7 , this embodiment relates to a process of how a computer device determines multiple updated similarity sequences based on a reference similarity sequence. In this embodiment, step 702 includes:

[0170] According to the sorting order of each image identifier in the reference similarity sequence, the sorting order of each image identifier in each other similarity sequence is adjusted to obtain each updated similarity sequence.

[0171] In an embodiment of the present application, after obtaining the benchmark similarity sequence, the sorting order of each image identifier in each other similarity sequence can be adjusted to the sorting order of each image identifier consistent with the benchmark similarity sequence according to the sorting order of each image identifier in the benchmark similarity sequence, thereby obtaining each updated similarity sequence.

[0172] For example, the benchmark similarity sequence is {(image identifier 1, sequence number 1, similarity 0.9), (image identifier 3, sequence number 2, similarity 0.8), (image identifier 6, sequence number 3, similarity 0.5)}, and the other similarity sequences are {(image identifier 1, sequence number 1, similarity 0.8), (image identifier 6, sequence number 2, similarity 0.7), (image identifier 3, sequence number 3, similarity 0.5)}, then according to the sorting order of each image identifier in the benchmark similarity sequence, the other similarity sequences are adjusted to the updated similarity sequences {(image identifier 1, sequence number 1, similarity 0.8), (image identifier 3, sequence number 3, similarity 0.5), (image identifier 6, sequence number 2, similarity 0.7)}.

[0173] The reordering method provided in the embodiment of the present application reorders the sorting order of each image identifier in each other similarity sequence based on the sorting order of each image identifier in the benchmark similarity sequence, so as to subsequently determine multiple similarities corresponding to the image identifier based on the other similarity sequences after the reordering, thereby greatly improving the efficiency of determining multiple similarities corresponding to each image identifier.

[0174] In one embodiment, based on Figure 4In the embodiment shown, see FIG8 , the first data item further includes a first serial number, and the second data item further includes a second serial number. This embodiment relates to how a computer device fuses multiple search results to obtain a target search result corresponding to a target image. As shown in FIG8 , in this embodiment, step 203 includes steps 801 to 803 shown in FIG8 :

[0175] Step 801 : correcting each first similarity in a first similarity sequence according to each first sequence number and the number of images of each candidate image to obtain a corrected first similarity sequence.

[0176] For each first data item in the first search result, the first data item includes not only the first similarity and the image identifier of the candidate image corresponding to the first similarity, but also the first serial number corresponding to the candidate image.

[0177] The number of each candidate image refers to the number M of candidate images in the image set obtained by first forming a candidate image set from each candidate image in a plurality of similarity sequences and then removing repeated candidate images in the candidate image set.

[0178] It should be noted that, assuming that the ranking information of candidate images in any similarity sequence is R{r1,r2...rm}, then 1<=rm<=M.

[0179] In an embodiment of the present application, after the first sequence numbers corresponding to the first data items in the first search result are obtained as described above, the first similarity correction values ​​corresponding to the candidate images in the first similarity sequence are determined according to the first sequence numbers and the number of images of the candidate images, and then the first similarities corresponding to the candidate images in the first similarity sequence are corrected according to the first similarity correction values ​​to obtain a corrected first similarity sequence.

[0180] For example, the first sequence number of a candidate image in the first similarity sequence is r i , then according to the number of images M of each candidate image, the first similarity correction value corresponding to the candidate image in the first similarity sequence can be expressed by the following formula (8):

[0181]

[0182] in, refers to the first similarity correction value corresponding to the candidate image in the first similarity sequence, It is a correction parameter and is a fixed value.

[0183] Furthermore, after the first similarity correction value corresponding to the candidate image in the first similarity sequence is obtained, the corrected first similarity corresponding to the candidate image in the first similarity sequence can be determined based on the first similarity correction value and the original similarity corresponding to the candidate image in the first similarity sequence, and the corrected first similarity sequence can be determined based on the corrected first similarities corresponding to each candidate image in the first similarity sequence.

[0184] Optionally, the process of determining the corrected first similarity corresponding to the candidate image in the first similarity sequence may refer to the following formula (9):

[0185]

[0186] in, 1 refers to the corrected first similarity corresponding to the candidate image in the first similarity sequence, 1 refers to the original similarity corresponding to the candidate image in the first similarity sequence.

[0187] Step 802: correct each second similarity in the second similarity sequence according to each second sequence number and the number of images of each candidate image to obtain a corrected second similarity sequence.

[0188] For each second data item in the second search result, the second data item includes not only the second similarity and the image identifier of the candidate image corresponding to the second similarity, but also the second serial number corresponding to the candidate image.

[0189] In an embodiment of the present application, after the second sequence numbers corresponding to the second data items in the second search results are obtained as described above, the second similarity correction values ​​corresponding to the candidate images in the second similarity sequence are determined according to the second sequence numbers and the number of images of the candidate images, and then the second similarities corresponding to the candidate images in the second similarity sequence are corrected according to the second similarity correction values ​​to obtain a corrected second similarity sequence.

[0190] Optionally, the second similarity correction value corresponding to the candidate image in the determined second similarity sequence may refer to the above formula (2).

[0191] Furthermore, after the second similarity correction value corresponding to the candidate image in the second similarity sequence is obtained, the corrected second similarity corresponding to the candidate image in the second similarity sequence can be determined based on the second similarity correction value and the original similarity corresponding to the candidate image in the second similarity sequence, and the corrected second similarity sequence can be determined based on the corrected second similarities corresponding to each candidate image in the second similarity sequence.

[0192] Optionally, the process of determining the corrected first similarity corresponding to the candidate image in the first similarity sequence may refer to the above formula (3).

[0193] Step 803 , fusing the corrected first similarity sequence with each corrected second similarity sequence to obtain a target retrieval result corresponding to the target image.

[0194] In an embodiment of the present application, after the corrected first similarity sequence and each corrected second similarity sequence are obtained as described above, the similarities of the corresponding candidate images in the corrected first similarity sequence and each corrected second similarity sequence can be fused to obtain the target similarity of each candidate image, and the corresponding target retrieval result can be determined based on the target similarity of each candidate image.

[0195] The method for obtaining target retrieval results provided in the present application corrects each first similarity in the first similarity sequence and corrects each second similarity in the second similarity sequence, so that the target retrieval results determined according to the corrected first similarity sequence and the corrected second similarity sequence are more accurate.

[0196] In one embodiment, based on Figure 3 - The embodiment shown in FIG8, see Fig. 9 In this embodiment, step 202 also includes Fig. 9 Steps 303 to 305 shown:

[0197] Step 303: Acquire multiple candidate images.

[0198] The candidate images refer to images to be stored in the image database.

[0199] In the embodiment of the present application, when a plurality of candidate images need to be stored in an image database, the plurality of candidate images may be acquired first.

[0200] Optionally, multiple candidate images to be stored in the image database may be acquired in real time.

[0201] Optionally, multiple candidate images to be stored in the image database may be collected in advance, and the multiple candidate images may be stored in a preset database. When the candidate images need to be stored in the image database, they may be obtained from the preset database.

[0202] Step 304 : for each candidate image, obtain candidate text features of the candidate image and candidate image features of at least one image perspective of the candidate image according to the candidate image.

[0203] The image perspective refers to different perspectives of the target object in the candidate image, including the main perspective of the image, the top-down perspective of the image, and the side perspective of the image.

[0204] In the embodiment of the present application, for each candidate image, candidate text features of the candidate image and candidate image features of at least one image perspective of the candidate image are respectively obtained.

[0205] Optionally, a feature extraction model, such as a CLIP model, can be pre-trained, and after acquiring a candidate image, the text information of the candidate image and the image information of at least one image perspective are first determined, and then the text information of the candidate image and the image information of at least one image perspective are input into the trained feature extraction model for feature extraction to obtain candidate text features of the candidate image and candidate image features of at least one image perspective of the candidate image.

[0206] Optionally, text features of multiple candidate images and image features of at least one image perspective of each candidate image can be acquired in advance, and the acquired text features of the multiple candidate images and image features of at least one image perspective of each candidate image can be stored in a feature database. When it is necessary to obtain features corresponding to the candidate images, the candidate text features of the candidate images and the candidate image features of at least one image perspective of the candidate images can be directly acquired from the feature database.

[0207] Next, a method for extracting text features of candidate images is provided. For extracting candidate text features, the following formula (10) needs to be satisfied:

[0208]

[0209] Among them, T refers to the candidate text features, refers to the text encoding function, and text refers to the text information of the candidate image.

[0210] Step 305: construct an image database based on the candidate text features of each candidate image and the candidate image features of at least one image perspective.

[0211] In an embodiment of the present application, after the candidate text features of each candidate image and the candidate image features of at least one image perspective of each candidate image are acquired as described above, based on the image identification of the candidate image, the candidate text features and the candidate image features of at least one image perspective are matched one-to-one with the image identification of the candidate image to construct an image database.

[0212] The method for constructing an image database provided in the embodiment of the present application makes a one-to-one correspondence between the candidate text features corresponding to the candidate images and the candidate image features of at least one perspective, thereby providing a data basis for subsequent image retrieval.

[0213] In one embodiment, based on Fig. 9 The embodiment shown, see Fig.10This embodiment relates to a process of how a computer device obtains candidate image features of at least one image perspective of a candidate image. Fig.10 As shown, in this embodiment, step 303 includes Fig.10 Step 901 and step 902 shown:

[0214] Step 901: for an image viewing angle, obtain image features of at least two different magnifications of the image viewing angle.

[0215] Among them, different magnifications mean different pixels, that is, images of multiple pixels with different pixel angles of an image are obtained, and then image features of each different pixel are extracted.

[0216] In an embodiment of the present application, after a candidate image is acquired, for an image viewing angle of the candidate image, at least two images of the image viewing angle with different magnifications are acquired, and then image features of the at least two images of the image viewing angle with different magnifications are extracted.

[0217] It should be noted that the method for extracting image features can be the extraction through the model provided above, as well as extraction from a feature database that pre-stores image features, which will not be described in detail here.

[0218] Optionally, the feature extraction network model (for example, the clip model) trained by the above process can be used to extract the features of the candidate image. Below, a method for extracting candidate image features of a candidate image is provided. For the candidate image features of an image viewing angle, it is necessary to obtain at least two image features of different magnifications. To obtain at least two image features of different magnifications, the following formula (11) needs to be satisfied:

[0219]

[0220] in, refers to the candidate image features of the first magnification, refers to the image encoding function, Refers to the image information of the first magnification of the candidate image at an image viewing angle, refers to the candidate image features of the second magnification, Refers to the image information of the candidate image at the second magnification under an image viewing angle. refers to the candidate image features of the third rate, It refers to the image information of the candidate image at the third magnification under an image viewing angle.

[0221] Step 902 , fusing at least two image features of different magnifications to obtain candidate image features of the image viewing angle of the candidate image.

[0222] In the embodiment of the present application, after acquiring at least two image features of different magnifications, the two image features of different magnifications may be spliced ​​to obtain candidate image features of the image viewing angle of the candidate image.

[0223] Next, a method for splicing candidate image features of a candidate image is provided, which satisfies the following formula (12):

[0224]

[0225] in, A candidate image feature representing an image view of the candidate image, It is a feature merging operation, which includes: 1) element-wise addition, which represents network structures such as FPN / ResNet / SENet; 2) feature map concatenation, which represents the network DenseNet.

[0226] A method for acquiring candidate image features of an image perspective provided in an embodiment of the present application obtains and fuses image features of at least two image perspectives with different magnifications to obtain a candidate image feature of an image perspective, thereby avoiding the problem of inaccurate retrieval results due to different image magnifications when performing image retrieval.

[0227] In one embodiment, see Fig.11 , also provides an image retrieval method, including:

[0228] Step 10, obtaining multiple candidate images;

[0229] Step 11, for each candidate image, obtaining candidate text features of the candidate image and candidate image features of at least one image perspective of the candidate image according to the candidate image;

[0230] Step 12, constructing an image database based on the candidate text features of each candidate image and the candidate image features of at least one image perspective;

[0231] Step 13, obtaining a target text feature of a target image and a target image feature of at least one image perspective of the target image;

[0232] Step 14, performing similarity matching between the target text feature and the candidate text feature of each candidate image in the image database, and obtaining a first similarity between the target text feature and each candidate text feature;

[0233] Step 15, determining a first similarity sequence according to a preset sorting order of each first similarity, and using the first similarity sequence as a first search result;

[0234] Step 16, for each target image feature, performing similarity matching between the target image feature and the candidate image features of each candidate image, to obtain a second similarity between the target image feature and each candidate image feature;

[0235] Step 17, determining a second similarity sequence according to a preset sorting order of each second similarity, and using the second similarity sequence as a second search result;

[0236] Step 18, for each image identifier, determining a reference similarity sequence from the first similarity sequence and each second similarity sequence;

[0237] Step 19, adjusting the sorting order of each image identifier in each other similarity sequence according to the sorting order of each image identifier in the reference similarity sequence to obtain each updated similarity sequence;

[0238] Step 20, according to the image identifier, determining the similarities corresponding to the image identifiers from the reference similarity sequence and each updated similarity sequence, to obtain multiple similarities;

[0239] Step 21, statistically processing multiple similarities to obtain a target similarity;

[0240] Step 22, taking the image identifier and the target similarity as the third data item;

[0241] Step 23: determine a third similarity sequence according to the third data item corresponding to each image identifier, and use the third similarity sequence as a target retrieval result.

[0242] The above method can more accurately obtain the features of the target image by combining the target text features of the target image and the target image features of at least one image perspective, and provide accurate data support for the subsequent determination of the target retrieval results based on the target text features and the target image features of at least one image perspective; at the same time, compared with the existing methods that determine the retrieval results based only on text features or image features, the above method is based on the retrieval results corresponding to the target text features and multiple target image features, and the retrieval results corresponding to the text features and the retrieval results corresponding to the multiple image features are mutually verified, making the target retrieval results more accurate.

[0243] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0244] Based on the same inventive concept, the embodiment of the present application also provides an image retrieval device for implementing the above-mentioned image retrieval method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image retrieval device embodiments provided below can refer to the limitations on the image retrieval method above, and will not be repeated here.

[0245] In an exemplary embodiment, as shown in FIG12 , an image retrieval device is provided, comprising: an acquisition module 20, a determination module 21 and a processing module 22, wherein:

[0246] The acquisition module 20 is used to acquire target text features of a target image and target image features of at least one image viewing angle of the target image.

[0247] The determination module 21 is used to determine the first search result corresponding to the target text feature according to the target text feature, and to determine the second search result corresponding to each target image feature according to each target image feature.

[0248] The processing module 22 is used to fuse the first search result and each second search result to obtain a target search result corresponding to the target image.

[0249] In an exemplary embodiment, the determination module 21 includes: a first matching unit and a first sorting unit, wherein:

[0250] A first matching unit is specifically used to perform similarity matching between the target text feature and the candidate text feature of each candidate image in the image database, so as to obtain a first similarity between the target text feature and each candidate text feature;

[0251] The first sorting unit is specifically used to determine a first similarity sequence according to a preset sorting order of each first similarity, and use the first similarity sequence as a first search result; each first data item in the first similarity sequence at least includes a first similarity and an image identifier of a candidate image corresponding to the first similarity.

[0252] In an exemplary embodiment, the determination module 21 further includes: a second matching unit and a second sorting unit, wherein:

[0253] A second matching unit is specifically used to perform similarity matching on each target image feature with the candidate image features of each candidate image, so as to obtain a second similarity between the target image feature and each candidate image feature;

[0254] The second sorting unit is specifically used to determine a second similarity sequence according to a preset sorting order of each second similarity, and use the second similarity sequence as a second search result; each second data item in the second similarity sequence at least includes a second similarity and an image identifier of a candidate image corresponding to the second similarity.

[0255] In an exemplary embodiment, the processing module 22 further includes: a first determining unit, a second determining unit and a third determining unit, wherein:

[0256] A first determining unit is specifically configured to determine, for each image identifier, a plurality of similarities corresponding to the image identifier based on the first similarity sequence and each second similarity sequence;

[0257] A second determining unit, specifically configured to determine a third data item corresponding to the image identifier according to the image identifier and the multiple similarities;

[0258] The third determining unit is specifically configured to determine a third similarity sequence according to the third data items corresponding to each image identifier, and use the third similarity sequence as a target retrieval result.

[0259] In an exemplary embodiment, the second determination unit is further configured to perform statistical processing on multiple similarities to obtain a target similarity; and use the image identifier and the target similarity as the third data item.

[0260] In an exemplary embodiment, the above-mentioned first determination unit is further specifically used to determine a baseline similarity sequence from the first similarity sequence and each second similarity sequence; according to the baseline similarity sequence, a plurality of other similarity sequences in the first similarity sequence and each second similarity sequence except the baseline similarity sequence are reordered to obtain a plurality of updated similarity sequences; according to the image identifier, the similarities corresponding to the image identifiers are determined from the baseline similarity sequence and each updated similarity sequence to obtain a plurality of similarities.

[0261] In an exemplary embodiment, the first determining unit is further configured to adjust the sorting order of each image identifier in each other similarity sequence according to the sorting order of each image identifier in the reference similarity sequence to obtain each updated similarity sequence.

[0262] In an exemplary embodiment, the first data item further includes a first serial number, the second data item further includes a second serial number, and the processing module 22 includes: a first correction unit, a second correction unit and a processing unit, which are specifically used to:

[0263] A first correction unit, specifically configured to correct each first similarity in the first similarity sequence according to each first sequence number and the number of images of each candidate image, to obtain a corrected first similarity sequence;

[0264] A second correction unit is specifically used to correct each second similarity in the second similarity sequence according to each second sequence number and the number of images of each candidate image to obtain a corrected second similarity sequence;

[0265] The processing unit is specifically used to fuse the corrected first similarity sequence with each corrected second similarity sequence to obtain a target retrieval result corresponding to the target image.

[0266] In an exemplary embodiment, the determination module 21 further includes: an acquisition unit, an extraction unit and a construction unit, wherein:

[0267] An acquisition unit, specifically used to acquire a plurality of candidate images;

[0268] An extraction unit, specifically configured to obtain, for each candidate image, a candidate text feature of the candidate image and a candidate image feature of at least one image perspective of the candidate image according to the candidate image;

[0269] The construction unit is specifically used to construct an image database according to the candidate text features of each candidate image and the candidate image features of at least one image perspective.

[0270] In an exemplary embodiment, the above-mentioned extraction unit is specifically used to obtain at least two image features of different magnifications of an image perspective for an image perspective; and fuse the at least two image features of different magnifications to obtain candidate image features of the image perspective of the candidate image.

[0271] Each module in the above-mentioned image retrieval device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0272] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG13. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image retrieval method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0273] Those skilled in the art will understand that the structure shown in FIG. 13 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0274] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0275] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0276] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0277] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0278] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0279] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image retrieval method, characterized in that: The method comprises: Acquire a target text feature of a target image and a target image feature of at least one image perspective of the target image; Determine a first search result corresponding to the target text feature according to the target text feature, and determine a second search result corresponding to each target image feature according to each target image feature; The first retrieval result and each of the second retrieval results are fused to obtain a target retrieval result corresponding to the target image.

2. The method according to claim 1, characterized in that The determining, according to the target text feature, a first search result corresponding to the target text feature comprises: Performing similarity matching on the target text feature and the candidate text features of each candidate image in the image database, respectively, to obtain a first similarity between the target text feature and each candidate text feature; According to the preset sorting order of each first similarity, a first similarity sequence is determined, and the first similarity sequence is used as the first search result; each first data item in the first similarity sequence includes at least the first similarity and an image identifier of a candidate image corresponding to the first similarity.

3. The method according to claim 2, characterized in that The determining, according to each of the target image features, a second search result corresponding to each of the target image features comprises: For each of the target image features, performing similarity matching between the target image feature and the candidate image features of each of the candidate images, to obtain a second similarity between the target image feature and each of the candidate image features; According to the preset sorting order of each second similarity, a second similarity sequence is determined, and the second similarity sequence is used as the second search result; each second data item in the second similarity sequence includes at least the second similarity and the image identifier of the candidate image corresponding to the second similarity.

4. The method according to claim 3, characterized in that The fusing the first search result and each of the second search results to obtain a target search result corresponding to the target image includes: For each of the image identifiers, determining a plurality of similarities corresponding to the image identifier based on the first similarity sequence and each of the second similarity sequences; Determining a third data item corresponding to the image identifier according to the image identifier and the multiple similarities; A third similarity sequence is determined according to the third data items corresponding to the image identifiers, and the third similarity sequence is used as the target retrieval result.

5. The method according to claim 4, characterized in that The determining, according to the image identifier and the multiple similarities, a third data item corresponding to the image identifier includes: Performing statistical processing on the multiple similarities to obtain a target similarity; The image identifier and the target similarity are used as the third data item.

6. The method according to claim 4, characterized in that The determining, based on the first similarity sequence and each of the second similarity sequences, a plurality of similarities corresponding to the image identifier comprises: Determine a reference similarity sequence from the first similarity sequence and each of the second similarity sequences; According to the reference similarity sequence, reordering a plurality of similarity sequences other than the reference similarity sequence in the first similarity sequence and each of the second similarity sequences to obtain a plurality of updated similarity sequences; According to the image identifier, the similarities respectively corresponding to the image identifiers are determined from the reference similarity sequence and each of the updated similarity sequences to obtain the multiple similarities.

7. The method according to claim 6, characterized in that The step of reordering the first similarity sequence and a plurality of other similarity sequences in each of the second similarity sequences except the reference similarity sequence according to the reference similarity sequence to obtain a plurality of updated similarity sequences includes: According to the sorting order of each image identifier in the reference similarity sequence, the sorting order of each image identifier in each other similarity sequence is adjusted to obtain each updated similarity sequence.

8. The method according to claim 3, characterized in that The first data item further includes a first serial number, the second data item further includes a second serial number, and the first search result and each of the second search results are fused to obtain a target search result corresponding to the target image, including: Correcting each of the first similarities in the first similarity sequence according to each of the first sequence numbers and the number of images of each of the candidate images to obtain a corrected first similarity sequence; According to each of the second sequence numbers and the number of images of each of the candidate images, each of the second similarities in the second similarity sequence is corrected to obtain a corrected second similarity sequence; The modified first similarity sequence is fused with each of the modified second similarity sequences to obtain a target retrieval result corresponding to the target image.

9. The method according to any one of claims 2 to 8, characterized in that: The method further comprises: Acquire a plurality of the candidate images; For each of the candidate images, obtaining, according to the candidate image, a candidate text feature of the candidate image and a candidate image feature of at least one image perspective of the candidate image; The image database is constructed according to the candidate text features of each of the candidate images and the candidate image features of the at least one image perspective.

10. The method according to claim 9, characterized in that The acquiring of the candidate text features of the candidate image and the candidate image features of at least one image perspective of the candidate image comprises: For one of the image viewing angles, acquiring image features of at least two different magnifications of the image viewing angle; The at least two image features of different magnifications are fused to obtain the candidate image features of the image viewing angle of the candidate image.

11. An image retrieval device, characterized in that: The device comprises: An acquisition module, used for acquiring a target text feature of a target image and a target image feature of at least one image viewing angle of the target image; A determination module, configured to determine a first search result corresponding to the target text feature according to the target text feature, and to determine a second search result corresponding to each target image feature according to each target image feature; The processing module is used to fuse the first retrieval result and each of the second retrieval results to obtain a target retrieval result corresponding to the target image.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images