Image Retrieval Method, Apparatus, Storage Medium and Electronic Device

Abstract

An embodiment of the present invention provides a method, apparatus, storage medium, and electronic device for image retrieval. The method includes: determining a first retrieval feature and a second retrieval feature of a retrieval object included in a retrieval image; determining a candidate image library from an image library based on the first retrieval feature and the second retrieval feature; determining a fusion false alarm expectation of the similarity between an object included in each candidate image included in the candidate image library and the retrieval object; and determining an image of an object corresponding to a target fusion false alarm expectation that satisfies a predetermined condition in the fusion false alarm expectations. Through the present invention, the problem of low accuracy of retrieval images in the related art is solved, and the effect of improving the accuracy of retrieval images is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular, to a method, device, storage medium, and electronic device for image retrieval. Background Art

[0002] Image retrieval technology can also be referred to as image search by image technology, that is, feature extraction is performed on the image to be retrieved, and it is compared with the corresponding types in the existing bottom library for features, and sorted according to a certain matching degree index to obtain the retrieval result. Person retrieval belongs to a sub-branch of image retrieval, and mainly obtains the retrieval result by extracting and comparing features such as the face and body of the retrieval target. Person retrieval has been widely used in the field of security at present.

[0003] However, in real monitoring scenarios, situations often occur where the features of the target person are missing or the feature reliability is poor due to occlusion, back view, posture, etc. of the target person. At this time, there are certain limitations in using single-modal information to retrieve the target person, and it is necessary to fuse multi-modal information to improve the retrieval reliability and retrieval effect. In related technologies, most multi-modal fusion person retrieval methods are implemented by summarizing the single-modal retrieval results. Its essence is still independent single-modal retrieval, and true fusion has not been achieved, resulting in inaccurate retrieved images.

[0004] It can be seen that there is a problem of low accuracy of retrieved images in related technologies.

[0005] For the above problems existing in related technologies, no effective solution has been proposed yet. Summary of the Invention

[0006] The embodiments of the present invention provide a method, device, storage medium, and electronic device for image retrieval, so as to at least solve the problem of low accuracy of retrieved images existing in related technologies.

[0007] According to an embodiment of the present invention, a method for image retrieval is provided, including: determining a first retrieval feature and a second retrieval feature of a retrieval object included in a retrieval image; determining a candidate image library from an image library based on the first retrieval feature and the second retrieval feature; determining a fusion false alarm expectation of the similarity between an object included in each candidate image included in the candidate image library and the retrieval object; and determining an image of an object corresponding to a target fusion false alarm expectation that satisfies a predetermined condition in the fusion false alarm expectations as a target image.

[0008] According to another embodiment of the present invention, there is provided an image retrieval device, including: a first determination module for determining a first retrieval feature and a second retrieval feature of a retrieval object included in a retrieval image; a second determination module for determining a candidate image library from an image library based on the first retrieval feature and the second retrieval feature; a third determination module for determining a fusion false alarm expectation of the similarity between an object included in each candidate image included in the candidate image library and the retrieval object; a fourth determination module for determining an image of an object corresponding to a target fusion false alarm expectation that satisfies a predetermined condition in the fusion false alarm expectations as a target image.

[0009] According to still another embodiment of the present invention, there is also provided a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any one of the above are implemented.

[0010] According to still another embodiment of the present invention, there is also provided an electronic device including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0011] Through the present invention, a first retrieval feature and a second retrieval feature of a retrieval object included in a retrieval image are determined, a candidate image library is obtained from an image library based on the first retrieval feature and the second retrieval feature, a fusion false alarm expectation of the similarity between an object included in each candidate image included in the candidate image library and the retrieval object is determined, and an image of an object corresponding to a target fusion false alarm expectation that satisfies a predetermined condition in the fusion false alarm expectations is determined as a target image. Since a candidate image library can be determined according to the first retrieval feature and the second retrieval feature, and a fusion false alarm expectation parameter is introduced, and a target image is determined according to the fusion false alarm expectation parameter. That is, multiple features are fused to determine the candidate image library, and a target image is determined from the candidate image library according to the fusion false alarm expectation parameter. Therefore, the problem of low accuracy of retrieved images in the related art can be solved, and the effect of improving the accuracy of retrieved images can be achieved. Description of the Drawings

[0012] Figure 1 is a hardware structural block diagram of a mobile terminal of an image retrieval method according to an embodiment of the present invention;

[0013] Figure 2 is a flowchart of an image retrieval method according to an embodiment of the present invention;

[0014] Figure 3 is a flowchart of an image retrieval method according to a specific embodiment of the present invention;

[0015] Figure 4It is a structural block diagram of an image retrieval device according to an embodiment of the present invention. Detailed implementation manners

[0016] In the following, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with embodiments.

[0017] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence.

[0018] The method embodiments provided in the embodiments of the present application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 It is a hardware structural block diagram of a mobile terminal of an image retrieval method according to an embodiment of the present invention. As Figure 1 shown, the mobile terminal may include one or more ( Figure 1 only one is shown in the figure) processors 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Among them, the above-mentioned mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned mobile terminal. For example, the mobile terminal may further include more or fewer components than Figure 1 shown in the figure, or have a different configuration from Figure 1 shown in the figure.

[0019] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the image retrieval method in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above-mentioned method. The memory 104 may include a high-speed random access memory, and may further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories may be connected to the mobile terminal through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0020] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of a mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0021] In this embodiment, a method for retrieving images is provided. Figure 2 It is a flowchart of the method for retrieving images according to an embodiment of the present invention, as Figure 2 shown, and the process includes the following steps:

[0022] Step S202, determining a first retrieval feature and a second retrieval feature of a retrieval object included in the retrieval image;

[0023] Step S204, determining a candidate image library from the image library based on the first retrieval feature and the second retrieval feature;

[0024] Step S206, determining a fusion false alarm expectation of the similarity between an object included in each candidate image included in the candidate image library and the retrieval object;

[0025] Step S208, determining an image of an object corresponding to a target fusion false alarm expectation that meets a predetermined condition in the fusion false alarm expectations as a target image.

[0026] In the above embodiment, the retrieval object may include a person, an animal, an object, etc. When the retrieval object is a person, the first retrieval feature may be a face feature, and the second retrieval feature may be a human body feature. According to the first retrieval feature and the second retrieval feature, the candidate image library can be determined from the image library. Among them, the images in the candidate image library may be images whose features have a similarity greater than a threshold with the first retrieval feature or the second retrieval feature. Among them, the first retrieval feature and the second retrieval feature may be features extracted by a deep neural network model.

[0027] In the above embodiment, according to the first retrieval feature and the second retrieval feature, the candidate image library can be determined from the image library, and the fusion false alarm expectation of each image in the candidate image library can be determined. That is, the mutual information between multiple modalities is fully utilized and effectively combined, significantly improving the ranking of real samples. True fusion is achieved, and on the basis of improving the retrieval recall rate, the retrieval accuracy is improved.

[0028] Optionally, the execution subject of the above steps may be a background processor, or other devices with similar processing capabilities, or may also be a machine at least integrated with an image acquisition device and a data processing device. Among them, the image acquisition device may include a graphic acquisition module such as a camera, and the data processing device may include terminals such as a computer and a mobile phone, but is not limited thereto.

[0029] Through the present invention, a first retrieval feature and a second retrieval feature of a retrieval object included in a retrieval image are determined, a candidate image library is determined from an image library based on the first retrieval feature and the second retrieval feature, and a fusion false alarm expectation of the similarity between an object included in each candidate image included in the candidate image library and the retrieval object is determined. An image of an object corresponding to a target fusion false alarm expectation that satisfies a predetermined condition in the fusion false alarm expectation is determined as a target image. Since a candidate image library can be determined according to the first retrieval feature and the second retrieval feature, and a fusion false alarm expectation parameter is introduced, and the target image is determined according to the fusion false alarm expectation parameter. That is, multiple features are fused to determine the candidate image library, and the target image is determined from the candidate image library according to the fusion false alarm expectation parameter. Therefore, the problem of low accuracy of retrieval images in the related art can be solved, and the effect of improving the accuracy of retrieval images can be achieved.

[0030] In an exemplary embodiment, determining a candidate image library from an image library based on the first retrieval feature and the second retrieval feature includes: determining a first image from a first type of image included in the image library, where a first similarity between a feature of an object included in the first image and the first retrieval feature is greater than a first threshold, and determining a second image from a second type of image included in the image library, where a second similarity between a feature of an object included in the second image and the second retrieval feature is greater than a second threshold; determining the candidate image library based on the first image and the second image. In this embodiment, the first type of image may be a face image, and the second type of image may be a human body image. That is, the image library may include face images and human body images. In the image library, the face image and the human body feature may be correlated, and the face and human body features belonging to the same bottom library target can be found through indexing. That is, the face image and the human body image of the same person have the same identification information.

[0031] In the above embodiment, a deep neural network model may be used to extract corresponding face features and human body features for retrieval respectively. The extracted face and human body features are respectively calculated for similarity with face and human body bottom libraries, and respective retrieval results are obtained after similarity ranking. That is, the first retrieval feature is calculated for similarity with the feature of the first type of image to determine a first image with a first similarity greater than the first threshold. The second retrieval feature is calculated for similarity with the feature of the second type of image to determine a second image with a second similarity greater than the second threshold.

[0032] In the above embodiments, after determining the first image and the second image, a candidate image library can be determined based on the first image and the second image, and the fusion false alarm expectation of the similarity between the object in each candidate image included in the candidate image library and the retrieval object can be determined. The image of the object corresponding to the target fusion false alarm expectation that meets the predetermined condition in the fusion false alarm expectation is determined as the target image. That is, when the retrieval object is a person, the first image can be a face image, and the second image can be a human body image. The candidate image library can be determined based on the determined face image and human body image.

[0033] In an exemplary embodiment, determining the candidate image library based on the first image and the second image includes: determining the target quality score and resolution of the retrieval image; performing regression processing on the target quality score, the resolution, and the first similarity to obtain a regression score; and determining the candidate image library based on the first image and the second image when the regression score is less than a predetermined threshold. In this embodiment, when the retrieval object is a person, the face quality score of the retrieval image can be calculated, and regression is performed based on the three indicators of the face quality score, the face resolution, and the similarity score between the face features of the retrieval picture and the face features of the retrieval result. A threshold is set for the regression score. Retrieval pictures with a score greater than the preset threshold do not need to be fused and retrieved, and the face retrieval result can be directly output. Otherwise, enter the fusion retrieval stage. Since high-quality face features have strong distinctiveness and robustness, this scheduling decision method can give full play to the advantages of high-quality faces, and at the same time avoid the interference caused by the non-robustness of human body features under the condition of high-quality faces. When the quality of the face features is poor, it is considered that the reliability of using only the face retrieval result is low. At this time, some external features such as human body features can be used to assist the face in fusion retrieval to improve the robustness of the retrieval result.

[0034] In an exemplary embodiment, determining a candidate image library based on the first image and the second image includes: determining the union of the first image and the second image as a candidate target set; determining candidate images from the candidate target set; and determining the candidate image library based on the candidate images and the candidate target set. In this embodiment, after the first image and the second image are determined, the union of the first image and the second image can be determined as the candidate target set, then candidate images are determined from the candidate target set, and the candidate image library is determined according to the candidate images and the candidate target set. When the first retrieval feature is a face feature and the second retrieval feature is a human body feature, a face image and a human body image are obtained through preliminary retrieval, and their union is used as the candidate target set. The complementarity of face and human body features is fully utilized, and the integrity of the candidate target set is improved. Candidate images, that is, samples with relatively high confidence, can be screened out from the first image and the second image. The high-confidence matching targets are used as new retrieval samples to start a secondary retrieval, thereby determining the candidate image library. Among them, the screening condition for the high-confidence matching samples can be the base library samples that simultaneously meet the preset face threshold and human body threshold for the face feature and human body feature of the retrieval object. A secondary retrieval is performed on the high-confidence samples in the first retrieval result, and the result is used to expand the initial retrieval result. This method effectively complements the multi-modal retrieval results and provides a complete and reliable retrieval result for subsequent fusion false alarm expectation re-ranking.

[0035] In an exemplary embodiment, determining candidate images from the candidate target set includes: determining a third image with a similarity greater than a third threshold from the first image, and a fourth image with a similarity greater than a fourth threshold from the second image; determining a fifth image with the same identification information included in the third image and the fourth image; and determining the fifth image as the candidate image. In this embodiment, a third image corresponding to the similarity with a similarity greater than the third threshold included in the first similarity can be determined, that is, an image corresponding to the first similarity greater than the third threshold is determined from the first image, a fourth image corresponding to the second similarity greater than the fourth threshold is determined from the second image, a fifth image with the same identification information is determined from the third image and the fourth image, and the fifth image is determined as the candidate image.

[0036] In an exemplary embodiment, determining the candidate image library based on the candidate image and the candidate target set includes: determining a sixth image from the image library, where the similarity between the features of the object included in the sixth image and the features of the object included in the first type of image included in the candidate image is greater than a fifth threshold; determining a seventh image from the image library, where the similarity between the features of the object included in the seventh image and the features of the object included in the second type of image included in the candidate image is greater than a sixth threshold; determining an eighth image with the same identification information included in the sixth image and the seventh image; and determining the union of the candidate target set and the eighth image as the candidate image library. In this embodiment, the candidate image can be used for secondary retrieval to determine the sixth image and the seventh image from the image library. That is, the high-confidence matching samples are used as new retrieval targets, and the original face and human body databases are used as retrieval databases. The K-nearest neighbors of the face and the human body of the new retrieval targets are calculated respectively, and the intersection of the two K-nearest neighbors (i.e., the eighth image) is selected as the expanded candidate target. Finally, the union of the eighth image and the candidate target set is de-duplicated as the final new candidate image library. By performing secondary retrieval on the high-confidence matching samples, the probability that the real target falls into the candidate target set can be increased, and the integrity and robustness of the candidate target set can be improved.

[0037] In an exemplary embodiment, determining the fusion false alarm expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieval object includes: determining the first-type false alarm rate of the first retrieval feature and the features of each first-type candidate image of the first type included in the candidate image; determining the second-type false alarm rate of the second retrieval feature and the features of each second-type candidate image of the second type included in the candidate image; and determining the fusion false alarm expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieval object based on the first-type false alarm rate and the second-type false alarm rate. In this embodiment, the first-type false alarm rate of the first retrieval feature and the features of each first-type candidate image of the first type included in the candidate image can be determined respectively. The second-type false alarm rate of the second retrieval feature and the features of the second-type candidate image can be determined respectively. The fusion false alarm expectation is determined according to the first-type false alarm rate and the second-type false alarm rate.

[0038] In the above embodiment, when the first retrieval feature is a face feature and the second retrieval feature is a human body feature, the first-type candidate image of the first type is a face candidate image, and the second-type candidate image of the second type is a human body candidate image. That is, the first-type false alarm rate of the face feature of the retrieval object and the features of each face candidate image and the second-type false alarm rate of the human body feature of the retrieval object and the features of each human body candidate image are determined respectively.

[0039] In an exemplary embodiment, determine a first type false positive rate of the first retrieval feature and the features of each first type of first type candidate image included in the candidate image: determine a third similarity between the features of the first type candidate image and the first retrieval feature, and determine the first type false positive rate based on the third similarity and a first correspondence relationship between the similarity and the false positive rate determined in advance; determining a second type false positive rate of the second retrieval feature and the features of each second type of second type candidate image included in the candidate image includes: determining a fourth similarity between the features of the second type candidate image and the second retrieval feature, and determining the second type false positive rate based on the fourth similarity and a second correspondence relationship between the similarity and the false positive rate determined in advance; determine the fusion false positive expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieval object based on the first type false positive rate and the second type false positive rate. In this embodiment, the third similarity between the features of each first type of first type candidate image included in the candidate image library and the first retrieval feature can be determined, and the first type false positive rate of the first type candidate image can be determined according to the third similarity and the first correspondence relationship. The fourth similarity between the features of each second type of second type candidate image included in the candidate image library and the second retrieval feature can be determined, and the second type false positive rate of the second type candidate image can be determined according to the fourth similarity and the second correspondence relationship. Determine the fusion false positive expectation according to the first type false positive rate and the second type false positive rate.

[0040] In the above embodiment, when the first type is the face type and the second type is the human body type, the similarity of the face features calculated between the current retrieval object and a certain target in the bottom library is S f (i.e., the third similarity), and the similarity calculated for the human body features is S b (i.e., the fourth similarity). Furthermore, the calculated face false positive rate F f (i.e., the first type false positive rate) and the relationship between the similarity S f is F f = f f (S f ), and the relationship between the human body false positive rate (i.e., the second type false positive rate) and the similarity S b is F b = f b (S b ). Determine the fusion false positive expectation according to F f and F b .

[0041] In an exemplary embodiment, determining the fusion false alarm expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieved object based on the first type of false alarm rate and the second type of false alarm rate includes: determining a first quantity of the first type of images included in the image library; determining a second quantity of the second type of images included in the image library; performing the following operations for each of the first type of candidate images included in the candidate image library: determining a first product of the first quantity and the first type of false alarm rate of the first type of candidate images; performing the following operations for each of the second type of candidate images included in the candidate image library: determining a second product of the second quantity and the second type of false alarm rate of the second type of candidate images; determining the fusion false alarm expectation based on the first product and the second product. In this embodiment, when determining the fusion false alarm expectation, the first quantity of the first type of images and the second quantity of the second type of images included in the image library can be determined first. Determine the first product of the first type of false alarm rate of each first type of candidate image and the first quantity, and determine the second product of the second type of false alarm rate of each second type of candidate image and the second quantity. Determine the fusion false alarm expectation according to the first product and the second product. For example, when the retrieved object is a person, the first quantity of the first type of images, that is, the face feature images, is N, and the second quantity of the second type of images, that is, the human body feature images, is M, then the first product can be expressed as E f =N×f f (S f ), and the second product can be expressed as E b =M×f b (S b ).

[0042] In an exemplary embodiment, determining the fusion false alarm expectation based on the first product and the second product includes: determining target candidate images with the same identification information included in the first type of candidate images and the second type of candidate images, and determining the product of the first product and the second product as the fusion false alarm expectation; determining the first product corresponding to the other images except the target candidate images included in the first type of candidate images as the fusion false alarm expectation; determining the second product corresponding to the other images except the target candidate images included in the second type of candidate images as the fusion false alarm expectation. In this embodiment, when the first type of candidate images and the second type of candidate images include the same target candidate images, the product of the first product and the second product can be determined as the fusion false alarm expectation. Determine the first product corresponding to the images except the target candidate images included in the first type of candidate images as the fusion false alarm expectation. Determine the second product corresponding to the images except the target candidate images included in the second type of candidate images as the fusion false alarm expectation.

[0043] In the above embodiments, when the retrieval object is a person and the face and the human body model are completely independent, the false alarm expectation of the fusion system during single retrieval, that is, the false alarm expectation of the fusion can be expressed as E mix = N × f f (S f ) × M × f b (S b ). According to a series of experimental verifications, the face and the human body model are not completely independent, but the correlation between them is very weak. By leveraging the weak correlation between the face and the human body model, the false alarm expectation of the current retrieval target and each target object in the candidate target set is calculated, and it is used as a new metric for the retrieval task. Emix = N × ff(Sf) × M × fb(Sb) is the calculation method when the face and human body features of the retrieval target and the gallery target coexist. In real-world scenarios, the pictures collected often have the problem of missing modal features of the retrieval target due to reasons such as occlusion, back view, and low resolution. In this case, only the false alarm expectation of the single modality needs to be calculated.

[0044] In the above embodiments, when the false alarm expectation of the fusion is determined, the candidate image library can be re-ranked according to the false alarm expectation of the fusion. That is, the smaller the false alarm expectation of the fusion, the greater the possibility of hitting the true target, and the higher the retrieval ranking. The re-ranked new retrieval result is output as the final retrieval result. That is, the false alarm expectations of the fusion can be sorted in ascending order, and the first N images can be determined as the target images, or the images with false alarm expectations of the fusion less than the predetermined false alarm expectation of the fusion can be determined as the target images.

[0045] In the above embodiments, a suitable modal feature can be selected according to the actual situation, and only the relationship between the false alarm rate and the similarity of the modality needs to be replaced. In addition, in terms of the number of modalities, for two or more modalities, the above method can be used to determine the false alarm expectation of the fusion.

[0046] The following describes the image retrieval method in combination with specific embodiments:

[0047] Figure 3 is the flowchart of the image retrieval method according to a specific embodiment of the present invention. As Figure 3 shown, this process includes:

[0048] Step S302, extracting face and human body features from the image to be retrieved (corresponding to the above retrieval image): The corresponding face features and human body features are respectively extracted from the image to be retrieved by using a deep neural network.

[0049] Step S304, face and human body retrieval: Calculate the similarity between the extracted face and human body features and the face and human body databases respectively. After sorting the similarities, the respective retrieval results are obtained. In the face and human body databases, the face and human body features of a certain database target are correlated with each other, and the face and human body features belonging to the same database target can be found through indexing.

[0050] Step S306, determine whether fusion is required. If the judgment result is yes, execute Step S308; if the judgment result is no, execute Step S316. The judgment method is as follows:

[0051] Calculate the face quality score of the current retrieved image, perform regression based on these three indicators: the face quality score, the face resolution, and the similarity score between the face features of the retrieved image and the face features of its retrieval result, and set a threshold for the regression score. The retrieved images that meet the preset threshold do not need to be fused for retrieval, and the face retrieval result can be directly output. Otherwise, enter the fusion retrieval stage.

[0052] Since high-quality face features have strong distinctiveness and robustness, this scheduling decision method can give full play to the advantages of high-quality faces, and at the same time avoid the interference caused by the non-robust human body features under the condition of high-quality faces. When the quality of face features is poor, it is considered that the reliability of using only the face retrieval result is low. At this time, some external features such as human body features can be used to assist the face in fusion retrieval to improve the robustness of the retrieval result.

[0053] Step S308, candidate target set screening: If in the face retrieval result and the human body retrieval result obtained in S304, take the union of the two results as the candidate target set. This method makes full use of the complementarity of face and human body features and improves the integrity of the candidate target set.

[0054] Step S310, candidate target set expansion: For the candidate target set constructed in Step S308, select the matching samples with higher confidence, and use the high-confidence matching targets as new retrieval samples to start a secondary retrieval. The screening conditions for high-confidence matching samples are: database samples that simultaneously meet the preset face and human body thresholds for the face and human body features of the retrieval target. The secondary retrieval steps are: Take the high-confidence matching sample as the new retrieval target, and the original face and human body databases as the retrieval databases. Calculate the K-nearest neighbors of the face and human body of the new retrieval target respectively, and select the intersection of the two K-nearest neighbors as the expanded candidate target. Finally, remove duplicates from the union of the candidate target set expanded through the secondary retrieval and the candidate target set screened in Step S308 as the final new candidate target set. By performing a secondary retrieval on high-confidence matching samples, the probability of the true target falling into the candidate target set can be increased, and the integrity and robustness of the candidate target set can be improved.

[0055] Step S312: For the selected candidate target set, calculate the fusion false alarm expectation (corresponding to the above-mentioned fusion false alarm expectation) for each sample, and use it as a new metric for measuring the retrieval ranking. The specific calculation method of the fusion false alarm expectation is as follows:

[0056] Assume that the size of the face feature database is N, the size of the human body feature database is M, and the face feature similarity calculated currently according to a certain retrieval target and a certain target in the database is S f , and the similarity calculated for the human body feature is S b . The face false alarm rate F f calculated based on the validation set and the similarity S f have the relationship that F f = f f (S f ), and the relationship between the human body false alarm rate and the similarity S b is that F b = f b (S b ).

[0057] At this time, the mathematical expectation of false alarms occurring in a single retrieval of the face recognition system is as shown in formula (5-1), and the mathematical expectation of false alarms occurring in a single retrieval of the human body system is as shown in formula (5-2).

[0058] E f = N × f f (S f ) (5-1)

[0059] E b = M × f b (S b ) (5-2)

[0060] When the face and human body models are completely independent, the fusion false alarm expectation of the fusion system in a single retrieval is as shown in formula (5-3).

[0061] E mix = N × f f (S f ) × M × f b (S b ) (5-3)

[0062] Based on a series of experimental verifications, it is known that the face and human body models are not completely independent, but the correlation between them is very weak. By leveraging the weak correlation between the face and human body models, calculate the fusion false alarm expectation of the current retrieval target and each target object in the candidate target set, and use it as a new metric for the retrieval task.

[0063] The above conditions are the calculation methods when the face and body features of the retrieval target and the database target exist simultaneously. In real-world scenarios, the images collected often have the problem of missing modal features of the retrieval target due to reasons such as occlusion, back view, and low resolution. In this case, only the false positive expectation of a single modality needs to be calculated.

[0064] Step S314, re-rank the retrieval results: According to the fused false positive expectations of each candidate target set in S312, re-rank the candidate databases. The smaller the fused false positive expectation, the greater the possibility of hitting the true target, and the higher the retrieval ranking.

[0065] Step S316, output the re-ranked new retrieval results as the final retrieval results.

[0066] In the foregoing embodiments, face and body feature extraction and retrieval are performed based on the image to be retrieved. Subsequently, it is jointly determined whether to enable the fusion retrieval mode according to the face retrieval results, face resolution, and face quality. If fusion is not required, the face retrieval results can be directly output. Otherwise, the fusion retrieval mode is enabled. If the fusion retrieval mode is enabled, candidate target sets are screened and expanded according to the face and body retrieval results. Subsequently, the fused false positive expectations of the expanded candidate objects are calculated respectively, and the fused false positive expectations are used as a new metric to re-rank the candidate target sets. The re-ranked results are used as the final retrieval results. This application makes full use of the mutual information between multiple modalities and realizes true fusion. On the basis of improving the recall rate of the retrieval results, the accuracy of the retrieval is effectively improved. In addition, the method has good scalability and portability. Moreover, the high-confidence samples in the initial multi-modal retrieval results are retrieved again, and the results are used to expand the initial retrieval results. This method effectively complements the multi-modal retrieval results and provides complete and reliable retrieval results for the subsequent fused false positive expectation re-ranking.

[0067] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0068] In this embodiment, a retrieval device for images is further provided. This device is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated here. As used hereinafter, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0069] Figure 4 is a structural block diagram of a retrieval device for images according to an embodiment of the present invention. As Figure 4 shown, the device includes:

[0070] A first determination module 402, configured to determine a first retrieval feature and a second retrieval feature of a retrieval object included in a retrieval image;

[0071] A second determination module 404, configured to determine a candidate image library from an image library based on the first retrieval feature and the second retrieval feature;

[0072] A third determination module 406, configured to determine a fusion false alarm expectation of the similarity between an object included in each candidate image included in the candidate image library and the retrieval object;

[0073] A fourth determination module 408, configured to determine an image of an object corresponding to a target fusion false alarm expectation that satisfies a predetermined condition in the fusion false alarm expectation as a target image.

[0074] In an exemplary embodiment, the second determination module 404 may implement determining a candidate image library from an image library based on the first retrieval feature and the second retrieval feature in the following manner: determining a first image from the first type of images included in the image library, where a first similarity between a feature of an object included in the first image and the first retrieval feature is greater than a first threshold, and determining a second image from the second type of images included in the image library, where a second similarity between a feature of an object included in the second image and the second retrieval feature is greater than a second threshold; determining a candidate image library based on the first image and the second image.

[0075] In an exemplary embodiment, the second determination module 404 may implement determining a candidate image library based on the first image and the second image in the following manner: determining a target quality score and a resolution of the retrieval image; performing a regression process on the target quality score, the resolution, and the first similarity to obtain a regression score; and determining a candidate image library based on the first image and the second image when the regression score is less than a predetermined threshold.

[0076] In an exemplary embodiment, the second determination module 404 may implement determining a candidate image library based on the first image and the second image in the following manner: determining the union of the first image and the second image as a candidate target set; determining candidate images from the candidate target set; and determining the candidate image library based on the candidate images and the candidate target set.

[0077] In an exemplary embodiment, the second determination module 404 may implement determining candidate images from the candidate target set in the following manner: determining a third image with a similarity greater than a third threshold from the first image, and determining a fourth image with a similarity greater than a fourth threshold from the second image; determining a fifth image with the same identification information included in the third image and the fourth image; and determining the fifth image as the candidate image.

[0078] In an exemplary embodiment, the second determination module 404 may implement determining the candidate image library based on the candidate images and the candidate target set in the following manner: determining a sixth image from the image library, where the similarity between the features of the object included in the sixth image and the features of the object included in the first type of image included in the candidate images is greater than a fifth threshold; determining a seventh image from the image library, where the similarity between the features of the object included in the seventh image and the features of the object included in the second type of image included in the candidate images is greater than a sixth threshold; determining an eighth image with the same identification information included in the sixth image and the seventh image; and determining the union of the candidate target set and the eighth image as the candidate image library.

[0079] In an exemplary embodiment, the third determination module 406 may implement determining the fusion false alarm expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieval object in the following manner: determining the first type of false alarm rate of the first retrieval feature and the features of each first type of first type candidate image included in the candidate image; determining the second type of false alarm rate of the second retrieval feature and the features of each second type of second type candidate image included in the candidate image; and determining the fusion false alarm expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieval object based on the first type of false alarm rate and the second type of false alarm rate.

[0080] In an exemplary embodiment, the third determination module 406 may determine the first type false alarm rate of the feature of the first retrieval feature and each first type of first type candidate image included in the candidate image in the following manner: determine the third similarity between the feature of the first type candidate image and the first retrieval feature, and determine the first type false alarm rate based on the third similarity and a pre-determined first correspondence between the similarity and the false alarm rate; the third determination module 406 may determine the second type false alarm rate of the feature of the second retrieval feature and each second type of second type candidate image included in the candidate image in the following manner: determine the fourth similarity between the feature of the second type candidate image and the second retrieval feature, and determine the second type false alarm rate based on the fourth similarity and a pre-determined second correspondence between the similarity and the false alarm rate.

[0081] In an exemplary embodiment, the third determination module 406 may determine the fused false alarm expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieval object based on the first type false alarm rate and the second type false alarm rate in the following manner: determine the first quantity of the first type of images included in the image library; determine the second quantity of the second type of images included in the image library; for each of the first type candidate images included in the candidate image library, perform the following operations: determine the first product of the first quantity and the first type false alarm rate of the first type candidate image; for each of the second type candidate images included in the candidate image library, perform the following operations: determine the second product of the second quantity and the second type false alarm rate of the second type candidate image; determine the fused false alarm expectation based on the first product and the second product.

[0082] In an exemplary embodiment, the third determination module 406 may determine the fused false alarm expectation based on the first product and the second product in the following manner: determine the target candidate images with the same identification information included in the first type candidate images and the second type candidate images, and determine the product of the first product and the second product as the fused false alarm expectation; determine the first product corresponding to the other images except the target candidate images included in the first type candidate images as the fused false alarm expectation; determine the second product corresponding to the other images except the target candidate images included in the second type candidate images as the fused false alarm expectation.

[0083] It should be noted that the above-mentioned respective modules may be implemented by software or hardware. For the latter, it may be implemented in the following manner, but not limited thereto: the above-mentioned modules are all located in the same processor; or, the above-mentioned respective modules are separately located in different processors in any combination form.

[0084] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. Wherein, when the computer program is executed by a processor, the steps of the method described in any one of the above are implemented.

[0085] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs that can store computer programs.

[0086] An embodiment of the present invention further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0087] In an exemplary embodiment, the above electronic device may further include a transmission device and an input / output device. Wherein, the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0088] Specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be elaborated herein.

[0089] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order from here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. In this way, the present invention is not limited to any specific combination of hardware and software.

[0090] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for retrieving an image, characterized in that, Including: Determining a first retrieval feature and a second retrieval feature of a retrieval object included in a retrieval image; Determining a candidate image library from an image library based on the first retrieval feature and the second retrieval feature; Determining a fusion false alarm expectation of the similarity between an object included in each candidate image included in the candidate image library and the retrieval object; Determining an image of an object corresponding to a target fusion false alarm expectation that satisfies a predetermined condition in the fusion false alarm expectations as a target image; Determining the fusion false alarm expectation of the similarity between an object included in each candidate image included in the candidate image library and the retrieval object includes: determining a first type false alarm rate of the first retrieval feature and the feature of each first type candidate image included in the candidate image; Determining a second type false alarm rate of the second retrieval feature and the feature of each second type candidate image included in the candidate image; determining the fusion false alarm expectation of the similarity between an object included in each candidate image included in the candidate image library and the retrieval object based on the first type false alarm rate and the second type false alarm rate.

2. The method according to claim 1, characterized in that, Determining a candidate image library from an image library based on the first retrieval feature and the second retrieval feature includes: Determining a first image from the first type of images included in the image library, where a first similarity between the feature of the object included in the first image and the first retrieval feature is greater than a first threshold, and determining a second image from the second type of images included in the image library, where a second similarity between the feature of the object included in the second image and the second retrieval feature is greater than a second threshold; Determining a candidate image library based on the first image and the second image.

3. The method according to claim 2, wherein Determining a candidate image library based on the first image and the second image includes: Determining a target quality score and a resolution of the retrieval image; Performing a regression process on the target quality score, the resolution, and the first similarity to obtain a regression score; When the regression score is less than a predetermined threshold, determining a candidate image library based on the first image and the second image.

4. The method according to claim 2, wherein Determining a candidate image library based on the first image and the second image includes: Determining a union of the first image and the second image as a candidate target set; Determining candidate images from the candidate target set; Determining the candidate image library based on the candidate images and the candidate target set.

5. The method according to claim 4, wherein Determining candidate images from the candidate target set includes: Determining a third image with a similarity greater than a third threshold from the first image, and determining a fourth image with a similarity greater than a fourth threshold from the second image; Determining a fifth image with the same identification information included in the third image and the fourth image; Determining the fifth image as the candidate image.

6. The method according to claim 4, wherein Determining the candidate image library based on the candidate images and the candidate target set includes: Determining a sixth image from the image library, where a similarity between the feature of the object included in the sixth image and the feature of the object included in the first type of image included in the candidate image is greater than a fifth threshold; Determine a seventh image from the image library, where the similarity between the features of the object included in the seventh image and the features of the object included in the second type of image included in the candidate image is greater than a sixth threshold; Determine an eighth image whose identification information included in the sixth image is the same as that included in the seventh image; Determine the union of the candidate target set and the eighth image as the candidate image library.

7. The method according to claim 1, wherein: Determining the first type of false alarm rate of the features of the first retrieval feature and each first type of first type candidate image included in the candidate image includes: determining the third similarity between the features of the first type candidate image and the first retrieval feature, and determining the first type of false alarm rate based on the third similarity and a first correspondence relationship between the similarity and the false alarm rate determined in advance; Determining the second type of false alarm rate of the features of the second retrieval feature and each second type of second type candidate image included in the candidate image includes: determining the fourth similarity between the features of the second type candidate image and the second retrieval feature, and determining the second type of false alarm rate based on the fourth similarity and a second correspondence relationship between the similarity and the false alarm rate determined in advance.

8. The method according to claim 1, wherein Based on the first type of false alarm rate and the second type of false alarm rate, determining the fusion false alarm expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieved object includes: Determine a first quantity of the first type of image included in the image library; Determine a second quantity of the second type of image included in the image library; Perform the following operations for each of the first type candidate images included in the candidate image library: determine a first product of the first quantity and the first type of false alarm rate of the first type candidate image; Perform the following operations for each of the second type candidate images included in the candidate image library: determine a second product of the second quantity and the second type of false alarm rate of the second type candidate image; Determine the fusion false alarm expectation based on the first product and the second product.

9. The method according to claim 8, wherein Determining the fusion false alarm expectation based on the first product and the second product includes: Determine a target candidate image whose identification information included in the first type candidate image and the second type candidate image is the same, and determine the product of the first product and the second product as the fusion false alarm expectation; Determine the first product corresponding to the other images except the target candidate image included in the first type candidate image as the fusion false alarm expectation; Determine the second product corresponding to the other images except the target candidate image included in the second type candidate image as the fusion false alarm expectation.

10. An image retrieval device, characterized in that, Including: A first determination module, configured to determine a first retrieval feature and a second retrieval feature of a retrieved object included in a retrieved image; A second determination module, configured to determine a candidate image library from the image library based on the first retrieval feature and the second retrieval feature; A third determination module, configured to determine the fusion false alarm expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieved object; A fourth determination module, configured to determine the image of the object corresponding to the target fusion false alarm expectation that satisfies a predetermined condition in the fusion false alarm expectations as the target image; The third determination module determines the fusion false alarm expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieved object in the following manner: determining the first type false alarm rate of the first retrieval feature and the feature of each first type candidate image included in the candidate image; determining the second type false alarm rate of the second retrieval feature and the feature of each second type candidate image included in the candidate image; determining the fusion false alarm expectation of the similarity between the object included in each candidate image included in the candidate image library and the retrieved object based on the first type false alarm rate and the second type false alarm rate.

11. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

12. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method described in any one of claims 1 to 9.

Images