Image retrieval method, device, and storage medium

Abstract

This application discloses an image retrieval method, device, and storage medium. The method includes: determining a target detection model corresponding to the scene type from a pre-constructed set of detection models based on the scene type of the acquired base images; performing target detection processing on each base image according to the target detection model to obtain target objects in each base image; responding to a received image to be retrieved, performing target detection processing on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved; and determining the target image from each base image based on the feature similarity between the object to be retrieved and each target object. This approach can improve the efficiency and accuracy of image retrieval.

Description

Technical Field

[0001] This application relates to the field of data retrieval technology, and in particular to an image retrieval method, device, and storage medium. Background Technology

[0002] Image retrieval technology is mainly divided into text-based image retrieval and image-based image retrieval.

[0003] In image-based image retrieval scenarios, the typical approach is to first acquire a target image, and then find images from the set of images to be retrieved that have a similarity higher than a similarity threshold to the target image as the retrieval result. This technique has wide applications in fields such as object detection and security.

[0004] However, current image retrieval schemes still have some problems that have not been effectively solved, such as poor retrieval results for small targets in images, poor retrieval results for local regions of target objects in images, and a large number of irrelevant image retrieval results, which makes the accuracy of image retrieval methods low in some application scenarios. Summary of the Invention

[0005] This application provides at least one image retrieval method, apparatus, device, and computer-readable storage medium.

[0006] A first aspect of this application provides an image retrieval method, comprising: determining a target detection model corresponding to a scene type from a pre-constructed set of detection models based on the scene type of the acquired base images; performing target detection processing on each base image according to the target detection model to obtain target objects in each base image; responding to a received image to be retrieved, performing target detection processing on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved; and determining target images from each base image based on the feature similarity between the object to be retrieved and each target object.

[0007] In one embodiment, determining the target detection model corresponding to the scene type from a pre-built set of detection models based on the scene type of the acquired base image includes: inputting the scene type of the base image into a pre-trained detection model decision agent to obtain the target detection model determined by the detection model decision agent from the set of detection models based on the scene type.

[0008] In one embodiment, the step of performing target detection processing on each database image according to the target detection model to obtain target objects in each database image includes: performing target detection on each database image according to the target detection model to obtain candidate objects in each database image; inputting each candidate object and target selection conditions into a pre-trained target selection decision agent to obtain the target object determined by the target selection decision agent from each candidate object according to the target selection conditions.

[0009] In one embodiment, before determining the target image from each base database image based on the feature similarity between the object to be retrieved and each target object, the method further includes: performing feature extraction processing on each target object according to a pre-trained feature extraction model to obtain target feature information; performing feature extraction processing on the object to be retrieved according to the feature extraction model to obtain retrieval feature information; and determining the feature similarity between the object to be retrieved and each target object based on the target feature information and the retrieval feature information.

[0010] In one embodiment, the step of performing target detection processing on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved includes: obtaining a target region in the image to be retrieved; performing target detection on the target region according to the target detection model to obtain a target detection result; in response to the target detection result indicating that the object to be retrieved does not exist in the target region, performing target detection on the image to be retrieved according to the target detection model to obtain neighboring objects in the image to be retrieved; and determining the object to be retrieved from each neighboring object according to the distance between the region where the neighboring object is located and the target region.

[0011] In one embodiment, after performing target detection on the target region according to the target detection model and obtaining the target detection result, the method further includes: in response to the target detection result indicating that the object to be retrieved exists in the target region, extracting the feature information of the object to be retrieved to obtain the feature to be retrieved; obtaining the feature similarity between the feature to be retrieved and the target features of each target object; and determining the target image from each base database image according to the feature similarity.

[0012] In one embodiment, after determining the object to be retrieved from each neighboring object based on the distance between the area where the neighboring object is located and the target area, the method further includes: extracting feature information of the target area to obtain a first feature; and extracting feature information of the neighboring objects to obtain a second feature; obtaining a first feature similarity between the first feature and the target features of each target object, and obtaining a second feature similarity between the second feature and the target features of each target object; and determining the target image from each base database image based on the first feature similarity and the second feature similarity.

[0013] In one embodiment, determining the target image from each database image based on the first feature similarity and the second feature similarity includes: determining a first image from each database image based on the first feature similarity and a preset sorting algorithm; determining a second image from each database image based on the second feature similarity and the preset sorting algorithm; performing deduplication processing on duplicate images between each first image and each second image to obtain remaining images; and determining the target image from each remaining image based on the feature similarity corresponding to each remaining image.

[0014] A second aspect of this application provides an image retrieval apparatus, comprising: a model selection module, configured to determine a target detection model corresponding to a scene type from a pre-constructed set of detection models based on the scene type of the acquired base images; a base image construction module, configured to perform target detection processing on each base image according to the target detection model to obtain target objects in each base image; a target detection module, configured to perform target detection processing on the received image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved; and an image retrieval module, configured to determine target images from each base image based on the feature similarity between the object to be retrieved and each target object.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the image retrieval method described above.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described image retrieval method.

[0017] The above scheme acquires base images and, based on the scene type of the acquired base images, determines the corresponding target detection model from a pre-built set of detection models. This allows for the selection of appropriate target detection models for different application scenarios, improving the accuracy of target object detection in the current application scenario. Target detection processing is performed on each base image using the target detection model to obtain the target objects in each base image, forming a base database. This reduces invalid targets in each base image and improves database quality. In response to a received image to be retrieved, target detection processing is performed on the image to be retrieved using the target detection model to obtain the target object in the image to be retrieved. Then, image retrieval is performed based on the feature similarity between the target object and each target object, determining the target image from each base image in the base database. This improves the efficiency and accuracy of the image retrieval process.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a flowchart illustrating an exemplary embodiment of the image retrieval method of this application;

[0021] Figure 2 This is an exemplary S1 data set construction flowchart in the image retrieval method of this application;

[0022] Figure 3 This is a flowchart illustrating the construction of an exemplary detection model decision agent in the image retrieval method of this application;

[0023] Figure 4 This is an exemplary S2 data set construction flowchart in the image retrieval method of this application;

[0024] Figure 5 This is a flowchart illustrating the construction of an exemplary target selection decision-making agent in the image retrieval method of this application;

[0025] Figure 6 This is a flowchart illustrating the construction of an exemplary database in the image retrieval method of this application;

[0026] Figure 7 This is an exemplary application flowchart of the image retrieval method in this application;

[0027] Figure 8This is a block diagram illustrating an image retrieval apparatus according to an exemplary embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0029] Figure 10 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0030] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0031] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0032] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0033] Please see Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of the image retrieval method of this application. Specifically, it may include the following steps:

[0034] Step S110: Based on the scene type of the acquired base image, determine the target detection model corresponding to the scene type from the pre-built detection model set.

[0035] The base image refers to images that have been pre-collected and can be used as a reference during the image retrieval process.

[0036] Scene type refers to the type of application scenario included in the image content. For example, scene types can include, but are not limited to, traffic scenes, market scenes, and industrial park / workshop scenes. Since the target objects to be detected typically differ in different scenarios, the level of expertise of various object detection models varies depending on the application scenario. Therefore, there are usually certain differences between object detection models corresponding to different application scenarios. This application can store one or more object detection models that support deployment in a detection model set. Then, based on the scene type of the acquired base image, the corresponding object detection model can be determined from the pre-built detection model set.

[0037] Step S120: Perform target detection processing on each base database image according to the target detection model to obtain the target objects in each base database image.

[0038] It's important to note that common image retrieval methods typically rely on the similarity between a database of images and the image to be retrieved. Essentially, these methods are based on the global image content of both database and target images. However, these methods are less effective at retrieving small targets within an image. Because the target object represents a small portion of the overall image, its information contribution is also limited. If the image also contains many irrelevant interfering targets, the accuracy of image retrieval will be significantly reduced. For example, in traffic scenarios, it's necessary to detect not only vehicles but also small items such as helmets, handheld objects, and backpacks belonging to non-motorized vehicle drivers.

[0039] This application uses an object detection model to perform object detection processing on each base database image, so as to detect in advance the target objects in each base database image that can be used as references in the image retrieval process, and store them in the base database (hereinafter referred to as the database). Therefore, when a retrieval image is received subsequently, the target objects in the base database can be compared with the retrieval image, thereby improving the image retrieval efficiency.

[0040] Optionally, in the specific implementation of this application, the local image of the target object in the base image may be saved to the database, and / or the feature information of the target object may be extracted and saved to the database.

[0041] Step S130: In response to the received image to be retrieved, target detection processing is performed on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved.

[0042] Based on the steps described above, after establishing the database according to the methods provided in the previous steps, the database can be used in the image retrieval process.

[0043] For example, in response to the received image to be retrieved, the same object detection model used when building the database can be used for object detection processing to obtain the object to be retrieved in the image. Similarly, the object to be retrieved can also be considered as a local image of the object to be retrieved within the image. Thus, image retrieval processing can be performed by comparing and analyzing the object to be retrieved with each target object in the database.

[0044] Step S140: Determine the target image from each base database image based on the feature similarity between the object to be retrieved and each target object.

[0045] In conjunction with the foregoing steps, the method in this application for determining target images similar to the image to be retrieved from each base database image may involve comparing the similarity of feature information between each target object and the object to be retrieved, determining the target object similar to the object to be retrieved from each target object based on the feature similarity, and determining the base database image corresponding to the target object similar to the object to be retrieved as the target image similar to the image to be retrieved.

[0046] If the database contains pre-stored feature information of the target objects, then before step S140, only feature information extraction of the image to be retrieved is required, followed by calculating the feature similarity between the object to be retrieved and each target object. If the database does not contain feature information of the target objects, then before step S140, feature information extraction of each target object and the object to be retrieved is required, followed by calculating the feature similarity between the object to be retrieved and each target object.

[0047] It should be noted that this application does not limit the number of target images; there can be zero, one, or more target images. For example, if the feature similarity between all target objects and the object to be retrieved is less than the similarity threshold, then it is determined that there are no target objects similar to the object to be retrieved, i.e., there are zero target images. As another example, among images with feature similarity greater than or equal to the similarity threshold, the target object with the highest feature similarity can be selected as the target object similar to the object to be retrieved, thus obtaining one target image. Alternatively, among images with feature similarity greater than or equal to the similarity threshold, several target objects can be selected based on feature similarity as the target objects similar to the object to be retrieved, thus obtaining multiple target images.

[0048] As can be seen, this application acquires base images and, based on the scene type of the acquired base images, determines the target detection model corresponding to the scene type from a pre-constructed set of detection models. This allows for the selection of appropriate target detection models for different application scenarios, improving the accuracy of target object detection in the current application scenario. By performing target detection processing on each base image according to the target detection model, the target objects in each base image are obtained, forming a base database. This reduces invalid targets in each base image and improves database quality. In response to a received image to be retrieved, target detection processing is performed on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image. Then, image retrieval can be performed based on the feature similarity between the object to be retrieved and each target object, determining the target image from each base image in the base database. This improves the efficiency and accuracy of the image retrieval process.

[0049] Based on the above embodiments, this application embodiment describes the steps of determining the target detection model corresponding to the scene type from a pre-built set of detection models according to the scene type of the acquired base image. Specifically, the method of this embodiment includes the following steps:

[0050] The scene type of the base image is input into the pre-trained detection model decision agent, and the target detection model is determined by the detection model decision agent from the detection model set according to the scene type.

[0051] In conjunction with the foregoing embodiments, the target detection model used for target detection in this application can be selected as needed based on the scene type of the base database images.

[0052] For example, the method for selecting an object detection model can be to determine the object detection model corresponding to the scene type from a set of detection models containing multiple types of detection models by using the scene type of the base image and a pre-set mapping relationship. The scene type of the base image can be pre-annotated information and / or information obtained by performing scene detection on the base image; this is not limited here. All detection models in the detection model set are pre-trained. The pre-set mapping relationship can include the mapping relationship between scene type and detection model, which can be, but is not limited to, a one-to-one relationship or a many-to-one relationship.

[0053] Another example is that the method for selecting an object detection model can be performed by a pre-built detection model decision agent. For instance, some or all of the base database images are input into the detection model decision agent, which analyzes the received image data to obtain the scene type of the base database images and determines the corresponding object detection model from the detection model set. Alternatively, the scene type of the base database images can be input into the detection model decision agent, which then determines the corresponding object detection model from the detection model set.

[0054] It should also be noted that when selecting the target detection model corresponding to the database images, in addition to referring to the method in the aforementioned embodiments to select the target detection model corresponding to the scene type of the database images, more image data can be referenced for selection to ensure that the most suitable detection model can be obtained. This image data may include, but is not limited to, the scene type, image resolution, size information, and category information of the target objects in the database images. The target objects may include the main target in the database images that need attention in the current scene type, and / or local targets (parts of the main target) in the database images that need attention in the current scene type, which can be specifically set according to the specific image retrieval requirements. For example, the detection model's decision-making intelligence may select the corresponding target detection model based on the scene type of the image and the category of the local target that needs attention in the image.

[0055] Specifically, you can refer to, for example Figure 2 As shown, Figure 2 This is an exemplary flowchart of the S1 data set construction process in the image retrieval method of this application. In the process of building the detection model decision agent, initial images of different scene types and image data of these initial images can be obtained first to obtain the initial image dataset S1 = {data1, data2, ..., dataN}.

[0056] Each `data` represents the image data of an initial image, for example, `data1 = {weight1, height1, scene1, Sobj1, SsubObj1, maxSize, minSize}`. Here, `weight1` represents the width of the image corresponding to this data point, `height1` represents the height of the image corresponding to this data point, `scene1` represents the scene type corresponding to this data point, `Sobj1` represents the main target category to be focused on within the corresponding scene type, `SsubObj1` represents the local target category to be focused on within the corresponding scene type, `maxSize` represents the maximum supported size of the local targets within the corresponding scene type, and `minSize` represents the minimum supported size of the local targets within the corresponding scene type. An initial image may include multiple main targets and multiple local targets, `Sobj1 = {obj1, obj2, ..., objm}`, and `SsubObj1 = {subobj1, subobj2, ..., subobjk}`.

[0057] Therefore, a corresponding detection model can be pre-defined for each image data, for example, S1 = {{data1,model1},{data2,model2},...,{datai,modeli}}. Here, model1 is the target detection model to be selected for the corresponding scene type of data1.

[0058] Then, you can refer to, for example Figure 3 As shown, Figure 3 This is a flowchart illustrating the construction of an exemplary detection model decision agent in the image retrieval method of this application. The detection model decision agent can be constructed based on the dataset obtained from the above implementation process. The detection model decision agent mainly consists of several classifiers, such as support vector machines, decision trees, random forests, K-nearest neighbors, etc., which are not limited here. These are trained using the data in the dataset S1 mentioned above to obtain several decision models, and the output of each model is summarized, such as: out = {w1*out1, w2*out2, ..., wn*outn}. Here, w1, ..., wn are the weights of each output, and out1, ..., outn are the outputs of each detection model. The value of each weight can be set to 1 / n. Finally, the output with the highest weight is taken as the final output, obtaining the detection model selected by the agent.

[0059] In summary, by using the steps provided in the example methods above, including but not limited to those steps, to build the detection model decision agent using the initial image, in subsequent application scenarios, the detection model decision agent can be input into the database images to obtain the target detection model determined by the detection model decision agent from the detection model set based on the scene type of the database images.

[0060] Based on the above embodiments, this application embodiment describes the steps of performing target detection processing on each database image according to the target detection model to obtain the target objects in each database image. Specifically, the method of this embodiment includes the following steps:

[0061] The target detection model is used to detect targets in each database image to obtain candidate objects in each database image. The candidate objects and target selection conditions are then input into a pre-trained target selection decision agent to obtain the target objects determined by the target selection decision agent from the candidate objects according to the target selection conditions.

[0062] In conjunction with the foregoing embodiments, after the target detection model performs target detection on each base database image, the obtained targets may include target objects and / or interfering targets. In this case, the detected targets are taken as candidate objects, and further screening of the candidate objects is required to obtain the target objects.

[0063] For example, the screening process for candidate objects in this application may involve comparing the image data of each candidate object (such as the width, height, and object category mentioned in the foregoing embodiments) with preset target screening conditions, determining the candidate objects that meet the target screening conditions as target objects, and determining the candidate objects that do not meet the target screening conditions as interference objects (interference targets).

[0064] In another exemplary embodiment, the candidate selection process in this application can involve inputting the candidate objects and their related data into a pre-built target selection decision-making agent. The target selection decision-making agent then selects the target object that meets the target selection criteria from among the candidate objects based on their related data. The target selection decision-making agent primarily consists of several differentiated neural networks, with the differences mainly lying in the number of network layers and nodes.

[0065] Optionally, in the process of comparing the data of each candidate object with the target screening conditions in this application, in addition to referring to the above example, the data richness of each candidate object can also be determined based on the amount of data obtained for each candidate object and the preset required data amount. By comparing the data richness of each candidate object with the preset richness threshold, candidate objects with a data richness greater than or equal to the richness threshold can be identified as target objects, and candidate objects with a richness less than the richness threshold can be identified as interference objects. For example, after target detection processing, each candidate object has its own corresponding image data (such as resolution, target category, target size, etc. mentioned in the previous embodiments, which are not limited here). If the preset required data amount is 3 (resolution, target category, target size), and the data amount of a detected candidate object is 1 (resolution), then the data richness of that candidate object can be determined to be 1 / 3. When the richness threshold is set to 2 / 3, the candidate object is judged as an interference object by the target selection decision-making agent. It should be noted that either of the two judgment criteria in the above example can be chosen, or both can be used simultaneously; this is not limited here.

[0066] Specifically, this will be illustrated using an image retrieval scenario for local targets, with examples to refer to. Figure 4 , Figure 4 This is an exemplary flowchart of the S2 data set construction process in the image retrieval method of this application. During the construction of the target selection decision-making agent, a candidate object set S2 = {D1, D2, ..., Dj} is obtained, which includes j candidate objects, each representing a local target. For each candidate object, Dj = {subobjj, weightj, heightj, scenej, objj}, which can be used as target filtering conditions for the candidate object. Here, subobjj is the local target category corresponding to the j-th candidate object, weightj is the width information of the local target corresponding to the j-th candidate object, heightj is the height information of the local target corresponding to the j-th candidate object, scenej is the scene category information corresponding to the j-th candidate object, and objj is the main target category corresponding to the j-th candidate object.

[0067] The data for these candidate objects are labeled, for example: S2 = {{D1, mark1}, {D2, mark2}, ..., {Dh, markh}}. Here, mark1 corresponds to the labeling result in the first data entry D1. The value of mark can be 0 or 1, where 0 indicates discarding the target, and 1 indicates saving the target (identifying it as the target object). This is used to determine whether the detected candidate objects are target objects, i.e., whether they meet the conditions for being stored in the database.

[0068] Furthermore, one can refer to, for example Figure 5 As shown, Figure 5 This is a flowchart illustrating the construction of an exemplary target selection decision-making agent in the image retrieval method of this application. The process of constructing the target selection decision-making agent may include, but is not limited to, training several neural networks with the data in the aforementioned dataset S2 to obtain several trained target selection decision-making models. The output of each model is then summarized, such as: O = {w1*out1, w2*out2, ..., wn*outn}. Here, w1, ..., wn are the weights of each model's output, and out1, ..., outn are the outputs of each model. The initial weights can all be set to 1 / n, and the output with the highest weight is taken as the final output. Subsequently, as the target selection decision-making agent participates in decision-making each time, the weights corresponding to each model are controlled to change accordingly, such as: W = w ± 0.01 * T. Here, w is the weight of each model in the previous decision. If the prediction of a certain decision model is consistent with the final result, then 0.01 times the preset weight threshold T is added; if the prediction of a certain decision model is inconsistent with the final result, then 0.01 times the threshold T is subtracted. This process iterates until the pre-set training completion condition is met, at which point the iteration stops, and the constructed target selection decision-making agent is obtained.

[0069] Based on the above embodiments, this application embodiment combines the detection model decision agent and target selection decision agent built in the foregoing embodiments to illustrate the specific application process. (See also...) Figure 6 , Figure 6This is a flowchart illustrating the construction of an exemplary database in the image retrieval method of this application. After obtaining the base database images, the base database images can be input into the detection model decision agent (relevant image data of the base database images can also be input into the detection model decision agent, such as the resolution of the base database images, scene type, the main target of interest in the corresponding scene type, the local targets of interest in the corresponding scene type, the maximum size of the local targets, the minimum size of the local targets, etc.), to obtain the target detection model selected by the detection model decision agent. This target detection model can be used to detect local targets (target objects) in the base database images. By using the target detection model to perform target detection processing on each base database image, each local target in each base database image can be obtained, for example, obtaining the local target set S3 = {subobj1, subobj2, ..., subobjn}. Each local target has corresponding image data, from which information such as image size, scene type, and main target category can be obtained. This data is then used as target filtering conditions and input into the target selection decision-making agent. The agent can select targets based on the filtering conditions, and the output of the agent is obtained. The agent determines whether the output is 0, discarding local targets with an output of 0 and retaining local targets with a non-zero output, thus obtaining the target objects in each base image.

[0070] Based on the above embodiments, this application embodiment describes the steps prior to determining the target image from each base database image based on the feature similarity between the object to be retrieved and each target object. Specifically, the method of this embodiment includes the following steps:

[0071] The target object is processed by extracting features from each target object according to the pre-trained feature extraction model to obtain target feature information; the object to be retrieved is processed by extracting features from the object to be retrieved according to the feature extraction model to obtain the retrieval feature information; the feature similarity between the retrieval object and each target object is determined according to the target feature information and the retrieval feature information.

[0072] Referring to the foregoing embodiments, after selecting the target objects from each base database image, a pre-trained feature extraction model can be used to extract features from each target object. The feature extraction model used in this application may include, but is not limited to, the multimodal model CLIP (Contrastive Language–Image Pre-training). The multimodal model extracts features from the target objects (or local targets in local target retrieval scenarios) to obtain target feature information, which can then be stored in the base database. The target feature information can be configured to include several dimensions of feature information according to the specific application scenario; this is not limited here, such as feature = {feature1, feature2, ..., featureg}, where featureg represents the extracted g-th dimension feature value.

[0073] Furthermore, after establishing the feature information (target feature information) in the database, if a retrieval image is received, the previously determined target detection model can be used to perform target detection processing on the retrieval image to determine whether the retrieval object (e.g., a local target) exists in the retrieval image. If the retrieval object exists, a multimodal model can be used to extract features of the retrieval object to obtain the retrieval feature information. Then, the feature similarity between each target feature information and the retrieval feature information is calculated, which is to obtain the feature similarity between each target object and the retrieval object.

[0074] Based on the above embodiments, this application embodiment describes the steps of performing target detection processing on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image. Specifically, the method of this embodiment includes the following steps:

[0075] The target region in the image to be retrieved is obtained; the target region is detected by the target detection model to obtain the target detection result; in response to the target detection result indicating that there is no object to be retrieved in the target region, the target image to be retrieved is detected by the target detection model to obtain the neighboring objects in the image to be retrieved; the object to be retrieved is determined from each neighboring object based on the distance between the area where the neighboring object is located and the target region.

[0076] Referring to the foregoing embodiments, in the process of using the object detection model to perform object detection on the target region, the target region in the image to be retrieved can be obtained first, and then the target region can be processed for object detection. The target region can be a pre-set portion or all of the image region; and / or it can be a target region determined in the image to be retrieved in response to a received user input region selection instruction (for example, the user can draw the target region to be retrieved on the image to be retrieved, thereby generating a corresponding region selection instruction), and the specific method is not limited.

[0077] If the object to be retrieved is detected in the target region, its features can be extracted. If the object to be retrieved is not detected in the target region, this application requires the use of an object detection model to perform object detection on the entire image to be retrieved, thereby obtaining neighboring objects in the image. Neighboring objects refer to targets detected in regions other than the target region in the image to be retrieved; essentially, neighboring objects are targets in other regions that are the same as or similar to the object to be retrieved. The detection of neighboring objects in this application aims to address the problem of reduced detection accuracy due to target movement in dynamic target retrieval scenarios.

[0078] Furthermore, if there are multiple neighboring objects, the object to be retrieved can be determined from the neighboring objects based on the distance between the area where each neighboring object is located and the target area. For example, based on the center point of the area where the neighboring object is located and the center point of the target area, the neighboring object in the area where the nearest neighboring object to the target area is located can be determined as the object to be retrieved.

[0079] Based on the above embodiments, this application embodiment describes the steps after target detection is performed on the target region according to the target detection model and the target detection result is obtained. Specifically, the method of this embodiment includes the following steps:

[0080] In response to the target detection result indicating the presence of a target object in the target region, the feature information of the target object is extracted to obtain the target feature; the feature similarity between the target feature and the target features of each target object is obtained; and the target image is determined from each base database image based on the feature similarity.

[0081] Referring to the foregoing embodiments, when a target object exists in the target area, a feature extraction model such as CLIP can be used to extract the feature information of the target object, obtaining the target feature feature_query1{feature1,feature2,...,featureg}. Then, all target features in the database can be traversed, and the feature similarity between the target feature and each target feature can be calculated. Finally, one or more target features with a feature similarity greater than a preset feature similarity threshold can be selected as the base image from the database.

[0082] For example, this application provides a similarity calculation method for reference, such as:

[0083]

[0084] Here, 'i' represents the feature dimension involved in the similarity calculation. This example demonstrates calculating the similarity between a target feature and n other features in the search. However, in specific applications, the method for calculating feature similarity can be flexibly set as needed, and may include, but is not limited to, the methods described above. For instance, one could select only a specific feature dimension for similarity calculation. It's understandable that Similarity represents feature similarity; a higher value indicates a higher similarity between two features.

[0085] Furthermore, the number of target images can be one or more, without limitation here. The method for determining the target image from each database image based on feature similarity may include, but is not limited to, selecting the database image with the highest feature similarity as the target image; or, based on the TopN algorithm, sorting each target feature in descending order of feature similarity, and then selecting the database images corresponding to the top N results from the sorting results as the target images.

[0086] Based on the above embodiments, this application embodiment describes the steps after determining the object to be retrieved from each neighboring object according to the distance between the area where the neighboring object is located and the target area. Specifically, the method of this embodiment includes the following steps:

[0087] Extract feature information of the target region to obtain a first feature; extract feature information of neighboring objects to obtain a second feature; obtain the first feature similarity between the first feature and the target features of each target object, and obtain the second feature similarity between the second feature and the target features of each target object; determine the target image from each base image library based on the first feature similarity and the second feature similarity.

[0088] Referring to the foregoing embodiments, if the target region does not contain the object to be retrieved, target detection needs to be performed on the entire image to be retrieved to identify neighboring objects in the image. Feature extraction is performed on the target region to obtain the first feature, feature_query2; feature extraction is performed on the neighboring objects to obtain the second feature, feature_query3.

[0089] For both the first and second features, calculate the first feature similarity between the first feature and the target features of each target object, and the second feature similarity between the second feature and the target features of each target object. Referring to the method in the previous embodiment, perform TopN calculations based on the feature similarity corresponding to each feature to obtain respective TopN results (for example, comparing the first feature with each target feature yields one set of TopN results, and comparing the second feature with each target feature yields another set of TopN results). Then, from the two sets of TopN results, select the base image corresponding to all or part of the target features as the target image.

[0090] Based on the above embodiments, this application embodiment describes the steps for determining a target image from each base database image based on a first feature similarity and a second feature similarity. Specifically, the method of this embodiment includes the following steps:

[0091] The first image is determined from each base image according to the first feature similarity and the preset sorting algorithm; and the second image is determined from each base image according to the second feature similarity and the preset sorting algorithm; duplicate images between each first image and each second image are deduplicated to obtain the remaining images; and the target image is determined from each remaining image according to the feature similarity corresponding to each remaining image.

[0092] In conjunction with the foregoing embodiments, in order to reduce interference data and redundancy, optimization processing can also be performed during the process of determining the target image in this application.

[0093] For example, the first image is the TopN result determined from each database image based on the first feature similarity and the TopN algorithm; the second image is the TopN result determined from each database image based on the second feature similarity and the TopN algorithm. The first image and the second image are merged, and duplicate images are removed from the first image and the second image (if two database images are identical, only one is retained), thus obtaining the remaining images. These remaining images can be used as target images, or the remaining images can be sorted according to their corresponding feature similarity from largest to smallest, and a predetermined number of the top remaining images can be selected as target images.

[0094] Referring to the foregoing embodiments, when calculating the feature similarity between the object to be retrieved and the target object, optionally, if the object to be retrieved is a local target (with relevant information about its parent target), then during feature comparison, the local feature similarity between the object to be retrieved and the target object, as well as the main feature similarity between the parent target to which the object to be retrieved belongs and the parent target to which the target object belongs, can be obtained. The weighted sum of the local feature similarity and the main feature similarity is then used as the feature similarity between the object to be retrieved and the target object. Specifically, this can be adaptively set according to whether constraints on the parent target to which the local target belongs are needed in the actual application scenario; no limitation is made here. If the object to be retrieved is a parent target (or does not have relevant information about its parent target), then the feature similarity between the two can be determined using conventional similarity calculation methods.

[0095] This application can also be combined with Figure 7 To understand the application process of image retrieval methods, Figure 7 This is a schematic diagram of an exemplary application process of the image retrieval method in this application. Figure 7 The specific steps can be found in the explanations of the foregoing embodiments, and will not be repeated here.

[0096] In summary, this application addresses several key challenges. First, by constructing a detection model decision agent, it enables the selection of the best-performing detection model based on the characteristics of different scenario types, thereby improving detection accuracy for customized scenarios and reducing the secondary investment costs for specialized algorithm engineers. Second, this detection model decision agent can integrate multiple different types of classification algorithms to obtain differentiated decision models, using the result with the highest weight as the final output, thus enhancing the robustness of the decision model. Third, this application can also construct a target selection decision agent, significantly reducing invalid targets, improving database quality, and ultimately enhancing overall performance. Specifically, this can be achieved by integrating differentiated neural networks to make the entire target selection decision agent robust. To further improve the selection performance of the target selection decision agent, this application can provide weighted feedback for each decision, allowing the agent to gradually adapt to the current application scenario.

[0097] On the other hand, during the retrieval phase, this invention can distinguish whether a target object has been detected in the target area, thereby further improving the retrieval effect. This avoids interference caused by continuous target movement or human misalignment during bounding, enhancing the generalization ability of the entire solution and enabling it to query unknown targets detected by the model. Furthermore, by comparing and sorting multi-dimensional feature values, it can prevent the loss of image information associated with the actual retrieval target.

[0098] In summary, this application can effectively improve the target search capability in image search solutions, reduce interfering search results, and reduce secondary human resource investment.

[0099] It should be further noted that the entity executing the image retrieval method can be an image retrieval device. For example, the image retrieval method can be executed by a terminal device, a server, or other processing devices. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the image retrieval method can be implemented by a processor calling computer-readable instructions stored in memory.

[0100] Figure 8 This is a block diagram illustrating an image retrieval apparatus according to an exemplary embodiment of this application. Figure 8 As shown, the exemplary image retrieval device 200 includes: a model selection module 210, a base database construction module 220, an object detection module 230, and an image retrieval module 240. Specifically:

[0101] The model selection module 210 is used to determine the target detection model corresponding to the scene type from a pre-built set of detection models based on the scene type of the acquired base image.

[0102] The base database construction module 220 is used to perform target detection processing on each base database image according to the target detection model to obtain the target objects in each base database image.

[0103] The target detection module 230 is used to respond to the received image to be retrieved, perform target detection processing on the image to be retrieved according to the target detection model, and obtain the object to be retrieved in the image to be retrieved.

[0104] The image retrieval module 240 is used to determine the target image from each base database image based on the feature similarity between the object to be retrieved and each target object.

[0105] In this exemplary image retrieval device, by acquiring base images and determining the target detection model corresponding to the scene type from a pre-built set of detection models based on the scene type of the acquired base images, appropriate target detection models can be selected for different application scenarios, improving the detection accuracy of target objects in the current application scenario. Target detection processing is performed on each base image according to the target detection model to obtain the target objects in each base image, forming a base database, reducing invalid targets in each base image and improving database quality. In response to a received image to be retrieved, target detection processing is performed on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved. Then, image retrieval can be performed based on the feature similarity between the object to be retrieved and each target object, determining the target image from each base image in the base database. This improves the efficiency and accuracy of the image retrieval process.

[0106] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0107] The functions of each module can be found in the image retrieval method implementation examples, and will not be repeated here.

[0108] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 100 includes a memory 101 and a processor 102. The processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above-described image retrieval method embodiments. In a specific implementation scenario, the electronic device 100 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 100 may also include mobile devices such as laptops and tablets, which are not limited here.

[0109] Specifically, processor 102 controls itself and memory 101 to implement the steps in any of the above-described image retrieval method embodiments. Processor 102 may also be referred to as a CPU (Central Processing Unit). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 102 may be implemented using integrated circuit chips.

[0110] In this exemplary electronic device, by acquiring base images and determining the corresponding target detection model from a pre-built set of detection models based on the scene type of the acquired base images, appropriate target detection models can be selected for different application scenarios, improving the detection accuracy of target objects in the current application scenario. Target detection processing is performed on each base image according to the target detection model to obtain the target objects in each base image, forming a base database, reducing invalid targets in each base image and improving database quality. In response to a received image to be retrieved, target detection processing is performed on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved. Then, image retrieval can be performed based on the feature similarity between the object to be retrieved and each target object, determining the target image from each base image in the base database. This improves the efficiency and accuracy of the image retrieval process.

[0111] Please see Figure 10 , Figure 10 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor. The program instructions 111 are used to implement the steps in any of the above-described image retrieval method embodiments.

[0112] In this exemplary storage medium, by running program instructions within the storage medium, base images are acquired. Based on the scene type of the acquired base images, a target detection model corresponding to the scene type is determined from a pre-built set of detection models. This allows for the selection of appropriate target detection models for different application scenarios, improving the accuracy of target object detection in the current application scenario. Target detection processing is performed on each base image according to the target detection model to obtain the target objects in each base image, forming a base database. This reduces invalid targets in each base image and improves database quality. In response to a received image to be retrieved, target detection processing is performed on the image according to the target detection model to obtain the object to be retrieved in the image. Then, image retrieval can be performed based on the feature similarity between the object to be retrieved and each target object, determining the target image from each base image in the base database. This improves the efficiency and accuracy of the image retrieval process.

[0113] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0114] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image retrieval method, characterized in that, The method includes: Based on the scene type of the acquired base image, determine the target detection model corresponding to the scene type from the pre-built detection model set; The target detection model is used to perform target detection processing on each base database image to obtain the target objects in each base database image; The step of performing target detection processing on each database image according to the target detection model to obtain the target objects in each database image includes: performing target detection on each database image according to the target detection model to obtain candidate objects in each database image; inputting each candidate object and target selection conditions into a pre-trained target selection decision agent to obtain the target objects determined by the target selection decision agent from each candidate object according to the target selection conditions. In response to the received image to be retrieved, the image to be retrieved is processed by the target detection model to obtain the object to be retrieved in the image to be retrieved; The step of performing target detection processing on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved includes: obtaining a target region in the image to be retrieved; performing target detection on the target region according to the target detection model to obtain a target detection result; responding to the target detection result indicating that the object to be retrieved does not exist in the target region, performing target detection on the image to be retrieved according to the target detection model to obtain neighboring objects in the image to be retrieved; and determining the object to be retrieved from each neighboring object according to the distance between the region where the neighboring object is located and the target region. The target image is determined from each base database image based on the feature similarity between the object to be retrieved and each target object.

2. The method according to claim 1, characterized in that, The step of determining the target detection model corresponding to the scene type from a pre-built set of detection models based on the scene type of the acquired base image database includes: The scene type of the base image is input into the pre-trained detection model decision agent to obtain the target detection model determined by the detection model decision agent from the detection model set according to the scene type.

3. The method according to claim 1, characterized in that, Before determining the target image from each base database image based on the feature similarity between the object to be retrieved and each target object, the method further includes: Based on a pre-trained feature extraction model, feature extraction processing is performed on each target object to obtain target feature information; The feature extraction model is used to extract features from the object to be retrieved to obtain the feature information to be retrieved. The feature similarity between the object to be retrieved and each target object is determined based on the target feature information and the feature information to be retrieved.

4. The method according to claim 1, characterized in that, After performing target detection on the target region according to the target detection model and obtaining the target detection result, the method further includes: In response to the target detection result indicating the presence of the object to be retrieved in the target region, the feature information of the object to be retrieved is extracted to obtain the features to be retrieved; Obtain the feature similarity between the feature to be retrieved and the target features of each target object; The target image is determined from each base database image based on the feature similarity.

5. The method according to claim 1, characterized in that, After determining the object to be retrieved from each neighboring object based on the distance between the area where the neighboring object is located and the target area, the method further includes: Extract feature information from the target region to obtain a first feature; and extract feature information from the neighboring objects to obtain a second feature; Obtain the first feature similarity between the first feature and the target features of each target object, and obtain the second feature similarity between the second feature and the target features of each target object; The target image is determined from each base database image based on the first feature similarity and the second feature similarity.

6. The method according to claim 5, characterized in that, Determining the target image from each base database image based on the first feature similarity and the second feature similarity includes: A first image is determined from each base database image based on the first feature similarity and a preset sorting algorithm; and a second image is determined from each base database image based on the second feature similarity and the preset sorting algorithm; The duplicate images between each first image and each second image are deduplicated to obtain the remaining images; The target image is determined from the remaining images based on the feature similarity corresponding to each remaining image.

7. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images