Image retrieval method and device and storage medium

Abstract

The invention discloses an image retrieval method and device and a storage medium, and the method comprises the steps: determining a target detection model corresponding to a scene type from a pre-constructed detection model set according to the scene type of an obtained bottom library image; performing target detection processing on each bottom library image according to a target detection model to obtain a target object in each bottom library image; in response to the received to-be-retrieved image, performing target detection processing on the to-be-retrieved image according to the target detection model to obtain a to-be-retrieved object in the to-be-retrieved image; and according to the feature similarity between the to-be-retrieved object and each target object, determining a target image from each base library image. According to the scheme, the image retrieval efficiency and precision can be improved.

Description

Technical Field

[0001] The present application relates to the field of data retrieval technology, and in particular to an image retrieval method, device, and storage medium. Background Art

[0002] Image retrieval technology is mainly divided into text-based image retrieval and image-based image retrieval.

[0003] In image-based image retrieval scenarios, a target image is typically acquired first, and then images with similarities to the target image exceeding a threshold are found from the image collection to be retrieved as retrieval results. This technical solution has a wide range of applications in target detection, security prevention, and other fields.

[0004] However, current image retrieval solutions still have some problems that have not been effectively solved, such as: poor retrieval effect on small targets in images, poor retrieval effect on local areas of target objects in images, and the existence of a large number of irrelevant image retrieval results, which makes the image retrieval method in some application scenarios less accurate. Summary of the Invention

[0005] This application at least provides an image retrieval method, apparatus, device, and computer-readable storage medium.

[0006] In a first aspect, the present application provides an image retrieval method, comprising: determining, based on the scene type of an acquired base image, a target detection model corresponding to the scene type from a pre-built detection model set; performing target detection processing on each base image according to the target detection model to obtain a target object in each base image; in response to a received image to be retrieved, performing target detection processing on the image to be retrieved according to the target detection model to obtain an object to be retrieved in the image to be retrieved; and determining a target image from each base image according to a feature similarity between the object to be retrieved and each target object.

[0007] In one embodiment, the method of determining the target detection model corresponding to the scene type from a pre-built detection model set based on the scene type of the acquired base image includes: inputting the scene type of the base image into a pre-trained detection model decision agent, and obtaining the target detection model determined by the detection model decision agent from the detection model set based on the scene type.

[0008] In one embodiment, the target detection processing is performed on each base image according to the target detection model to obtain the target object in each base image, including: performing target detection on each base image according to the target detection model to obtain candidate objects in each base image; inputting each candidate object and target screening conditions into a pre-trained target selection decision agent to obtain the target object determined by the target selection decision agent from each candidate object according to the target screening conditions.

[0009] In one embodiment, before determining the target image from each base library image based on the feature similarity between the object to be retrieved and each target object, the method further includes: performing feature extraction processing on each target object based on a pre-trained feature extraction model to obtain target feature information; performing feature extraction processing on the object to be retrieved based on the feature extraction model to obtain feature information to be retrieved; and determining the feature similarity between the object to be retrieved and each target object based on the target feature information and the feature information to be retrieved.

[0010] In one embodiment, the target detection processing is performed on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved, including: obtaining a target area in the image to be retrieved; performing target detection on the target area according to the target detection model to obtain a target detection result; in response to the target detection result indicating that the object to be retrieved does not exist in the target area, target detection is performed on the image to be retrieved according to the target detection model to obtain neighboring objects in the image to be retrieved; and determining the object to be retrieved from each neighboring object according to the distance between the area where the neighboring object is located and the target area.

[0011] In one embodiment, after performing target detection on the target area according to the target detection model and obtaining a target detection result, the method further includes: in response to the target detection result indicating that the object to be retrieved exists in the target area, extracting feature information of the object to be retrieved to obtain a feature to be retrieved; obtaining a feature similarity between the feature to be retrieved and a target feature of each target object; and determining the target image from each base library image according to the feature similarity.

[0012] In one embodiment, after determining the object to be retrieved from each neighboring object based on the distance between the area where the neighboring object is located and the target area, the method further includes: extracting feature information of the target area to obtain a first feature; and extracting feature information of the neighboring objects to obtain a second feature; obtaining a first feature similarity between the first feature and a target feature of each target object, and obtaining a second feature similarity between the second feature and the target feature of each target object; and determining the target image from each base library image based on the first feature similarity and the second feature similarity.

[0013] In one embodiment, determining the target image from each base image based on the first feature similarity and the second feature similarity includes: determining a first image from each base image based on the first feature similarity and a preset sorting algorithm; and determining a second image from each base image based on the second feature similarity and the preset sorting algorithm; deduplicating duplicate images between each first image and each second image to obtain remaining images; and determining the target image from each remaining image based on the feature similarity corresponding to each remaining image.

[0014] The second aspect of the present application provides an image retrieval device, including: a model selection module, used to determine, based on the scene type of the acquired base image, a target detection model corresponding to the scene type from a pre-built detection model set; a base image construction module, used to perform target detection processing on each base image according to the target detection model to obtain the target object in each base image; a target detection module, used to respond to the received image to be retrieved, perform target detection processing on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved; an image retrieval module, used to determine the target image from each base image according to the feature similarity between the object to be retrieved and each target object.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-mentioned image retrieval method.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned image retrieval method when the program instructions are executed by a processor.

[0017] The above scheme obtains a base image and, based on the scene type of the acquired base image, determines the target detection model corresponding to the scene type from a pre-built detection model set. This allows the selection of appropriate target detection models for different application scenarios, thereby improving the detection accuracy of target objects in the current application scenario. Target detection processing is performed on each base image according to the target detection model to obtain the target objects in each base image, forming a base database, reducing invalid targets in each base image and improving the database quality. In response to the received image to be retrieved, target detection processing is performed on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved. Image retrieval can then be performed based on the feature similarity between the object to be retrieved and each target object, and the target image can be determined from each base image in the base database. This can improve the efficiency and accuracy of the image retrieval process.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0020] Figure 1 is a flowchart of an exemplary embodiment of the image retrieval method of the present application;

[0021] Figure 2 This is an exemplary S1 data set construction flow chart in the image retrieval method of the present application;

[0022] Figure 3 This is a flowchart for building an exemplary detection model decision agent in the image retrieval method of this application;

[0023] Figure 4 This is an exemplary S2 data set construction flow chart in the image retrieval method of the present application;

[0024] Figure 5 This is a flowchart for building an exemplary target selection decision agent in the image retrieval method of this application;

[0025] Figure 6 This is a flowchart of an exemplary database construction in the image retrieval method of the present application;

[0026] Figure 7 This is a schematic diagram of an exemplary application process of the image retrieval method of the present application;

[0027] Figure 8is a block diagram of an image retrieval device shown in an exemplary embodiment of the present application;

[0028] Figure 9 This is a schematic structural diagram of an embodiment of an electronic device of the present application;

[0029] Figure 10 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0031] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0032] The term "and / or" in this article is simply a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0033] See also Figure 1 , Figure 1 This is a flowchart of an exemplary embodiment of the image retrieval method of the present application. Specifically, it may include the following steps:

[0034] Step S110 , according to the scene type of the acquired base image, determine the target detection model corresponding to the scene type from a pre-built detection model set.

[0035] Among them, the base image refers to the pre-collected image that can be used as a reference in the image retrieval process.

[0036] The scene type refers to the type of application scene included in the image content. For example, the scene type may include but is not limited to traffic scenes, market scenes, park workshop scenes, etc. Since the target objects that need to be detected in different scenes are usually different, the professionalism of various target detection models in performing target detection for different application scenarios is also different. Therefore, there are usually certain differences between the target detection models corresponding to different application scenarios. The present application can store one or more target detection models that support deployment and use in a detection model set. Then, according to the scene type of the acquired base image, the corresponding target detection model can be determined from the pre-built detection model set.

[0037] Step S120 , performing target detection processing on each base image according to the target detection model to obtain the target object in each base image.

[0038] It should be noted that common image retrieval methods usually perform retrieval based on the similarity between the base image and the image to be retrieved. This is equivalent to common image retrieval methods being based on the global image content of the base image and the image to be retrieved. However, this type of method is less effective when retrieving small targets in an image. Since the target object to be retrieved in an image accounts for a small proportion of its information in the overall image, if there are many irrelevant interfering targets in the image, the image retrieval accuracy will be greatly reduced. For example, in traffic scenarios, it is necessary not only to detect vehicles, but also small items such as helmets, handheld objects, and backpacks of non-motor vehicle drivers.

[0039] This application can use the target detection model to perform target detection processing on each base image to detect the target objects in each base image in advance, which can be used as a reference in the image retrieval process, and store them in the base database database (hereinafter referred to as the database). Therefore, when the image to be retrieved is subsequently received, the target objects in the base database can be compared with the image to be retrieved, thereby improving the efficiency of image retrieval.

[0040] Optionally, in the specific implementation process of the present application, the local image where the target object is located in the base image may be saved in the database, and / or the feature information of the target object may be extracted and saved in the database.

[0041] Step S130 , in response to the received image to be retrieved, performing target detection processing on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved.

[0042] In conjunction with the above steps, after a database is established according to the method provided in the above steps, the database can be used in the image retrieval process.

[0043] For example, in response to a received image to be retrieved, the same target detection model used when establishing the database can be used to perform target detection processing to obtain the target object in the image to be retrieved. Similarly, the target object can also be equivalent to a partial image of the image to be retrieved where the target object is located. Thus, image retrieval processing can be performed by comparing and analyzing the target object with the target objects in the database.

[0044] Step S140 : determining a target image from each base library image based on the feature similarity between the object to be retrieved and each target object.

[0045] In conjunction with the above steps, the method of determining a target image similar to the image to be retrieved from each base library image in the present application can be to compare the similarity of feature information between each target object and the object to be retrieved, determine a target object similar to the object to be retrieved from each target object based on the feature similarity, and determine the base library image corresponding to the target object similar to the object to be retrieved as the target image similar to the image to be retrieved.

[0046] If the target object's feature information is pre-stored in the database, then before step S140, feature information extraction can be performed on the image to be retrieved, and then the feature similarity between the target object to be retrieved and each target object can be calculated. If the target object's feature information is not pre-stored in the database, then before step S140, feature information extraction must be performed on each target object and the target object to be retrieved, and then the feature similarity between the target object to be retrieved and each target object must be calculated.

[0047] It should be noted that this application does not limit the number of target images, and there can be zero, one, or multiple target images. For example, if the feature similarity between all target objects and the object to be retrieved is less than the similarity threshold, it is determined that there is no target object similar to the object to be retrieved, that is, the target image is zero. For another example, in images with feature similarity greater than or equal to the similarity threshold, the target object with the largest feature similarity can be selected as the target object similar to the object to be retrieved, that is, one target image is obtained. Alternatively, in images with feature similarity greater than or equal to the similarity threshold, several target objects can be selected as target objects similar to the object to be retrieved based on the feature similarity, that is, multiple target images are obtained.

[0048] It can be seen that the present application obtains a base image and, based on the scene type of the obtained base image, determines the target detection model corresponding to the scene type from a pre-built detection model set. This allows the application to select appropriate target detection models for different application scenarios, thereby improving the detection accuracy of target objects in the current application scenario. Target detection processing is performed on each base image according to the target detection model to obtain the target objects in each base image, forming a base database, reducing invalid targets in each base image and improving the database quality. In response to the received image to be retrieved, target detection processing is performed on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved. Image retrieval can then be performed based on the feature similarity between the object to be retrieved and each target object, and the target image can be determined from each base image of the base database. This can improve the efficiency and accuracy of the image retrieval process.

[0049] Based on the above embodiment, the present embodiment describes the steps of determining the target detection model corresponding to the scene type from a pre-built detection model set based on the scene type of the acquired base image. Specifically, the method of this embodiment includes the following steps:

[0050] The scene type of the base image is input into the pre-trained detection model decision agent, and the target detection model is obtained by the detection model decision agent from the detection model set according to the scene type.

[0051] With reference to the above embodiments, the target detection model used for target detection in the present application can be selected as needed according to the scene type of the base library image.

[0052] Exemplarily, a method for selecting a target detection model may be to determine a target detection model corresponding to a scene type from a detection model set comprising multiple types of detection models through the scene type of the base image and a pre-set mapping relationship. The scene type of the base image may be pre-labeled information and / or information obtained by performing scene detection on the base image, which is not limited here. The various detection models in the detection model set are pre-trained. The preset mapping relationship may include a mapping relationship between a scene type and a detection model, which may include but is not limited to a one-to-one relationship or a many-to-one relationship.

[0053] Alternatively, the method for selecting an object detection model can be performed by a pre-built detection model decision agent. For example, a portion or all of the underlying image data can be input into the detection model decision agent, which then analyzes the received image data, determines the scene type of the underlying image data, and determines the corresponding object detection model from a set of detection models. Alternatively, the scene type of the underlying image data can be input into the detection model decision agent, which then determines the corresponding object detection model from a set of detection models.

[0054] It should also be noted that when selecting the target detection model corresponding to the base image, in addition to referring to the method of the aforementioned embodiment to select the target detection model corresponding to the scene type of the base image, you can also refer to more image data for selection to ensure that the most suitable detection model can be obtained. These image data may include but are not limited to the scene type of the base image, image resolution, size information of the target object in the base image, category information and other image data. Among them, the target object may include the main target in the base image that needs to be paid attention to in the current scene type, and / or the local target (part of the main target) in the base image that needs to be paid attention to in the current scene type, which can be set according to the specific image retrieval requirements. For example, the detection model decision intelligence will select the corresponding target detection model according to the scene type of the image and the category of the local target that needs to be paid attention to in the image.

[0055] Specifically, please refer to Figure 2 As shown, Figure 2 This is an exemplary S1 data set construction flowchart in the image retrieval method of the present application. In the process of building the detection model decision-making intelligent agent, the initial images of different scene types and the image data of these initial images can be obtained first to obtain the initial image data set S1 = {data1, data2, ..., dataN}.

[0056] Each data can represent the image data of an initial image, for example, data1 = {weight1, height1, scene1, Sobj1, SsubObj1, maxSize, minSize}. Weight1 represents the width of the image corresponding to the data, height1 represents the height of the image corresponding to the data, scene1 represents the scene type corresponding to the data, Sobj1 is the main target category that needs to be paid attention to in the corresponding scene type, SsubObj1 represents the local target category that needs to be paid attention to in the scene type corresponding to the data, maxSize represents the maximum size supported by the local target in the scene type corresponding to the data, and minSize represents the minimum size supported by the local target in the scene type corresponding to the data. An initial image may include multiple main targets and multiple local targets, Sobj1 = {obj1, obj2, ..., objm}, SsubObj1 = {subobj1, subobj2, ..., subobjk}.

[0057] Thus, a corresponding detection model can be pre-set for each image data, for example, S1 = {{data1, model1}, {data2, model2}, ..., {datai, modeli}}, where model1 is the target detection model to be selected for the corresponding scene type of data1.

[0058] Then, you can refer to Figure 3 As shown, Figure 3 This is a flowchart for building an exemplary detection model decision agent in the image retrieval method of the present application. The detection model decision agent can be built according to the data set obtained in the above implementation process. The detection model decision agent is mainly composed of several classifiers, such as support vector machines, decision trees, random forests, K-nearest neighbors, etc., which are not limited here, and is trained with the data in the above-mentioned data set S1 to obtain several decision models, and summarize the output of each model, such as: out = {w1*out1, w2*out2, ..., wn*outn}. Among them, w1, ..., wn are the weights of each output, out1, ..., outn are the outputs of each detection model, and the value of each weight can be set to 1 / n. Finally, the output with the highest weight is used as the final output to obtain the detection model selected by the agent.

[0059] In summary, after using the initial image to build a detection model decision agent through steps including but not limited to those provided in the above example method, in subsequent application scenarios, the base library image can be input into the detection model decision agent to obtain a target detection model determined by the detection model decision agent from the detection model set according to the scene type of the base library image.

[0060] Based on the above embodiment, the present embodiment of the application describes the steps of performing target detection processing on each base image according to the target detection model to obtain the target object in each base image. Specifically, the method of this embodiment includes the following steps:

[0061] According to the target detection model, target detection is performed on each base image to obtain candidate objects in each base image; each candidate object and target screening conditions are input into a pre-trained target selection decision agent to obtain the target object determined by the target selection decision agent from each candidate object according to the target screening conditions.

[0062] With reference to the above-mentioned embodiment, after the target detection model performs target detection on each base library image, the obtained targets may include target objects and / or interference targets. At this time, the detected targets are regarded as candidate objects, and the candidate objects need to be further screened to obtain the target objects.

[0063] Exemplarily, the screening process of candidate objects in the present application may be to compare the image data of each candidate object (such as the width, height, and object category mentioned in the aforementioned embodiments) with preset target screening conditions, determine the candidate objects that meet the target screening conditions as target objects, and determine the candidate objects that do not meet the target screening conditions as interference objects (interference targets).

[0064] In another exemplary embodiment, the candidate object screening process in this application may include inputting the candidate objects and their related data into a pre-established target selection decision agent, which then selects target objects that meet the target screening criteria from among the candidate objects based on the candidate objects' related data. The target selection decision agent is primarily composed of several different neural networks, where the differences between the different neural networks primarily lie in the number of network layers and nodes.

[0065] Optionally, in the process of comparing the data of each candidate object with the target screening conditions in the present application, in addition to referring to the above examples, the data richness of each candidate object can also be determined based on the amount of data obtained for each candidate object and the preset required data amount. According to the comparison of the data richness of each candidate object and the preset richness threshold, the candidate object whose data richness is greater than or equal to the richness threshold can be determined as the target object, and the candidate object whose data richness is less than the richness threshold can be determined as the interference object. For example, after the target detection process, each candidate object has its own corresponding image data (such as the resolution, target category, target size, etc. mentioned in the above embodiment, which are not limited here). If the preset required data amount is 3 (resolution, target category, target size), and the data amount for a certain detected candidate object is 1 (resolution), it can be determined that the data richness of the candidate object is 1 / 3. When the richness threshold is set to 2 / 3, the candidate object is determined to be an interference object by the target selection decision agent. It should be noted that the two judgment bases in the above examples can be selected either one or both at the same time, which is not limited here.

[0066] Specifically, we take the local target image retrieval scenario as an example. Figure 4 , Figure 4 This is an exemplary S2 data set construction flow chart in the image retrieval method of the present application. In the process of building a target selection decision-making agent, a candidate object set S2 = {D1, D2, ..., Dj} is obtained, which includes j candidate objects, and each candidate object can represent a local target. For each candidate object, Dj = {subobjj, weightj, heightj, scenej, objj}, these data are equivalent to target screening conditions that can be used as candidate objects. Among them, subobjj is the local target category corresponding to the j-th candidate object, weightj is the local target width information corresponding to the j-th candidate object, heightj is the local target height information corresponding to the j-th candidate object, scenej is the scene category information corresponding to the j-th candidate object, and objj is the main target category corresponding to the j-th candidate object.

[0067] The data for these candidate objects is labeled, for example: S2 = {{D1, mark1}, {D2, mark2}, ..., {Dh, markh}}. Mark1 corresponds to the labeling result of the first data item D1, and the value of mark can be 0 or 1, with 0 indicating that the object is discarded and 1 indicating that the object is retained (determined as the target object). This determines whether the detected candidate object is the target object, that is, whether it meets the conditions for storage in the database.

[0068] Further, you can refer to Figure 5 As shown, Figure 5 This is a flowchart for building an exemplary target selection decision agent in the image retrieval method of the present application. The process of building the target selection decision agent may include, but is not limited to: training several neural networks using the data in the aforementioned data set S2 to obtain several trained target selection decision models, summarizing the outputs of each model, such as: O = {w1*out1,w2*out2,...,wn*outn}. Where w1,...,wn are the weights of each model output, and out1,...,outn are the outputs of each model. The initial weights can all be set to 1 / n, with the output with the highest weight being the final output. Subsequently, as the target selection decision agent participates in each decision, the weights corresponding to each model are controlled to change, such as: W = w±0.01*T. Where w is the weight of each model's previous decision. If the prediction of a decision model is consistent with the final result, 0.01 times the preset weight threshold T is added. If the prediction of a decision model is inconsistent with the final result, 0.01 times the threshold T is subtracted. This process is repeated until the pre-set training completion condition is met, at which point the iterations can be stopped, resulting in the built target selection decision agent.

[0069] On the basis of the above embodiments, the present application embodiment integrates the detection model decision agent and target selection decision agent built in the above embodiments to illustrate the specific application process. Figure 6 , Figure 6This is a flowchart for constructing an exemplary database in the image retrieval method of the present application. After obtaining the base image, the base image can be input into the detection model decision agent (the relevant image data of the base image can also be input into the detection model decision agent, such as the resolution of the base image, the scene type, the main target of interest in the corresponding scene type, the local target of interest in the corresponding scene type, the maximum size of the local target, the minimum size of the local target, etc.), to obtain the target detection model selected by the detection model decision agent. The target detection model can be used to detect the local target (target object) in the base image. By using the target detection model to perform target detection processing on each base image, each local target in each base image can be obtained, for example, a local target set S3 = {subobj1, subobj2, ..., subobjn} is obtained. Each local target has corresponding image data, from which information including but not limited to the image size of each local target, the scene type to which it belongs, the main target category, etc. can be obtained, and these data are input into the target selection decision-making agent as target screening conditions. The target selection decision-making agent can make a selection based on the screening conditions to obtain the output result of the target selection decision-making agent, and judge whether the output result is 0, discard the local targets with an output result of 0, and retain the local targets with an output result not of 0, that is, obtain the target objects in each base image.

[0070] Based on the above embodiment, the present embodiment describes the steps before determining the target image from each base image based on the feature similarity between the object to be retrieved and each target object. Specifically, the method of this embodiment includes the following steps:

[0071] According to the pre-trained feature extraction model, feature extraction processing is performed on each target object to obtain target feature information; according to the feature extraction model, feature extraction processing is performed on the object to be retrieved to obtain feature information to be retrieved; based on the target feature information and the feature information to be retrieved, the feature similarity between the object to be retrieved and each target object is determined.

[0072] With reference to the above-mentioned embodiment, after screening out the target objects in each base library image, a pre-trained feature extraction model can be used to perform feature extraction processing on each target object. Among them, the feature extraction model used in this application may include but is not limited to the multimodal model CLIP (Contrastive Language–Image Pre-training), and the multimodal model is used to extract features of the target object (local target in the local target retrieval scenario) to obtain target feature information, which can be stored in the base library database. Among them, the target feature information can include feature information of several dimensions according to the specific application scenario, which is not limited here, such as feature = {feature1, feature2, ..., featureg}, featureg represents the extracted g-th dimension feature value.

[0073] Furthermore, after establishing the feature information (target feature information) in the database, if an image to be retrieved is received, the previously determined target detection model can be used to perform target detection processing on the image to be retrieved to determine whether the object to be retrieved (such as a local target) exists in the image to be retrieved. If the object to be retrieved exists, the multimodal model can be used to extract features of the object to be retrieved to obtain the feature information to be retrieved, and then the feature similarity between each target feature information and the feature information to be retrieved is calculated, that is, the feature similarity between each target object and the object to be retrieved is obtained.

[0074] Based on the above embodiment, the embodiment of the present application describes the steps of performing target detection processing on the image to be retrieved based on the target detection model to obtain the object to be retrieved in the image to be retrieved. Specifically, the method of this embodiment includes the following steps:

[0075] Acquire a target area in the image to be retrieved; perform target detection on the target area according to a target detection model to obtain a target detection result; in response to the target detection result indicating that the object to be retrieved does not exist in the target area, perform target detection on the image to be retrieved according to the target detection model to obtain neighboring objects in the image to be retrieved; determine the object to be retrieved from each neighboring object according to the distance between the area where the neighboring object is located and the target area.

[0076] In conjunction with the aforementioned embodiment, in the process of using the target detection model to perform target detection on the target area, it is also possible to first obtain the target area in the image to be searched, and then perform target detection processing on the target area. The target area can be a pre-set part or all of the image area; and / or can be a target area determined in the image to be searched according to the area selection instruction in response to a region selection instruction received from the user (for example, the user can draw the target area to be searched on the image to be searched, thereby generating a corresponding region selection instruction), without specific limitation.

[0077] If the object to be retrieved is detected in the target area, the features of the object to be retrieved can be extracted. If the object to be retrieved is not detected in the target area, this application needs to use the target detection model to perform target detection on the entire image to be retrieved to obtain the neighboring objects in the image to be retrieved. Among them, the neighboring objects refer to the targets detected in other areas other than the target area in the image to be retrieved, which is equivalent to the neighboring objects being targets that are the same or similar to the object to be retrieved in other areas. The purpose of detecting neighboring objects in this application is to solve the problem of reducing detection accuracy due to target movement in dynamic target retrieval scenarios.

[0078] Furthermore, if there are multiple neighboring objects, the object to be retrieved can be determined from each neighboring object based on the distance between the area where each neighboring object is located and the target area. For example, based on the center point of the area where the neighboring object is located and the center point of the target area, the neighboring object in the area where the neighboring object closest to the target area is located is determined as the object to be retrieved.

[0079] Based on the above embodiment, the embodiment of the present application describes the steps after performing target detection on the target area according to the target detection model and obtaining the target detection result. Specifically, the method of this embodiment includes the following steps:

[0080] In response to the target detection result indicating that there is an object to be retrieved in the target area, feature information of the object to be retrieved is extracted to obtain features to be retrieved; feature similarities between the features to be retrieved and target features of each target object are obtained; and a target image is determined from each base library image based on the feature similarities.

[0081] To illustrate this with reference to the previous embodiment, if a target object exists in the target region, a feature extraction model such as CLIP can be used to extract the feature information of the target object, resulting in the target feature feature_query1{feature1, feature2,..., featureg}. All target features in the database can then be traversed to calculate the feature similarity between the target feature and each target feature. The base image corresponding to one or more target features whose feature similarity exceeds a preset feature similarity threshold can then be selected as the target image.

[0082] For example, this application provides a similarity calculation method for reference, such as:

[0083]

[0084] Among them, i represents the feature dimension involved in the similarity calculation. Here, an example is given of calculating the similarity between the features to be retrieved with n dimensions and a target feature. However, in specific application scenarios, the method for calculating feature similarity (Similarity) can be flexibly set as needed, and may include but is not limited to the above calculation method. For another example, only features of a specified dimension can be selected for similarity calculation. It can be understood that Similarity is feature similarity, and the larger its value, the higher the similarity between the two features.

[0085] Furthermore, the number of target images may be one or more, and is not limited here. Methods for determining the target image from the base images based on feature similarity may include, but are not limited to, selecting the base image with the greatest feature similarity as the target image; or sorting the target features from largest to smallest based on feature similarity using a TopN algorithm, and then selecting the base images corresponding to the top N results from the sorted results as the target image.

[0086] Based on the above embodiment, the present embodiment describes the steps after determining the object to be retrieved from each neighboring object based on the distance between the area where the neighboring object is located and the target area. Specifically, the method of this embodiment includes the following steps:

[0087] Extract feature information of the target area to obtain a first feature; and extract feature information of adjacent objects to obtain a second feature; obtain a first feature similarity between the first feature and the target feature of each target object, and obtain a second feature similarity between the second feature and the target feature of each target object; and determine a target image from each base library image based on the first feature similarity and the second feature similarity.

[0088] In conjunction with the above embodiment, if the target region does not contain the object to be retrieved, target detection is performed on the entire image to be retrieved to determine the neighboring objects in the image to be retrieved. Feature extraction is performed on the target region to obtain the first feature, feature_query2; feature extraction is performed on the neighboring objects to obtain the second feature, feature_query3.

[0089] For the first feature and the second feature, respectively, calculate the first feature similarity between the first feature and the target feature of each target object, and the second feature similarity between the second feature and the target feature of each target object. Referring to the method of the aforementioned embodiment, perform TopN calculations based on the feature similarities corresponding to each feature to obtain respective TopN results (for example, one set of TopN results can be obtained by comparing the first feature with each target feature, and another set of TopN results can be obtained by comparing the second feature with each target feature). Then, from the two sets of TopN results, the base image corresponding to all or part of the target features can be selected as the target image.

[0090] Based on the above embodiment, the present embodiment describes the steps of determining the target image from each base image based on the first feature similarity and the second feature similarity. Specifically, the method of this embodiment includes the following steps:

[0091] A first image is determined from each base image based on a first feature similarity and a preset sorting algorithm; and a second image is determined from each base image based on a second feature similarity and a preset sorting algorithm; duplicate images between each first image and each second image are deduplicated to obtain remaining images; and a target image is determined from each remaining image based on the feature similarity corresponding to each remaining image.

[0092] With reference to the above embodiments, in order to reduce interference data and redundancy, the present application may also perform optimization processing during the process of determining the target image.

[0093] For example, the first image is the TopN result determined from each base image using the first feature similarity and the TopN algorithm; the second image is the TopN result determined from each base image using the second feature similarity and the TopN algorithm. The first and second images are merged, and duplicate images are removed from the first and second images (if there are two identical base images, one is retained), thereby obtaining residual images. These residual images can be used as target images, or the residual images can be sorted from highest to lowest according to their corresponding feature similarities, and a preset number of the top residual images are selected as target images.

[0094] In conjunction with the above embodiment, when calculating the feature similarity between the object to be retrieved and the target object, optionally, if the object to be retrieved is a local target (there is relevant information about the main target to which it belongs), then when comparing features, the local feature similarity between the object to be retrieved and the target object can be obtained, as well as the main feature similarity between the main target to which the object to be retrieved belongs and the main target to which the target object belongs can be obtained, and the weighted summation calculation result of the local feature similarity and the main feature similarity is used as the feature similarity between the object to be retrieved and the target object. Specifically, it can be adaptively set according to whether it is necessary to constrain the main target to which the local target belongs in the actual application scenario, which is not limited here. If the object to be retrieved is the main target (or there is no relevant information about the main target to which it belongs), the feature similarity between the two can be determined according to the conventional similarity calculation method.

[0095] This application can also be combined Figure 7 Understand the application process of image retrieval methods, Figure 7 This is an exemplary application flow diagram of the image retrieval method of this application. Figure 7 The specific steps in the embodiment can be referred to the explanation of the above embodiment and will not be repeated here.

[0096] In summary, on the one hand, the present application can build a detection model decision-making agent so that the detection model with the best target detection effect can be selected according to the characteristics of different scene types in the future, thereby improving the detection accuracy of customized scenes and reducing the secondary investment cost of professional algorithm personnel. Furthermore, the detection model decision-making agent can obtain differentiated decision-making models by integrating multiple different types of classification algorithms, and use the result with the largest weight as the final output to improve the robustness of the decision-making model. Furthermore, the present application can also build a target selection decision-making agent, which can greatly reduce invalid targets, improve the quality of the database, and thus improve the overall effect. Specifically, the entire target selection decision-making agent can be made robust by integrating differentiated neural networks, in order to further improve the selection effect of the target selection decision-making agent. The present application can provide weighted feedback for each decision, so that the agent gradually adapts to the current application scenario.

[0097] Furthermore, during the retrieval phase, the present invention can distinguish whether the target object in the target area is detected, further improving retrieval results. This avoids interference caused by continuous target movement, artificial frame selection misalignment, and other factors, enhancing the generalization of the entire solution and enabling the query of unknown targets in the detection model. Furthermore, by comparing and sorting multidimensional feature values, it can prevent the loss of image information associated with the actual retrieval target.

[0098] As described above, the present application can effectively improve the target search capability in the image search solution, reduce interference search results, and reduce secondary manpower investment.

[0099] It should be further noted that the image retrieval method may be executed by an image retrieval device. For example, the image retrieval method may be executed by a terminal device, a server, or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. In some possible implementations, the image retrieval method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0100] Figure 8 FIG. 1 is a block diagram of an image retrieval device according to an exemplary embodiment of the present application. Figure 8 As shown, the exemplary image retrieval device 200 includes: a model selection module 210, a base library construction module 220, a target detection module 230 and an image retrieval module 240. Specifically:

[0101] The model selection module 210 is used to determine the target detection model corresponding to the scene type from a pre-built detection model set according to the scene type of the acquired base image.

[0102] The base image building module 220 is used to perform target detection processing on each base image according to the target detection model to obtain the target object in each base image.

[0103] The target detection module 230 is configured to, in response to the received image to be retrieved, perform target detection processing on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved.

[0104] The image retrieval module 240 is used to determine the target image from each base library image based on the feature similarity between the object to be retrieved and each target object.

[0105] In this exemplary image retrieval device, a target detection model corresponding to the scene type is determined from a pre-built detection model set by acquiring a base image and, based on the scene type of the acquired base image, selecting an appropriate target detection model for different application scenarios, thereby improving the detection accuracy of the target object in the current application scenario. Target detection processing is performed on each base image according to the target detection model to obtain the target object in each base image, forming a base database, thereby reducing invalid targets in each base image and improving the database quality. In response to a received image to be retrieved, target detection processing is performed on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved. Image retrieval is then performed based on the feature similarity between the object to be retrieved and each target object, and the target image is determined from each base image in the base database. This improves the efficiency and accuracy of the image retrieval process.

[0106] It should be noted that the apparatus provided in the above embodiments and the methods provided in the above embodiments are based on the same concept. The specific manner in which the various modules and units perform their operations has been described in detail in the method embodiments and will not be repeated here. In actual applications, the apparatus provided in the above embodiments can, as needed, allocate the above functions to different functional modules, i.e., divide the internal structure of the apparatus into different functional modules to perform all or part of the functions described above. This is not a limitation herein.

[0107] The functions of each module can be found in the embodiment of the image retrieval method and will not be described in detail here.

[0108] See also Figure 9 , Figure 9 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. Electronic device 100 includes memory 101 and processor 102. Processor 102 is configured to execute program instructions stored in memory 101 to implement the steps of any of the above-described image retrieval method embodiments. In a specific implementation scenario, electronic device 100 may include, but is not limited to, a microcomputer and a server. Furthermore, electronic device 100 may also include mobile devices such as laptops and tablet computers, without limitation herein.

[0109] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned image retrieval method embodiments. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0110] In this exemplary electronic device, by acquiring a base image and determining the target detection model corresponding to the scene type from a pre-built detection model set based on the scene type of the acquired base image, it is possible to select an appropriate target detection model for different application scenarios, thereby improving the detection accuracy of the target object in the current application scenario; target detection processing is performed on each base image according to the target detection model to obtain the target object in each base image, forming a base database, reducing invalid targets in each base image and improving the database quality; in response to the received image to be retrieved, target detection processing is performed on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved; then, image retrieval can be performed based on the feature similarity between the object to be retrieved and each target object, and the target image can be determined from each base image in the base database. This can improve the efficiency and accuracy of the image retrieval process.

[0111] See also Figure 10 , Figure 10 The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor, and the program instructions 111 are used to implement the steps of any of the above-mentioned image retrieval method embodiments.

[0112] In this exemplary storage medium, by executing program instructions in the storage medium, a base image is acquired, and based on the scene type of the acquired base image, a target detection model corresponding to the scene type is determined from a pre-built detection model set. This allows the selection of appropriate target detection models for different application scenarios, thereby improving the detection accuracy of target objects in the current application scenario. Target detection processing is performed on each base image according to the target detection model to obtain the target objects in each base image, forming a base database, reducing invalid targets in each base image and improving the database quality. In response to a received image to be retrieved, target detection processing is performed on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved. Then, image retrieval can be performed based on the feature similarity between the object to be retrieved and each target object, and the target image can be determined from each base image of the base database. This can improve the efficiency and accuracy of the image retrieval process.

[0113] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0114] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0116] In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. An image retrieval method, characterized in that: The method comprises: According to the scene type of the acquired base image, determine the target detection model corresponding to the scene type from the pre-built detection model set; Performing target detection processing on each base image according to the target detection model to obtain the target object in each base image; In response to the received image to be retrieved, performing target detection processing on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved; According to the feature similarity between the object to be retrieved and each target object, a target image is determined from each base library image.

2. The method according to claim 1, characterized in that The step of determining, based on the scene type of the acquired base image, a target detection model corresponding to the scene type from a pre-built detection model set includes: The scene type of the base image is input into a pre-trained detection model decision agent to obtain a target detection model determined by the detection model decision agent from the detection model set according to the scene type.

3. The method according to claim 1, characterized in that The performing target detection processing on each base image according to the target detection model to obtain the target object in each base image includes: Performing target detection on each base image according to the target detection model to obtain candidate objects in each base image; The candidate objects and target screening conditions are input into a pre-trained target selection decision agent to obtain a target object determined by the target selection decision agent from the candidate objects according to the target screening conditions.

4. The method according to claim 1, wherein Before determining the target image from each base image based on the feature similarity between the object to be retrieved and each target object, the method further includes: Perform feature extraction processing on each target object according to the pre-trained feature extraction model to obtain target feature information; Performing feature extraction processing on the object to be retrieved according to the feature extraction model to obtain feature information to be retrieved; The feature similarity between the object to be retrieved and each target object is determined according to the target feature information and the feature information to be retrieved.

5. The method according to claim 1, wherein The performing target detection processing on the image to be retrieved according to the target detection model to obtain the object to be retrieved in the image to be retrieved includes: Acquire a target area in the image to be retrieved; Performing target detection on the target area according to the target detection model to obtain a target detection result; In response to the target detection result indicating that the object to be retrieved does not exist in the target area, performing target detection on the image to be retrieved according to the target detection model to obtain neighboring objects in the image to be retrieved; The object to be retrieved is determined from each neighboring object according to the distance between the area where the neighboring object is located and the target area.

6. The method according to claim 5, characterized in that After performing target detection on the target area according to the target detection model to obtain a target detection result, the method further includes: In response to the target detection result indicating that the object to be retrieved exists in the target area, feature information of the object to be retrieved is extracted to obtain features to be retrieved; Obtaining feature similarity between the feature to be retrieved and the target feature of each target object; The target image is determined from each base library image according to the feature similarity.

7. The method according to claim 5, characterized in that After determining the object to be retrieved from the neighboring objects based on the distance between the area where the neighboring objects are located and the target area, the method further includes: Extracting feature information of the target area to obtain a first feature; and extracting feature information of the neighboring objects to obtain a second feature; Obtaining a first feature similarity between the first feature and a target feature of each target object, and obtaining a second feature similarity between the second feature and the target feature of each target object; The target image is determined from each base library image according to the first feature similarity and the second feature similarity.

8. The method according to claim 7, characterized in that The determining the target image from each base library image according to the first feature similarity and the second feature similarity includes: Determine a first image from each base image according to the first feature similarity and a preset sorting algorithm; and determine a second image from each base image according to the second feature similarity and the preset sorting algorithm; performing deduplication processing on duplicate images between each first image and each second image to obtain a residual image; The target image is determined from the remaining images according to the feature similarities corresponding to the remaining images.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images