Retrieval result determination method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention provide a retrieval result determination method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining an input image; determining an open set target of the input image, and extracting a category label of the open set target; generating a content description text for the category label, and obtaining text features of the content description text; acquiring position information of the open set target; generating an unstructured visual coding feature and an attribute feature based on the category label and the position information; an open set target index is constructed through the text features, the unstructured visual coding features and the attribute features; when the retrieval request is obtained, the retrieval result corresponding to the retrieval request is determined based on the open set target index, and therefore the situation that when a new target type appears every time, a model needs to be retrained is avoided, and retrieval for the new target type can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of retrieval result determination, and in particular to a retrieval result determination method, a retrieval result determination device, an electronic device and a computer-readable storage medium. Background Art

[0002] Video content structuring is a technology that converts unstructured visual information in a video into structured data. By extracting objects such as people and cars in the video and describing them as computer-understandable text or graphic information, it is possible to achieve refined management of surveillance videos. This structured processing can not only effectively remove redundant information and simplify video data, but also greatly improve the retrieval efficiency of video data.

[0003] However, existing video structuring techniques are usually limited to predefined object categories, and once new object types appear, the model needs to be retrained, which greatly limits its scope of application. Summary of the invention

[0004] The embodiments of the present invention provide a search result determination method, device, electronic device and computer-readable storage medium to overcome the above problems or at least partially solve the above problems.

[0005] The embodiment of the present invention discloses a method for determining a search result, comprising:

[0006] Get the input image;

[0007] Determining an open set target of the input image, and extracting a category label of the open set target;

[0008] Generate a content description text for the category label, and obtain text features of the content description text;

[0009] Obtaining location information of the open set target;

[0010] generating unstructured visual coding features and attribute features based on the category label and the position information;

[0011] Constructing an open set target index through the text features, the unstructured visual encoding features and the attribute features;

[0012] When a search request is obtained, a search result corresponding to the search request is determined based on the open set target index.

[0013] The embodiment of the present invention also discloses a search result determination device, comprising:

[0014] An input image acquisition module is used to acquire an input image;

[0015] A category label extraction module, used to determine the open set target of the input image and extract the category label of the open set target;

[0016] A text feature acquisition module, used to generate a content description text for the category label and acquire text features of the content description text;

[0017] A location information acquisition module, used to acquire the location information of the open set target;

[0018] An unstructured visual coding feature generation module, used to generate unstructured visual coding features and attribute features based on the category label and the position information;

[0019] An open set target index module, used for constructing an open set target index through the text features, the unstructured visual encoding features and the attribute features;

[0020] The retrieval result determination module is used to determine the retrieval result corresponding to the retrieval request based on the open set target index when a retrieval request is obtained.

[0021] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0022] The memory is used to store computer programs;

[0023] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0024] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0025] The embodiments of the present invention include the following advantages:

[0026] The embodiment of the present invention achieves the following beneficial effects by acquiring an input image; determining an open set target of the input image and extracting a category label of the open set target; generating a content description text for the category label and acquiring text features of the content description text; acquiring location information of the open set target; generating an unstructured visual coding feature and an attribute feature based on the category label and the location information; constructing an open set target index through the text features, the unstructured visual coding features and the attribute features; and when a retrieval request is acquired, determining a retrieval result corresponding to the retrieval request based on the open set target index:

[0027] Open set object recognition: can identify target categories that have never appeared in the training data.

[0028] Multimodal feature fusion: combines image features, text features, and attribute features to provide more comprehensive information.

[0029] Flexible search: supports multiple types of search methods to meet different user needs.

[0030] Efficient retrieval: By building an index, the retrieval efficiency is improved.

[0031] This avoids the need to retrain the model whenever a new target type appears, greatly expanding its scope of application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flowchart of a method for determining a search result provided in an embodiment of the present invention;

[0033] Figure 2 is a flow chart of a method for determining a search result provided in an embodiment of the present invention;

[0034] Figure 3 It is a schematic diagram of an optimization training process of an open set target feature and a multi-attribute extraction subnetwork provided in an embodiment of the present invention;

[0035] Figure 4 It is a schematic diagram of another optimization training process of an open set target feature and a multi-attribute extraction subnetwork provided in an embodiment of the present invention;

[0036] Figure 5 is a structural block diagram of a search result determination device provided in an embodiment of the present invention;

[0037] Figure 6 is a hardware structure block diagram of an electronic device provided in an embodiment of the present invention;

[0038] Figure 7 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Video content structuring refers to a comprehensive technology that extracts structured information from unstructured visual data in videos. The extracted structured information is generally text information or visual graphic information that can be understood and processed by computers or humans. Monitoring video storage can remove a large amount of redundant or useless information on the one hand, and facilitate subsequent retrieval of monitoring targets on the other hand. By understanding and describing the target metadata, the amount of video monitoring data has been greatly reduced and compressed, while the information density has been greatly improved. The structuring of monitoring information is also conducive to actual business needs such as database storage and query.

[0041] The video content structuring methods of related technologies are limited to existing target categories. For example, in the monitoring structured scenes of pedestrians, motor vehicles and non-motor vehicles, it is necessary to complete the detection model for the determined different categories of targets, and then classify the attributes of the respective targets to complete the content structuring scheme in the existing scene. However, the existing structuring scheme cannot meet the needs of searching for new targets, thus limiting its application scenarios.

[0042] The present invention proposes a video structuring scheme based on open world detection, and based on the scheme proposes a flexible retrieval method for arbitrary targets, which supports text-based target retrieval, image-based retrieval and image-text hybrid retrieval.

[0043] In order to enable those skilled in the art to better understand the embodiments of the present invention, the reference method involved in the present invention is introduced as follows:

[0044] (1) RAM (Recognize Anything, AStrong Image Tagging Model), this method adopts a new image tagging paradigm, which can recognize any common category with high accuracy and give corresponding descriptions. It uses large-scale image-text pairs for training instead of manual annotation. It can obtain unannotated image labels on a large scale through automatic text semantic parsing, and uses a unified title and tagging task to train a preliminary model for automatic annotation, which is supervised by the original text and parsed labels respectively. It uses a data engine to generate additional annotations and remove incorrect annotations, and uses the processed data to retrain the model and fine-tune it using a smaller but higher quality dataset. In terms of building structure, methods based on RAM, groudingdino and clip are adopted;

[0045] (2)Groudingdino, Grounding DINO is a transformer-based object detection model. It contains an image feature extraction module, a text feature extraction module, a feature enhancement module, a language-guided query selection module, and a cross-modal decoder module.

[0046] (3) Clip (Contrastive Language-Image Pre-Training), which is a contrastive language-image pre-training, is an open source, multimodal, zero-shot model. Given an image and a text description, the model can predict the most relevant text description for the image without optimizing for a specific task.

[0047] Reference Figure 1 , shows a flowchart of a method for determining a search result provided in an embodiment of the present invention, which may specifically include the following steps:

[0048] Step 101, obtaining an input image;

[0049] Step 102, determining an open set target of the input image, and extracting a category label of the open set target;

[0050] Step 103, generating a content description text for the category label, and obtaining text features of the content description text;

[0051] Step 104, obtaining the location information of the open set target;

[0052] Step 105, generating unstructured visual coding features and attribute features based on the category label and the position information;

[0053] Step 106, constructing an open set target index through the text features, the unstructured visual encoding features and the attribute features;

[0054] Step 107: When a search request is obtained, a search result corresponding to the search request is determined based on the open set target index.

[0055] In a specific implementation, the embodiment of the present invention mainly includes a stage of constructing an open set target index and arbitrary retrieval of the open set target.

[0056] In a specific implementation, the embodiment of the present invention can obtain an input image to obtain the original image data to be processed, thereby providing a data basis for subsequent feature extraction and index construction.

[0057] In a specific implementation, an embodiment of the present invention can determine the open set targets of the input image and extract the category labels of the open set targets; so as to identify all targets of interest in the image and assign a category label to each target, thereby providing target information for subsequent feature extraction and indexing.

[0058] The embodiment of the present invention uses an advanced target detection algorithm to perform pixel-level classification on an image, thereby determining what types of targets are contained in the image, and accurately identifying various types of targets, such as pedestrians, vehicles, roadblocks, etc., in complex traffic scenes.

[0059] Open-Set Object

[0060] In the field of computer vision, especially in object detection tasks, we often encounter a problem: when training a model, we can only use known categories for training, but in actual applications, new categories that have never been seen in the training set may appear. These categories that have never been seen are called open set targets.

[0061] Simply put, open set targets are those unknown object categories that were not included in the training model.

[0062] Class labels for open set targets:

[0063] Traditional object detection: In traditional object detection tasks, each object is assigned a certain category label, such as "cat", "dog", "car", etc. These labels are defined in advance, and the model will only make predictions in these known categories.

[0064] Category labels of open-set targets: For open-set targets, since their categories are unknown, we cannot directly assign them a definite category label. Therefore, the category labels of open-set targets are usually expressed as "unknown" or "other".

[0065] Open set target: unknown object categories outside the training set.

[0066] Category label of open-set target: usually expressed as "unknown" or "other", indicating that the target does not belong to a known category.

[0067] In a specific implementation, an embodiment of the present invention can generate content description text for the category label and obtain text features of the content description text; generate a text description for the image and extract semantic features of the text, thereby enriching the representation of the image and providing support for text-based retrieval.

[0068] By generating a natural language description of the image content based on the extracted target category label, a comprehensive semantic description is generated for the entire image. By using a text encoding model (such as BERT, GPT, etc.) to map the text into a high-dimensional vector space, the text description is converted into a feature vector that can be processed by a computer.

[0069] In the stage of building the open-set target index, the global content of the image is described through the text encoding module, and the corresponding text features are extracted.

[0070] These text features play a very important role in subsequent retrieval and comprehension.

[0071] 1. As a semantic representation of an image:

[0072] Enriching image information: Although the pixel values ​​of an image contain rich visual information, they cannot directly express the semantic meaning of the image. Text features can convert the content of the image into a language description that humans can understand, thereby more accurately expressing the semantic information of the image.

[0073] Easy to retrieve: With text features, we can use natural language to retrieve images. For example, we can enter "find a red car", and the system can find images that meet the conditions based on the text features.

[0074] 2. Used to build indexes:

[0075] Build text index: Store the extracted text features in the index database and establish a mapping relationship between text features and images.

[0076] Accelerated retrieval: When a user enters a text query, the system can quickly locate relevant images through text indexing, greatly improving retrieval efficiency.

[0077] 3. Assisted target detection:

[0078] Providing prior knowledge: Text features can provide some prior knowledge for object detection. For example, if the text feature contains the word "person", the object detection model can prioritize the location of the person in the image.

[0079] Improve detection accuracy: By combining text features with visual features, the accuracy of object detection can be improved, especially for some difficult-to-distinguish categories.

[0080] 4. Support multimodal retrieval:

[0081] Image-text matching: Text features can be matched with the visual features of the image to achieve the image-text matching task.

[0082] Cross-modal retrieval: Through text features, we can implement cross-modal retrieval, for example, find related images based on a text description.

[0083] In a specific implementation, the embodiment of the present invention can obtain the position information of the open set targets to determine the specific position of each target in the image and provide positioning information for subsequent visual feature extraction.

[0084] The specific location of each target is marked through methods such as bounding boxes, providing a basis for subsequent feature extraction and accurately determining the location of each target in the image.

[0085] In a specific implementation, an embodiment of the present invention can generate unstructured visual coding features and attribute features based on the category label and the location information to extract the visual features and attribute features of the target and provide feature representation for subsequent indexing and similarity calculations.

[0086] By extracting the visual appearance features of the target, such as color, texture, shape, etc., to distinguish different types of targets, extracting the attribute features of the target, such as the gender and age of pedestrians, the brand and model of vehicles, etc., to provide a more fine-grained description. Using the CLIP model, the image and text features are mapped to the same embedding space to achieve the joint representation of image and text, thereby improving the expressiveness of the features, thereby enriching the representation of the target and improving the accuracy of retrieval.

[0087] In image recognition and computer vision tasks, the relationship between category labels, location information, unstructured visual encoding features, and attribute features together constitute a comprehensive description and understanding of the image content.

[0088] Category label and location information:

[0089] Category label: Indicates the category to which the target in the image belongs, such as "person", "car", "cat", etc. It provides a high-level semantic information, telling us what is in the image.

[0090] Position information: Indicates the specific location of the target in the image, usually represented by a bounding box. It provides a spatial positioning information, telling us where the target is.

[0091] Their relationship:

[0092] Complementarity: Category labels and location information are complementary. Category labels tell us "what" and location information tells us "where". The combination of the two can fully describe an object.

[0093] Dependency: In many tasks, location information is the basis for determining the category label. For example, in object detection, we first need to locate the location of the object before we can classify it.

[0094] Unstructured visual encoding features and attribute features:

[0095] Unstructured visual encoding features: It is a mathematical description of the visual appearance of an image or an object in an image. It is usually a high-dimensional vector that contains information such as texture, color, shape, etc. These features are extracted from images through deep learning models (such as convolutional neural networks).

[0096] Attribute features: descriptions of the attributes of the target, such as "color", "size", "material", etc. These attributes can be qualitative (such as "red", "large") or quantitative (such as "length", "width").

[0097] Their relationship:

[0098] Hierarchical relationship: Unstructured visual coding features are low-level features, which are directly extracted from image pixels. Attribute features are high-level features, which are the abstraction and generalization of visual features.

[0099] Complementary relationship: Unstructured visual encoding features contain rich visual information, but this information may be relatively abstract. Attribute features provide more specific and intuitive descriptions.

[0100] The relationship between the four:

[0101] There is a certain correspondence between category labels and attribute features. For example, the attribute features corresponding to the category label "cat" may include "furry", "four legs", "tail", etc.

[0102] There is also a connection between position information and unstructured visual encoding features. By extracting features from image regions at different locations, we can obtain features that describe different parts of the target.

[0103] Category labels, location information, unstructured visual encoding features, and attribute features together constitute a comprehensive description of the image content:

[0104] Category labels provide information at the semantic level; location information provides spatial location information; unstructured visual encoding features provide underlying visual features; and attribute features provide high-level semantic features.

[0105] By combining this information, we can understand the image content more accurately and perform more complex image analysis tasks such as object detection, image classification, image retrieval, etc.

[0106] In a specific implementation, an embodiment of the present invention can construct an open set target index through the text features, the unstructured visual coding features and the attribute features; store the extracted features in an index database, establish a mapping relationship between the features and the image, and provide a basis for efficient retrieval.

[0107] Exemplarily, an open set target index can be constructed in the following ways. Feature extraction and representation: Text features: Use natural language processing technology (such as word embedding, TF-IDF) to convert text into numerical vectors. Visual features: Use deep learning models (such as CNN) to extract deep features of images and convert them into high-dimensional vectors. Attribute features: Quantify or encode attribute values ​​into numerical vectors. Feature fusion: Simple concatenation: Directly concatenate feature vectors of different modalities into a long vector. Weighted fusion: Weighted average the features of each modality according to the importance of different features. Deep fusion: Use attention mechanisms or multimodal fusion models to learn the interaction between features of different modalities. Index construction: Vector database: Use vector databases such as Faiss and Milvus to store and index the fused feature vectors. Hash index: Use technologies such as local sensitive hashing (LSH) to map high-dimensional vectors to low-dimensional space to accelerate neighbor search. Tree index: Use space partitioning methods such as KD trees and ball trees to build indexes to accelerate searches. Similarity search: Nearest neighbor search: Given a query vector, find the most similar target in the index. Range search: Find targets with similarity to the query vector within a certain range. By storing the extracted features (such as image features, text features, attribute features) in the index database, an index is established for fast search, enabling fast retrieval of related images.

[0108] In a specific implementation, when a search request is obtained, the embodiment of the present invention can determine the search result corresponding to the search request based on the open set target index, so as to find the most relevant image from the index database according to the user's search request.

[0109] refer to Figure 2 , Figure 2 is a flow chart of a method for determining a search result provided in an embodiment of the present invention;

[0110] GroundingDINO: An object detection model for localizing objects in images.

[0111] CLIP: Contrastive Language-Image Pre-training, a pre-training model that can map images and text into the same embedding space.

[0112] fea: visual feature, visual feature.

[0113] attrs: attribute feature, attribute feature.

[0114] embedding: Feature embedding, mapping high-dimensional features to low-dimensional space representation.

[0115] C1, C2, ... Cn: Use target detection models such as GroundingDINO to detect and classify objects in the image and obtain multiple target categories.

[0116] Index building phase:

[0117] Image input: Input the image to be indexed into the system.

[0118] Object detection: Use object detection models such as GroundingDINO to detect and classify objects in the image and obtain multiple object categories (C1, C2, ... Cn).

[0119] Feature extraction:

[0120] Visual feature extraction: For each detected target, the CLIP model is used to extract its visual features (fea) and its corresponding attribute features (attrs).

[0121] Global image features: extract a global visual feature from the entire image.

[0122] Feature fusion: The visual features and attribute features of each target and the global image features are fused to obtain a comprehensive feature representation.

[0123] Index construction: Store the fused features in the database and create an index for subsequent rapid retrieval.

[0124] Retrieval phase:

[0125] Retrieval request: The user can enter a text description (prompt) or query an image.

[0126] Feature extraction: Extract features from the input text or image to obtain the corresponding feature vector.

[0127] Similarity calculation: Calculate the similarity between the query features and the features in the index database.

[0128] Return results: Returns the images or objects most similar to the query.

[0129] Exemplarily, the present invention may construct an open set target index in the following manner.

[0130] 1. Image preprocessing and feature extraction:

[0131] Multi-category target recognition and classification: For the input large traffic scene images, the system can accurately identify and classify a variety of open-set targets, such as pedestrians, motor vehicles, non-motor vehicles, roadblocks, trash cans, etc.

[0132] Global semantic description generation: Based on the extracted target labels, the system can generate a global semantic description of the entire image content. For example, for an image containing pedestrians crossing the road and vehicles driving, the system can generate a description such as "pedestrians crossing the road, vehicles driving".

[0133] Text feature encoding: Convert the generated global semantic description text into a computer-understandable feature vector to provide a basis for subsequent retrieval.

[0134] 2. Target positioning and location information extraction:

[0135] Refined target positioning: The system can accurately locate the position of each target in the image, including pedestrians, vehicles, roadblocks, etc.

[0136] Multi-target processing: Even if the image contains multiple targets of the same type, the system can locate them one by one and extract their position information.

[0137] 3. Multimodal feature extraction:

[0138] Visual feature extraction: Using advanced technologies such as the CLIP model, the unstructured visual coding features of the target are extracted from the image to capture the visual appearance information of the target.

[0139] Attribute feature extraction: Extract corresponding attribute features for different types of targets. For example, for pedestrians, extract gender, age, race and other attributes; for vehicles, extract brand, color, type and other attributes.

[0140] 4. Index database construction:

[0141] Multi-level index: The system has established a multi-level index database, including:

[0142] Global feature index: A feature vector that stores the global semantic description of an image.

[0143] Object visual feature index: stores the unstructured visual encoding features of each object.

[0144] Target attribute feature index: stores the attribute features of the target.

[0145] Efficient retrieval: By building these indexes, the system can quickly and accurately find matching images or objects based on user queries.

[0146] By extracting global features and local target features (including visual features and attribute features) from the input image, the extracted features are stored in the index database to prepare for subsequent retrieval. After completing the open-set target index construction stage, any retrieval stage of the open-set target can be entered. By converting text queries into feature vectors, video clips with similar features can be found in the index database; or, image queries can be converted into feature vectors, and video clips with similar features can be found in the index database; or, image and text information can be used for retrieval at the same time to improve retrieval accuracy.

[0147] Through text-based retrieval: according to the text description entered by the user, search for matching images in the index database; image-based retrieval: according to the input image, search for similar images in the index database; mixed image and text-based retrieval: use image and text information for retrieval at the same time to improve the accuracy of retrieval and support multiple types of retrieval.

[0148] The embodiment of the present invention achieves the following beneficial effects by acquiring an input image; determining an open set target of the input image and extracting a category label of the open set target; generating a content description text for the category label and acquiring text features of the content description text; acquiring location information of the open set target; generating an unstructured visual coding feature and an attribute feature based on the category label and the location information; constructing an open set target index through the text features, the unstructured visual coding features and the attribute features; and when a retrieval request is acquired, determining a retrieval result corresponding to the retrieval request based on the open set target index:

[0149] Open set object recognition: can identify target categories that have never appeared in the training data.

[0150] Multimodal feature fusion: combines image features, text features, and attribute features to provide more comprehensive information.

[0151] Flexible search: supports multiple types of search methods to meet different user needs.

[0152] Efficient retrieval: By building an index, the retrieval efficiency is improved.

[0153] This avoids the need to retrain the model once a new target type appears, greatly expanding its scope of application.

[0154] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0155] In an optional embodiment of the present invention, the unstructured visual coding feature is a global unstructured visual coding feature for the global content of the input image, further comprising:

[0156] A global unstructured visual coding feature sub-index database is constructed for the global unstructured visual coding feature of the input image.

[0157] In a specific implementation, the embodiment of the present invention can extract global unstructured visual coding features of an input image, and construct a global unstructured visual coding feature sub-index database for the global unstructured visual coding features.

[0158] Global unstructured visual encoding features:

[0159] Global: It means that the features are extracted from the entire input image rather than a specific area in the input image.

[0160] Unstructured: Compared with structured features (such as histograms), unstructured features are usually high-dimensional vectors that can better capture the overall information of the image.

[0161] Visual encoding: refers to the process of converting the visual information of an image (such as color, texture, shape) into a numerical representation that a computer can understand.

[0162] The global unstructured visual encoding feature is a high-dimensional numerical representation of the entire image, which can capture the overall visual content of the image.

[0163] Methods for extracting global unstructured visual encoding features:

[0164] Convolutional Neural Network (CNN):

[0165] The output of the last fully connected layer of CNN is usually regarded as the global feature of the image.

[0166] Pre-trained CNN models (such as VGG, ResNet, Inception) can be directly used to extract features.

[0167] CNNs can be fine-tuned to make them more suitable for specific tasks.

[0168] Autoencoder:

[0169] Autoencoders can extract compact representations of images by learning latent representations of data.

[0170] Global Average Pooling:

[0171] The features on each channel of the convolutional feature map are averaged to obtain a feature vector as the global feature.

[0172] The way to construct the global unstructured visual encoding feature sub-index database is as follows:

[0173] Feature extraction:

[0174] For each image, a global unstructured visual encoding feature is extracted using a selected method.

[0175] Feature Normalization:

[0176] The extracted feature vectors are normalized to make them more comparable.

[0177] Index building:

[0178] Select an appropriate index structure (such as inverted index, vector database) to store these feature vectors.

[0179] For each feature vector, an index is created to facilitate subsequent fast retrieval.

[0180] Why build a global unstructured visual encoding feature sub-index database?

[0181] Efficient retrieval: Through the index structure, the image with the highest similarity to the query image can be quickly found.

[0182] Support similar image search: You can search for similar images based on image content.

[0183] As the basis for other tasks: It can be used as the basis for tasks such as image classification, image retrieval, and image generation.

[0184] The process of building a global unstructured visual encoding feature sub-index database is essentially to convert the visual content of an image into a numerical representation that can be understood by a computer and to create an index for fast retrieval. This process has a wide range of applications in image search, image classification, image recommendation and other fields.

[0185] In an optional embodiment of the present invention, when a search request is obtained, the step of determining a search result corresponding to the search request based on the open set target index includes:

[0186] When a search request is obtained, and the search request includes a first target text for describing the entire image scene, obtaining a first target text feature vector of the first target text;

[0187] Based on the first target text feature vector and the open set target index, a first target image corresponding to the first target text is determined from a global unstructured visual encoding feature sub-index database.

[0188] For example, the first target text used to describe the entire image scene may be: "Find a red BMW X3 crossing a zebra crossing without yielding to two pedestrians on the zebra crossing", which contains all important feature elements of the entire image scene. The embodiment of the present invention may convert the first target text into a numerical vector that can be understood by a computer, namely, a first target text feature vector, which generally represents the semantic information of the text.

[0189] As can be seen from the above, the embodiments of the present invention can store the global unstructured visual coding features of the image in a database and create an index so as to quickly find images similar to the query text.

[0190] The retrieval process can be as follows:

[0191] Text encoding:

[0192] The first target text input by the user (such as "find a red BMW X3 crossing the zebra crossing without giving way to two pedestrians on the zebra crossing") is input into a pre-trained text encoding model (such as BERT, GPT, etc.).

[0193] The model converts the text into a high-dimensional vector, namely the first target text feature vector, which contains the semantic information of the text.

[0194] Find similar images in an indexed database:

[0195] The similarity between the encoded text vector and the image feature vector stored in the index database is calculated.

[0196] Commonly used similarity calculation methods include cosine similarity, Euclidean distance, etc.

[0197] The image corresponding to the image feature vector with the highest similarity to the query text vector is found as the retrieval result.

[0198] Example description:

[0199] Suppose we have a database containing a large number of car images, and we have extracted global unstructured visual encoding features for each image. Now, we want to find images with the first target text "a red BMW X3 crossed the zebra crossing without giving way to 2 pedestrians on the zebra crossing".

[0200] Text encoding: Input this sentence into a pre-trained text encoding model to obtain a text vector (the first target text feature vector).

[0201] Similarity calculation: Calculate the similarity between the obtained text vector and the feature vector of each image in the database.

[0202] Sorting: Sort all images based on their similarity scores.

[0203] Return results: Return several pictures with the highest similarity as search results.

[0204] In the above way, by learning a large amount of image and text data, the model can accurately understand the image content. By learning a large amount of image and text data, the model can accurately understand the image content. Using the index structure, the retrieval results can be quickly returned. By mapping text and images to the same feature space and using similarity calculation, we can achieve text-based image retrieval. This technology has broad application prospects in many fields.

[0205] In an optional embodiment of the present invention, the unstructured visual coding feature is an open set target unstructured visual coding feature for an open set target of the input image, and further includes:

[0206] Constructing an open-set target unstructured visual coding feature sub-index database for the open-set target unstructured visual coding feature of the input image;

[0207] According to the attribute features, a multi-attribute feature sub-index database is constructed.

[0208] The embodiments of the present invention can extract the unstructured visual coding features and attribute features of each target simultaneously for multiple open-set targets in an input image, construct an open-set target unstructured visual coding feature sub-index database and a multi-attribute feature sub-index database, and construct an index.

[0209] Open set targets: In traditional machine learning, we usually assume that all possible categories are known during training. However, in practical applications, we often encounter some categories that have never been seen during training. These categories are called open set targets. Multiple open set targets refer to objects that may contain multiple unknown categories in an image.

[0210] Open-set object unstructured visual encoding feature: This is a high-dimensional numerical representation of open-set objects in an image. It does not rely on the specific properties or structure of the object, but is learned from the pixel-level information of the image. This feature is usually extracted by deep learning models (such as convolutional neural networks).

[0211] Sub-index database: The extracted features are stored in a database and indexed for fast retrieval.

[0212] The method of extracting the unstructured visual encoding features and the attribute features is as follows:

[0213] 1. Object Detection

[0214] The input image is processed using an object detection algorithm (such as YOLOv5, Faster R-CNN), all objects in the image are identified, and a bounding box is generated for each object.

[0215] 2. Feature extraction:

[0216] Visual feature extraction:

[0217] For each detected object, the corresponding image region is cropped.

[0218] The cropped image region is input into a pre-trained visual encoding model (such as ResNet, ViT) to extract the unstructured visual encoding features of the target.

[0219] Attribute feature extraction:

[0220] A variant of Region-based CNN (R-CNN) can be used to extract attribute features while detecting objects.

[0221] Alternatively, the attention mechanism can be used to focus on the attribute-related areas in the image to extract more accurate attribute features.

[0222] 3. Feature normalization:

[0223] The extracted visual features and attribute features are normalized to make them more comparable.

[0224] Index building includes:

[0225] Visual feature index:

[0226] The visual feature vector of each target is stored in an open-set target unstructured visual coding feature sub-index database, thereby completing the construction of the open-set target unstructured visual coding feature sub-index database.

[0227] For each feature vector, an index is created to quickly find feature vectors similar to the query vector.

[0228] Attribute feature index:

[0229] The attribute feature vector of each target is stored in another attribute feature sub-index database to complete the construction of the attribute feature sub-index database.

[0230] For each attribute feature vector, an index is created to quickly find objects with specific attributes.

[0231] By extracting visual features and attribute features from multiple open-set objects in the input image and building a corresponding index database, more flexible and accurate image retrieval can be achieved. This method has broad application prospects in computer vision, artificial intelligence and other fields.

[0232] In an optional embodiment of the present invention, when a search request is obtained, the step of determining a search result corresponding to the search request based on the open set target index includes:

[0233] When a search request is obtained, and the search request includes a second target text for describing the open set target, obtaining a second target text feature vector of the second target text;

[0234] Based on the second target text feature vector and the open-set target index, a second target image corresponding to the second target text is determined from the open-set target unstructured visual encoding feature sub-index database.

[0235] The CLIP_attr network is a model that maps text and image features into the same embedding space. Through CLIP_attr, we can convert text descriptions (such as "a deliveryman from Merchant X riding a purple electric bike") and visual features of images into similar vector representations.

[0236] Unstructured visual encoding features: This is a high-level semantic representation of image content, usually extracted by deep learning models (such as ResNet, ViT, etc.).

[0237] Open-set target unstructured visual encoding feature sub-index database: This is a database that stores the visual features of a large number of images, each feature vector represents an image. By searching the database for the feature vector that is most similar to the query text vector, the image that best matches the text description can be found.

[0238] The retrieval process is as follows:

[0239] Text encoding:

[0240] The second target text input by the user to describe the open set target (such as "a takeaway deliveryman from Merchant X who rides a purple electric bike") is directly input into the CLIP_attr network. The CLIP_attr network encodes the text into a dense second target text feature vector representation, which contains the semantic information of the second target text.

[0241] Find similar images in an indexed database:

[0242] The encoded second target text feature vector is similarly calculated with the image feature vector stored in the open set target unstructured visual encoding feature sub-index database.

[0243] Commonly used similarity calculation methods include cosine similarity, Euclidean distance, etc.

[0244] The image corresponding to the image feature vector with the highest similarity to the query text vector is found as the retrieval result.

[0245] Example description:

[0246] Assume that there is an open-set target unstructured visual encoding feature sub-index database containing a large number of urban street scene images, and the open-set target unstructured visual encoding features are extracted for each image. Now, you want to find a picture of "a takeaway deliveryman from merchant X riding a purple electric bike".

[0247] Text encoding: Input this sentence as the second target text into the CLIP_attr network to obtain a second target text feature vector.

[0248] Similarity calculation: The obtained second target text feature vector is similarly calculated with the feature vector of each image in the open set target unstructured visual encoding feature sub-index database.

[0249] Sorting: Sort all images based on their similarity scores.

[0250] Return results: Return several pictures with the highest similarity as the search results, i.e. the second target images.

[0251] The advantages of CLIP_attr networks are:

[0252] End-to-end training: The CLIP_attr network is pre-trained on a large number of image-text pairs, and can directly map text and images into the same embedding space without additional training.

[0253] Multimodal Understanding: CLIP_attr is able to understand the semantic relationship between text and images, enabling more accurate image retrieval.

[0254] Flexible: It can handle a wide variety of text descriptions, including complex scenes and objects.

[0255] Technical details and advantages:

[0256] CLIP_attr network structure: CLIP_attr usually consists of an image encoder and a text encoder. The image encoder encodes the image into visual features, and the text encoder encodes the text into text features.

[0257] Loss function: The training goal of CLIP_attr is to make the image features and text features as close as possible in the embedding space.

[0258] Index database: Using efficient index structures (such as Faiss and Milvus) can speed up retrieval.

[0259] In this way, text can be directly input into the CLIP_attr network without complex preprocessing; CLIP_attr performs well in multimodal tasks and can accurately match text and images; it can be applied to various image retrieval tasks.

[0260] In an optional embodiment of the present invention, when a search request is obtained, the step of determining a search result corresponding to the search request based on the open set target index includes:

[0261] When a search request is obtained, and the search request includes a search request image and a third target text for describing an object of interest of the search request image, obtaining the object of interest position information of the object of interest;

[0262] Extracting an object of interest image of an object of interest from the retrieval request image based on the object of interest position information;

[0263] Acquire an unstructured visual coding feature of an object of interest of the image of the object of interest;

[0264] Based on the unstructured visual coding feature of the object of interest and the open set object index, a third object image is determined from the open set object unstructured visual coding feature sub-index database and the multi-attribute feature sub-index database.

[0265] Target of interest: A specific object or area that the user hopes to find in the retrieval request image.

[0266] Patch: An image region cropped from the original image that contains the object of interest.

[0267] CLIP_attr network: A model that maps images and text into the same embedding space to calculate the similarity between images and text.

[0268] Unstructured visual encoding features of interest objects: semantic representation of the interest objects in the retrieval request image, which does not depend on the specific structure of the image.

[0269] Attribute characteristics: describe the specific attributes of the target, such as color, size, shape, etc.

[0270] Open-set target unstructured visual encoding feature sub-index database and multi-attribute feature sub-index database: store the visual features and attribute features of the image respectively for fast retrieval.

[0271] Retrieval process:

[0272] Target positioning:

[0273] The retrieval request includes: a retrieval request image and a description text (such as "a food delivery driver from merchant X riding a purple electric bike").

[0274] Target localization module: Use target detection algorithms (such as YOLOv5, Faster R-CNN) or text-based image segmentation methods to locate the location information of the target of interest in the retrieval request image according to the text description.

[0275] Obtain patch: According to the position information of the target of interest, crop the target image patch of interest containing the image area of ​​the target of interest from the retrieval request image.

[0276] Feature extraction:

[0277] CLIP_attr network: Input the target image patch into the CLIP_attr network to obtain the corresponding unstructured visual encoding features of the target.

[0278] Attribute feature extraction: Attribute features can be extracted using additional attribute prediction models or directly from the output of CLIP_attr.

[0279] Index search:

[0280] Visual feature retrieval: The extracted unstructured visual encoding features of the target of interest are similarly calculated with the features in the visual feature sub-index database to find the most similar image.

[0281] Attribute feature retrieval: Calculate the similarity between the extracted attribute features and the features in the attribute feature sub-index database to find targets with similar attributes.

[0282] Comprehensive retrieval: The retrieval results of visual features and attribute features can be integrated to obtain the final retrieval results.

[0283] Through the above method, the following beneficial effects can be achieved:

[0284] High accuracy: By combining target localization and feature extraction, targets of interest can be found more accurately.

[0285] Strong flexibility: Flexible retrieval can be performed based on different text descriptions.

[0286] Good scalability: The retrieval accuracy can be improved by adding new attribute features.

[0287] In an optional embodiment of the present invention, when a search request is obtained, the step of determining a search result corresponding to the search request based on the open set target index includes:

[0288] When a retrieval request is obtained, and the retrieval request includes a retrieval request image and a cutout image of an object of interest of the retrieval request image, the cutout image is input into a pre-trained similarity calculation model, and a cutout image unstructured visual coding feature of the cutout image is obtained;

[0289] Based on the cutout image unstructured visual coding feature and the open set target index, a fourth target image is determined from the open set target unstructured visual coding feature sub-index database and the multi-attribute feature sub-index database.

[0290] Cutout image: Separate the object of interest from the original image to generate a new image.

[0291] Similarity calculation model: CLIP_attr network, a model that can map images and texts into the same embedding space, used to calculate the similarity between images and texts.

[0292] Unstructured visual encoding features of cutout images: The semantic representation of cutout images does not depend on the specific structure of the image.

[0293] Attribute characteristics: describe the specific attributes of the target, such as color, size, shape, etc.

[0294] Open-set target unstructured visual encoding feature sub-index database and multi-attribute feature sub-index database: store the visual features and attribute features of the image respectively for fast retrieval.

[0295] Retrieval process:

[0296] Input: Retrieve the requested image to retrieve the matted image of the object of interest of the requested image.

[0297] Feature extraction: Input the cutout image into the CLIP_attr network to obtain the corresponding unstructured visual encoding features of the cutout image.

[0298] Attribute feature extraction: Attribute features can be extracted using additional attribute prediction models or directly from the output of CLIP_attr.

[0299] Index search:

[0300] Visual feature retrieval: The extracted unstructured visual encoding features of the cutout image are used to calculate the similarity with the features in the visual feature sub-index database to find the most similar image.

[0301] Attribute feature retrieval: Calculate the similarity between the extracted attribute features and the features in the attribute feature sub-index database to find targets with similar attributes.

[0302] Comprehensive retrieval: The retrieval results of visual features and attribute features can be integrated to obtain the final retrieval results.

[0303] By retrieving images in the above manner, the following beneficial effects can be achieved.

[0304] High accuracy: By cutting out the target, background interference is reduced and the accuracy of retrieval is improved.

[0305] Strong flexibility: can handle targets in various complex backgrounds.

[0306] Good scalability: The retrieval accuracy can be improved by adding new attribute features.

[0307] In order to enable those skilled in the art to better understand the embodiment of the present invention, an example is used below to illustrate the embodiment of the present invention.

[0308] It can be based on the video structuring scheme of open world detection, and supports text-based target retrieval, image-based retrieval and image-text hybrid retrieval.

[0309] (1) Open set target index construction stage:

[0310] A. Obtain labels of all valid targets in the image content and extract unstructured visual encoding features of the entire image:

[0311] The RAM model can be used to describe the image in detail, and the label information of common targets in the corresponding image can be obtained. The tag2text method can also be used to obtain the unstructured description information of the entire image and perform unstructured visual coding feature extraction.

[0312] refer to Figure 2 , Figure 2 1 is a flow chart of a method for determining retrieval results provided in an embodiment of the present invention; the label information of the target and the description information of the whole image are obtained through text encoding to complete the extraction of unstructured features of the whole image. The image is obtained through the RAM model for all the extractions in the image. At the same time, the obtained unstructured features of the whole image are sent to the retrieval database for storage to facilitate retrieval.

[0313] B. Multi-target label screening: In order to adapt to specific scenarios, such as the large traffic scenario (which will also be used as an example in the future), filter out the label information of the corresponding range and send it to the subsequent process.

[0314] C. Send the filtered valid label information of the whole image in the above steps to the open set position information acquisition module, such as the Groudingdino module, to obtain all the frame position information corresponding to the valid label information, so as to facilitate the subsequent feature expression of each frame. In order to enhance the open set detection capability of the position information acquisition module in specific scenes, we collected a large number of target images and their position information including hundreds of categories such as pedestrians, motor vehicles and non-motor vehicles, trash cans, etc., and fine-tuned the model to further improve the accuracy;

[0315] D. Cut out the corresponding target area and the corresponding label information of all the box positions in the above steps in the original image, and send them to the joint extraction module CLIP_attr of open set target unstructured visual coding features and multi-attribute features of open set targets based on CLIP network extension. For different patch images, their unstructured visual coding features are obtained by sending them to the image encoder layer of CLIP_attr. In the encoding layer of specific features, multiple attribute classification heads are introduced to perform attribute analysis on different patch images. The unstructured visual coding features corresponding to each patch image and their corresponding attribute features are sent to the database for retrieval.

[0316] E. Build an index database. For each image, save the unstructured visual coding feature extraction of the entire image, as well as the unstructured visual coding features and multi-attribute features of each target in the image, and complete the corresponding index database construction.

[0317] (2) Arbitrary search of open set targets:

[0318] After completing the above-mentioned open set target indexing stage, flexible retrieval can be achieved, supporting text-based target retrieval, image-based retrieval and image-text hybrid retrieval.

[0319] A. If the search request is a descriptive text for the entire image, the text is encoded with text features, and the search is directly performed in the corresponding full-image unstructured visual coding feature extraction data sub-database and the search results are returned;

[0320] B. If the search request is a descriptive text for a specific target, it can be directly sent to the CLIP_attr network. After obtaining the text encoding features, it is compared with the unstructured visual encoding features pre-saved in the database and the search results are returned.

[0321] C. If the retrieval request is a full image and the description information of a target of interest in the image, the detection box can be obtained by inputting the information into Groundingdino, and the patch is obtained from the original image and sent to the pre-trained CLIP_attr network to obtain the unstructured visual coding features and attribute features corresponding to the patch, and then compared with the unstructured visual coding features and attribute features pre-saved in the database, and the results are retrieved;

[0322] D. If the search request is for a specific target patch, it can be directly sent to the pre-trained CLIP_attr network to obtain the unstructured visual coding features and attribute features corresponding to the patch, and then compared with the unstructured visual coding features and attribute features pre-saved in the database, and the search results are retrieved;

[0323] In summary, this solution supports a variety of powerful and flexible retrieval methods in different forms.

[0324] (3) Optimization training of open set target features and multi-attribute extraction sub-network.

[0325] A. Fine-tuning CLIP in specific scenarios: Improving the expressiveness of unstructured visual encoding features in specific scenarios.

[0326] a) Fine-tuning data preparation for specific scenarios: In order to improve the feature expression ability of the CLIP model in the specific scenario of large traffic lights, the present invention prepares multiple image-text pairs of traffic scenarios. The targets involved are various common targets in traffic scenes, such as pedestrians, motor vehicles, non-motor vehicles, roadblocks, trash cans, etc. The image is a compact frame of each target, including each target. For example, the corresponding text form can be listed as follows:

[0327] Image 1, a car / truck / SUV of xx make and xx style in xx color

[0328] Image 2, a pedestrian / person of xx age and xx gender wearing xx top and xx bottoms;

[0329] Image 3, a pedestrian / person of xx gender riding an xx electric bike of xx color.

[0330] refer to Figure 3 , Figure 3 It is a schematic diagram of an optimization training process of an open set target feature and a multi-attribute extraction subnetwork provided in an embodiment of the present invention;

[0331] b) Training strategy: First, use the existing image pre-training model and text pre-training model to initialize the twin towers of the CLIP model respectively, and freeze the image pre-training parameters so that the language pre-training model is associated with the existing image pre-feature representation space; then partially unfreeze the image side parameters, and fine-tune the image and text data in a specific scenario through comparative learning to improve the model's feature expression ability in that scenario.

[0332] Text / Patch: Input data, which can be a text description or a region (patch) of an image.

[0333] Text Encoder: Text encoder converts text input into text feature vector.

[0334] Image Encoder: Image encoder converts an image or image patch into an image feature vector.

[0335] Feature: The final feature vector is the representation of text features and image features in the same embedding space.

[0336] Clip loss: The loss function of the CLIP model is used to shorten the feature distance of semantically related image-text pairs and push away semantically unrelated pairs.

[0337] B. Joint training module of multi-attribute features: After completing the fine-tuning of the CLIP model under specific scene data, all image pre-training parameters and language pre-training parameters are frozen, and the attribute feature extraction layer is added to complete the training.

[0338] a) In a specific scene, many different types of targets are involved, but only some types of targets are often given attribute attention. For example, in a large traffic scene, the attributes of targets such as pedestrians, vehicles, and cyclists are often more important and need more attention. Therefore, this module only adds corresponding multi-attribute training and feature extraction for important categories of targets;

[0339] During model training, all image pre-training parameters and language pre-training parameters are frozen, and multiple attribute extraction sub-networks are added to the back of a specific layer in the image pre-training model. The attribute extraction sub-network can be composed of one or more fully connected layers, and the cross entropy loss is introduced to complete the training of specific attributes. It should be noted that the attributes of the targets of each key category are inconsistent, and the number of attributes is also inconsistent. For example, the attributes of pedestrians include gender, age, hairstyle, etc., and the attributes of cars include color, type and brand, etc. Therefore, the multiple attribute training of different targets all share the same visual encoding network to reduce the number of parameters and calculations of the model. In the attribute training module, the training data of each target only participates in the parameter training of the corresponding multiple attributes. Therefore, the multi-attribute loss of the mth target category among all M key targets can be defined as follows m=1,...,M. Among them, for the kth attribute of the target, Indicates the number of categories of the k-th attribute of the m-th target category.

[0340] in For the prediction result of the k-th attribute of the m-th target category, Attr_loss m is the classification loss of the corresponding attribute. is the predicted value of the j-th dimension of the k-th attribute of the m-th target category, which is 1 if the original label is correct and 0 otherwise.

[0341] The training loss of the entire model is as follows, where γ is the weighted weight of the two losses to balance the CLIP backbone network loss and the multi-attribute loss of all M key targets.

[0342]

[0343] After training, for each of the M key target types, the visual encoding layer features and corresponding attribute prediction features from the CLIP backbone network are extracted and stored separately;

[0344] refer to Figure 4 , Figure 4 It is a schematic diagram of another optimization training process of an open set target feature and a multi-attribute extraction subnetwork provided in an embodiment of the present invention;

[0345] c) After completing the model training, the corresponding visual coding layer features and attribute set features are extracted for key targets of different categories, and used to construct a visual coding layer feature sub-index database and a multi-attribute feature sub-index database; for targets of other categories, only their visual coding layer features are extracted and a visual coding layer feature sub-index database is constructed;

[0346] d) When searching for attributes in the key target category, each target may only have a few of these attributes. After extracting the location information of the corresponding target from the Groudingdino module, it also contains its corresponding confidence information, which also helps to obtain more reliable attribute information of the corresponding target. Therefore, when predicting the kth attribute of the mth target category, it needs to be multiplied by the confidence information in the previous step to get the final judgment.

[0347]

[0348] Here is the score m is the confidence of the mth key target. For the prediction result of the kth attribute of the mth target category, the attribute threshold of the kth attribute of the mth target category will be set If the requirements are met, the final category information of the attribute will be given.

[0349]

[0350] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0351] Reference Figure 5 , shows a structural block diagram of a search result determination device provided in an embodiment of the present invention, which may specifically include the following modules:

[0352] The input image acquisition module 501 is used to acquire the input image;

[0353] A category label extraction module 502, used to determine an open set of objects in the input image and extract category labels of the open set of objects;

[0354] A text feature acquisition module 503 is used to generate a content description text for the category label and acquire text features of the content description text;

[0355] A location information acquisition module 504 is used to acquire the location information of the open set target;

[0356] An unstructured visual coding feature generation module 505, configured to generate unstructured visual coding features and attribute features based on the category label and the position information;

[0357] An open set target index module 506, configured to construct an open set target index through the text features, the unstructured visual encoding features and the attribute features;

[0358] The search result determination module 507 is used to determine the search result corresponding to the search request based on the open set target index when a search request is obtained.

[0359] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0360] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0361] Memory 603, used for storing computer programs;

[0362] The processor 601 is used to implement any of the search result determination methods described in the above embodiments when executing the program stored in the memory 603:

[0363] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0364] The communication interface is used for communication between the above terminal and other devices.

[0365] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0366] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0367] like Figure 7 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 701 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the search result determination method described in the above embodiment.

[0368] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0369] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0370] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0371] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0372] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0373] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0374] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0375] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for determining a search result, characterized in that: include: Get the input image; Determining an open set target of the input image, and extracting a category label of the open set target; Generate a content description text for the category label, and obtain text features of the content description text; Obtaining location information of the open set target; generating unstructured visual coding features and attribute features based on the category label and the position information; Constructing an open set target index through the text features, the unstructured visual encoding features and the attribute features; When a search request is obtained, a search result corresponding to the search request is determined based on the open set target index.

2. The method according to claim 1, characterized in that The unstructured visual coding feature is a global unstructured visual coding feature for the global content of the input image, and further includes: A global unstructured visual coding feature sub-index database is constructed for the global unstructured visual coding feature of the input image.

3. The method according to claim 2, characterized in that When a search request is obtained, the step of determining a search result corresponding to the search request based on the open set target index comprises: When a search request is obtained, and the search request includes a first target text for describing the entire image scene, obtaining a first target text feature vector of the first target text; Based on the first target text feature vector and the open set target index, a first target image corresponding to the first target text is determined from the global unstructured visual encoding feature sub-index database.

4. The method according to claim 1, characterized in that: The unstructured visual coding feature is an open set target unstructured visual coding feature for an open set target of the input image, and further includes: Constructing an open-set target unstructured visual coding feature sub-index database for the open-set target unstructured visual coding feature of the input image; According to the attribute features, a multi-attribute feature sub-index database is constructed.

5. The method according to claim 4, characterized in that When a search request is obtained, the step of determining a search result corresponding to the search request based on the open set target index comprises: When a search request is obtained, and the search request includes a second target text for describing the open set target, obtaining a second target text feature vector of the second target text; Based on the second target text feature vector and the open-set target index, a second target image corresponding to the second target text is determined from the open-set target unstructured visual encoding feature sub-index database.

6. The method according to claim 5, characterized in that When a search request is obtained, the step of determining a search result corresponding to the search request based on the open set target index comprises: When a search request is obtained, and the search request includes a search request image and a third target text for describing an object of interest of the search request image, obtaining the object of interest position information of the object of interest; Extracting an object of interest image of an object of interest from the retrieval request image based on the object of interest position information; Acquire an unstructured visual coding feature of an object of interest of the image of the object of interest; Based on the unstructured visual coding feature of the object of interest and the open set object index, a third object image is determined from the open set object unstructured visual coding feature sub-index database and the multi-attribute feature sub-index database.

7. The method according to claim 5, characterized in that When a search request is obtained, the step of determining a search result corresponding to the search request based on the open set target index comprises: When a retrieval request is obtained, and the retrieval request includes a retrieval request image and a cutout image of an object of interest of the retrieval request image, the cutout image is input into a pre-trained similarity calculation model, and a cutout image unstructured visual coding feature of the cutout image is obtained; Based on the cutout image unstructured visual coding feature and the open set target index, a fourth target image is determined from the open set target unstructured visual coding feature sub-index database and the multi-attribute feature sub-index database.

8. A search result determination device, characterized in that: include: An input image acquisition module is used to acquire an input image; A category label extraction module, used to determine the open set target of the input image and extract the category label of the open set target; A text feature acquisition module, used to generate a content description text for the category label and acquire text features of the content description text; A location information acquisition module, used to acquire the location information of the open set target; An unstructured visual coding feature generation module, used to generate unstructured visual coding features and attribute features based on the category label and the position information; An open set target index module, used for constructing an open set target index through the text features, the unstructured visual encoding features and the attribute features; The retrieval result determination module is used to determine the retrieval result corresponding to the retrieval request based on the open set target index when a retrieval request is obtained.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images