Image retrieval method, device and equipment for target image information
By constructing a structured index library for image data, the automated parsing and fusion of multimodal data is realized, solving the problems of high manual annotation costs and insufficient semantic understanding in existing image retrieval technologies, improving retrieval efficiency and accuracy, and supporting multimodal joint retrieval and complex queries.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA AI MEDIA&ENTERTAINMENT TECH CO LTD
- Filing Date
- 2025-12-26
- Publication Date
- 2026-05-12
AI Technical Summary
Existing image retrieval technologies rely on manual annotation, which is costly and inefficient, and makes it difficult to deeply understand the semantic content of images. This results in a large discrepancy between the retrieval results and the user's intent. Furthermore, the lack of effective correlation and fusion of multimodal data makes it impossible to support complex retrieval needs.
By automating the parsing of multimodal data and constructing a structured index library for image data, we can achieve efficient understanding and accurate matching of deep semantic content in images, including multimodal fusion analysis of images, videos and text and generation of structured descriptive information.
It significantly reduces reliance on manual annotation, improves the efficiency and objectivity of processing massive amounts of data, increases the precision and recall of retrieval, supports cross-modal joint retrieval and complex query methods, and makes retrieval results closer to the user's true intent.
Smart Images
Figure CN122019833A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of image retrieval technology, and in particular to an image retrieval method, apparatus and device for target image information. Background Technology
[0002] Current image retrieval technologies primarily rely on manually labeled keywords or simple metadata (such as capture time and file size), as well as matching based on low-level visual features like color and texture. This approach has significant drawbacks: First, manual labeling is costly, inefficient, and highly subjective, making it difficult to handle massive amounts of data. Second, simple metadata and low-level features cannot deeply understand the semantic content of images (such as specific targets, behaviors, and events), leading to significant discrepancies between search results and the user's true intent, resulting in low precision and recall. Third, multimodal data such as images, videos, and their associated text are often managed in isolation, lacking effective association and fusion, making cross-modal verification and joint retrieval impossible. Finally, it struggles to support demands such as image-based search and complex retrieval based on natural language descriptions. Summary of the Invention
[0003] The technical problem to be solved by the embodiments of the present invention is to provide an image retrieval method, apparatus and device for target image information, which can achieve efficient understanding and accurate matching of deep semantic content of images by automatically parsing multimodal data and building a structured index library.
[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: A method for retrieving target image information, comprising: Obtain the target image information to be retrieved; The target image information is analyzed and extracted to obtain target image feature information; The target image feature information is matched in a preset image data structured index library to retrieve search results containing the matched target image data and the structured description information associated with the target image data. Output the search results; wherein the image data structured index is obtained through the following process: Acquire multimodal raw data containing images, videos, and text descriptions of the same target object; perform content parsing and feature extraction on the multimodal raw data to obtain corresponding structured description information; store the structured description information in a preset storage format as an index structure that supports feature-based matching and retrieval to obtain the image data structured index library.
[0005] Optionally, the multimodal raw data is subjected to content parsing and feature extraction to obtain corresponding structured description information, including: The images in the multimodal raw data are preprocessed and feature extracted to obtain the first structured description information; The video in the multimodal raw data is preprocessed, keyframes are extracted, and features are extracted to obtain the second structured description information; Natural language processing and information extraction are performed on the text descriptions in the multimodal raw data to obtain third structured description information; The first structured description information, the second structured description information, and the third structured description information are associated to obtain the associated structured description information.
[0006] Optionally, the first structured description information, the second structured description information, and the third structured description information are associated to obtain associated structured description information, including: The target category label is extracted from the first structured description information, the second structured description information, and the third structured description information, respectively. Establish cross-modal data associations using common target category labels as association keys; For each target category, all associated multimodal structured information is integrated to obtain the associated structured description information.
[0007] Optionally, the structured description information is stored in a preset storage format as an index structure that supports feature-based matching and retrieval, resulting in the image data structured index library, including: From the structured description information, feature vectors of all modalities are extracted to construct a multimodal feature vector library; Create an inverted index using the target category label as the index key; Create a structured descriptive index entry for each target category label; The multimodal feature vector library, inverted index terms, and index entries are encapsulated into a structured index library for image data.
[0008] Optionally, the target image information is parsed and extracted to obtain target image feature information, including: Based on the target image information, determine the modality type of the target image information; Based on the modality type, the target image information is parsed and extracted to obtain target image feature information.
[0009] Optionally, the target image feature information is matched in a preset image data structured index library to retrieve search results containing the matched target image data and the structured description information associated with the target image data, including: Calculate the comprehensive similarity between the target image feature information and the content feature information in the image data structured index library; Based on the comprehensive similarity, the candidate items are sorted and filtered to obtain a sorted list of candidate items. The target image data and structured description information are retrieved and assembled from the image data structured index library to obtain the retrieval results.
[0010] Optionally, the above methods also include: Obtain the secondary filtering instruction triggered by the user based on the search results; The search results are filtered, reordered, and updated according to the secondary filtering instructions.
[0011] Embodiments of the present invention also provide an image retrieval device for target image information, comprising: The acquisition module is used to acquire the target image information to be retrieved; The processing module is used to parse and extract the target image information to obtain target image feature information; match the target image feature information in a preset image data structured index library to retrieve retrieval results containing the matched target image data and the structured description information associated with the target image data; and output the retrieval results. The image data structured index library is obtained through the following process: acquiring multimodal raw data containing images, videos, and text descriptions of the same target object; parsing and extracting features from the multimodal raw data to obtain corresponding structured description information; and storing the structured description information in a preset storage format as an index structure supporting feature-based matching retrieval to obtain the image data structured index library.
[0012] Embodiments of the present invention also provide a computing device, comprising: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to perform the method as described above.
[0013] Embodiments of the present invention also provide a computing device readable storage medium storing a program that, when executed by a processor, implements the method described above.
[0014] The above-described solutions of the embodiments of the present invention have at least the following beneficial effects: The above-described solution in this invention significantly reduces reliance on manual annotation by automating content parsing and feature extraction of multimodal raw data, thereby improving the efficiency and objectivity of processing massive amounts of data and reducing subjective errors.
[0015] By performing multimodal fusion analysis of images, videos, and associated text, structured descriptive information is generated, enabling a deeper understanding of the semantic content of specific targets, behaviors, and events in the images, thereby significantly improving the precision and recall of retrieval.
[0016] Based on standardized organization and index construction, it realizes effective association of multimodal data and cross-modal joint retrieval, and supports complex query methods such as image search and natural language description, making the retrieval closer to the user's true intent. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating the image retrieval method for target image information provided in an embodiment of the present invention.
[0018] Figure 2 This is a schematic diagram of a module of an image retrieval device for target image information provided in an embodiment of the present invention. Detailed Implementation
[0019] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
[0020] like Figure 1 As shown, an embodiment of the present invention provides an image retrieval method for target image information, including: Step 11: Obtain the target image information to be retrieved; Step 12: Analyze and extract the target image information to obtain target image feature information; Step 13: Match the target image feature information in a preset image data structured index library to retrieve the retrieval results containing the matched target image data and the structured description information associated with the target image data. Step 14: Output the search results; The image data structured index library is obtained through the following process: Step 101: Obtain multimodal raw data containing images, videos, and text descriptions of the same target object; Step 102: Perform content parsing and feature extraction on the multimodal raw data to obtain the corresponding structured description information; Step 103: Store the structured description information in a preset storage format as an index structure that supports feature-based matching and retrieval, thereby obtaining the image data structured index library.
[0021] In this example, by automating content parsing and feature extraction of multimodal raw data, the reliance on manual annotation is significantly reduced, the efficiency and objectivity of processing massive amounts of data are improved, and subjective errors are reduced.
[0022] By performing multimodal fusion analysis of images, videos, and associated text, structured descriptive information is generated, enabling a deeper understanding of the semantic content of specific targets, behaviors, and events in the images, thereby significantly improving the precision and recall of retrieval.
[0023] Based on standardized organization and index construction, it realizes effective association of multimodal data and cross-modal joint retrieval, and supports complex query methods such as image search and natural language description, making the retrieval closer to the user's true intent.
[0024] In an optional embodiment of the present invention, in step 101, multimodal raw data containing images, videos, and text descriptions of the same target object are acquired.
[0025] Specifically, through multi-source data access, images, videos, and text descriptions of the same target object from different devices and sources are collected synchronously to form a multimodal raw dataset. The target object can be military equipment, such as a tank. Specifically, using USB interfaces, network transmission protocols, or cloud storage interfaces, standard format image files, video streams, and corresponding text reports, annotations, or metadata are acquired in real time or in batches from devices such as cameras, scanners, satellites, drones, and mission recording terminals. During the acquisition process, an association mapping table is established for the same target object targeted by the images, videos, and text descriptions to ensure that each image and video data establishes a one-to-one correspondence with its related text description information.
[0026] This example demonstrates the simultaneous acquisition and format unification of multimodal data, including images, videos, and text, laying a high-quality data foundation for subsequent integrated processing. By establishing an association mapping table centered on the target object, semantic consistency and traceability among different modal data are ensured, overcoming the shortcomings of data isolation and reliance on manual association. In typical application scenarios such as military reconnaissance, it can support the real-time aggregation of key target image information from multi-dimensional sensing devices such as satellites, drones, and ground terminals, significantly improving the completeness, timeliness, and accuracy of data acquisition.
[0027] In an optional embodiment of the present invention, step 102 involves performing content parsing and feature extraction on the multimodal raw data to obtain corresponding structured description information, including: Step 1021 involves preprocessing and feature extraction of the images in the multimodal raw data to obtain first structured description information; specifically, this includes: Step 1021a: Perform preprocessing on the images in the multimodal raw data, including resolution unification, color space conversion, noise filtering, and illumination equalization, to obtain standardized images; Step 1021b, according to This yields a set of all detected bounding boxes. in, B The set of all detected bounding boxes. b j For the target bounding box, j = 1, 2, ..., M, M To standardize the total number of targets detected in the image, b j = ( x j , y j , w j , h j , c j ), x j and y j The coordinates of the center point of the target box. w j and h j The width and height of the target bounding box. c j The target category is labeled (e.g., tank, vehicle, or person), and `Dectect` is the target detection function. I norm To standardize images, E det For target detection parameters; Step 1021c, according to Visual features are extracted from all detected target bounding boxes to obtain corresponding visual feature vectors (encoding essential visual information such as the shape, texture, color, and structure of the target). in, f j For the corresponding visual feature vector, ψ visual For visual feature extraction function, E feat Extract parameters for visual features; Step 1021d, according to Attribute features are extracted from all detected target bounding boxes to obtain the corresponding attribute feature vectors (describe the dynamic or state information of the target). in, a jFor the corresponding attribute feature vector, ψ attr For attribute feature extraction function, E attr Extract parameters for attribute features; Step 1021e, according to This yields the first structured description information; in, F The first structured description information (is a collection containing the complete file of each target in the image); Step 1022 involves preprocessing the video in the multimodal raw data, extracting keyframes, and extracting features to obtain second structured description information; specifically, this includes: Step 1022a: Perform preprocessing on the video in the multimodal raw data, including format unification, resolution unification, frame rate standardization and color space conversion, to obtain a standardized video; Step 1022b, according to The standardized video is segmented into scenes to obtain a set of video clips; in, S A collection of video clips, s k The video segments are the result of the segmentation. k =1, 2, ..., N , N The number of segments into which the video is divided, and Segment is the scene segmentation function. V norm To standardize videos, E seg These are the parameters for scene segmentation. Step 1022c, according to Keyframes are extracted from a set of video clips to obtain a set of keyframes for the video clips. in, H A collection of keyframes from a video clip. h k，t Keyframes of a video clip t =1, 2, ..., T , T This represents the number of keyframes in the video clip, and `ExrtctKeyFrames` is the keyframe extraction function. E key Extract parameters for keyframes; Step 1022d: Perform feature extraction on the keyframe set of the video segment (consistent with the feature extraction process in step 1021) to obtain the target detection results, visual feature vectors, and attribute feature vectors of the keyframes; Step 1022e, according to Motion feature extraction is performed on the target detection results of the keyframe to obtain a motion feature vector; in, d （k，m） For motion feature vectors, ψ motion For motion feature extraction function, br （k，m） The target detection result for the keyframe. E motion Parameters are extracted for motion features. m =1, 2, ..., g , g In the k The total number of target objects with independent trajectories identified and associated in each video clip; Step 1022f, according to Behavioral features are extracted from the target detection results of the keyframes to obtain behavioral feature vectors; in, q （k，m） For behavioral feature vectors, ψ behavior For behavioral feature extraction function, br （k，m） The target detection result for the keyframe. E behavior Extract parameters for behavioral features; Step 1022g, according to This yields the second structured description information; in, V struct This is the second structured description information. s k The video segments are the result of the segmentation. h k，t Keyframes of a video clip c k For the target category label, X k，t This is a set of visual feature vectors and attribute feature vectors of the target object. G k，m This is a set of motion feature vectors and behavior feature vectors of the target object. Step 1023 involves performing natural language processing and information extraction on the text description in the multimodal raw data to obtain third structured description information; specifically, this includes: Step 1023a: Preprocessing the text description in the multimodal raw data to obtain standardized text; Step 1023b, according to Entities are extracted from standardized text to obtain an entity set; in, E For a collection of entities, e u The extracted entities (such as tanks, moving objects, and positions). u =1, 2, ..., V , V Where is the total number of entities, and EntityExtract is the entity extraction function. T norm For standardized text, E entity Parameters for entity extraction; Step 1023c, according to Extract the relationships between entities from standardized text to obtain a set of relationships; in, R For a set of relations, r v For relationships between entities (such as being located, belonging to, and executing). v =1, 2, ..., U , U Let represent the total number of relationships between entities, and RelationExtract be the relationship extraction function. T norm For standardized text, E relation Extract parameters for the relation; Step 1023d, according to This yields the third structured description information; in, T struct This is the third structured description information. E For a collection of entities, R For a set of relations; Step 1024: Associate the first structured description information, the second structured description information, and the third structured description information to obtain the associated structured description information; specifically, this includes: Step 1024a: Extract the target category label from the first structured description information, the second structured description information, and the third structured description information respectively (the target category label is extracted from the first structured description information). c j The target category label is extracted from the second structured description information. c k Entities are extracted from the third structured description information. e u From the entity e uThe system filters out entities that match a preset target category (such as tanks, vehicles, or personnel) as target category labels. c i ); Step 1024b: Using the common target category label as the association key, establish the association relationship between cross-modal data (including the first structured description information, the second structured description information, and the third structured description information). Specifically, generate a category association mapping table to aggregate images, video clips, and text descriptions belonging to the same target category. Step 1024c: For each target category, integrate all associated multimodal structured information to obtain associated structured description information (which includes all image features, video clips (including motion / behavioral features), and text semantic information describing the same type of target).
[0028] In this example, by performing standardized preprocessing, object detection, feature extraction, and semantic parsing on three types of heterogeneous data—images, videos, and text—the originally unstructured raw data is transformed into semantically rich, machine-understandable structured descriptive information (first, second, and third structured descriptive information). This process is completed by a pre-defined algorithm function, which significantly reduces the reliance on manual annotation and interpretation and greatly improves the efficiency of data processing.
[0029] For image / video keyframes: not only are visual feature vectors (shape, texture, and color) of the essential attributes of the target extracted, but also attribute feature vectors describing the target's dynamics or state are extracted, forming a dual description of the static target's appearance and state.
[0030] For video sequences: motion feature vectors reflecting spatiotemporal changes of the target and behavioral feature vectors representing the target's action intentions were further extracted, realizing the semantic extension from static appearance to dynamic behavior.
[0031] For text descriptions: structured factual knowledge (a set of entities and relations) is extracted through entity extraction and relation extraction.
[0032] Multi-level, fine-grained feature extraction enables the generated structured description information to deeply characterize the visual attributes, dynamic behavior, and textual semantics of the target object, laying a rich data foundation for subsequent accurate matching.
[0033] From the structured descriptions of images, videos, and texts, common target category labels are extracted as unified association keys; by generating a category association mapping table, image features, video clips (including motion / behavior), and text semantic information describing the same category of targets (such as all tanks) are aggregated.
[0034] It achieves deep semantic fusion and association of image, video and text data, and constructs a multimodal unified view with the target object as the core; it overcomes the defects of isolated storage of different modal data, difficulty in association or reliance on fragile manual rules, so that data is no longer an island, but forms an organic whole; it provides direct and efficient support for supporting complex cross-modal joint retrieval such as searching for video / text by image and searching for image / video by text.
[0035] In an optional embodiment of the present invention, in step 103, the structured description information is stored in a preset storage format as an index structure that supports feature-based matching and retrieval, thereby obtaining the image data structured index library, including: Step 1031: Extract feature vectors for all modalities from the structured description information to construct a multimodal feature vector library; specifically, the feature vectors for all modalities include: Visual feature vectors are visual feature vectors and attribute feature vectors extracted from first structured description information (images) and second structured description information (videos); Dynamic feature vectors are motion and behavioral feature vectors extracted from the second structured descriptive information (video). Semantic feature vectors are obtained by converting the entity set and relation set into dense vector representations from the third structured description information (text description) through word embedding or sentence encoding models. The normalized modal feature vectors are organized according to their source and target category label and stored in the vector database to form a multimodal feature vector library. Step 1032: Create an inverted index using the target category label as the index key; specifically, this includes: Traverse the multimodal feature vector library and create an inverted index for each unique target category label; Each inverted index entry records the storage location identifiers of all feature vectors belonging to that target category label; This inverted index allows for quick location of all relevant visual, dynamic, and semantic feature vectors via category labels; Step 1033: Create a structured description index entry for each target category label; specifically, this includes: category label; identifiers of associated image, video, and text data sources; metadata of each modality feature vector; and auxiliary fields such as spatiotemporal information and acquisition device information. Organize index entries into a tree or hash structure that supports fast queries; Step 1034: Encapsulate the multimodal feature vector library, inverted index terms, and index entries into a structured index library for image data.
[0036] In this example, the multimodal feature vector library constructed in step 1031 normalizes and centrally stores heterogeneous feature vectors (visual, attribute, dynamic, and semantic) from images, videos, and text, achieving unified management and efficient access to underlying features. The inverted index established in step 1032, with the target category label as the key, provides an extremely efficient retrieval entry point. Based on the target category (such as tank) specified in the user's query or filtering conditions, it can instantly locate the feature vectors of all modalities related to the category within constant or logarithmic time complexity, avoiding full database scanning and greatly improving retrieval response speed.
[0037] The structured description index entries established in step 1033 not only associate the identifiers of the original data, but also integrate the metadata of each modality feature vector and rich auxiliary fields such as collection time, geographical coordinates, and device information. These entries are organized into efficient tree or hash structures. When a candidate target is found through inverted index or feature matching, the complete file of the target can be obtained immediately through this index, including all its associated original data, extracted features, and key spatiotemporal context information. This elevates discrete feature matching to information retrieval with complete semantics and context, so that the returned results are not only similar images or videos, but also a collection of all known information about a target.
[0038] In an optional embodiment of the present invention, step 11, obtaining the target image information to be retrieved, may include: Step 111: Receive a search request initiated by a user or upper-layer application through a user interface, application programming interface or message queue; the search request carries the target image information to be searched, and the target image information can be one or more of the following forms: image file, video file, video stream or text description; Step 112: Perform parsing and standardization preprocessing on the received input data to ensure that it meets the input requirements of the subsequent feature parsing and extraction module. If the input is an image or video file: perform format validation and convert it to the standardized format required for processing (such as specifying the encoding format, resolution, and color space). If the input is a text description: perform natural language preprocessing on it, such as word segmentation, removal of stop words, and conversion into a normalized text sequence; Step 113: The standardized input data is encapsulated into a unified internal system data structure describing the target object to be retrieved; this structure must contain at least the following fields: The unique identifier for this search request; The modality type of the input data, such as image, video, text, or a mixture; A copy of the original data or its reference address in the storage system; Relevant metadata, such as input time, source channel, and user identifier.
[0039] This example supports receiving one or more forms of search requests, such as image files, video files, video streams, and text descriptions. It breaks the limitation of traditional search systems that usually only support a single modality (such as only uploading images or only entering keywords), allowing users to initiate queries in the most natural and convenient way based on the information they have. For example, reconnaissance personnel can directly upload a blurry photo of a target taken at the border, or enter a text description from an intelligence briefing (such as discovering a suspected new type of tank), or submit both for a combined query. This significantly lowers the barrier to entry and makes the search process closer to actual workflows and user intuition.
[0040] After receiving heterogeneous raw input, for images / videos, format verification and standardization conversion (unified encoding, resolution and color space) are performed; for text, preprocessing such as word segmentation and stop word removal is performed; raw data from different sources and with different specifications are cleaned and converted into a standardized format; this ensures that feature parsing and extraction can work stably on a consistent and standardized data basis, thereby guaranteeing the accuracy and consistency of the final retrieval results.
[0041] In an optional embodiment of the present invention, step 12 involves parsing and extracting the target image information to obtain target image feature information, including: Step 121: Determine the modality type of the target image information based on the target image information; specifically, this includes: Parse the description of the target object to be retrieved generated in step 11 to determine the modality type of the input data, including at least one of image, video or text; Step 122: Based on the modality type, analyze and extract the target image information to obtain target image feature information; specifically, this includes: Step 1221, if the modality type includes an image: then process it according to the same or compatible feature extraction process as step 1021, specifically including: Image preprocessing includes resolution unification, color space conversion, noise filtering, and illumination equalization to obtain a standardized image; Perform object detection on the standardized image to obtain a set of bounding boxes; For each bounding box, extract its visual feature vector and attribute feature vector; The extracted visual feature vector set and attribute feature vector set are added to the target image feature information; Step 1222, if the modality type includes video: then process according to the same or compatible feature extraction process as step 1022, specifically including: The video is preprocessed, including format unification, resolution unification, frame rate standardization, and color space conversion, to obtain a standardized video. Perform scene segmentation and keyframe extraction on standardized videos; For the extracted keyframes, perform the same object detection and feature extraction process as for the image to obtain visual feature vectors and attribute feature vectors; For video clips, extract motion feature vectors and behavior feature vectors; The extracted visual feature vectors, attribute feature vectors, motion feature vectors, and behavioral feature vectors are added to the target image feature information; Step 1223, if the modality type contains text: then process it according to the same or compatible feature extraction process as step 1023, specifically including: The text is preprocessed to obtain standardized text; Convert standardized text into semantic feature vectors; The extracted semantic feature vectors are added to the target image feature information; Step 1224: Organize all extracted feature vectors according to their modality and type to form a structured target image feature information object. Its data structure may include the following fields: List of visual feature vectors; List of attribute feature vectors; List of motion feature vectors; List of behavioral feature vectors; List of semantic feature vectors.
[0042] In this example, step 121 clearly identifies the modal type of the query input (image, video, text, or a combination thereof). Based on this determination, the feature extraction process, which is completely consistent with the one used in step 102 when building the index library, is adaptively invoked. Whether it is a query image uploaded by the user, a video clip, or a text description, the system can understand its input form and perform deep analysis using the same feature representation as the data in the library. Based on the adaptive input design and the standardization of backend processing, the system can intelligently and accurately handle various forms of user query intent, even mixed forms, greatly enhancing the system's applicability and user-friendliness.
[0043] Step 122, the processing of query data (preprocessing, detection, and feature extraction), strictly follows the methods used when constructing the index library. For example, the target detection model and visual feature extraction function of the query image are consistent with the model used for image processing in the library. This ensures that the target image feature information extracted from the query data is in the same feature space as the content feature information pre-stored in the index library, and has direct comparability. This is the fundamental technical guarantee that the subsequent similarity calculation can be carried out effectively and fairly, and ultimately achieve high precision and recall, avoiding matching deviations or failures caused by inconsistent feature extraction standards.
[0044] In an optional embodiment of the present invention, step 13, matching the target image feature information in a preset image data structured index library to retrieve a retrieval result containing the matched target image data and the structured description information associated with the target image data, may include: Step 131: Calculate the comprehensive similarity between the target image feature information and the content feature information in the image data structured index library; specifically, this includes: Step 1311: Using the target image feature information generated in step 12 as the query vector, and the content feature information (including visual feature vectors, dynamic feature vectors, semantic feature vectors and their metadata) stored in the image data structured index library constructed in step 103 as the matching target, perform a similarity calculation operation. Step 1312, according to To determine the similarity between the target image feature information and each content feature information; in, XSD dy ( de The similarity between the target image feature information and each content feature information is denoted as . dy=1, 2, ..., ds, ds The total number of feature vectors representing the feature information of the target image. de=1, 2, ..., dw, dw is the total number of content feature information in the index, and sim is the similarity calculation function. XL dy This refers to the feature vector in the target image feature information. YL dy，de These are feature vectors from the content feature information in the index library; Step 1313, according to Determine the comprehensive similarity between the target image feature information and the content feature information in the image data structured index library; in, S ( de The similarity score is calculated as the combined similarity between the target image feature information and the content feature information in the image data structured index library. w dyThese are the weights of each feature vector; Step 132: Based on the comprehensive similarity, sort and filter the candidate items to obtain a sorted list of candidate items; specifically, this includes: Step 1321: Sort the set of comprehensive similarity scores calculated in step 131 in descending order to obtain a sorted list of candidate entries; Step 1322: Based on the preset similarity threshold, filter out all candidate items to form a preliminary matching result set; Step 1323: Record the storage location in the index of the target image data identifier and its associated structured description information corresponding to each matching entry; Step 133: Retrieve and assemble matching target image data and structured description information from the image data structured index library to obtain retrieval results; specifically, this includes: Step 1331: Based on the matching entry identifier and storage location obtained in step 132, retrieve the corresponding original target image data (such as image files, video clips) and its complete structured description information (including visual features, attributes, motion behavior, semantic entities and relationships, etc.) from the image data structured index library. Step 1332: Assemble the retrieved data into a unified search result data structure; each search result entry must contain at least: A unique identifier for the target image data; Access address or thumbnail of the raw image data; This data is associated with a complete structured descriptive information; Matching similarity scores; Auxiliary information, such as collection time, location, and source device; Step 1333: Sort all the search results of matching entries in descending order of similarity score to form the final search result list.
[0045] This example not only supports similarity calculation for single-modal features (such as pure visual or pure text), but also implements adaptive weighted fusion matching of multimodal feature vectors. The system can assign appropriate weights to different types of features (visual, semantic, and dynamic features, etc.) according to the specific composition of the query (such as mixed text and image queries) and the actual application scenario. This enables the retrieval process to comprehensively consider multiple aspects of information such as the static appearance, dynamic behavior, and text description of the target, achieving a more accurate understanding and matching of the user's complex and comprehensive query intent. This fundamentally improves the precision rate and can adapt to diverse retrieval needs, avoiding the limitations of single feature matching.
[0046] Step 132 performs rapid sorting and threshold filtering on a massive number of candidate items based on the comprehensive similarity score; descending sorting ensures that the most relevant results to the query are presented first; while filtering based on preset thresholds can effectively filter out noisy data with low relevance and control the quality and size of the returned result set; this process greatly optimizes the retrieval efficiency and result purity, so that users do not need to manually filter through a large number of irrelevant results and can quickly focus on high-value information.
[0047] Step 133 retrieves the original image data and its associated complete structured description information from the index database based on the matching identifier, and integrates them with auxiliary information such as matching score and spatiotemporal metadata to assemble them into a unified search result entry. What is returned to the user is not just a link to an image or a video, but an information package containing a comprehensive profile of the target: it includes raw data that can be previewed intuitively, structured features and semantic relationships that can be analyzed in depth by machines, similarity scores that indicate credibility, and key contextual information; which greatly enriches the information dimension and practical value of the search results.
[0048] In an optional embodiment of the present invention, step 14, outputting the search results, includes: The matched target image data and its associated structured description information are uniformly formatted and organized to generate entries containing thumbnails, feature summaries, similarity scores, and spatiotemporal metadata. Rendering and displaying multiple views (list, grid, and details) through a graphical user interface, supporting result sorting, filtering, and multimodal data linkage viewing; Meanwhile, it provides standard data interfaces and file export functions for other systems to call or generate analysis reports; in addition, the system records the retrieval process and user feedback to optimize retrieval performance.
[0049] In this example, a unified formatting and multi-view visualization rendering transforms the results into an intuitive display; it generates clear entries containing thumbnails, feature summaries, similarity and spatiotemporal metadata, making complex multimodal matching results readily apparent; and operators can quickly grasp the core information of the search results.
[0050] In an optional embodiment of the present invention, after outputting the search results in step 14, the method further includes: Step 15: Obtain the secondary filtering instruction triggered by the user based on the search results; specifically, it includes: When a user interacts with the visualization results interface generated in step 14, the system listens for and captures in real time interactive events indicating the user's intention to refine and sort the current search results. These interactive events include, but are not limited to: selecting specific attribute conditions (such as target type or time range) in a drop-down menu, checking checkboxes to filter data sources, or clicking the sort button to change the result organization method. This interactive event is parsed into a structured secondary filtering instruction, which clearly includes the filtering dimensions, specific condition values, and sorting preferences added or modified by the user. Step 16: Filter, reorder, and update the search results according to the secondary filtering instructions; specifically, this includes: Step 161: Parse the secondary filtering instruction and extract the filtering conditions and sorting parameters; specifically, this includes: Semantic parsing and structural transformation are performed on the secondary filtering instructions obtained in step 15: the user-specified filtering conditions are extracted from the instructions, including but not limited to target object type, target status, time range, geographical region and acquisition device model; at the same time, the user-specified sorting parameters are extracted, including but not limited to ascending / descending order by acquisition time, geographical distance from near to far, number of targets in the image from most to least, or restored to sorting based on initial similarity; Step 162: Filter the search results one by one according to the filtering conditions; specifically, this includes: Using the final search result list output in step 133 and the filtering conditions extracted in step 161 as input, a traversal filtering operation is performed: for each target image data and its associated structured description information in the search results, it is determined whether all filtering conditions are met; the specific judgment logic includes: matching the target object type specified by the user based on the target category label in the structured description information; matching the target state specified by the user based on the attribute or behavioral characteristics; determining whether it is within the time range specified by the user based on the acquisition timestamp; determining whether it is located within the geographical area specified by the user based on the geographical coordinates; the system retains only the entries that fully meet all filtering conditions, forming a filtered data subset; Step 163: Reorder the filtered data subset according to the sorting parameters; specifically, this includes: The filtered data subset obtained in step 162 and the sorting parameters extracted in step 161 are sorted as follows: If the sorting parameter is specified as sorting by time, the data is sorted in ascending or descending order based on the collection time field of each data point; if the sorting parameter is specified as sorting by geographical distance, the Euclidean distance between each data point and the reference coordinates (such as the current observation point) provided by the user is calculated, and the data is sorted from closest to furthest; if the sorting parameter is specified as sorting by the number of targets, the data is sorted based on the number of targets detected in the data; if the initial similarity sorting is specified as restoring the initial similarity sorting, the data is sorted in descending order based on the original similarity scores calculated in step 131; finally, a result list is generated that is reorganized according to the user's intention. Step 164: Integrate and output the final search results after secondary filtering; specifically, this includes: The reordered list obtained in step 163 is assembled and output. The list is merged and its consistency is adjusted with other matching data that need to be processed synchronously in the search results (such as associated video clips and text reports) to ensure that the cross-modal data correlation is maintained. Finally, the system outputs a structured final search result that has been filtered and sorted twice, and immediately updates and displays it in the visualization interface to replace the initial result output in step 14, thus completing the complete closed loop from initial search to user interaction optimization.
[0051] This example introduces a targeted optimization step based on real-time user interaction. When the initial search results (output of step 14) are still too broad or the sorting does not match the user's current focus, the user does not need to rethink and initiate a completely new query. Instead, they can directly express their immediate and more refined intentions on the results interface through intuitive interactive operations (such as drop-down menus, sliders, and checkboxes). Step 15 captures and parses these intentions in real time to form structured secondary filtering instructions. This makes the search process no longer one-way and one-time, but evolves into a collaborative iterative process in which the system initially casts a wide net and the user interacts to precisely capture the net, significantly improving the efficiency and accuracy of users in quickly locking in high-value targets from massive amounts of information.
[0052] The filtering logic in step 162 relies entirely on specific fields in this information: target category, status attribute, collection timestamp, and geographic coordinates, etc. Users can not only perform category-based filtering, but also perform complex combination filtering based on status (such as in motion), spatiotemporal (such as within the past 24 hours, east of area A), and even collection source. This makes in-depth analysis and contextualized focusing of search results exceptionally convenient and powerful.
[0053] The reordering function in step 163 gives users the ability to customize the information organization logic; users can choose to sort by time to follow the latest developments based on the urgency of the task; choose to sort by distance based on geographical location to focus on nearby targets; or choose to sort by quantity based on the degree of target clustering to discover key areas; allowing users to reorganize the results into the form most beneficial to their decision-making, so that the information presentation can dynamically adapt to the changing practical or analytical scenarios, improving the operability and insight of the results.
[0054] Example 1 Take military reconnaissance missions as an example; A method for retrieving target image information, comprising: Step 21: Obtain multimodal raw data containing the same target object. By accessing data from multiple sources, multimodal raw datasets targeting the same region are collected synchronously from different devices and sources: Image data: High-resolution remote sensing images of the area acquired via satellite; Video data: Continuous reconnaissance is conducted using drones to generate surveillance video streams; Related text data: Synchronously acquire text reports such as reconnaissance briefings, target identification descriptions, and coordinate parameter records for this mission; During the data collection process, a one-to-one correspondence between image, video, and text data is established through related information such as task identifiers, timestamps, and geographic coordinates. Step 22: Perform content parsing and feature extraction on the multimodal raw data to obtain the corresponding structured description information. The collected multimodal raw data were analyzed and features were extracted respectively: Image data parsing: After standardizing and preprocessing the original image, the target detection and recognition model is called for processing; the generated first structured description information includes: the detected target category (e.g., tank), quantity, location coordinates, attribute status (e.g., stationary, camouflaged) and corresponding visual feature vector; Video data parsing: After standardizing the video and performing scene segmentation and keyframe extraction, the same target detection and feature extraction as for images are performed on the keyframes, and the motion trajectory and behavior patterns of the targets are further analyzed; the generated second structured description information includes: the time range of the video segment, the keyframe sequence, the target category, the set of visual and attribute features, and the motion features and behavior feature vectors of the targets; Text data parsing: Natural language processing is performed on reconnaissance briefings to extract key entities (such as tanks and positions) and relationships between entities (such as location and movement); the generated third structured description information is a structured set of entities and relationships; Extract target category labels (e.g., tanks) from the three types of structured description information mentioned above; use the common target category label as the association key to generate a category association mapping table, and aggregate all image features, video clips (including motion / behavioral features) and text semantic information describing the same type of target to obtain the associated structured description information; Step 23: Store the structured description information as an image data structured index library. From the associated structured description information, feature vectors (visual, attribute, motion, behavior, and semantic) of all modalities are extracted to construct a multimodal feature vector library; An inverted index is built using the target category label as the index key to record the storage location of all feature vectors under each category; A structured descriptive index entry is created for each target category, including category labels, associated data source identifiers, feature metadata, and spatiotemporal information of data collection, and organized into an efficient query structure; The above components are encapsulated to form a structured index library for image data that supports fast feature-based matching and retrieval. Step 24: Obtain the target image information to be retrieved. Users can upload a query image containing the target object through the system interface and enter relevant text descriptions (such as geographical location and time range). Step 25: Analyze and extract the target image information to obtain target image feature information. The system preprocesses, detects, and extracts features from the user-submitted query images to obtain visual feature vectors and attribute feature vectors; it also processes the input text descriptions to obtain semantic feature vectors; and finally organizes them into structured target image feature information. Step 26: Match the target image feature information in a preset image data structured index library to obtain the search results. Calculate the comprehensive similarity between the feature information of the target image and the content feature information in the index; All candidate entries are sorted and filtered based on comprehensive similarity to form a preliminary matching result set; The target image data (such as images and video clips) with the highest matching degree and its complete structured description information are retrieved from the index and assembled into a list of search results; Step 27: Output the search results. The system displays the search results in a visual format; each result includes a thumbnail of the matched image, a summary of key information (such as target type, coordinates and status), and a similarity score; Step 28: Obtain the secondary filtering instruction triggered by the user based on the search results. Users can further specify filtering conditions (such as target type being tank and status being moving) through interactive controls on the results interface (such as drop-down menus and checkboxes); the system captures and parses this interactive event into a structured secondary filtering instruction; Step 29: Filter, reorder, and update the search results according to the secondary filtering instructions. Parse the instructions to extract specific filtering conditions and sorting parameters; Based on the filtering criteria, the output search results list is filtered item by item, retaining only the data that meets all the criteria; The filtered data subset is reordered based on sorting parameters (such as sorting by the latest collection time). The system integrates and outputs the final search results, which are then updated and displayed on the interface, allowing users to quickly focus on the target image information that best matches their precise intent.
[0055] This invention significantly reduces reliance on manual annotation and improves processing efficiency and objectivity through automatic parsing and feature extraction of multimodal data. By constructing a structured index library, it achieves deep fusion and efficient organization of multimodal data, supporting rapid retrieval based on categories and features. During the retrieval stage, the system can intelligently parse diverse query inputs and accurately return results through multimodal feature fusion matching, greatly improving precision and recall. The results are presented intuitively in an interactive interface and support users to perform secondary filtering and reordering, realizing a closed-loop optimization from initial system retrieval to human-computer collaborative refinement. The overall solution effectively solves problems such as insufficient semantic understanding, modal isolation, and low efficiency in image retrieval, and is particularly suitable for scenarios with extremely high requirements for timeliness and accuracy, such as military reconnaissance.
[0056] like Figure 2 As shown, embodiments of the present invention also provide an image retrieval device 20 for target image information, comprising: The acquisition module 21 is used to acquire the target image information to be retrieved; Processing module 22 is used to parse and extract the target image information to obtain target image feature information; match the target image feature information in a preset image data structured index library to retrieve retrieval results containing the matched target image data and the structured description information associated with the target image data; and output the retrieval results; wherein, the image data structured index library is obtained through the following process: acquiring multimodal raw data containing images, videos, and text descriptions of the same target object; performing content parsing and feature extraction on the multimodal raw data to obtain corresponding structured description information; and storing the structured description information in a preset storage format as an index structure that supports feature-based matching retrieval to obtain the image data structured index library.
[0057] Optionally, the multimodal raw data is subjected to content parsing and feature extraction to obtain corresponding structured description information, including: The images in the multimodal raw data are preprocessed and feature extracted to obtain the first structured description information; The video in the multimodal raw data is preprocessed, keyframes are extracted, and features are extracted to obtain the second structured description information; Natural language processing and information extraction are performed on the text descriptions in the multimodal raw data to obtain third structured description information; The first structured description information, the second structured description information, and the third structured description information are associated to obtain the associated structured description information.
[0058] Optionally, the first structured description information, the second structured description information, and the third structured description information are associated to obtain associated structured description information, including: The target category label is extracted from the first structured description information, the second structured description information, and the third structured description information, respectively. Establish cross-modal data associations using common target category labels as association keys; For each target category, all associated multimodal structured information is integrated to obtain the associated structured description information.
[0059] Optionally, the structured description information is stored in a preset storage format as an index structure that supports feature-based matching and retrieval, resulting in the image data structured index library, including: From the structured description information, feature vectors of all modalities are extracted to construct a multimodal feature vector library; Create an inverted index using the target category label as the index key; Create a structured descriptive index entry for each target category label; The multimodal feature vector library, inverted index terms, and index entries are encapsulated into a structured index library for image data.
[0060] Optionally, the target image information is parsed and extracted to obtain target image feature information, including: Based on the target image information, determine the modality type of the target image information; Based on the modality type, the target image information is parsed and extracted to obtain target image feature information.
[0061] Optionally, the target image feature information is matched in a preset image data structured index library to retrieve search results containing the matched target image data and the structured description information associated with the target image data, including: Calculate the comprehensive similarity between the target image feature information and the content feature information in the image data structured index library; Based on the comprehensive similarity, the candidate items are sorted and filtered to obtain a sorted list of candidate items. The target image data and structured description information are retrieved and assembled from the image data structured index library to obtain the retrieval results.
[0062] Optionally, the retrieval device 20 further includes: Obtain the secondary filtering instruction triggered by the user based on the search results; The search results are filtered, reordered, and updated according to the secondary filtering instructions.
[0063] It should be noted that this device is a device corresponding to the above method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.
[0064] Embodiments of the present invention also provide a computing device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.
[0065] Embodiments of the present invention also provide a computing device readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.
[0066] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computing device software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0067] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0068] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0069] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0070] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0071] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computing device-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computing device software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a personal computing device, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.
[0072] Furthermore, it should be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of the present invention. Moreover, the steps performing the above-described series of processes can naturally be executed in the order described, but are not necessarily required to be executed in chronological order; some steps can be executed in parallel or independently of each other. Those skilled in the art will understand that all or any step or component of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software, or a combination thereof. This is something that those skilled in the art can achieve using basic programming skills after reading the description of the present invention.
[0073] Therefore, the object of the present invention can also be achieved by running a program or a set of programs on any computing device. The computing device can be a known general-purpose device. Therefore, the object of the present invention can also be achieved simply by providing a program product containing program code implementing the method or apparatus. That is, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any known storage medium or any storage medium developed in the future. It should also be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent to the present invention. Furthermore, the steps performing the above series of processes can naturally be performed in the order described, but are not necessarily required to be performed in chronological order. Some steps can be performed in parallel or independently of each other.
[0074] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for retrieving target image information, characterized in that, include: Obtain the target image information to be retrieved; The target image information is analyzed and extracted to obtain target image feature information; The target image feature information is matched in a preset image data structured index library to retrieve search results containing the matched target image data and the structured description information associated with the target image data. Output the search results; wherein the image data structured index is obtained through the following process: Acquire multimodal raw data containing images, videos, and text descriptions of the same target object; The multimodal raw data is parsed and its features are extracted to obtain the corresponding structured description information; The structured description information is stored in a preset storage format as an index structure that supports feature-based matching and retrieval, thus obtaining the image data structured index library.
2. The image retrieval method for target image information according to claim 1, characterized in that, The multimodal raw data is subjected to content parsing and feature extraction to obtain corresponding structured description information, including: The images in the multimodal raw data are preprocessed and feature extracted to obtain the first structured description information; The video in the multimodal raw data is preprocessed, keyframes are extracted, and features are extracted to obtain the second structured description information; Natural language processing and information extraction are performed on the text descriptions in the multimodal raw data to obtain third structured description information; The first structured description information, the second structured description information, and the third structured description information are associated to obtain the associated structured description information.
3. The image retrieval method for target image information according to claim 2, characterized in that, The first structured description information, the second structured description information, and the third structured description information are associated to obtain the associated structured description information, including: The target category label is extracted from the first structured description information, the second structured description information, and the third structured description information, respectively. Establish cross-modal data associations using common target category labels as association keys; For each target category, all associated multimodal structured information is integrated to obtain the associated structured description information.
4. The image retrieval method for target image information according to claim 3, characterized in that, The structured description information is stored in a preset storage format as an index structure that supports feature-based matching and retrieval, resulting in the image data structured index library, including: From the structured description information, feature vectors of all modalities are extracted to construct a multimodal feature vector library; Create an inverted index using the target category label as the index key; Create a structured descriptive index entry for each target category label; The multimodal feature vector library, inverted index terms, and index entries are encapsulated into a structured index library for image data.
5. The image retrieval method for target image information according to claim 1, characterized in that, The target image information is parsed and extracted to obtain target image feature information, including: Based on the target image information, determine the modality type of the target image information; Based on the modality type, the target image information is parsed and extracted to obtain target image feature information.
6. The image retrieval method for target image information according to claim 1, characterized in that, The target image feature information is matched against a preset image data structured index library to retrieve search results containing the matched target image data and the structured description information associated with the target image data, including: Calculate the comprehensive similarity between the target image feature information and the content feature information in the image data structured index library; Based on the comprehensive similarity, the candidate items are sorted and filtered to obtain a sorted list of candidate items. The target image data and structured description information are retrieved and assembled from the image data structured index library to obtain the retrieval results.
7. The image retrieval method for target image information according to claim 1, characterized in that, Also includes: Obtain the secondary filtering instruction triggered by the user based on the search results; The search results are filtered, reordered, and updated according to the secondary filtering instructions.
8. An image retrieval device for target image information, characterized in that, include: The acquisition module is used to acquire the target image information to be retrieved; The processing module is used to parse and extract the target image information to obtain target feature information; The target feature information is matched in a preset image data structured index library to retrieve search results containing the matched image data and the structured description information associated with the image data. Output the search results; wherein, the construction process of the image data structured index library includes: acquiring multimodal raw data containing images, videos and associated text; performing content parsing and feature extraction on the multimodal raw data to obtain corresponding structured description information; organizing the structured description information according to a preset specification and storing it as an index structure that supports feature-based matching retrieval to obtain the image data structured index library.
9. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 7.
10. A computing device readable storage medium, characterized in that, The computing device readable storage medium stores a program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.