Image retrieval method and electronic equipment

Abstract

The invention provides an image retrieval method and electronic equipment. The image retrieval method comprises the following steps: generating a question vector according to an input question; according to the problem vector, at least one target image is determined from the multiple images, each image in the multiple images comprises at least one image block, different image blocks of the images represent different semantics, the target image comprises at least one target image block, and the similarity between the block vector corresponding to the target image block and the problem vector meets a first target condition; the block vectors represent the features of the corresponding blocks; and determining a retrieval result of the input question according to the target image blocks in the target images.

Description

Technical Field

[0001] This disclosure relates to the field of image retrieval technology, and in particular to an image retrieval method and electronic device. Background Technology

[0002] In the field of image-based knowledge retrieval, the common strategy is "whole-image search, whole-image return," which utilizes whole-image-level multimodal vectors for retrieval. However, when a user's query intent is only related to local content in an image (such as a specific graphic, text block, or logical module), the whole-image-level vectors dilute the local semantics within the global context, leading to a decrease in search accuracy. Summary of the Invention

[0003] This disclosure provides an image retrieval method and an electronic device.

[0004] According to one aspect of this disclosure, an image retrieval method is provided, comprising: generating a question vector based on an input question; determining at least one target image from a plurality of images based on the question vector, wherein each of the plurality of images includes at least one patch, different patches of each image represent different semantics, the target image includes at least one target patch, the similarity between the block vector corresponding to the target patch and the question vector satisfies a first target condition, and the block vector represents the features of its corresponding patch; and determining the retrieval result of the input question based on the target patches in each target image.

[0005] According to embodiments of this disclosure, the method further includes: performing semantic segmentation on each of the multiple images to obtain at least one patch for each image; and generating a block vector corresponding to each patch based on the at least one patch for each image.

[0006] According to embodiments of this disclosure, generating a block vector corresponding to each image based on at least one image patch includes: obtaining content information, location information, and relationship information for each image patch, wherein the content information represents the content features of each image patch, the location information represents the location features of each image patch in its corresponding image, and the relationship information represents the relationship features between each image patch and other images in its corresponding image; and generating a block vector corresponding to each image patch based on the content information, location information, and relationship information.

[0007] According to embodiments of this disclosure, generating a block vector corresponding to each tile based on content information, location information, and relationship information includes: using a deep learning model to calculate the attention weights of the content information, location information, and relationship information respectively, and thereby generating a block vector corresponding to each tile.

[0008] According to embodiments of this disclosure, determining at least one target image from a plurality of images based on a question vector includes: determining at least one candidate image from the plurality of images based on the question vector and graph vectors corresponding to each image in the plurality of images, wherein the graph vectors characterize the features of their corresponding images, and the similarity between the graph vectors corresponding to the candidate images and the question vectors satisfies a second objective condition; determining at least one target patch based on the question vector and the block vectors of each patch in each candidate image; and determining at least one target image from at least one candidate image based on at least one target patch.

[0009] According to embodiments of this disclosure, the method further includes: generating a graph vector corresponding to each image based on the content features of each image in the plurality of images.

[0010] According to embodiments of this disclosure, determining the retrieval result of the input question based on target patches in each target image includes: generating a first similarity score for each target patch in at least one target image based on the similarity between the question vector and the block vector corresponding to the target patch; generating a second similarity score for each target image based on the similarity between the question vector and the graph vector corresponding to the target image, wherein the graph vector represents the features of its corresponding image; sorting at least one target image based on the first similarity score of the target patch in each target image and the second similarity score of each target image to obtain a target arrangement order; and arranging at least one target image according to the target arrangement order to obtain the retrieval result.

[0011] According to embodiments of this disclosure, sorting at least one target image based on a first similarity score of target patches in each target image and a second similarity score of each target image to obtain a target arrangement order includes: using a visual language model to generate a confidence score for each target image based on an input question and each target image, wherein the visual language model is trained based on sample images and sample text corresponding to the sample images; and sorting at least one target image based on the first similarity score of target patches in each target image, the second similarity score of each target image, and the confidence score to obtain a target arrangement order.

[0012] According to embodiments of this disclosure, arranging at least one target image in a target order to obtain retrieval results includes: arranging at least one target image in a target order and adding corresponding target identifiers to target patches in each target image to obtain retrieval results, wherein the target identifier represents the position information of the target patch in its corresponding target image.

[0013] Another aspect of this disclosure provides an electronic device, comprising: an acquisition module for acquiring an input question; a processor communicatively connected to the acquisition module for generating a question vector based on the input question; determining at least one target image from a plurality of images based on the question vector, wherein each image in the plurality of images includes at least one patch, different patches of each image represent different semantics, the target image includes at least one target patch, the similarity between the block vector corresponding to the target patch and the question vector satisfies a first target condition, and the block vector represents the features of its corresponding patch; determining retrieval results for the input question based on the target patches in each target image; and a display screen for displaying the retrieval results.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0016] Figure 1 This is a flowchart of an image retrieval method according to an embodiment of the present disclosure;

[0017] Figure 2 This is a schematic diagram of semantic segmentation of each image in a plurality of images according to embodiments of the present disclosure;

[0018] Figure 3 This is a flowchart of an image retrieval method according to another embodiment of the present disclosure;

[0019] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure; and

[0020] Figure 5 This is a schematic block diagram of an example electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0023] Figure 1 This is a flowchart of an image retrieval method according to an embodiment of the present disclosure.

[0024] like Figure 1 As shown, the image retrieval method of this embodiment includes operations S110-S130.

[0025] In operation S110, a question vector is generated based on the input question.

[0026] In embodiments of this disclosure, the input question may be a search request submitted by a user in natural language, keywords, or other forms.

[0027] For example, the input question could be "Find all pictures that contain a red car dashboard." The input question serves as the starting point and objective of the retrieval process.

[0028] In embodiments of this disclosure, the question vector can be a vector representation of the input question obtained after feature extraction and numericalization. For example, the semantics of the input question can be mapped to a vector space using a text encoding model to obtain the question vector. The question vector encodes the overall semantic intent of the question.

[0029] In operation S120, at least one target image is determined from multiple images based on the question vector. Each image in the multiple images includes at least one patch, and different patches of each image represent different semantics. The target image includes at least one target patch, and the similarity between the block vector corresponding to the target patch and the question vector satisfies the first objective condition. The block vector represents the features of its corresponding patch.

[0030] In embodiments of this disclosure, the multiple images constitute a set of images to be retrieved. For example, the multiple images may be from a database, a gallery, or any group of images input in real time. These images may come from diverse sources and have heterogeneous content.

[0031] In the embodiments of this disclosure, a tile refers to a local region unit obtained by segmenting a complete image. Each tile carries a portion of the visual information of the image, and different tiles represent different semantics. For example, a car image may be segmented into "headlight tiles," "tire tiles," "steering wheel tiles," etc., and each tile expresses a relatively independent semantic concept.

[0032] In embodiments of this disclosure, a block vector is a vector that characterizes the features of its corresponding tile. For example, a block vector can be a deep abstraction and numerical representation of the visual, texture, color, and other information of a tile.

[0033] In embodiments of this disclosure, at least one target image is determined from multiple images based on the question vector. All tiles in each of the multiple images are traversed, and the similarity between the block vector of each tile and the question vector is calculated. For any image, if one or more of its tiles satisfy a first target condition, the image is determined to be a target image. A target image refers to an image selected from multiple images that is relevant to the input question, and its determination criterion is that the image contains at least one target tile.

[0034] A target tile is a tile whose similarity to the question vector satisfies the first objective condition. The first objective condition can be a set similarity threshold; for example, a cosine similarity greater than 0.8 between a tile vector and the question vector is considered to satisfy the first objective condition. Alternatively, the first objective condition can be selected by choosing the top three tiles with the highest similarity ranking; if the similarity between a tile vector and the question vector is among the top three, then the first objective condition is satisfied.

[0035] In the embodiments of this disclosure, there is a difference in granularity between the question vector and the block vector, making it difficult to achieve accurate local intent capture through direct similarity comparison. The question vector can be output as a fixed-dimensional vector using a text encoder. The question vector encodes the overall intent of the question and the global semantic context, providing a macroscopic and comprehensive representation. The block vector is generated from individual image tiles after image segmentation. For example, the block vector can incorporate visual, textual, and positional features of the region. Each block vector only encodes isolated information of its corresponding local region, providing a microscopic and detailed representation. Therefore, a granular preprocessing step is required before performing similarity comparison between the question vector and the block vector.

[0036] In embodiments of this disclosure, block vectors satisfying a first target condition can be determined based on the granularity of the question vector, thereby identifying at least one target image from multiple images. Specifically, fine-grained question sub-vectors can be generated based on the question vector, ensuring that the granularity of the question sub-vectors and the block vectors are at the same semantic granularity level. Similarity calculation can be performed directly between the question sub-vectors and the block vectors.

[0037] In the embodiments of this disclosure, the image retrieval method can be applied to precise knowledge retrieval of structured images such as PPT documents. A PPT page is essentially a highly information-dense visual representation, containing mixed modal information such as text, graphics, charts, and layout. Each slide in a PPT can be considered an independent image. Each PPT page can be segmented into multiple semantic tiles, and a corresponding block vector can be generated for each tile. If the similarity between a tile and the question vector meets the first target condition, then the PPT page to which it belongs is determined as the target image.

[0038] In operation S130, the retrieval results for the input question are determined based on the target patches in each target image.

[0039] In embodiments of this disclosure, the method does not simply return images, but incorporates tile information. For example, images with the highest similarity target tiles can be prioritized for display, or the location of the target tiles can be directly marked on the returned images.

[0040] In embodiments of this disclosure, the form of the search results is determined based on the target image tiles. The search results can be a directly returned list of highly relevant target images, or enhanced results with target images marked. The search results are used to directly and clearly respond to the query intent corresponding to the user's input question.

[0041] By introducing a fine-grained tile semantic representation and matching mechanism, the embodiments of this disclosure effectively solve the problem of local semantic dilution caused by whole-image retrieval. The image is decomposed into multiple semantically independent tiles and block vectors are generated, refining the retrieval granularity from the image level to the region level. By calculating the similarity between the question vector and the block vector, highly relevant local regions are accurately located, ensuring that the returned target image contains semantically matching content. Finally, retrieval results are generated based on the target tiles, improving the accuracy of local queries, avoiding information overload caused by returning the entire image, and thus improving the accuracy of the image retrieval method of this embodiment.

[0042] In some embodiments of this disclosure, the method further includes: performing semantic segmentation on each of the multiple images to obtain at least one patch for each image; and generating a block vector corresponding to each patch based on the at least one patch for each image.

[0043] In embodiments of this disclosure, semantic segmentation is performed on each of a plurality of images to obtain at least one patch for each image. Semantic segmentation refers to the process of dividing a complete image into multiple regional units with independent and complete semantics using image processing and machine learning techniques. Pre-trained deep learning models can be used for semantic segmentation.

[0044] For example, optical character recognition models can be used to extract text boxes from each image. Graphical analysis models can be used to identify charts, tables, arrows, module borders, and other content within each image. Layout modeling models can be used to infer the logical structure of each image. Semantic segmentation of each image can be performed using these models to obtain at least one image patch for each image.

[0045] Figure 2 This is a schematic diagram of semantic segmentation of each image in a plurality of images according to embodiments of the present disclosure.

[0046] like Figure 2 As shown, Figure 2 This is a schematic diagram of semantic segmentation of any one of multiple images. After semantic segmentation of any image 200, patch 1, patch 2, patch 3, and patch 4 can be obtained.

[0047] In the embodiments of this disclosure, a tile is an image sub-region obtained after semantic segmentation, and is a basic unit carrying independent semantics. Each tile is a local region of the original image, physically composed of a set of continuous pixels, and semantically representing a relatively complete information subject. For example, for a PowerPoint slide, its tiles can be a "title text box", "a step graphic in a flowchart", "a data area in a table", etc.

[0048] In embodiments of this disclosure, a block vector corresponding to each image is generated based on at least one patch. A block vector is a vector characterizing the features of its corresponding patch. A block vector can be a point in a vector space that maps the visual, texture, and structural information of a patch to a point in a vector space using a feature extraction model. The block vector is a deep abstract representation of the patch's features.

[0049] Through the embodiments of this disclosure, images are deconstructed into semantically independent units, avoiding global semantic dilution and creating conditions for accurate local matching. At the same time, visual information is converted into numerical vectors, so that semantic similarity can be calculated mathematically, thereby improving the accuracy and efficiency of image retrieval.

[0050] In some embodiments of this disclosure, generating a block vector corresponding to each image patch based on at least one patch of each image includes: obtaining content information, location information, and relationship information of each patch, wherein the content information represents the content features of each patch, the location information represents the location features of each patch in its corresponding image, and the relationship information represents the relationship features between each patch and other patches in its corresponding image; and generating a block vector corresponding to each patch based on the content information, location information, and relationship information.

[0051] In the embodiments of this disclosure, content information represents the content features of each tile. Content information can be the visual and textual content features contained within the tile. Content information can include visual features, such as low-level visual attributes like color, texture, shape, and outline. Content information can include textual content, such as text information extracted within the tile using optical character recognition technology. Content information can also include semantic content, such as high-level semantic information like the theme or concept expressed by the tile.

[0052] Location information represents the positional characteristics of each tile within its corresponding image. Location information can include the spatial positional characteristics of the tile within its respective image. Location information can include absolute position, such as the tile's coordinates within the image. Location information can include relative position, such as the tile's positional relationship relative to the image center or other tiles. Location information can also include layout information, such as the tile's role in the overall layout (e.g., a centered title, sidebar, etc.).

[0053] Relationship information represents the relationship features between each tile and other tiles in its corresponding image. Relationship information can include the association features between a tile and other tiles in its image. It can include spatial relationships, such as adjacency, containment, and overlap. It can also include logical relationships, such as arrow pointing, referencing, and flow order. Finally, it can include visual relationships, such as color matching, size contrast, and alignment.

[0054] In the embodiments of this disclosure, content information, location information, and relationship information of each tile are obtained. For example, content information includes text information and image information. Text information (e.g., text content, contextual meaning) can be obtained through a bidirectional encoder representation transformation model or document layout model, and image information (e.g., background graphic features, style, color, icon type) can be obtained through a contrastive language-image pre-trained model or visual transformer. For example, location information (e.g., spatial coordinates, module alignment, hierarchical structure, etc.) can be obtained through positional encoding techniques, and relationship information (e.g., arrow / contrast / flow relationships with other tiles) can be obtained through graph convolutional networks or relationship graphs.

[0055] In the embodiments of this disclosure, block vectors corresponding to each tile are generated based on content information, location information, and relationship information. The content information, location information, and relationship information can be converted into vector representations respectively, and multimodal features can be integrated into a unified block vector representation using feature concatenation, weighted summation, or attention-based fusion methods to generate a fusion vector that comprehensively represents the semantics of the tiles.

[0056] Through the embodiments of this disclosure, multimodal feature fusion enables block vectors to simultaneously encode the complete information of a block, enriching the dimensions of semantic representation and thus significantly improving the accuracy of block semantic understanding.

[0057] In some embodiments of this disclosure, generating a block vector corresponding to each tile based on content information, location information, and relation information includes: using a deep learning model to calculate the attention weights of the content information, location information, and relation information respectively, and thereby generating a block vector corresponding to each tile.

[0058] In embodiments of this disclosure, a deep learning model refers to a deep neural network architecture capable of automatically learning data features. In embodiments, it can be a neural network model used for multimodal feature processing. For example, a deep learning model can be a Transformer model. Attention weights are numerical indicators representing the importance of different information dimensions, calculated through an attention mechanism.

[0059] In the embodiments of this disclosure, a deep learning model is used to calculate the attention weights of content information, location information, and relational information, thereby generating block vectors corresponding to each tile. Content information, location information, and relational information can be converted into feature vectors of a unified dimension through an embedding layer. A query-key-value attention mechanism is used to calculate the attention weights of each feature vector. The attention weights automatically learn the importance of different information dimensions; for example, for text-dense tiles, content information may receive higher weights, while for structure-dense tiles, location and relational information may be more important. The final block vector is generated by weighted summation. It should be noted that the sum of the attention weights corresponding to content information, location information, and relational information is 1.

[0060] Through the embodiments of this disclosure, attention weights can automatically adjust the importance of each information dimension according to the characteristics of the image patch, thereby improving the accuracy and adaptability of feature representation. The method of this embodiment can improve the discriminativeness and expressiveness of the block vector, and thus improve the accuracy and robustness of the entire image retrieval system.

[0061] In some embodiments of this disclosure, determining at least one target image from a plurality of images based on a question vector includes: determining at least one candidate image from the plurality of images based on the question vector and graph vectors corresponding to each image in the plurality of images, wherein the graph vectors characterize the features of their corresponding images, and the similarity between the graph vectors corresponding to the candidate images and the question vectors satisfies a second objective condition; determining at least one target patch based on the question vector and the block vectors of each patch in each candidate image; and determining at least one target image from at least one candidate image based on at least one target patch.

[0062] In the embodiments of this disclosure, the graph vector corresponding to each image refers to a vector representation that characterizes the features of the entire image. Graph vectors can encode the visual, semantic, and structural features of the entire image. The graph vectors corresponding to each image can be extracted using a deep convolutional neural network or a visual Transformer, and a single graph vector can represent the content of the entire image.

[0063] In embodiments of this disclosure, at least one candidate image is determined from a plurality of images based on the question vector and the graph vectors corresponding to each image. A candidate image refers to a subset of potentially related images initially selected based on the similarity between the graph vector and the question vector. The selection criteria are based on a second objective condition, which may include setting a similarity threshold, such as a cosine similarity greater than 0.6. For example, the second objective condition may include selecting the top 10% of images by similarity as candidate images. The second objective condition may also include a comprehensive selection combining the threshold and the ranking.

[0064] In embodiments of this disclosure, at least one target patch is determined based on the question vector and the block vectors of each patch in each candidate image. For each candidate image, the block vectors of all its patches are extracted, the similarity between the question vector and each block vector is calculated, and the target patch is determined based on a first target condition. For example, the patch with the highest similarity or all patches exceeding a threshold can be selected.

[0065] In embodiments of this disclosure, at least one target image is determined from at least one candidate image based on at least one target patch. The final target image is determined from the candidate images based on the presence of the target patch. For example, a candidate image containing at least one target patch can be considered the target image. For example, a preset quantity threshold N can be used, and only candidate images containing at least N target patches are determined as target images. For example, a weighted scoring method can be used to filter images based on a comprehensive score of the number and similarity of target patches.

[0066] Through the embodiments of this disclosure, coarse-grained screening using graph vectors significantly reduces computational complexity, enabling the system to rapidly process massive amounts of images. Fine-grained screening using block vectors on the candidate images ensures the accuracy and local relevance of the final results. The two-stage architecture of this embodiment effectively improves the efficiency and accuracy of large-scale image retrieval.

[0067] In some embodiments of this disclosure, the method further includes generating a graph vector corresponding to each image based on the content features of each image in the plurality of images.

[0068] In the embodiments of this disclosure, content features refer to various feature elements in an image that can characterize its visual and semantic information. Content features may include low-level visual features of the image, such as basic visual attributes like color distribution, texture, edge information, and shape contours. Content features may include high-level semantic features of the image, such as object category, scene type, and emotional atmosphere. Content features may include structural layout features of the image, such as the image's composition, spatial layout, and visual center of gravity.

[0069] In the embodiments of this disclosure, a graph vector refers to a fixed-dimensional numerical vector representation generated based on image content features. Graph vectors can comprehensively reflect the visual and semantic content of the entire image.

[0070] In the embodiments of this disclosure, a graph vector corresponding to each image is generated based on the content features of each image in a plurality of images. The content features of each image are dimensionally unified and numerically standardized, and weights are assigned to different features according to their importance. The high-dimensional features can be compressed into fixed-dimensional vectors by an autoencoder to obtain the graph vector.

[0071] Through the embodiments of this disclosure, by comprehensively extracting multi-dimensional content features, it is ensured that the graph vector can fully and accurately represent the semantics of the whole image, providing a reliable basis for similarity calculation. The method of this embodiment can improve the response speed and processing capability of large-scale image retrieval systems.

[0072] Figure 3 This is a flowchart of an image retrieval method according to another embodiment of the present disclosure.

[0073] like Figure 3 As shown, the retrieval results for the input question are determined based on the target patches in each target image, including operations S310-S340.

[0074] In operation S310, based on the similarity between the question vector and the block vector corresponding to the target patch, a first similarity score is generated for each target patch in the target image.

[0075] In the embodiments of this disclosure, the first similarity score is a score calculated based on the similarity between the question vector and the block vector corresponding to the target tile. The first similarity score can reflect the degree of semantic matching between the question vector and the block vector corresponding to the target tile.

[0076] In embodiments of this disclosure, a first similarity score is generated for each target patch in at least one target image based on the similarity between the question vector and the corresponding block vector of the target patch. A distance metric function can be used to calculate the similarity value between the question vector and the corresponding block vector of each target patch. The similarity values ​​are standardized to ensure a uniform scoring range; for example, the similarity values ​​can be normalized to the 0-1 range. The highest similarity value among all patches can be taken as the first score for the image.

[0077] In operation S320, a second similarity score is generated for each target image based on the similarity between the question vector and the graph vector corresponding to the target image, where the graph vector represents the features of its corresponding image.

[0078] In the embodiments of this disclosure, the second similarity score is a score calculated based on the similarity between the question vector and the graph vector corresponding to the target image. The second similarity score is used to evaluate the overall relevance between the question vector and the whole image, reflecting macroscopic semantic consistency.

[0079] In embodiments of this disclosure, the second similarity score can be generated using the same method as in operation S310, which will not be elaborated here.

[0080] In operation S330, at least one target image is sorted according to the first similarity score of the target patches in each target image and the second similarity score of each target image to obtain the target arrangement order.

[0081] In operation S340, at least one target image is arranged according to the target arrangement order to obtain the retrieval results.

[0082] In embodiments of this disclosure, the target arrangement order is the final ranking result obtained by fusing a first similarity score and a second similarity score. For example, the final score of the target image can be determined by weighting the first similarity score and the second similarity score, and the target arrangement order can be obtained based on the final score.

[0083] In the embodiments of this disclosure, the weight ratio can be dynamically adjusted according to the query type of the input question. When a specific query is involved, the first similarity score is given a higher weight; for example, the input question involves specific content such as a red car. When a generalized query is involved, the second similarity score is given a higher weight; for example, the input question involves generalized content such as transportation.

[0084] By considering both local feature matching degree and global semantic relevance in the embodiments of this disclosure, the limitations of single-dimensional ranking can be avoided, making the results more comprehensive and accurate. The method of this embodiment effectively improves the accuracy of image retrieval.

[0085] In some embodiments of this disclosure, sorting at least one target image according to a first similarity score of target patches in each target image and a second similarity score of each target image to obtain a target arrangement order includes: using a visual language model to generate a confidence score for each target image based on an input question and each target image, wherein the visual language model is trained based on sample images and sample text corresponding to the sample images; and sorting at least one target image according to the first similarity score of target patches in each target image, the second similarity score of each target image, and the confidence score to obtain a target arrangement order.

[0086] In the embodiments of this disclosure, a multimodal artificial intelligence model is used that can simultaneously understand and process visual and linguistic information using a visual language model. The visual language model can establish deep semantic relationships between image content and text descriptions. The visual language model is trained based on sample images and corresponding sample text; that is, it is pre-trained on massive amounts of image-text pairing data to learn rich visual-language correspondences.

[0087] For example, visual language models can include contrastive language-image pre-trained models, aligned and then fused models, visual-language transformer models, etc.

[0088] In the embodiments of this disclosure, a visual language model is used to generate a confidence score for each target image based on the input question and each target image. The confidence score is a probabilistic evaluation of the semantic matching degree between the target image and the input question by the visual language model. The evaluation of the confidence score can take into account the overall context of the image and the implicit meaning of the question. The confidence score can output a confidence value between 0 and 1, representing the degree of certainty of the match; a high score indicates that the visual language model is confident that the image and the question are semantically highly matched.

[0089] In embodiments of this disclosure, the visual language model is trained based on sample images and corresponding sample text. The sample images and corresponding sample text constitute training data pairs for training the visual language model, typically containing hundreds of millions of paired samples (images, text descriptions). The text descriptions in the training data pairs must accurately reflect the image content to ensure that the visual language can learn the correct correspondences.

[0090] In the embodiments of this disclosure, the first similarity score, the second similarity score, and the confidence score can all reflect the correlation between the input question and the target image. However, a single signal may be biased. By fusing the three, robustness can be improved.

[0091] In embodiments of this disclosure, at least one target image is sorted according to a first similarity score of target patches in each target image, a second similarity score of each target image, and a confidence score to obtain a target arrangement order. For example, the first similarity score, the second similarity score, and the confidence score can be weighted and summed to determine the final score.

[0092] The first similarity score can represent the best-case scenario for fine-grained matching, such as the maximum score across all blocks, or the overall situation across all blocks. The second similarity score can represent the confidence level of the initial screening. The confidence score provides a more global, semantically understanding-based judgment, preventing the neglect of the overall context due to excessive focus on local details.

[0093] In the embodiments of this disclosure, the weights of the first similarity score, the second similarity score, and the confidence score can be dynamically adjusted according to the query complexity. By calibrating the weights of each score, the reliability of the score can be ensured.

[0094] Through the embodiments of this disclosure, confidence scoring provides a more reliable relevance assessment and effectively filters out interference results that are superficially similar but semantically mismatched. The method of this embodiment enables the retrieval system to better understand the user's true intent and provide more accurate and reliable ranking results.

[0095] In some embodiments of this disclosure, arranging at least one target image according to a target arrangement order to obtain retrieval results includes: arranging at least one target image according to a target arrangement order, and adding corresponding target identifiers to target patches in each target image to obtain retrieval results, wherein the target identifier represents the position information of the target patch in its corresponding target image.

[0096] In embodiments of this disclosure, the target ranking order can be the final ranking sequence obtained after multi-dimensional score fusion calculation. The target ranking order ensures that the most relevant results are placed at the top, reducing user browsing costs. For example, the target ranking order can be determined based on a first similarity score and a second similarity score, or it can be determined based on a first similarity score, a second similarity score, and a confidence score.

[0097] In embodiments of this disclosure, target identifiers may include visual elements in the target image used to mark and highlight target tiles. For example, target identifiers may be bounding boxes, highlight marks, arrow indicators, color marks, number labels, and the coordinates of the target tiles in the corresponding image.

[0098] In embodiments of this disclosure, the target identifier may include a text label or target text paragraph corresponding to the target tile. Here, the target text paragraph may be used to describe an overall summary of the target tile's content, or it may be used to describe the association between the target tile and the input question.

[0099] In the embodiments of this disclosure, after completing the preliminary retrieval and matching work in the foregoing embodiments, retrieval results can be obtained. The retrieval results contain three key elements: first, several of the most relevant images (target images) selected; second, the location information of the specific regions (target tiles) in each image that are highly relevant to the user's question; and finally, a summary of the content of these relevant regions (corresponding text tags or brief explanations).

[0100] Further, in the user-facing presentation stage, the search results can be displayed on the screen. The first step is to summarize and semantically analyze the search results. The system will first inform the user of the macro-level results of this search, namely, "XX related knowledge items were found." This is a quantitative feedback that allows the user to immediately establish an expectation of the scale of the results. Based on a deep understanding of the content of all target images, a concise "overall content summary" is generated. This summary aims to extract the core themes shared by these images or their connections to the user's input question from different perspectives, providing the user with a global, semantic overview.

[0101] Furthermore, after completing the overall overview, each target image is introduced and highlighted individually. This is a cyclical process designed to provide a detailed interpretation of each target image. The system displays each selected target image sequentially. When displaying a single image, a brief introduction is first provided, highlighting the image's core theme or its specific relevance to the user's question. Then, using the coordinate information of the obtained target tiles, visualization operations are performed on the target image. For example, borders can be drawn or semi-transparent masks can be added to precisely highlight the areas (target tiles) deemed most relevant. This embodiment effectively guides the user's attention from the overall result to a single image, and then focuses on the most critical information points within the image, greatly improving the efficiency of information delivery and the intuitiveness of the user experience.

[0102] Through the embodiments of this disclosure, accurate location identification can directly guide users to focus on relevant areas, significantly reducing information location time and improving retrieval efficiency. Diverse visual identifiers and hierarchical designs can intuitively convey the strength of relevance and enhance the understandability of the results.

[0103] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure.

[0104] Another aspect of this disclosure provides an electronic device 400, comprising: an acquisition module 410 for acquiring an input question; a processor 420 communicatively connected to the acquisition module for generating a question vector based on the input question; determining at least one target image from a plurality of images based on the question vector, wherein each image in the plurality of images includes at least one patch, different patches of each image represent different semantics, the target image includes at least one target patch, the similarity between the block vector corresponding to the target patch and the question vector satisfies a first target condition, and the block vector represents the features of its corresponding patch; determining retrieval results for the input question based on the target patches in each target image; and a display screen 430 for displaying the retrieval results.

[0105] In this embodiment of the disclosure, the processor 420 can execute the processing method described above.

[0106] By introducing a fine-grained tile semantic representation and matching mechanism, the embodiments of this disclosure effectively solve the problem of local semantic dilution caused by whole-image retrieval. The image is decomposed into multiple semantically independent tiles and block vectors are generated, refining the retrieval granularity from the image level to the region level. By calculating the similarity between the question vector and the block vector, highly relevant local regions are accurately located, ensuring that the returned target image contains semantically matching content. Finally, retrieval results are generated based on the target tiles, improving the accuracy of local queries, avoiding information overload caused by returning the entire image, and thus improving the accuracy of the image retrieval method of this embodiment.

[0107] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0108] Figure 5 This is a schematic block diagram of an example electronic device used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0109] like Figure 5As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0110] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0111] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as image retrieval methods. For example, in some embodiments, the image retrieval method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the image retrieval method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the image retrieval method by any other suitable means (e.g., by means of firmware).

[0112] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: electronic devices for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0116] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0117] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0118] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image retrieval method, comprising: Generate a question vector based on the input question; Based on the question vector, at least one target image is determined from multiple images, wherein each of the multiple images includes at least one patch, and different patches of each image represent different semantics; the target image includes at least one target patch, and the similarity between the block vector corresponding to the target patch and the question vector satisfies a first objective condition; the block vector represents the features of its corresponding patch. The retrieval results for the input question are determined based on the target patches in each target image.

2. The method according to claim 1, further comprising: Semantic segmentation is performed on each of the plurality of images to obtain at least one patch of each image; Based on at least one patch of each image, generate a block vector corresponding to each patch.

3. The method according to claim 2, wherein generating a block vector corresponding to each image patch based on at least one image patch comprises: The content information, location information, and relationship information of each image block are obtained, wherein the content information represents the content features of each image block, the location information represents the location features of each image block in its corresponding image, and the relationship information represents the relationship features between each image block and other images in its corresponding image; Based on the content information, the location information, and the relationship information, a block vector corresponding to each tile is generated.

4. The method according to claim 3, wherein generating a block vector corresponding to each tile based on the content information, the location information, and the relationship information comprises: Using a deep learning model, the attention weights of the content information, the location information, and the relationship information are calculated, thereby generating block vectors corresponding to each tile.

5. The method according to claim 1, wherein determining at least one target image from a plurality of images based on the problem vector comprises: Based on the question vector and the graph vectors corresponding to each image in the plurality of images, at least one candidate image is determined from the plurality of images, wherein the graph vector represents the features of its corresponding image, and the similarity between the graph vector corresponding to the candidate image and the question vector satisfies the second objective condition; Based on the question vector and the block vector of each patch in each candidate image, at least one target patch is determined; The at least one target image is determined from the at least one candidate image based on the at least one target patch.

6. The method according to claim 5, further comprising: Based on the content features of each of the multiple images, a graph vector corresponding to each image is generated.

7. The method according to claim 1, wherein determining the retrieval result of the input question based on target patches in each target image includes: Based on the similarity between the question vector and the block vector corresponding to the target image, a first similarity score is generated for each target image in the at least one target image; Based on the similarity between the question vector and the graph vector corresponding to the target image, a second similarity score is generated for each target image, wherein the graph vector represents the features of its corresponding image; Based on the first similarity score of the target patches in each target image and the second similarity score of each target image, the at least one target image is sorted to obtain the target arrangement order; The at least one target image is arranged according to the target arrangement order to obtain the retrieval result.

8. The method according to claim 7, wherein sorting the at least one target image based on a first similarity score of the target patches in each target image and a second similarity score of each target image to obtain a target arrangement order comprises: Using a visual language model, a confidence score for each target image is generated based on the input question and each target image, wherein the visual language model is trained based on sample images and sample text corresponding to the sample images; Based on the first similarity score of the target patches in each target image, the second similarity score of each target image, and the confidence score, the at least one target image is sorted to obtain the target arrangement order.

9. The method according to claim 7, wherein arranging the at least one target image according to the target arrangement order to obtain the retrieval result includes: The at least one target image is arranged according to the target arrangement order, and a corresponding target identifier is added to the target patch in each target image to obtain the retrieval result, wherein the target identifier represents the position information of the target patch in its corresponding target image.

10. An electronic device, comprising: The acquisition module is used to acquire input questions; A processor, communicatively connected to the acquisition module, is used to generate a question vector based on the input question; Based on the question vector, at least one target image is determined from multiple images, wherein each of the multiple images includes at least one tile, and different tiles of each image represent different semantics; the target image includes at least one target tile, and the similarity between the block vector corresponding to the target tile and the question vector satisfies a first target condition; the block vector represents the features of its corresponding tile; and the retrieval result of the input question is determined based on the target tiles in each target image. A display screen is used to display the search results.

Images