Multi-mode image searching method based on images

Abstract

The invention discloses a multi-modal image searching method, and solves the problems that in the prior art, effective utilization of document image text information is lacked, and the multi-modal retrieval requirement in a complex scene is difficult to meet. According to the invention, the input image is classified by using the deep learning model, and the image classification result is obtained; the image classification result is a text image or a non-text image; generating a multi-dimensional composite feature vector for the text image; for a non-text image, generating a visual feature vector; retrieving a text image based on the multi-dimensional composite feature vector, calculating a comprehensive score by fusing the text semantic similarity, the abstract semantic similarity and the format visual similarity, and returning a retrieval result according to the comprehensive score; and for a non-text image, performing retrieval based on the cosine similarity of the visual feature vector, and returning a retrieval result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of picture search, in particular to a picture search method. BACKGROUND

[0002] Current picture search technology mainly targets common person and object images, and performs similarity matching through convolutional neural network (CNN) to extract visual features (such as color, texture, shape, etc.). However, for document images (such as bills, certificates, book pages, etc.), the core information is often contained in the text content. The traditional method only relies on visual features, which will lead to the loss of semantic information and insufficient search accuracy. The existing technology lacks effective utilization of text information of document images, and it is difficult to meet the multi-modal search requirements in complex scenarios. SUMMARY

[0003] The present application provides a multi-modal picture search method, aiming to solve the problem that the existing technology lacks effective utilization of text information of document images, and it is difficult to meet the multi-modal search requirements in complex scenarios.

[0004] In a first aspect, a multi-modal picture search method is provided, comprising:

[0005] An image classification step: using a deep learning model to classify the input image to obtain an image classification result; the image classification result is a text image or a non-text image;

[0006] A double-branch feature extraction step: for a text image, performing OCR recognition, abstract generation, and layout visual feature extraction, and performing weighted fusion to generate a multi-dimensional composite feature vector; for a non-text image, performing visual feature extraction to generate a visual feature vector;

[0007] A cross-modal search step, comprising:

[0008] a) Search execution: for a text image, performing search based on the multi-dimensional composite feature vector, and fusing text semantic similarity, abstract semantic similarity, and layout visual similarity to calculate a comprehensive score, and returning the search result according to the comprehensive score; for a non-text image, performing search based on the cosine similarity of the visual feature vector, and returning the search result;

[0009] b) Cross-modal association: based on the search execution step, through a unified vector index framework, cross-modal association search of non-text images and text images is realized, and a cross-modal association search result is returned.

[0010] Optionally, in the image classification step, the deep learning model is improved based on ResNet-50, a global average pooling layer is added after the last convolutional block, a 2-dimensional classification vector is output, and a classification probability is calculated through a Softmax activation function.

[0011] Optionally, before the cross-modal retrieval step, the method further comprises: using a high-performance vector retrieval library, and respectively establishing independent index structures for different distribution characteristics of text image text features being sparse and non-text image visual features being dense.

[0012] Optionally, for the text image, the OCR recognition, the abstract generation, the layout visual feature extraction are performed, and weighted fusion is performed to generate a multi-dimensional composite feature vector, specifically comprising:

[0013] The text image is processed by using an OCR recognition engine to generate structured text, and the structured text is a JSON format file containing paragraph, line, and character three-level coordinate information;

[0014] The core semantic information of the structured text is extracted by using the adjusted BERT model to generate an abstract;

[0015] The structured text and the abstract are converted into fixed-length text feature vectors by using a pre-trained text embedding model;

[0016] The layout features of the text image are extracted to generate a layout feature vector;

[0017] The text feature vector and the layout feature vector are weighted and fused to generate a multi-dimensional composite feature vector.

[0018] Optionally, for the non-text image, the visual feature extraction is performed to generate a visual feature vector, specifically comprising:

[0019] For the non-text image, the visual feature extraction is performed by using a visual feature network to generate a visual feature vector;

[0020] In the preprocessing stage, the non-text image is subjected to random flipping, brightness adjustment, and Gaussian blur operation to improve the generalization ability of the model.

[0021] Optionally, before the cross-modal retrieval step, the method further comprises:

[0022] A relational database and a vector database are established to store basic information and feature vectors of images respectively; the relational database is used to store basic information of images, including image metadata and abstract; the image metadata includes ID, type, size, upload time and structured text; the vector database is used to store feature vectors of images and to establish indexes for text image features and non-text image features respectively: the text feature index is suitable for high-dimensional sparse features; and the visual feature index is suitable for high-dimensional dense features.

[0023] In the above scheme, optionally, in the cross-modal retrieval step, the comprehensive score of the text image is composed of the following parts:

[0024] Text semantic similarity: the cosine similarity of the text feature vector of the retrieval image and the text feature vector of the image in the database is calculated;

[0025] Abstract semantic similarity: the abstracts of the retrieval image and the database image are encoded into vectors using the Sentence-BERT model, and the cosine similarity of the two is calculated;

[0026] Layout visual similarity: the Euclidean distance of the layout feature vectors of the retrieval image and the database text image is calculated;

[0027] The text semantic similarity is regarded as the first weight, the abstract semantic similarity is regarded as the second weight, and the layout visual similarity is regarded as the third weight. For text images, the text semantic similarity, the abstract semantic similarity and the layout visual similarity are fused, and the comprehensive score of each retrieval result is calculated combined with the respective weights. The retrieval results are ranked in descending order of the comprehensive score.

[0028] In the above scheme, optionally, in the cross-modal retrieval step, for non-text images, the cosine similarity of the query visual feature vector and the database visual feature vector is directly calculated, the K-nearest neighbor algorithm is used for retrieval, the retrieval results are ranked in descending order of the cosine similarity of the visual feature vector, and the top n images are returned, n being a preset number.

[0029] In the above scheme, optionally, after the cross-modal retrieval step, the fusion and display of the retrieval results are further included:

[0030] For text image retrieval results, the ranking is in descending order of the comprehensive score; for non-text image retrieval results, the ranking is in descending order of the cosine similarity of the visual feature vector;

[0031] For text images, the overlapping keywords of the retrieval abstract and the hit abstract are highlighted, and the document layout matching area is labeled; for non-text images, the matching key visual area is highlighted by superimposing a heat map.

[0032] Optionally, in the cross-modal retrieval step, the cross-modal association step specifically comprises: realizing cross-modal association retrieval of text images and non-text images by designing a unified retrieval interface.

[0033] Compared with the prior art, the present application has at least the following beneficial effects:

[0034] Based on further analysis and research of the problems of the prior art, the present application realizes that the prior art lacks effective utilization of document image text information, and it is difficult to meet the multi-modal retrieval demand in complex scenarios. The method distinguishes text images and non-text images through image classification, and extracts multi-dimensional composite feature vectors and visual feature vectors respectively. In retrieval, the text image fusion text semantic, abstract semantic and layout visual similarity are calculated to obtain a comprehensive score, and the non-text image is retrieved based on the cosine similarity of the visual feature vector. At the same time, the cross-modal association retrieval is realized through a unified vector index framework, which significantly improves the retrieval accuracy and efficiency, and is suitable for intelligent document management, legal document comparison, e-commerce search and other complex scenarios, and meets the actual demand of multi-modal retrieval.

[0035] The method distinguishes text images and non-text images using a deep learning model through an image classification step, and constructs independent processing branches respectively, realizing differentiated processing of different types of images. For text images, the present application not only performs OCR recognition to extract structured text, but also further generates an abstract and extracts layout visual features, and finally generates a multi-dimensional composite feature vector through weighted fusion, thereby fully utilizing the "content" and "form" information of the text image. For non-text images, a visual feature vector is generated through efficient visual feature extraction, retaining its visual characteristics.

[0036] In the cross-modal retrieval step, the present application uses a high-performance vector retrieval library, and establishes independent index structures for different distribution characteristics of text image and non-text image features, significantly improving the retrieval efficiency. For text images, the text semantic similarity, abstract semantic similarity and layout visual similarity are fused to calculate a comprehensive score and return the retrieval result, thereby realizing accurate retrieval of text images. For non-text images, retrieval is performed based on the cosine similarity of the visual feature vector, ensuring the efficiency and accuracy of the retrieval result. In addition, the present application supports cross-modal association retrieval of non-text images and text images through a unified vector index framework, breaking down the barriers between modalities and realizing more extensive retrieval capabilities.

[0037] Through the above design, the retrieval accuracy of the document image is significantly improved, and the retrieval efficiency is optimized, so that it can meet the multi-modal retrieval demand in complex scenes. The method is suitable for various application scenarios such as intelligent document management, legal document comparison, e-commerce multi-modal search, and provides an efficient and accurate image retrieval solution for users. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 A flowchart of a multi-modal image search method provided for the first embodiment of the application is shown in the figure.

[0039] Figure 2 A schematic diagram of a double-branch multi-modal fusion architecture provided for an embodiment of the application is shown in the figure.

[0040] Figure 3 A flowchart of a multi-modal image search method provided for the second embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0042] In the description of the application: unless otherwise specified, the terms "first", "second", "third" and the like in the application are intended to distinguish the objects referred to, and do not have special meanings in the technical connotation aspect (for example, should not be understood as emphasizing importance or order, etc.). "Include", "contain", "have" and the like, also mean "not limited to" (some units, components, materials, steps, etc.).

[0043] (I) Current situation of traditional article image retrieval technology

[0044] The existing mainstream method is based on visual feature extraction and metric learning, which specifically includes:

[0045] 1. Shallow feature stage: Early use of hand-designed features combined with bag-of-words model to construct image feature vectors, and calculate similarity through Euclidean distance or cosine distance. However, hand-designed features are not robust to complex lighting and viewing angle changes, and cannot capture high-level semantic information.

[0046] 2. Deep feature stage: With the development of deep learning, end-to-end methods based on CNN have become mainstream, such as AlexNet and VGGNet, which extract hierarchical visual features through multiple layers of convolution and achieve excellent classification performance on datasets such as ImageNet. Further optimization of the feature space through metric learning methods such as triplet loss makes images of the same class closer in Euclidean space. Mainly for non-text images such as commodities and natural scenery, relying on color, texture, shape, and other visual features to achieve retrieval.

[0047] (B) The particularity of document image retrieval technology and existing solutions.

[0048] For document image retrieval (such as bills, contracts, and ancient book pages), there are two independent technical routes in existing technology, but neither has effectively integrated with traditional image search technology:

[0049] 1. Text retrieval based on OCR: First, convert the document image into text through OCR technology, and then use keyword matching, TF-IDF, and other text retrieval techniques to achieve query. This method only uses text character information and completely ignores the layout of the document (such as table structure, paragraph layout, seal position, and other visual features), and the OCR accuracy of low-quality documents (such as handwritten and blurred images) is significantly reduced, leading to retrieval failure.

[0050] 2. Document image retrieval based on vision: A few studies attempt to follow the methods of item image retrieval by extracting visual features (such as edges, grid lines, and character distribution patterns) from document images using CNN, but fail to exploit the semantic information of the text content.

[0051] (Three) Bottlenecks of cross-modal retrieval technology.

[0052] In existing technology, retrieval systems for mixed modalities (text images + non-text images) have the following core defects:

[0053] 1. Modality processing fragmentation: Most systems treat text images and non-text images as independent tasks and do not establish a unified multi-modal feature representation space. For example, the image search function of an e-commerce platform only processes product pictures, while document images need to be queried through a separate text search portal, resulting in fragmented user experience.

[0054] 2. Insufficient semantic fusion:

[0055] For text images, traditional image search technology only uses visual features, extracting edge and contour features of document images through CNN, completely ignoring the semantic information of OCR text, resulting in a "semantic gap" problem - visually similar documents may have different content (such as similar format reports of different years), and content-related documents may have significant differences in visual features due to font and layout differences.

[0056] For non-textual images, the prior art lacks the ability to search for semantic associations with document images, such as retrieving a physical picture of a product from an image containing a screenshot of the product's manual.

[0057] Indexing efficiency problem: When dealing with a million-level image library, the traditional linear scanning matching method (such as brute force cosine distance calculation) has an exponential increase in retrieval time, while the existing vector indexing technology (such as KD tree) has a significant decrease in retrieval accuracy in high-dimensional feature space (such as 2048-dimensional visual features).

[0058] (Four) Essential defects of the prior art.

[0059] In summary, the core deficiency of the prior art is that a unified retrieval framework that integrates visual features and text semantics has not been established, especially lacking the collaborative use of "text content-visual layout" dual-modal information in document images. This leads to the one-sided problem of traditional image search technology in document image retrieval scenarios, which can only match the format or only match the characters, and cannot meet the demand for comprehensive matching of multi-modal information in financial bill review, legal document comparison, and education resource retrieval.

[0060] In addition, the existing image search technology has the following three major essential defects:

[0061] (1) Modality processing is fragmented, lacking a unified retrieval framework.

[0062] Traditional technology treats text images (such as documents) and non-text images (such as items) as independent retrieval tasks, without establishing a unified processing flow that integrates the features of both types of images. For example, document images rely on pure text retrieval portals, while item images rely on pure visual retrieval portals, and users need to switch retrieval methods according to the type of image, leading to fragmented experience in multi-modal retrieval scenarios.

[0063] (2) Semantic and visual information are separated, and the retrieval accuracy is insufficient.

[0064] 1. Text image retrieval is one-sided:

[0065] Text retrieval based on OCR only uses character information, losing visual features such as layout, seal position, etc.; document retrieval based on vision only extracts low-level features such as edges and contours, ignoring semantic information of text content (such as contract terms, bill amount, etc.), leading to "similar format but unrelated content" or "related content but large format difference" false detection problems.

[0066] 2. Lack of cross-modal association:

[0067] The semantic association retrieval between text images and non-text images (such as retrieving a corresponding real object image through a product manual document image) cannot be achieved, which limits the application scenarios of the technology.

[0068] (3) Low efficiency of large-scale data retrieval.

[0069] The existing vector index technology (such as KD tree) has a "dimension disaster" problem when processing high-dimensional visual features (such as 2048-dimensional CNN features) or text semantic features, and the retrieval time increases exponentially with the growth of data size, which is difficult to meet the real-time retrieval demand of a million-level image library.

[0070] The present application is aimed at the above technical blank, and proposes a retrieval method capable of simultaneously processing two types of images and fusing multi-modal features, which effectively solves the problem of insufficient retrieval accuracy of existing technology in document image retrieval through the construction of a double-branch feature extraction architecture and a cross-modal weight fusion strategy.

[0071] In one embodiment, with reference to Figure 1 , Figure 2 and Figure 3 , a multi-modal image search method is provided, comprising:

[0072] Image classification step: using a deep learning model to classify the input image to obtain an image classification result; the image classification result is a text image or a non-text image;

[0073] Double-branch feature extraction step: for a text image, performing OCR recognition, abstract generation, and layout visual feature extraction, and performing weighted fusion to generate a multi-dimensional composite feature vector; for a non-text image, performing visual feature extraction to generate a visual feature vector;

[0074] Cross-modal retrieval step: comprising:

[0075] a) Retrieval execution: for a text image, performing retrieval based on the multi-dimensional composite feature vector, and fusing text semantic similarity, abstract semantic similarity, and layout visual similarity to calculate a comprehensive score, and returning a retrieval result according to the comprehensive score; for a non-text image, performing retrieval based on the cosine similarity of the visual feature vector, and returning a retrieval result;

[0076] b) Cross-modal association: based on the retrieval execution step, realizing cross-modal association retrieval between non-text images and text images through a unified vector index framework, and returning a cross-modal association retrieval result.

[0077] In view of the above defects, the core purpose of the present application is to provide a unified retrieval method fusing multi-modal features, specifically comprising:

[0078] (1) Constructing a double-modal processing architecture.

[0079] The text and non-text images are distinguished by the image classification module, and independent feature extraction branches (OCR semantic branch of text image + visual feature branch of non-text image) are respectively designed to realize the differential processing and unified retrieval of the two types of images and solve the modality split problem.

[0080] (2) Fusion of text semantics and visual features.

[0081] 1. For text images, the structured text, semantic summary and layout visual features (extracted by a low-level visual model) recognized by OCR are first weighted and fused to form a feature vector containing “content-form” dual-modal information, which makes up for the defects of traditional methods relying on a single modality.

[0082] 2. For non-text images, the efficient visual feature extraction process is retained, and at the same time, the unified vector index framework is supported to support cross-modal associated retrieval with text images (such as retrieving corresponding item images through document image retrieval).

[0083] (3) Optimization of large-scale data retrieval efficiency.

[0084] A high-performance vector retrieval library (such as FAISS) is used to construct a layered scalable index, and independent index parameters are designed for the different distribution characteristics of text features and visual features (text features are sparse, and visual features are dense) to realize second-level retrieval response for millions of images and solve the efficiency bottleneck of traditional index technology.

[0085] Through the above design, the application first realizes the unified retrieval of text images and non-text images, while retaining the advantages of traditional item image retrieval, significantly improving the retrieval accuracy and cross-modal retrieval capability of document images, and is suitable for complex scenarios such as intelligent document management, legal document comparison, e-commerce multi-modal search, etc.

[0086] The application adopts a dual-branch multi-modal fusion architecture (as shown in Figure 2 ), which includes two core processes of image data preprocessing and image retrieval, and dynamically routes to the text image processing branch or the non-text image processing branch through the classification module to realize differential feature extraction and cross-modal retrieval matching.

[0087] In one embodiment, in the image classification step, the deep learning model is improved based on ResNet-50, a global average pooling layer is added after the last convolutional block, a 2-dimensional classification vector is output, and the classification probability is calculated through the Softmax activation function.

[0088] In this embodiment, by adding a global average pooling layer on the basis of ResNet-50 and using a Softmax activation function to calculate the classification probability, the model can efficiently and accurately classify the input image. This improvement not only improves the robustness of the model, but also provides a solid foundation for subsequent multi-modal image processing and retrieval.

[0089] In one embodiment, before the cross-modal retrieval step, it also includes: using a high-performance vector retrieval library, for the different distribution characteristics of text image text features sparse and non-text image visual features dense, respectively establishing independent index structures.

[0090] Using a high-performance vector retrieval library (such as FAISS) to construct a hierarchical scalable index, and designing independent index parameters for the different distribution characteristics of text features and visual features (text features sparse, visual features dense), a million-level image can be retrieved in seconds, solving the efficiency bottleneck of traditional index technology.

[0091] In one embodiment, for text images, OCR recognition, abstract generation, and layout visual feature extraction are performed, and a weighted fusion is performed to generate a multi-dimensional composite feature vector, specifically including:

[0092] OCR recognition: using an OCR recognition engine to process the text image to generate structured text, the structured text being a JSON format file containing paragraph, line, and character three-level coordinate information;

[0093] Abstract generation: using an adjusted BERT model to extract core semantic information of the structured text to generate an abstract;

[0094] Text vectorization: using a pre-trained text embedding model to convert the structured text and the abstract into a fixed-length text feature vector;

[0095] Layout visual feature extraction: extracting layout features of the text image to generate a layout feature vector;

[0096] Feature fusion: performing weighted fusion of the text feature vector and the layout feature vector to generate a multi-dimensional composite feature vector.

[0097] In one embodiment, for non-text images, visual feature extraction is performed to generate a visual feature vector, specifically including:

[0098] For non-text images, visual feature extraction is performed through a visual feature network to generate a visual feature vector;

[0099] In the preprocessing stage, random flipping, brightness adjustment, and Gaussian blur operations are performed on the non-text images to improve the generalization ability of the model.

[0100] In this embodiment, through a detailed double-branch feature extraction process, rich feature information is extracted for text images and non-text images respectively. Specifically: text images: through OCR recognition, summary generation, text vectorization, and layout feature extraction, a multi-dimensional composite feature vector containing "content" and "form" is generated. Non-text images: visual features are extracted through a visual feature network, and the generalization ability of the model is improved through data augmentation. This feature extraction method can provide high-quality feature representation for subsequent cross-modal retrieval, significantly improving retrieval accuracy and efficiency.

[0101] In one embodiment, before the cross-modal retrieval step, it further includes:

[0102] A relational database and a vector database are established to store the basic information and feature vectors of images respectively; wherein the relational database is used to store the basic information of images, including image metadata, abstract; the image metadata includes ID, type, size, upload time, structured text; the vector database is used to store the feature vectors of images, and indexes are established for text image features and non-text image features respectively: text feature index is suitable for high-dimensional sparse features; visual feature index is suitable for high-dimensional dense features.

[0103] By establishing a relational database and a vector database, this embodiment provides efficient data management and index support for cross-modal retrieval. The relational database stores the basic information of images, which is convenient for management and query; the vector database stores the feature vectors and establishes special indexes for different types of feature vectors, which significantly improves the retrieval efficiency. This design not only improves the performance of the system, but also provides strong support for cross-modal retrieval.

[0104] In one embodiment, in the cross-modal retrieval step, the comprehensive score of the text image is composed of the following parts:

[0105] Text semantic similarity: calculate the cosine similarity of the text feature vector of the retrieval image and the text feature vector of the image in the database;

[0106] Abstract semantic similarity: use the Sentence-BERT model to encode the abstract of the retrieval image and the abstract of the database image into a vector, and calculate the cosine similarity between the two;

[0107] Layout visual similarity: calculate the Euclidean distance between the layout feature vectors of the retrieval image and the database text image;

[0108] The text semantic similarity is regarded as a first weight, the abstract semantic similarity is regarded as a second weight, and the layout visual similarity is regarded as a third weight. For a text image, the text semantic similarity, the abstract semantic similarity and the layout visual similarity are fused, and a comprehensive score of each search result is calculated by combining the respective weights. The search results are ranked in descending order of the comprehensive score.

[0109] In one embodiment, in the cross-modal retrieval step, for a non-text image, a cosine similarity of a query visual feature vector and a database visual feature vector is directly calculated, a K nearest neighbor algorithm is used for retrieval, and search results are ranked in descending order of the cosine similarity of the visual feature vector. The top n images in the similarity are returned, and n is a preset number.

[0110] In this embodiment, for a text image, a comprehensive score is calculated by fusing the text semantic similarity, the abstract semantic similarity and the layout visual similarity and combining the weights, so that a more comprehensive similarity evaluation is realized. For a non-text image, a K nearest neighbor search is performed directly using the cosine similarity of the visual feature vector, so that an efficient similar image search is realized.

[0111] In one embodiment, after the cross-modal retrieval step, a fusion and display of the search results are further included.

[0112] For a text image search result, the search results are ranked in descending order of the comprehensive score. For a non-text image search result, the search results are ranked in descending order of the cosine similarity of the visual feature vector.

[0113] For a text image, overlapping keywords of a search abstract and a hit abstract are highlighted, and a document layout matching area is labeled. For a non-text image, a key visual matching area is highlighted by superimposing a heat map.

[0114] In this embodiment, the fusion and display step of the search results significantly improves the readability of the search results and the user experience by using a sorting strategy and visual enhancement.

[0115] The sorting strategy: for a text image, the text content, the semantic and the layout features are fused by sorting according to the comprehensive score. For a non-text image, the visual similarity is directly reflected by sorting according to the cosine similarity of the visual feature vector.

[0116] The visual enhancement: for a text image, the matching points are quickly identified by highlighting the overlapping keywords and labeling the layout matching area. For a non-text image, the key visual matching area is directly displayed by superimposing a heat map.

[0117] In one embodiment, for cross-modal associated search results, a cross-modal associated search module can be added. When the user clicks on this module, it will jump to the corresponding detailed search page. For example, for the display of text-image as the main query, the main result area: display the search results of text-image according to the original logic (highlight the keywords and mark the matching area). Cross-modal associated search area: add a "cross-modal associated search" module to display the associated non-text images (such as real object images and scene images) in the form of thumbnail grids, and mark the association strength (such as similarity score). The user can click on the cross-modal results to jump to the detailed search page of the non-text image.

[0118] For the display of non-text image as the main query, the main result area: display the search results of non-text image according to the original logic (heat map to mark the key visual area). Cross-modal associated search area: add a "cross-modal associated search" module to display the associated text images (such as specifications and contracts) in the form of document summary cards, and highlight the text segments related to the semantic of the query image. The user can click on the summary card to expand the complete document image and OCR text.

[0119] In one embodiment, for text-image search results / non-text image search results, and cross-modal search results, cross-modal weights are assigned, and the results are reordered and displayed according to the weights. For example, text-image is sorted according to the comprehensive score, and the search results are returned. At the same time, cross-modal associated search is performed, and the search results of non-text images associated with the cross-modal are returned. Cross-modal weights are assigned to the text-image (after sorting) and the non-text images associated with the cross-modal, and the search results are displayed again according to the weights. Similarly, for non-text images, cross-modal weights are assigned to the non-text images (sorted according to the cosine similarity of the visual feature vector) and the text images associated with the cross-modal, and the search results are displayed again according to the weights.

[0120] In one embodiment, in the cross-modal search step, the cross-modal association step specifically includes: through the design of a unified search interface, the cross-modal association search between text-image and non-text image is realized.

[0121] In this embodiment, cross-modal associated search refers to the ability to search between different modalities (such as text-image and non-text image) and find results similar to the query image in terms of semantics or vision. This search method can break through the limitations of a single modality and provide more comprehensive search results.

[0122] The unified search interface is a key component of the cross-modal search system, and its main functions include: (1) Support multi-modal input: allow users to input text-image or non-text image as a query. (2) Unified search logic: integrate the search logic of different modalities into one interface for seamless switching. (3) Cross-modal matching: can retrieve relevant results from text-image and non-text image according to the features of the query image.

[0123] The scheme of the present application is described below from the image search model training process and the image search process respectively.

[0124] Image search model training process:

[0125] 1. Input image standardization.

[0126] Implementation details: Support common formats such as JPEG, PNG, BMP, etc., and realize format conversion through OpenCV library. Adaptive size normalization strategy is adopted.

[0127] 2. Image classification module.

[0128] (1) Model architecture: based on ResNet-50 improvement, add global average pooling layer after the last convolution block, output 2-dimensional classification vector (text image / non-text image), Softmax activation function calculates classification probability.

[0129] (2) Training data: mixed dataset contains 150,000 labeled images:

[0130] ① Text image subset: 80,000 (including bills, contracts, book pages, from self-collected dataset + public dataset);

[0131] ② Non-text image subset: 70,000 (including commodities, natural scenes, faces, from ImageNet subset + desensitization data).

[0132] (3) Classification threshold: when the text image classification probability is greater than or equal to 0.7, it is determined as a text image, otherwise it is a non-text image (the optimal threshold is determined by adjusting the validation set).

[0133] 3. Text image feature extraction branch.

[0134] (1) OCR recognition sub-module: use self-developed OCR recognition engine to generate structured text (JSON format storage, including paragraph, line, character three-level coordinate information).

[0135] (2) Abstract generation sub-module: based on BERT model fine-tuning.

[0136] (3) Text vectorization: concatenate with text semantic vector to form 700-dimensional text image feature vector.

[0137] 4. Non-text image feature extraction branch.

[0138] (1) Visual feature network: extract 2048-dimensional feature vector output by Conv5_3 layer.

[0139] (2) Data augmentation: Randomly flip, adjust brightness (±20%), and apply Gaussian blur (σ=1.5) to non-text images in the preprocessing stage to improve the model's generalization ability.

[0140] 5. Feature storage and indexing.

[0141] Database design: Relational database stores image metadata (ID, type, size, upload time, OCR text (if text image), and summary (if text image)); vector database stores feature vectors, and text image features and non-text image features are indexed separately:

[0142] ① Text feature index: suitable for high-dimensional sparse features;

[0143] ② Visual feature index: suitable for high-dimensional dense features.

[0144] Image search process:

[0145] 1. Query image preprocessing.

[0146] Consistent with the data preprocessing process, generate the multi-modal feature vector (text feature / visual feature) and summary (if text image) of the query image.

[0147] 2. Cross-modal retrieval matching.

[0148] Text image query scenario: The retrieval score consists of three parts: ① Text semantic similarity (W1=0.6): Calculate the cosine similarity between the query text feature vector and the database text feature vector; ② Summary semantic similarity (W2=0.3): Use the Sentence-BERT model to encode the query summary and database summary into 384-dimensional vectors, and calculate the cosine similarity; ③ Layout visual similarity (W3=0.1): Calculate the Euclidean distance between the layout feature vectors (200-dimensional) of the query image and the database text image.

[0149] Non-text image query scenario: Directly calculate the cosine similarity between the query visual feature vector and the database visual feature vector, use K-nearest neighbor algorithm (K=20) for retrieval, and return the top 20 similar images.

[0150] 3. Result fusion and display.

[0151] Sorting strategy: Text image retrieval results are ranked in descending order of comprehensive score; non-text image retrieval results are ranked in descending order of visual similarity.

[0152] Visualization enhancement:

[0153] Text image: Highlight the overlapping keywords between the query summary and the hit summary, and mark the document layout matching area;

[0154] Non-text image: superimposed heat map, visualizing the key visual areas of the retrieved matches.

[0155] The key of the present application is:

[0156] 1. Dual-modal classification and differential processing architecture: dynamically distinguish text images and non-text images through image classification module, and construct independent feature extraction branches:

[0157] Text image branch: fusion of OCR text recognition (structured text), semantic summary generation (BERT model) and layout visual features (VGG extracts low-level features), to solve the defects of traditional methods relying on single modal;

[0158] Non-text image branch: use efficient ResNet visual feature extraction to retain the advantages of item image retrieval.

[0159] 2. Multi-dimensional feature fusion retrieval strategy: design a three-dimensional fusion retrieval model of "text semantics + summary semantics + layout visual" for text images:

[0160] Text semantics (OCR text vectorization) captures accurate character information;

[0161] Summary semantics (BERT generation) extracts core document semantics;

[0162] Layout visual (VGG features) retains document layout, typesetting and other form features.

[0163] 3. Efficient vector indexing and cross-modal retrieval mechanism:

[0164] According to the characteristics of text features (sparse) and visual features (dense), indexes are constructed respectively to support second-level retrieval of millions of images;

[0165] Design a unified retrieval interface to support cross-modal associated retrieval of text images and non-text images (such as retrieving corresponding items through document images).

[0166] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of technical features in the above embodiments are described, however, as long as the combinations of technical features do not exist contradictory, they should be considered as the scope of the present application.

Claims

1. A multimodal image search method, characterized in that: include: Image classification step: using a deep learning model to classify the input image to obtain an image classification result; the image classification result is a text image or a non-text image; Dual-branch feature extraction steps: For text images, perform OCR recognition, summary generation, layout visual feature extraction, and weighted fusion to generate a multi-dimensional composite feature vector; for non-text images, perform visual feature extraction to generate a visual feature vector; Cross-modal retrieval steps include: a) Retrieval execution: For text images, retrieval is performed based on the multi-dimensional composite feature vector, and a comprehensive score is calculated by integrating text semantic similarity, abstract semantic similarity, and layout visual similarity, and retrieval results are returned based on the comprehensive score; for non-text images, retrieval is performed based on the cosine similarity of the visual feature vector, and retrieval results are returned; b) Cross-modal association: Based on the above retrieval execution steps, a unified vector indexing framework is used to implement cross-modal association retrieval of non-text images and text images, and the cross-modal association retrieval results are returned.

2. The multimodal image search method according to claim 1, characterized in that: In the image classification step, the deep learning model is improved based on ResNet-50, a global average pooling layer is added after the last convolution block, a 2-dimensional classification vector is output, and the classification probability is calculated through the Softmax activation function.

3. The multimodal image search method according to claim 1, characterized in that: Before the cross-modal retrieval step, the method further includes: using a high-performance vector retrieval library to establish independent index structures for different distribution characteristics of sparse text features of text images and dense visual features of non-text images.

4. The multimodal image search method according to claim 1, characterized in that: For the text image, performing OCR recognition, summary generation, layout visual feature extraction, and weighted fusion to generate a multi-dimensional composite feature vector specifically includes: The text image is processed using an OCR recognition engine to generate structured text, which is a JSON format file containing three-level coordinate information of paragraphs, lines, and characters; Using the adjusted BERT model, the core semantic information of the structured text is extracted to generate a summary; Using a pre-trained text embedding model, convert the structured text and summary into a fixed-length text feature vector; Extract the layout features of the text image and generate a layout feature vector; The text feature vector and the layout feature vector are weightedly fused to generate a multi-dimensional composite feature vector.

5. The multimodal image search method according to claim 1, characterized in that: The method of performing visual feature extraction on the non-text image to generate a visual feature vector specifically includes: For non-text images, visual feature extraction is performed through the visual feature network to generate a visual feature vector; In the preprocessing stage, non-text images are randomly flipped, brightness adjusted, and Gaussian blurred to improve the generalization ability of the model.

6. The multimodal image search method according to claim 1, characterized in that: Before the cross-modal retrieval step, the method further includes: A relational database and a vector database are established to store basic image information and feature vectors, respectively. The relational database is used to store basic image information, including image metadata and summaries. The image metadata includes ID, type, size, upload time, and structured text. The vector database is used to store feature vectors of images and to establish indexes for text image features and non-text image features, respectively. The text feature index is applicable to high-dimensional sparse features, while the visual feature index is applicable to high-dimensional dense features.

7. The multimodal image search method according to claim 1, characterized in that: In the cross-modal retrieval step, the comprehensive score of the text image is composed of the following parts composition: Text semantic similarity: Calculate the cosine similarity between the text feature vector of the retrieved image and the text feature vector of the image in the database; Abstract semantic similarity: Use the Sentence-BERT model to encode the abstract of the retrieved image and the abstract of the database image into vectors and calculate the cosine similarity between the two; Layout visual similarity: Calculate the Euclidean distance between the layout feature vectors of the retrieved image and the database text image; The text semantic similarity is regarded as the first weight, the abstract semantic similarity is regarded as the second weight, and the layout visual similarity is regarded as the third weight. For text images, the text semantic similarity, abstract semantic similarity and layout visual similarity are integrated and combined with their respective weights to calculate the comprehensive score of each search result, and the search results are arranged in descending order according to the comprehensive score.

8. The multimodal image search method according to claim 1, characterized in that: In the cross-modal retrieval step, for non-text images, the cosine similarity between the query visual feature vector and the database visual feature vector is directly calculated, and the K-nearest neighbor algorithm is used for retrieval. The retrieval results are sorted in descending order according to the cosine similarity of the visual feature vector, and the top n images with the highest similarity are returned, where n is a preset number.

9. The multimodal image search method according to claim 1, wherein: After the cross-modal retrieval step, the retrieval results are integrated and displayed: For text image retrieval results, they are sorted in descending order according to the comprehensive score; for non-text image retrieval results, they are sorted in descending order according to the cosine similarity of the visual feature vectors; For text images, the overlapping keywords between the search summary and the hit summary are highlighted, and the document layout matching areas are marked; for non-text images, the key visual areas of the search match are highlighted by overlaying heat maps.

10. The multimodal image search method according to claim 1, wherein: In the cross-modal retrieval step, the cross-modal association step specifically includes: implementing cross-modal association retrieval of text images and non-text images by designing a unified retrieval interface.

Images